Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

co19 spawnUri and spawn tests broken #15974

Open
vsmenon opened this issue Jan 8, 2014 · 10 comments
Open

co19 spawnUri and spawn tests broken #15974

vsmenon opened this issue Jan 8, 2014 · 10 comments
Labels
area-test Cross-cutting test issues (use area- labels for specific failures; not used for package:test). gardening library-async P2 A bug or feature request we're likely to work on type-bug Incorrect behavior (everything from a crash to more subtle misbehavior)

Comments

@vsmenon
Copy link
Member

vsmenon commented Jan 8, 2014

The following tests are running very slowly on our test bots (e.g., 1 min each on the dartium-linux bots). I am marking them skipped for Dartium and the standalone VM as they significantly impact the overall bot cycle times.

LibTest/isolate/Isolate/spawnUri_A01_t02
LibTest/isolate/Isolate/spawnUri_A01_t03
LibTest/isolate/Isolate/spawnUri_A02_t02
LibTest/isolate/Isolate/spawnUri_A02_t03
LibTest/isolate/Isolate/spawnUri_A02_t04
@iposva-google
Copy link
Contributor

Slow == forever in this case. The error from the isolate is not forwarded to the spawning isolate. There is a mismatch in expectations about the future completing without errors.

@lrhn
Copy link
Member

lrhn commented May 5, 2014

Added Library-Async label.

@alan-knight
Copy link
Contributor

Looking at this, these tests seem broken.

Normally, if you use extra files, the test framework expects them to be listed with a // OtherScripts=
comment. These tests don't have that. They seem to work anyway in the VM, but in Dartium or in dart2js the A01_t01 test, which is the only one not suppressed, passes even if I change the spawned Uri to gibberish and the Expect to always be false. I'm not sure why it doesn't just time out, as it doesn't seem like it can call asyncEnd without going through a failing Expect. It may be that the asyncStart/End is not working correctly either.

The other tests seem to be testing for an arbitrary exception, so anything that goes wrong (including file not found) counts as passing for the tests that are verifying for errors.

cc @whesse.
Removed Priority-Unassigned, Area-Library labels.
Added Priority-Medium, Area-Test labels.
Changed the title to: "co19 spawnUri tests broken".

@alan-knight
Copy link
Contributor

Oops, the reason A01_t01 was passing on Dartium and dart2js was because it was listed as Fail there.

@alan-knight
Copy link
Contributor

Issue #13992 has been merged into this issue.

cc @ricowind.
cc @floitschG.
cc @munificent.

@whesse
Copy link
Member

whesse commented Mar 9, 2015

To clarify the status of these tests: They aren't copying the additional files they need to the generated test directories. To fix this, requires filing a co19 bug.
It looks like the OtherScripts= comments are not enabled for co19 though, so there may be a fix in test infrastructure needed.

@vsmenon vsmenon added Type-Defect area-test Cross-cutting test issues (use area- labels for specific failures; not used for package:test). library-async labels Mar 9, 2015
@kevmoo kevmoo added P2 A bug or feature request we're likely to work on type-bug Incorrect behavior (everything from a crash to more subtle misbehavior) and removed triaged labels Mar 1, 2016
@whesse
Copy link
Member

whesse commented Aug 29, 2016

Here are some more tests that are timing out on the VM, but not on dart2js:
co19/LibTest/isolate/Isolate/spawn_A03_t02: Timeout
co19/LibTest/isolate/Isolate/spawn_A04_t01: Timeout
co19/LibTest/isolate/Isolate/spawn_A04_t04: Timeout
co19/LibTest/isolate/Isolate/spawn_A06_t03: Timeout
co19/LibTest/isolate/Isolate/spawn_A06_t05: Timeout

Skipping them in the co19-runtime.status file.

@dcharkes
Copy link
Contributor

dcharkes commented Jan 4, 2019 •
edited

2019 update:

$ git grep 'spawnUri_'
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t01: Fail # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t02: Fail # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t03: Fail # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t04: Fail # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t05: Fail # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t06: RuntimeError # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t07: RuntimeError # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A02_t01: RuntimeError, Pass # Dart issue 15617
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t01: Skip # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t02: Skip # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A01_t03: Skip # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A02_t01: RuntimeError # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A02_t02: Skip # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A02_t03: Skip # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawnUri_A02_t04: Skip # Please triage this failure.
tests/co19_2/co19_2-kernel.status:LibTest/isolate/Isolate/spawnUri_A01_t02: RuntimeError
tests/co19_2/co19_2-kernel.status:LibTest/isolate/Isolate/spawnUri_A05_t05: RuntimeError
tests/co19_2/co19_2-kernel.status:LibTest/isolate/Isolate/spawnUri_A05_t04: RuntimeError, Pass

$ git grep 'spawn_A'
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A02_t02: RuntimeError, Pass # Dart issue 15617
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t01: RuntimeError # Dart issue 15974
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A06_t06: Skip # Times out. Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t04: Fail # Issue 27558
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t04: Fail # Issue 27558
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A03_t01: RuntimeError # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A03_t03: RuntimeError # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A03_t04: RuntimeError # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t02: RuntimeError # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t03: Fail # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t05: Fail # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A06_t02: Fail # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A06_t03: Fail # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A06_t04: RuntimeError # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A06_t05: RuntimeError # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A06_t07: Fail # Please triage this failure
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A03_t01: Pass # co19 issue 77
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A04_t02: RuntimeError # Please triage this failure.
pkg/status_file/test/data/co19-dart2js.status:LibTest/isolate/Isolate/spawn_A01_t05: Skip # Times out. Please triage this failure
tests/co19_2/co19_2-kernel.status:LibTest/isolate/Isolate/spawn_A06_t03: Crash, Pass
tests/co19_2/co19_2-kernel.status:LibTest/isolate/Isolate/spawn_A04_t04: RuntimeError, Pass

Build configurations for reliably failing LibTest/isolate/Isolate/spawnUri_* and LibTest/isolate/Isolate/spawn_* tests:

[ $compiler == dart2js ]
LibTest/isolate/Isolate/spawnUri_A01_t01: Fail # Dart issue 15974
LibTest/isolate/Isolate/spawnUri_A01_t02: Fail # Dart issue 15974
LibTest/isolate/Isolate/spawnUri_A01_t03: Fail # Dart issue 15974
LibTest/isolate/Isolate/spawnUri_A01_t04: Fail # Dart issue 15974
LibTest/isolate/Isolate/spawnUri_A01_t05: Fail # Dart issue 15974
LibTest/isolate/Isolate/spawnUri_A01_t06: RuntimeError # Dart issue 15974
LibTest/isolate/Isolate/spawnUri_A01_t07: RuntimeError # Dart issue 15974
LibTest/isolate/Isolate/spawn_A02_t02: RuntimeError, Pass # Dart issue 15617
LibTest/isolate/Isolate/spawn_A04_t01: RuntimeError # Dart issue 15974

[ $compiler == dart2js && $runtime == chromeOnAndroid ]
LibTest/isolate/Isolate/spawnUri_A02_t01: RuntimeError # Please triage this failure.

[ $compiler == dart2js && $runtime == jsshell ]
LibTest/isolate/Isolate/spawn_A04_t04: Fail # Issue 27558
LibTest/isolate/Isolate/spawn_A03_t01: RuntimeError # Please triage this failure
LibTest/isolate/Isolate/spawn_A03_t03: RuntimeError # Please triage this failure
LibTest/isolate/Isolate/spawn_A03_t04: RuntimeError # Please triage this failure
LibTest/isolate/Isolate/spawn_A04_t02: RuntimeError # Please triage this failure
LibTest/isolate/Isolate/spawn_A04_t03: Fail # Please triage this failure
LibTest/isolate/Isolate/spawn_A04_t05: Fail # Please triage this failure
LibTest/isolate/Isolate/spawn_A06_t02: Fail # Please triage this failure
LibTest/isolate/Isolate/spawn_A06_t03: Fail # Please triage this failure
LibTest/isolate/Isolate/spawn_A06_t04: RuntimeError # Please triage this failure
LibTest/isolate/Isolate/spawn_A06_t05: RuntimeError # Please triage this failure
LibTest/isolate/Isolate/spawn_A06_t07: Fail # Please triage this failure

[ $compiler == dart2js && $fast_startup && $runtime == d8]
LibTest/isolate/Isolate/spawn_A04_t04: Fail # Issue 27558
LibTest/isolate/Isolate/spawn_A04_t02: RuntimeError # Please triage this failure.

[ $compiler == dartkp ]
LibTest/isolate/Isolate/spawnUri_A01_t02: RuntimeError
LibTest/isolate/Isolate/spawnUri_A05_t05: RuntimeError

Build configurations for flaky LibTest/isolate/Isolate/spawnUri_* and LibTest/isolate/Isolate/spawn_* tests:

[ $compiler == dart2js ]
LibTest/isolate/Isolate/spawnUri_A02_t01: RuntimeError, Pass # Dart issue 15617

[ $strong && ($compiler == dartk || $compiler == dartkb) ]
LibTest/isolate/Isolate/spawnUri_A05_t04: RuntimeError, Pass
LibTest/isolate/Isolate/spawn_A04_t04: RuntimeError, Pass

[ $runtime == vm && $system == linux && ($compiler == dartk || $compiler == dartkb) ]
LibTest/isolate/Isolate/spawn_A06_t03: Crash, Pass

Build configurations for reliably failing or flaky (skipped) LibTest/isolate/Isolate/spawnUri_* and LibTest/isolate/Isolate/spawn_* tests:

[ $compiler == dart2js ]
LibTest/isolate/Isolate/spawn_A06_t06: Skip # Times out. Please triage this 

[ $compiler == dart2js && $runtime == chromeOnAndroid ]
LibTest/isolate/Isolate/spawnUri_A01_t01: Skip # Please triage this failure.
LibTest/isolate/Isolate/spawnUri_A01_t02: Skip # Please triage this failure.
LibTest/isolate/Isolate/spawnUri_A01_t03: Skip # Please triage this failure.
LibTest/isolate/Isolate/spawnUri_A02_t02: Skip # Please triage this failure.
LibTest/isolate/Isolate/spawnUri_A02_t03: Skip # Please triage this failure.
LibTest/isolate/Isolate/spawnUri_A02_t04: Skip # Please triage this failure.

[ $compiler == dart2js && $runtime == chrome && $system == macos ]
LibTest/isolate/Isolate/spawn_A01_t05: Skip # Times out. Please triage this failure

@dcharkes dcharkes mentioned this issue Jan 4, 2019
Closed
@dcharkes dcharkes changed the title co19 spawnUri tests broken co19 spawnUri and spawn tests broken Jan 4, 2019
@sigmundch
Copy link
Member

Couple notes:

  • .status files under pkg/status_file/ are only test data of the status-file logic and not used by the bots.
  • the isolate library is no longer supported on the web, so it is intended to fail on dart2js (we should probably just skip them instead)
  • dart2js is now using the new bot workflow and doesn't have a .status entries for co19 tests (except some skip sections).

@dcharkes
Copy link
Contributor

dcharkes commented Jan 7, 2019

@sigmundch Thanks!

This leaves two reliably failing LibTest/isolate/Isolate/spawnUri_* vm tests:

[ $compiler == dartkp ]
LibTest/isolate/Isolate/spawnUri_A01_t02: RuntimeError
LibTest/isolate/Isolate/spawnUri_A05_t05: RuntimeError

And one flaky LibTest/isolate/Isolate/spawnUri_* and two flaky LibTest/isolate/Isolate/spawn_* vm tests:

[ $strong && ($compiler == dartk || $compiler == dartkb) ]
LibTest/isolate/Isolate/spawnUri_A05_t04: RuntimeError, Pass
LibTest/isolate/Isolate/spawn_A04_t04: RuntimeError, Pass

[ $runtime == vm && $system == linux && ($compiler == dartk || $compiler == dartkb) ]
LibTest/isolate/Isolate/spawn_A06_t03: Crash, Pass

@crelier crelier mentioned this issue Aug 27, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-test Cross-cutting test issues (use area- labels for specific failures; not used for package:test). gardening library-async P2 A bug or feature request we're likely to work on type-bug Incorrect behavior (everything from a crash to more subtle misbehavior)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@kevmoo @sigmundch @vsmenon @alan-knight @dcharkes @lrhn @whesse @iposva-google