Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handling flaky tests is time consuming #12966

Closed
peter-ahe-google opened this issue Sep 1, 2013 · 6 comments
Closed

Handling flaky tests is time consuming #12966

peter-ahe-google opened this issue Sep 1, 2013 · 6 comments
Labels
area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. P2 A bug or feature request we're likely to work on type-bug Incorrect behavior (everything from a crash to more subtle misbehavior)

Comments

@peter-ahe-google
Copy link
Contributor

Rico has several times stated that he wants the Dart team to follow a certain procedure when they see a flaky test.

This weekend, I have tried handling all flaky tests I saw when submitting r26974. However, due to the long cycle times of dartium builders, I was not able to complete this in one day.

I agree that we need to do something about flaky tests, however, it is not reasonable to have a procedure which only works if you dedicate a weekend to follow it.

My suggestion is to place reasonable time requirements on non-FYI builders: how long may a status-only build cycle take, how long is the expected build time on an average CL which requires a rebuild.
@ricowind
Copy link
Contributor

The cycle time of the dartium inc builders have been greatly improved:
  Better incremental builds + windows ninja
  Using the browser controller for running the tests in dartium

We still need to shard this across more vms to get really fast cycle times, but we are at least <30 mins now, tentatively assigning to Martin, please figure out if you or bill will do the new dartium sharding when he is back

cc @whesse.
Set owner to @mkustermann.
Added this to the 1.2 milestone.
Removed Priority-Unassigned, Area-Test labels.
Added Priority-Medium, Area-Infrastructure labels.

@mkustermann
Copy link
Member

Dartium inc builders are now at 22 minutes (when we don't have to compile anything). I think as long as we don't have tests timing out, we can run all tests on dartium-inc and don't need to shard here (they are faster than dart2js builders).

It's different for debug builders: They are extremely slow and in addition to that, they execute also DRT tests (the full builders do this as well). We could shard these, but they are pretty flaky anyway. I suspect, Peter would say we should move the debug ones to FYI.

Unassigning me for now, since I don't plan to work on this anytime soon.

Removed the owner.

@kasperl
Copy link

kasperl commented Jun 4, 2014

Removed this from the 1.2 milestone.
Added this to the 1.6 milestone.

@kasperl
Copy link

kasperl commented Jul 10, 2014

Removed this from the 1.6 milestone.
Added Oldschool-Milestone-1.6 label.

@kasperl
Copy link

kasperl commented Aug 4, 2014

Removed Oldschool-Milestone-1.6 label.

@peter-ahe-google peter-ahe-google added Type-Defect area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. labels Aug 4, 2014
@kevmoo kevmoo added P2 A bug or feature request we're likely to work on type-bug Incorrect behavior (everything from a crash to more subtle misbehavior) and removed triaged labels Feb 29, 2016
@bkonyi
Copy link
Contributor

bkonyi commented Jun 27, 2018

Closing due to inactivity.

@bkonyi bkonyi closed this as completed Jun 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. P2 A bug or feature request we're likely to work on type-bug Incorrect behavior (everything from a crash to more subtle misbehavior)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@kevmoo @kasperl @ricowind @peter-ahe-google @mkustermann @bkonyi