Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request: Marking a test as "expected to fail" #20036

Closed
sethladd opened this issue Jul 14, 2014 · 6 comments
Closed

Request: Marking a test as "expected to fail" #20036

sethladd opened this issue Jul 14, 2014 · 6 comments
Labels
area-pkg Used for miscellaneous pkg/ packages not associated with specific area- teams. P3 A lower priority bug or feature request type-enhancement A request for a change that isn't a bug

Comments

@sethladd
Copy link
Contributor

We're looking at doing more automated test runs, using only off-the-shelf testing infrastructure. We'd like to allow developers to mark a test as "known to fail", so users aren't tracking down false negatives.

@sethladd
Copy link
Contributor Author

@DartBot
Copy link

DartBot commented Jul 14, 2014

This comment was originally written by @zoechi


isn't that what throwsA is about?

expect(() => throw 'some exception', throwsA(new isInstanceOf<String>()));

(there is also throws but I failed earlier this day in understanding how to use it and therefore sticked with throwsA which fits perfectly but I didn't care what type is actually thrown)

@sethladd
Copy link
Contributor Author

After looking into more, it's not exactly throwsA, it's "I expect this test to fail".

Bob knows more about the use case.


cc @munificent.

@srawlins
Copy link
Member

srawlins commented Oct 1, 2014

+1

The concept is the same as the status files in Dart's repo: In your project, a user reports a bug. You wish to track the bug and add a test case for it immediately, but until you've written a fix, you wish to mark the test as failing. This should be 2 new methods:

* failing_test(String spec, TestFunction body)

  • failing_group(String description, void body())

(akin to the other oddly underscored skip_test, solo_test skip_group, solo_group)

Running the unit tests would run all tests defined with failing_test, and would anticipate them to fail. I think this would then yield a final output of something like:

PASS: Something here
PASS: Something else here
ANTICIPATED FAIL: Some not-yet-fixed bug

2 tests passed. 1 test anticipated to fail indeed failed.

It's important to run the test: if at some point the test starts to succeed, the developer surely wants to know this, and investigate! Thus:

PASS: Something here
PASS: Something else here
UNANTICIPATED PASS: Some not-yet-fixed bug

2 PASSED, 1 UNANTICIPATED PASS, 0 FAILED, 0 ERRORS

I think this helps with Seth's use case: If I want to use package foo, I could (in the future) look at the test results on pub.dartlang, see "100 tests pass, 3 anticipated failures" and imagine that these are related to open bugs.

I'm happy to implement this.

@sethladd
Copy link
Contributor Author

sethladd commented Oct 1, 2014

cc @nicolasgarnier.

@sethladd sethladd added Type-Enhancement P3 A lower priority bug or feature request area-pkg Used for miscellaneous pkg/ packages not associated with specific area- teams. labels Oct 1, 2014
@DartBot
Copy link

DartBot commented Jun 5, 2015

This issue has been moved to dart-lang/test#254.

@DartBot DartBot closed this as completed Jun 5, 2015
@kevmoo kevmoo added type-enhancement A request for a change that isn't a bug and removed triaged labels Mar 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-pkg Used for miscellaneous pkg/ packages not associated with specific area- teams. P3 A lower priority bug or feature request type-enhancement A request for a change that isn't a bug
Projects
None yet
Development

No branches or pull requests

4 participants