bbreport should be able to recognize suspicious failures, like: 1) new failures; 2) intermittent failures; 3) solved failures; 4) detect the branches/platforms affected; 5) detect the revision that introduced the problem; 6) detect tests that fail together and might be related;

It should either have a "smart" flag or print by default at the end of the normal report another report like: """ After r79000 test_foo started failed on Windows machines on trunk. There are still intermittent failures in test_bar and test_baz on Freebsd. test_xyz doesn't fail anymore after r79000. """

Another suggestion: associate a failure to an issue. I don't know how, but it would be nice to match a failure using different patterns and to associate it to an issue number. Then we will be able to see failures without associated issue and ignore (hide) known issues. The detection doesn't have to be perfect, it's just an indicator.

Examples: - trunk and py3k are currently affected by test_ssl and test_ftplib failures on a specific builder (a specific version of OpenSSL) => http://bugs.python.org/issue8108 - trunk and py3k sometimes hung (1800 seconds timeout) on test_subprocess or test_multiprocessing => issue #8429 and issue #8430 - test_xmlrpc failed on a specific builder (antoine's non-ascii buildbot) => #7606 - etc.

So we should be able to match on: - failing test name - buildslave name - failure type (test failed, hung, ...)

In a prototype, a config file edied manually would be enough. Something like:

issues = { 8429: {'test': 'test_multiprocessing', 'status': 'hung'}, 8430: {'test': 'test_subprocess', 'status': 'hung'}, 8108: {'test': 'test_ftplib', 'buildslaves': ('alpha Debian 3.x', 'alpha Debian trunk'}, 7606: {'test': 'test_xmlrpc', 'buildslaves: ('AMD64 Ubuntu 3.x',), }

It should be quite easy to associate it to the revision where it started failing, and possibly retrieve the commit message for that revision (or even extract an issue number from there). I don't see any easy way to find other issues opened to track failures. We can do it manually, but IMHO is not worth doing it and update the config file every time a new issue is opened or closed or when the test is fixed on a particular buildbot.

It's wasn't difficult to associate an issue to a failed test :-) Here is an implementation.