Improve the integration of ctest and our buildbot #368
Comments
|
From bruen...@google.com on April 28, 2011 07:58:40
2,3,4) CTest was chosen because it integrates nicely with CMake. for test suite results, CTest integrates with CDash. I have set up a CDash dashboard before: it was very easy, and it comes with scripts to pull out and display individual results at the configure, build, and test steps. see http://www.corp.google.com/~bruening/cdash-snapshot.jpg for a snapshot of the main dashboard page for the DynamoRIO tests. each item can be clicked on to drill down for more information. CDash can be configured to send email on certain types of failures: e.g., we had it sending on all configure perhaps rather than writing our own scripts to extract ctest results and display them in the current waterfall view we should just install CDash. IMHO I prefer the CDash results view anyway. |
|
From timurrrr@google.com on May 03, 2011 08:59:24 The grep trick from (3) is done in r274 .
Regarding CDash - it will likely require some hacking (or adding more port forwarding exceptions) btw, looks like some opensource projects use both buildbot and CDash: http://liblas.org/development/buildbot.html |
|
From bruen...@google.com on May 03, 2011 10:21:39 what you're calling "ctest" (as taking 10+mins) is really "ctest invoked on runsuite.cmake". runsuite.cmake is executed outside of any pre-existing build. it builds multiple configurations and runs tests for some of them. it is meant to be the pre-commit test suite and with a non-default arg the long/nightly/bot test suite. I only run it when I have a diff that's ready to commit. for normal development, you'd be inside a particular build dir that you've been working with (it could be one built by runsuite, though usually not). from within a build dir, ctest can be used to run some or all tests. it has a number of options ("ctest --help"). most useful are -V (verbose), -R (run tests matching regexp), -E (exclude regexp), -N (don't run just list), -I (run by number).
e.g.:
|
|
From bruen...@google.com on May 03, 2011 10:33:37 I often run something like "ctest -j4" for parallel tests |
|
From timurrrr@google.com on May 04, 2011 05:02:43 Recent Buildbot improvements:
|
|
From timurrrr@google.com on May 04, 2011 05:03:12 Discussed via e-mail: |
From timurrrr@google.com on April 28, 2011 06:20:12
Current problems:
All the builds are actually red and we're not quite aware of this.
We should try hard to make "ctest" run green on all platforms. Otherwise buildbot usage
is suboptimal. As a temporary solution we should
the same way (I'm talking about "4/5 leaks" Win version
differences on suppress tests),
{ctest = "build" + "run all tests"} is run as a single step.
as "ctest -VV -S /tests/runsuite.cmake,drmemory_only".
This doesn't fit well into the buildbot philosophy.
It would be better if we could separate "build" and "test" into separate commands,
like "make ALL_BUILD" and "make test" (or .sh/.bat scripts or whatever).
I believe this would also be very valuable for local testing.
Maybe this is already there but I don't know how to run it separately -
(I know "ctest run" but it didn't work quite well for me. Also, is there "ctest build"?)
Derek, can you please comment on this?
If possible, ctest should return non-zero code in case of any test failures.
If this is not possible - I can grep the logs and set non-zero code on the buildbot side
but it looks like a hack.
Individual test failures from "ctest" should be shown on the waterfall / build page.
Currently we only see an orange box (= no errors, only warnings)
with "stdio" link (leads to full ctest log - too much info and requires ctest knowledge)
and "warnings" link (leads to VS build warnings, see issue Visual Studio build warnings while building Dr. Memory #367 )
For example,
grep "Test project\|\*\*\*Failed" \<full ctest log>gives much morebuildbot-friendly information (and user- too!), e.g.
Test project E:/b/build/slave/win_xp_drm_checkout/build/build_drmemory-dbg-32
5/20 Test
#5: cs2bug ..........................._Failed 7.61 sec20/20 Test
#20: nudge ............................_Failed 18.16 secTest project E:/b/build/slave/win_xp_drm_checkout/build/build_drmemory-rel-32
5/20 Test
#5: cs2bug ..........................._Failed 2.67 sec18/20 Test
#20: nudge ............................_Failed 1.48 secTest project E:/b/build/slave/win_xp_drm_checkout/build/build_drmemory-cyg-dbg-32
5/18 Test
#5: cs2bug ..........................._Failed 6.56 sec15/18 Test
#12: suppress ........................._Failed 120.07 sec16/18 Test
#13: suppress-genoffs ................._Failed 123.38 sec17/18 Test
#14: suppress-gensyms ................._Failed 123.27 sec18/18 Test
#18: nudge ............................***Failed 204.63 sec<- this is easy to read. And I can easily extract the list of test failures for each test failure.
Derek, (how?) is it possible to get individual test logs from CTest so we easily access them
from the buildbot, without downloading logs as a .7z file (as we do now)
Don't worry, I don't advocate to abandon it :) just out of curiosity and for my personal
"recipe of success for a new opensource project".
If (just imagine) you were starting from scratch - would you still use ctest?
Looks like many opensource projects tend to use buildbot, not ctest http://wiki.winehq.org/UnitTestSuites
Original issue: http://code.google.com/p/drmemory/issues/detail?id=368
The text was updated successfully, but these errors were encountered: