Introduction

This is a rebaselining tool that automatically produces baselines using the layout test results from buildbot for all platforms (win, linux and mac). The goal is that after running this tool, the working copy will be ready to create a change containing all new baselines for all three platforms.

The tool is written in Python and live in webkit/tools/layout_tests.

How does this tool work?

The script does the following for each platform:

• Compile a list of tests that need rebaseline.
• Download test result archive from buildbot for the platform.
• Extract baselines from the archive file for all identified files.
• Add new baselines to SVN repository. For GIT repository, you need to manually "git add" all the new baseline files.
• For each test that has been rebaselined, remove this platform option from the test in test_expectation.txt. If no other platform options remaining after removal, delete the test from the file.

After new baselines are generated, the tool launches a html page to compare the old and new baselines. The html can be disabled by option "--no_html_results".

How to use it?

• Make sure the test files (.html) exist at src\webkit\data\layout_tests in repository. If not, add the html files there and commit them. By default, the rebaselining tool retrieves new baselines from the release version of buildbot layout test archive. If you add a new test, you need to wait for buildbot to run layout tests and the actual result for the test has been archived before you can rebaseline it using this tool. Check the RELEASE version of buildbot layout archive to see whether your new test is available.
• Update test_expectations.txt at src\webkit\tools\layout_tests: add option REBASELINE to the tests that need to be rebaselined.
• Run rebaselining script: rebaseline.bat for Windows and rebaseline.sh for Linux and Mac. The script can be found at src\webkit\tools\layout_tests. If you want to pull new baselines from webkit.org canary, run rebaseline.bat or rebaseline.sh with option --webkit_canary.
• If the script runs successfully, new baselines (expected files) are added to SVN repository and the rebaselined tests are removed from test_expectations.txt.
• The tool launches a html page to compare the old and new baselines. Check the new baseline changes are expected.
• Prepare a patch with the new baselines and test_expectations.txt, review the changes and then commit the patch.

Examples

We need rebaseline test LayoutTests/foo1.html for all three platforms and test LayoutTests/foo2.html for win and linux.

• Make sure test files foo1.html and foo2.html have been committed at src\webkit\data\layout_tests\LayoutTests. If these are tests just added, make sure the layout tests ran at buildbot and the test results have been archived.
• Update test_expectations.txt to include the following lines:

BUG1234 REBASELINE : LayoutTests/foo1.html = FAIL
BUG5678 REBASELINE WIN LINUX : LayoutTests/foo2.html = FAIL

If foo1 and foo2 already exist, add option "REBASELINE" to the existing tests.

• Run rebaseline script on Windows machine:

  open a command window
  change the directory to [YOUR CHROMUIM DIRECTORY]\src\webkit\tools\layout_tests
  rebaseline.bat

• Run rebaseline script on Linux or Mac machine

  open a shell
  change the directory to [YOUR CHROMUIM DIRECTORY]/src/webkit/tools/layout_tests
  ./rebaseline.sh

NOTE: you only need to run the script on ONE platform. The tool will auto generate new baselines for ALL three platforms.

• Check tests foo1 and foo2 are removed from test_expectations.txt.

• Create a patch

gcl change [YOUR PATCH NAME]

Review the changes and then commit it. Don't check in with any tests still marked REBASELINE, as this will prevent others from cleanly using the tool.

• The tool launches a html to compare new baselines with old baselines. Here is an example of the baseline result comparison html:

Baseline Result Comparison HTML

Notes to GIT user

If you are using GIT repository, you need to manually "git add" all the new baseline files before creating patch.