Philosophy

There have been quite a few python test frameworks right now. So why is thumb.py? Because It just works.

I've tried many python coverage, auto regression test framework, well, actually almost all of them. But most of them requires non-trivial tweaks before you could fit them into your real life project.

thumb.py is designed with two philosophies in mind:

• It Just Works: everything is contained in a single file, you could download it and it's ready to go.
• Nothing is more flexible than modifying the source code: 'Reality is rarely simple and never pure.' Instead of building a one-size-fits-it-all test framework, you could put all your environment setup tasks in one python file and let thumb.py load it for you.

Download

svn checkout http://thumbpy.googlecode.com/svn/trunk/ thumbpy

Features

• test case auto discovery in current folder
• 'nosy' style daemon mode maximize productivity by accelerating edit-test-edit cycle
• Colorful HTML based code coverage report

Usage Examples

1. thumb.py: execute last modified test case
2. thumb.py -f: execute full regression test
3. thumb.py -d: enter fast 'edit-test-edit' mode
4. thumb.py -c or thumb.py -l -c: generate coverage report for last modified test case
5. thumb.py -f -c: generate full regression test coverage report. (Your need a faster CPU to run this. :-)

Command Line options

Usage: thumb.py [options]

Options:
  -h, --help           show this help message and exit
  -d, --daemon         Stay in daemon mode
  -c, --coverage       Generate code coverage reporting
  -f, --full           Execute all test cases as full regression suite
  -l, --last_modified  Execute only test cases found in last modifed test file

Default Setting File

Your could customize your testing environment by having a thumb_conf.py in current folder. You could have as many thumb_conf.py as you want. Thus offering a project level control over the thumb.py behavior.

With a 'from thumb_conf import ', all variables defined in this file will be exposed to the testing environment.

# -------------------------------------------
#    THUMB.PY SETTINGS FILE
# -------------------------------------------
# set `verbosity` level for unittest.TextTestRunner as
# defined in http://docs.python.org/lib/unittest-contents.html. 
# default is 1, which is quite terse.
# set to 2 if you want more details
thumb_verbosity 	    = 1  

# test file patterns 
# use '\/Test.*\.py$' if your test files look like 'TestModels.py'
#  or '\\Test.*\.py$' if you're on windows
thumb_pattern		    = '\/.*_test\.py$'

# list of files you want to ignore from the coverage report
# For example: ['local_conf_sample.py', 'test/test_factory.py']
thumb_ignore_files      = ['local_conf.py', 'manage.py']
thumb_ignore_folders    = ['test',]

# Percentage of pass coverage. Default is 100, which is 100%
thumb_coverage_level    = 100


# -------------------------------------------
#   CUSTOMIZATION STARTS 
# -------------------------------------------
# Your own project specific settings goes here. 
# Here I'm changing django's database settings to 
# make it point to a test database
import settings
settings.DATABASE_NAME      +=  '_test'

Known Problems

I use '\n'.join(...) to work around Python interpreter bug 1184112. But the walk around couldn't handle source code files end with empty lines. So if you see any error raised from parser.suite, try to see which file causes the problem and clean the file up and try again.

Under the Hood of Coverage

Introdcution

The coverage part is based heavily on figleaf project. I actually shameless stole the whole LineGrabber algorithm from there.

Here are a few features I intentionally wrote to make it different from major coverage test frameworks like coverage.py, pycover and figleaf.

1. I will log coverage information only for files in the code repository, which is specified when starting the coverage testing by calling start(include, exclude). The biggest benefit is a huge performance boost from 2 min to 5 sec.
2. Instead of seperating execution log and reporting tools into two pieces, I eliminate all of the intermediate on-disk log files, like *.cover, .figleaf, and pass everything via memory.
3. For the report, instead of the pycover's console result or figleaf's multiple html files, I generated one single html files with outline and detail mode.

How it Works

Here is a brief introduction on how this coverage works.

Collect Execution Result

The key is to use sys.settrace and threading.settrace. In order to collect data only for our source code, there is a global filter that will check the current executed file's name and decide whether to proceed with the result recording.

The real line-by-line recording is done by trace, which is a callback method. Python interpreter will pass the execution info to this method.

The data structure I used to hold the execution result is very simple: {filename: ... }. lineno is sorted list.

Tell whether a line is hit

For any line in source code, there are three possible status: hit, missed or ignored. After the first result collection step, we could know for sure that which line is hit. But we need a little bit magic to tell whether a non-hit line is a comment line, or a dict/list/tuple declaration line, which are not supposed to be hit, or a real code line.

This magic is done via figleaf's wonderful LineGrabber class, which builds itself on top of python's parser package. There are a couple of bugs in the current interpreter implementation that might cause problems. Please refer to known problems section for detail.

Generate HTML report

Please play with a report and look at the HTML code generated. The basic flow is we first generate the styles and scripts, then for each file, we generate each line with current styles. This is quite straight-forward so I'll just shut up.

Credit

To C. Titus Brown for his excellent work in figleaf. My Chinese based photo blog on 好看簿