Introduction

The Automated Testing Framework (ATF) package is a collection of libraries to simplify the creation of test programs in a variety of languages. The supported languages include C, C++ and POSIX shell, which are respectively offered by the atf-c, atf-c++ and atf-sh components. Test programs written using these libraries all expose a unified command-line interface (CLI). This common CLI permits the Kyua runtime system to run all the test cases in a controlled and automated manner.

• Introduction
• Status of the ATF project
• Downloads
• Documentation
• Mailing lists
• Source repository
• Frequently asked questions
• Why are test programs installed?
• How do I run a test suite?
• How do I generate machine-parseable test results?
• How do I write a C test program?
• How do I write a C++ test program?
• How do I write a shell test program?
• How do I make atf-run recognize my test programs?

Status of the ATF project

The ATF project started in 2007 as part of the Google Summer of Code program. Initially, ATF included both the libraries described above and the set of tools to permit the execution of such tests. Since November 2010, the tool components of ATF have been rewritten under the Kyua name (but not the libraries yet) due to a variety of reasons.

The current status of the ATF components is as follows:

Up until February 2012, ATF had been hosted in a very adhoc manner: its web site and download files were located in jmmv's private space of the NetBSD web/FTP server, and its source code was kept in a Monotone repository hosted by some personal servers. These facts hindered development due to the need of maintaining a custom infrastructure, and also made it very hard for external contributors to contact the project due to the rarity of the hosting platform. As a result, on February 26th, 2012, ATF's web site was torn down and replaced by this page, and the Monotone repository was migrated to Git and put in this hosting platform.

Downloads

The latest formal release is 0.15 and was released on January 16th, 2012.

ATF formal releases can be found in Kyua's downloads page along with the other components of Kyua. Every release file is annotated with the list of major changes in that particular release, although you can also see the NEWS file for a unified list of changes.

Documentation

The main source of documentation of ATF is in the form of manual pages. See the atf(7) manual page after installation for a quick reference of all other available pages.

You can also find some valuable information in the Creating atf-based tests for NetBSD src tutorial.

Mailing lists

The official mailing list for ATF-specific discussions still lives in the NetBSD mailing lists site and is appropriately named atf-devel. Please follow the instructions in that page to subscribe to this mailing list.

Source repository

Since ATF's creation, the source code had been maintained in a Monotone repository hosted by a minor unknown provider. The use of such an uncommon tool was a constant source of trouble to potential contributors, as the entry barrier to downloading the repository was pretty high. As such, the repository was migrated to Git on February 26th, 2012, and can now be found in the atf module of Kyua's repository.

Frequently asked questions

Why are test programs installed?

One of the major features of ATF is that it encourages test programs to be installed on the live system as part of the installation package. The reason for this is to allow the end user to run the test suite of the program by himself, which will result in better confidence from his side that the program is working as intended.

I have sometimes heard claims that running test programs is the task of the Quality Assurance (QA) department. Unfortuantely, most open source projects cannot afford having people dedicated to such tasks. And, even if they do, they cannot test the program under all possible scenarios. These scenarios include infinite hardware combinations and infinite combinations of the dependencies of the program. By allowing the user to run the tests, and by encouraging him to run them periodically, he can make sure that things do not break unexpectedly across e.g. daily system upgrades or when the hardware undergoes maintenance.

How do I run a test suite?

To run a test suite (or just a collection of test programs), go to the directory where the test programs live in and execute the following command:

$ atf-run | atf-report

You can recognize that a directory contains a test suite by checking whether it contains an Atffile file or not.

How do I generate machine-parseable test results?

Test cases can write their actual results (i.e. whether they passed, failed or were skipped, alongside with any associated information) in a machine-parseable format. Use the -r option of the containing test program to specify where to store such output. For example:

$ ./t_env -r results-file test_env_has

Similary, atf-run generates, by default, a machine-parseable output format. You can use atf-report to convert the output of atf-run into XML if this is deemed more appropriate. For example:

$ atf-run >raw-output.txt
$ atf-report -o xml:output.xml

See atf-formats(4) for more details.

How do I write a C test program?

Use the following template:

#include <atf-c.h>

ATF_TC(my_test_case, tc);
ATF_TC_HEAD(my_test_case, tc)
{
    atf_tc_set_md_var(tc, "descr", "This test case ensures that...");
}
ATF_TC_BODY(my_test_case, tc)
{
    ATF_CHECK(returns_a_boolean()); /* Non-fatal test. */
    ATF_REQUIRE(returns_a_boolean()); /* Fatal test. */

    ATF_CHECK_EQ(4, 2 + 2); /* Non-fatal test. */
    ATF_REQUIRE_EQ(4, 2 + 2); /* Fatal test. */

    if (!condition)
        atf_tc_fail("Condition not met!"); /* Explicit failure. */
}

ATF_TP_ADD_TCS(tp)
{
    ATF_TP_ADD_TC(tp, my_test_case);
}

Define the following variables:

$ CFLAGS="$(pkg-config --cflags atf-c)"
$ LDFLAGS="$(pkg-config --libs-only-L --libs-only-other atf-c)"
$ LIBS="$(pkg-config --libs-only-l atf-c)"

And build your test program with the following commands:

$ gcc ${CFLAGS} -o sample_test.o -c sample_test.c
$ gcc ${LDFLAGS} -o sample_test sample_test.o ${LIBS}

How do I write a C++ test program?

Use the following template:

#include <atf-c++.hpp>

ATF_TEST_CASE(my_test_case);
ATF_TEST_CASE_HEAD(my_test_case)
{
    set_md_var("descr", "This test case ensures that...");
}
ATF_TEST_CASE_BODY(my_test_case)
{
    ATF_CHECK(returns_a_boolean());
    ATF_CHECK_EQUAL(4, 2 + 2);

    if (!condition)
        ATF_FAIL("Condition not met!"); /* Explicit failure. */
}

ATF_INIT_TEST_CASES(tcs)
{
    ATF_ADD_TEST_CASE(tcs, my_test_case);
}

Define the following variables:

$ CXXFLAGS="$(pkg-config --cflags atf-c++)"
$ LDFLAGS="$(pkg-config --libs-only-L --libs-only-other atf-c++)"
$ LIBS="$(pkg-config --libs-only-l atf-c++)"

And build your test program with the following commands:

$ g++ ${CXXFLAGS} -o sample_test.o -c sample_test.cpp
$ g++ ${LDFLAGS} -o sample_test sample_test.o ${LIBS}

How do I write a shell test program?

Use the following template:

#! /usr/bin/env atf-sh

atf_test_case my_test_case
my_test_case_head() {
    atf_set "descr" "This test case ensures that..."
}
my_test_case_body() {
    touch file1 file2

    cat >expout <<EOF
file1
file2
EOF
    atf_check -s eq:0 -o file:expout -e empty 'ls'

    atf_check_equal 4 $((2 + 2))

    if [ 'a' != 'b' ]; then
        atf_fail "Condition not met!"  # Explicit failure.
    fi
}

atf_init_test_cases() {
    atf_add_test_case my_test_case
}

And then turn it into an executable:

$ chmod +x sample.sh

How do I make atf-run recognize my test programs?

Create an Atffile in the directory where your programs list and list the name of your programs:

Content-Type: application/X-atf-atffile; version="1"

prop: test-suite = "examples"

tp: sample1_test
tp: sample2_test

Alternative, use a wildcard:

Content-Type: application/X-atf-atffile; version="1"

prop: test-suite = "examples"

tp-glob: *