Introduction

One of the major goals of the Kyua project is to replace the Automated Testing Framework (ATF) with a more modular and robust implementation. This is a worthy goal, but a complete reimplementation of ATF involves a lot of work and, if there is no clear transition plan and if such transition does not happen incrementally, the rewrite will fail and will frustrate users.

The purpose of this document is to describe how the implementation will shift from ATF to Kyua and how users should see these changes.

Tentative schedule

Developing Kyua from the ground up is a huge engineering effort. The following table provides a list of independent targets and the tentative schedule for their readiness; the entries are sorted by the planned release date:

As mentioned, these components are independent and each will replace a small part of ATF. The components not touched by a component will continue to be redistributed and branded as ATF, so you can be confident in relying on them. When the replacement is ready for release, the code in ATF of the replaced components will not disappear overnight.

You should join the kyua-discuss mailing list to keep informed about the development process of Kyua and to get information about the transition procedure when the new releases happen.

Transition plan

The key idea of the transition is to replace the components in ATF one at a time. All the components rewritten in Kyua should slowly vanish from the ATF codebase. Kyua and ATF will be installable side by side and Kyua will remain compatible with ATF thorough the development process of Kyua. Incompatibilities will be introduced later once the transition has happened.

Trimming down atf

Kyua will depend on ATF until atf-c++ is replaced by kyua-c++. Until that happens, Kyua will have a quite heavy dependency (no generally-available packages, hard to install from source) that will scare users away.

In order to mitigate this problem, the ATF distribution will be trimmed down as its components are rewritten under Kyua. This will reduce the size of the legacy code (simplifying portability and building in other systems) and will also help in "forcing" users to migrate to the new tools.

Such removal of legacy code will not happen right away though to prevent breakage in user procedures. See the next sections for details.

Lastly, when a replacement for an ATF component in Kyua is in the works, the ATF version will enter maintenance mode. This means that:

• No new major features will be introduced.
• No code cleanups/refactorings.
• New features might be added if there is real/immediate pressure for them (as has happened in the past), but the implementation may be a ugly ad-hoc hack.
• Bug fixes will be allowed.

Replacing the command-line tools

The atf-run and atf-report pipeline is the part of ATF that receives most complaints from users. These complaints are generally in the form of usability annoyances, but are also feature requests that are very hard to implement with the current design of ATF. Therefore, the rewrite in Kyua targets these at first:

1. Implement a minimal kyua CLI tool to replace atf-run and atf-report.
• The goal is a reliable runtime engine: should tolerate all predictable test program crashes (and more!).
• Do not bother to provide nice reporting at this stage.
• The baseline goal is the ability to self-test.
2. Change atf-run and atf-report to display a deprecation warning and to point users to Kyua.
• Publish a new ATF formal release at this point.
• Maybe make atf-run and atf-report delegate on Kyua if the kyua binary is present.
3. Improve the kyua CLI tool:
• Provide proper generation of reports.
• Add a migration script to convert Atffile files to Kyuafile files (once we decide on a new file format).
4. Remove atf-run and atf-report from the distribution.
• Publish a new ATF formal release at this point.

Replacing the C library

This is what BSD developers care about, so rewriting this binding should be a high priority.

1. Implement the kyua-c library to provide a C binding for test programs.
• Must be much more transparent than atf-c. In particular, avoid using macros wherever possible.
• Redesign the CLI of test programs. The current CLI is confusing because some flags change the whole behavior of the test program.
2. Change atf-c to become a simple wrapper library over kyua-c and add deprecation warnings.
• Publish a new ATF formal release at this point.
3. Work with users of atf-c to migrate:
• This means I convert the NetBSD tests.
• ... but also need to talk to FreeBSD to do the same.
4. Kill atf-c from the distribution.
• Publish a new ATF formal release at this point.

Replacing the C++ library

1. Implement the kyua-c++ library to provide a C++ binding for test programs.
• Must be much more transparent than atf-c. In particular, avoid using macros wherever possible.
2. Change atf-c++ to become a simple wrapper library over kyua-c++ and add deprecation warnings.
• Publish a new ATF formal release at this point.
3. Wait some reasonable time; I'm unsure as to how many users of atf-c++ are there.
4. Kill atf-c++ from the distribution.
• Publish a new ATF formal release at this point.

Replacing the shell library

1. Implement the kyua-sh library to provide a shell binding for test programs.
2. Change atf-sh to become a simple wrapper library over kyua-sh and add deprecation warnings.
• Publish a new ATF formal release at this point.
3. Work with users of atf-sh to migrate:
• This means I conver the NetBSD tests.
• ... but also need to talk to FreeBSD to do the same.
4. Strip the atf-sh library and the atf-sh and atf-check binaries from the distribution.
• Publish a new ATF formal release at this point.

End of ATF

Once atf is empty, we can consider Kyua a success.