The GNU Testing Framework January 2006 Rob Savoye Ben Elliston 2006 Free Software Foundation, Inc. 0.6.2 2002-07-16 rob Add new tutorial as a new sect1. 0.6.1 2001-02-16 rob Add info on the new dejagnu.h file. 0.6 2001-02-16 rob Updated for new release. 0.5 2000-01-24 rob Initial version after conversion to DocBook. This document describes the functionality of &dj;, the testing framework of the GNU project. &dj; is written in Expect, which uses Tcl as a command language. Expect acts as a very programmable shell. As with other Unix command shells, you can run any program, but once the program is started, your test script has programmable control over its input and output. This does not just apply to the programs under test; expect can also run any auxiliary program, such as diff or sh, with full control over its input and output. &dj; itself is merely a framework for the creation of testsuites. Testsuites are distributed with each application. &dj; is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool. Each program can have multiple testsuites, all supported by a single test harness. &dj; is written in Expect, which in turn uses Tcl -- Tool command language. There is more information on Tcl at the Tcl/Tk web site and the Expect web site is at NIST. Julia Menapace first coined the term ``&dj;'' to describe an earlier testing framework at Cygnus Support she had written for GDB. When we replaced it with the Expect-based framework, it was like &dj; all over again. More importantly, it was also named after my daughter, Deja Snow Savoye, who was a toddler during &dj;'s beginnings. &dj; offers several advantages for testing: The flexibility and consistency of the &dj; framework make it easy to write tests for any program, with either batch oriented, or interactive programs. &dj; provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for GDB can run from any supported host system on any supported target system. &dj; runs tests on many single board computers, whose operating software ranges from a simple boot monitor to a real-time OS. All tests have the same output format. This makes it easy to integrate testing into other software development processes. &dj;'s output is designed to be parsed by other filtering script and it is also human readable. Using Tcl and Expect, it's easy to create wrappers for existing testsuites. By incorporating existing tests under &dj;, it's easier to have a single set of report analyse programs.. Running tests requires two things: the testing framework and the testsuites themselves. Tests are usually written in Expect using Tcl, but you can also use a Tcl script to run a testsuite that is not based on Expect. Expect script filenames conventionally use .exp as a suffix; for example, the main implementation of the &dj; test driver is in the file runtest.exp.) This release has a number of substantial changes over version 1.3. The most visible change is that the version of Expect and Tcl included in the release are up-to-date with the current stable net releases. The biggest change is years of modifications to the target configuration system, used for cross testing. While this greatly improved cross testing, it has made that subsystem very complicated. The goal is to have this entirely rewritten using iTcl by the next release. Other changes are: More built-in support for building target binaries with the correct linker flags. Currently this only works with GCC as the cross compiler, preferably with a target supported by . Lots of little bug fixes from years of heavy use at Cygnus Solutions. &dj; now uses Automake for Makefile configuration. Updated documentation, now in DocBook XML. Windows support. There is beta level support for Windows that is still a work in progress. This requires the Cygwin POSIX subsystem for Windows. To use &dj; on Windows, you need to first install the Cygwin release. This works as of the B20.1 release. Cygwin is a POSIX system for Windows. This covers both utility programs and a library that adds POSIX system calls to Windows. Among them is pseudo tty support for Windows that emulates the POSIX pty standard. The latest Cygwin is always available from this location. This works well enough to run "make check" of the GNU development tree on Windows after a native build. But the nature of ptys on Windows is still evolving. Your mileage may vary. &dj; grew out of the internal needs of Cygnus Solutions, the company formerly known as Cygnus Support. Cygnus maintained and enhanced a variety of free programs in many different environments and we needed a testing tool that: was useful to developers while fixing bugs; automated running many tests during a software release process; was portable among a variety of host computers; supported cross-development testing; permitted testing interactive programs, like GDB; and permitted testing batch oriented programs, like GCC. Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to make sure that GDB works as well when cross-debugging as it does in a native configuration. Probably the greatest challenge was testing in a cross-development environment. Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communication interfaces vary from a serial line to Ethernet. &dj; was designed with a modular communication setup, so that each kind of communication can be added as required and supported thereafter. Once a communication procedure is coded, any test can use it. Currently &dj; can use rsh, rlogin, telnet, tip, kermit and mondfe for remote communications. &dj; conforms to the POSIX 1003.3 standard for test frameworks. Rob Savoye was a member of that committee. The POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET. TET was created by Unisoft for a consortium comprised of X/Open, Unix International and the Open Software Foundation. The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says ``The sun shall shine'', a corresponding assertion might be ``The sun is shining.'' A test based on this assertion would pass or fail depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written. As there is no testsuite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of POSIX conforming systems and a fifth for a conditional feature. &dj; supports the use of all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case: PASS A test has succeeded. That is, it demonstrated that the assertion is true. XFAIL POSIX 1003.3 does not incorporate the notion of expected failures, so PASS, instead of XPASS, must also be returned for test cases which were expected to fail and did not. This means that PASS is in some sense more ambiguous than if XPASS is also used. FAIL A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework. As with PASS, POSIX tests must return FAIL rather than XFAIL even if a failure was expected. UNRESOLVED A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when writing tests to not carelessly use Tcl commands like return---if you alter the flow of control of the Tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up UNRESOLVED: A test's execution is interrupted. A test does not produce a clear result. This is usually because there was an ERROR from &dj; while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened---and to improve the test case. A test depends on a previous test, which fails. The test was set up incorrectly. UNTESTED A test was not run. This is a place-holder, used when there is no real test case yet. The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional: UNSUPPORTED There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. &dj; also uses this message when a testing environment (often a ``bare board'' target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor. &dj; uses the same output procedures to produce these messages for all testsuites and these procedures are already known to conform to POSIX 1003.3. For a &dj; testsuite to conform to POSIX 1003.3, you must avoid the setup_xfail procedure as described in the PASS section above and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above. &user; &ref;