path: root/doc/dejagnu.xml

<?xml version="1.0"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
    "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [
  <!ENTITY legal SYSTEM "legal.xml">
  <!ENTITY appversion "1.5.1">
  <!ENTITY version "1.5.1">
  <!ENTITY manrevision "1.0">
  <!ENTITY date "Feb 2004">
  <!ENTITY app "<application>DejaGnu</application>">
  <!ENTITY appname "DejaGnu">
  <!ENTITY dj "DejaGnu">
  <!-- The reference material -->
  <!ENTITY ref SYSTEM "ref.xml">
  <!-- The user manual -->
  <!ENTITY user SYSTEM "user.xml">
]>
<!-- Begin Document Specific Declarations -->
<article>
  <articleinfo>
    <title>&dj;</title>
    <subtitle>The GNU Testing Framework</subtitle>
    <date>January 2006</date>
    <authorgroup>
      <author>
        <firstname>Rob</firstname>
        <surname>Savoye</surname>
      </author>
      <author>
        <firstname>Ben</firstname>
        <surname>Elliston</surname>
      </author>
    </authorgroup>

    <copyright>
       <year>2006</year>
       <holder>Free Software Foundation, Inc.</holder>
    </copyright>
    <!-- <legalnotice>
      <para> -->
        <!-- [FIXME: must put legal notice here] -->
      <!-- </para> -->
    <!-- </legalnotice> -->
    <revhistory>
      <revision>
        <revnumber>0.6.2</revnumber>
        <date>2002-07-16</date>
        <authorinitials>rob</authorinitials>
        <revremark>Add new tutorial as a new sect1.</revremark>
      </revision>
      <revision>
        <revnumber>0.6.1</revnumber>
        <date>2001-02-16</date>
        <authorinitials>rob</authorinitials>
        <revremark>Add info on the new dejagnu.h file.</revremark>
      </revision>
      <revision>
        <revnumber>0.6</revnumber>
        <date>2001-02-16</date>
        <authorinitials>rob</authorinitials>
        <revremark>Updated for new release.</revremark>
      </revision>
      <revision>
        <revnumber>0.5</revnumber>
        <date>2000-01-24</date>
        <authorinitials>rob</authorinitials>
        <revremark>Initial version after conversion to DocBook.</revremark>
      </revision>
    </revhistory>

  </articleinfo>

 <sect1 id="preface">
    <title>Abstract</title>

  <para>This document describes the functionality of &dj;, the
    testing framework of the GNU project. &dj; is written in
    <productname>Expect</productname>, which uses
    <productname>Tcl</productname> as a command
    language. <productname>Expect</productname> acts as a very
    programmable shell.  As with other Unix command shells, you can
    run any program, but once the program is started, your test script
    has programmable control over its input and output.  This does not
    just apply to the programs under test; <command>expect</command>
    can also run any auxiliary program, such as
    <command>diff</command> or <command>sh</command>, with full
    control over its input and output.</para>

    <para>&dj; itself is merely a framework for the creation of
    testsuites. Testsuites are distributed with each
    application.</para>

  </sect1>

  <sect1 id="overview" xreflabel="Overview">
    <title>Overview</title>
    <sect2 id="whatis" xreflabel="What is &dj; ?">
      <title>What is &dj; ?</title>

    <para><productname>&dj;</productname> is a framework for
	testing other programs.  Its purpose is to provide a single
	front end for all tests. Think of it as a custom library of
	Tcl procedures crafted to support writing a test harness. A
	<emphasis>Test Harness</emphasis> is the testing
	infrastructure that is created to support a specific program
	or tool. Each program can have multiple testsuites, all
	supported by a single test harness. &dj; is written in
	<productname>Expect</productname>, which in turn uses
	<productname>Tcl</productname> -- Tool command
	language. There is more information on Tcl at the <ulink
	url="http://www.scriptics.com">Scriptics</ulink> web site and the
	Expect web site is at <ulink
	url="http://expect.nist.gov">NIST</ulink>.</para>
	
   <para>Julia Menapace first coined the term ``&dj;'' to describe
        an earlier testing framework at Cygnus Support she had written
        for <command>GDB</command>. When we replaced it with the
        Expect-based framework, it was like &dj; all over again.
        More importantly, it was also named after my daughter, <ulink
        url="http://www.welcomehome.org/deja/">Deja Snow Savoye</ulink>
        (now 14 years old as of Feb 2004), who was a toddler
        during &dj;'s beginnings.</para>

    <para>&dj; offers several advantages for testing:</para>

    <itemizedlist mark="bullet" spacing="compact">

      <listitem><para>The flexibility and consistency of the &dj;
	framework make it easy to write tests for any program, with
	either batch oriented, or interactive programs.</para>
      </listitem>

      <listitem><para>&dj; provides a layer of abstraction which
	allows you to write tests that are portable to any host or
	target where a program must be tested. For instance, a test
	for <command>GDB</command> can run from any supported host
	system on any supported target system. &dj; runs tests on
	many single board computers, whose operating software ranges
	from a simple boot monitor to a real-time OS.</para>
	</listitem>

      <listitem><para>All tests have the same output format. This
	makes it easy to integrate testing into other software
	development processes. &dj;'s output is designed to be
	parsed by other filtering script and it is also human
	readable.</para>
      </listitem>

      <listitem><para>Using Tcl and Expect, it's easy to create wrappers
      for existing testsuites. By incorporating existing tests under
      &dj;, it's easier to have a single set of report analyse
      programs..</para>

      </listitem>
    </itemizedlist>

    <para>Running tests requires two things: the testing framework and
    the testsuites themselves. Tests are usually written in
    <productname>Expect</productname> using Tcl, but you can also use
    a Tcl script to run a testsuite that is not based on
    <productname>Expect</productname>. <productname>Expect</productname>
    script filenames conventionally use <emphasis>.exp</emphasis> as a
    suffix; for example, the main implementation of the &dj; test
    driver is in the file
    <productname>runtest.exp</productname>.)</para>
	
  </sect2>

  <sect2 id="new" xreflabel="Release Notes">
    <title>New In This Release</title>

    <para>This release has a number of substantial changes over version
    1.3. The most visible change is that the version of Expect and Tcl
    included in the release are up-to-date with the current stable net
    releases. The biggest change is years of modifications to the
    target configuration system, used for cross testing. While this
    greatly improved cross testing, it has made that subsystem very
    complicated. The goal is to have this entirely rewritten using
    <productname>iTcl</productname> by the next release. Other changes
    are:</para>

    <itemizedlist>
      <listitem>
        <para>More built-in support for building target binaries
          with the correct linker flags. Currently this only works with
          <productname>GCC</productname> as the cross compiler,
	  preferably with a target supported by
	  <xref linkend="libgloss"/>.
        </para>
      </listitem>

      <listitem><para>Lots of little bug fixes from years of heavy
        use at Cygnus Solutions.</para></listitem>

      <listitem><para>&dj; now uses
      <productname>Automake</productname> for Makefile
      configuration.</para></listitem>

      <listitem><para>Updated documentation, now in DocBook XML.
      </para></listitem>
	
      <listitem><para>Windows support. There is beta level support for
      Windows that is still a work in progress. This requires the
      <ulink url="http://www.cygwin.com/">Cygwin</ulink> POSIX
      subsystem for Windows.</para></listitem>

    </itemizedlist>

    <sect3 id="cygwin" xreflabel="Windows Support">
      <title>Windows Support</title>

      <para>To use &dj; on Windows, you need to first install the
	<ulink url="http://www.cygwin.com/">Cygwin</ulink>
	release. This works as of the B20.1 release. Cygwin is a POSIX
	system for Windows. This covers both utility programs and a library
	that adds POSIX system calls to Windows. Among them is pseudo tty
	support for Windows that emulates the POSIX pty standard. The
	latest Cygwin is always available from <ulink
	url="http://www.cygwin.com/">this location</ulink>. This
	works well enough to run <emphasis>"make check"</emphasis> of
	the GNU development tree on Windows after a native build. But the
	nature of ptys on Windows is still evolving. Your mileage may
	vary.</para>

    </sect3>
	
  </sect2>

  <sect2 id="designgoals" xreflabel="Design Goals">
    <title>Design Goals</title>

    <para>&dj; grew out of the internal needs of Cygnus Solutions,
    the company formerly known as Cygnus Support. Cygnus maintained
    and enhanced a variety of free programs in many different
    environments and we needed a testing tool that:</para>

    <itemizedlist mark="bullet">
      <listitem><para>was useful to developers while fixing
      bugs;</para></listitem>
      <listitem><para>automated running many tests during a software
      release process;</para></listitem>
      <listitem><para>was portable among a variety of host
      computers;</para></listitem>
      <listitem><para>supported cross-development
      testing;</para></listitem>
      <listitem><para>permitted testing interactive programs, like
      <command>GDB</command>; and </para></listitem>
      <listitem><para>permitted testing batch oriented programs, like
      <command>GCC</command>.</para></listitem>
    </itemizedlist>

    <para>Some of the requirements proved challenging.  For example,
    interactive programs do not lend themselves very well to automated testing.
    But all the requirements are important: for instance, it is imperative to
    make sure that <command>GDB</command> works as well when cross-debugging
    as it does in a native configuration. </para>

    <para>Probably the greatest challenge was testing in a
    cross-development environment.  Most cross-development
    environments are customized by each developer.  Even when buying
    packaged boards from vendors there are many differences.  The
    communication interfaces vary from a serial line to Ethernet.
    &dj; was designed with a modular communication setup, so that
    each kind of communication can be added as required and supported
    thereafter.  Once a communication procedure is coded, any test can
    use it.  Currently &dj; can use <command>rsh</command>,
    <command>rlogin</command>, <command>telnet</command>,
    <command>tip</command>, <command>kermit</command> and
    <command>mondfe</command> for remote communications.</para>

    </sect2>

    <sect2 id="posix" xreflabel="A POSIX Conforming Test Framework">
      <title>A POSIX conforming test framework</title>

      <para>&dj; conforms to the POSIX 1003.3 standard for test
      frameworks. Rob Savoye was a member of that committee.</para>

      <para>The POSIX standard 1003.3 defines what a testing framework needs to
      provide, in order to permit the creation of POSIX conformance test
      suites. This standard is primarily oriented to running POSIX conformance
      tests, but its requirements also support testing of features not related
      to POSIX conformance.  POSIX 1003.3 does not specify a particular testing
      framework, but at this time there is only one other POSIX conforming test
      framework: TET. TET was created by Unisoft for a consortium comprised of
      X/Open, Unix International and the Open Software Foundation.</para>

      <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
      An assertion is a description of behavior.  For example, if a standard
      says ``The sun shall shine'', a corresponding assertion might be ``The
      sun is shining.''  A test based on this assertion would pass or fail
      depending on whether it is day or night.  It is important to note
      that the standard being tested is never 1003.3; the standard being tested
      is some other standard, for which the assertions were written.</para>

      <para>As there is no testsuite to test testing frameworks for POSIX
      1003.3 conformance, verifying conformance to this standard is done by
      repeatedly reading the standard and experimenting.  One of the main
      things 1003.3 does specify is the set of allowed output messages and
      their definitions.  Four messages are supported for a required feature of
      POSIX conforming systems and a fifth for a conditional feature. &dj;
      supports the use of all five output messages.  In this sense a testsuite
      that uses exactly these messages can be considered POSIX conforming.
      These definitions specify the output of a test
      case:</para>

      <variablelist>
	<varlistentry>
	  <term>PASS</term>
	  <listitem><para>A test has succeeded.  That is, it demonstrated that
	  the assertion is true.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>XFAIL</term>
	  <listitem><para>POSIX 1003.3 does not incorporate the notion of
	  expected failures, so <emphasis>PASS</emphasis>, instead of
	  <emphasis>XPASS</emphasis>, must also be returned for test cases
	  which were expected to fail and did not.  This means that
	  <emphasis>PASS</emphasis> is in some sense more ambiguous than if
	  <emphasis>XPASS</emphasis> is also used.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>FAIL</term>
	  <listitem><para>A test has produced the bug it was intended to
	  capture.  That is, it has demonstrated that the assertion is false.
	  The <emphasis>FAIL</emphasis> message is based on the test case only.
	  Other messages are used to indicate a failure of the framework. As
	  with <emphasis>PASS</emphasis>, POSIX tests must return
	  <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
	  if a failure was expected.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>UNRESOLVED</term>
	  <listitem><para>A test produced indeterminate results.  Usually, this
	  means the test executed in an unexpected fashion; this outcome
	  requires that a human being go over results, to determine if the test
	  should have passed or failed.  This message is also used for any test
	  that requires human intervention because it is beyond the abilities
	  of the testing framework.  Any unresolved test should resolved to
	  <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
	  run can be considered finished.</para>

	  <para>Note that for POSIX, each assertion must produce a test result
	  code.  If the test isn't actually run, it must produce
	  <emphasis>UNRESOLVED</emphasis> rather than just leaving that test
	  out of the output.  This means that you have to be careful when
	  writing tests to not carelessly use Tcl commands like
	  <emphasis>return</emphasis>---if you alter the flow of control of the
	  Tcl code you must insure that every test still produces some result
	  code.</para>

	  <para>Here are some of the ways a test may wind up
	  <emphasis>UNRESOLVED</emphasis>:</para></listitem>

	</varlistentry>
	</variablelist>

          <itemizedlist>
	    <listitem><para>A test's execution is
	    interrupted.</para></listitem>

	    <listitem><para>A test does not produce a clear
	    result. This is usually because there was an
	    <emphasis>ERROR</emphasis> from &dj; while processing
	    the test, or because there were three or more
	    <emphasis>WARNING</emphasis> messages. Any
	    <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
	    messages can invalidate the output of the test.  This
	    usually requires a human being to examine the output to
	    determine what really happened---and to improve the test
	    case.</para></listitem>

	    <listitem><para>A test depends on a previous test, which
	    fails.</para></listitem>

	    <listitem><para>The test was set up
	    incorrectly.</para></listitem>
	  </itemizedlist>

	<variablelist>
	  <varlistentry>
	    <term>UNTESTED</term>
	    <listitem><para>A test was not run.  This is a place-holder, used
	    when there is no real test case yet.</para></listitem>
	  </varlistentry>
	</variablelist>

	  <para>The only remaining output message left is intended to test
	  features that are specified by the applicable POSIX standard as
	  conditional:</para>

	<variablelist>
	  <varlistentry>
  	    <term>UNSUPPORTED</term>
	    <listitem><para>There is no support for the tested case.  This may
  	    mean that a conditional feature of an operating system, or of a
  	    compiler, is not implemented.  &dj; also uses this message when
  	    a testing environment (often a ``bare board'' target) lacks basic
  	    support for compiling or running the test case.  For example, a
  	    test for the system subroutine <emphasis>gethostname</emphasis>
  	    would never work on a target board running only a boot
  	    monitor.</para></listitem>
	  </varlistentry>
	</variablelist>

        <para>&dj; uses the same output procedures to produce these messages
	for all testsuites and these procedures are already known to conform
	to POSIX 1003.3.  For a &dj; testsuite to conform to POSIX 1003.3,
	you must avoid the <emphasis>setup</emphasis>xfail} procedure as
	described in the <emphasis>PASS</emphasis> section above and you must
	be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
	as described in the <emphasis>UNRESOLVED</emphasis> section
	above.</para>
     </sect2>
  </sect1>

  <!-- include the user manual -->
  &user;

  <!-- include the reference manual -->
  &ref;

</article>

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-omittag:t
sgml-shorttag:t
sgml-namecase-general:t
sgml-general-insert-case:lower
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:nil
sgml-parent-document:nil
sgml-exposed-tags:nil
sgml-local-catalogs:nil
sgml-local-ecat-files:nil
End:
-->