diff options
Diffstat (limited to 'Help/manual/ctest.1.rst')
| -rw-r--r-- | Help/manual/ctest.1.rst | 800 |
1 files changed, 641 insertions, 159 deletions
diff --git a/Help/manual/ctest.1.rst b/Help/manual/ctest.1.rst index a89c4e9bb..e29ebcaf6 100644 --- a/Help/manual/ctest.1.rst +++ b/Help/manual/ctest.1.rst @@ -3,20 +3,27 @@ ctest(1) ******** +.. contents:: + Synopsis ======== .. parsed-literal:: ctest [<options>] + ctest --build-and-test <path-to-source> <path-to-build> + --build-generator <generator> [<options>...] + [--build-options <opts>...] [--test-command <command> [<args>...]] + ctest {-D <dashboard> | -M <model> -T <action> | -S <script> | -SP <script>} + [-- <dashboard-options>...] Description =========== -The "ctest" executable is the CMake test driver program. +The **ctest** executable is the CMake test driver program. CMake-generated build trees created for projects that use the -ENABLE_TESTING and ADD_TEST commands have testing support. This -program will run the tests and report results. +:command:`enable_testing` and :command:`add_test` commands have testing support. +This program will run the tests and report results. Options ======= @@ -26,8 +33,21 @@ Options Some CMake-generated build trees can have multiple build configurations in the same tree. This option can be used to specify - which one should be tested. Example configurations are "Debug" and - "Release". + which one should be tested. Example configurations are ``Debug`` and + ``Release``. + +``--progress`` + Enable short progress output from tests. + + When the output of **ctest** is being sent directly to a terminal, the + progress through the set of tests is reported by updating the same line + rather than printing start and end messages for each test on new lines. + This can significantly reduce the verbosity of the test output. + Test completion messages are still output on their own line for failed + tests and the final test summary will also still be logged. + + This option can also be enabled by setting the environment variable + :envvar:`CTEST_PROGRESS_OUTPUT`. ``-V,--verbose`` Enable verbose output from tests. @@ -49,22 +69,35 @@ Options ``--output-on-failure`` Output anything outputted by the test program if the test should fail. - This option can also be enabled by setting the environment variable - ``CTEST_OUTPUT_ON_FAILURE``. + This option can also be enabled by setting the + :envvar:`CTEST_OUTPUT_ON_FAILURE` environment variable ``-F`` Enable failover. - This option allows ctest to resume a test set execution that was + This option allows CTest to resume a test set execution that was previously interrupted. If no interruption occurred, the ``-F`` option will have no effect. ``-j <jobs>, --parallel <jobs>`` Run the tests in parallel using the given number of jobs. - This option tells ctest to run the tests in parallel using given - number of jobs. This option can also be set by setting the - environment variable ``CTEST_PARALLEL_LEVEL``. + This option tells CTest to run the tests in parallel using given + number of jobs. This option can also be set by setting the + :envvar:`CTEST_PARALLEL_LEVEL` environment variable. + + This option can be used with the :prop_test:`PROCESSORS` test property. + + See `Label and Subproject Summary`_. + +``--resource-spec-file <file>`` + Run CTest with :ref:`resource allocation <ctest-resource-allocation>` enabled, + using the + :ref:`resource specification file <ctest-resource-specification-file>` + specified in ``<file>``. + + When ``ctest`` is run as a `Dashboard Client`_ this sets the + ``ResourceSpecFile`` option of the `CTest Test Step`_. ``--test-load <level>`` While running tests in parallel (e.g. with ``-j``), try not to start @@ -74,7 +107,7 @@ Options ``TestLoad`` option of the `CTest Test Step`_. ``-Q,--quiet`` - Make ctest quiet. + Make CTest quiet. This option will suppress all the output. The output log file will still be generated if the ``--output-log`` is specified. Options such @@ -84,37 +117,47 @@ Options ``-O <file>, --output-log <file>`` Output to log file. - This option tells ctest to write all its output to a log file. + This option tells CTest to write all its output to a ``<file>`` log file. -``-N,--show-only`` +``-N,--show-only[=<format>]`` Disable actual execution of tests. - This option tells ctest to list the tests that would be run but not + This option tells CTest to list the tests that would be run but not actually run them. Useful in conjunction with the ``-R`` and ``-E`` options. + ``<format>`` can be one of the following values. + + ``human`` + Human-friendly output. This is not guaranteed to be stable. + This is the default. + + ``json-v1`` + Dump the test information in JSON format. + See `Show as JSON Object Model`_. + ``-L <regex>, --label-regex <regex>`` Run tests with labels matching regular expression. - This option tells ctest to run only the tests whose labels match the + This option tells CTest to run only the tests whose labels match the given regular expression. ``-R <regex>, --tests-regex <regex>`` Run tests matching regular expression. - This option tells ctest to run only the tests whose names match the + This option tells CTest to run only the tests whose names match the given regular expression. ``-E <regex>, --exclude-regex <regex>`` Exclude tests matching regular expression. - This option tells ctest to NOT run the tests whose names match the + This option tells CTest to NOT run the tests whose names match the given regular expression. ``-LE <regex>, --label-exclude <regex>`` Exclude tests with labels matching regular expression. - This option tells ctest to NOT run the tests whose labels match the + This option tells CTest to NOT run the tests whose labels match the given regular expression. ``-FA <regex>, --fixture-exclude-any <regex>`` @@ -137,10 +180,13 @@ Options ``-D <dashboard>, --dashboard <dashboard>`` Execute dashboard test. - This option tells ctest to act as a CDash client and perform a - dashboard test. All tests are <Mode><Test>, where Mode can be - Experimental, Nightly, and Continuous, and Test can be Start, - Update, Configure, Build, Test, Coverage, and Submit. + This option tells CTest to act as a CDash client and perform a + dashboard test. All tests are ``<Mode><Test>``, where ``<Mode>`` can be + ``Experimental``, ``Nightly``, and ``Continuous``, and ``<Test>`` can be + ``Start``, ``Update``, ``Configure``, ``Build``, ``Test``, + ``Coverage``, and ``Submit``. + + See `Dashboard Client`_. ``-D <var>:<type>=<value>`` Define a variable for script mode. @@ -153,35 +199,33 @@ Options ``-M <model>, --test-model <model>`` Sets the model for a dashboard. - This option tells ctest to act as a CDash client where the ``<model>`` + This option tells CTest to act as a CDash client where the ``<model>`` can be ``Experimental``, ``Nightly``, and ``Continuous``. Combining ``-M`` and ``-T`` is similar to ``-D``. + See `Dashboard Client`_. + ``-T <action>, --test-action <action>`` Sets the dashboard action to perform. - This option tells ctest to act as a CDash client and perform some + This option tells CTest to act as a CDash client and perform some action such as ``start``, ``build``, ``test`` etc. See `Dashboard Client Steps`_ for the full list of actions. Combining ``-M`` and ``-T`` is similar to ``-D``. -``--track <track>`` - Specify the track to submit dashboard to - - Submit dashboard to specified track instead of default one. By - default, the dashboard is submitted to Nightly, Experimental, or - Continuous track, but by specifying this option, the track can be - arbitrary. + See `Dashboard Client`_. ``-S <script>, --script <script>`` Execute a dashboard for a configuration. - This option tells ctest to load in a configuration script which sets + This option tells CTest to load in a configuration script which sets a number of parameters such as the binary and source directories. - Then ctest will do what is required to create and run a dashboard. + Then CTest will do what is required to create and run a dashboard. This option basically sets up a dashboard and then runs ``ctest -D`` with the appropriate options. + See `Dashboard Client`_. + ``-SP <script>, --script-new-process <script>`` Execute a dashboard for a configuration. @@ -190,19 +234,15 @@ Options script may modify the environment and you do not want the modified environment to impact other ``-S`` scripts. -``-A <file>, --add-notes <file>`` - Add a notes file with submission. - - This option tells ctest to include a notes file when submitting - dashboard. + See `Dashboard Client`_. ``-I [Start,End,Stride,test#,test#|Test file], --tests-information`` Run a specific number of tests by number. - This option causes ctest to run tests starting at number Start, - ending at number End, and incrementing by Stride. Any additional - numbers after Stride are considered individual test numbers. Start, - End,or stride can be empty. Optionally a file can be given that + This option causes CTest to run tests starting at number ``Start``, + ending at number ``End``, and incrementing by ``Stride``. Any additional + numbers after ``Stride`` are considered individual test numbers. ``Start``, + ``End``, or ``Stride`` can be empty. Optionally a file can be given that contains the same syntax as the command line. ``-U, --union`` @@ -214,11 +254,11 @@ Options ``--rerun-failed`` Run only the tests that failed previously. - This option tells ctest to perform only the tests that failed during - its previous run. When this option is specified, ctest ignores all + This option tells CTest to perform only the tests that failed during + its previous run. When this option is specified, CTest ignores all other options intended to modify the list of tests to run (``-L``, ``-R``, ``-E``, ``-LE``, ``-I``, etc). In the event that CTest runs and no tests - fail, subsequent calls to ctest with the ``--rerun-failed`` option will run + fail, subsequent calls to CTest with the ``--rerun-failed`` option will run the set of tests that most recently failed (if any). ``--repeat-until-fail <n>`` @@ -234,41 +274,154 @@ Options name which can be very annoying. ``--interactive-debug-mode [0|1]`` - Set the interactive mode to 0 or 1. + Set the interactive mode to ``0`` or ``1``. - This option causes ctest to run tests in either an interactive mode + This option causes CTest to run tests in either an interactive mode or a non-interactive mode. On Windows this means that in non-interactive mode, all system debug pop up windows are blocked. - In dashboard mode (Experimental, Nightly, Continuous), the default + In dashboard mode (``Experimental``, ``Nightly``, ``Continuous``), the default is non-interactive. When just running tests not for a dashboard the default is to allow popups and interactive debugging. ``--no-label-summary`` Disable timing summary information for labels. - This option tells ctest not to print summary information for each + This option tells CTest not to print summary information for each label associated with the tests run. If there are no labels on the tests, nothing extra is printed. -``--build-and-test <path-to-source> <path-to-build>`` - Configure, build and run a test. + See `Label and Subproject Summary`_. + +``--no-subproject-summary`` + Disable timing summary information for subprojects. + + This option tells CTest not to print summary information for each + subproject associated with the tests run. If there are no subprojects on the + tests, nothing extra is printed. + + See `Label and Subproject Summary`_. + +``--build-and-test`` +See `Build and Test Mode`_. + +``--test-output-size-passed <size>`` + Limit the output for passed tests to ``<size>`` bytes. + +``--test-output-size-failed <size>`` + Limit the output for failed tests to ``<size>`` bytes. - This option tells ctest to configure (i.e. run cmake on), build, - and or execute a test. The configure and test steps are optional. - The arguments to this command line are the source and binary - directories. - The ``--build-generator`` option *must* be provided to use - ``--build-and-test``. If ``--test-command`` is specified then that will be - run after the build is complete. Other options that affect this - mode are ``--build-target``, ``--build-nocmake``, ``--build-run-dir``, - ``--build-two-config``, ``--build-exe-dir``, - ``--build-project``, ``--build-noclean`` and ``--build-options``. +``--overwrite`` + Overwrite CTest configuration option. + + By default CTest uses configuration options from configuration file. + This option will overwrite the configuration option. + +``--force-new-ctest-process`` + Run child CTest instances as new processes. + + By default CTest will run child CTest instances within the same + process. If this behavior is not desired, this argument will + enforce new processes for child CTest processes. + +``--schedule-random`` + Use a random order for scheduling tests. + + This option will run the tests in a random order. It is commonly + used to detect implicit dependencies in a test suite. + +``--submit-index`` + Legacy option for old Dart2 dashboard server feature. + Do not use. + +``--timeout <seconds>`` + Set the default test timeout. + + This option effectively sets a timeout on all tests that do not + already have a timeout set on them via the :prop_test:`TIMEOUT` + property. + +``--stop-time <time>`` + Set a time at which all tests should stop running. + + Set a real time of day at which all tests should timeout. Example: + ``7:00:00 -0400``. Any time format understood by the curl date parser + is accepted. Local time is assumed if no timezone is specified. + +``--print-labels`` + Print all available test labels. + + This option will not run any tests, it will simply print the list of + all labels associated with the test set. + +.. include:: OPTIONS_HELP.txt + +.. _`Label and Subproject Summary`: + +Label and Subproject Summary +============================ + +CTest prints timing summary information for each ``LABEL`` and subproject +associated with the tests run. The label time summary will not include labels +that are mapped to subprojects. + +When the :prop_test:`PROCESSORS` test property is set, CTest will display a +weighted test timing result in label and subproject summaries. The time is +reported with `sec*proc` instead of just `sec`. + +The weighted time summary reported for each label or subproject ``j`` +is computed as:: + + Weighted Time Summary for Label/Subproject j = + sum(raw_test_time[j,i] * num_processors[j,i], i=1...num_tests[j]) + + for labels/subprojects j=1...total + +where: + +* ``raw_test_time[j,i]``: Wall-clock time for the ``i`` test + for the ``j`` label or subproject +* ``num_processors[j,i]``: Value of the CTest :prop_test:`PROCESSORS` property + for the ``i`` test for the ``j`` label or subproject +* ``num_tests[j]``: Number of tests associated with the ``j`` label or subproject +* ``total``: Total number of labels or subprojects that have at least one test run + +Therefore, the weighted time summary for each label or subproject represents +the amount of time that CTest gave to run the tests for each label or +subproject and gives a good representation of the total expense of the tests +for each label or subproject when compared to other labels or subprojects. + +For example, if ``SubprojectA`` showed ``100 sec*proc`` and ``SubprojectB`` showed +``10 sec*proc``, then CTest allocated approximately 10 times the CPU/core time +to run the tests for ``SubprojectA`` than for ``SubprojectB`` (e.g. so if effort +is going to be expended to reduce the cost of the test suite for the whole +project, then reducing the cost of the test suite for ``SubprojectA`` would +likely have a larger impact than effort to reduce the cost of the test suite +for ``SubprojectB``). + +.. _`Build and Test Mode`: + +Build and Test Mode +=================== + +CTest provides a command-line signature to configure (i.e. run cmake on), +build, and/or execute a test:: + + ctest --build-and-test <path-to-source> <path-to-build> + --build-generator <generator> + [<options>...] + [--build-options <opts>...] + [--test-command <command> [<args>...]] + +The configure and test steps are optional. The arguments to this command line +are the source and binary directories. The ``--build-generator`` option *must* +be provided to use ``--build-and-test``. If ``--test-command`` is specified +then that will be run after the build is complete. Other options that affect +this mode include: ``--build-target`` Specify a specific target to build. - This option goes with the ``--build-and-test`` option, if left out the - ``all`` target is built. + If left out the ``all`` target is built. ``--build-nocmake`` Run the build without running cmake first. @@ -299,91 +452,77 @@ Options Specify the name of the project to build. ``--build-makeprogram`` - Override the make program chosen by CTest with a given one. + Specify the explicit make program to be used by CMake when configuring and + building the project. Only applicable for Make and Ninja based generators. ``--build-noclean`` Skip the make clean step. ``--build-config-sample`` A sample executable to use to determine the configuration that - should be used. e.g. Debug/Release/etc. + should be used. e.g. ``Debug``, ``Release`` etc. ``--build-options`` - Add extra options to the build step. - - This option must be the last option with the exception of - ``--test-command`` + Additional options for configuring the build (i.e. for CMake, not for + the build tool). Note that if this is specified, the ``--build-options`` + keyword and its arguments must be the last option given on the command + line, with the possible exception of ``--test-command``. ``--test-command`` - The test to run with the ``--build-and-test`` option. - -``--test-output-size-passed <size>`` - Limit the output for passed tests to ``<size>`` bytes. - -``--test-output-size-failed <size>`` - Limit the output for failed tests to ``<size>`` bytes. + The command to run as the test step with the ``--build-and-test`` option. + All arguments following this keyword will be assumed to be part of the + test command line, so it must be the last option given. ``--test-timeout`` - The time limit in seconds, internal use only. + The time limit in seconds -``--tomorrow-tag`` - Nightly or experimental starts with next day tag. - - This is useful if the build will not finish in one day. - -``--ctest-config`` - The configuration file used to initialize CTest state when submitting dashboards. +.. _`Dashboard Client`: - This option tells CTest to use different initialization file instead - of CTestConfiguration.tcl. This way multiple initialization files - can be used for example to submit to multiple dashboards. +Dashboard Client +================ -``--overwrite`` - Overwrite CTest configuration option. +CTest can operate as a client for the `CDash`_ software quality dashboard +application. As a dashboard client, CTest performs a sequence of steps +to configure, build, and test software, and then submits the results to +a `CDash`_ server. The command-line signature used to submit to `CDash`_ is:: - By default ctest uses configuration options from configuration file. - This option will overwrite the configuration option. + ctest (-D <dashboard> | -M <model> -T <action> | -S <script> | -SP <script>) + [-- <dashboard-options>...] -``--extra-submit <file>[;<file>]`` - Submit extra files to the dashboard. +Options for Dashboard Client include: - This option will submit extra files to the dashboard. +``--group <group>`` + Specify what group you'd like to submit results to -``--force-new-ctest-process`` - Run child CTest instances as new processes. - - By default CTest will run child CTest instances within the same - process. If this behavior is not desired, this argument will - enforce new processes for child CTest processes. + Submit dashboard to specified group instead of default one. By + default, the dashboard is submitted to Nightly, Experimental, or + Continuous group, but by specifying this option, the group can be + arbitrary. -``--schedule-random`` - Use a random order for scheduling tests. + This replaces the deprecated option ``--track``. + Despite the name change its behavior is unchanged. - This option will run the tests in a random order. It is commonly - used to detect implicit dependencies in a test suite. +``-A <file>, --add-notes <file>`` + Add a notes file with submission. -``--submit-index`` - Legacy option for old Dart2 dashboard server feature. - Do not use. + This option tells CTest to include a notes file when submitting + dashboard. -``--timeout <seconds>`` - Set a global timeout on all tests. +``--tomorrow-tag`` + ``Nightly`` or ``Experimental`` starts with next day tag. - This option will set a global timeout on all tests that do not - already have a timeout set on them. + This is useful if the build will not finish in one day. -``--stop-time <time>`` - Set a time at which all tests should stop running. +``--extra-submit <file>[;<file>]`` + Submit extra files to the dashboard. - Set a real time of day at which all tests should timeout. Example: - ``7:00:00 -0400``. Any time format understood by the curl date parser - is accepted. Local time is assumed if no timezone is specified. + This option will submit extra files to the dashboard. ``--http1.0`` - Submit using HTTP 1.0. + Submit using `HTTP 1.0`. - This option will force CTest to use HTTP 1.0 to submit files to the - dashboard, instead of HTTP 1.1. + This option will force CTest to use `HTTP 1.0` to submit files to the + dashboard, instead of `HTTP 1.1`. ``--no-compress-output`` Do not compress test output when submitting. @@ -392,26 +531,6 @@ Options this to maintain compatibility with an older version of CDash which doesn't support compressed test output. -``--print-labels`` - Print all available test labels. - - This option will not run any tests, it will simply print the list of - all labels associated with the test set. - -.. include:: OPTIONS_HELP.txt - -.. _`Dashboard Client`: - -Dashboard Client -================ - -CTest can operate as a client for the `CDash`_ software quality dashboard -application. As a dashboard client, CTest performs a sequence of steps -to configure, build, and test software, and then submits the results to -a `CDash`_ server. - -.. _`CDash`: http://cdash.org/ - Dashboard Client Steps ---------------------- @@ -606,7 +725,7 @@ Configuration settings to specify the version control tool include: The source tree is updated by ``git fetch`` followed by ``git reset --hard`` to the ``FETCH_HEAD``. The result is the same - as ``git pull`` except that any local moficiations are overwritten. + as ``git pull`` except that any local modifications are overwritten. Use ``GITUpdateCustom`` to specify a different approach. ``GITInitSubmodules`` @@ -716,6 +835,8 @@ Configuration settings to specify the version control tool include: * :module:`CTest` module variable: ``UPDATE_TYPE`` if set, else ``CTEST_UPDATE_TYPE`` +.. _`UpdateVersionOnly`: + ``UpdateVersionOnly`` Specify that you want the version control update command to only discover the current version that is checked out, and not to update @@ -723,7 +844,18 @@ Configuration settings to specify the version control tool include: * `CTest Script`_ variable: :variable:`CTEST_UPDATE_VERSION_ONLY` +.. _`UpdateVersionOverride`: +``UpdateVersionOverride`` + Specify the current version of your source tree. + + When this variable is set to a non-empty string, CTest will report the value + you specified rather than using the update command to discover the current + version that is checked out. Use of this variable supersedes + ``UpdateVersionOnly``. Like ``UpdateVersionOnly``, using this variable tells + CTest not to update the source tree to a different version. + + * `CTest Script`_ variable: :variable:`CTEST_UPDATE_VERSION_OVERRIDE` Additional configuration settings include: @@ -758,6 +890,16 @@ Configuration settings include: * :module:`CTest` module variable: :variable:`CMAKE_COMMAND` followed by :variable:`PROJECT_SOURCE_DIR` +``LabelsForSubprojects`` + Specify a semicolon-separated list of labels that will be treated as + subprojects. This mapping will be passed on to CDash when configure, test or + build results are submitted. + + * `CTest Script`_ variable: :variable:`CTEST_LABELS_FOR_SUBPROJECTS` + * :module:`CTest` module variable: ``CTEST_LABELS_FOR_SUBPROJECTS`` + + See `Label and Subproject Summary`_. + .. _`CTest Build Step`: CTest Build Step @@ -778,7 +920,17 @@ Configuration settings include: * `CTest Script`_ variable: :variable:`CTEST_CONFIGURATION_TYPE` * :module:`CTest` module variable: ``DEFAULT_CTEST_CONFIGURATION_TYPE``, - initialized by the ``CMAKE_CONFIG_TYPE`` environment variable + initialized by the :envvar:`CMAKE_CONFIG_TYPE` environment variable + +``LabelsForSubprojects`` + Specify a semicolon-separated list of labels that will be treated as + subprojects. This mapping will be passed on to CDash when configure, test or + build results are submitted. + + * `CTest Script`_ variable: :variable:`CTEST_LABELS_FOR_SUBPROJECTS` + * :module:`CTest` module variable: ``CTEST_LABELS_FOR_SUBPROJECTS`` + + See `Label and Subproject Summary`_. ``MakeCommand`` Command-line to launch the software build process. @@ -815,6 +967,21 @@ Arguments to the command may specify some of the step settings. Configuration settings include: +``ResourceSpecFile`` + Specify a + :ref:`resource specification file <ctest-resource-specification-file>`. See + :ref:`ctest-resource-allocation` for more information. + +``LabelsForSubprojects`` + Specify a semicolon-separated list of labels that will be treated as + subprojects. This mapping will be passed on to CDash when configure, test or + build results are submitted. + + * `CTest Script`_ variable: :variable:`CTEST_LABELS_FOR_SUBPROJECTS` + * :module:`CTest` module variable: ``CTEST_LABELS_FOR_SUBPROJECTS`` + + See `Label and Subproject Summary`_. + ``TestLoad`` While running tests in parallel (e.g. with ``-j``), try not to start tests when they may cause the CPU load to pass above a given threshold. @@ -853,6 +1020,8 @@ Configuration settings include: * `CTest Script`_ variable: :variable:`CTEST_COVERAGE_EXTRA_FLAGS` * :module:`CTest` module variable: ``COVERAGE_EXTRA_FLAGS`` + These options are the first arguments passed to ``CoverageCommand``. + .. _`CTest MemCheck Step`: CTest MemCheck Step @@ -945,7 +1114,7 @@ Configuration settings include: * :module:`CTest` module variable: ``BUILDNAME`` ``CDashVersion`` - Specify the version of `CDash`_ on the server. + Legacy option. Not used. * `CTest Script`_ variable: none, detected from server * :module:`CTest` module variable: ``CTEST_CDASH_VERSION`` @@ -974,55 +1143,58 @@ Configuration settings include: * :module:`CTest` module variable: ``CTEST_CURL_OPTIONS`` ``DropLocation`` - The path on the dashboard server to send the submission. + Legacy option. When ``SubmitURL`` is not set, it is constructed from + ``DropMethod``, ``DropSiteUser``, ``DropSitePassword``, ``DropSite``, and + ``DropLocation``. * `CTest Script`_ variable: :variable:`CTEST_DROP_LOCATION` * :module:`CTest` module variable: ``DROP_LOCATION`` if set, else ``CTEST_DROP_LOCATION`` ``DropMethod`` - Specify the method by which results should be submitted to the - dashboard server. The value may be ``cp``, ``ftp``, ``http``, - ``https``, ``scp``, or ``xmlrpc`` (if CMake was built with - support for it). + Legacy option. When ``SubmitURL`` is not set, it is constructed from + ``DropMethod``, ``DropSiteUser``, ``DropSitePassword``, ``DropSite``, and + ``DropLocation``. * `CTest Script`_ variable: :variable:`CTEST_DROP_METHOD` * :module:`CTest` module variable: ``DROP_METHOD`` if set, else ``CTEST_DROP_METHOD`` ``DropSite`` - The dashboard server name - (for ``ftp``, ``http``, and ``https``, ``scp``, and ``xmlrpc``). + Legacy option. When ``SubmitURL`` is not set, it is constructed from + ``DropMethod``, ``DropSiteUser``, ``DropSitePassword``, ``DropSite``, and + ``DropLocation``. * `CTest Script`_ variable: :variable:`CTEST_DROP_SITE` * :module:`CTest` module variable: ``DROP_SITE`` if set, else ``CTEST_DROP_SITE`` ``DropSitePassword`` - The dashboard server login password, if any - (for ``ftp``, ``http``, and ``https``). + Legacy option. When ``SubmitURL`` is not set, it is constructed from + ``DropMethod``, ``DropSiteUser``, ``DropSitePassword``, ``DropSite``, and + ``DropLocation``. * `CTest Script`_ variable: :variable:`CTEST_DROP_SITE_PASSWORD` * :module:`CTest` module variable: ``DROP_SITE_PASSWORD`` if set, else ``CTEST_DROP_SITE_PASWORD`` ``DropSiteUser`` - The dashboard server login user name, if any - (for ``ftp``, ``http``, and ``https``). + Legacy option. When ``SubmitURL`` is not set, it is constructed from + ``DropMethod``, ``DropSiteUser``, ``DropSitePassword``, ``DropSite``, and + ``DropLocation``. * `CTest Script`_ variable: :variable:`CTEST_DROP_SITE_USER` * :module:`CTest` module variable: ``DROP_SITE_USER`` if set, else ``CTEST_DROP_SITE_USER`` ``IsCDash`` - Specify whether the dashboard server is `CDash`_ or an older - dashboard server implementation requiring ``TriggerSite``. + Legacy option. Not used. * `CTest Script`_ variable: :variable:`CTEST_DROP_SITE_CDASH` * :module:`CTest` module variable: ``CTEST_DROP_SITE_CDASH`` ``ScpCommand`` - ``scp`` command-line tool to use when ``DropMethod`` is ``scp``. + Legacy option. Not used. * `CTest Script`_ variable: :variable:`CTEST_SCP_COMMAND` * :module:`CTest` module variable: ``SCPCOMMAND`` @@ -1035,15 +1207,325 @@ Configuration settings include: * :module:`CTest` module variable: ``SITE``, initialized by the :command:`site_name` command +``SubmitURL`` + The ``http`` or ``https`` URL of the dashboard server to send the submission + to. + + * `CTest Script`_ variable: :variable:`CTEST_SUBMIT_URL` + * :module:`CTest` module variable: ``SUBMIT_URL`` if set, + else ``CTEST_SUBMIT_URL`` + ``TriggerSite`` - Legacy option to support older dashboard server implementations. - Not used when ``IsCDash`` is true. + Legacy option. Not used. * `CTest Script`_ variable: :variable:`CTEST_TRIGGER_SITE` * :module:`CTest` module variable: ``TRIGGER_SITE`` if set, else ``CTEST_TRIGGER_SITE`` +.. _`Show as JSON Object Model`: + +Show as JSON Object Model +========================= + +When the ``--show-only=json-v1`` command line option is given, the test +information is output in JSON format. Version 1.0 of the JSON object +model is defined as follows: + +``kind`` + The string "ctestInfo". + +``version`` + A JSON object specifying the version components. Its members are + + ``major`` + A non-negative integer specifying the major version component. + ``minor`` + A non-negative integer specifying the minor version component. + +``backtraceGraph`` + JSON object representing backtrace information with the + following members: + + ``commands`` + List of command names. + ``files`` + List of file names. + ``nodes`` + List of node JSON objects with members: + + ``command`` + Index into the ``commands`` member of the ``backtraceGraph``. + ``file`` + Index into the ``files`` member of the ``backtraceGraph``. + ``line`` + Line number in the file where the backtrace was added. + ``parent`` + Index into the ``nodes`` member of the ``backtraceGraph`` + representing the parent in the graph. + +``tests`` + A JSON array listing information about each test. Each entry + is a JSON object with members: + + ``name`` + Test name. + ``config`` + Configuration that the test can run on. + Empty string means any config. + ``command`` + List where the first element is the test command and the + remaining elements are the command arguments. + ``backtrace`` + Index into the ``nodes`` member of the ``backtraceGraph``. + ``properties`` + Test properties. + Can contain keys for each of the supported test properties. + +.. _`ctest-resource-allocation`: + +Resource Allocation +=================== + +CTest provides a mechanism for tests to specify the resources that they need +in a fine-grained way, and for users to specify the resources availiable on +the running machine. This allows CTest to internally keep track of which +resources are in use and which are free, scheduling tests in a way that +prevents them from trying to claim resources that are not available. + +A common use case for this feature is for tests that require the use of a GPU. +Multiple tests can simultaneously allocate memory from a GPU, but if too many +tests try to do this at once, some of them will fail to allocate, resulting in +a failed test, even though the test would have succeeded if it had the memory +it needed. By using the resource allocation feature, each test can specify how +much memory it requires from a GPU, allowing CTest to schedule tests in a way +that running several of these tests at once does not exhaust the GPU's memory +pool. + +Please note that CTest has no concept of what a GPU is or how much memory it +has, nor does it have any way of communicating with a GPU to retrieve this +information or perform any memory management. CTest simply keeps track of a +list of abstract resource types, each of which has a certain number of slots +available for tests to use. Each test specifies the number of slots that it +requires from a certain resource, and CTest then schedules them in a way that +prevents the total number of slots in use from exceeding the listed capacity. +When a test is executed, and slots from a resource are allocated to that test, +tests may assume that they have exclusive use of those slots for the duration +of the test's process. + +The CTest resource allocation feature consists of two inputs: + +* The :ref:`resource specification file <ctest-resource-specification-file>`, + described below, which describes the resources available on the system. +* The :prop_test:`RESOURCE_GROUPS` property of tests, which describes the + resources required by the test. + +When CTest runs a test, the resources allocated to that test are passed in the +form of a set of +:ref:`environment variables <ctest-resource-environment-variables>` as +described below. Using this information to decide which resource to connect to +is left to the test writer. + +The ``RESOURCE_GROUPS`` property tells CTest what resources a test expects +to use grouped in a way meaningful to the test. The test itself must read +the :ref:`environment variables <ctest-resource-environment-variables>` to +determine which resources have been allocated to each group. For example, +each group may correspond to a process the test will spawn when executed. + +Note that even if a test specifies a ``RESOURCE_GROUPS`` property, it is still +possible for that to test to run without any resource allocation (and without +the corresponding +:ref:`environment variables <ctest-resource-environment-variables>`) +if the user does not pass a resource specification file. Passing this file, +either through the ``--resource-spec-file`` command-line argument or the +``RESOURCE_SPEC_FILE`` argument to :command:`ctest_test`, is what activates the +resource allocation feature. Tests should check the +``CTEST_RESOURCE_GROUP_COUNT`` environment variable to find out whether or not +resource allocation is activated. This variable will always (and only) be +defined if resource allocation is activated. If resource allocation is not +activated, then the ``CTEST_RESOURCE_GROUP_COUNT`` variable will not exist, +even if it exists for the parent ``ctest`` process. If a test absolutely must +have resource allocation, then it can return a failing exit code or use the +:prop_test:`SKIP_RETURN_CODE` or :prop_test:`SKIP_REGULAR_EXPRESSION` +properties to indicate a skipped test. + +.. _`ctest-resource-specification-file`: + +Resource Specification File +--------------------------- + +The resource specification file is a JSON file which is passed to CTest, either +on the :manual:`ctest(1)` command line as ``--resource-spec-file``, or as the +``RESOURCE_SPEC_FILE`` argument of :command:`ctest_test`. The resource +specification file must be a JSON object. All examples in this document assume +the following resource specification file: + +.. code-block:: json + + { + "version": { + "major": 1, + "minor": 0 + }, + "local": [ + { + "gpus": [ + { + "id": "0", + "slots": 2 + }, + { + "id": "1", + "slots": 4 + }, + { + "id": "2", + "slots": 2 + }, + { + "id": "3" + } + ], + "crypto_chips": [ + { + "id": "card0", + "slots": 4 + } + ] + } + ] + } + +The members are: + +``version`` + An object containing a ``major`` integer field and a ``minor`` integer field. + Currently, the only supported version is major ``1``, minor ``0``. Any other + value is an error. + +``local`` + A JSON array of resource sets present on the system. Currently, this array + is restricted to being of size 1. + + Each array element is a JSON object with members whose names are equal to the + desired resource types, such as ``gpus``. These names must start with a + lowercase letter or an underscore, and subsequent characters can be a + lowercase letter, a digit, or an underscore. Uppercase letters are not + allowed, because certain platforms have case-insensitive environment + variables. See the `Environment Variables`_ section below for + more information. It is recommended that the resource type name be the plural + of a noun, such as ``gpus`` or ``crypto_chips`` (and not ``gpu`` or + ``crypto_chip``.) + + Please note that the names ``gpus`` and ``crypto_chips`` are just examples, + and CTest does not interpret them in any way. You are free to make up any + resource type you want to meet your own requirements. + + The value for each resource type is a JSON array consisting of JSON objects, + each of which describe a specific instance of the specified resource. These + objects have the following members: + + ``id`` + A string consisting of an identifier for the resource. Each character in + the identifier can be a lowercase letter, a digit, or an underscore. + Uppercase letters are not allowed. + + Identifiers must be unique within a resource type. However, they do not + have to be unique across resource types. For example, it is valid to have a + ``gpus`` resource named ``0`` and a ``crypto_chips`` resource named ``0``, + but not two ``gpus`` resources both named ``0``. + + Please note that the IDs ``0``, ``1``, ``2``, ``3``, and ``card0`` are just + examples, and CTest does not interpret them in any way. You are free to + make up any IDs you want to meet your own requirements. + + ``slots`` + An optional unsigned number specifying the number of slots available on the + resource. For example, this could be megabytes of RAM on a GPU, or + cryptography units available on a cryptography chip. If ``slots`` is not + specified, a default value of ``1`` is assumed. + +In the example file above, there are four GPUs with ID's 0 through 3. GPU 0 has +2 slots, GPU 1 has 4, GPU 2 has 2, and GPU 3 has a default of 1 slot. There is +also one cryptography chip with 4 slots. + +``RESOURCE_GROUPS`` Property +---------------------------- + +See :prop_test:`RESOURCE_GROUPS` for a description of this property. + +.. _`ctest-resource-environment-variables`: + +Environment Variables +--------------------- + +Once CTest has decided which resources to allocate to a test, it passes this +information to the test executable as a series of environment variables. For +each example below, we will assume that the test in question has a +:prop_test:`RESOURCE_GROUPS` property of +``2,gpus:2;gpus:4,gpus:1,crypto_chips:2``. + +The following variables are passed to the test process: + +.. envvar:: CTEST_RESOURCE_GROUP_COUNT + + The total number of groups specified by the :prop_test:`RESOURCE_GROUPS` + property. For example: + + * ``CTEST_RESOURCE_GROUP_COUNT=3`` + + This variable will only be defined if :manual:`ctest(1)` has been given a + ``--resource-spec-file``, or if :command:`ctest_test` has been given a + ``RESOURCE_SPEC_FILE``. If no resource specification file has been given, + this variable will not be defined. + +.. envvar:: CTEST_RESOURCE_GROUP_<num> + + The list of resource types allocated to each group, with each item + separated by a comma. ``<num>`` is a number from zero to + ``CTEST_RESOURCE_GROUP_COUNT`` minus one. ``CTEST_RESOURCE_GROUP_<num>`` + is defined for each ``<num>`` in this range. For example: + + * ``CTEST_RESOURCE_GROUP_0=gpus`` + * ``CTEST_RESOURCE_GROUP_1=gpus`` + * ``CTEST_RESOURCE_GROUP_2=crypto_chips,gpus`` + +.. envvar:: CTEST_RESOURCE_GROUP_<num>_<resource-type> + + The list of resource IDs and number of slots from each ID allocated to each + group for a given resource type. This variable consists of a series of + pairs, each pair separated by a semicolon, and with the two items in the pair + separated by a comma. The first item in each pair is ``id:`` followed by the + ID of a resource of type ``<resource-type>``, and the second item is + ``slots:`` followed by the number of slots from that resource allocated to + the given group. For example: + + * ``CTEST_RESOURCE_GROUP_0_GPUS=id:0,slots:2`` + * ``CTEST_RESOURCE_GROUP_1_GPUS=id:2,slots:2`` + * ``CTEST_RESOURCE_GROUP_2_GPUS=id:1,slots:4;id:3,slots:1`` + * ``CTEST_RESOURCE_GROUP_2_CRYPTO_CHIPS=id:card0,slots:2`` + + In this example, group 0 gets 2 slots from GPU ``0``, group 1 gets 2 slots + from GPU ``2``, and group 2 gets 4 slots from GPU ``1``, 1 slot from GPU + ``3``, and 2 slots from cryptography chip ``card0``. + + ``<num>`` is a number from zero to ``CTEST_RESOURCE_GROUP_COUNT`` minus one. + ``<resource-type>`` is the name of a resource type, converted to uppercase. + ``CTEST_RESOURCE_GROUP_<num>_<resource-type>`` is defined for the product + of each ``<num>`` in the range listed above and each resource type listed in + ``CTEST_RESOURCE_GROUP_<num>``. + + Because some platforms have case-insensitive names for environment variables, + the names of resource types may not clash in a case-insensitive environment. + Because of this, for the sake of simplicity, all resource types must be + listed in all lowercase in the + :ref:`resource specification file <ctest-resource-specification-file>` and + in the :prop_test:`RESOURCE_GROUPS` property, and they are converted to all + uppercase in the ``CTEST_RESOURCE_GROUP_<num>_<resource-type>`` environment + variable. + See Also ======== .. include:: LINKS.txt + +.. _`CDash`: http://cdash.org/ |