NAME

test-driver — scaffolding for executing test(3) based tests from the command-line

SYNOPSIS

test-executable

[-c artefact-archive-name] [-l] [-n run-name] [-p search-path-directory] [-R runtime-base-directory] [-s execution-style] [-t test-selector] [-T seconds] [-v]

DESCRIPTION

The test-driver library provides a main() function that will execute the test(3) based tests in an executable according to the options specified on the command-line. The test-driver library usually used in conjunction with code generated by the make-test-scaffolding(1) utility.

Test executables built using test-driver recognize the following command-line options:

-c archive-name

If this option is specified, then the test-driver provided scaffolding will copy test outputs and other artefacts from the test run to the archive named by argument archive-name. The format of the archive is specified by the path name suffix of the archive name. The supported output formats are those supported by libarchive(3).

-l

If this option is specified, then the test-driver utility will list the selected tests and exit.

-n run-name

Use the specified test run name in status messages and to name any files and directories created during the test run. If this option is not specified, then the base name of the test executable is used.

-p search-path-directory

Add the argument search-path-directory to the list of directories searched for by test(3) utility functions.

-R runtime-base-directory

Set the runtime base directory to the directory specified by the argument runtime-base-directory. Tests execute with their current directory set to a subdirectory within this directory. The path specified by argument runtime-base-directory must exist, and must name a directory.

If this option is not specified, then the TEST_TMPDIR environment variable will be examined. If set to a non-empty value, then its value will be used. Otherwise, the value of the TMPDIR environment variable will be used, if non-empty. If neither of the environment variables TEST_TMPDIR and TMPDIR contain a non-empty value, then the path “/tmp” will be used.

-s execution-style

Set the desired execution style to that specified by argument execution-style. Legal values for execution-style are:

atf: Be compatible with NetBSD atf(9).
tap: Be compatible with TAP (Test Anything Protocol).
test: Be compatible with libtest (this test framework).

The default is to use libtest semantics.

-t test-selector

Select (or deselect) tests to execute according to the argument test-selector.

Test selectors are specified using the following syntax:

[-]c:pattern: Select test cases whose names match pattern. Selecting a test case will cause all of its contained test functions to be selected.
[-]f:pattern: Select test functions whose names match pattern.
[-]t:pattern: Select the test cases and test functions associated with tags matching pattern.
[-]pattern: If the c, f or t qualifiers were not specified, then the pattern is matched against the names of test cases.

The pattern fields of test selectors use shell wildcard syntax, as implemented by fnmatch(3).

If no test selectors are specified then all the tests present in the test executable will be run. Otherwise, the test selectors specified are processed in the order specified on the command line.

A test selector that does not start with a “-” will add the entries that it matches to the currently selected list of tests. A test selector that starts with a “-” will remove the entries that it matches from the currently selected list of tests.

If at least one test selector was specified, and if the result of applying the specified test selectors was an empty list of tests, then the test-driver library will exit with an error message.

-T seconds

Set the timeout for individual tests to seconds. If a test function fails to return with the specified number of seconds then it is treated as having failed. The default is to wait indefinitely for the test function to complete.

-v

Increase verbosity level by 1. The default verbosity level is 0.

Link-time Pre-requisites

The test-driver library expects the following symbols to be present in the test executable it is linked with:

struct test_case_descriptor test_cases[]: An array of test cases descriptors. Test case descriptors described by test_case(5).
int test_case_count: The number of entries in the test_cases array.

Test Execution

At start up, the main() function provided by test-driver will select tests (and test cases) to execute, based on the test selection options specified.

For each selected test case, test execution proceeds as follows:

The runtime directory for the test case is created.
The test process forks, with test execution continuing in the child.
(Child) The current directory of the process is changed to the runtime directory.
(Child) The test case's set up function is then executed. If this function returns an error then test case execution is aborted.
(Child) Each selected test function in the test case is then executed and its status is output to stdout (or stderr) according to the test execution style selected.
(Child) The test case's tear down function is then executed.
If test artefacts need to be preserved, then these are copied to the specified archive.
The test's runtime directory is then deleted.

After all test cases have been attempted, the main() function exits with the exit code appropriate for the test execution style selected.

EXAMPLES

To run all tests in the binary named tc_example, copying test artefacts to a cpio(1) archive named /tmp/tc_example.cpio, use:

tc_example -c /tmp/tc_example.cpio

To execute tests in the test case “tc1” alone, use:

tc_example -t 'c:tc1'

To execute tests in the test case “tc1” but not the test functions associated with tag tag1, use:

tc_example -t 'c:tc1' -t '-t:tag1'

DIAGNOSTICS

Test programs built with the test-driver library will exit with an exit code of 0 if all of the selected tests passed when run, and with a non-zero exit code if an error occurred during test execution.