To run one or more tests, use the qmtest run command. Each invocation of the qmtest run command is a single test run, and produces a single set of test results and statistics. Specify as arguments the names of tests and test suites to run. Even if you specify a test more than once, either directly or by incorporation in a test suite, QMTest runs it only once.
If you wish to run all tests in the test database, use the implicit test suite . (a single period; see Section 3.1.5.1), or omit all IDs from the command line.
QMTest can run tests in multiple concurrent threads of execution or on multiple remote hosts. See the documentation for the run command for details.
Given one or more input test names and test suite names, QMTest employs the following procedure to determine which tests and resources to run and the order in which they are run.
QMTest resolves test names and test suite names. Test suites are expanded into the tests they contain. Since test suites may contain other test suites, this process is repeated until all test suites have been expanded. The result is a set of tests that are to be run.
QMTest computes a schedule for running the tests to be run such that a test's prerequisites are run before the test itself is run. Prerequisites not included in the test run are ignored. Outside of this condition, the order in which tests are run is undefined.
If QMTest is invoked to run tests in parallel or distributed across several targets, the tests are distributed among them as well. QMTest does not guarantee that a test's prerequisites are run on the same target, though. On each target, tests are assigned to the next available concurrent process or thread.
QMTest determines the required resources for the tests to be run. If several tests require the same resource, QMTest attempts to run all of the tests on the same target. In this case, the resource is set up and cleaned up only once. In some cases, QMTest may schedule the tests on multiple targets; in that case, the resource is set up and cleaned up once on each target.
In some cases, a test, resource setup function, or resource cleanup function is not executed:
A test specifies for each of its prerequisite tests an expected outcome. If the prerequisite is included in the test run and the actual outcome of the prerequisite test is different from the expected outcome, the test is not run. Instead, it is given an UNTESTED outcome.
If a test's prerequisite is not included in the test run, that prerequisite is ignored.
If a setup function for one of the resources required by a test fails, the test is given an UNTESTED outcome.
The cleanup function of a resource is run after the last test that requires that resource, whether or not that test was run. The cleanup function is run even if the setup function failed.
QMTest passes a context object to the Run method of a test that is run and to the SetUp method of a resource.
Most of the properties of the context are the same for all tests and resource functions run during a single test run. These properties are configured as part of the test run. For example, when you run tests using the qmtest run command, you may specify individual context properties with the --context (-c) or --load-context (-C) options.
In addition, a resource setup function may add additional properties to a context. These added properties do not become part of the common context; they are hidden from other tests and resources except that the properties added by a resource are visible to tests that require that resource.
For instance, a resource SetUp function might allocate the resource and place a handle to it (for instance, a temporary directory name or a database session key) in the context as a context property. Tests that require that resource have access to the temporary resource via the handle stored in the context. The resource's cleanup function also uses the handle to deallocate the resource. That information should be stored in the resource object itself since no context is made available to the CleanUp.