All QMTest functionality is available using the qmtest command.
These options can be used with any QMTest command, and must precede the command name on the command line.
All options are available in a "long form" prefixed with "--" (two hyphens). Some options also may be specified in a "short form" consisting of a single hyphen and a one-letter abbreviation. Short-form options may be combined; for example, -abc is equivalent to -a -b -c.
Use the test database located in the directory given by path. This flag overrides the value of the environment variable QMTEST_DB_PATH. If neither this flag nor the environment variable is specified, QMTest assumes that the current directory should be used as the database. See Section 3.1.6.
Display help information, listing commands and general options for the qmtest command.
Describe the version of QMTest in use.
Additional options are available for specific commands; these are presented with each command. Options specific to a command must follow the command on the command line. Specify the --help (-h) option after the command for a description of the command and a list of of available options for that command.
The qmtest create creates a new extension instance. For example, this command can be used to create a new test or resource. For a list of the kinds of extensions supported by QMTest, run qmtest extensions. The kind must be one of these extension kinds.
The descriptor specifies an extension class and (optionally) attributes for that extension class. The form of the descriptor is class(attributes), where the attributes are of the form attr = "val". If there are no attributes, the parentheses may be omitted.
The class may be either the path to an extension file or a QMTest class name in the form module.class. If the class is the path to an extension file (such as an existing test or resource file), the class name used is the one provided in the file; otherwise the class named used is the name provided on the command line.
The attributes used to construct the extension instance come from three sources: the attributes in the extension file (if the class is the path to an extension file), the --attribute options provided on the command line, and the explicit attributes provided in the descriptor. If multiple values for the the same attribute name are provided, the value used is taken from the first source in the following list for which there is a value: the rightmost attribute provided in the descriptor, the extension file, or the rightmost --attribute present on the command line.
The new extension file is written to the file specified with the --output option, or to the standard output if no --output is specified.
The create command accepts these options:
Set the target class argument name to value. The set of valid argument names and valid values is dependent on the extension class in use.
Write a description of the extension instance to file.
The qmtest create-target command creates a new target. A target is an entity that runs tests; normally, a target corresponds to a particular machine.
The target's name and class must be specified. An optional group may also be specified. When QMTest decides which target to use to run a particular tests, it will select a target that matches the test's requested target group.
The create-target command accepts these options:
Set the target class argument name to value. The set of valid argument names and valid values is dependent on the target class in use.
Write the target description to the indicated file. If there are already targets listed in file, they will be preserved, except that any target with the same name as the new target will be removed. If this option is not present, the file used will be the QMTest/targets file in the test database directory.
The qmtest create-tdb command creates a new, empty test database. A test database is a directory in which QMTest stores configuration files, tests, and other data. Certain test database classes may also store data elsewhere, such as in an external relational database.
The test database is created in the directory specified by --tdb (-D) option or by setting the QMTEST_DB_PATH environment variable. If no database path is specified, QMTest assumes that the current directory is the test database.
By default, QMTest creates a new test database that uses the standard XML-based implementation. (See Section 4.5 for information about writing a test database class.)
The create-tdb command accepts these options:
Set the database attribute name to value. The set of attribute names and valid values is dependent on the database class in use. The default database class accepts no attributes.
Use the test database class given by class. The class may have the general form described in Section 3.4.2. Once you create a test database, you cannot change the test database implementation it uses. If you do not use this option, QMTest will use the default test database implementation, which uses an XML file format to store tests.
The qmtest gui starts the graphical user interface. The graphical user interface is accessed through a web browser. You must have a web browser that supports JavaScript to use the graphical interface. QMTest has been tested with recent versions of Internet Explorer and Netscape Navigator. Other web browsers may or may nor work with QMTest.
The gui command accepts these options:
Bind the server to the indicated internet address, which should be a dotted quad. By default, the server binds itself to the address 127.0.0.1, which is the address of the local machine. If you specify another address, the server will be accessible to users on other machines. QMTest does not perform any authentication of remote users, so you should not use this option unless you have a firewall in place that blocks all untrusted users.
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
Run the QMTest GUI as a daemon. In this mode, QMTest will detach from the controlling terminal and run in the background until explicitly shutdown.
For details about this option, see the description of the qmtest run command.
Do not attempt to start a web browser when starting the GUI. QMTest will still print out the URL at which the server can be accessed. You can then connect to this URL manually using the browser of your choice.
For details about this option, see the description of the qmtest run command.
Specify the path to which the QMTest GUI will write its process ID. This option is useful if you want to run QMTest as a daemon. If this option is not provided, no PID file is written. If you specify this option, but path is the empty string, QMTest will check the .qmrc configuration file for a pid-file entry. If there is no such entry, QMTest will use an appropriate platform-specific default value.
Specify the port on which the QMTest GUI will listen for connections. If this option is not provided, QMTest will select an available port automatically.
For details about this option, see the description of the qmtest run command.
The qmtest extensions lists available extension classes and provides a brief description of each class. You can use this command to list all of the available extension classes, or to list all of the available extension classes of a particular type. For example, you can use this command to list all of the available test classes.
The extensions command accepts these options:
List the available extension classes of the indicated kind. The kind must be one of test, resource, target, or database.
The qmtest register registers an extension class with QMTest. As part of this process, QMTest will load your extension class. If the extension class cannot be loaded, QMTest will tell you what went wrong.
QMTest will search for your extension class in the directories it would search when running tests, including those given by the environment variable QMTEST_CLASS_PATH.
The kind argument tells QMTest what kind of extension class you are registering. If you invoke qmtest register with no arguments it will provide you with a list of the available extension kinds.
The class-name argument gives the name of the class in the form module.Class. QMTest will look for a file whose basename is the module name and whose extension is either py, pyc, or pyo.
The qmtest run command runs tests and displays the results. If no test or suite names are specified, QMTest runs all of the tests in the test database. If test or suite names are specified, only those tests or suites are run. Tests listed more than once (directly or by inclusion in a test suite) are run only once.
The run command accepts these options:
Add a property to the test execution context. The name of the property is name, and its value is set to the string value.
This option may be specified multiple times.
Read properties for the test execution context from the file file.
The file should be a text file with one context property on each line, in the format name=value. Leading and trailing whitespace on each line are ignored. Also, blank lines and lines that begin with "#" (a hash mark) are ignored as comments.
This option may be specified more than once, and used in conjunction with the --context option. All of the context properties specified are added to the eventual context. If a property is set more than once, the last value provided is the one used.
If this option is not specified, but a file named context exists in the current directory, that file is read. The properties specified in this file are processed first; the values in this file can be overridden by subsequent uses of the --context option on the command line.
Control the format used when displaying results. The format specified must be one of full, brief, stats, batch, or none. The brief format is the default if QMTest was invoked interactively; the batch format is the default otherwise. In the full format, QMTest displays any annotations provided in test results. In the brief mode only the causes of failures are shown; detailed annotations are not shown. In the stats format, no details about failing tests are displayed; only statistics showing the number of passing and failing tests are displayed. In the batch mode, the summary is displayed first, followed by detailed results for tests with unexpected outcomes. In the none mode, no results are displayed, but a results file is still created, unless the --no-output option is also provided.
Run tests in multiple count concurrent processes on the local computer. On multiprocessor machines, the processes may be scheduled to run in parallel on different processors. QMTest automatically collects results from the processes and presents combines test results and summary. By default, one process is used.
This option may not be combined with the --targets (-T) option.
Do not produce a test results file.
Write full test results to file. Specify "-" (a hyphen) to write results to standard output. If neither this option nor --no-output is specified, the results are written to the file named results.qmr in the current directory.
Treat file as a set of expected outcomes. The file must have be a results file created either by qmtest run, or by saving results in the graphical user interface. QMTest will expect the results of the current test run to match those specified in the file and will highlight differences from those results.
Run the tests in a random order.
This option can be used to find hidden dependencies between tests in the testsuite. (You may not notice the dependencies if you always run the tests in the same order.)
Rerun only those tests that had unexpected outcomes.
The tests run are determined as follows. QMTest starts with all of the tests specified on the command line, or, if no tests are explicitly specified, all of the tests in the database. If no expectations file is specified (see the description of the --outcomes option), then all tests that passed in the results file indicated by the --rerun option are removed form the set of eligible tests. If an expectations file is specified, then the tests removed are tests whose outcome in the results file indicated by the --rerun option is the same as in the expectations file.
The --rerun provides a simple way of rerunning failing tests. If you run your tests and notice failures, you might try to fix those failing tests. Then, you can rerun the failing tests to see if you succeeded by using the --rerun option.
Specify an additional output result stream. The descriptor is in the format described in Section 3.4.2.
If the --random is used, QMTest randomizes the order in which tests are run, subject to the constraints described in Section 3.2.1. By default, the random number generator is seeded using the system time.
For debugging purposes, it is sometimes necessary to obtain a reproducible sequence of tests. Use the --seed option to specify the seed for the random number generator.
Note that even with the same random number seed, if tests are run in parallel, scheduling uncertainty may still produce variation in the order in which tests are run.
Use targets specified in target specification file file. If this option is not present, the QMTest/targets in the test database directory will be used. If that file is not present, the tests will be run in serial on the local machine.
The qmtest summarize extracts information stored in a results file and displays this information on the console. The information is formatted just as if the tests had just been run, but QMTest does not actually run the tests.
The summarize command accepts the following options:
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
Specify an additional output result stream. The descriptor is in the format described in Section 3.4.2.
QMTest recognizes the following environment variables:
If this environment variable is set, it should contain a list of directories in the same format as used for the system's PATH environment variable. These directories are searched (before the directories that QMTest searches by default) when looking for extension classes such as test classes and database classes.
If this environment variable is set, its value is used as the location of the test database, unless the --tdb (-D) option is used. If this environment variable is not set and the --tdb option is not used, the current directory is used as the test database.
These configuration variables are used by QMTest. You should define them in the [qmtest] section of your QM configuration file.
The default path to use when creating a PID file with the --pid-file option. (See Section 3.4.5 for more information about this option.) If this entry is not present, an appropriate platform-specific default value is used.
If QMTest successfully performed the action requested, QMTest returns 0. For the qmtest run or qmtest summarize commands, success implies not only that the tests ran, but also that all of the tests passed (if the --outcomes option was not used) or had their expected outcomes (if the --outcomes option was used).
If either the run command or the summarize command was used, and at least one test failed (if the --outcomes option was not used) or had an unexpected outcome (if the --outcomes option was used), qmtest returns 1.
If QMTest could not perform the action requested, qmtest returns 2.