3.1. QMTest Concepts

This section presents the concepts that underlie QMTest's design. By understanding these concepts, you will be able to better understand how QMTest works. In addition, you will find it easier to extend QMTest to new application domains.

3.1.1. Tests

A test checks for the correct behavior of the target application. What constitutes correct behavior will vary depending on the application domain. For example, correct behavior for a database might mean that it is able to retrieve records correctly while correct behavior for a compiler might mean that it generates correct object code from input source code.

Every test has a name that uniquely identifies the test, within a given test database. Test names must be composed entirely of lowercase letters, numbers, the "_" character, and the "." character. You can think of test names like file names. The "." character takes the place of "/"; it allows you to place a test in a particular directory. For example, the test name a.b.c names a test named c in the directory a.b. The directory a.b is a subdirectory of the directory a.

Every test is an instance of some test class. The test class dictates how the test is run, what constitutes success, and what constitutes failure. For example, the command.ExecTest class that comes with QMTest executes the target application and looks at its output. The test passes if the actual output exactly matches the expected output.

The arguments to the test parameterize the test; they are what make two instances of the same test class different from each other. For example, the arguments to command.ExecTest indicate which application to run, what command-line arguments to provide, and what output is expected.

Sometimes, it may be pointless to run one test unless another test has passed. Therefore, each test can have a set of associated prerequisite tests. If the prerequisite tests did not pass, QMTest will not run the test that depends upon them.

3.1.2. Resources

Some tests take a lot of work to set up. For example, a database test that checks the result of SQL queries may require that the database first be populated with a substantial number of records. If there are many tests that all use the same set of records, it would be wasteful to set up the database for each test. It would be more efficient to set up the database once, run all of the tests, and then remove the databases upon completion.

You can use a resource to gain this efficiency. If a test depends on a resource, QMTest will ensure that the resource is available before the test runs. Once all tests that depend on the resource have been run QMTest will destroy the resource.

Just as every test is an instance of a test class, every resource is an instance of a resource class. The resource class explains how to set up the resource and how to clean up when it is no longer needed. The arguments to the resource class are what make two instances of the same resource class different from each other. For example, in the case of a resource that sets up a database, the records to place in the database might be given as arguments. Every resource has a name, using the same format that is used for tests.

Under some circumstances (such as running tests on multiple machines at once), QMTest may create more than one instance of the same resource. Therefore, you should never depend on there being only one instance of a resource. In addition, if you have asked QMTest to run tests concurrently, two tests may access the same resource at the same time. You can, however, be assured that there will be only one instance of a particular resource on a particular target at any one time.

Tests have limited access to the resources on which they depend. A resource may place additional information into the context (Section 3.1.3) that is visible to the test. However, the actual resource object itself is not available to tests. (The reason for this limitiation is that for a target consisting of multiple processes, the resource object may not be located in the process as the test that depends upon it.)

Setting up or cleaning up a resource produces a result, just like those produced for tests. QMTest will display these results in its summary output and record them in the results file.

3.1.3. Context

When you create a test, you choose arguments for the test. The test class uses this information to run the test. However, the test class may sometimes need information that is not available when the test is created. For example, if you are writing compiler tests to verify conformance with the C programming language specification, you do not know the location of the C compiler itself. The C compiler may be installed in different locations on different machines.

A context gives users a way of conveying this kind of information to tests. The context is a set of key/value pairs. The keys are always strings. The values of all context properties provided by the user are strings. In general, all tests in a given use of QMTest will have the same context. However, when a resource is set up, it may place additional information in the context of those tests that depend upon it. The values inserted by the resource may have any type, so long as they can be "pickled" by Python.

All context properties whose names begin with "qmtest." are reserved for use by QMTest. The values inserted by QMTest may have any type. Test and resource classes should not depend on the presence or absence of these properites.

3.1.4. Test Results

A result is an outcome together with some annotations. The outcome indicates whether the test passed or failed. The annotations give additional information about the result, such as the manner in which the test failed, the output the test produced, or the amount of time it took to run the test.

3.1.4.1. Outcomes

The outcome of a test indicates whether it passed or failed, or whether some exceptional event occurred. There are four test outcomes:

  • PASS: The test succeeded.

  • FAIL: The test failed.

  • ERROR: A problem occurred in the test execution environment, rather than in the tested system. For example, this outcome is used when the test class attempted to run an executable in order to test it, but could not because the system call to create a new process failed.

    This outcome may also indicate a defect in QMTest or in the test class.

  • UNTESTED: QMTest did not attempt to execute the test. For example, this outcome is used when QMTest determines that a prerequisite test failed.

3.1.4.2. Annotations

An annotation is a key/value pair. Both the keys and values are strings. The value is HTML. When a test (or resource) runs it may add annotations to the result. These annotations are displayed by QMTest and preserved in the results file. If you write your own test class, you can use annotations to store information that will make your test class more informative.

3.1.5. Test Suite

A test suite is a collection of tests. QMTest can run an entire test suite at once, so by grouping tests together in a test suite, you make it easier to run a number of tests at once. A single test can be a member of more than one test suite. A test suite can contain other test suites; the total set of tests in a test suite includes both those tests included directly and those tests included as part of another test suite. Every test suite has a name, following the same conventions given above for tests and resources.

One use of test suites is to provide groups of tests that are run in different situations. For example, the nightly test suite might consist of those tests that should be run automatically every night, while the checkin test suite might consist of those tests that have to pass before any changes are made to the target application.

3.1.5.1. Implicit Test Suites

Section 3.1.1 explains how you may arrange tests in a tree hierarchy, using a period (".") as the path separator in test names. QMTest defines an implicit test suite for each directory. The name of these implicit test suites is the same as the name of the directory. The implicit test suite corresponding to a directory contains all tests in that directory or its subdirectories.

Consider, for example, a test database which contains tests with these names:

back_end.db_1
back_end.db2
front_end.cmdline
front_end.gui.widget_1
front_end.gui.widget_2

For this test database, QMTest defines implicit test suites with IDs back_end, front_end, and front_end.gui. The test suite front_end contains the tests front_end.cmdline, front_end.gui.widget_1, and front_end.gui.widget_2.

The suite named "." (a single period) is the implicit test suite corresponding to the root directory in the test database. This suite therefore contains all tests in the database. For example, the command


> qmtest run .
    
is equivalent to:

> qmtest run
    
Both commands run all tests in the database.

3.1.6. Test Database

A test database stores tests, test suites, and resources. When you ask QMTest for a particular test by name, it queries the test database to obtain the test itself. QMTest stores a test database in a single directory, which may include many files and subdirectories.

In general, QMTest can only use one test database at a time. However, it is possible to create a test database which contains other test databases. This mechanism allows you to store the tests associated with different parts of a large application in different test databases, and still combine them into a single large test database when required.

A single test database can store many different kinds of tests. By default, QMTest stores tests, resources, and test suites in the test database using subdirectories containing XML files. Generally, there should be no need to examine or modify these files directly. However, the use of an XML format makes it easy for you to automatically generate tests from another program, if required.

3.1.7. Targets

A target is QMTest's abstraction of a machine. By using multiple targets, you can run your tests on multiple machines at one. If you have many tests, and many machines, you can greatly reduce the amount of time it takes to run all of your tests by distributing the tests across multiple targets.

By default, QMTest uses only one target: the machine on which you are running QMTest. You may specify other targets by creating a target file, which lists the available targets and their attributes, and specifying the target file when you invoke qmtest.

Each target is a member of a single target group. All machines in the same target group are considered equivalent. A target group is specified by a string. If you are testing software on multiple platforms at once, the target group might correspond to machines running the same operating system. For example, all Intel 80386 compatible machines running GNU/Linux might be in the "i386-pc-linux-gnu" target group.

Section 3.6 describes how you specify and use targets with QMTest.