Concepts: Test-Ideas List

Disciplines > Test > Concepts > Test-Ideas List

Topics

Introduction

Information used in designing tests is gathered from many places: design models, classifier interfaces, statecharts, and code itself. At some point, this source document information must be transformed into executable tests:

Specific inputs given to the software under test,
in a particular hardware and software configuration,
initialized to a known state,
with specific results expected.

It is possible to go directly from source document information to executable tests. But it's often useful to add an intermediate step. In it, test ideas are written into a Test-Ideas List. The list is used to create executable tests.

What are Test Ideas?

A test idea (AKA test requirement) is a brief statement about a test that could be performed. As a simple example, let's take a function that calculates a square root. Here's a list of test ideas:

give as input a number that's barely less than zero
give zero as the input
test a number that's a perfect square, like 4 or 16 (is the result exactly 2 or 4?)

Each of these could readily be converted into an executable test with exact descriptions of inputs and expected results.

There are two advantages to this less-specific intermediate form. One is that test ideas are more reviewable and understandable than complete tests - it's easier to understand the reasoning behind them. The other is that test ideas support more powerful tests, as described below in Test design using the list.

The square root examples all describe inputs, but test ideas can describe any of the elements of an executable test. For example, "print to a LaserJet IIIp" describes the test configuration, while "test with database full" describes system setup and initialization. These latter test ideas are very incomplete in themselves: Print what to the printer? Do what with that full database? They do, however, ensure that important ideas aren't forgotten, ideas that will be fleshed out later in test design.

Test ideas are often based on fault models, notions of which faults are plausible in software and how those faults can best be uncovered. For example, consider boundaries. It's safe to assume the square root function will be implemented something like this:

double sqrt(double x) {
    if (x < 0) 
      // signal error
    ...

It's plausible that the < will be mistyped as <=. People make that kind of mistake all the time, so it's worth checking for. The fault cannot be detected with X having the value 2, because both the incorrect expression (x<=0) and the correct expression (x<0) will take the same branch of the if statement. Similarly, giving X the value -5 cannot find the fault. The only way is to give X the value 0. That justifies the second test idea.

In this case, the fault model is explicit. In other cases, it's implicit. For example, whenever a program manipulates a linked structure, it's good to test it against a circular one. There are many possible faults that could lead to a mishandled circular structure. For the purposes of testing, they needn't be enumerated - it suffices to know that some fault is likely enough that the test is worth running.

The following links provide information on getting test ideas from different kinds of fault models. The first two are explicit fault models; the last one uses implicit ones.

These fault models can be applied to many different artifacts. For example, the first one describes what to do with boolean expressions. Such expressions can be found in code, in guard conditions in statecharts and sequence diagrams, and in natural-language descriptions of method behaviors (such as might be found in a published API).

It's also occasionally helpful to have guidelines for specific artifacts. Here's what's available:

Guidelines: Test Ideas for Statechart and Flow Diagrams

Notice that a particular Test-Ideas List may contain test ideas from many fault models. It may also contain test ideas derived from more than one artifact.

Test design using the list

Let's suppose you're designing tests for a method that searches for a string in a sequential collection. It can either obey case or ignore case in its search, and it returns the index of the first match found or -1 if no match is found.

int Collection.find(String string,
                    Boolean ignoreCase);

Here are some test ideas for this method:

match found in the first position
match found in the last position
no match found
two or more matches in the collection
case is ignored; match found, but it wouldn't match if case were obeyed
case is obeyed and an exact match is found
case is obeyed; a string that would have matched if case were ignored is skipped

It would be simple to implement seven tests, one for each test idea. However, different test ideas can be combined into a single test. For example, this test satisfies test ideas 2, 6, and 7:

Setup: collection initialized to ["dawn", "Dawn"]
Invocation: collection.find("Dawn", false)
Expected result: return value is 1 (it would be 0 if "dawn" were not skipped)

(Notice that making test ideas nonspecific makes them easier to combine.)

It's possible to satisfy all the test ideas in three tests. Why would three tests that satisfy seven test ideas be better than seven separate tests?

When creating a large number of simple tests, it's common to create test N+1 by copying test N and tweaking it just enough to satisfy the new test idea. The result, especially in more complex systems, is that test N+1 probably exercises the program in almost the same way as test N: it takes almost exactly the same paths through the code.

A smaller number of tests, each satisfying several test ideas, doesn't allow a "copy and tweak" approach. Each test will be rather different from the last, exercising the code in different ways, taking different paths.

Why would that be better? If the Test-Ideas List were complete, with a test idea for every fault in the program, it wouldn't matter how you wrote the tests. But the list is always missing some test ideas that could find bugs. By having each test do very different things from the last one—by adding seemingly-unneeded variety—you increase the chance that one of the tests will stumble over a bug by sheer dumb luck. In effect, smaller, more complex tests increase the chance the test will satisfy a test idea that you didn't know you needed.
Sometimes when creating more complex tests, new test ideas pop into your mind. That happens less often with simple tests, because so much of what you're doing is exactly like the last test that your mind gets dulled.

There are reasons for not creating complex tests, though:

If each test satisfies a single test idea, and the test for idea 2 fails, you know immediately the most likely cause: the program doesn't handle a match in the last position. If a test satisfies ideas 2, 6, and 7, isolating the failure is harder.
Complex tests are more difficult to understand and maintain. It's less obvious what the test is testing.
Complex tests are more difficult to create. Constructing a test that satisfies five test ideas often takes more time than constructing five tests that each satisfy one. Moreover, it's easier to make mistakes—to think you're satisfying all five when you're only satisfying four.

In practice, you must find a reasonable balance between complexity and simplicity. For example, the first suite of tests you subject the system to (the smoke tests) should be simple, easy to understand and maintain, intended to catch the most obvious problems. Later tests should be more complex, but not so complex they are unmaintainable.

After you've finished a set of tests, it's good to check them against the characteristic test design mistakes discussed in Concepts: Developer Testing.

Using test ideas before testing

A Test-Ideas List is useful for reviews and inspections of design artifacts. For example, consider this part of a design model:

Fig1: Association between Department and Employee Classes

The rules for creating test ideas from such a model would ask you to consider the case where a department has many employees. By walking through a design and asking "what if, at this point, the department has many employees?", you might discover design or analysis errors. For example, you might realize that only one employee at a time can be transferred between departments. That might be a problem if the corporation is prone to sweeping reorganizations in which many employees need to be transferred.

Such faults, cases where a possibility was overlooked, are called faults of omission. Just like the faults themselves, you have probably omitted tests that detect these faults from your testing effort. (See, for example, [GLA81], [OST84], [BAS87], [MAR00], and other studies that show how often faults of omission escape into deployment.)

The role of testing during design is further discussed here: Concepts: Test-first Design.

Test ideas and Traceability

Traceability is a matter of tradeoffs. Is its value worth the cost of maintaining it? This question needs to be considered during Activity: Define Assessment and Traceability Needs.

When traceability is worthwhile, it's conventional to trace tests back to the artifacts that inspired them. For example, you might have traceability between an API and its tests. If the API changes, you know which tests to change. If the code that implements the API changes, you know which tests to run. If a test puzzles you, you can find the API it's intended to test.

The Test-Ideas List adds another level of traceability. You can trace from a test to the test ideas it satisfies, and then from the test ideas to the original artifact.

Rational Unified Process