Activity:  Implement Test

Select appropriate implementation technique

Select the most appropriate technique to implement the test. For each test that you want to conduct, consider implementing at least one Test Script. In some instances, the implementation for a given test will span multiple Test Scripts. In others, a single Test Script will provide the implementation for multiple tests.

Typical methods for implementing Test Scripts include the manual, programming, recording and generation. Each method is discussed in the following sections.

As with most approaches, we recommend you'll get more useful results if you use a mixture of the following techniques. While you don't need to use them all, you shouldn't confine yourself to using a single technique either.

Manual Test Scripts

Many tests make sense to be conducted manually. Usability tests are an area where manual testing is in most cases a better solution than an automated one. Also tests that require validation of the accuracy and quality of the physical outputs from a software system generally require manual validation. As a general heuristic, it's a good idea to begin the first tests of a particular Target Test Item with a manual implementation; this approach allows the tester to learn about the target item, adapt to unexpected behavior from it, and apply human judgment to determine the next appropriate action to be taken.

Sometimes manually conducted tests will be subsequently automated and reused as part of a regression testing strategy. Note however that it isn't necessary or desirable—or even possible—to automate every test that you could otherwise conduct manually. Automation brings certain advantages in speed and accuracy of test execution, visibility and collation of detailed test outcomes and in efficiency of creating and maintaining complex tests, but like all useful tools, it isn't the solution to all your needs.

Automation comes with certain disadvantages: these basically amount to an absence of human judgment and reasoning during test execution. Current automation solutions don't have the cognitive abilities that a human does—and it's arguably unlikely that they ever will. During implementation of a manual test, human reasoning can be applied to the observed system responses to stimulus. Automated test techniques and their supporting tools typically have minimal ability to notice the implications of certain system behaviors, and currently no ability to infer possible problems through deductive reasoning.

Programmed Test Scripts

Arguably the method of choice practiced by most testers who use test automation. In it's purest form, this practice is performed in the same manner and using the same general principles as software programming. As such, most methods and tools used for software programming are generally applicable and useful to test automation programming.

Using either a standard software development environment (such as Microsoft Visual Studio or IBM Visual Age) or a specialized test automation development environment (such as the IDE provided with Rational Robot), the tester is free to harness the features and power of the development environment to best effect.

The negative aspects of programming automated tests are related to the negative aspects of programming itself as a general technique. For programming to be effective, some consideration should be given to appropriate design: without this the implementation will likely fail. If the developed software will likely be modified by different people over time—the usual situation—then some consideration must be given to adopting a common style and form to be used in program development, and ensuring it's correct use. Arguably the two most important concerns relate to the misuse of this technique.

First, there is a risk that a tester will become engrossed in the features of the programming environment, and spend too much time crafting elegant and sophisticated solutions to problems that could be achieved by simpler means. The result is that the tester wastes precious time on what are essentially programming tasks to the detriment of time that could be spent actually testing and evaluating the Target Test Items. It requires both discipline and experience to avoid this pitfall.

Secondly, there is the risk that the program code used to implement the test will itself have bugs introduced through human error or omission. Some of these bugs will be easy to debug and correct in the natural course of implementing the automated test: others won't. Just as errors can be elusive to detect in the Target Test Item, it can be equally difficult to detect errors in test automation software. Furthermore, errors may be introduced where algorithms used in the automated test implementation are based on the same faulty algorithms used by the software implementation itself. This results in errors going undetected, hidden by the false security of automated tests that apparently execute successfully. Mitigate this risk by using different algorithms in the automated tests wherever possible.

Recorded or captured Test Scripts

There are a number of test automation tools that provide the ability to record or capture human interaction with a software application and produce a basic Test Script. There are a number of different tool solutions for this. Most tools produce a Test Script implemented in some form of a high-level, normally editable, programming language. The most common designs work in one of the following ways:

• by capturing the interaction with the client UI of an application based on intercepting the inputs sent from the client hardware peripheral input devices: mouse, keyboard and so forth to the client operating system. In some solutions, this is done by intercepting high-level messages exchanged between the operating system and the device driver that describe the interactions in a somewhat meaningful way; in other solutions this is done by capturing low-level messages, often based at the level of time-based movements in mouse coordinates or key-up and key-down events.
• by intercepting the messages sent and received across the network between the client application and one or more server applications. The successful interpretation of those messages relies typically on the use of standard, recognized messaging protocols, such as HTTP, SQL, Tuxedo and so forth. Some tools also allow the capture of "base" communications protocols such as TCP/IP, however it can be more complex to work with Test Scripts of this nature.

While these techniques are generally useful to include as part of your approach to automated testing, some practitioners feel these techniques have limitations. One of the main concerns is that some tools simply capture application interaction and do nothing else. Without the additional inclusion of observation points that capture and compare system state during subsequent script execution, the basic Test Script cannot be considered to be a fully-formed test. Where this is the case, the initial recording will need to be subsequently augmented with additional custom program code to implement observation points within the Test Script.

Various authors have published books and essays on this and other concerns related to using test procedure record or capture as a test automation technique. To gain a more in-depth understanding of these issues, we recommend reviewing the work available on the Internet by the following authors: James Bach, Cem Kaner, Brian Marick and Bret Pettichord, and the relevant content in the book Lessons Learned in Software Testing [KAN99]

Generated Tests

Some of the more sophisticated test automation software enables the actual generation of various aspects of the test—either the procedural aspects or the Test Data aspects of the Test Script—based on generation algorithms. This type of automation can play a useful part in your test effort, but shouldn't be considered as the only approach used. Both Rational TestFactory and the Rational TestManager datapool generation feature are examples of implementations of this type of technology.

Set up test environment preconditions

Setup the test environment to ensure that all the needed components (hardware, software, tools, data, etc.) have been implemented and are in the test environment, ready in the correct state to enable the tests to be conducted. Typically this will involve some form of basic environment reset (e.g. Registry and other configuration files), restoration of underlying databases to known state, and so forth in addition to tasks such as loading paper into printers. While some tasks can be performed automatically, some aspects typically require human attention.

(Optional) Manual walk-through of the test

Especially applicable to automated Test Scripts, it can be beneficial to initially walk-through the test manually to confirm expected prerequisites are present. During the walk-through, you should verify the integrity of the environment, the software and the test design. The walk-through is most relevant where you are using an interactive recording technique, and least relevant where you are programming the Test Script.

Where the software is known to be sufficiently stable or mature, you way elect to skip this step where you deem the risk of problems occurring in the areas the manual walk-through addresses are relatively low.

Identify and confirm appropriateness of Test Oracles

During this walk-through, confirm that the Test Oracles you plan to use are appropriate. Where they have not already been identified, now is the time for you to do so.

You should try to confirm through alternative means that the chosen Test Oracle(s) will provide accurate and reliable results. For example, if you plan to validate test results using a field displayed via the application's GUI that indicates a database update has occurred, consider independently querying the back-end database to verify the state of the corresponding records in the database. Alternatively, you might ignore the results presented in an update confirmation dialog, and instead confirm the update by querying for the record through an alternative front-end function or operation.

Reset test environment and tools

Next you should restore the environment back to it's original state. Typically this will involve some form of basic operating environment reset (e.g. Registry and other configuration files), restoration of underlying databases to a known state, and so forth in addition to tasks such as loading paper into printers. While some reset tasks can be performed automatically, some aspects typically require human attention.

Set the implementation options of the supporting tools. Depending on the sophistication of the tool, this may be many options to consider. Failing to set these options appropriately may reduce the usefulness and value of the resulting test assets. Where possible, you should try to store these tool options and settings so that they can be reloaded easily based on one or more predetermined profiles.

In the case of automated test implementation tools, there may be many different settings to be considered. In the case of manual testing, it may be a simple matter of partitioning a new entry in a support system for logging results, signing into issue and changes request logging systems.

Implement the test

Using the Test-Ideas List, one or more selected Test Cases or the Workload Analysis Model, begin to implement the test. Start by giving the test a uniquely identifiable name (if it does not already have one) and prepare the IDE, capture tool or document to begin recording the specific steps of the test. Work through the following sections as many times as are required to implement the test.

Implement navigation actions

Program, record or generate the required navigation actions. Start by selecting your appropriate navigation method of choice. For most classes of system these days, a "Mouse" or other pointing device is the preferred and primary medium for navigation. For example, the pointing and scribing device used with a Personal Digital Assistants (PDA) is conceptually equivalent to a Mouse.

The secondary navigation means is generally that of keyboard interaction. In most cases, navigation will be made up of a combination of mouse-driven and keyboard-driven actions.

In some cases, you will need to consider voice-activated, light, visual and other forms of recognition. These can be more troublesome to automate tests against, and may require the addition of special test extensions to the application to allow audio and visual elements to be loaded and processed from file rather than captured dynamically.

In some situations, you may want to—or need to—perform the same test using multiple navigation methods. There are different approaches you can take to achieve this, for example: automate all the tests using one method and manually perform all or some subset of the tests using others; separate the navigation aspects of the tests from the Test Data that characterize the specific test, providing and building a logical navigation interface that allows either method to be selected to drive the test; simply mix and match navigation methods.

Implement observation points

At each point in the Test Script where an observation should be taken, use the appropriate Test Oracle to capture the desired information. In many cases, the information gained from the observation point will need to be recorded and retained to be referenced during subsequent control points.

Where this is an automated test, decide how the observed information should be reported from the Test Script. In most cases it usually appropriate simply to record the observation in a central Test Log relative to it's delta-time from the start of the Test Script; in other cases specific observations might be output separately to a spreadsheet or data file for more sophisticated uses.

Implement control points

At each point in the Test Script where a control decision should be taken, obtain and assess the appropriate information to determine the correct branch for the flow of control to follow. The data retrieved form prior observation points are usually input to control points.

Where a control point occurs, and a decision made about the next action in the flow-of-control, we recommend you record the input values to the control point, and the resulting flow that is selected in the Test Log.

Resolve implementation errors

During test implementation, you'll encounter errors, omissions and other issues that need to be resolved before the test can be implemented completely. Identify each error you encounter and work through addressing them.

In the case of test automation, this might include completion errors due to undeclared variables and functions, or invalid use of those functions. Work your way through the error listings from the compiler and other sources until the Test Script is free of syntactical and other obvious errors.

Establish external data sets

In many cases it's more appropriate to maintain your Test Data external to the Test Script. This provides flexibility, simplicity and security in Test Script and Test Data maintenance. External data sets provide value to test in the following ways:

• Test Data is external to the Test Script eliminating hard-coded references in the Test Script
• external Test Data can be modified easily, usually with minimal Test Script impact
• additional Test Cases can easily be supported by the Test Data with little or no Test Script modifications
• external Test Data can be shared with many Test Scripts
• Test Scripts can be developed to use external Test Data to control the conditional branching logic within the Test Script.

Recover test environment to known state

Setup tools and initiate test execution

When you have completed the basic implementation of the Test Script, it should be tested to ensure it implements the individual tests appropriately and that they execute properly.

It's a good idea to perform this step using the same Build version of the software used to implement the Test Scripts. This eliminates the possibility of problems due to introduced errors in subsequent builds.

Resolve execution errors

It's pretty common that things done and approaches used during implementation will need some degree of "tweaking" to adjust the test to run in one or more Test Environment Configurations.

Be prepared to spend some time checking and the tests "function within tolerances" and adjusting them until they do before you declare the test as implemented. While you can delay step until later, we recommend that you don't: otherwise you could end up with a significant backlog of failures to subsequently be addressed.

Restore test environment to known state

While this step might seem trivial, but it's an important good habit to form to work effectively with the other testers on the team—especially where the implementation environment is shared. It's also important to establish a routine that makes thinking of the system state second nature.

While in a primarily manual testing effort, it's often simple to identify and fix environment restore problems, remember that test automation has much less ability to tolerate unanticipated problems with environment state.

Maintain traceability relationships

Using the Traceability requirements outlined in the Test Plan, update the traceability relationships as required.

Evaluate and verify your results

Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".

Have the people performing the downstream activities that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input artifacts to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input artifact review your work on this basis.

Try to remember that that RUP is an iterative process and that in many cases artifacts evolve over time. As such, it is not usually necessary—and is often counterproductive—to fully-form an artifact that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the artifact will change—and the assumptions made when the artifact was created proven incorrect—before the artifact is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.

Copyright  © 1987 - 2001 Rational Software Corporation

Rational Unified Process