Satisfice, Inc.

James Bach, Principal

http://www.satisfice.com [email protected]
Copyright (c) 1999, Satisfice, Inc. Version 1.12 9/25/99

Test Plan Evaluation Model

The answer to the question “How good is this test plan?” can only be given with reference to an idea of
what a test plan should be. Although there are a number of public standards that specify test plan document
formats, they provide little basis for distinguishing a better plan from a worse plan. This model identifies
basic concepts, functions that a test plan serves, criteria that a test plan should satisfy, and some heuristics
to assist in determining if the criteria are satisfied with respect to the functions:

Terms and Concepts

Test Plan. The test plan is the set of ideas that guide or represent the intended test process.
Often those ideas are only partially documented, spread across multiple documents, and
subject to change as the project evolves.

Test Plan Document. A test plan document is any document intended to convey test plan
information. However, test plan documents are not the only source of information about the
test plan. Test plan information is also contained in the oral tradition of the project and the
culture of the company.

Test Strategy. The test strategy is the way tests will be designed and executed to support an
effective quality assessment. Test strategy is the plan for what parts of the product will be
covered by tests and what test techniques will be used. Test strategy is distinct from the
logistics of implementing the strategy. Test strategy is essentially the “brains” of the test
process.

Test Project. The test project is the means by which the test strategy is implemented and
results delivered. The test project is the “brawn” of the test process.

Test Plan Functions

Test plan functions are what a test plan is supposed to do. Below is a list of functions served by an ideal
test plan. However, a test plan document may only address a subset of these functions—the rest handled in
other documents or managed directly by the test manager or individual tester without the support of any
document. Thus, a test plan should be judged only with regard to those functions that it intends to serve, or
are insufficiently served by other means.

Support the development of a quality assessment that enables wise and timely decisions to
be made concerning the product.

Describe and justify the test strategy (including proposed test coverage) in relation to
technical requirements and technical risk. Promote awareness of the benefits and limitations
of the test strategy.

Page 1 of 5

Describe and justify any special requirements or entry criteria that must be met in order for
the test project to proceed, as well as any exit or process for determining when to stop
testing.

Support the initiation and organization of the test project, including preparations, staffing,
delegation of responsibilities, facility acquisition, task planning, and scheduling.

Support daily management and evaluation of the test project and test strategy.

Support effective coordination, collaboration, and other relations among members of the test
team, and between the test team and the rest of the project.

Identify and manage any risks or issues that may impact the project.

Specify the deliverables of the test project, and the delivery process.

Record historical information in support of process audits, process improvement and future
test projects.

Test Plan Quality Criteria

These criteria relate to how well a test plan performs its functions. A test plan is good to the extent that it
satisfies these criteria. Exactly how good is “good enough” depends on situational factors and judgments.

Usefulness. Will the test plan effectively serve its intended functions?

Accuracy. Is it accurate with respect to any statements of fact?

Efficiency. Does it make efficient use of available resources?

Adaptability. Will it tolerate reasonable change and unpredictability in the project?

Clarity. Is the test plan self-consistent and sufficiently unambiguous?

Usability. Is the test plan document concise, maintainable, and helpfully organized?

Compliance. Does it meet externally imposed requirements?

Foundation. Is it the product of an effective test planning process?

Feasibility. Is it within the capability of the organization that must perform it?

Test Plan Heuristics

In order to determine how well the test plan meets the criteria, heuristics are used. That is to say, a test plan
evaluation is based on generally accepted rules of thumb we have collected through experience and study.

Each of the heuristics in the table below relates to one or more of the criteria and functions identified
above. The only criterion in the list for which there is no corresponding heuristic is compliance. This is
because compliance to externally imposed requirements requires specific knowledge of those
requirements.

Each heuristic is described in terms of a general rule, and a brief basis for that rule. The basis is intended to
help determine when and where a heuristic applies.

Page 2 of 5
Heuristic Basis for heuristic
1. Testing should be optimized to find The later in the project that a problem is found, the
important problems fast, rather than greater the risk that it will not be safely fixed in
attempting to find all problems with equal time to ship. The sooner a problem is found after it
urgency. is created, the lesser the risk of a bad fix.

2. Test strategy should focus most effort on Complete testing is impossible, and we can never
areas of potential technical risk, while still know if our perception of technical risk is
putting some effort into low risk areas just in completely accurate.
case the risk analysis is wrong.

3. Test strategy should address test platform Sloppiness or neglect within any of these four
configuration, how the product will be basic testing activities will increase the likelihood
operated, how the product will be observed, that important problems will go undetected.
and how observations will be used to
evaluate the product.

4. Test strategy should be diversified in terms No single test technique can reveal all important
of test techniques and perspectives. Methods problems in a linear fashion. We can never know
of evaluating test coverage should take into for sure if we have found all the problems that
account multiple dimensions of coverage, matter. Diversification minimizes the risk that the
including structural, functional, data, test strategy will be blind to certain kinds of
platform, operations, and requirements. problems.

5. The test strategy should specify how test It is common for the test strategy to be organized
data will be designed and generated. around functionality or code, leaving it to the
testers to concoct test data on the fly. Often that
indicates that the strategy is too focused on
validating capability and not focused enough on
reliability.

6. Not all testing should be pre-specified in A rigid test strategy may make it more likely that a
detail. The test strategy should incorporate particular subset of problems will be uncovered,
reasonable variation and make use of the but in a complex system it reduces the likelihood
testers’ ability to use situational reasoning to that all important problems will be uncovered.
focuse on important, but unanticipated Reasonable variability in testing, such as that
problems. which results from interactive, exploratory testing,
increases incidental test coverage, without
substantially sacrificing essential coverage.

7. It is important to test against implied Testing only against explicit written requirements
requirements—the full extent of what the will not reveal all important problems, since
requirements mean, not just what they say. defined requirements are generally incomplete and
natural language is inherently ambiguous.

Page 3 of 5
Heuristic Basis for heuristic
8. The test project should promote Other teams and stakeholders often have
collaboration with all other functions of the information about product problems or potential
project, especially developers, technical problems that can be of use to the test team. Their
support, and technical writing. Whenever perspective may help the testers make a better
possible, testers should also collaborate with analysis of risk. Testers may also have information
actual customers and users, in order to better that is of use to them.
understand their requirements.

9. The test project should consult with The likelihood that a test strategy will serve its
development to help them build a more purpose is profoundly affected by the testability of
testable product. the product.

10. A test plan should highlight the non-routine, Virtually every software project worth doing
project-specific aspects of the test strategy involves special technical challenges that a good
and test project. test effort must take into account. A completely
generic test plan usually indicates a weak test
planning process. It could also indicate that the test
plan is nothing but unchanged boilerplate.

11. The test project should use humans for what Many test projects suffer under the false belief that
humans do well and use automation for human testers are effective when they use
what automation does well. Manual testing exactingly specified test scripts, or that test
should allow for improvisation and on the automation duplicates the value of human
spot critical thinking, while automated cognition in the test execution process. Manual
testing should be used for tests that require and automated testing are not two forms of the
high repeatability, high speed, and no same thing. They are two entirely different classes
judgment. of test technique.

12. The test schedule should be represented and A monolithic test schedule in a test plan often
justified in such a way as to highlight any indicates the false belief that testing is an
dependencies on the progress of independent activity. The test schedule can stand
development, the testability of the product, alone only to the extent that the product the highly
time required to report problems, and the testable, development is complete, and the test
project team’s assessment of risk. process is not interrupted by the frequent need to
report problems.

13. The test process should be kept off of the This is important in order to deflect pressure to
critical path to the extent possible. This can truncate the testing process.
be done by testing in parallel with
development work, and finding problems
worth fixing faster than the developers fix
them.

Page 4 of 5
Heuristic Basis for heuristic
14. The feedback loop between testers and This is important in order to maximize the
developers should be as tight as possible. efficiency and speed of quality improvement. It
Test cycles should be designed to provide also helps keep testing off of the critical path.
rapid feedback to developers about recent
additions and changes they have made
before a full regression test is commenced.
Whenever possible testers and developers
should work physically near each other.

15. The test project should employ channels of By examining product quality information
information about quality other than formal gathered through various means beyond the test
testing in order to help evaluate and adjust team, blind spots in the formal test strategy can be
the test project. Examples of these channels uncovered.
are inspections, field testing, or informal
testing by people outside of the test team.

16. All documentation related to the test Tunnel-vision is the great occupational hazard of
strategy, including test cases and procedures, testing. Review not only helps to reveal blind spots
should be undergo review by someone other in test design, but it can also help promote dialog
than the person who wrote them. The review and peer education about test practices.
process used should be commensurate with
the criticality of the document.

Page 5 of 5