Quick Notes

CH5 – Test Management

Number of Questions – 8 (K1-3, K2-3, K3-2) Duration – 170 minutes

1] Test organization

a) Test organization and independence

 The effectiveness of finding defects by testing and reviews can be improved

by using independent testers. Options for independence are:
 No independent testers. Developers test their own code.
 Independent testers within the development teams.
 Independent test team or group within the organization, reporting to
project management or executive management
 Independent testers from the business organization or user community.
 Independent test specialists for specific test targets such as usability
testers, security testers or certification testers (who certify a software
product against standards and regulations).
 Independent testers outsourced or external to the organization.

The benefits of independence include:

 Independent testers see other and different defects, and are unbiased.
 An independent tester can verify assumptions people made during
specification and implementation of the system.

Drawbacks include:

 Isolation from the development team (if treated as totally independent).

 Independent testers may be the bottleneck as the last checkpoint.
 Developers may lose a sense of responsibility for quality.

Quick Notes 5: Test Management

Created by: Sagar Joshi 1
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
b) Tasks of the test leader and tester

Test leader tasks may include:

 Coordinate the test strategy and plan with project manager

 Write or review a test strategy for the project, and test policy for the
organization
 Plan the tests
 Initiate the specification, preparation, implementation and execution of
tests, monitor the test results and check the exit criteria
 Setup adequate configuration management of test ware for traceability
 Introduce suitable metrics for measuring test progress and evaluating the
quality of the testing and the product
 Decide what should be automated, to what degree, and how
 Select tools to support testing and organize the training in tool use for
testers
 Decide about the implementation of the test environment
 Write test summary reports

Tester tasks may include:

 Review and contribute to test plans

 Analyze, review and assess user requirements, specifications and models for
traceability
 Create test specification
 Set up test environment
 Prepare and acquire test data
 Implement tests on all test levels, execute and log the tests, evaluate test
results and document the deviations from expected results
 Use test administration or management tools and test monitoring tools as
required
 Automate tests
 Measure performance of components and systems
 Review tests developed by others

Quick Notes 5: Test Management

Created by: Sagar Joshi 2
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
c) Defining skills test staff need

Now days a testing professional must have ‘application’ or ‘business domain’

knowledge and ‘Technology’ expertise apart from ‘Testing’ Skills

2] Test planning and estimation

a) Test planning activities

 Determining the scope and risks, and identifying the objectives of testing.
 Defining the overall approach of testing (the test strategy), including the
definition of the test levels and entry and exit criteria.
 Integrating and coordinating the testing activities into the software life
cycle activities: acquisition, supply, development, operation and
maintenance.
 Making decisions about what to test, what roles will perform the test
activities, how the test activities should be done, and how the test results
will be evaluated.
 Scheduling test analysis and design activities.
 Scheduling test implementation, execution and evaluation.
 Assigning resources for the different activities defined.
 Defining the amount, level of detail, structure and templates for the test
documentation.
 Selecting metrics for monitoring and controlling test preparation and
execution, defect resolution and risk issues.
 Setting the level of detail for test procedures in order to provide enough
information to support reproducible test preparation and execution.

b) Exit criteria

The purpose of exit criteria is to define when to stop testing, such as at the end of
a test level or when a set of tests has a specific goal.

Quick Notes 5: Test Management

Created by: Sagar Joshi 3
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
Typically exit criteria may consist of:

 Thoroughness measures, such as coverage of code, functionality or risk.

 Estimates of defect density or reliability measures.
 Cost.
 Residual risks, such as defects not fixed or lack of test coverage in certain
areas.
 Schedules such as those based on time to market.

c) Test estimation

Two approaches for the estimation of test effort are covered in this syllabus:

 The metrics-based approach: estimating the testing effort based on metrics

of former or similar projects or based on typical values.
 The expert-based approach: estimating the tasks by the owner of these
tasks or by experts.

Once the test effort is estimated, resources can be identified and a schedule can
be drawn up.

The testing effort may depend on a number of factors, including:

 Characteristics of the product: the quality of the specification and other

information used for test models (i.e. the test basis), the size of the
product, the complexity of the problem domain, the requirements for
reliability and security, and the requirements for documentation.
 Characteristics of the development process: the stability of the
organization, tools used, test process, skills of the people involved, and
time pressure.
 The outcome of testing: the number of defects and the amount of rework
required.

d) Test approaches (test strategies)

One way to classify test approaches or strategies is based on the point in time at
which the bulk of the test design work is begun:

Quick Notes 5: Test Management

Created by: Sagar Joshi 4
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
  Preventative approaches, where tests are designed as early as possible.
 Reactive approaches, where test design comes after the software or system
has been produced.

Typical approaches or strategies include:

 Analytical approaches, such as risk-based testing where testing is directed

to areas of greatest risk
 Model-based approaches, such as stochastic testing using statistical
information about failure rates (such as reliability growth models) or usage
(such as operational profiles).
 Methodical approaches, such as failure-based (including error guessing and
fault-attacks), experienced-based, check-list based, and quality
characteristic based.
 Process- or standard-compliant approaches, such as those specified by
industry-specific standards or the various agile methodologies.
 Dynamic and heuristic approaches, such as exploratory testing where testing
is more reactive to events than pre-planned, and where execution and
evaluation are concurrent tasks.
 Consultative approaches, such as those where test coverage is driven
primarily by the advice and guidance of technology and/or business domain
experts outside the test team.
 Regression-averse approaches, such as those that include reuse of existing
test material, extensive automation of functional regression tests, and
standard test suites.

Different approaches may be combined, for example, a risk-based dynamic

approach.

The selection of a test approach should consider the context, including:

 Risk of failure of the project, hazards to the product and risks of product
failure to humans, the environment and the company.
 Skills and experience of the people in the proposed techniques, tools and
methods.
 The objective of the testing endeavour and the mission of the testing team.
Quick Notes 5: Test Management
Created by: Sagar Joshi 5
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
  Regulatory aspects, such as external and internal regulations for the
development process.
 The nature of the product and the business.

3] Test progress monitoring and control

a) Test progress monitoring

 Percentage of work done in test case preparation (or percentage of planned

test cases prepared).
 Percentage of work done in test environment preparation.
 Test case execution (e.g. number of test cases run/not run, and test cases
passed/failed).
 Defect information (e.g. defect density, defects found and fixed, failure
rate, and retest results).
 Test coverage of requirements, risks or code.
 Subjective confidence of testers in the product.
 Dates of test milestones.
 Testing costs, including the cost compared to the benefit of finding the next
defect or to run the next test.

b) Test Reporting

 What happened during a period of testing, such as dates when exit criteria
were met.
 Analyzed information and metrics to support recommendations and
decisions about future actions, such as an assessment of defects remaining,
the economic benefit of continued testing, outstanding risks, and the level
of confidence in tested software.

Metrics should be collected during and at the end of a test level in order to assess:

 The adequacy of the test objectives for that test level.

 The adequacy of the test approaches taken.
 The effectiveness of the testing with respect to its objectives.
Quick Notes 5: Test Management
Created by: Sagar Joshi 6
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
c) Test control

Test control describes any guiding or corrective actions taken as a result of

information and metrics gathered and reported. Actions may cover any test
activity and may affect any other software life cycle activity or task.

Examples of test control actions are:

 Making decisions based on information from test monitoring.

 Re-prioritize tests when an identified risk occurs (e.g. software delivered
late).
 Change the test schedule due to availability of a test environment.
 Set an entry criterion requiring fixes to have been retested (confirmation
tested) by a developer before accepting them into a build.

4] Configuration management

The purpose of configuration management is to establish and maintain the

integrity of the products (components, data and documentation) of the software or
system through the project and product life cycle.

For testing, configuration management may involve ensuring that:

 All items of testware are identified, version controlled, tracked for changes,
related to each other and related to development items (test objects) so
that traceability can be maintained throughout the test process.
 All identified documents and software items are referenced unambiguously
in test documentation

For the tester, configuration management helps to uniquely identify (and to

reproduce) the tested item, test documents, the tests and the test harness.

During test planning, the configuration management procedures and infrastructure

(tools) should be chosen, documented and implemented.

Quick Notes 5: Test Management

Created by: Sagar Joshi 7
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
5] Risk and testing

a) Project risks

Project risks are the risks that surround the project’s capability to deliver its
objectives, such as:

Organizational factors:

 skill and staff shortages;

 personal and training issues;
 political issues, such as
o problems with testers communicating their needs and test results;
o failure to follow up on information found in testing and reviews (e.g.
not
 improving development and testing practices).
 improper attitude toward or expectations of testing (e.g. not appreciating
the value of finding defects during testing).

Technical issues:

 problems in defining the right requirements;

 the extent that requirements can be met given existing constraints;
 the quality of the design, code and tests.

Supplier issues:

 failure of a third party;

 contractual issues.

b) Product risks

Potential failure areas (adverse future events or hazards) in the software or system
are known as product risks, as they are a risk to the quality of the product, such
as:

 Failure-prone software delivered.

Quick Notes 5: Test Management

Created by: Sagar Joshi 8
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
  The potential that the software/hardware could cause harm to an individual
or company.
 Poor software characteristics (e.g. functionality, reliability, usability and
performance).
 Software that does not perform its intended functions.

Risks are used to decide where to start testing and where to test more; testing is
used to reduce the risk of an adverse effect occurring, or to reduce the impact of
an adverse effect.

Product risks are a special type of risk to the success of a project. Testing as a
risk-control activity provides feedback about the residual risk by measuring the
effectiveness of critical defect removal and of contingency plans.

A risk-based approach to testing provides proactive opportunities to reduce the

levels of product risk, starting in the initial stages of a project. It involves the
identification of product risks and their use in guiding test planning and control,
specification, preparation and execution of tests. In a risk-based approach the risks
identified may be used to:

 Determine the test techniques to be employed.

 Determine the extent of testing to be carried out.
 Prioritize testing in an attempt to find the critical defects as early as
possible.
 Determine whether any non-testing activities could be employed to reduce
risk (e.g. providing training to inexperienced designers).

Risk-based testing draws on the collective knowledge and insight of the project
stakeholders to determine the risks and the levels of testing required to address
those risks.

To ensure that the chance of a product failure is minimized, risk management

activities provide a disciplined approach to:

 Assess (and reassess on a regular basis) what can go wrong (risks).

 Determine what risks are important to deal with.

Quick Notes 5: Test Management

Created by: Sagar Joshi 9
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
  Implement actions to deal with those risks.

In addition, testing may support the identification of new risks, may help to
determine what risks should be reduced, and may lower uncertainty about risks.

6] Incident management

Since one of the objectives of testing is to find defects, the discrepancies between
actual and expected outcomes need to be logged as incidents. Incidents should be
tracked from discovery and classification to correction and confirmation of the
solution. In order to manage all incidents to completion, an organization should
establish a process and rules for classification.

Incidents may be raised during development, review, testing or use of a software

product. They may be raised for issues in code or the working system, or in any
type of documentation including requirements, development documents, test
documents, and user information such as “Help” or installation guides.

Incident reports have the following objectives:

 Provide developers and other parties with feedback about the problem to
enable identification, isolation and correction as necessary.
 Provide test leaders a means of tracking the quality of the system under test
and the progress of the testing.
 Provide ideas for test process improvement.

Details of the incident report may include:

 Date of issue, issuing organization, and author.

 Expected and actual results.
 Identification of the test item (configuration item) and environment.
 Software or system life cycle process in which the incident was observed.
 Description of the incident to enable reproduction and resolution, including
logs, database dumps or screenshots.
 Scope or degree of impact on stakeholder(s) interests.

Quick Notes 5: Test Management

Created by: Sagar Joshi 10
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/
  Severity of the impact on the system.
 Urgency/priority to fix.
 Status of the incident (e.g. open, deferred, duplicate, waiting to be fixed,
fixed awaiting retest, closed).
 Conclusions, recommendations and approvals.
 Global issues, such as other areas that may be affected by a change
resulting from the incident.
 Change history, such as the sequence of actions taken by project team
members with respect to the incident to isolate, repair, and confirm it as
fixed.
 References, including the identity of the test case specification that
revealed the problem.

Quick Notes 5: Test Management

Created by: Sagar Joshi 11
https://www.udemy.com/istqb-foundation-level-certification-ctfl-training/