ISEB Foundation Certificate in Software Testing

Testing Terminology

© SIM Group Ltd., SQS Group AG, 2002

Testing Terminology

‰ IT is an industry riddled with mysterious words, terms and

TLA’s

‰ Software testing is no exception

‰ Historically there has not been a generally accepted set of

testing definitions

© SIM Group Ltd., SQS Group AG, 2002

Testing Terminology

‰ To resolve this issue, the BCS SIGIST created a Standard

Glossary of Testing Terms
z British Computer Society
z Special Interest Group In Software Testing

‰ This is now a British Standard

z BS7925-1

© SIM Group Ltd., SQS Group AG, 2002

Testing Terminology

‰ You will need to read the document and start to learn the
terminology used in software testing

‰ You will be exposed to the terminology and customers will

assume you know what they mean

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Why Testing is Necessary

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ In this session we will

‰ Understand what testing is and why it is necessary

‰ Define some of the words used in software testing

‰ Look at the cause and cost of errors

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Why Test?
To find bugs

What is a bug?

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

What is Testing?

“…the process of executing a program with the intent to certify its

Quality”
Mills

“…the process of executing a program with the intent of finding

failures / faults”
Myers

“…the process of exercising software to detect bugs & to verify that

it satisfies specified functional & non-functional requirements”

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Why is Testing becoming more important?

‰ The Y2K problem

‰ EMU

‰ e Commerce

‰ Increased user base

‰ Increased complexity

‰ Speed to Market

© SIM Group Ltd., SQS Group AG, 2002

Definitions

© SIM Group Ltd., SQS Group AG, 2002

Definitions

ERROR
“A human action that produces an incorrect result”

FAULT
“A manifestation of an ERROR in software”

© SIM Group Ltd., SQS Group AG, 2002

Definitions

FAULT
A Fault, if encountered, may cause a failure

FAILURE
“A deviation of the software from its expected delivery or
service”

© SIM Group Ltd., SQS Group AG, 2002

Definitions

DEFECT
“The departure of a quality characteristic from its specified value
that results in a product or service not satisfying its normal
usage requirements”

© SIM Group Ltd., SQS Group AG, 2002

Definitions

RELIABILITY
“The probability that software will not cause the failure of a
system for a specified period of time under specified
conditions”

© SIM Group Ltd., SQS Group AG, 2002

Definitions

QUALITY
“The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs”

© SIM Group Ltd., SQS Group AG, 2002

Testing & Quality

„ Does testing improve quality?

‰ Testing does not build Quality into the software

‰ Testing is a means of determining the Quality of the Software

under Test

© SIM Group Ltd., SQS Group AG, 2002

Definitions

QUALITY ASSURANCE

“All those planned actions used to fulfil the requirements for

quality”

© SIM Group Ltd., SQS Group AG, 2002

Why do errors occur?

© SIM Group Ltd., SQS Group AG, 2002

Why do errors occur?

„ How Errors Occur

‰ No one is perfect!
We all make mistakes or omissions

‰ The more pressure we are under the more likely we are to

make mistakes

‰ In IT development we have time and budgetary deadlines to

meet

‰ Poor Training

© SIM Group Ltd., SQS Group AG, 2002

Why do errors occur?

„ How Errors Occur

‰ Poor Communication

‰ Requirements not clearly defined

‰ Requirements change & are not properly documented

‰ Data specifications not complete

‰ ASSUMPTIONS!

© SIM Group Ltd., SQS Group AG, 2002

Why do errors occur?

„ The Cost Of Errors

‰ A single failure may incur little cost - or millions

‰ Report layouts may be wrong - little true cost

‰ Or a significant error may cost millions…

Ariane V, Venus Probe, Mars Explorer and Polar Lander

© SIM Group Ltd., SQS Group AG, 2002

Why do errors occur?

„ The Cost Of Errors

‰ In extreme cases a software or systems error may cost

LIVES

‰ Usually safety critical systems are tested exhaustively

Aeroplane, railway, nuclear power systems etc

‰ Unfortunately there are exceptions to this

London Ambulance Service

© SIM Group Ltd., SQS Group AG, 2002

Why do errors occur?

„ The Cost Of Errors

‰ The cost of defects increases proportionately (tenfold?) with

the passing of each successive stage in the system
development process before they are detected

‰ To correct a problem at requirements stage may cost £1

‰ To correct the problem post-implementation may cost £000’s

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Why Test?

‰ Identifies faults
Reduces live defects
Improves the quality of the users application
Increases reliability
To help ensure live failures don’t impact costs & profitability
To help ensure requirements are satisfied
To ensure legal requirements are met
To help maintain the organisation’s reputation

‰ Provides a measure of quality

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Quality Measurements

‰ Quality can be measured by testing for

correctness
reliability
usability
maintainability
testability
reusability

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ How much is enough?

‰ How do we know when to stop?

There are many factors to consider

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Exhaustive Testing

‰ Find all faults by testing everything?

‰ To test everything is rarely possible

‰ The time required makes it impractical

‰ The resource commitment required makes it impractical

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ If we can’t test everything, what can we do?

‰ Managing and reducing Risk

‰ Carry out a Risk Analysis of the application

‰ Prioritise tests to focus on the main areas of risk

‰ Apportion time relative to the degree of risk involved

‰ Understand the risk to the business of the software not

functioning correctly

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Should you stop testing when the product goes live?

‰ Continuing testing beyond the implementation date should be

considered

‰ Better that you find the errors than the users

© SIM Group Ltd., SQS Group AG, 2002

Why Testing is Necessary

„ Summary

‰ The purpose of testing is to find faults

Faults can be fixed, making better software
Better software is more reliable, less prone to failures
Establish the relationship between the software and its’
specification

‰ Testing enables us to measure the quality of the software

‰ This enables us to understand & manage the risk to the

business

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

The Test Process

© SIM Group Ltd., SQS Group AG, 2002

The Test Process

„ In this session we will

‰ Look at the steps involved in the test process

‰ Look in detail at each step

© SIM Group Ltd., SQS Group AG, 2002

The Test Process

„ What is the objective of a test?

‰ A “successful” test is one that does detect a fault

© SIM Group Ltd., SQS Group AG, 2002

The Test Process

TEST MANAGEMENT

SPECIFICATION

COMPLETION
RECORDING
EXECUTION
PLANNING

TEST ASSET MANAGEMENT

TEST ENVIRONMENT MANAGEMENT

© SIM Group Ltd., SQS Group AG, 2002

The Test Process

„ Five steps in the Test Process are

‰ Test Planning
‰ Test Analysis and Specification
‰ Test Execution
‰ Test Recording (Verification)
‰ Checking for Completion

© SIM Group Ltd., SQS Group AG, 2002

Test Planning

„ The Test Plan describes how

the Test Strategy is
implemented

© SIM Group Ltd., SQS Group AG, 2002

Test Planning

„ Contents of a Test Plan include

‰ Background ‰ Resources
‰ Reference documents ‰ Dependencies
‰ Approach ‰ Reporting
‰ Method ‰ Test Asset Identification
‰ Timetable ‰ Exit Criteria

© SIM Group Ltd., SQS Group AG, 2002

Test Planning

„ The most critical stage of the

process

„ Effort spent now will be

rewarded later

„ The foundation on which

testing is built

© SIM Group Ltd., SQS Group AG, 2002

Test Specification

„ Three step process

‰ Preparation & analysis

‰ Building test cases

‰ Define expected results

© SIM Group Ltd., SQS Group AG, 2002

Test Specification

„ Test preparation

‰ Analyse the Application

‰ Identify good test conditions
‰ Identify test cases
‰ Document thoroughly
‰ Cross-referencing

© SIM Group Ltd., SQS Group AG, 2002

Test Specification

„ Build Test Cases

‰ Test cases comprise:

z Standing Data
z Transaction Data
z Actions
z Expected Results

© SIM Group Ltd., SQS Group AG, 2002

Test Specification

„ Expected Results

‰ The outcome of each action

‰ The state of the application

during & after

‰ The state of the data during

& after

© SIM Group Ltd., SQS Group AG, 2002

Test Specification

„ Cross-Referencing &
Classification

‰ Enables maintainability of
Test Assets

‰ Allows testing to be
performed in a focussed
manner directed at specific
areas

© SIM Group Ltd., SQS Group AG, 2002

Test Execution

„ Test Execution checklist

‰ Test execution schedule / log

‰ Identify which tests are to be run
‰ Test environment primed & ready
‰ Resources ready, willing & able
‰ Back-up & recovery procedures in place
‰ Batch runs planned & scheduled

„ Then we are ready to run the tests

© SIM Group Ltd., SQS Group AG, 2002

Test Recording

„ Test verification

‰ If planning and preparation is sufficiently detailed this is the

easy part of testing

‰ The test is run to verify the application under test

‰ The test itself either passes or fails!

© SIM Group Ltd., SQS Group AG, 2002

Test Recording

„ The test log should record

‰ Software and test versions

‰ Specifications used as test base
‰ Test timings
‰ Test results
z Actual results
z Expected results

‰ Defect details for erroneous tests

© SIM Group Ltd., SQS Group AG, 2002

Test Completion

„ Test Exit Criteria

„ Used to determine when to

implement the software

‰ Key Functionality tested

‰ Test coverage
‰ Budget used?
‰ Defect detection rate
‰ Performance satisfactory

© SIM Group Ltd., SQS Group AG, 2002

The Test Process

„ Summary

‰ There are five steps in the Test Process

z Test Planning
z Test Analysis and Specification
z Test Execution
z Test Recording (Verification)
z Checking for Completion

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

The Psychology Of Testing

© SIM Group Ltd., SQS Group AG, 2002

The Psychology Of Testing

„ In this session we will

‰ Understand what qualities make good testers

‰ Look at a testers’ relationship with developers

‰ Look at a testers’ relationship with management

‰ Understand the issues with testing independence

© SIM Group Ltd., SQS Group AG, 2002

What makes a Tester?

„ Testing is primarily to find faults

‰ Can be regarded as ‘destructive’

‰ Development is ‘constructive’
‰ Testing asks questions
‰ Testers need to ask questions

„ A tester needs many qualities...

© SIM Group Ltd., SQS Group AG, 2002

What makes a Tester?

„ Intellectual qualities

‰ Can absorb incomplete facts

‰ Can work with incomplete facts
‰ Can learn quickly on many levels
‰ Good verbal communication
‰ Good written communication
‰ Ability to prioritise
‰ Self-organisation

© SIM Group Ltd., SQS Group AG, 2002

What makes a Tester?

„ Knowledge

‰ How projects work

‰ How computer systems and business needs interact
‰ What makes IT tick - technology
‰ What makes IT tick - commercial aspects
‰ Testing techniques
‰ Testing best practice
‰ To be able to think inside and outside of a system
specification

© SIM Group Ltd., SQS Group AG, 2002

What makes a Tester?

„ More skills to acquire

‰ How to find bugs - planning, preparation & execution

‰ How to understand systems
‰ How to read specifications
‰ How to extract testable functionality
‰ How to work efficiently
‰ How to focus on essentials

© SIM Group Ltd., SQS Group AG, 2002

Reporting Defects

„ Defects need to be reported to

‰ Developers to enable them to fix them

‰ Management so they can track progress

„ Communication to both groups is vital

© SIM Group Ltd., SQS Group AG, 2002

Communication with Developers

„ A good relationship is vital

‰ Developers need to keep testers up to date with changes to

the application

‰ Testers need to inform developers of defects to allow fixes to

be applied

© SIM Group Ltd., SQS Group AG, 2002

Communication with Management

„ Managers need progress reports

„ The best way is through metrics

‰ Number of tests planned & prepared
‰ Number of tests executed to date
‰ Number of defects raised & fixed
‰ How long planning, preparation and execution stages take

© SIM Group Ltd., SQS Group AG, 2002

 The Psychology of Testing

Testing Independence

© SIM Group Ltd., SQS Group AG, 2002

Testing Independence

„ It is important that testing is separate from

development

‰ The developer is likely to confirm adherence not deviation

‰ The developer will make assumptions - the same when

testing as developing

© SIM Group Ltd., SQS Group AG, 2002

Testing Independence

„ Levels of Independence

‰ Low - Developers write their own tests

‰ Medium - Tests are written by another developer

‰ High - Tests written by an independent body

z Tests written by another section
z Tests written by another organisation

‰ Utopia - Tests generated automatically

© SIM Group Ltd., SQS Group AG, 2002

The Psychology Of Testing

„ Summary

‰ Testers require a particular set of skills

z The desire to break things
z The desire to explore and experiment
z Communication
z Questioning

‰ Testing requires a different mentality to development

z “Destroying” things rather than creating them
z Testing should be separate from development

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Re-Testing & Regression Testing

© SIM Group Ltd., SQS Group AG, 2002

Re-Testing & Regression Testing

„ In this session we will

‰ Understand how fault fixing and re-testing are key to the

overall testing process

‰ Look at how test repeatability helps

‰ Understand what regression testing is and where it fits in

‰ Understand how to select test cases for regression testing

© SIM Group Ltd., SQS Group AG, 2002

Re-Testing

„ Testing is directed at finding faults

‰ These faults will need to be corrected and a new version of

the software released

‰ The tests will need to be run again to prove that the fault is
fixed

‰ This is known as Re-Testing

© SIM Group Ltd., SQS Group AG, 2002

Re-Testing

„ The need for re-testing needs to be planned & designed

for

‰ Schedules need to allow for re-testing

‰ Tests need to be easily re-run

‰ The environment needs to be easily restored

© SIM Group Ltd., SQS Group AG, 2002

Regression Testing

“Re-testing of a previously tested program following

modification to ensure that FAULTS have not been
introduced or uncovered as a result of the changes
made”

© SIM Group Ltd., SQS Group AG, 2002

Regression Testing

8
Module 93 v1.3.5
How can you fix this
but break that? ~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
9
9
~~~~~~~~~~~~~~~~
Module 14 v1.4.2 ~~~~~~~~~~~~~~~~
Print (x);
Module 14 v1.4.3
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~ 9 ~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Declare String x =Date();
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Global String x = Date();
~~~~~~~~~~~~~~~~

9
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~ Module 41 v1.8.9
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
Declare Integer x = 0;
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
8
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
For x = 1 to 5
~~~~~~~~~~~~~~~~

© SIM Group Ltd., SQS Group AG, 2002

Regression Testing

„ Tests will also need to be re-run when checking

software upgrades

‰ Regression tests should be run whenever there is a change

to the software or the environment

‰ Regression tests are executed to prove aspects of a system

have not changed

‰ Regression Testing is a vital testing technique

© SIM Group Ltd., SQS Group AG, 2002

Regression Testing

„ Selecting cases for regression:

‰ Tests that cover safety or business critical functions

‰ Tests that are repetitive

‰ Tests that need accurate data

‰ Tests for areas that change regularly

‰ Tests of functions that have a high level of defects

© SIM Group Ltd., SQS Group AG, 2002

Regression Testing

„ Regression Testing is the ideal foundation for

Automation

‰ Selecting test cases is vital, and requires a degree of

knowledge of the applications and their expected evolution

© SIM Group Ltd., SQS Group AG, 2002

Re-Testing & Regression Testing

„ Summary

‰ Once a fault has been fixed the software MUST be re-tested

z New faults may have been introduced by the fix
z Existing faults may have been uncovered by the fix
z Tests need to be written to enable their re-use

‰ Re-testing is the rerunning of failed tests once a fix has been

implemented to check that the fix has worked

‰ Regression testing is running of a wider test suite to check for

unexpected errors in unchanged code

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Expected Results

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ In this session we will

‰ Understand the need to define expected results

‰ Understand where expected results can be found

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ Expected results = expected

outcomes

„ Identifying required behaviour

„ If the expected outcome of a

test is not defined the actual
output may be misinterpreted

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ Running a test with only a

general concept of the
outcome is fatal

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ It is vital the Expected Results

are defined with the tests,
before they are used

„ You cannot decide whether a

test has passed just because
it looks right

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ It may be possible to identify

the expected result of a test
based on knowledge or
experience of the
application… but not the code

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ Consult the Oracle!

© SIM Group Ltd., SQS Group AG, 2002

Expected Results

„ Summary

‰ Without defining expected results how do you know if a test

has passed or failed?

‰ Expected results can be found in the system specification and

by asking experienced business users

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Prioritisation of Tests

© SIM Group Ltd., SQS Group AG, 2002

Prioritisation of Tests

„ In this session we will

‰ Understand why we need to prioritise tests

‰ Understand how we decide the priority of individual tests

© SIM Group Ltd., SQS Group AG, 2002

Prioritisation of tests

„ It is not possible to test

everything

‰ Therefore faults will get

through to the live system

‰ We must do the best testing

possible in the time available

‰ This means we must

prioritise and focus testing
on the priorities

© SIM Group Ltd., SQS Group AG, 2002

Aspects to Consider

„ Severity „ Frequency of Change

„ Probability „ Vulnerability to error

„ Visibility „ Technical criticality

„ Priority of Requirements „ Complexity

„ Customer Requirements „ Resource availability

© SIM Group Ltd., SQS Group AG, 2002

Business Criticality

„ What elements of the

application are essential to
the success of the
organisation?

© SIM Group Ltd., SQS Group AG, 2002

Customer factors

„ How visible would a failure

be?

„ What does the customer

want?

© SIM Group Ltd., SQS Group AG, 2002

Technical Factors

„ How complex is it?

„ How likely is an error?

„ How often does this change?

© SIM Group Ltd., SQS Group AG, 2002

Prioritisation of Tests

„ Summary

‰ Due to the last minute nature of testing, it is necessary to

prioritise tests

‰ There will never be enough time to complete all tests

‰ Therefore those tests that cover those areas deemed most

important (to the business, highest risk etc) must be tested
first

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Models for Testing

© SIM Group Ltd., SQS Group AG, 2002

Models for Testing

„ In this session we will

‰ Look at two of the most commonly accepted testing models

© SIM Group Ltd., SQS Group AG, 2002

Testing Models

„ There are many approaches to testing

‰ Of these 2 are widely used & referred to:

z V,V&T
z The V-Model

‰ There are various other models

z They are not part of this course

© SIM Group Ltd., SQS Group AG, 2002

V,V&T

‰ Verification
z “The process of evaluating a system or component to determine
whether the products of the given development phase satisfies
the conditions imposed at the start of that phase”

‰ Validation
z “Determination of the correctness of the products of software
development with respect to the user needs and requirements”

‰ Testing
z “process of exercising software to verify that it satisfies specified
requirements and to detect faults”

© SIM Group Ltd., SQS Group AG, 2002

V-Model

„ The V-Model is the most commonly used model in

testing

‰ It represents the Software Development Life Cycle

‰ Shows the various stages in development and testing

‰ Shows the relationships between the various stages

© SIM Group Ltd., SQS Group AG, 2002

A simple V-Model

high level

T
requirements UAT
T
design system
T test
code unit
test
low level time

© SIM Group Ltd., SQS Group AG, 2002

 A typical development life-cycle V-Model

Business Service Level

Requirements Testing

Operational User Acceptance

Requirements Testing

Design Integration
Specifications Testing (large)
Functional System
Specification Testing
level
Program Component
Specification Testing
time

© SIM Group Ltd., SQS Group AG, 2002

Models for Testing

V-Model - where do we start testing?

Requirements T UAT

Design T System Test

Code T Unit Test

level

time

© SIM Group Ltd., SQS Group AG, 2002

Models for Testing

V-Model - Start Testing Early

R PR R T T T
Reqs
UAT
RPR T T
Spec System Test
R P T
Code Unit Test
level

time

© SIM Group Ltd., SQS Group AG, 2002

Models for Testing

„ Summary

‰ The V-Model is the most common approaches model for

testing

‰ There are other models used for testing, but these are less
widely used
z Inc. V,V & T
z Other models have been created

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Economics of Testing

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ In this session we will

‰ Look at the cost of testing (or the cost of not testing)

‰ Appreciate the value of good, timely testing

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ The earlier a fault is found, the cheaper it is to remedy

‰ Faults in the requirements can lead to major re-engineering of

the entire system

‰ Many faults can be found using reviews

z Reviews of documentation and / or code

‰ Allows testing to find more ‘substantial’ faults

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ The average cost of fixing a defect increases tenfold

with every step of the development process

‰ In code reviews you will find and fix defects in an average of

1 - 2 minutes

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ In initial testing, defect fix times will average between

10 to 20 minutes

„ In integration testing each defect can cost an hour or

more

„ In system test each defect can cost 10 to 40 or more

engineer hours

Watts Humphrey

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Early test design can prevent fault multiplication

‰ Analysis of specification during test preparation often brings

faults in the specification to light

‰ If faults in documentation are not found then the system may

be developed incorrectly

© SIM Group Ltd., SQS Group AG, 2002

 Economics of testing

An example

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in the requirements

‰ Change requirements document

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in the specifications

‰ Change requirements document

‰ Change specifications

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in the code

‰ Change requirements document

‰ Change specifications

‰ Change the code

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in system testing

‰ Change requirements document

‰ Change specifications

‰ Change the code

‰ Change the tests, re-test & regression test

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in UAT

‰ Change requirements document

‰ Change specifications

‰ Change the code

‰ Change the tests, re-test & regression test

‰ Re-do UAT

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in Live

‰ Change requirements document

‰ Change specifications

‰ Change the code

‰ Change the tests, re-test & regression test

‰ Re-do UAT
Plus...

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Finding the error in Live

‰ User to report and log fault

‰ Users must be advised of fix, work-around etc

‰ The live database may need to be fixed

‰ Deploying the new build may take hours

‰ Loss of business

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Additionally

‰ Faults found in coding or later require changes to the code

library

‰ Documentation and help files will have to be changed

‰ Every fault requires investigating and logging

‰ Administration costs

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ The cost of not Testing

‰ What is the cost of testing?

‰ Which is cheaper?
z Testing and finding faults before release
z Not testing and finding faults once the system is live

‰ Companies do not typically have figures for either

© SIM Group Ltd., SQS Group AG, 2002

Economics of Testing

„ Summary

‰ The purpose of testing is to find faults

‰ The earlier a fault is found the cheaper (and easier) it is to fix

‰ Starting testing early will help find faults quicker

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

High Level Test Planning

© SIM Group Ltd., SQS Group AG, 2002

High Level Test Planning

„ In this session we will

‰ Look at how a test plan is put together

‰ Understand how it should be used and maintained

‰ Understand why they are so important to a testing project

© SIM Group Ltd., SQS Group AG, 2002

High Level Test Planning

„ What is a test plan?

‰ A project plan for testing

‰ Covering all aspects of testing

© SIM Group Ltd., SQS Group AG, 2002

High Level Test Planning

„ Before you plan

‰ A test strategy must be in place

© SIM Group Ltd., SQS Group AG, 2002

The Different Sections of a Test Plan

‰ Test plan identifier ‰ Test deliverables

‰ Introduction ‰ Testing tasks

‰ Test items ‰ Environment

‰ Features to be tested ‰ Responsibilities

‰ Staffing and training needs
‰ Features not to be tested
‰ Schedules
‰ Approach
‰ Risks and contingencies
‰ Item pass / fail criteria
‰ Approvals
‰ Suspension criteria &
resumption criteria

Based upon IEEE 829-1998 standard for

software test documentation

© SIM Group Ltd., SQS Group AG, 2002

Test Plan Identifier

„ A unique identifier for the test plan

© SIM Group Ltd., SQS Group AG, 2002

Introduction

„ The introduction should

‰ Give an overview of the plan

z A summary of the requirements
z Discuss what needs to be achieved
z Detail why testing is needed

‰ Reference to other documents

z Quality assurance and configuration management

© SIM Group Ltd., SQS Group AG, 2002

Test Items

„ The various items to be used in the tests

‰ The software items

‰ Their versions numbers / identifiers

‰ How they will be handed over to testing

‰ References to relevant documents

© SIM Group Ltd., SQS Group AG, 2002

Features to Be Tested

„ List all features of the SUT that will be tested under this
plan

© SIM Group Ltd., SQS Group AG, 2002

Features Not to Be Tested

„ List all features of the SUT that will not be tested under
this plan

Between the test items, features to be tested and features not to be

tested we have scope of the project

© SIM Group Ltd., SQS Group AG, 2002

Approach

„ Describes the approach to testing the SUT

‰ This should be high level, but sufficient to estimate the time

and resources required
z What this approach will achieve
z Specify major activities
z Testing techniques
z Testing tools / aids
z Constraints to testing
z Support required - environment & staffing

© SIM Group Ltd., SQS Group AG, 2002

Item Pass / Fail Criteria

„ How to judge whether a test item has passed

‰ Expected vs. Actual results

‰ Certain % of tests pass

‰ Number of faults remaining (known and estimated)

‰ Should be defined for each test item

© SIM Group Ltd., SQS Group AG, 2002

Suspension criteria & resumption requirements

„ Reasons that would cause testing to be suspended

‰ Steps necessary to resume testing

© SIM Group Ltd., SQS Group AG, 2002

Test Deliverables

„ Everything that goes to make up the tests

‰ All documentation
z e.g. Specification, test plans, procedures, reports

‰ Code releases

‰ Testing tools
z Test management tools, automation tools, excel, word etc

‰ Test systems
z Manual and automated test cases

© SIM Group Ltd., SQS Group AG, 2002

Testing Tasks

‰ Preparation to perform testing

z Test case identification
z Test case design
z Test data storage
z Baseline application

‰ Special skills needed

z Spreadsheet skills, test analysis, automation etc

‰ Inter-dependencies

© SIM Group Ltd., SQS Group AG, 2002

Environment

„ Requirements for test environment

‰ Hardware & software

z PCs, servers, routers etc
z SUT, interfaced applications, databases

‰ Configuration
z Maybe operating systems or middleware to test against

‰ Facilities
z Office space, desks, internet access

© SIM Group Ltd., SQS Group AG, 2002

Responsibilities

„ Who is responsible?

‰ For which activities

‰ For which deliverables

‰ For the environment

© SIM Group Ltd., SQS Group AG, 2002

Staffing and Training Needs

‰ Staff required
z Test managers, team leaders, testers, test analysts

‰ Skill levels required

z Automation experience
z Spreadsheet skills, etc

‰ Training requirements
z Tools specific training
z Refresher courses, etc.
z Overall project resource requirements

© SIM Group Ltd., SQS Group AG, 2002

Schedule

„ Timescales, dates and milestones

‰ Resources required to meet milestones

‰ Availability of software and environment

‰ Deliverables

© SIM Group Ltd., SQS Group AG, 2002

Risks and Contingencies

„ What might go wrong?

‰ Actions for minimising impact on testing should things go

wrong

© SIM Group Ltd., SQS Group AG, 2002

Approvals

„ Who has approved the test plan

‰ Names and dates of approval

‰ Why is it so important
z Evidence that the document has been viewed
z Shows that the approach has been agreed and has the backing
of those who matter
z You have commitment, now make them stick to it!

© SIM Group Ltd., SQS Group AG, 2002

High Level Test Planning

„ Summary

‰ Test plans are created to ensure that the requirements are

understood

‰ To ensure that maximum test coverage is achieved

‰ To identify all items that will form the test

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Acceptance Testing

© SIM Group Ltd., SQS Group AG, 2002

Acceptance Testing

„ In this session we will

‰ Understand what acceptance testing is

z Why you would want to do it
z How you would plan it and prepare for it
z What you need to actually do it

‰ Understand the different types of acceptance testing

© SIM Group Ltd., SQS Group AG, 2002

Introduction

„ What is Acceptance Testing ?

"Formal testing conducted to enable a user, customer or other

authorised entity to determine whether to accept a system or
component."

© SIM Group Ltd., SQS Group AG, 2002

User Acceptance Testing

„ What is User Acceptance Testing (UAT)?

‰ Exactly what it says it is!!

‰ Users of end product conducting the tests

‰ The set-up will represent a working Environment

‰ Covers all areas of a project, not just the system

‰ A.K.A Business Acceptance Testing or Business Process

Testing

© SIM Group Ltd., SQS Group AG, 2002

Where does it fit in?

Unit
Test

Systems
Test

UAT

Alpha /
Beta

© SIM Group Ltd., SQS Group AG, 2002

A typical development life-cycle V-Model

Business Service Level

Requirements Testing

Operational User Acceptance

Requirements Testing

Design Integration
Specifications Testing (large)
Functional System
Specification Testing
level
Program Component
Specification Testing
time

© SIM Group Ltd., SQS Group AG, 2002

Planning UAT

„ Why plan?

„ Things to consider

„ Preparing Tests
‰ Manual
‰ Automated

© SIM Group Ltd., SQS Group AG, 2002

Why Plan?

„ If you don’t then how do you know you have achieved

what you set out to do!

‰ Avoids repetition

‰ Test according to code releases

‰ Makes efficient and effective use of time and resources

© SIM Group Ltd., SQS Group AG, 2002

Things to consider

„ Timescales

„ Resources

„ Availability

© SIM Group Ltd., SQS Group AG, 2002

Preparing your tests

„ Take a logical approach

‰ Identify the business processes

‰ Separate into everyday business scenarios

© SIM Group Ltd., SQS Group AG, 2002

How the business fits together...

The Business

Process Process Process

Task Task Task Task Task Task Task Task Task

© SIM Group Ltd., SQS Group AG, 2002

Data Requirements

„ Copied Environments

„ Created Environments

© SIM Group Ltd., SQS Group AG, 2002

Running the Tests

„ Order of Tests

„ Confidence Checks

„ Automated and Manual test runs

© SIM Group Ltd., SQS Group AG, 2002

Contract Acceptance

„ “A demonstration of the acceptance criteria”

‰ Acceptance criteria will have been defined in the contract

‰ Before the software is accepted it is necessary to show that

its matches its’ specification as defined in the criteria

© SIM Group Ltd., SQS Group AG, 2002

Alpha Testing

„ Letting your customers do your testing

‰ Requires a stable version of the software

‰ People who represent your target market use the product in

the same way(s) as if they had bought the finished version

‰ Alpha testing is conducted in-house (at the developers site)

© SIM Group Ltd., SQS Group AG, 2002

Beta Testing

„ As Alpha testing but users perform tests at their site

© SIM Group Ltd., SQS Group AG, 2002

Acceptance Testing

„ Summary

‰ You can UAT anything!

‰ Every Deliverable should pass through UAT

‰ User representation is VITAL

‰ If the product does not pass UAT then a decision about

implementation needs to be made

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Integration Testing in the Large

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ In this session we will

‰ Understand what Integration Testing is

‰ Understand what Integration Testing in the Large is

‰ Look at how & why we test system integration

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

‰ Integration testing
z “Testing performed to expose faults in the interfaces and in the
interaction between integrated components”

‰ Integration testing in the large

z Integration Testing in the Large is testing performed to expose
faults in the interfaces and in the interaction between systems

‰ Why do we need to do integration testing in the large

z Just because the SUT works in the test lab doesn’t mean it will
work in the real world

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ All components of a computer system work on only

one thing

DATA

‰ Data is manipulated within the SUT

‰ It may then passed to another system

‰ This is done to satisfy a business process requirement

© SIM Group Ltd., SQS Group AG, 2002

A typical system

Env Env Env

SUT SUT SUT

Data Model

Databases

Data Ext apps.

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ Integration Testing on a Large Scale

‰ Test the way the SUT interfaces with external systems

‰ The elements may run on the same and / or different

platforms

‰ They may be located in separate locations and use

communication protocols to pass data back & forth

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ Typical approach

‰ Test the system

z In a test environment
z Use stubs & drivers where external systems aren’t available

‰ Test the system “in situ”

z In a replica of the production / live environment
z Use stubs & drivers where external systems aren’t available

‰ Test the system and its’ interfaces with other systems

z In a replica of the production / live environment
z Access test versions of the external systems

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ Fundamentally we are still looking at

DATA

‰ & how that data is obtained, processed, manipulated, stored

and made available or transported for use elsewhere within or
outside the organisation

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ Additional challenges posed by large scale:

‰ Multiple Platforms
‰ Communications between platforms
‰ Management of the environments
‰ Coordination of changes
‰ Different technical skills required

© SIM Group Ltd., SQS Group AG, 2002

The approach used

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing

„ Planning testing

‰ We need to determine when and at what levels Integration

Testing will be performed

‰ Identify the boundaries

‰ A similar process will need to be followed for all elements

z Once these elements have been tested individually they are
further integrated to support the business process

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing

„ Approaches to integration testing

‰ Incremental
z Top down
z Bottom-up
z Functional

‰ Non-incremental
z “Big bang”

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Large

„ Summary

‰ Integration testing is needed to test how components interact

with each other and how the system under test works with
other systems

‰ Large scale integration testing is needed to test how the

system under test works with other systems

‰ Just because a system works in a test environment doesn’t

mean it will work in a live environment

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Non-Functional System Testing

© SIM Group Ltd., SQS Group AG, 2002

Non-Functional System Testing

„ In this session we will

‰ Understand what Non-Functional System Testing is

‰ Understand the need for non-functional system testing

‰ Look at the various types of non-functional system testing

© SIM Group Ltd., SQS Group AG, 2002

Non-Functional System Testing

„ Non-Functional System Testing is defined as:

“Testing Of Those requirements that do not relate to the

Functionality e.g. Performance, Load, usability, security etc”

‰ Users, generally, focus on the Functionality of the system

© SIM Group Ltd., SQS Group AG, 2002

Non-Functional System Testing

‰ Functional testing may show that the system performs as per

the requirements

‰ Will the system work if now deployed?

‰ There are other factors critical to the successful use of a

system

© SIM Group Ltd., SQS Group AG, 2002

Security Testing

“Testing whether the system meets specified security

requirements”

‰ How easy is it for an unauthorised user to access the

system?

‰ How easy is it for an unauthorised person to access the data?

© SIM Group Ltd., SQS Group AG, 2002

Usability Testing

“Testing the ease with which users can learn and use the
product”

‰ Employ people to use the application as a user and study

how they use it

‰ A beta-test may be a cheap way of doing Usability testing

‰ There is no simple way to examine HOW people use the

product

© SIM Group Ltd., SQS Group AG, 2002

Storage Testing

“Testing whether the system meets its specified storage

objectives”

‰ Study how memory and storage is used by the product

‰ Predict when extra capacity may be needed

© SIM Group Ltd., SQS Group AG, 2002

Installability Testing

“Testing concerned with the installation procedures for the

system”

‰ Does the installation work?

‰ Is it easy to carry out?

‰ Does it uninstall correctly?

© SIM Group Ltd., SQS Group AG, 2002

Documentation Testing

“Testing concerned with the accuracy of the documentation”

© SIM Group Ltd., SQS Group AG, 2002

Recovery Testing

“Testing aimed at verifying the system’s ability to recover from

varying degrees of failure”

‰ Should include recovery from system back-ups

© SIM Group Ltd., SQS Group AG, 2002

Load Testing

“Testing geared to assessing the applications ability to deal with

the expected throughput of data & users”

© SIM Group Ltd., SQS Group AG, 2002

Stress Testing

“Assesses individual components by exercising them to and

beyond the limits of expected use”

© SIM Group Ltd., SQS Group AG, 2002

Performance Testing

“Tests the efficiency of individual components of an application”

© SIM Group Ltd., SQS Group AG, 2002

Volume Testing

“Testing where the system is subjected to large volumes of

data”

© SIM Group Ltd., SQS Group AG, 2002

Non-Functional System Testing

„ Summary

‰ Just because a system’s functions have been tested doesn’t

mean that testing is complete

‰ There are a range of non-functional tests that need to be

performed upon a system

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Functional System Testing

© SIM Group Ltd., SQS Group AG, 2002

Functional System Testing

„ In this session we will

‰ Understand what functional system testing is

‰ Understand the benefits of functional system testing

© SIM Group Ltd., SQS Group AG, 2002

Functional System Testing

„ What is System Testing?

‰ Testing of the complete system

z May be the last step in integration testing in the small
z May be the first time that enough of the system has been put
together to make a working system

‰ Ideally done by an independent test team

‰ Two types - functional and non-functional

© SIM Group Ltd., SQS Group AG, 2002

Functional System Testing

„ A Functional Requirement is

“A requirement that specifies a function that a system or system

component must perform”

‰ Functional System testing is geared to checking the function

of the system against specification

‰ May be requirements based or business process based

© SIM Group Ltd., SQS Group AG, 2002

Functional System Testing

„ Testing Based on Requirements

‰ Requirements specification used to derive test cases

‰ System is tested to ensure the requirements can be met

© SIM Group Ltd., SQS Group AG, 2002

Functional System Testing

„ Testing carried out against Business processes

‰ Based on expected use of the system

‰ Builds use cases - test cases that reflect actual or expected

use of the system

© SIM Group Ltd., SQS Group AG, 2002

© SIM Group Ltd., SQS Group AG, 2002
Functional System Testing

„ Summary

‰ Functional system testing allows us to test the system against

the specifications, user requirements and business processes

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Integration Testing in the Small

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

„ In this session we will

‰ Understand what “Integration Testing in the Small” is

‰ Look at how & why we do “Integration Testing in the Small”

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

‰ Integration testing
z “Testing performed to expose faults in the interfaces and in the
interaction between integrated components”

‰ Integration testing in the small

z Testing performed to expose faults in the interfaces and in the
interaction between components

© SIM Group Ltd., SQS Group AG, 2002

A sample system

Databases

The SUT
- made up of 000’s of units

External
systems

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

„ All components of a computer application work on only

one thing

DATA

‰ Data is passed from one component to another

‰ Data is passed from one system to another

‰ Software components will be required to add, amend, &

delete information, validate the information and report on the
information

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

„ Integration testing

‰ Testing should start with the smallest, definable component

‰ Building tested components up to the next level of definition

‰ Eventually the complete business processes is tested ‘End to

End’

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

„ Testing components

‰ Each component needs to be tested individually to ensure it

processes the data as required

‰ This is Component Testing

‰ All possible options & paths should be tested

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

„ Once components have been tested we can link them

together to see if they can work together

‰ This is Integration Testing in the Small

z Linking components together to ensure that they communicate
correctly
z Gradually linking more and more components together
z Prove that the data is passed between components as required
z Steadily increase the number of components and create & test
sub-systems and then the complete system

© SIM Group Ltd., SQS Group AG, 2002

The approach used

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing

„ Planning testing

‰ We need to determine when and at what levels Integration

Testing will be performed

‰ Identify the boundaries

‰ A similar process will need to be followed for all elements

z Once these elements have been tested individually they are
further integrated to support the business process

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing

„ Stubs and drivers

‰ Used to emulate the “missing bits”

z External systems
z Sub-systems
z Missing components

‰ May need to be written specifically for the SUT

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing

„ Approaches to integration testing

‰ Incremental
z Top down
z Bottom-up
z Functional

‰ Non-incremental
z “Big bang”

© SIM Group Ltd., SQS Group AG, 2002

Integration Testing in the Small

„ Summary

‰ Integration testing is needed to test how components interact

with each other and how the system under test works with
other systems

‰ Integration testing in the small is concerned with the

interaction between the various components of a system

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Component Testing

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ In this session we will

‰ Understand what Component Testing is

‰ Look at the Component Testing Process

‰ Look at the myriad types of Component Testing

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ What is component testing?

‰ “The testing of an individual software component”

„ What is a component?

‰ “A minimal software item for which a separate specification is

available”

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ Component testing is the lowest form of testing

‰ At the bottom of the V-model

‰ Each component is tested in isolation

z Prior to being integrated into the system

‰ Involves testing the code itself

z Test the code in the greatest detail
z Usually done by the components developer

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ Component Test Process

‰ Component Test Planning

‰ Component Test Specification
‰ Component Test Execution
‰ Component Test Recording
‰ Checking for Component Test Completion

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ Testing techniques

‰ Equivalence Partitioning ‰ Data Flow Testing

‰ Boundary Value Analysis ‰ Branch Condition Testing

‰ Branch Condition
‰ State Transition Testing
Combination Testing
‰ Cause-Effect Graphing ‰ Modified Condition Decision
‰ Syntax Testing Testing
‰ LCSAJ Testing
‰ Statement Testing
‰ Random Testing
‰ Branch/Decision Testing
‰ Other Testing Techniques

For definitions see BS7925-

BS7925-2

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ What about “Playtime”?

‰ Often a short unscripted session with a component can be of

benefit

‰ More common at later stages (once formal test techniques

have been completed)

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ Not done too well?

„ Some basics would quickly improve it:

‰ Checklists
‰ Standards
‰ Procedures
‰ Code & version control
‰ Sign off

© SIM Group Ltd., SQS Group AG, 2002

Component Testing

„ Summary

‰ Component testing is intended to test a software component

before it is knitted into the system

‰ There are lots of different approaches

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Maintenance Testing

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ In this session we will

‰ Look at the challenges that face testers when an application

changes post implementation

‰ See how to ensure that maintenance applied to the system

does not cause failures

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ What is Maintenance Testing?

‰ Testing of changes to existing, established systems

z Where maintenance has been performed on the (old) code

‰ Checking that the fixes have been made and that the system
has not regressed

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ The Challenge

‰ Old Code

‰ No Documentation

‰ Out of Date Documentation (worse)

‰ How much testing to do and when to stop testing

z Mitigation of risk vs. lost revenue

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ Often applications will need maintenance

‰ In doing this minor changes will be required

‰ These need to be tested quickly & effectively to allow service

to be restored

‰ These systems may have been in place and working for

years
z They are likely to be vital to the running of the business
z They may be in use 24/7

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ You need to be able to test the changes quickly and

effectively

‰ As well as being able to prove that the change introduced

does not impact the other functions

‰ A Regression test is in order

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ A regression test is

‰ “Retesting of a previously tested program following

modification to ensure that faults have not been introduced or
uncovered as a result of the changes made”

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ Other Risks

‰ Reactive, high risk when changing

‰ Impact analysis is vital but is difficult

‰ Stopping testing too soon - errors may be missed

‰ Stopping testing too late - missed business through delayed

implementation

© SIM Group Ltd., SQS Group AG, 2002

Maintenance Testing

„ Summary

‰ All established systems need maintenance from time to time

‰ The changed code / function will need to be tested

‰ A regression test will need to be executed to ensure that the

change(s) have been made correctly and that they have not
affected some other part of the system

‰ Impact analysis is key, but difficult

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Black & White Box Testing

© SIM Group Ltd., SQS Group AG, 2002

Black & White Box Testing

„ In this session we will

‰ Understand the differences between Black & White Box

testing and where they feature in a testing lifecycle

‰ Understand how a systematic approach provides confidence

‰ Understand how tools can be used to improve and increase

productivity

© SIM Group Ltd., SQS Group AG, 2002

Black & White Box Testing

„ Black Box testing

‰ “Test Case selection that is based on an analysis of the

specification of a component without reference to its internal
workings”

„ White Box testing

‰ “Test Case selection based on an analysis of the internal

structure of a component”

© SIM Group Ltd., SQS Group AG, 2002

Black Box Testing

„ Concentrates on the Business Function

‰ Can be used throughout testing cycle

‰ Dominates the later stages of testing

z Although is relevant throughout the development life cycle

‰ Little / No knowledge of the underlying code needed

© SIM Group Ltd., SQS Group AG, 2002

Black Box Testing

„ We have all done it!

‰ What do you do when

changing a light bulb?

‰ `Remove Old Bulb

‰ Insert New Bulb

‰ Switch light / lamp on

‰ Switch light / lamp off

© SIM Group Ltd., SQS Group AG, 2002

White Box Testing

„ Testing at a detailed level

‰ A.K.A. “Glass Box testing” or “Structural testing”

‰ Focuses on Lines of Code

‰ Looks at specific conditions

‰ Looks at the mechanics of the Application

‰ Useful in the early stages of testing.

© SIM Group Ltd., SQS Group AG, 2002

White Box Testing

„ Requires a detailed knowledge of the code

‰ Business Function / Process not a prime consideration

‰ Aim to achieve code coverage, not just functional coverage

‰ Plays a lesser role later in the testing cycle

© SIM Group Ltd., SQS Group AG, 2002

White Box Testing

„ Code Coverage objective

‰ To devise tests that exercise each significant line of program

code at least once

‰ This does not mean every combination of paths and data -

which is normally unattainable

© SIM Group Ltd., SQS Group AG, 2002

Black & White Box Testing

„ Techniques and measurements

‰ Systematic techniques exist for black and white box testing

z Give us a systematic approach to testing
z Test cases are developed in a logical manner
z Enable us to have a sub-set of tests that have a high probability
of finding faults

‰ By using defined techniques we should be able to produce

corresponding measurements
z Means we can see how testing is progressing

© SIM Group Ltd., SQS Group AG, 2002

Black & White Box Testing

„ Software Testing Tools

‰ Are useful for both Black & White Box Testing

‰ Can increase productivity and quality of tests

‰ Particularly useful for white box testing

© SIM Group Ltd., SQS Group AG, 2002

Black & White Box Testing

„ Summary

‰ A systematic approach is needed for both

‰ Tests need to be planned, prepared, executed and verified

against the code

‰ Expected results need to be defined and understood

‰ Tools can help increase productivity and quality

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Black box test techniques

© SIM Group Ltd., SQS Group AG, 2002

Black box test techniques

„ In this session we will

‰ Understand what black box testing is

‰ Look at some of the different types of black box testing

© SIM Group Ltd., SQS Group AG, 2002

Black box test techniques

„ Black box testing is

‰ “Test Case selection that is based on an analysis of the

specification of a component without reference to its internal
workings”

© SIM Group Ltd., SQS Group AG, 2002

Black box test techniques

„ Why do we need black box test techniques?

‰ Exhaustive testing is not possible

z Due to the constraints of time, money and resources

‰ Therefore we must create a sub-set of tests

z These must have fewer tests, but should not reduce coverage

‰ We should also focus on areas of likely risk

z Those places where mistakes may occur

© SIM Group Ltd., SQS Group AG, 2002

Black box test techniques

„ Each black box test technique has

‰ A technique
z i.e. how to do it

‰ A test case design approach

z How to create test cases using the approach

‰ See BS7925-2 for detailed information

© SIM Group Ltd., SQS Group AG, 2002

Black box test techniques

„ BS7925-2 lists all the black box test techniques

‰ Equivalence Partitioning
‰ Boundary Value Analysis
‰ State Transition
‰ Cause & Effect Graphing
‰ Syntax Testing
‰ Random Testing

© SIM Group Ltd., SQS Group AG, 2002

Equivalence Partitioning

„ Uses a model of the component to

‰ Partition Input & Output values into sets such that each value
within a set can be reasonably expected to be treated in the
same manner

‰ Therefore only one example of that set needs to be input to or

resultant from the component

© SIM Group Ltd., SQS Group AG, 2002

Equivalence Partitioning

„ Test Case Design

‰ Inputs to the component

‰ Partitions exercised:
z Valid partitions
z Invalid partitions

‰ Expected results

© SIM Group Ltd., SQS Group AG, 2002

Equivalence Partitioning

„ Example

‰ Course grade

z Coursework - out of 25 marks

z Examination - out of 75 marks

z Grades A-D based on Coursework and Examination marks

z Identify Valid Partition and Invalid Partitions

z Define test cases including expected results

© SIM Group Ltd., SQS Group AG, 2002

Boundary Value Analysis

„ Uses a model of the component to

‰ Identify the values close to and on partition boundaries

‰ For input and output, valid & invalid data

‰ Chosen specifically to exercise the boundaries of the area

under test

© SIM Group Ltd., SQS Group AG, 2002

Boundary Value Analysis

„ Test Case Design

‰ Inputs to the component

‰ Boundaries to be exercised:
z Just below
z On
z Just above

‰ Expected results

© SIM Group Ltd., SQS Group AG, 2002

Boundary Value Analysis

„ Examples

‰ Using the grading system defined earlier

‰ The exam mark and the coursework mark both have two
boundaries

‰ Select values on, immediately above and immediately below

each boundary

© SIM Group Ltd., SQS Group AG, 2002

State Transition

„ Uses analysis of the specification of the component to

model its behaviour by state transitions

‰ Identifies
z The various states that the software may be in
z The transitions between these points
z The events that cause the transactions
z Actions resulting from the transactions

© SIM Group Ltd., SQS Group AG, 2002

State Transition

„ Test Case Design

‰ Identify possible transitions

‰ Identify initial state and the data / events that will trigger the
transition

‰ Identify the expected results

© SIM Group Ltd., SQS Group AG, 2002

State Transition

„ Example

‰ Using the course grade example

‰ There is a grade indicator on screen

z When coursework and examination marks are entered it
displays the persons grade (A-D)

‰ Select sets of data to test for each grade

z And extra sets of data to change from one grade to the next

© SIM Group Ltd., SQS Group AG, 2002

Other techniques

‰ Cause & Effect Graphing & Tabling

z Analysis of the specification to produce a graphical model of
causes and their effects
z Cases expressed as Boolean conditions in the form of a
decision table

‰ Syntax Testing
z Uses the syntax of the components’ inputs as the basis for the
design of the test case

‰ Random Testing
z Not as odd a concept as it sounds
z Not Ad-Hoc testing, but a means of testing functionality with a
randomly selected set of values

© SIM Group Ltd., SQS Group AG, 2002

Black Box Testing Techniques

„ Summary

‰ Black box testing concentrates on testing the features of the

system

‰ Techniques enable us to maximise testing

z Create an achievable set of tests that offer maximum coverage
z Ensure possible areas of risk are tested

‰ Black box testing is relevant throughout the testing process

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

White Box Test Techniques

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ In this session we will

‰ Understand what White Box testing is

‰ Look at some of the different types of White Box testing

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ White Box testing

‰ “Test Case selection based on an analysis of the internal

structure of a component”

‰ Also known as Glass Box testing, Clear Box testing

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Why do we need White Box

techniques?

‰ Provide formal structure to

testing code

‰ Enable us to measure how

much of a component has been
tested

‰ Example
z <100 lines of code
z 100,000,000,000,000 possible
paths
z At 1,000 tests per second
would still take 3,170 years to
test all paths

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ To plan and design effective cases requires a

knowledge of the

‰ Programming language used

‰ Databases used

‰ Operating system(s) used

‰ And ideally knowledge of the code itself

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ BS7925-2 lists all the white box test techniques

‰ Statement Testing
‰ Branch / Decision Testing
‰ Branch Condition Testing
‰ Branch Condition Combination Testing
‰ Modified Condition Decision Testing
‰ Linear Code Sequence & Jump
‰ Data Flow Testing

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Statement Testing

‰ “A test case design technique for a component in which test

cases are designed to execute statements”

‰ Test cases are designed and run with the intention of

executing every statement in a component

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Statement Testing example

a;
if (b)
{
c;
}
d;

Any test case with b TRUE will achieve full statement coverage

NOTE: Full statement coverage can be achieved without

exercising with b FALSE

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Branch / Decision testing

‰ A technique used to execute all branches the code may take

based on decisions made

‰ Test Cases designed to ensure all branches & decision points

are covered

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Branch / Decision testing example

a;
if (b)
{
c;
}
d;

100% statement coverage requires 1 test case (b = True)

100% branch / decision coverage requires 2 test cases

(b = True & b = False)

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Other techniques

‰ Branch Condition testing, Branch Condition Combination

testing & Modified Condition Decision testing
z Condition testing is based upon an analysis of the conditional
control flow within the component

‰ LCSAJ
z Linear Code Sequence And Jump

‰ Data flow testing

z Tests the flow of data through a component

© SIM Group Ltd., SQS Group AG, 2002

White Box Test Techniques

„ Summary

‰ White box testing can be done immediately after code is

written
z Doesn’t need the complete system
z Does need knowledge of the code

‰ A combination of all techniques are required for a successful

test
z Don’t rely on just one technique

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Error Guessing

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ In this session we will

‰ Understand how you can use experience to ‘predict’ where

errors may occur

‰ Understand how this can benefit testing

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ Used to guess where errors may be lurking

‰ Based on information about how the system has been put

together and previous experience of testing it

‰ Use to complement more systematic techniques not instead

of

‰ Not ad-hoc testing, but testing that targets certain parts of the
application

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ Based on the user's or tester's experience of the

commercial aspects of the system

‰ e.g.

z Table based calculations such as calculating benefits payments

z Where the user is allowed a high degree of flexibility in GUI

navigation using multiple windows

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ Experience of the developers or the development cycle

‰ e.g.

z Knowledge of an individual's ‘style’

z Knowledge of the development lifecycle and especially the

change management process

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ Experience of the operating systems used

‰ e.g.

z It is known that Windows NT is better at storage management

than Windows 95

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ Should still be planned

‰ Is part of the test process

z Should be used as a supplement to systematic techniques

‰ Ad-Hoc testing can be encouraged if

z The tester maintains a log of those tests performed & is able to
reasonably recollect what actions were performed

‰ If you have the time to run an extra (Ad-Hoc) test that passes
there is little loss
z If it finds a defect then that’s a bonus

© SIM Group Ltd., SQS Group AG, 2002

Error Guessing

„ Summary

‰ You can never do enough testing!

z The more tests that you can run the more errors you may find

‰ Error guessing can help ensure that areas where defects are
likely to occur are fully tested

‰ Uses experience and gut feeling to supplement systematic

testing techniques

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Reviews and the Test Process

© SIM Group Ltd., SQS Group AG, 2002

Static Testing

„ Over the next three sessions we will

‰ Discover the various approaches to Static Testing

‰ Discover the benefits of Static Testing

‰ Discover the issues with Static Testing

© SIM Group Ltd., SQS Group AG, 2002

Static Testing

„ What is Static Testing?

‰ “Testing of an object without execution on a computer”

© SIM Group Ltd., SQS Group AG, 2002

Static Testing

Session Topics

1. Reviews and the Test Process

2. Types of Review
3. Static Analysis

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ Session Objectives

‰ To identify what and when to Review

‰ To cover the Costs & Benefits of Reviews

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ Testing without data

‰ “Testing of an object without execution on a computer”

„ How is this done

‰ By reviewing the system deliverables

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ Why review

‰ To identify faults as soon as possible in the development

lifecycle

‰ Reviews offer the chance to find faults in the system

specifications

‰ This should lead to:

z Development productivity improvements
z Reduced development time-scales
z Lifetime cost reductions
z Reduced fault levels

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ When to review?

‰ As soon as an object is ready, before it is used as a product

or the basis for the next step in development

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ What to Review

‰ Anything and everything can be reviewed

‰ Requirements, System and Program Specifications should be

reviewed prior to publication

‰ System Design deliverables should be reviewed both in terms

of functionality & technical robustness

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ What should be reviewed?

‰ Program specifications should be reviewed before

construction

‰ Programs should be reviewed before execution

‰ Test Plans should be reviewed before execution

‰ Test Results should be reviewed before implementation

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ Reviews can be hazardous

‰ If misused they can lead to friction

‰ The errors & omissions found should be regarded as a good

thing

‰ The author(s) should not take errors & omissions personally

‰ It is a positive step to identify defects and issues before the

next stage

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ Building a Quality culture

‰ In order for them to work, Reviews must be regarded as a

positive step and must be properly organised

‰ This is often a part of the company’s Quality culture

z Define the review panel
z Define the roles of its’ members
z Define the review procedures
z “Sell” the quality culture to staff

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ There are other dangers in a regime of regular reviews

‰ Lack of Preparation

‰ “Familiarity breeds contempt”

‰ No follow up to ensure correction has been made

‰ The wrong people doing them

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ Systems development is very complex

‰ This means that it is difficult to build a system with all

elements completely & accurately specified

‰ By reviewing a deliverable, its deficiencies can be identified

and remedied earlier – saving time and money

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

What else can Testers review?

„ Test plans „ Causes of defect
„ Test cases „ Test metrics
„ Test results „ Development metrics
„ Defects „ Operational defects

We can measure how well we do

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ The costs of reviews

‰ On-going reviews cost approximately 15% of development

budget

‰ This includes activities such as the review itself, metric

analysis & process improvement

‰ Reviews are highly cost effective

z Finding and fixing faults in reviews is significantly cheaper than
finding and fixing faults in later stages of testing

© SIM Group Ltd., SQS Group AG, 2002

Reviews & the Test Process

„ The benefits of reviews

‰ Reviews save time and money

‰ Development productivity improvements

z People take greater care
z They have more pride in doing good work
z They have a better understanding of what they are required to
deliver

‰ Reduced development costs

z Reduced fault levels
z Reduced lifetime cost reductions

© SIM Group Ltd., SQS Group AG, 2002

Reviews and the Test Process

„ Summary

‰ Reviews enable us to test that the systems specification is

correct and relates to the users’ requirements

‰ Anything generated by a project can be reviewed

‰ In order for them to be effective, reviews must be well

managed

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Types of Review

© SIM Group Ltd., SQS Group AG, 2002

Static Testing

Session Topics

1. Reviews and the Test Process

2. Types of Review
3. Static Analysis

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ In this session we will

‰ Look at the types of review & the activities performed

‰ Look at goals & deliverables
‰ Look at roles & responsibilities
‰ Examine the suitability of each type of review
‰ Discover review pitfalls

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

‰ Author’s Review
‰ Informal Review
‰ Walkthroughs
‰ Presentations
‰ Prototypes
‰ Checklists
‰ Templates
‰ Technical Review
‰ Inspections

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Author’s review

‰ Before issuing a document, the author should ALWAYS

check
z Conformance to company standards
z Fitness for purpose
z Presentation
z Spelling

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Informal Review

‰ It is very helpful to ask a colleague to read through the

deliverable to
z Identify jargon to eliminate or translate
z Make sure it all makes sense
z Rationalise the sequence
z Identify where the author’s familiarity with the subject assumes
the reader knows more than he or she actually does

‰ This is the least effective form of review as there is no

recognised way of measuring it

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Walkthrough

‰ For some technical deliverables or complex processes it is

very useful for the author to walk through the document

‰ Reviewers are asked to comment and query the concepts

and processes that are to be delivered

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Presentations

‰ Used to explain to users your understanding of their

Requirements

‰ To explain how you will test those requirements i.e. your

approach

‰ To gain agreement for the next step

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Prototypes

‰ Allows the non-technician to see what is going to be built

‰ Enables the business to refine their business flows or adjust

sequences on input screens

‰ Is used to demonstrate and display features of the [proposed]

system
z GUI screens, Database designs, Network architecture etc
z Has been around for years in the form of mock ups and screen
layouts
z Much improved with the advent of GUI’s and tools to paint
screens quickly and effectively

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Checklists

‰ Extremely simple and useful method of ensuring that all

elements of a deliverable have been considered

‰ Tends to ensure conformity of content, sequence and style

‰ Can be devised for almost any activity

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Templates

‰ Very useful to standardise the style, sequence & content of

code and documents

‰ Saves time re-inventing

‰ Requires agreement by all who will use them

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Technical (Peer) Review

‰ Your Peers can also perform more formal technical reviews in

larger groups

‰ Often used by workgroups as a self-checking mechanism

‰ It’s format is defined, agreed and practised with documented

processes and outputs

‰ Requires technical expertise

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Inspections

‰ Focus on reviewing code or documents

‰ Formal process based on rules and checklists

z Chaired by Moderator
z Every attendee has a defined role
z Includes metrics and entry & exit criteria
z Training is required

‰ Reviewers must be prepared prior to attendance

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Fagan Inspections

‰ Are very effective - to the point where they have replaced

unit testing

‰ Defects are reported - no debate

‰ Contentious issues and queries are resolved outside the

inspection and the result reported back to the Moderator

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Goals

‰ Validation and verification against specifications and

standards

‰ Achieve consensus before moving on to next stage

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Activities

‰ Reviews need to be planned

‰ They may require overview meetings

‰ Preparation

‰ The review meeting

‰ Follow-up

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Roles & responsibilities

‰ For each type of review (especially the formal ones) roles and
responsibilities must be clearly defined

‰ Training may be required to understand and perform role

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Deliverables

‰ Any project deliverable can (and should) be reviewed

‰ Reviewing these items can lead to

z Product changes
z Source document changes
z Improvements (to both reviews and development)

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Pitfalls

‰ Lack of training
z And therefore understanding of role, responsibilities and
purpose of review

‰ Lack of documentation

‰ Lack of management support

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Review Management

‰ Formal reviews require proper management

z Allow people time to prepare
z Issue guidance notes (if not in Standards)
z Give plenty of notice
z Practice Presentations & Walkthroughs
z Sound out “contentious issues”
z ALWAYS document outcomes & Actions

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

Peer Technical
Walkthrough Inspections
Review Reviews
What Req.'s & Designs Everything Everything Specs & Code

Approach Informal Informal Formal Formal

Why / when Before detail Always* Before detail Before sign off

By Who Author(s) Peer Author(s) Moderator

To Whom IT & Business Author IT & Business Reviewers

Always* : can be done at all points when a document is released.

© SIM Group Ltd., SQS Group AG, 2002

Types of Review

„ Summary

‰ There are various types of reviews

z Companies must decide which one(s) are best for them

‰ In order to gain maximum benefit from them they must be

organised and implemented

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Static Analysis

© SIM Group Ltd., SQS Group AG, 2002

Static Testing

Session Topics

1. Reviews and the Test Process

2. Types of Review
3. Static Analysis

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ In this session we will:

‰ Understand what static analysis is

‰ Look at some of the elements of Static Analysis

z Compilers
z Static analysis tools
z Data flow analysis
z Control flow analysis
z Complexity analysis

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ What is Static Analysis?

‰ Analysis of the code without dynamic execution

z Examine the logic and syntax of the code

‰ Attempting to identify faults in the code

‰ Provides metrics on flow through the SUT and its complexity

‰ A form of automation - usually done with tools

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ What do we hope to find?

‰ Faults in the code

z Unreachable code
z Undeclared variables
z Parameter type mismatches
z Uncalled functions and procedures
z Possible array boundary violations

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ What else will we get?

‰ Information on the code

z Its complexity
z The flow of data through it
z The control flow through it

z Metrics and measurements

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ Compilers

‰ Convert program language code into Machine Code for faster

execution

‰ Will perform some basic Static Analysis

z Usually produces a source code list
z Often produces a memory map
z Often produces a cross-reference of labels and variables
z Will do a syntax check on the code prior to compilation

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ Static Analysis Tools

‰ A Static Analyser is used to parse the Source Code & identify

defects, such as

z Missing branches and links

z Non returnable “performs”
z Variables not initialised
z Standards not met
z Unreachable code

‰ Will also find any faults found by a Compiler

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ Data Flow Analysis

‰ Technique to follow the path that various specific items of

data take through the code of a program

‰ Looking for possible anomalies in the way the code handles

the data items
z Variables used without being first created
z Variables being “used” after they have been “killed”

‰ Consider a CRUD matrix to detail which components affect

which data items

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ Control Flow Analysis

‰ Examines the flow of control through the code

‰ Can identify
z Infinite loops
z Unreachable code
z Cyclomatic complexity and other metrics

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

Control Flow Graph

B1 B2 B3 B4 B8

B9 B5 B6

B7

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ Metrics

‰ Static Analysers generate metrics based on the code

‰ Cyclomatic Complexity
z Identifies the number of decision points in a control flow
z Can be calculated by counting the number of decision points in
a control flow graph and adding 1

‰ Other Complexity Metrics

z Number of Lines of Code
z Number of nested statements

© SIM Group Ltd., SQS Group AG, 2002

Static Analysis

„ Summary

‰ The inspection of program code and logic for errors

‰ The code is analysed without being executed by the tool

‰ Errors are highlighted and metrics are produced

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Test Organisation

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ In this session we will

‰ Gain an appreciation of the many different testing structures

in place

‰ Understand that it is likely that every organisation will have a

unique testing set-up

‰ Look at some of the more common, general set-ups and

appreciate the impact they may have on the success of the
testing carried out

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Testing may be the responsibility of the individual

developers

‰ Developers are only responsible for testing their own work

‰ Goes against the ideal that testing ought to be independent

‰ Tends to confirm the system works rather than detect errors

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Testing may be the responsibility of the developer

team

‰ A “Buddy system”

‰ Each developer is responsible for testing his / her colleagues

work

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Each team has a single tester

‰ Is not involved in the actual development, solely concentrates

on testing

‰ Able to work with the team throughout the development

process

‰ May be too close to the team to be objective

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Dedicated test teams may be in place

‰ These do no development whatsoever

‰ Take a more objective view of the system under test

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Internal Test Consultants

‰ Provide testing advice to project teams

‰ Involvement throughout the lifecycle

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Testing outsourced to specialist agencies

‰ Guarantees independence

‰ Empathy with the Business of the organisation?

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Multi-disciplined teams needed to cover

‰ Production of Testing ‰ Logging Results

Strategies
‰ Executing necessary re-tests
‰ Creation of Test Plans
‰ Automation expertise
‰ Production of Testing Scripts
‰ Technical support
‰ Proving of Testing Scripts z Database admin,
environment admin
‰ Execution of Testing Scripts
‰ User Interface expertise

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Testing teams will typically include

‰ Test analysts to prepare, execute and analyse tests

‰ Test Consultants prepare strategies and plans
‰ Test automation experts
‰ Database administrator or designer
‰ User interface experts
‰ Test environment management
‰ And others..

© SIM Group Ltd., SQS Group AG, 2002

Test Organisation

„ Summary

‰ Testing structures will vary between organisations

‰ Testing teams require a range of skills

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Configuration Management

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ In this session we will

‰ Gain an understanding of Configuration Management

‰ Look at the potential impact of poor Configuration

Management on testing

‰ And understand how introducing Configuration Management

helps address those problems

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ What is Configuration Management?

‰ Ensuring that no-one is able to change a part of the system

without proper procedures being in place

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Closely linked to Version Control

‰ Version Control looks at each component

‰ Holds the latest version of each component

‰ What versions of components works with others in a

configuration

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Without it, testing can be seriously impeded

‰ Developers may not be able to match source to object code

‰ Simultaneous changes may made to the same source

z Although this may be allowed

‰ Unable to identify source code changes made in a particular

version of the software

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Critical to successful testing

‰ It associates programs with specifications, with test plans,

with test data

‰ When a component is successfully tested test results can be

included

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ It enables you to understand what versions of

components work with each other

‰ It allows you to understand the relationship between test

cases, specifications, and components

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ If any configuration component is out of step - Results

are unpredictable

‰ This applies throughout the project lifecycle, and beyond

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

‘…is as old as the pyramids. It allows you at any point in

time to examine the product of your work. (a program,
requirements, tests etc.,) and know: Who, What When,
Where, Why & How’

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Everything intended to last must be subject to

Configuration Management

‰ Anything that isn’t doesn’t exist as part of the product

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Other Configuration Management terms

‰ Configuration identification
z Requires that all Configuration Items (CI’s) and their versions
are known

‰ Configuration control
z Maintenance of the Configuration Items in a library and
maintenance of records on how Configuration Items change
over time

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Other Configuration Management terms

‰ Status accounting
z The function recording and tracking problem reports, change
requests etc

‰ Configuration auditing
z The function to check on the contents of libraries etc for
standards compliance

© SIM Group Ltd., SQS Group AG, 2002

Configuration Management

„ Summary

‰ Configuration Management is a necessary part of any system

development

‰ All assets must be known and controlled

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Test Estimation, Monitoring & Control

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ In this session we will

‰ Understand how we estimate how long we need for testing

‰ Understand how we monitor the progress of testing once it

has started

‰ Understand what steps we take to ensure that the effort

progresses as smoothly as possible

© SIM Group Ltd., SQS Group AG, 2002

 Test Estimation, Monitoring & Control

Test Estimation

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Test Estimation

‰ The same as estimating for any project

‰ We need to identify the number of tasks (tests) to be

performed, the length of each test, the skills and resources
required and the various dependencies

‰ Testing has a high degree of dependency

z You cannot test something until it has been delivered
z Faults found need to be fixed and re-tested
z The environment must be available whenever a test is to be run

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Factors to Consider

‰ Risk of Failure

‰ Complexity of Code / Solution

‰ Criticality

‰ Coverage

‰ Stability Of SUT

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Testing is a tool to aid Risk Management

‰ Consider the cost to the business of failure of a feature

‰ Test the most important features as soon as possible

‰ Be prepared to repeat these tests

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Consider the complexity of the supporting code

‰ Estimate & plan tests for these ASAP

‰ These are likely to be re-run many times, allow for this in

plans

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Data is critical to testing

‰ You can in certain situations load the data for testing & test in
one pass

‰ These tests will be run first but only once… unless...

‰ The testing environment may need to be re-set before a test

can be re-run

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Use previous experience or best estimates to predict

the reliability of components

‰ Plan to run tests for the unreliable elements sooner rather

than later

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Allow time for re-testing

‰ Time must be included for identifying, investigating & logging

faults

‰ Faults that have been “fixed” must be re-tested

‰ Whenever a change is made to the SUT or the environment a

regression test should be run

‰ How do we build these factors into the estimation?

© SIM Group Ltd., SQS Group AG, 2002

 Test Estimation, Monitoring & Control

Test Monitoring

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Test Preparation

‰ Estimate number of tests needed

‰ Estimate time to prepare

‰ Refine these figures

‰ Track percentage of tests prepared

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Test Execution

‰ Assess the coverage achieved to date by testing

z Coverage - amount of tests completed, amount of code
exercised by tests, amount of features tested so far

‰ Estimate time required to run the test suite

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Whilst testing you can monitor

‰ The number of tests run

‰ The number of tests passed & failed

‰ The number of defects raised

z These can be categorised by Severity, Priority & Probability

‰ The number of re-tests - that pass & that fail

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ The status of the project should be regularly reported

‰ Any deviations from the schedule raised ASAP

‰ Any critical faults found should be raised immediately

© SIM Group Ltd., SQS Group AG, 2002

 Test Estimation, Monitoring & Control

Test Control

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Controlling measures

‰ Assign extra resource

‰ Re-allocate resource

‰ Adjust the test schedule

‰ Arrange for extra test environments

‰ Refine the completion criteria

© SIM Group Ltd., SQS Group AG, 2002

Test Estimation, Monitoring & Control

„ Summary

‰ Multiple factors must be considered when estimating the

length of time we need to perform testing

‰ Once testing has started it is necessary to monitor the

situation as it progresses

‰ Careful control must be kept to ensure project success

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Incident Management

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ In this session we shall

‰ Understand what an “incident” is

‰ Understand the impact they have on the testing process

‰ Understand how they are logged, and tracked

‰ Understand why & how they should be analysed to help

ensure they don’t happen again

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ An ‘incident’ is when testing cannot progress as

planned

‰ The application does not function as anticipated

‰ Components are missing etc

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ What is an incident?

‰ When actual results differ from expected results

‰ They may be raised against

z Documentation
z The code
z The System Under Test
z The Test Environment
z Or the tests themselves

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ The incident needs to be viewed in terms of its impact

on testing

‰ As testing progresses through its lifecycle priorities can

change

‰ The progress of the testing activity can be monitored by

analysing incidents

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ Top Priority must be given to any issues that prevent

testing from progressing

‰ At the second level we need to assign those that can be

worked around

‰ In the lower categories we log what are minor issues &

questions

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ As project deadlines get nearer what is considered

high priority may well change

‰ The emphasis will move towards the product and what will
compromise its success

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ When logging incidents testers should include

‰ Test ID, System ID, Testers ID

‰ Expected and Actual results

‰ Environment in use

‰ Severity or priority of the incident

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ Service levels should be set up for each category of

incident

‰ Progress monitored accordingly

‰ Involve Development, Testers and Users

‰ Issue regular status reports

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ STATUS should also be tracked

‰ This ensures that we are able to swiftly establish the situation

with an incident

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ A typical status flow is

‰ When first raised an incident is OPEN

‰ It then passes to development to fix - WITH DEVELOPMENT
‰ Development may question the incident - PENDING
‰ Or fix the problem and return it for - RETEST
‰ Should the retest prove successful the defect is CLOSED

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ Analysing incidents

‰ Incidents may be analysed to help monitor the test process

‰ And to aid in improvement of the test process

© SIM Group Ltd., SQS Group AG, 2002

Incident Management

„ Summary

‰ An incident is “any significant, unplanned event that occurs

during testing that requires subsequent investigation and / or
correction”

‰ Incidents should be recorded and tracked

‰ Analysis of incidents will enable us to see where problems

arose and to aid in test process improvement

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Standards for Testing

© SIM Group Ltd., SQS Group AG, 2002

Standards for Testing

„ In this session we will

‰ Understand the various standards that may influence testing

© SIM Group Ltd., SQS Group AG, 2002

Standards for Testing

„ There are three types of standards that affect testing

‰ Quality Assurance standards

‰ Industry specific standards

‰ Testing Standards

© SIM Group Ltd., SQS Group AG, 2002

Standards for Testing

„ Quality Assurance standards

‰ Only specify that testing should be done

z e.g. ISO 9000-3:1991

Quality management and quality assurance standards-Part 3:

Guidelines for the application of ISO 9001 to the development,
supply and maintenance of software

© SIM Group Ltd., SQS Group AG, 2002

Standards for Testing

„ Industry specific standards

‰ Specify what level of testing to perform

z e.g. railway, medical, insurance

ASTM F153-95 Standard Test Method for Determining the Yield of

Wide Inked Computer Ribbons

© SIM Group Ltd., SQS Group AG, 2002

Standards for Testing

„ Testing standards

‰ Specify how to perform testing

z e.g. BS7925-1 Software Testing vocabulary

BS7925-2 Software component testing
IEEE 829 Standard for Test Documentation

© SIM Group Ltd., SQS Group AG, 2002

Standards for Testing

„ Summary

‰ There are a range of standards that can influence testing

‰ These come from different sources and affect different areas

of testing

‰ Additionally, various industries have legal requirements that

will influence testing

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Types of CAST Tool

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ In this session we will

‰ Understand that help is at hand in the form of CAST Tools

z CAST - Computer Aided Software Testing

‰ Look at the range of tools available in the market today

‰ Identify the areas in which tools can best help the testing
process

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Why do we need / use them?

‰ To automate the most repetitious and time consuming tasks

‰ Tools are able to do some things that people cannot easily do

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Requirements testing „ Drivers & Simulators

„ Static Analysis „ Performance
„ Test Design „ Dynamic Analysis
„ Data Preparation „ Debugging
„ Character Based Test „ Comparison
Running „ Test Management
„ GUI Test Running „ Coverage Measurement
„ Test Harnesses

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Requirements Testing tools

‰ There are tools available to help verify and validate

requirements

‰ They use requirements models

‰ Can check consistency and animations

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Static Analysis tools

‰ There are tools on the market that can help with analysing
code

‰ These tools analyse the quality and complexity of the code

and paths through the code

‰ Many use McCabe, Halstead or McClure metrics

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Test Design tools

‰ Help with designing of test cases

‰ These generate test cases from a specification

z Normally held in a CASE tool repository
z Or from formally specified requirements held in the tool itself

‰ Some tools generate test cases from analysis of the code

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Data Preparation tools

‰ Help with data preparation

‰ These enable data to be selected from existing databases or

created, generated, manipulated and edited for use in tests

‰ May create data in a simple database or a complex

mainframe database

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Character Based Testing tools

‰ Used with character based applications

z Terminals on Mainframes, Unix and mid-range systems (System
36/38)
z Early PC (DOS) systems

‰ The tools simulate user entered keystrokes and capture

screen responses for comparison
z Normally used to automate regression and functional system
testing

‰ Many tools available to cover this

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ GUI Test Running tools

‰ Testing for GUI based applications

‰ Testing software running in graphical operating systems i.e.

Windows
z Most of our work in this area
z Mostly for functional and regression tests

‰ Tools must be able to identify and interact with the objects on

screen

‰ Tests scripted in a Test Script Language

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Web Testing tools

‰ Testing web based applications

‰ Increasingly tools are becoming available to help test E-

Commerce, web-based applications
z Existing tools that have “evolved” to include web testing
z Dedicated web testing tools

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool Test

„ Harnesses & Drivers

‰ These are key to building up integration tests

‰ Used to execute the software under test

z i.e. for systems that do not have a user interface

‰ These are small modules that drive data into the area under
test

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Simulators

‰ Simulate events or systems

‰ There will be times when it is necessary to test a system that

interacts with an event or system that does not physically
exist

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Performance Testing tools

‰ Measure what a system operates like in normal

circumstances
z Drive the SUT & monitor the response time for specified
transactions

‰ Load generation
z How the system performs when accessed by many users
z The tool simulates a number of users accessing the SUT

‰ The tools can simulate multiple sessions from one device

z LoadRunner, QALoad etc

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Dynamic Analysis tools

‰ Provide run time information on the state of the executing

software

‰ Typically used to monitor system resources

‰ Find errors that can be difficult to find “statically”

‰ i.e. use and de-allocation of system memory and flag memory

leaks

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Debugging tools

‰ Used mainly by programmers to reproduce bugs and

investigate the state of programs

‰ Debuggers enable programmers to execute programs line by

line, to halt the program at any statement and to set and
examine program variables

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Comparison tools

‰ Comparing two elements is a key activity in testing

‰ ‘Eyeballing’ information is labour intensive and prone to error

‰ Tools are available on many platforms to help here

‰ Comparison tools detect differences between files

z Typically actual and expected results files

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Test Management tools

‰ Enable testers to manage their tests

‰ Tests are created in the tool, linked together to form workflow,

tests executed and results collated

‰ Some of the better known tools

z i.e. TestDirector, SQA Manager, QADirector

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Coverage Measurement tools

‰ Report amount of code exercised by test executions

‰ The code is instrumented before compilation

‰ As the program is executed the instrument code records the

coverage data

‰ Reports generated can show the amount of code exercised

during tests
z The most common coverage measures are statement and
branch coverage

© SIM Group Ltd., SQS Group AG, 2002

Types of CAST Tool

„ Summary

‰ CAST Tools can help automate areas of the testing process

‰ There is a wide range of functions that can be automated

© SIM Group Ltd., SQS Group AG, 2002

 ISEB Foundation Certificate in Software Testing

Tool Selection & Implementation

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ In the session we will

‰ Examine the processes involved in identifying where tools

can help the testing process

‰ Learn how to approach the selection and implementation of

those tools

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ As indicated tools can help many aspects of testing

‰ How do you select where you want to start introducing CAST

support?

‰ The selection and implementation of a tool is a major task

and should be regarded as a project in its own right

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ If testing is currently badly organised tools may not be

the immediate answer

‰ Automating chaos simply speeds the delivery of chaos

‰ The benefits of tools usually depend on a systematic and

disciplined test process being in place

‰ Address the test processes and introduce disciplines first

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Examine the test process in detail

‰ Identify the weaknesses, and the cause of those weaknesses

‰ Find tools that fit in with your processes and address the
weak areas
z This may be more important than selecting the tool with the
most number of features

‰ Prioritise the areas of greatest importance

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Pick a number of tools to trial in a certain area

‰ Define clear requirements & success criteria

‰ Understand each tools’ capabilities & weaknesses

‰ Run trials or small pilot exercises to evaluate the tools

‰ Review and select tool

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Examine the Test Environment(s)

‰ One of the most crucial areas for tool implementation

‰ Shared environments will not accept automation unless the

SUT allows the data to be entered fresh every time

‰ Automation may also fill an environment up and make it run

out of space as it allows a large amount of data to be entered
in a short space of time

‰ Test facilities may be (will be!!!) less than Production facilities

and use of the tools with the SUT can cause systems to run
out of memory

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Funding must be secured

‰ Software testing tools are not low cost items and there is
usually a large amount of man hours required as well

‰ It is necessary to budget for both the tool and the resources

required to implement

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Evaluate the results of the trial

‰ Select the best candidate - pilot project

‰ Plan the implementation & use of that tool

‰ Implement & use

‰ Broadcast the benefits

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Plan the roll-out of the tool

‰ Gain commitment from the tools users

‰ Ensure new projects are aware of the cost of introducing a

tool

© SIM Group Ltd., SQS Group AG, 2002

Tool Selection & Implementation

„ Summary

‰ Tools can improve the organisation and efficiency of a test

process

‰ Before tools can be implemented a sound testing process

must be in place

‰ Tool selection and implementation must be carefully

considered and managed

© SIM Group Ltd., SQS Group AG, 2002