Software Testing Strategies

1
1
• Software testing needs to be planned in
advance and conducted systematically
• A number of software testing strategies have
been proposed in the literature
• All of them have these four generic
characteristics

2
2
Characteristics of Testing Strategies - 1
• (1) Testing begins at the component level and
works “outward” toward the integration of the
entire computer-based system
• (2) Different testing techniques are appropriate
at different points in time

3
3
Characteristics of Testing Strategies - 2
• (3) Testing is conducted by the developer of
the software and (for large projects) an
independent test group
• (4) Testing and debugging are different
activities, but debugging must be
accommodated in any testing strategy

4
4
 Testing Strategy - 1
• Unit testing
• Integration testing
• Validation testing
• System testing

5
5
 Testing Strategy

unit test integration

system validation
test test

6
 Testing Strategy - 2
• Unit testing concentrates on each unit (i.e.,
component) of the software as implemented in
source code
• Integration testing is next, where the focus is
on design and the construction of the software
architecture

7
7
 Testing Strategy - 3
• Next is the validation testing, where
requirements established as part of software
requirements analysis are validated against the
software that has been constructed
• Finally, we have system testing, where the
software and other system elements are tested
as a whole

8
8
 Testing Strategy: Unit Testing
• Unit testing makes heavy use of white-box
testing techniques, exercising specific paths in
a module’s control structure to ensure
complete coverage and maximum error
detection

9
9
 Testing Strategy: Integration Testing
• Next, components must be assembled or
integrated to form the complete software
package
• Integration testing addresses the issues
associated with the dual problems of
verification and program construction
• BB test case design techniques are the most
prevalent during integration

10
10
Testing Strategy: Validation Testing
• After software has been integrated, a set of
high-order tests are conducted
• Validation criteria (established during
requirements analysis) must be tested
• Validation testing provides final assurance that
software meets all functional, behavioral, and
performance requirements
• BBT techniques are used exclusively here
11
11
 Testing Strategy: System Testing
• System testing is the last high-order testing
and it falls outside the boundaries of software
engineering and into the broader context of
computer system engineering
• System testing verifies that all elements
(hardware, people, databases) mesh properly
and that overall system function/performance
is achieved

12
12
 The “V” Model of Software Testing

Requirements Validation

Design Integration

Coding Unit

13
 Strategic Issues - 1
• Specify product requirements in a quantifiable
manner long before testing commences
– NFRs
• State testing objectives explicitly
• Understand the users of the software and
develop a profile for each user category
– Actors/users of use-cases

14
14
 Strategic Issues - 2
• Develop a testing plan that emphasizes “rapid
cycle testing”
• Build “robust” software that is designed to test
itself
– Software should be capable of diagnosing certain
classes of errors
• Use effective formal technical reviews as a
filter prior to testing

15
15
 Strategic Issues - 3
• Conduct formal technical reviews to assess the
test strategy and test cases themselves
• Develop a continuous improvement approach
for the testing process

16
16
 Unit Testing - 1
• Unit testing focuses verification effort on the
smallest unit of software design – the software
component or module
• The relative complexity of tests and uncovered
errors is limited by the constrained scope
established for unit testing
• Different units can be tested in parallel

17
17
 Unit Testing

module
to be
tested

interface
local data structures

boundary conditions
independent paths
error handling paths

test cases

18
 Unit Testing - 2
• The module interface is tested to ensure that
information properly flows into and out of the
program unit under test
• The local data structure is examined to ensure
that data stored temporarily maintains its
integrity during all steps in an algorithm’s
execution

19
19
 Unit Testing - 3
• Boundary conditions are tested to ensure that
the module operates properly at boundaries
established to limit or restrict processing
• All independent paths through the control
structure are exercised to ensure that all
statements in a module have been executed
• All error handling paths are tested

20
20
 Unit Testing - 4
• Tests of data flow across a module interface
are required before any other test is initiated

21
21
 Unit Testing: Common Errors Found
• Among the more common errors in
computation are
– Misunderstood or incorrect arithmetic precedence
– Mixed mode operations
– Incorrect initialization
– Precision inaccuracy
– Incorrect symbolic representation of expression

22
22
 Unit Testing: Test Cases
• Test cases should uncover errors such as
– Comparison of different data types
– Incorrect logical operators or precedence
– Expectation of equality when precision error makes
equality unlikely
– Incorrect comparison of variables
– Improper or non-existent loop termination
– Failure to exit when divergent iteration is encountered
– Improperly modified loop variables

23
23
 Unit Testing: Error Handling
• Among the potential errors that should be tested when
error handling is evaluated are
– Error description is unintelligible
– Error noted does not correspond to error encountered
– Error condition causes system intervention prior to error
handling
– Exception-condition processing is incorrect
– Error description does not provide enough information to
assist in the location of the cause of the error

24
24
 Unit Testing: Boundary Testing
• Boundary testing is the last and probably the
most important task of unit testing step
• Software often fails at its boundaries
• Test cases that exercise data structure, control
flow, and data values just below, at, and just
above maxima and minima are very likely to
uncover errors

25
25
• Unit testing is simplified when a component
with high cohesion is designed
• When only one function is addressed by a
component, the number of test cases is
reduced and errors can be more easily
predicted and uncovered
• Let’s see a unit test environment

26
26
 Unit Test Environment

driver

interface
local data structures
Module boundary conditions
independent paths
error handling paths
stub stub

test cases

RESULTS

27
 Integration Testing - 1
• “If they all work individually, why do you
doubt that they’ll work when we put them
together?”
• The problem, of course, is “putting them
together” – interfacing

28
28
 Integration Testing - 2
• Data can be lost across an interface
• One module can have an inadvertent, adverse
affect on another
• Sub-functions, when combined, may not produce
the desired major function
• Individually acceptable imprecision may be
magnified to unacceptable levels
• Global data structures can present problems

29
29
Software Integration

30
 Integration Testing - 3
• Integration testing is a systematic technique
for constructing the program structure while at
the same time conducting tests to uncover
errors associated with interfacing
• Individual components have already been unit
tested and the structure is dictated by design

31
31
 Integration Testing - 4
• There is tendency, sometimes, to attempt non-
incremental integration – a sort of “big bang”
approach
• All components are combined in advance
• The entire software is tested as a whole
• And chaos usually results!
• We should rather use incremental integration

32
32
 Integration Testing Approaches
• Top-down integration
• Bottom-up integration
• Sandwich testing and integration

33
33
 Top Down Integration

A
top module is tested with
stubs

B F G

stubs are replaced one at

34
 Bottom-Up Integration

B F G

drivers are replaced one at a

worker modules are grouped into

cluster

35
Sandwich Testing and Integration
A
Top modules are
tested with stubs

B F G

Worker modules are grouped into

cluster

36
 Test Case Design - 1
• Any engineered product can be tested in one
of two ways
– Knowing the specified function that a product has
been designed to perform
– Knowing the internal workings of a product

37
 Test Case Design - 2
• In the first case, tests can be conducted that
demonstrate each function is fully operational
while at the same time searching for errors in
each function
• In the second case, tests can be conducted to
ensure that internal operations are performed
according to the specifications and all internal
components have been adequately exercised

38
 Test Case Design - 3
• In the first case, testing is focused on the
external behavior of a software system or its
various components, and we cannot see inside
the components
• While in the second case, testing is focused on
the internal implementation, and we must see
inside the component

39
 Testing Techniques
• Black-box testing (BBT)
– aka functional/behavioral testing
• White-box testing (WBT)
– aka structural/glass-box testing

40
 Black-Box Testing - 1
• Black-box testing alludes to tests that are
conducted at the software interface
• Although they are designed to uncover errors,
they are also used to demonstrate that software
functions are operational, that input is properly
accepted and output is correctly produced, and
that the integrity of external information is
maintained

41
 Black-Box Testing - 2
• A block-box test examines some fundamental
aspect of a system with little regard for the
internal logical structure of the software

42
 Black-Box Testing - 3
• The inner structure or control flow of the
application is not known or viewed as
irrelevant for constructing test cases. The
application is tested against external
specifications and/or requirements in order to
ensure that a specific set of input parameters
will in fact yield the correct set of output
values

43
 Black-Box Testing - 4
• It is useful for ensuring that the software more
or less is in concordance with the written
specifications and written requirements
• The simplest form of BBT is to start running
the software and make observations in the
hope that it is easy to distinguish between
expected and unexpected behavior

44
 Black-Box Testing - 5
• This is ad-hoc testing and it is easy to identify
some unexpected behavior, like system crash
• With repeated executions, we can determine
the cause to be related to software and then
pass that information to the people responsible
for repairs

45
 Black-Box Testing Approaches
• System testing of full application
• New function testing
• Lab testing
• Customer acceptance testing
• Field (Beta) testing

46
 White-Box Testing - 1
• White-box testing of software is predicated on
close examination of procedural detail
• Logical paths through the software are tested
by providing test cases that exercise specific
sets of conditions and/or loops
• The “status of the programs” may be examined
at various points to determine if the
expected/asserted status corresponds to the
actual status
47
 White-Box Testing - 2
• The test developer is privy to inner structure of
the application and knows the control flow
through the application, or at least knows the
control if the software works correctly
• It is useful for ensuring that all or at least most
paths through the applications have been
executed in the course of testing

48
 White-Box Testing - 3
• Using white-box testing methods, software
engineers can derive test cases that
– Guarantee that all independent paths within a
module have been exercised at least once
– Exercise all logical decisions on their true and false
sides
– Execute all loops at their boundaries and within
their operational bounds
– Exercise internal data structures to ensure their
validity
49
 White-Box Testing - 4
• The simplest form of WBT is statement
coverage testing through the use of various
debugging tools, or debuggers, which help us
in tracing through program executions
• By doing so, the tester can see if a specific
statement has been executed, and if the result
or behavior is expected

50
 White-Box Testing - 5
• One of the advantages is that once a problem
is detected, it is also located
• However, problems of omission or design
problems cannot be easily detected through
white-box testing, because only what is
present in the code is tested
• Another important point is that the tester needs
to be very familiar with the code under testing
to trace through it executions
51
 White-Box Testing - 6
• Therefore, typically white-box testing is
performed by the programmers themselves

52