Lecture 5

Outline
 Introduction
 Software Life Cycle
 Quality Control
 Infrastructure
 Management
 Standards
 Conclusion & Summary
III.1 Foundations

Quality Control is the series of

inspections, reviews and tests used
throughout the development cycle to
ensure that each work product meets the
requirements placed upon it.
[Pressman2004]
 Verification & Validation
Verification: refers to the set of activities that
ensure that softwarecorrectly implements a specific function.
Validation: refers to a different set of activities that ensure that
the software that has been built is traceable to customer
requirements.
Boehm [Boehm81]:
Verification: “Are we building the product right?”
Validation: “Are we building the right product?”
The definition of V&V encompasses many of SQA activities,
including
formal technical reviews, quality and configuration audits
performance monitoring, different types of software testing
feasibility study and simulation
 Terminology: Dynamic/Static
Analysis
Dynamic analysis:
The process of evaluating a system or component based on
its behavior during execution.
Contrast with: static analysis.
See also: demonstration; testing.
[IEEE-610.12-1990]

Static analysis:
The process of evaluating a system or component based on
its form, structure, content, or documentation.
Contrast with: dynamic analysis.
See also: inspection; walk-through.
[IEEE-610.12-1990]
 Dynamic vs. Static Analysis
Dynamic analysis (testing):
execution of system components;
running the software
Static analysis:
investigation without operation;
pencil and paper reviews etc.
Modelling (mathematical representation)
 Terminology: Testing

Several definitions:
“Testing is the process of executing a program or
system with the intent of finding errors.”
[Myers1979]

Testing is the process used to verify or validate a

system or its components.
[Storey1996]
 Testing (according to IEEE)
(1) The process of operating a system or component under specified
conditions, observing or recording the results, and making an
evaluation of some aspect of the system or component.
(2) (IEEE Std 829-1983 [5]) The process of analyzing a software item
to detect the
differences between existing and required conditions (that is, bugs)
and to evaluate the features of the software items. See also:
acceptance testing; benchmark; checkout; component testing;
development testing; dynamic analysis; formal testing; functional
testing; informal testing; integration testing; interface testing;
loopback testing; mutation testing; operational testing; performance
testing; qualification testing; regression testing;stress testing;
structural testing; system testing; unit testing.
[IEEE-610.12-1990]
 Quality Control
Dynamic Activities
Life cycle phase Static &
Modelling
analysis
Process
analysis
Requirements X X
Top-level design X X
Detailed design X X
Implementation X X
Integration testing X X X
System validation X X
 III.2 Static Analysis Techniques
Overview
 Reviews and Inspections
Walkthroughs, inspections, personal
reviews
Formal technical reviews
Summary
Other Techniques
Control-flow analysis, data-flow analysis,
metrics, …
 II.2.1 Reviews and Inspections
A family of techniques
 Personal reviews
 Inspections
 Walkthroughs
 Formal technical reviews
Review / inspect
 To examine closely, With an eye toward
correction or appraisal
 People (peers) are the examiners
 Purpose/Objectives
 Verify that software meets its requirements software
follows predefined standards software is developed in
uniform manner
 Catching errors
Sooner
More and different
 Make projects more manageable
To identify new risks likely to affect the project
 Improving communication
Crossing organization boundaries
 Providing Education
 Making software visible
 (1) Personal Review
Features
 Informal
 Done by the producer
Implications
 Not objective
 Available to any developer
 Different mindset (A personal review often
reflects the developer's own thought process,
which may differ from that of external
reviewers, potentially limiting critical insights.)
 Need for review
 Conducting personal reviews helps identify issues

early, ensuring that the product is thoroughly

checked before completion.

 Product completion
 Personal reviews contribute to the overall

progress toward product completion by

addressing preliminary flaws efficiently.
 (2) Inspections
Inspection is More formal type of walkthrough
• Used to identify specific errors or problems
• assume details are correct, but look for missing items at a global level
• look for occurrences of a particular kind of bug such as buffer
overflow.
Features
 Team reviews materials separately
 Team and producers meet to discuss
 May review selected product aspects only
Implications
 Focus on important issues
If you know what they are
 More material per meeting
 Less preparation time
Process and Participants
 (3) Walkthroughs
Features
 Less formal
 Producer presents or provides information
Implications
 Larger groups can attend (education)
 More material per meeting
 Less preparation time

Disadvantage: Harder to separate

Product and presenter
 The walkthrough process may focus more on the presenter’s
performance than on critically evaluating the actual product.
Explanation and justification
 During the walkthrough, explanations of the product may
unintentionally turn into justifications for its shortcomings, making it
difficult to identify real issues.
Process and Participants
What to omit and what to include?
Inclusion:
Sections of complicated logic Critical sections,
where defects severely damage essential system
capability Sections dealing with new environments
Sections designed by new or inexperienced team
members
Omission:
“Straightforward” sections (no complications)
Sections of a type already reviewed by the team in
similar past projects Sections that, if faulty, are not
expected to effect functionality Reused design and
code Repeated parts of the design and code
 (4) Formal Technical Review
Features
 Formal
 Scheduled event
 Defined procedure
 Reported result
 Independent review team
 Producers not present
Implications
 More preparation time
 Less material per meeting
Product must stand or fall on its own
 Formal Technical Review (FTR)
 Process
Phases (Orientation) and procedures (Checklists)
 Roles
Author, Moderator, Reader, Reviewer, Recorder
 Objectives
Defect removal, requirements elicitation, etc.
 Measurements
Forms, consistent data collection, etc.
 FTR Process
 How much to review
 Review pacing
 When to review
 Pre-meeting preparation
 Meeting pace
 How Much to Review?
 Tied into meeting time (hours)
 Should be manageable
 Break into chunks if needed
 Review Pacing
 How long should the meeting last?
Based on:
 Lines per hour?
 Pages?
 Specific time frame?
 When to Review?
 How much work should be completed before the
review
 Set out review schedule with project planning
 Again, break into manageable chunks
 Prioritize based on impact of code module to overall
project
 Pre-Meeting Preparation
 Materials to be given to reviewers
 Time expectations prior to the meeting
 Understand the roles of participants
 Training for team members on their various
roles
 Expected end product
 Pre-Meeting Preparation (2/2)
How is document examination conducted?
 Ad-hoc
 Checklist
 Specific reading techniques (scenarios or
perspective-based reading)

Preparation is crucial to effective reviews

 FTR Team Roles

 Select the correct participants for each role

 Understand team review psychology
 Choose the correct team size
 FTR Team Roles (2/2)
 Author
 Moderator
 Reader
 Reviewer
 Recorder (optional?)

Who should not be involved and why?

 Team Participants
 Must be actively engaged
 Must understand the “bigger picture”
 Team Psychology
Stress
Conflict resolution
Perceived relationship to performance reviews
 Team Size
 What is the ideal size for a team?
 Less than 3?
 3-6?
 Greater than 6?
 What is the impact of large, complex projects?
 How to work with globally distributed teams?
 FTR Objectives
 Review meetings can take place at various stages of
the project lifecycle
 Understand the purpose of the review
 Requirements elicitation
 Defect removal
 Other
 Goal of the review is not to provide solutions
 Raise issues, don’t resolve them
 FTR Measurements
 Documentation
 Sample forms
 Inspection metrics
 Documentation
 Forms used to facilitate the process
 Documenting the meeting
 Use of standards
 How is documentation used by:
 Managers
 Developers
 Team members
 Sample Forms

 NASA Software Formal Inspections Guidebook

 Sample checklists
Architecture design
Detailed design
Code inspection
Functional design
Software requirements
 Beyond the FTR Process
 Impact of reviews on the programmer
 Post-meeting activities
 Review challenges
 Survey of reviews and comparisons
 Future of FTR
 Impact on the Programmer
 Should reviews be used as a measure of
performance during appraisal time?
 Can it help to improve commitment to their
work?
 Will it make them a better reviewer when
roles are reversed?
 Improve teamwork?
 Post-Meeting Activities
Defect correction
How to ensure that identified defects are
corrected?
What metrics or communication tools are
needed?
Follow-up
Feedback to team members
Additional phases of reviews
Data collection for historical purposes
Gauging review effectiveness
 Review Challenges
 Distributed, global teams
 Large teams
 Complex projects
 Virtual vs. face-to-face meetings
 Survey of Reviews
 Reviews were integrated into software development
with a range of goals
 Early defect detection
 Better team communication
 Review approaches vary widely
 Tending towards nonsystematic methods and
techniques

Source: Ciolkowski, M., Laitenberger, O., & Biffl, S. (2003). Software reviews,
the state of the practice. Software, IEEE, 20(6), 46-51.
 Survey of Reviews (2/2)
 What were common review goals?
 Quality improvement
 Project status evaluation
 Means to enforce standards
 Common obstacles
 Time pressures
 Cost
 Lack of training (most train by participation)
 Do We Really Need a Meeting?
 “Phantom Inspector” (Fagan)
The “synergism” among the review team
that can lead to the discovery of defects not
found by any of the participants working
individually
 Meetings are perceived as higher quality
 What about false positives and duplicates?
 A Study of Review Meetings
 The need for face-to-face meetings has never
been questioned
 Meetings are expensive!
 Simultaneous attendance of all participants
 Preparation
 Readiness of work product under review
 High quality moderation
 Team personalities
 Adds to project time and cost
(15-20% overhead)
A Study of Review Meetings (2/3)
 Studied the impact of:
 Real (face-to-face) vs. nominal (individual)
 groups
 Detection effectiveness (number of defects
detected)
 Detection cost
 Significant differences were expected
 A Study of Review Meetings (3/3)
 Results
 Defect detection effectiveness was not
significantly different for either group
 Cost was less for nominal than for real groups
(average time to find defects was higher)
 Nominal groups generated more issues, but
had higher false positives and more duplication
 Does Openness and Anonymity
Impact Meetings?
 “Working in a group helped me find errors
faster and better.”
 “Working in a group helped me understand
the code better.”
 “You spent less time arguing the issue
validity when working alone.”
 “I could get a lot more done in a given
amount of time when working by myself.”
 Study on Successful Industry Uses
 Lack of systematic execution during preparation
and detection
 60% don’t prepare at all, only 50% use
checklist, less than 10% use advance reading
techniques
 Reviews are not part of an overall improvement
program
 Only 23% try to optimize the review process
 Study on Successful Industry Uses
(2/3)
 Factors for sustained success
 Top-management support is required
 Need evidence (external, internal) to warrant
using reviews
 Process must be repeatable and measurable for
continuous improvement
 Techniques need to be easily adaptable to
changing needs
 Study on Successful Industry Uses
(3/3)
 Repeatable success tends to use well defined
techniques
 Reported success (NASA, Motorola, IBM)
 95% defect detection rates before testing
 50% overall cost reduction
 50% reduction in delivery time
 Future of FTR
1. Provide tighter integration between FTR and the
development method
2. Minimize meetings and maximize asynchronicity in
FTR
3. Shift the focus from defect removal to improved
developer quality
4. Build organizational knowledge bases on review
5. Outsource review and in-source review knowledge
6. Investigate computer-mediated review technology
7. Break the boundaries on review group size
Category/Attribute Management Technical Inspection Walkthrough
Objective Ensure Progress. Evaluate conformance to Detect and identify Detect defects.
Recommend corrective specifications and plans. defects. Examine alternatives.
action. Ensure proper Ensure change integrity. Verify resolution. Forum for learning.
allocation of resources.
Delegated Controls Management team charts Review team petitions Team chooses All decisions made by
Decision making course of action. management of technical predefined product producer. Change is the
Decisions leadership to act on dispositions. Defects prerogative of the
are made at the meeting recommendations. must be removed. producer.
or as a result of
recommendations.

Change verification Change verification left to Leader verifies as part of Moderator verifies rewo rk.Change verification left to
other project controls. review other project controls.
report.
Group Dynamics 2 or more people 3 or more people 3-6 people 2-7 people
Recommended size
Attendance Management, technical Technical leadership and College of peers meet Technical leadership and
leadership, and peer mix. peer mix. with documented peer mix.
attendance.
Leadership Usually the responsible Usually the lead engineer. Trained moderator. Usually producer.
manager.
Procedures Moderate to high, Moderate to high, Relatively low. Relatively low.
Material volume depending depending
on the specific on the specific
“statement “statement
of objectives” of objective”
at the for the
meeting. meeting.
Presenter Project representative Software element Presentation by “reader” Usually the producer
representative. other than producer.
Data collection As required by applicable Not a formal project Formally required. Not a formal project
policies, standards, or requirement. May be requirement. May be
plans. done locally. done locally.
Outputs Management review Technical review report. Defect list. Defect Walkthrough report.
Reports report. summary. Inspection
report.

Database entries Any schedule changes No formal database Defect counts, No formal database
must be entered into the required. characteristics, severity, required.
 Cost-Benefit Analysis
 Fagan reported that IBM inspections found
90% of all defects for a 9% reduction in
average project cost
 Johnson estimates that rework accounts for
44% of development cost
 Finding defects, finding defects early and
reducing rework can impact the overall cost of
a project
 Cost of Defects
 What is the impact of the annual cost of software
defects in the US?
$59 billion
 Estimated that $22 billion could be avoided by
introducing a best-practice defect detection
infrastructure
 Source: NIST, The Economic Impact of Inadequate
Infrastructure for Software Testing, May 2002
 Software Inspections
Why are software inspections not widely used?
 Lack of time
 Not seen as a priority
 Not seen as value added (measured by loc)
 Lack of understanding of formalized techniques
 Improper tools used to collect data
 Lack of training of participants
 Pits programmer against reviewers
 Other Techniques
Covered:  Static analysis:
Static Code Analysis
The process of
Formal Methods
evaluating a system
Model Checking
or component based
on its form,
Static analysis:
structure, content,
investigation without
operation; or documentation.
pencil and paper reviews [IEEE-610.12-1990]
etc.
Modelling (mathematical
representation)
 Static Analysis Techniques (1/2)
Static Code Analysis
 Control flow analysis
 data flow analysis
 symbolic execution
 Timing analysis
Performance/scheduling theories
Formal Methods
Discrete mathematics, logics, temporal logics, process
algebra, …
Model Checking
Fully automatic analysis of formal models w.r.t. a
formal specification
 III.3 Dynamic Testing Techniques
Common definition [Myers1979] :
Testing is a process of executing a program with the
intent of finding an error.
A good test case is one that has high probability of finding
an undiscovered error.
A successful test is one that uncovers an as-yet
undiscovered error.
Also: testing to assess reliability
(not considered here)
 Test:

(1) An activity in which a system or component is

executed under specified conditions, the results are
observed or recorded, and an evaluation is made of some
aspect of the system or component.
(2) To conduct an activity as in (1).
(3) (IEEE Std 829-1983 [5]) A set of one or more test
cases.
(4) (IEEE Std 829-1983 [5]) A set of one or more test
procedures.
(5) (IEEE Std 829-1983 [5]) A set of one
or more test cases and procedures.
[IEEE-610.12-1990]
 Objectives
 Design tests that systematically uncover defects
 Maximize bug count: uncover as many defects (or bugs) as possible
 Using the minimum cost and efforts, within the limits budgetary/scheduling
 Guide correction such that an acceptable level of quality is reached
 Help managers make ship / no-ship decisions
 Assess quality (depends on the nature of the product)
 Block premature product releases; Minimize technical support costs
 It is impossible to verify correctness of the product by testing, but you can
 prove that the product is not correct or you can demonstrate that you didn’t
 find any errors in a give time period.
 Minimize risks (especially safety-related lawsuit risk):
 Assess conformance to specification
 Conform to regulations
 Find safe scenarios for use (find ways to get it to work, in spite of the bugs)
 Indirect objectives
To compile a record of software defects for use in error prevention
(by corrective and preventive actions)
 Testing Fundamentals
Software Testing is a critical element of Software QualityAssurance
 It represents the ultimate review of the requirements specification,
the design, and the code.
 It is the most widely used method to insure software quality.
 Many organizations spend 40-50% of development time in testing.
Testing is the one step in software engineering process that could be
viewed as destructive rather than constructive.
“A successful test is one that breaks the software.” [McConnell 1993]
A successful test is one that uncovers an as yet undiscovered
defect.
Testing can not show the absence of defects, it can only show that
software defects are present.
For most software exhaustive testing is not possible.
 Test Cases & Set of Test Cases
“Adequate” Set of Test Cases?
Describe how to test a
system/module/function
Objective:
Description must identify to uncover as many defects as
 system state before possible
executing the test
 system/module/function Test set criteria:
does the set cover the system in
to be tested a complete/sufficient manner
 parameter values for the test
 expected outcome of the test Constraints:
with a minimum/appropriate of
Should enable automation! effort and time

So, how many? Depends on method!

Model of Testing (1/2)
Inputs causing
anomalous
behaviour
Input test data I

System

Outputs which reveal

the presence of
defects
Output test results
Oe
 Model of Testing (1/2)Unexpected
outcome
The Environment
environment model

The Program
program model Tests

Nature and Bug

psychology model Expected
outcome
 Testing Approaches
Black box testing
1. Testing that ignores the internal mechanism of the
system or component and focuses solely on the outputs in
response to selected inputs and execution conditions
2. Testing conducted to evaluate the compliance of a system
or component with specified functional requirements
(sometimes named functional testing)
White box testing
Testing that takes into account the internal mechanism of a
system or component(sometimes named structural testing)
IEEE
 III.3.2 Black Box Testing
Techniques
An approach to testing Black box testing:
where the program is (1) Function testing
considered as a “black- (2) Domain testing
box” (3) Specification-based testing
(4) Risk-based testing
requirements (5) Stress testing
(6) Regression testing
(7) User testing
input output (8) Scenario testing
(9) State-model based testing
events (10) High volume automated testing
(11) Exploratory testing

Not orthogonal!
 (1) Function Testing
variable in isolation. Test each function / feature /
Usually start with fairly simple function testing
focuses on a single function and tests it with
middle-of-the-road values don’t expect the
program to fail but if it fails the algorithm is
fundamentally wrong, the build is broken, or …
Later switch to a different style which often
involves the interaction of several functions
These tests are highly credible and easy to evaluate
but not particularly powerful
 (2) Domain Testing
Essence: sampling
 Equivalence class testing:
Look at each variable domains (including invalid values)
reduce a massive set of possible tests (domain) into a small group by
partitioning
picking one or two representatives from each subset.
look for "best representative“: a value that is at least as likely to
expose an error as any other
 Boundary value analysis: A good test set for a numeric variable hits
every boundary value because boundary / extreme-value errors are
common use the minimum, the maximum, a value barely below the
minimum, and a value barely above the maximum bugs are sometimes
dismissed, especially when you test extreme values of several variables at
the same time (corner cases)
 Equivalence Partitioning
 Input data and output
results often fall into
Valid inputs
different classes Invalid inputs

 Each of these classes is an

equivalence partition
where the program behaves System

in an “equivalent” way
 Test cases should be
chosen from each partition
Outputs
 Equivalence Partitioning
an approach that divides the input domain of a program
into classes of data from which test cases can be derived.
Example: Suppose a program computes the value of the
function. ( X − 1) ∗ ( X + 2) This function defines
the following
valid and invalid equivalence classes:
X < = -2 valid
-2 < X < 1 invalid
X >= 1 valid
Test cases would be selected from each equivalence
class.
 Boundary Value Analysis (1/2)
 a testing technique where test cases are
designed to test the boundary of an input
domain. “Bugs lurk in corners
 Boundary value analysis complements and congregate at
boundaries…”
and can be used in conjunction with Boris Beizer

equivalence partitioning.
Example (cont.): boundary values of the
three ranges have to be included in the test
data. That is, we might choose the following
test cases:
X <= -2 MIN, -100, -2.1, -2
-1.9, -1, 0, 0.9
-2 < X < 1 1, 1.1,100, MAX
X >= 1
 Boundary Value Analysis (2/2)
If an input condition specifies a range [a, b]: Example: [-3, 10],
test values: 0 (interior point); 0 is always good candidate!
test values: -3, 10 (extreme points)
test values: -2, 9 (boundary points)
test values: -4, 11 (off points)
If an input condition specifies a number values:
minimum and maximum
values just above and below minimum and
maximum
Also for more complex data (data structures):
array
input condition: empty, single element, full element, out-of-
boundary
search for element: element is inside array or not
You can think about other data structures: list, set, stack, queue,
and tree
 (3) Specification-Based Testing
Check the program against a reference document design specification,
a requirements list, a user interface description, a model, or a user
manual.
 Rational:
the specification might be part of a contract products must conform to
their advertisements safety-critical products must conform to any
safety-related specification.
 Observation
Specification-driven tests are often weak, not particularly powerful
representatives of the class of possible tests
 Different approaches:
focus on the specification: the set of test cases should include an
unambiguous and relevant test for each claim made in the spec. look
further, for problems in the specification: explore ambiguities in the
spec. or examine aspects of the product that were not well-specified
 (4) Risk-Based Testing
Process
(1) Imagine a way the program could fail (cf. error guessing)
(2) design one or more tests to check for it
a good risk-based test = a powerful representative that address
a given risk.
“complete”: a list of every way the program could fail.
If tests tie back to
significant failures in the field or
well known failures in a competitor’s product failure will be
highly credible and highly motivating
However, many risk-based tests are dismissed as academic
(unlikely to occur in real use).
Test for possible problem carry high information value
(We learn a lot whether the program passes the test or not)
 (5) Stress Testing
 Different definitions of stress testing:
Hit the program with a peak burst of activity and see it fail.
Testing conducted to evaluate a system or component at or beyond the
limits of its specified requirements with the goal of causing the system to
fail. [IEEE Standard 610.12-1990]
Driving the program to failure in order to watch how the program fails.
Keep increasing the size or rate of input until either the program finally fails
or
you become convinced that further increases won’t yield a failure.
Look at the failure and ask what vulnerabilities have been exposed and which
of them might be triggered under less extreme circumstances.
 Discussion:
stress test results as not representative of customer use
One problem with stress testing is that a failure may not be easily analyzed
(must includes enough diagnostic support)
stress-like tests can expose failures that are hard to see if the system is not
running with a high load (e.g., many concurrent tasks)
 (6) Regression Testing
Process
(1) design tests with the intent of regularly reusing them
(2) repeat the tests after making changes to the program.
A good regression test is designed for reuse.
adequately documented and maintainable
designed to be likely to fail if changes induce errors
in the addressed part of the program
Discussion:
First run of the tests may have been powerful,
but after passed
many times detecting defects is not very
likely (unless there have
been major changes or changes in part of the
code directly involved with this test.) Usually carries little
information value
Automation can make regression testing very “cheap”
 Regression Testing Process
 Requirements changes
Affect design, coding, and Software Change
Analysis
testing document update
 Design changes Software Change
Impact Analysis

Affect coding, tests, associated

components, system architecture, Define regression
testing strategy
related component interactions
 Implementation changes Build Regression
Test Suite
Affect test cases, test data, test scripts,
test specification (see also code change Run RegressionTests at
impact) different levels

 Test changes Report Retest

Affect other tests and test documentation Results

 Document changes
Affect other document
 Regression Testing Strategies
Random: The test cases are randomly selected from the
existing test suite.
Retest-all: Run all the tests in the existing suite.
Safe: The test selection algorithm excludes no test from the
original test suite that if executed would reveal faults in the
modified program.
Based on modifications: Place an emphasis on selecting
existing
test cases to cover modified program components and
those that
may be affected by the modifications.
Dataflow/coverage based: Select tests that exercise data
interactions that have been affected by modifications.

Remark ([Juriso+2004]):
For small programs and set of test cases is small, then retest-all.
Use safe techniques for large programs and programs with a
large
number of test cases.
 (7) User Testing
The testing done by real users (Not testers pretending to be users)
User tests might be designed by
the users,
the testers, or
other people (sometimes even by lawyers;
acceptance tests in a contract)
Continuum:
tests can be designed in such detail that the user merely executes them
reports whether the program passed or failed
a carefully scripted demonstration, without much opportunity to detect
defects
Leave the user some freedom to
cognitive activity while providing enough structure to ensure effective reports (in a
way that helps readers understand and troubleshoot the problem)
detect problems a user will encounter in real use is much more
difficult
Failures found in user testing are typically credible
Beta tests:
often described as cheap, effective
user tests
but in practice they can be quite
expensive to administer
you may not yield much information.
automation required!
 (8) Scenario Testing
Check how the program copes with a scenario
Good scenario test should be
credible,
motivating,
easy to evaluate, and
complex.
How often run a given scenario test?
Some create a pool of scenario tests as regression tests
Others run a scenario once or a small number of times and then design
another scenario rather than sticking with the ones they’ve used before
Scenarios help to develop insight into the product
Early in testing (understand the product at all) or
Late in testing (understand advanced uses of a stabilized product)
harsher test: rarely be used by typical users (cf. stress testing)
 (9) State-Model-Based Testing
Model the visible behavior of the program as a state machine
Checking for conformance with the state machine model
Approach is credible, motivating and easy to troubleshoot
Tradeoffs:
state-based testing often involves simplification and if the model is
oversimplified, failures exposed by the model can be false negatives
and
difficult to troubleshoot
more detailed models find more bugs, but can be much harder to
read and
maintain
Experiences shows that much of the bug-finding happens while
modeling the state models rather than testing
coverage criteria / stopping rule (cf. [Al-Ghafees&Whittaker2002])
 State-Model-Based Testing
 Uncovers the following issues:
Wrong number of states.
Wrong transition for a given state-input combination.
Wrong output for a given transition.
States or sets of states that have become dead.
States or sets of states that have become unreachable.
 Test sequence:
Define a set of covering input sequences that
starting from the initial state and get back to the initial state
For each step in the input sequence, defines the expected next state, the
expected transition, and the expected output code
 Remarks:
Completeness: transitions and outputs for every combination of input and
states
Instrument the transitions to also capture the sequence of states (not just the
outputs)
Count the states
 Variant: Syntax Testing
Test model: BNF-based syntax instead of state machine
E.g., for testing command-driven software
Dirty syntax testing:
a) to exercise a good sample of single syntactic errors in all
commands in an attempt to break the software.
b) to force every diagnostic message appropriate to that
command to be executed.
Example test cases: Syntax Graph for <real_number>:
.
correct ones <digital> <digital>
1.54683e5,
1.6548
incorrect without loops:
.05 E
1. e <digital>
1.1e
D +
incorrect with loops: d -
123456.78901.1
1.12345e6789D-123
 (10) High-Volume Automated Testing
Massive numbers of tests, comparing the results against a partial
oracles.
The simplest partial oracle: crashing
State-model-based testing (if the stopping rule is not based on a
coverage
criterion) or also more general notion of stochastic state-based
testing
[Whittaker1997].
Randomly corrupt input data
Repeatedly feeds random data to the application under test and a
reference
oracle in parallel [Kaner2000] (back-to-back testing)
Run arbitrarily long random sequence of regression tests to let
memory
leaks, stack corruption, wild pointers or other garbage that
cumulates over
time finally causes failures. Probes (tests built into the program)
can be
 (11) Exploratory Testing
 any testing which uses information gained while testing to better test
continuum between
purely scripted (the tester does precisely what the script
specifies and nothing else) to purely exploratory (no pre-specified
activities, not documentation beyond bug reports is required)
 prototypic case/ “freestyle exploratory testing”
exploratory testers continually learn about the software, the
context, the ways it could fail, its
weaknesses, and the best ways to test the software.
At the same time they are also test the software and report the
problems
Test cases are good to the extent that they advance the tester’s
knowledge (goal-driven)
as the tester gains new knowledge the goals may also change
quickly
which type of testing: what is most likely to reveal the
information the tester is looking for
 Not purely spontaneous:
studying competitive products,
review failure histories of this and
analogous products
interviewing programmers and users,
reading specifications, and
working with the product.
 state models can serve as an exploratory testing
tool
III.3.3 White Box Testing Techniques
Black box testing:
Requirements
fulfilled
Interfaces
available
and working
But can we also exploit
the internal
structure
of a
component,
interactions
between
objects?
white box
testing
White box testing:
 (1) Control Flow Testing: Example
procedure XYZ is 1
A,B,C: INTEGER;
begin 2
1. GET(A); GET(B); GET(C);
2. if (A > 15 and C > 20) then
3 9
3. if B < 10 then
4. B := A + 5;
5. else 4 6
6. B := A - 5;
7. end if
8. else 7
9. A := B + 5;
10. end if;
10
end XYZ;
 Control Flow Testing
a) Statement coverage: The test cases are
generated so that all the program statements are
executed at least once.
b) Decision coverage (branch coverage): The test
cases are generated so that the program
decisions take the value true or false.
c) Condition coverage: The test cases are
generated so that all the program conditions
(predicates) that form the logical expression of the
decision take the value true or false.
d) Path coverage: Test cases are generated to
execute all/some program paths.
 a) Statement Coverage
Statement 1

coverage: 2
The test cases are generated
so that all the program
statements are executed at 3 9
least once.
4 6

10
b) Decision / Branch Coverage
Decision 1

coverage 2
(branch
coverage): The 3 9
test cases are
generated so 4 6
that the program
decisions take
the value true or 7
false.
10
 c) Condition Coverage (1/2)
1
Conditioncoverage: The
test cases are generated 2
so that all the program
conditions (predicates) 3 9
that form the logical
expression of the
decision take the value 4 6

true or false.
7

10
Simple Condition Coverage
Composed condition:
((A and B) or C) A B C (A and B) or C
Test cases must cover all TRUE TRUE TRUE TRUE
the values true or false for TRUE TRUE FALSE TRUE
each conditions (predicates)
TRUE FALSE TRUE TRUE
of the logical expression
TRUE FALSE FALSE FALSE
combinatorial explosion
only coverage for simple FALSE TRUE TRUE TRUE
condition coverage FALSE TRUE FALSE FALSE
FALSE FALSE TRUE TRUE
Simple condition: (A) FALSE FALSE FALSE FALSE
degenerates to decision
coverage but simple Remarks:
condition coverage does not complete condition evaluation
ensure decision coverage assumed
Otherwise compiler dependent
execution semantics
 Other Condition Coverage Types
Simple condition coverage: require that the tests must
cover both true and false for each conditions
(predicates) of the logical expression.
Condition/decision coverage: require in addition that
the coverage w.r.t. the decision is also complete.
Minimal multiple condition coverage: also cover all
sub expressions of the Boolean formula
Modified condition/decision coverage: requires that
every atomic condition can influence the overall output
(required for level A equipment by RTCA DO-178 B).
Multiple condition coverage: test all combinations of
the atomic conditions.
 d) Path Coverage
1
Path coverage: Test cases are
generated to 2
execute all/some program paths.
Why path coverage?
logic errors and incorrect 3 9
assumptions are inversely
proportional to a path’s execution
4 6
probability we often believe that
a path is not likely to be
executed; in fact, reality is often 7
counter intuitive it’s likely that
untested paths will contain some
undetected defects 10
Exhaustive Testing Not Possible
There are 1014 possible
paths!
If we execute one test
per millisecond, it
would take 3.170 years
to test this program!!

Out of question
 Basic Path Testing
 Terminology:
execution path: a path that connects a start node and a
terminal node.
Two paths are independent if they have disjoint node or edge
sets
 Steps:
convert the unit into a “flow graph”
compute a measure of the unit's logical complexity
use the measure to derive a “basic” set of independent
execution paths for which test cases are determined
Remark: Path coverage set is not unique!
 Cyclomatic Complexity
Cyclomatic complexity (McCabe complexity) is a metric,
V(G), that
describes the logical complexity of a flow graph, G.
V(G) = E - N + 2 where E = #edges in G and N = #nodes in G
Studies have shown:
V(G) is directly related to the number of errors in source
code
V(G) = 10 is a
practical upper modules in this range
limit for testing are more error prone

modules

V(G)
 Path Coverage Testing
 Next, we derive the independent paths:
 Since V(G) = 4, we have 4 paths 1
 Path 1: 1,2,3,6,7,8
 Path 2: 1,2,3,5,7,8 2

 Path 3: 1,2,4,7,8 4 3
 Path 4: 1,2,4,7,2,4…7,8 5 6
 Finally, we derive test cases to
 exercise these paths, i.e. choose
 inputs that lead to traversing the
 paths 7

8
 Testing Loops
simple loop

nested
loops
concatenated unstructured
loops loops
 Loop Testing: Simple Loops
Minimum conditions - simple loops
1. skip the loop entirely
2. only one pass through the loop
(boundary)
3. two passes through the loop
(interior)
4. m passes through the loop m < n
5. (n-1), n, and (n+1) passes through the loop

where n is the maximum number of allowable

passes
 Testing Nested Loops
Just extending simple loop testing:
number of tests explodes
Reduce the number of tests:
start at the innermost loop; set all other
loops to minimum values
conduct simple loop test; add out of range
or excluded values
work outwards while keeping inner nested
loops to typical values
continue until all loops have been tested
 Comparison: Control Flow Testing
a) Statement coverage: The test cases are generated so
that all the program statements are executed at
least once.
b) Decision coverage (branch coverage): The test cases are
generated so that the program decisions take the
value true or false.
c) Condition coverage: The test cases are generated so
that all the program conditions (predicates) that
form the logical expression of the decision take the value
true or false.
d) Path coverage: Test cases
are generated to execute
all/some program paths.
a) Statement Simple condition
coverage coverage

b) Decision
Condition/decision
coverage
coverage

Minimal multiple
condition coverage
Basic
path
coverage
Modified
condition/decision
coverage

Path
coverage
Multiple condition
coverage
 (2) Data Flow Testing
Basic Definitions:
1
V = the set of variables.
N = the set of nodes
E = the set of edges 2 read(x,y) def (2) = {i}
i:=1; c-use(2) = {x,y}
Definition/write on variables (nodes):
p-use(3,4) = {i}
def(i) = { x in V | x has a global i<=2 3 i>2 p-use(3,5) = {i}
def in block I }
4 5
p-use(6,8) = {y}
Computational-use of variables (nodes): writeln(“hello”); p-use(6,7) = {y}
y>=0
y<0 6
c-use(i) = { x in V | x has a global i:= i+1;
8
c-use in block i } def (4) = {i} 7
c-use(4) = {i}
Predicate-use of variables (edges): 9 writeln(x);
c-use(8) = {x}
p-use(i,j) = { x in V | x has a p-use
in edge (i,j) } 10
 Idea of Data Flow Testing
Idea: coverage of the a def-clear 1

path between definition and

usage 2 read(x,y) def (2) = {i}
i:=1; c-use(2) = {x,y}
Examples:
p-use(3,4) = {i}
One path per def (all-defs) i<=2 3 i>2 p-use(3,5) = {i}
One path per def to all c-uses
(all-c-uses) 4 5
p-use(6,8) = {y}
One path per def to all p-uses writeln(“hello”); p-use(6,7) = {y} 6
y<0
y>=0
(all-p-uses) i:= i+1;
def (4) = {i} 7 8
One path per def to all p-uses or c-use(4) = {i}
c-uses (all-uses) 9 writeln(x
c-use(8) = {x}
All path (without loops) from defs
to all p-uses or c-uses (all du- 10

paths)
 Data Flow Testing
1
All node with an effective path from a def to a c-use
for variable x:
dcu(x, j) = { j in N | x in c-use(j) and read(x,y) def (2) = {i,x,y}
there is a def-clear path wrt 2
x from i to j } i:=1; c-use(2) = {}

All edges with an effective path from a def to a p-p-use(3,4) = {i} i>2 p-use(3,5) = {i}
use for variable x: i<=2 3

dpu(x, j) = {(j,k) in E | x in p-use(j,k) and

there is a def-clear path wrt x 4 5
from i to (j,k) } p-use(6,8) = {y}
writeln(“hello”); p-use(6,7) = {y} y>=0
All effective path from a def to use without loops: i:= i+1; y<0 6

du-c(x,i,j) = { i,…,k,j in N* | x in def(i), x in

c-use(j), the path is def-clear wrt x, def (4) = {i} 7 8
and no node is visit twice} c-use(4) = {i}
9
writeln(x);
du-p(x,i,j) = { i,…,k,j in N* c-use(8) = {x}
p-use(k,j), and the path i,…,k is
def-clear wrt x, and no node i,…,k 10
is visit twice}
du(x,i,j) = du-c(x,i,j) ∪ du-c(x,i,j)
 Types of Data Flow Testing
All-defs: Test cases are generated to cover each definition of each
variable for at least one use of the variable.
All c-uses: Test cases are generated so that there is at least one path
of each variable definition to each c-use of the variable.
All p-uses: Test cases are generated so that there is at least one path
of each variable definition to each p-use of the variable.
All-c-uses/some-p-uses: Test cases are generated so that there is at
least one path of each variable definition to each c-use of the
variable. If
there are variable definitions not covered, use p-uses.
All-p-uses/some-c-uses: Test cases are generated so that there is at
least one path of each variable definition to each p-use of the
variable. If
there are variable definitions not covered, use c-uses.
 Types of Data Flow Testing
All uses: Test cases are generated so that there is at least
one path of
each variable definition to each use (read: p-use and c-
use) of the
definition.
All-du paths: Test cases are generated for all the
possible paths of
each definition of each variable to each use of the
definition not taking
loops into account.
 Comparison: Data Flow Testing
If time is an issue: All-Paths
All p-uses should be used instead
of all-uses All-DU-Paths
All-uses should be used
All-Uses
instead of all-du-paths
to generate fewer tests: All-C-Uses/Some-P-Uses All-P-Uses/Some-C-Use

use the test cases

covered by the (stronger) All-C-Uses All-DEFS All-P-Uses
criteria.
decision
All-du-paths is usable in coverage
practice.
[Juriso+2004] statement
coverage
 (3) Mutation Testing
Strong (standard) mutation: Test cases are generated
to cover all the mutants generated by applying all the
mutation operators defined for the programming
language in question.
Selective (or constrained) mutation: Test cases are
generated to cover all of the mutants generated by
applying some of the mutation operators defined for the
programming language in questions.
Weak mutation: Test cases are generated to cover a
given percentage of mutants generated by applying all
the mutation operators defined for the programming
language in question.
 III.3.4 Comparison
Black box tests: White box tests:
tester has no access to tester can access
information about the information about the
system implementation system implementation

Characteristics: Characteristics:
Good for independence of Simplifies diagnosis of
tester results
But not good for formative Can compromise
tests independence
Hard to test individual How much do they need to
modules... know?
III.4 Testing Process & Activities
Beforehand:
Requirement analysis
design analysis
Testing Process & Activities
(1) Unit test
(2) Integration test
(3) Function test
(4) Performance test
System test
(5) Acceptance test
(6) Installation test
 Testing Activities
Unit Design System Other Customer User
test specifications functional software requirements environment
requirements requirements specification
Unitcode

Unit
test
Unitcode
Integration Function Performance Acceptance Installation
test test test test test

Integrated Functioning Verified, Accepted

modules system validated system
software
Unit
test SYSTEM
IN USE!
Unitcode
 III.4.1 Unit Testing
 Individual components are tested independently to ensure their
quality.
The focus is to uncover errors in design and implementation,
including
data structure in a component
program logic and program structure in a component
component interface
functions and operations of a component
There is some debate about what constitutes a “unit”. Some
common definitions:
the smallest chunk that can be compiled by itself
a stand alone procedure of function
something so small that it would be developed by a single person
 Unit testers: developers of the components.
 Example: Test GCD Algorithm
Testing a unit designed to compute the “greatest
common divisor” (GCS) of a pair of integers (not both
zero).
GCD(a,b) = c where
c is a positive integer
c is a common divisor of a and b (e.g., c divides a and c divides
b)
c is greater than all other common divisors of a and b.
For example
GCD(45, 27) = 9
GCD (7,13) = 1
GCD(-12, 15) = 3
GCD(13, 0) = 13
GCD(0, 0) undefined
 Test Planning
 How do we proceed to determine the tests cases?
1. Determine appropriate equivalence classes for the
2. input data.
3. Determine the boundaries of the equivalence classes.
4. Design an algorithm for the GCD function.
5. Analyze the algorithm using basic path analysis.
6. Then, choose tests cases that include the basic path
7. set, data form each equivalence class, and data at and
8. near the boundaries.
 GCD Algorithm
1. function gcd (int a, int b) {
2. int temp, value;
3. a := abs(a); 1
4. b := abs(b);
5. if (a = 0) then
6. value := b; // b is the GCD 5
else if (b = 0) then
8. raise exception; 7
9. else
10. loop 6
11. temp := b; 9
12. b := a mod b;
13. a := temp;
14. until (b = 0) 10
15. value := a;
16. end if; 17
17. return value;
18. end gcd
18
 Example: GCD Test Planning
Equivalence Classes
Although the the GCD algorithm should accept any
integers as input, one could consider 0, positive
integers and negative integers as “special” values.
This yields the following classes:
a < 0 and b < 0, a < 0 and b > 0, a > 0 and b < 0
a > 0 and b > 0, a = 0 and b < 0, a = 0 and b > 0
a > 0 and b = 0, a > 0 and b = 0, a = 0 and b = 0
Boundary Values
a = -231, -1, 0, 1, 231-1 and b = -231, -1, 0, 1, 231-1
Basic Path Set
V(G) = 4
(1,5,6,17,18), (1,5,7,18), (1,5,7,9,10,17,18),
(1,5,7,9,10,9,10,17,18)
Example GCD Test Plan
Test Description / Data Expected Results Test Experience / Actual Results
Basic Path Set

path (1,5,6,17,18) (0, 15) 15

path (1,5,7,18) (15, 0) 15

path (1,5,7,9,10,17,18) (30, 15) 15

path (1,5,7,9,10,9,10,17,18) (15, 30) 15

Equivalence Classes

a < 0 and b < 0 (-27, -45) 9

a < 0 and b > 0 (-72, 100) 4

a > 0 and b < 0 (121, -45) 1

a > 0 and b > 0 (420, 252) 28

a = 0 and b < 0 (0, -45) 45

a = 0 and b > 0 (0 , 45) 45

a > 0 and b = 0 (-27, 0) 27

a > 0 and b = 0 (27, 0) 27

a = 0 and b = 0 (0 , 0) exception raised

Boundary Points

(1 , 0) 1

(-1 , 0) 1

(0 , 1) 1

(0 , -1) 1
 Test Implementation
Once one has determined the testing strategy, and the
units to tested, and completed the unit test plans, the
next concern is how to carry on the tests.
If you are testing a single, simple unit that does not
interact with other units (like the GCD unit), then one can
write a program that runs the test cases in the test plan.

However, if you are testing a unit that must interact with

other units, then it can be difficult to test it in isolation.

The next slide defines some terms that are used in

implementing and running test plan.
 Test Implementation Terms
 Test Driver
a class or utility program that applies test cases to a component
being tested.
 Test Stub
a temporary, minimal implementation of a component to increase
controllability and observability in testing.
When testing a unit that references another unit, the unit must
either be complete (and tested) or stubs must be created that can
be used when executing a test case referencing the other unit.
 Test Harness
A system of test drivers, stubs and other tools to support test
execution
 Test Implementation Steps
 Here is a suggested sequence of steps for testing a unit:

1. Once the design for the unit is complete, carry out

Complete a test plan for a unit.
create stubs for not yet completed referenced other units
Create a driver (or set of drivers) for the unit
construction of test case data (from the test plan)
2. Once the implementation is complete, carry out
Use the driver (or set of drivers) to test the unit
execution of the unit, using the test case data
provision for the results of the test case execution to be printed
or logged as appropriate
 Testing Tools
There are a number of tools that have been
developed to support the testing of a unit or system.
googling “Software Testing Tools” will yield
thousands of results.
JUnit testing (http://www.junit.org/index.htm) is a
popular tool/technique that can be integrated into the
development process for a unit coded in Java.
 III.4.2 Integration Testing
A group of dependent components are tested together
to ensure their the quality of their integration unit.

The focus is to uncover errors in:

Design and construction of software architecture
Integrated functions or operations at sub-system level
Interfaces and interactions between them
Resource integration and/or environment integration

Integration testers:
either developers and/or test engineers.
 III.4.2 Integration Testing
Tests complete systems or subsystems composed of
integrated components
Integration testing should be black-box testing with tests
derived from the
specification
Main difficulty is localising errors
Incremental integration testing reduces this problem
 Integration Testing Strategies
 Incremental testing strategies:
 Bottom-up testing: Integrate individual components in levels
until the complete system is created
 Top-down testing: Start with high-level system and integrate
from the top-down replacing individual components by stubs
where appropriate
 Outside-in integration: Do bottom-up and top-down testing
at the same time such that the final integration step occurs in a
middle layer
 Non incremental testing strategies:
 Big bang testing: put all together and test it
 Remark: In practice, most integration involves a combination
of these strategies
 Big Bang Integration Testing
combine (or integrate) all parts at once.
Advantages:
simple
Disadvantages:
hard to debugging, not easy to isolate errors
not easy to validate test results
impossible to form an integrated system
 Bottom Up Integration Testing
Idea:Modules at the lowest levels are integrated at first,
then by moving upward through the control structure.
Integration process (five steps):
1. Low-level modules are combined into clusters that
perform a specific
software sub-function
2. A driver is written to coordinate test case input and
output
3. Test cluster is tested
4. Drivers are removed and clusters are combined
moving upward in the
program structure
 Bottom Up Integration Testing
Pros and cons of bottom-up integration:
no stubs cost
need test drivers
no controllable system until the last step
 Bottom Up Integration Testing
Stage 4 M11
Integration B Integration c

Stage 3 M9 M10
Integration A

Stage 2 M8

Stage 1 M1 M2 M3 M4 M5 M6 M7
 Top Down Integration Testing
Idea:
Modules are integrated by moving downward through
the control structure.
Modules subordinate to the main control module are
incorporated into the
system in either a depth-first or breadth-first manner.
Integration process (five steps):
1. the main control module is used as a test driver, and the
stubs are substituted
for all modules directly subordinate to the main control
module.
2. subordinate stubs are replaced one at a time with actual
modules
3. tests are conducted as each module is integrated
4. On completion of each set of tests, another stub is replaced
with the real
module
5. regression testing may conducted
Pros and cons top-down integration:
stub construction cost
major control function can be tested early.
 Top Down Integration Testing
Integration D
Integration C
Integration B
Integration A
Stage 1 M11

Stage 2 M9 M10

Stage 3 M8

Stage 4 M6 M7

Stage 5 M1 M2

Stage 6 M3 M4 M5
 Test Stubs and Test Drivers
Top-down testing of module M8 Bottom-up testing of module M8

Module
tested in an Drive
M9 earlier of M9
stage

M8 Module M8 Module
on test on test

Modules
Stub Stub tested in an
of M1 of M2 M1 M2 earlier
stage
 Regression Testing & Integration
A module firewall:
closure of all possible
affected modules and
in a program based on a
control-flow graph. Main

Reduction of the 4
software regression
testing to a smaller M1 M2 M3
scope (the firewall)
3
This implies that: Changed Module
re-test of the changed
M4 1 M5 M6 M7
module and its affected
modules 2

re-integration for all of A module firewall:

M8 - M5, M1, and Main
related integration links Re-testing at the unit level: M5
Re-integration steps: 2, 3, 4
 Comparison of the Approaches
Problems of non-incremental integration:
hard to debugging, hard to isolate errors not easy to validate test
results
impossible to form an integrated system
Pros and cons top-down integration:
stub construction cost
major control function is tested early
Pros and cons of bottom-up integration:
no stubs cost
need test drivers
no controllable system until the last step
Pros and cons outside-in integration:
stub construction cost
major control function is tested early
need test drivers
Architectural validation
Top-down/Outside-in integration testing is better at
discovering errors in the system architecture
System demonstration
Top-down/Outside-in integration testing allows a
limited demonstration at an early stage in the
development
Test implementation
Often easier with bottom-up integration testing
Test observation
Problems with all approaches. Extra code may be
required to observe tests
 III.4.3 Function Testing
The integrated software is tested based on requirements
to ensure that
we have a right product (validate functional
requirements).
The focus is to uncover errors in:
System input/output
System functions and information data External interfaces
System interfaces with external partsUser System
User interfaces interface
User (Operations &
System behavior Functions
& Behavior)
Function testers:
test engineers or SQA people.
 III.4.4 Performance Testing
Performance testing is designed to test Relevant attributes:
run-time performance of software within System process speed
the context of an integrated system (Max./Min./Average)
(validate non-functional requirements).
System throughput
The focus areas are: (Max./Min./Average)
Confirm and validate the specified system System latency
performance requirements. (Max./Min./Average)
Check the current product capacity to System utilization
answer the questions from customers and (Max./Min./Average)
marketing people. System availability
Identify performance issues and (component-level/system-level)
performance degradation in a given System reliability (component-
system level/system-level)
System behavior in the special load System scalability
conditions (component-level/system-level)
System successes/failures rates
Performance testers:
test engineers or SQA people.
 III.4.5 Acceptance Testing
The system software is tested as a whole. It verifies all
elements mesh properly to make sure that all system
functions and performance are achieved in the target
environment.
The focus areas are:
System functions and performance
System reliability and recoverability (recovery test)
System behavior in the special conditions (stress)
System user operations (acceptance test/alpha test)
Hardware and software integration and collaboration
Integration of external software and the system
System testers:
test engineers or SQA people.
 III.4.6 Installation Testing
The installation testing is designed to test the
installation procedure and its supported software.
The focus areas are:
Hardware and software integration and collaboration
Integration of external software and the system
Installation testers: test engineers or SQA people.
 III.5 Management & Automation
Management
The testing process
Scope: Unit, integration & system testing
Test methodology
Test planning
Test execution
Alpha and beta site testing programs
Scope: customer or user
Automation:
Cost effective testing
 Test Management
Test management encompass:
management of a test team
management of a test process
management of test projects
Management role:
Motivation: Motivate quality work and stimulate creative
solutions
Methodology: Test methodology, process, standards, and test
criteria
Mechanization/Selection/Development: test tools,
configuration management of test suites, integrated test
environment
Measurement: Test cost, complexity and efforts, engineer
performance, test effectiveness, product quality
 Roles in Testing
A test manager’s role: A test engineer’s role:
Responsibilities:
Play a leadership role in: Ensure that testing is
planning projects performed and documented as
motivating people defined by the employed
build a team and manage testing approach
engineers
Play as a controller in Basic skills:
product evaluation Technical awareness of testing
performance evaluation methods, tools, and criteria
Play as a supporter in: Organized; systematic
assist and train engineers planning
enforce and control test Inner determination to
methods, standards, and discover
criteria problems Pay good attention
select and develop test to the details
tools Ability to related to others and
resolve conflicts
 III.5.1 The Testing Process
Phases: Determining the test
(1) Determining the test methodology
methodology phase
Planning the tests
(2) Planning the tests
(3) Test design
Designing the tests
(4) Test implementation

Performing the tests

(implementation)
(1) Determining the Test Methodology
Determining the appropriate SQA standard
Depends on the application domain
The risk associated with the product

Determine the software testing strategy

Big Bang, Bottom-Up, Top-Down, …
Where to use white-box testing
Where can automated testing be applied?
 (2) Planning the Test
Scope:
Unit test, Integration test, system test

Must address:
a) What to test?
b) Which sources to be used for the test?
c) Who is performing the tests?
d) Where to perform the tests?
e) When to terminate the tests?
 a) What to test?
Must address:
Which modules should be unit tested?
Which integrations should be tested?
Determine priorities for allocating test resources
to the individual tasks or applications

Rating:
Damage severity level: estimate the severity of
results in case the module/application fails
Software risk level: probability of failure
 Damages to Customers and Users
1. Endangers the safety of human beings
2. Affects an essential organizational function with no system
replacement capability available
3. Affects functioning of firmware, causing malfunction of an
entire system
4. Affects an essential organizational function but a
replacement is available
5. Affects proper functioning of software packages for
business applications
6. Affects proper functioning of software packages for a
private customer
7. Affects functioning of a firmware application but without
affecting the entire system.
8. Inconveniences the user but does not prevent accomplishment of the
system’s capabilities
Damages to the Software Developer
1. Financial losses
Damages paid for physical injuries
Damages paid to organizations for
malfunctioning of software
Purchase cost reimbursed to customers
High maintenance expanses for repair of failed
systems
2. Non-quantitative damages
Expected to affect future sales
Substantially reduced current sales
Issues Affecting Software Risk Level
Module/application issues
1. Magnitude
2. Complexity and difficulty
3. Percentage of original software (vs. percentage of
reused software)
Programmer issues
4. Professional qualifications
5. Experience with the module's specific subject matter.
6. Availability of professional support (backup of
knowledgeable and experience).
7. Acquaintance with the programmer and the ability to
evaluate his/her capabilities.
 Priorities/Ratings for
Modules/Applications
A = Damage severity level
B = Software risk level
Combined Ratings
C=A+B
C=k*A+m*B
C=A*B (≈ Risk = Severity x Probability)
Employ Ratings to:
Assign testing priorities
Allocate resources for testing
Determine required techniques
 Example: Super Teacher
Combined rating method
Application Damage Software A +B 7xA+2xB AxB
Severity risk
Factor A Factor B
1. Input of test results 3 2 5 (4) 25 (5) 6 (4)
2. Interface for input and output of pupils’ data to 4 8 (1) 36 (1) 16 (1)
4
and from other teachers
3. Preparation of lists of low achievers 2 2 4 (5-6) 18 (7) 4 (5-6)
4. Printing letters to parents of low achievers 1 2 3 (7-8) 11 (8) 2 (8)
5. Preparation of reports for the school principal 3 3 6 (3) 27 (3) 9 (3)
6. Display of a pupil’s achievements profile 4 3 7 (2) 34 (2) 12 (2)

7. Printing of pupil’s term report card 3 1 3 (7-8) 23 (6) 3 (7)

8. Printing of pupil’s year-end report card 4 1 4 (5-6) 26 (4) 4 (5-6)

Super Teacher is a software package designed to support

teacher in managing the grades of elementary school pupils.
The package contains eight applications.
b) Which Sources to be used for the test?
Alternatives:
Random samples of real life cases
(Preferable – Stratified sampling of real life cases )
Taken from real application context
Refers to existing customers, shipment,
products, …
Synthetic test cases (simulated test cases)
Prepared by the test designer
Refers to combinations of system operating
conditions
and parameters (set of input data)
Designed to cover all known/expected operating
situations
 Comparison of Test Data Sources
Implication Random Synthetic
Effort Low, especially where expected results High; parameters must be
are available determined and expected
results calculated
Size Relatively high; repetitions are Relatively small; repetitions
frequent; can be avoided
in order to obtain a sufficient number
of
non-standard situations a relatively
large
number is required
Efforts to High efforts (low efficiency) Low effort (high efficiency);
perform tests
Effectiveness Relatively low; Relatively high; due to good
(probability of No coverage of erroneous cases coverage
error Some ability to identify unexpected Good coverage of
detection) errors erroneous situations
Little possibility to identify
unexpected errors
 c) Who is performing the tests?
Unit test:
developer or tester
Integration test:
developer or tester
System test:
independent tester
Large Systems: developer independent tester
Multiple test teams Understands the Must learn about the
system but, will system, but, will
Small organizations:
test “gently ” and, attempt to break it
Testing by another is driven by and, is driven by
development team “ delivery ” quality
Outsourcing of testing
 d) Where to perform the tests?
Unit test & Integration test:
Developer site
System test:
Developer Site or target site?
If external tester: consultant site
Simulate target site as good as possible
Still unpredicted failures of the installed system possible!
 e) When to terminate the tests?
Alternatives:

i. The completed implementation route

ii. The mathematical model application route
iii. The error seeding route
iv. The dual independent testing team route
v. Termination after resources have petered
vi. out
i. Completed Implementation Route
Approach:
Testing is terminated once:
The entire test plan has been carried out
Error free (“clean”) results have been
achieved for all regression tests
Discussion:
Fits to perfection approach
Disregards budget and timetable constraints
 ii. Mathematical Model Application
Route
Approach:
A mathematical model is used to estimate the
percentage of
undetected errors based on the rate of error detection
Define acceptable level of undetected errors
Determine an abort rate that corresponds to a
predefined
level of undetected errors which is considered
acceptable
Testing terminates when
The error detection rate declines below the abort
rate
 ii. Mathematical Model Application
Route
Discussion:
The mathematical model may not fully represent the
project
characteristics
Simple models provide no estimates about the severity
of the undetected errors
 iii. Error Seeding Route
Approach:
Errors of various types are seeded (hidden) in the tested
software prior to testing
Assumption: percentage of discovered seeded errors
corresponds to
the percentage of real errors
Define acceptable percentage of undetected errors
Testing terminates when
The residual percentage of undetected seeded
errors declines
below the acceptable percentage
 iii. Error Seeding Route
Discussion:
Additional workload for tester
Seeding distribution is totally based on past experience
while testers
might experience “new and original” errors in every
new project
only applicable for similar/familiar systems
 iv. Dual Independent Testing Team Route
Approach:
Two teams implement the testing
process independently
Define acceptable percentage or
number of undetected errors
Assumption: statistically independent
testing by both teams and random
detection of errors
Testing terminates when
The residual percentage/number
of
undetected errors is below the
acceptable percentage/number
Discussion:
Only applicable when there are two
independent test teams
Random detection of errors is a
strong assumption
Methodological basis may be
questioned when both teams share
similar testing experience or
methodology
 v. Termination after Resources have
petered out …
Approach:
Testing terminates when
budget or time allocated for testing run out
Discussion:
Undesirable but not uncommon
Systematic follow-up (for all cases):
Data collection about errors detected during development and regular
use (first
6 or 12 month)
Analysis of the error data: compare estimates supplied by the models
with the real figures
Comparative analysis of the severity of errors: Are the errors
detected during
regular use (first 6 or 12 month) more sever than that detected during
development?
 Software Test Plan (STP) - Template
1 Scope of the tests 3.3 Cross-reference to the relevant design
1.1 The software package to be tested document and the requirement documen
(name, version and revision) 3.4 Test class
3.5 Test level (unit, integration or system
1.2 The documents that provide the basis tests)
for the planned tests 3.6 Test case requirements
2 Testing environment 3.7 Special requirements (e.g.,
2.1 Sites measurements of response times,
security requirements)
2.2 Required hardware and firmware 3.8 Data to be recorded
configuration
2.3 Participating organizations 4 Test schedule (for each test or
2.4 Manpower requirements test group) including time
2.5 Preparation and training required of estimates for:
the test team 4.1 Preparation
4.2 Testing
3 Tests details (for each test) 4.3 Error correction
3.1 Test identification 4.4 Regression tests
3.2 Test objective
 (3) Test Design
Product of the test design stage:
Detailed design and procedures for each test
Test case database/file

Is carried out on the basis of the software test plan

(as documented by a STP)
The test case database/file may be documented by
a “software test procedure” and “test case file”
document
or in a single document called “software test
description”
 Software Test Description (STD) -
Template
1 Scope of the tests 3. Testing process
1.1 The software package to be tested 3.1 Instructions for input, detailing every
(name, version and revision) step of the input process
1.2 The documents providing the basis 3.2 Data to be recorded during the tests
for the designed tests (name and
version for each document) 4. Test cases (for each case)
2 Test environment (for each test) 4.1 Test case identification details
2.1 Test identification (the test details 4.2 Input data and system settings
are documented in the STP) 4.3 Expected intermediate results (if
2.2 Detailed description of the operating applicable)
system and hardware configuration 4.4 Expected results (numerical,
and the required switch settings for message, activation of equipment,
the tests etc.)
2.3 Instructions for software loading 5. Actions to be taken in case of
program failure/cessation
6. Procedures to be applied
according to the test results
summary
(4) Test Implementation
Phase activities:
Testing
Corrections of
detected errors
Re-testing
(regression
testing)
Is the quality of
the software
acceptable?
(Determine
termination)
 Test Execution
Test execution can be performed:
using manual approach
using a systematic approach
using a semi-automatic approach
Basis activities in test execution are:
Select a test case
Set up the pre-conditions for a test case
Set up test data
Run a test case following its procedure
Track the test operations and results
Monitor the post-conditions of a test case & expect the test
results
Verify the test results and report the problems if there is any
Record each test execution
Software Test Report (STR) - Template
1. Test identification, site, schedule and participation
1.1 The tested software identification (name, version and
revision)
1.2 The documents providing the basis for the tests (name and
version for each document)
1.3 Test site
1.4 Initiation and concluding times for each testing session
1.5 Test team members
1.6 Other participants
1.7 Hours invested in performing the tests
2. Test environment
2.1 Hardware and firmware configurations
2.2 Preparations and training prior to testing
 III.5.2 Alpha and Beta Site Tests
Programs
Alpha & Beta tests:
Test from the user perspective
Replace acceptance test
Should NOT replace testing by the developer/tester
Possible usually only for COTS
Alpha Site Tests:
Customer examine the program by applying to it the
specific
requirements of his organization
Is done at the developer site
Beta Site Tests:
Free offer to potential users
Is done at the “early” user sites (called “beta sites”)
Often hundreds or even thousands participants
Pro & Cons of Alpha and Beta Site Tests
Advantages
Identification of unexpected errors.
A wider population in search of errors.
Low costs.
Disadvantages
A lack of systematic testing.
Low quality error reports.
Difficult to reproduce the test environment.
Much effort is required to examine reports.
Remark:
Support for online crash reports
everybody becomes a “beta” tester
 III.5.3 Test Automation
 What is Software Test Automation?
Software test automation refers to the activities and
efforts that intend to automate engineering tasks and
operations in a software test process using well-defined
strategies and systematic solutions.
 Major objectives:
Reduce software testing time, cost, and efforts
Increase the quality of software testing
Apply well-defined test methods through tools
Relieve the complicated and redundant work from test
engineers
 Scope of Test Automation
Could be performed in three different scopes:
Enterprise-oriented test automation, where the major
focus of test automation efforts is to automate an enterprise-
oriented test process so that it could be used and reused to
support different product lines and projects in an
organization.
Product-oriented test automation, where test automation
activities are performed to focus on a specific software
product line to support its related testing activities.
Project-oriented test automation, where test automation
effort is aimed at a specific project and its test process.
 Main Types of Automated Testing
Code auditing
Module size, levels of nesting, levelsof subroutine nesting,
prohibited
constructs (GOTO), check naming conventions, check
programming
styles, check documentation styles, and other forms of
automated
static analysis: unreachable code, …
Test Management
Test plans, test results, and correction follow-up
E.g., Coverage monitoring: collect coverage information
such as
line coverage
Test execution
Functional testing: Automation regression testing; Output comparator
Load testing: Simulate load characteristics
Maintenance follow-up
 Automated and Manual Testing by Phase

Testing process phase Automat Manua

ed l
testing testin
g
Test planning M M
Test design M M
Preparing test cases M M
Performance of the tests A M
Preparing the test log and test A M
reports
Regression tests A M
Preparing the tests log and test M M
reports M = phase performed manually,
including comparative reports
A= phase performed automatically
Maturity Levels of Test Automation
Level 4: Optimal
A systematic solutions to manage test information, execute tests, and generate
tests, and measure test coverage.
Level 3: Automatic
Besides the test management and test execution tools, a
Level 4: Optimal
software test process at this level is supported with additional
solutions to generate software tests using systematic methods.
Systematic Test Level 2: Repeatable
Measurement &
Optimization Level 3:Automatic Also systematic solutions to execute software
Systematic
tests in a systematic manner.
Test Generation Level 2: Repeatable

Systamatic
Level 1: Initial Test Execution
Systematic solutions to Control
Level 1: Initial
create, update, and Systematic Test
manage all types of Information
software test information Management
Maturity Levels and
Level 4: Automatic test measurement
Tools test coverage and metrics
measurement
test cost and complexity measurement
Level 3: Automatic test generation
black-box/white-box test generation
Level 2: Automatic test execution
Level 4: Optimal test control and execution
Level 1: Automatic test management
Systematic Test test case management, and
Measurement &
Optimization Level 3:Automatic documentation
test script management
Systematic
Test Generation Level 2: Repeatable

Systematic
Test Execution
Control Level 1: Initial

Systematic Test
Information
Management
 Essential Needs & Basic Issues
Essential Needs: Basic Issues:
A dedicated work force for Poor manually performed
test automation software test process
The commitment from Late engagement of
senior managers and software test automation in
a software product life
engineers cycle
The dedicated budget and Unrealistic goals and
project schedule unreasonable expectations
A well-defined plan and Organization issues
strategy Lack of good
Talent engineers and cost- understanding and
effective testing tools experience of software test
automation
Maintenance of automated
software tests and tools
A Software Test Automation Process
Steps:
Test automation planning: identify test
automation focuses, objectives, strategies,
Plan Software requirements, schedule and budget.
Test Automation
Test automation design: draw out the
detailed test automation solutions to
Design Test Automation achieve the major objectives and meet the
Strategies & Solutions
given requirements in a test automation
Select and plan.
Develop & Implement Test
Automation Solutions
Evaluate
Available
Test tool development: realize the test
Software
Testing
automation solutions; the developed tools
Introduce and Deploy Test
Tools must be reliable, usable and reusable
Automation Solutions Test tool deployment: introduce and
deploy the tools; including basic user
Review and Evaluate
training and user support
Software Test Automation Review and evaluation : a review should
be conducted to identify limitations and
evaluate the provided features; valuable
feedback to the test automation group for
further improvements and enhancements.
 Test Tool Types
Test Tool Types Basic Descriptions of Different Types of Test Tools
Test Information Systematic solutions and tools support test engineers and
Management quality assurance people to
create, update, and maintain diverse test information,
including test cases, test scripts,
test data, test results, and discovered problems.
Test Execution and Systematic solutions and tools help engineer set up and run
Control tests, and collect and
validate test results.
Test Generation Systematic solutions and tools generate program tests in an
automatic way.
Test Coverage Systematic solutions and tools analyze the test coverage
Analysis during a test process based
on selected test criteria.
Performance Systematic solutions and tools support program
Testing and performance testing and performance
Measurement measurement.
Software Programs are developed to simulate the functions and
Simulators behaviors of external systems,
or dependent subsystems/components for a under test
program.
Regression Testing Test tools support the automation performance of
regression testing and activities,
including test recording and re-playing.
 Classification of Software Test Tools
There are three types of test information
management systems and tools.
Test information management tool: It supports engineers to create, update,
and
manage all types of test information, such as testing requirements, test cases,
data, procedures, and results.
Test suite management tool: It enables engineers to create, update, and
manage various software test scripts for test execution.
Problem management tool: It helps engineer
bookkeeping and manage the discovered problems during a test process.
There are three types of software simulators:
Model-driven program simulators: based on a specific model, such as a finite
state machine
Data-driven program simulators: functional program based on the given
inputs and return the pre-defined outputs.
Message-driven program simulators: communication process to generate
outgoing messages, based on the pre-defined incoming messages.
Regression test tools:
Software change analysis: This refers to a systematic facility that
identifies various types of software changes and discovers their ripple effects
and impacts.
Software test selection: This refers to a systematic facility that assists
engineers to select reusable tests for regression testing based on software
change information.
Test change analysis: This refers to a systematic solution that identifies the
necessary test changes based on the given software changes. The major task of
test change analysis is to find reusable tests and identify obsolete tests so that
new tests could be added, and existing tests will be updated.
Test recorder and re-player: This refers to a software tool that records the
executed tests, and re-plays them for re-testing. Mercury
Interactive’s Winrunner is a typical re-testing
tool that is able to record and re-play user-
system interactive sequences for window-
based programs using pre-defined test scripts.
 Classification of Software Test Tools
Types of Test Tools Test Tool Vendors Test Tools
Problem Management Tools Rational Inc. ClearQust, ClearDDTS
Microsoft Corp. PVCS Tracker
Imbus AG Imbus Fehlerdatenbank
Test Information Management Rautional Inc. TestManager
Tools Mercury Interactive TestDirectory
Evalid TestSuiter
Test Suite Management Tools Rational Inc. TestFactory
SUN JavaTest, JavaHarness
White-Box Test Tools McCabe & Associates McCabe IQ2
Junit
IBM IBM COBOL Unit Tester
IBM ATC
- Coverage Assistant
- Source Audit Assistant
- Distillation Assistant
- Unit Test Assistant
Classification of Software Test Tools
Test Execution Tools OC Systems Aprob
Softbridge ATF/TestWright
AutoTester AutoTester
Rational Inc. Visual Test
SQA Robot
Mercury Interactive WinRunner
Sterling Software Vision TestPro
Compuware QARun
Seque Software SilkTest
RSW Software Inc. e-Test
Cyrano Gmbh Cyrano Robot
Code Coverage Analysis Tools Case Consult Corp. Analyzer, Analyzer Java
OC Systems Aprob
IPL Software Product Cantata/Cantata++
Group
ATTOL Testware SA Coverage
Compuware NuMega TruCoverage
Software Research TestWorks Coverage
Rational Inc PureCoverage
SUN JavaScope
ParaSoft TCA
Software Automation Inc Panorama
Classification of Software Test Tools
Load Test and Performance Tools Rational Inc. Rational Suite PerformanceStudio
InterNetwork AG sma@rtTest

Compuware QA-Load
Mercury Interactive LoadRunner
RSW Software Inc. e- Load
SUN JavaLoad
Seque Software SilkPerformer
Client/Server Solutions, Inc. Benchmark Factory
Regression Testing Tools IBM Regression Testing Tool(ARTT)
Distillation Assistant
GUI Record/Replay Software Research eValid
Mercury Interactive Xrunner
Astra Astra QuickTest
AutoTester AutoTester, AutoTester One
 Automated Testing
Advantages
Reduce manual software testing operations and eliminate redundant
testing efforts.
Few manpower resources required for performing of tests.
Shorter duration of testing.
Execute much more software tests and achieve a better testing
coverage in a very limited schedule.
Accuracy and completeness of performance.
Accuracy of results log and summary reports.
Comprehensiveness of information.
Produce more systematic repeatable software tests, and generate more
consistent testing results.
Performance of complete regression tests.
Performance of test classes beyond the scope of manual testing.
 Automated Testing
Disadvantages
High investments required in package purchasing and
training.
High package development investment costs.
High manpower requirements for test preparation.
Considerable testing areas left uncovered.
 III.6 Paradigm & Domain Specific
Aspects
Paradigms: object-oriented, component-
based, agent-oriented, …
Testing of systems which apply object-
oriented technologies (OOA/OOD/OOP/…)
Applications domains: business, finance,
web, embedded systems, …
Testing embedded systems
III.6.1 Testing Object-Oriented Systems
Relatively little attention has been paid to considerations for testing OO
implementations
Lack of attention is not lack of need; New opportunities for error come with
the increased power of OO languages:
each lower level in an inheritance hierarchy is a new context for
inherited features
correct behavior at an upper level in no way guarantees behavior at a
lower level
polymorphism with dynamic binding dramatically increases the number
of execution paths static analysis of source code is of relatively little
help with these situations
encapsulation is an obstacle to controllability and operability of
implementation state
Components offered for reuse should be highly reliable:
each reuse is a new context of usage and re-testing is warranted
likely that more testing is needed to obtain high reliability in OO systems
 OO & Unit Testing
Unit of testing in OO context:
encapsulation drives definition of classes and objects:
each class and class instance (object), package
attributes (data),
and operations (methods or services) that manipulate
data
smallest testable unit is encapsulated class or object, rather
than the conventional module, because
a class can have a number of operations/methods and
particular
methods may exist in a number of different classes
class testing for OO software ≡ unit testing for
conventional
software
class testing driven by encapsulated operations and state
behavior of class
 OO Testing: Easier vs. Harder

Easier
Modularity Inheritance Harder
Quicker
Polymorphism
development
Dynamic
Encapsulation binding
Small methods
Complex
Reuse interfaces
Interfaces
identified More integration
early
 Class Testing & States
Functional
model of
processing

OO model of
processing

Problem: The state

often due information
hiding not directly
accessible, but can only
be accessed using public
class operations
 Testing & Inheritance
Problems:
Inheritance defines new context for methods
in the child class, the method may be redefined and
must
then be tested when the child is implemented
behavior of inherited methods can be changed
because
methods, that are called within methods, have to be
tested per class
Polymorphism
can result in an number of alternative paths as object
of
the parent class can be substituted by any parent
class
 Object-Oriented Testing Issues
The object class is the basic testing unit for
system developed with OO techniques.
This is especially appropriate if strong cohesion
and
loose coupling is applied effectively in the class
design.
An incremental testing approach is dictated by
The package/class design decomposition.
The incremental nature of the development
process.
Information hiding restricts/constrains testing to
using a white-box approach.
Encapsulation motivates testing based on the class
 Levels of Testing
1. Testing individual methods associated with objects:
black-box and white-box testing
2. Testing individual object classes:
black-box testing with “equivalence class” testing extended to
cover related operation sequences
structural (white-box) testing (requires a different type of analysis)
3. Testing clusters of objects :
strict top-down/bottom-up iteration inappropriate to create groups
or related groups explicit clustering
scenario-based testing appropriate
4. Testing the object-oriented system:
verification/validation against system requirements is carried out
as for any other type of system
 Class Testing (1/2)
Class test cases are created by examining the
specification of the class.
If you have a state model for the class, test
each transition - devise a driver that sets an
object of the class in the source state of the
transition and generates the transition event.
Class Constraints/Invariants should be
incorporated into the class test.
All class methods should be tested, but testing
a method in isolation from the rest of the class
is usually meaningless
 Class Testing (2/2)
To test a class, you may need instances
of other classes.
Interaction diagrams provide useful
guidance in constructing class test cases:
They provide more specific knowledge about
how objects of one class interact with
instances of other classes.
Messages sent to a class object provide a
guidance on which test cases are most critical for
that class.
 Method Testing (1/2)
A public method in a class is typically tested
using a black-box approach.
Start from the specification, no need to look at
the
method body.
Consider each parameter in the method
signature,
and identify its equivalence classes.
Incorporate pre-conditions and post-conditions
in
your test of a method.
Test exceptions.
For complex logic, also use white-box testing or
 Method Testing (2/2)
For private methods, either
modify the class (temporarily) so that it can
be tested externally:
change the access to public
or incorporate a test driver within the
class
or use static test methods, such as program
tracing or symbolic execution
 Incremental Testing
Incremental testing indicates that we test each
unit in isolation, and then integrate each unit,
one at a time, into the system, testing the
overall system as we go.
Classes that are dependent on each other
called class clusters, are good candidates for
an increment integration.
Candidate class clusters:
Classes in a package
Classes in a class hierarchy.
Classes associated with the interaction diagram
for a
use case.
A Class Test Order for OO Programs

3
4
4.1 ButtonList Sensor
AG 3.4
AS AS
AG 3.3
AS
AS
AS
4.2 Event Scene
Button

AS AS I 3.1
I 2
ButtonState AG
3 TextButton 3.2 World I
AS
AS
AS
I
1 I 1 2
1.1 5.2 I
Subject Control CanvasRep Canvas
AS
I
AS
AS AS AS
MonoScene

1.2 InstructorItr 5 ControlState 5.1

AG
 Integration Test Plan
An Integration Test checks and verifies that the
various components in a class cluster
communicate properly with each other.
With this plan, the emphasis is not on whether
system functionality is correctly implemented,
but rather do all the cluster classes “fit” together
properly.
A single test plan, for each cluster, which tests
the communication between all cluster
components is typically sufficient.
 Increment Test Planning
Determine the classes to be tested in an
increment.
Determine the appropriate “class clusters”.
Develop a unit test plan and test driver for each
class.
Develop an integration test plan and test driver
for each class cluster.
Develop a test script that details the
components to be tested and the order in which
the plans will be executed.
 III.6.1 Testing Embedded Systems
What is an embedded system?
What is the correct behavior?
Development Stages
Simulation
Prototyping
Pre-Production
Production
 What is an Embedded System?
Software
embedded into a
technical
system that
interacts with the
real physical
world, controlling
some specific
hardware.
Interaction with
 What is the Correct Behavior?
Continuous and discrete
signals in the
Time domain
Value domain
Exact match is not realistic
tolerance in
value and time
 Different Development Stages
(1) Simulation
(2) Prototyping
(3) Pre-Production
(4) Production
 (1) Simulation
One-way simulation (model test)
Test plant or embedded
software model

Feedback-simulation (model-in-the-loop)
Let the models of the plant and the
embedded software interact

Rapid Prototyping
Let the “real” plant (or a
simplified version) interact
with the model of the
embedded software (on
high performance hardware;
floating-point arithmetic)
 (2) Prototyping (1/3)
(1) Embedded software
Compiled for host
Compiled for target
platform
(2) Processor
High performance host
Target platform
emulator
(3) Rest of the embedded
system
Simulated on the host
Experimental hardware
platform
Prototype hardware
platform
(4) Plant
Statically (signal
generator)
Dynamically
(simulation model)
 Prototyping (2/3)
(1) Software unit and software
integration test
Same environment as MIL
Final software instead of model
a) Host environment
b) Target (emulator)
(2) Hardware/software
integration
test
Test hardware integration
Signal processors
Output monitoring with
oscilloscopes
or logic analyzers
In-circuit test equipment (not
restricted to outputs)
Real-time simulation of the plant
 Prototyping (3/3)
(3) System integration test
Integrate all hardware parts (often prototypes)
Similar to hardware integration (but complete!)
(4) Environmental test
Measuring influence of the system on the environment
(EMC)
Measuring roboustness of the system w.r.t.
environment
conditions (temperature, humidity, EMI, shock or
vibration, …)
 (3) Pre-Production
Test a pre-production unit Applicable tests:
Use pre-production unit Acceptance test
Execute with the real plant Qualification test
Safety execution test
Goals: Test of the production and
Prove that all requirements are maintenance
met test facilities
Demonstrate conformity with Inspection and/or test by government
officials
relevant
standards/norms Typical test techniques:
Demonstrate that production Real-life testing
effort and schedule can be met Random testing
Demonstrate maintainability Fault injection
and mean- time-to-repair Remark:
requirements Special tools for gathering and
Demonstrate product to recording
(potential) the data as well as fault injection are
customers required
Often environment test facilities
(climate
chambers, vibration tables, …) are also
used
 (4) Production
Production (or Post-development stage):
Development and test of the production
facilities (often equally important for quality)
Often a first article inspection of similar
style as the pre-production test is required
Production and maintenance tests on
production units may be required for quality
control
Comparison of the Test Levels
 III.7 Discussion & Summary
Besides dynamic analysis (testing), which includes the execution
of the software, also effective manual static analysis techniques
such as reviews or inspections are available.
Automatic or semi-automatic static analysis techniques are static
code analysis, formal methods, or model checking.
Two main approaches to testing (dynamic analysis) are black
box
testing, which ignores the internal mechanism of the system or
component (sometimes named functional testing), or white box
testing, which takes into account the internal mechanism of a
system or component (sometimes named structural testing).
The testing activities during development should include, unit
tests,
integration tests, and system tests. System tests may be divided
into
function tests, performance tests, acceptance tests, and
installation
tests.
 III.7 Discussion & Summary
Test management should result in a testing process which includes (1)
determining the test methodology, (2) planning the tests, (3) design the
individual tests, and (4) implement the designed tests.
The criteria when to terminate the tests is of critical importance.
Mathematical models, error seeding, or dual independent testing teams
are available approaches to guide termination.
Test automation can help to reduce software testing time, cost, and
efforts while increase the quality of software testing by applying well-
defined test methods through tools. Systematically employed can they
relieve some complicated and redundant work from test engineers.
The paradigm employed for the system development is relevant for
testability: e.g., object-oriented systems have advantages as well as
drawbacks w.r.t. testing
The application domain is relevant for the required testing: e.g.,
embedded systems and their software must be subject to a whole series
of test configurations