Software Testing
&
Security Testing
Types of Testing
• Unit (Module) testing
– testing of a single module in an isolated environment
• Integration testing
– testing parts of the system by combining the modules
• System testing
– testing of the system as a whole after the integration phase
• Acceptance testing
– testing the system as a whole to find out if it satisfies the
requirements specifications
Integration Testing
• Integration testing: Integrated collection of
modules tested as a group or partial system
– Tests complete systems or subsystems
composed of integrated components
• Integration plan specifies the order in which to
combine modules into partial systems
Integration testing
• Tests complete systems or subsystems
composed of integrated components
• Integration testing should be black-box
testing with tests derived from the
specification
• Main difficulty is localising errors
• Incremental integration testing reduces
this problem
Incremental integration testing
Approaches to integration
testing
• Top-down testing
– Start with high-level system and integrate
from the top-down replacing individual
components by stubs where appropriate
• Bottom-up testing
– Integrate individual components in levels until
the complete system is created
• In practice, most integration involves a
combination of these strategies
System Testing, Acceptance Testing
• System and Acceptance testing follows the integration phase
– testing the system as a whole
• Test cases can be constructed based on the the requirements
specifications
– main purpose is to assure that the system meets its requirements
• Manual testing
– Somebody uses the software on a bunch of scenarios and records
the results
– Use cases and use case scenarios in the requirements
specification would be very helpful here
– manual testing is sometimes unavoidable: usability testing
System Testing, Acceptance Testing
• Stress testing
– push system to extreme situations and see if it fails
– large number of data, high input rate, low input rate, etc.
– Stressing the system often causes defects to come to light
– Stressing the system test failure behaviour.. Systems should not
fail catastrophically. Stress testing checks for unacceptable loss of
service or data
– Particularly relevant to distributed systems which can exhibit
severe degradation as a network becomes overloaded
Regression testing
• You should preserve all the test cases for a program
• During the maintenance phase, when a change is made to the
program, the test cases that have been saved are used to do
regression testing
– figuring out if a change made to the program introduced any faults
• Regression testing is crucial during maintenance
– It is a good idea to automate regression testing so that all test
cases are run after each modification to the software
• When you find a bug in your program you should write a test case
that exhibits the bug
– Then using regression testing you can make sure that the old
bugs do not reappear
Mutation Testing
– To determining if a set of test data or test
cases is useful, by deliberately introducing
various bugs.
– Re-testing with the original test data/cases to
determine if the bugs are detected.
– Mutation analysis is used to figure out
the quality of a test set
Mutation Testing
• Mutation analysis creates mutants of a program by making changes
to the program (change a condition, change an assignment, etc.)
• Each mutant program and the original program are executed using
the test set
• If a mutant and the original program give different results for a test
case then the test set detected that the mutant is different from the
original program, hence the mutant is said to be dead
• If test set does not detect the difference between the original program
and some mutants, these mutants are said to be live
• We want the test set to kill as many mutants as possible
– Mutant programs can be equivalent to the original program, hence
no test set can kill them
Test Oracle
• a test oracle, or just oracle, is a mechanism for determining whether
a test has passed or failed
• a source of information about whether the output of a program (or
function or method) is correct or not.
• The oracle might be
– a program (separate from the system under test) which takes the
same input and produces the same output
– documentation that gives specific correct outputs for specific given
inputs
– a documented algorithm that a human could use to calculate
correct outputs for given inputs
– a human domain expert who can somehow look at the output and
tell whether it is correct
– or any other way of telling that output is correct
Test Automation
• If a manual test costs $X to run the first
time, it will cost $X to run every time
thereafter.
• An automated test can cost 3 to 30 times
$X the first time, but will cost about $0 after
that.
