Software Testing Background

• Reading:
– Ron Patton, Software Testing, 2nd ed., 2006.
• Chapter 1
– Daniel Michaels and Andy Pasztor, “Incidents
Prompt New Scrutiny Of Airplane Software
Glitches”, Wall Street Journal, Vol. 247, no.
125, May 30, 2006.
– Safety Critical Overview
 Outline
• Why Software Testing?
• What is a Bug?
• Why do Bugs Occur, Projects Fail?
• What is the Cost of Bugs?
• Software Testers
– What they do?
– Who are the Good Ones?
 Motivation (1)
• Why Software Testing?
– The Lion King Animated Storybook, Fall 1994
• Disney’s first multimedia CD-ROM game for children.
• Sales were huge.
• The game to buy for children that holiday season.
• On December 26, Disney's customer support phones began
to ring, and ring, and ring….from angry parents with crying
children who couldn't get the software to work. Numerous
stories appeared in newspapers and on TV news.
• The software development team have tried their software
on specific PC platforms. It failed on many very popular PC
platforms!
 Motivation (2)
• Why Software Testing?
– Intel Pentium Floating-Point Division Bug, 1994
• Enter the following equation into your PC's calculator:
(4195835 / 3145727) * 3145727 - 4195835
If the answer is zero, your computer is just fine. If you get anything
else, you have an old Intel Pentium CPU with a floating-point division
bug, a software bug burned into a computer chip and reproduced over
and over in the manufacturing process.
• On October 30, 1994, Dr. Thomas R. Nicely of Lynchburg (Virginia)
College traced an unexpected result from one of his experiments to an
incorrect answer by a division problem solved on his Pentium PC. He
posted his finding on the Internet and soon afterward a firestorm
erupted as numerous other people duplicated his problem and found
additional situations that resulted in wrong answers.
– Fortunately, these cases were rare and resulted in wrong answers only for
extremely math-intensive, scientific, and engineering calculations.
 Motivation (3)
• Why Software Testing?
– NASA Mars Polar Lander, 1999
• On December 3, 1999, NASA's Mars Polar Lander
disappeared during its landing attempt on the
Martian surface.
• A Failure Review Board investigated the failure and
determined that the most likely reason for the
malfunction was the unexpected setting of a single
data bit.
• Most alarming was why the problem wasn't caught
by internal tests.
 Motivation (3)
• Why Software Testing?
– Malaysia Airlines jetliner, August 2005
• As a Malaysia Airlines jetliner cruised from Perth, Australia, to Kuala
Lumpur, Malaysia, it suddenly took on a mind of its own and zoomed
3,000 feet upward.
• The captain disconnected the autopilot and pointed the Boeing 777’s
nose down to avoid stalling, but was jerked into a steep dive. He
throttled back sharply on both engines, trying to slow the plane.
Instead, the jet raced into another climb.
• The crew eventually regained control and manually flew their 177
passengers safely back to Australia.
• Investigators quickly discovered the reason for the plane’s roller-
coaster ride 38,000 feet above the Indian Ocean. A defective software
program had provided incorrect data about the aircraft’s speed and
acceleration, confusing flight computers. The computers had also
failed, at first, to respond to the pilot’s commands.
 Motivation (4)
• Why Software Testing?
– http://www.aonix.com/safety_critical_overview.html
• A passenger airplane is circling in a prearranged location off
the coast of Florida. The landing is delayed because of bad
weather conditions. As the plane is banking into a turn, a
sudden updraft causes the plane to roll much faster than the
software control system expects. The software "assumes" a
glitch, and the computers are set into an automatic reboot
process. The pilot looks on with horror as all of the navigation
displays turn blue with a white line through them. At a most
crucial moment, when the pilot needs information to stabilize
the aircraft, the computers are performing memory checks and
restarting the display software.
 What is a Bug?
• Informally, it is “what happens when software
fails”, whether the failure was
– Inconvenient
– Catastrophic
• Terms for software failure
* Fault * Anomaly* Problem * Inconsistency
* Failure * Incident * Error * Feature
* Defect * Variance * Bug
 What is a Bug?
• Product Specification: “an agreement among the software
development team. It defines the product they are
creating, detailing what it will be, how it will act, what it
will do, and what it won't do”
• Formally, we say that a s/w bug occurs when one or more
of the following five rules is true: when the software
– doesn't do something that the product specification says it
should do.
– does something that the product specification says it shouldn't
do.
– does something that the product specification doesn't mention.
– doesn't do something that the product specification doesn't
mention but should.
– is difficult to understand, hard to use, slow, or in the software
tester's eyes will be viewed by the end user as just plain not
right.
Why do Bugs Occur?
 Symptoms and Root Causes of Bugs
• Inaccurate understanding of end user needs
• Inability to deal with changing requirements
• Modules that don’t fit together
• Software that is hard to maintain or extend
• Late discovery of serious project flows
• Poor software quality
• Unacceptable software performance
• Team members in each other’s way, making it
impossible to reconstruct who changed what, when, were
and why
• An untrustworthy build-and-release process
 Root Causes of Project Failures
• Ad hoc requirement management
• Ambiguous and imprecise communication
• Brittle architectures
• Overwhelming complexity
• Undetected inconsistencies in requirement, design,
and implementation
• Insufficient testing
• Subjective assessment of project status
• Failure to attack risk
• Uncontrolled change propagation
• Insufficient automation
 The Cost of Bugs

Remember Disney’s Lion King CD!

What does a Software Tester Do?
• The goal of a software tester is to
– find bugs
– find bugs and find them as early as possible
– find bugs, find them as early as possible, and
make sure they get fixed
What Makes a Good Software Tester?
• They are explorers.
• They are troubleshooters. They love puzzles.
• They are relentless. Software testers keep trying.
• They are creative.
• They are (mellowed) perfectionists.
• They exercise good judgment.
• They are tactful and diplomatic.
• They are persuasive.

Software Testing is Fun

What goes into a SW product?

“Software Testing”
by Ron Patton
What parts make up a SW
product?

“Software Testing”
by Ron Patton
 Optimal test effort

“Software Testing” by Ron Patton

 Precision vs accuracy
• Testing a simulation game like a flight
simulator..
– Should you test precision or accuracy?
• How about a calculator?
• Depends on what the product is and what
the development team is aiming at !!
Neither accurate nor precise

“Software Testing” by Ron Patton

 Precise, but not accurate

“Software Testing” by Ron Patton

 Accurate, but not precise

“Software Testing” by Ron Patton

 Accurate and precise

“Software Testing” by Ron Patton

 Try this
• Start the Windows Calculator program
• Type 5,000-5 (comma is important)
• Look at result
• Is it a bug or not??
 Quiz
• Q: Given that its impossible to test a program completely,
what information do you think should be considered when
declaring whether its time to stop testing?

• A: There is no correct answer when to stop testing. Each

project is different. Things to consider would be, Are lots
of bugs being found?, Is the team satisfied with the
number and types of tests that have been run?, Has the
product been validated against the users requirements?
 Quiz
• Q: If you were testing a flight
simulator, what would be more
important, accuracy or
precision?

• A: Simulator should look and

feel like flying a real plane.
What’s more important is how
 Quiz
• Q: Is it possible to have a high-
quality and low reliability
product? What might an
example be?

• A: Yes, but it depends on the

customers expectations for
 Quiz
• Q: Why is it impossible to test a
program completely?

• A: There are too many inputs,

too many outputs, and too many
path combinations to fully test.
Also, software specs can be