SOFTWARE QUALITY

ENGINEERING
Lecture # 1: Overview

Ayesha Kanwal
Department of Computing (DoC), SEECS, NUST
Course topics overview

Software quality and quality assurance

Testing concepts, issues, types and techniques

Test activities, management, and automation

Software inspection

Comparison of various QA techniques

Quantifiable quality improvement: defects and risk

identification
Quality processed and standards: ISO Standards, CMMI , TQM
Defect analyses

Defect removal models

Software reliability, failures and faults

Text book
 Tian, Jeff (2005). Software Quality
Engineering: Testing, Quality Assurance,
and Quantifiable Improvement | ISBN-10:
0471713457 | ISBN-13: 978-0471713456
| Edition: 1
Grading Policy

 30% mid term

 10% Assignments
 10% Quizzes
 10% Project
 40% ESE
Class Policy

 Assignments:
 Each assignment will count towards the
total.
 Late assignments will not be
accepted / graded.
 Quiz policy:
 Quizzes may be announced or
unannounced.
 No best of quizzes policy.
6 Why Software Quality Assurance?

Software errors cost the U.S.

economy $60 billion annually in rework,
lost productivity and actual damages. We
all know software bugs can be annoying,
but faulty software can also be
expensive, embarrassing, destructive and
deadly. Following are some of the famous
software “disasters”:
 1. Mariner Bugs Out (1962)
7

 Disaster: The Mariner 1 rocket with a space probe

headed for Venus diverted from its intended flight
path shortly after launch. Mission Control destroyed
the rocket 293 seconds after liftoff.

 Cost: $18.5 million

 Cause: A programmer incorrectly transcribed a
handwritten formula into computer code, missing a
single superscript bar. Without the smoothing
function indicated by the bar, the software treated
normal variations of velocity as if they were serious,
causing faulty corrections that sent the rocket off
course.
 2. CIA Gives the Soviets Gas (1982)
8

 Disaster: Control software went haywire and

produced intense pressure in the Trans-Siberian
gas pipeline, resulting in the largest man-made
non-nuclear explosion in Earth’s history.
 Cost: Millions of dollars, significant damage to
Soviet economy
 Cause: CIA operatives allegedly planted a bug in
a Canadian computer system purchased by the
Soviets to control their gas pipelines. The
purchase was part of a strategic Soviet plan to
steal or secretly obtain sensitive U.S. technology.
When the CIA discovered the purchase, they
sabotaged the software so that it would pass
 3. World War III… Almost
9
(1983)
 Disaster: The Soviet early warning system
falsely indicated the United States had launched
five ballistic missiles. Fortunately the Soviet duty
officer had a “funny feeling in my gut” and
reasoned if the U.S. was really attacking they
would launch more than five missiles, so he
reported the apparent attack as a false alarm.

 Cost: Nearly all of humanity

 Cause: A bug in the Soviet software failed to
filter out false missile detections caused by
 4. Mars Climate Crasher
10
(1998)
 Disaster: After a 286-day journey from
Earth, the Mars Climate Orbiter fired its
engines to push into orbit around Mars. The
engines fired, but the spacecraft fell too far
into the planet’s atmosphere, likely causing
it to crash on Mars.

 Cost: $125 million

 Cause: The software that controlled the
Orbiter thrusters used imperial units
(pounds of force), rather than metric units
INTRODUCTION TO
SQE
General Expectations
 Good software quality
 People: Consumers vs producers
 quality expectations by consumers
 to be satisfied by producers through software
quality engineering (SQE)
 Deliver software system that...
 does what it is supposed to do
 needs to be “validated”
 does the things correctly
 needs to be “verified”
Main tasks for software quality
engineering

 Organized into three major topics:

 Software testing, as a primary means to ensure
software quality
 Software quality assurance (SQA), that includes:
 Defect prevention
 Process improvement
 Inspection
 Formal verification
 Fault tolerance etc.
 Measurement and analysis to close the feedback
loop for quality assessment and quantifiable
improvement.
Testing, quality assurance (QA), and
quality engineering

Testing
Quality
Assurance

Software quality
engineering
What is software quality??

 Conformance to requirements
(Crosby)
 Problems:
 What if requirements are wrong?
 How do you know if requirements are
being met?
 Fitness For Use (Juran/Gruna)
 Problems:
 How many different ways are there for a
customer to ‘use’ a product?
 Customer’s view of Quality
 Perceived value of the product based on price,
performance, reliability, and satisfaction
Two perspectives
 “small q”
 Intrinsic product quality
 defect rate - how many bugs, or missing
functions
 What is considered a defect to the customer?
 reliability - how often it fails

 “big Q”
 Broader level of quality
 product quality
 process quality
 customer satisfaction
Two perspectives
 Will a good ‘q’ guarantee customer
satisfaction?

 Can you achieve a good “Q” without a

good “q”?
Two perspectives

 Will a good “q” guarantee customer

satisfaction?
 Issues
 Performance
 Requirements
 Service
 Documentation

 Can you achieve a good “Q” without a

good “q”?
 Bugs and poor reliability lead to poor
customer satisfaction
Views of quality

Transcendental view, quality is hard

Vie to define or describe in abstract
ws terms
User view, quality is fitness for
purpose
Manufacturing view, quality means
conformance to process standards
Product view, the focus is on
inherent characteristics in the
product
Value-based view, quality is the
customers’ willingness to pay for a
software
 ISO-9126 for Quality

 ISO-9 126 (ISO, 2001) provides a hierarchical

framework for quality definition, organized into quality
characteristics and sub-characteristics.
Functionality: Suitability - Accuracy -
Interoperability - Security

Reliability Maturity- Fault tolerance -

Recoverability

Usability: Understandability- Learnability

- Operability
Efficiency: Time behavior
- Resource behavior
Maintainability: Analyzability - Changeability
- Stability - Testability
Portability: Adaptability - Installability
- Conformance - Replaceability
Defects

 Defect definition – a problem with the

software, either with its external
behavior or internal characteristics

 What does Quality Assurance do?

 Focuses on ‘Correctness’ aspect of quality
 Focuses on defect prevention, detection,
removal and containment
 Focuses on testing activities
Correctness

 Many definitions:
 The degree to which a software entity's
behavior matches its specification
 The degree to which a system is free
from defects in its specification, design
and implementation.
 The ability of software products to
perform their exact tasks as defined by
their specification.
 Free from error, accurate, in
accordance with specifications.
Correctness and defects

High quality ≈ low defect

 intuitive notion related to correctness
 quality problem ≈ defect impact
 widely accepted, but need better
definitions
Error, fault, failure, and defect

 Failure: The inability of a system or

component to perform its required
functions within specified
performance requirements.
 Fault: An incorrect step, process, or
data definition in a computer
program.
 Error: A human action that produces
an incorrect result.
 Relation

Dormant faults
 Cost of removing defects
26

 The cost of removing defects increases

exponentially. A defect caught in
requirement and design phase costs less
to fix than an error caught in the
software maintenance cycle.

Cost of defect increases with each phase

How do we deal with
defects ?

 Defect Prevention
 Defect Detection and Removal
 Defect Containment
Defect Prevention

 Eliminating certain error

sources, such as eliminating
ambiguities or correcting human
misconceptions

 Fault prevention or blocking by

directly correcting or blocking these
missing or incorrect human actions
Defect Prevention through Formal
Verification
 Motivation
 Fault present: revealed through testing/inspection/etc.
 Fault absent: formally verify.

 Basic ideas
 Behavior formally specified:
 pre/post conditions, or
 as mathematical functions.
 Verify “correctness":
 intermediate states/steps,
 axioms and compositional rules.
 Approaches: axiomatic/functional/etc
Software Defect Prevention Opportunity Tree

IPT, JAD, WinWin,…

People PSP, Cleanroom, Dual
practices development,…
Defect
Prevention Manual execution, scenarios,...
Staffing for dependability
Standards
Rqts., Design, Code,…
Languages Interfaces, traceability,…
Prototyping
Checklists
Modeling & Simulation
Reuse
Root cause analysis
Defect detection and removal

 A method for identifying and removing defects

at every phase of the SDLC
 Checkpoints for reviews of the artifacts, i.e.,
software requirement specifications, design
document, test plan and test cases, software
code.
 Common techniques
 Inspection, faults are discovered
 Testing, failures trace back to faults
 Results of defect removal:
 Reduction in product expense
 Reduction in product development time
 Increase in development team’s productivity

 Software Defect Detection Opportunity
Tree
Completeness checking
Automated Consistency checking
Analysis - Views, interfaces,
behavior, pre/post
conditions
Defect Traceability checking
Detection Compliance checking
and Removal - Models, assertions,
standards
Peer reviews,
- Rqts.
- Design inspections
- Code Reviewing Architecture Review
Boards
Pair programming
Requirements & design
Structural

Testing Operational profile

Usage (alpha, beta)
Regression
Value/Risk - based
Test automation
Defect containment:

 Fault tolerance
 Fault present in the software but removal
is infeasible or impractical
 Fault tolerance - > contain defects
 Fault tolerance techniques:
 Recovery: rollback and redo
 Block the fault (N-version programming)
 Defect Impact Reduction Opportunity
Tree
Business case analysis

Value/Risk - Pareto (80-20) analysis

Based Defect V/R-based reviews
Reduction
V/R-based testing
Decrease
Defect Cost/schedule/quality

Impact, as independent variable

Size (Loss)
Fault tolerance
Self-stabilizing SW
Graceful
Degredation Manual Backup
Rapid recovery
Generic ways to deal with defects
 Thank you !!!