Solving Medical Device Development Headaches

A closer look at Testing and Development Documentation

Neil Langmead, Technical Consultant, Emenda

Anders Emmerich, CEO, Aligned AG

March 4th 2014

1
Agenda – Part 1
08:30 – 09:00 Welcome and Coffee
09:00 – 09:45 FDA Software Development: two bottlenecks and current
challenges
Achieving FDA Guidelines on Software Architecture
Achieving FDA Guidelines on Unit and Integration Testing
New Guidelines from the FDA on Software Security

09:45-10:30 Technical Demonstration of Total Testing Platform

Multi dimensional Software Analysis, Advanced Defect Detection
FDA-tuned Reporting for Architecture, Bugs, Security
Analysing Software Architecture and Dependencies
Agenda – Part 2
10:45 – 11:30 Design History File Bad Practices
How Not to Manage your Design History File
Typical Examples from Real World Cases

11:30 – 12:30 Automated Design History File Management

Saving Time Through Automation and Control
Improving Quality with Continuous Consisency Analysis
Single Click Document Generation

12:15- 12:30 Buffet lunch, discussions, further demonstrations as required

Bottlenecks & Challenges
Testing and Documentation are both known to be problematic, difficult to automate,
challenging problems in FDA projects, and more generally in all safety critical software
projects.

This short seminar looks at possible solutions to the two above problems.

We will focus only on technical solutions to real world problems in this presentation
and demonstration
Why test? In Software generally…
Therac-25 Radiation
Therapy Machine
Death of 3 patients due
to massive overdose of
radiation

Toyota Prius
100K+ vehicle recalls

Ariane 5 Flight 501

Loss of $370M spacecraft
Why else?
>
Medical Device Software Testing Guidelines
Functional Safety Standards Overview

• DO-178b/c for aviation software

• IEC 61508 for industry and more…

• IEC 60880 for nuclear power

• EN 50128 / 50129 for railway applications (CENELEC)

• ISO 26262 for automotive software

• IEC 62304 / FDA for medical device software

What all standards have in common

• Define some rules that can be checked. For example:

– McCabe Cyclomatic Complexity < 10
– Max nesting level < 4
– No GOTO statements
– All global variables prefixed with glb_
– No dead code
– No unused variables
– etc

• Justify why these are sufficient

• Generate evidence that your developments meet these rules

What’s in the FDA Guidelines?
Also applies for Iterative/Agile

Testing Development

Deployment Design
 IEC 62304 - Software Testing Statements

“Appropriate consideration of software architecture (e.g., modular structure) during

design can reduce the magnitude of future validation efforts when software changes are
needed. “

“static analyses provide a very effective means to detect errors before execution of the
code. They allow for examination of each error in isolation and can also help in focusing
later dynamic testing of the software“

“Structural testing is accomplished primarily with unit (module) level testing, but can be
extended to other levels of software testing.”
MISRA standards in FDA projects
• Source Code Analysis (SCA) referenced as a technique in FDA Guidelines

• MISRA is a Coding standard, not functional safety standard

– MISRA C 2004
– MISRA C++ 2008
– MISRA C 3 / 2012

• Prevalent in the automotive sector but also now being adopted by many medical
companies as a standard (including two well known ones in Switzerland…)

• The MISRA approach is to write safer software from the beginning be using a very
restrictive subset of the language
– Example: MISRA C 2004: Rule 20.4 (required): Dynamic heap memory allocation shall not be used.
– C library dynamic memory – surely better the devil you know?
– What if we can check for correct usage?
What is Static Code Analysis?
Code Format Static Analysis Refactoring

Code Standards Defect Finding

Static Code
Analysis (SCA)

Code Metrics Security Checks

IP Compliance Architecture
At first there was Lint…

The term was derived

from the name of the
undesirable bits of
fibre and fluff found
in sheep's wool.

Lint, a C program checker. Stephen C. Johnson, Bell Laboratories,

December 1977.
Evolution of Source Code Analysis

1970s 2000s Today

Basic Whole Partial &

checking for program inter- incremental
suspicious & procedural whole program
non-portable control/data- analysis
constructs flow analysis ‘on-the-fly’
SCA Evolution: Structural code analysis
Structural Whole Smart
code program on-the-fly
analysis analysis analysis

• At first there was Lint…

– And then a few others

• What were they doing?

– Analysis of the compiler Abstract Syntax Tree (AST)
– On a translation-unit basis
– Looking for “suspicious and non-portable constructs” (likely to be bugs)

• Intended as a developer’s tool

– Intended to “just give better compiler errors to the developer”
– Quick
– Easy
– Noisy
Structural defects
• Simple coding mistakes…
• Think:
– Assignments in conditions
– Suspicious semi-colons
– Variables uninitialized on
declaration
• Would probably be picked up
with code reviews but quicker
and more certain with a static
code analysis platform
Inter-procedural control/data-flow defects

• Often not local to one particular module

• Not easy to spot with the human eye
– Not general found by code review
– Traditionally found with dynamic testing

• Real bugs, not just typing errors

– Buffer overflows (code injection)
– Memory leaks, integer taint (DoS)
– Uninitialized data usage (data injection)
– String taint (MitM)
– Platform/OS specifics (process injection, privilege escalation, …)
– Environment specifics (XSS/CSRF, SQL injection, …)
– Concurrency
SCA Evolution: Whole-Program Analysis Structural
code
Whole
program
Smart
on-the-fly
analysis analysis analysis

• Provided better core analysis capabilities

extended beyond syntactical and semantic
analysis to include:
– Sophisticated inter-procedural, control- and data-flow analysis
– Simulating potential runtime behaviour
• Can provide an automated view of every possible execution path, rather
than just those aspects of easily observed runtime behaviour
– New approaches for pruning false paths
– Estimating the values that variables will assume

• Understanding the entire build is key

• But moved away from being a developer tool

Anything else SCA can help with?
• More advanced SCA also weed out bugs, early!

• Finding bugs early means no time wasted with approvals, etc

• Unit/integration testing then focussed on finding design issues
FDA modelling and coding guidelines

To support the correctness of the design and implementation, the design and coding guidelines for the
modelling, or programming languages, shall address the topics listed below:

Topics ASIL
A B C D
1a Enforcement of low complexity ++ ++ ++ ++
1b Use of language subsets ++ ++ ++ ++
1c Enforcement of strong typing ++ ++ ++ ++
1d Use of defensive implementation techniques o + ++ ++
1e Use of established design principles + + + ++
1f Use of unambiguous graphical representation + ++ ++ ++
1g Use of style guides + ++ ++ ++
1h Use of naming conventions ++ ++ ++ ++
FDA Developer Testing Guidelines
U.S. FDA, China cFDA
General Principles of Software Validation;
Final Guidance for Industry and FDA Staff
Section 5.2.5
Testing by the Software Developer
• Unit Testing
• Integration Testing
• System Testing
 FDA Developer Testing Guidelines

“Testing entails running

“A software products testing
“Testing starts with unit test software products under
can be organized into unit,
and concludes with system known conditions with defined
integration, and system
testing” inputs and documented
testing”
outcomes”

“Regression analysis and

“The amount of structural testing are employed to
“Essential element of test is
coverage is commensurate provide assurance that a
the expected result”
with the level of risk posed” change has not created a
problem”
 Medical Device Software Defects

“An FDA analysis of 3,140 medical device recalls conducted between 1992 and 1998
reveals that 79% of software-related recalls were caused by software defects
introduced when changes were made to the software after its initial production and
distribution.

The primary cause of these critical failures: repeatability. A software development

process with a lack of repeatability is destined to failure.”
International Standard for Medical Device Software

IEC 62304
Medical Device Software –
Software Lifecycle Processes
First edition 2006-05

Section 5.5
Software Unit Implementation and Verification

Section 5.6
Software Integration and Integration Testing

Section 5.7
Software System Testing
 IEC 62304 - Software Safety Classifications

“The MANUFACTURER shall assign to each SOFTWARE SYSTEM a software safety class
(A, B, or C) according to the possible effects on the patient, operator, or other people
resulting from a HAZARD to which the SOFTWARE SYSTEM can contribute.””

The software safety classes shall initially be assigned based on severity as follows:

• Class A: No injury or damage to health is possible

• Class B: Non-SERIOUS INJURY is possible
• Class C: Death or SERIOUS INJURY is possible
 IEC 62304 Safety Classification Effects
 

Software
Documentation Class A Class B Class C
Software Must contain contents to sections 5.1 IEC 62304:2006. The plan's content list increases as the class increases, but a plan is required
development plan for all classes.

Software Software requirements specification conforming to 5.2 IEC 62304:2006. The content list for the software requirements specification
requirements increases as the class increases, but a document is required for all classes.
specification

Software Not required. Software architecture to 5.3 IEC 62304:2006. Refined to software unit level for Class C.
architecture

Software detailed Not required. Document detailed design for software

design units. (5.4).

Software unit All units are implemented, documented and source controlled (5.5.1).
implementation

Software unit Not required. Define process, tests and acceptance Define additional tests and acceptance
verification criteria (5.5.2, 5.5.3). criteria (5.5.2, 5.5.3, 5.5.4).
Carry out verification (5.5.5) Carry out verification (5.5.5).

Software Not required. Integration testing to 5.6 IEC 62304:2006.

integration and
integration
testing

Software system Not required. System testing to 5.7 IEC 62304:2006.

testing

Software release Document the version of the software List of remaining software anomalies, annotated with an explanation of the
product that is being released (5.8.4). impact on safety or effectiveness, including operator usage and human factors.

Summary of safety classification effects on the code development documentation and process
>
The Software Architecture Headache
Architecture Blueprint

Hard to determine which

dependencies are bad

Dependency Structure Matrix is

the most scalable way to
visualize dependencies &
specify architecture
Example of Software Architecture

“clearly structured…into several modules

and layers”
Software Architecture Erosion

Erosion results from dependencies which

violate the intended architecture,
crossing layer or component boundaries

These couplings prevent modularity

and lead to higher defect levels,
while also increasing the costs
to build, test, and maintain your system

“Spaghetti Code”  
What’s a DSM?

A Simple DSM Create Layers

DSM Partitioning Hierarchical DSM

Transform DSM

Worklist tracks changes during

transformation

Place the optional

modules in the
architecture
Establish Rules - Independence
 Establish Rules - Independence

Optional Modules
Violations
are are
independent
identified but
at every
can use
mandatory level
Modules
Establish Rules – Sublayering

Mandatory Modules should not use

Optional Modules
Establish Rules - API

API is visible, as are private

members
Fix Violations

Decouple driver from scheduler

 Measure, Report, Track, & Compare

Charts show
comparisons of
key metrics
across a
selected range
of Snapshots

Delta Reports compares the active Snapshot with any

previous one to see changes in metrics and new/missing
dependencies and violations
>
The Software Security Headache
Software Security
• Growing Concerns about security in medical devices
• The FDA is developing a cybersecurity laboratory in which a fuzz testing capability is to
be integrated
Software Security – our coverage
• OWASP Top 10 which was recently updated in June of 2013
– TTP supports seven (7) out of the 10. Of the remaining three, only one is statically
verifiable.
– This is an improvement over the previous version of the standard.
• CWE/SANS Top 25 Most Dangerous Software Errors
– We support twelve (12) of the Top 25. This is excellent since a large majority of the
standard is not statically verifiable
– Our nearest competitor only support eight (8).
– Major focus area for the next release.
• CWE – Common Weakness Enumeration
– Have excellent support for the entire list of CWE entries
• We also support
– CERT Secure Coding for C/C++ and Java
– The latest version of the DISA STIG “Application Security and Development” at
Version 3, Release 5.
>
The Emenda Total Testing Platform
Emenda – Total Testing Platform
Licence conflicts FOSS
e.g.src calls lib,
lib should not use src Architecture e.g. deadlocks, mutex,
Threads, semaphores,
Buffer overruns, Security Sleep issues
Denial of service
SQL injections Threads/Deadlocks
e.g. pointer dereference
e.g. class interfaces,
Function call chains
Interprocedural p->q;
where p = NULL

Crash Causing
e.g.
e.g. uninitialised
Variables
Logic Flow if (a = b) {
if (a != b) { }
}
Control Flow
e.g. if (a = b)
Syntactically ok, but
Meaning is not correct
Semantic Detected by most
Compilers and simple
Analysis, code
Syntactic Walthroughs, etc
Emenda – Total Testing Platform
DASHBOARD
Project Comparator, KPIs, Metrics, Architectural Dependency Structure
Project Health Scores, Drill-down to issues

System level tests,

Understand your Simulating DoS
e.g. class interfaces,
Code base through
Function call chains
Visualisation Attacks, robustness
Flowcharts, graphing testing
Fuzzing
Partitioning Algorithms
Management of
Open source components
FOSS Discovery
Cluster Analysis
Rules
Architecture
Unit Testing
Integration Testing
Code Coverage Dynamic Testing
Embedded Systems Defects
Security
Static Analysis Developer & System level
Emenda – Total Testing Platform
Static Analysis Testing
• Klocwork for C, C++, Java, C#. Advanced Defect
detection for improved security, reliability and
maintainability.
• Understand – Visualisation of your source code,
flow charts, heat maps, metrics…

Dynamic Testing
• VectorCast for C, C++ and Ada.
• Unit / Integration Dynamic Testing
• System Test / Code coverage
• Regression Test Management
• Requirements Traceability
• Target Test Execution

Architecture
• Lattix
• Solution to improve and enforce the
architecture of the application via DSM

47
Emenda – Software Testing Platform
Open Source Management
• Black Duck Software
• Licensing and compliance with the use of Open
Source code
• Security challenges associated with Open
Source Software (OSS)
Software Quality Management
• SQUORE KPI dashboard
• Valuing software capital
• Visualizing Technical-Debt
• Monitoring software quality and security

CI / Release Management
• Hudson / Jenkins
• Compilation, analysis, test… fully automated
• Klocwork, Understand, VectorCast plugins
• ElectricCloud EAccelerator and ECommader
• Reduce your build times by as much as 20x
• Ensure you release what you should release

48
 Our Solution- Total Testing Platform
Full Embedded Software Testing
> Unit and Integration testing True Embedded Test Automation
> Code coverage analysis through lifecycle
> > “Easiest to Use” test tool on the market
Automated regression testing
> > Easy continuous integration and test for repeatability
Testing on embedded targets and simulators
> > Integration with best of breed static analysis tools
Qualification materials for FAA, FDA, and others

Total Testing
World-Class Service and Support
> Our engineers are our support staff
> Local support via phone and email
SEMI-AUTOMATED
> Consulting and Technical Training TOOLS

Benefits of using Emenda TTP FREE AND OTHER

MANUAL TOOLS
> Reduced Time-To-Market
> Easier Certification / Qualification
NO
> Verifiable savings of time and money AUTOMATION
Dashboarding
Code analyzers and other tools Business Intelligence

Capitalization Dashboard

50
Architectural analysis
Defect density

Heatmap showing
areas of greatest
concern
52
Build analysis & acceleration

53
Release management automation

54
Characteristics of an Ideal Dynamic Analysis System

• Fully automates harness generation to support low-level

module testing

• Supports and fully automates integration with a wide array of

embedded targets

• Allows stand-alone code coverage to support testing legacy code

• Integrates with your existing requirements-tracking tool to

support requirements traceability
 Regression Testing leads to Better Quality!

“79% of all software errors in medical devices arise from the

addition of new functionality and regression testing issues.”
-- FDA

An ideal platform should…

• Provide a full command line interface to support test repeatability and integration with
existing testing infrastructure

• Encourages continuous integration and test repeatability

> Customer Success Stories
 Customer Characteristics
95%
95% of our customers are using VectorCAST to test embedded applications

50%
50% of our customers use live hardware and the rest use a target simulator

70%
70% of our new customers are using C++ and the rest are using C; with many long-time
customers still developing or maintaining systems in Ada

All of our customers want to be able to automate regression testing – not all of them are there yet
Emenda – Professional Services for medical projects

Software Verification and Validation

• Functional safety standards processes and compliance

• Implementation of unit and integration testing strategy and techniques

• Static code analysis, code review, inspections and defect triaging

• Coding standards/conventions enforcement

• Custom metrics and management reporting

• Automation of regression test suite

• Software licensing and IP audits

• Software design and architectural review

59
Emenda – Code Clinic
• Code Clinic workshops provided by Emenda
• Gives immediate, take-home information on the quality, security,
maintainability and/or legal implications of the code
• Emenda Total Testing Platform

60
Boston Scientific Corporation

Boston Scientific Corporation

Developer of the COGNIS® Cardiac
Resynchronization Therapy Defibrillator

Use Emenda TTP with MC/DC for the most

safety critical aspects of their application.
 Hoana Medical, Inc.

Hoana Medical, Inc.

Developer of the Lifebed™ Patient
Vigilance System

Use VectorCAST applications to test and

certify the software embedded on its
first FDA cleared product.
Sirona Dental Systems

Sirona Dental Systems

Developer of the GALILEOS 3D Digital Imaging
Systems

Use VectorCAST/C++ and Manage

As part of their IEC 62304 development process .
Terumo Heart, Inc.

Terumo Heart, Inc.

Developer of the DuraHeart™
Left Ventricular Assist System (LVAS)

Use VectorCAST applications to test and

certify embedded software running on
their patented left ventricle assist
implantable device.
 Medical Customer Characteristics

95%
95% of our customers are using Emenda TTP to test embedded applications

50%
50% of our customers use live hardware and the rest use a target simulator

70%
70% of our new customers are using C++ and the rest are using C; with many long-time
customers still developing or maintaining systems in Ada

All of our customers want to be able to automate regression testing – not all of them are there yet
 Connect With Us
Emenda Software Website
Emenda.eu

Automated Software Testing Blog User Driven Community

vectorcast.com/blog testresponsibly.com

Emenda Professional Network

linkedin.com

Emenda Fans
facebook.com/emenda

www.emenda.eu
News and Events
twitter.com/emenda
www.aligned.ch

Tutorials and Demonstrations

youtube.com/emenda
The End – Any Questions?

67