Mitigating Software-Related

Business Risk Requires Systems

Perspective
April 2017

Prepared by:
Peter Kaminski, Senior Consultant, Cutter Consortium

Cutter Consortium
37 Broadway, Arlington, MA 02474-5552
Tel: +1 781 648 8700; Fax: +1 781 648 8707
Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

Table of Contents
Executive Summary .................................................................................................................... 2

Introduction ................................................................................................................................. 3

Software Risk Equals Business Risk .......................................................................................... 3

Organizational Structure and Software Risk............................................................................ 3

Risk: The Flip Side of Quality .................................................................................................... 4

Run, Build, Transform ................................................................................................................ 6

The Software Risk Management Tool Landscape ................................................................... 9

Business Benefits of Managing Software Risk ....................................................................... 12

About Peter Kaminski ............................................................................................................... 13

Cutter Consortium • Document prepared for CISQ 1

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

Executive Summary
Software-related business risk is a serious matter; US companies lose $26.5 billion in
revenue due to downtime each year, and there are many other deleterious effects in
terms of security, reliability, performance efficiency, and maintainability of software
systems.

Organizationally, software risk management can fall between the cracks.

Several business departments must be involved, but product management or
the product owner should assume ultimate responsibility for it.

The Consortium for IT Software Quality (CISQ) has published industry frameworks
for measuring software quality. CISQ is an IT industry leadership group that develops
international standards for measurement of software size and source code structural
quality, jointly organized by the Object Management Group (OMG) and the Software
Engineering Institute (SEI) at Carnegie Mellon University. The frameworks can be
adapted for structuring software risk management.

Risk can be measured and mitigated at two complementary levels: the component level
and the overall system level. There are powerful static code analysis tools available
for both levels. Choice of analysis type depends on where the system is within its
development and operation lifecycle of the software portfolio.

Systemizing software risk management offloads automatable work of software

architects and engineers so that they can focus instead on knowledge work and
innovation for business benefit. Doing so allows the organization to improve
development velocity, reduce the chance of outages or security breaches, and
compete for current and future business more effectively.

Cutter Consortium • Document prepared for CISQ 2

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

Introduction
Driving business risk down is just smart business. Software-related business risk is
an increasing portion of business risk, so knowing how to assiduously reduce software
risk has become part and parcel of today’s business reality. Fortunately, there is an
array of tools and methods that you can apply across your portfolio of software assets
and development projects to manage software risk. Industrializing software risk
management is critical for organizations in the digital age. It unleashes the “smarts”
in developers so that they can work on the difficult parts of building and delivering
applications for the future, while ensuring current, past, and future risk is baked out of
applications, putting both human intelligence and software intelligence to their best use.

Software Risk Equals Business Risk

Software systems provide a great many benefits to business, but like any kind of system,
they are susceptible to problems. There can be direct failures (e.g., outages or intrusions)
or indirect failures (e.g., missed deadlines or requirements/features mismatches).
Problems with software systems have direct business impact on customer satisfaction,
revenue, stock price, and employee retention. Indeed, it’s become a major problem for
US companies who are losing $26.5 billion in revenue due to downtime each year.

Companies that develop software for themselves or others have an affirmative

responsibility to minimize the business risk associated with the delivery, operation, and
maintenance of its software systems. As software systems encompass larger and larger
portions of business processes and continue to grow in complexity, reducing risk
inherent in software becomes not only more challenging, but critically important.

Organizational Structure and Software Risk

Many IT organizations are structured in a way that works against effective management
of software risk. Who is organizationally responsible for managing software risk? Who
should be responsible?

The chief information security officer (CISO) is responsible for overall security, but that
person may not be directly involved with the day-to-day engineering of application
development. Furthermore, the CISO does not have responsibility for the other kinds
of software risk; namely, reliability, performance efficiency, and maintainability.

Architects are responsible for setting up the architectural standards that should
determine how developers reduce risk over the course of developing applications.
But they are often disconnected from development while the application is built.

Cutter Consortium • Document prepared for CISQ 3

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

Consequently, development work might unintentionally drift from the intended

architecture unbeknownst to the architecture team.

QA engineers are nominally in charge of product quality and, therefore, in reducing

software risk, but they lack insight into how the software is engineered, resulting in an
inability to test edge cases and other blind spots. Often, they don’t even have time to test
all the functionality — which is their primary focus.

Thus, responsibility for mitigating software risk is by default left to developers. This
makes sense for risks that exist within the purview of the developer; but the growing
size and complexity of applications, new architectural frameworks, and pace of delivery
all contribute to the developer’s diminishing view into modern day software systems.
This reduced visibility, or lack of context into the entire system, becomes an insur-
mountable blind spot for most organizations, especially for something so critical to
the business — the safety and soundness of core applications.

Our recommendation is that product management or the product owner be ultimately

responsible for managing and mitigating product risk, leveraging contextual software
analysis and the engineering teams to install and run it.

Risk: The Flip Side of Quality

In this report, we aim to provide an overview of software risk — how to reason about it
and then how to manage it.

We first must acknowledge that IT leaders who are interested in building resilient, fast,
responsive, safe, and secure systems generally talk in terms of “software quality” rather
than “software risk.” That is, they describe a positive outcome, rather than the avoidance
of a negative outcome.

For those managing the business outcomes of deploying software systems, it is

appropriate to have a balanced view of positive (software quality) and negative
(software risk) outcomes in order to deploy typical business risk management analyses
in an optimal manner.

Since software risk and software quality are two sides of the same coin, we can leverage
frameworks used to classify and promote software quality to organize our examination
of software risk.

For this report, let’s use the most broadly accepted framework for measuring software
quality, the Automated Quality Characteristic Measures standard, published by CISQ.
The standard lays out two broad dimensions to organize software quality measurement
— or, for us, measurement of software risk (see Figure 1).

Cutter Consortium • Document prepared for CISQ 4

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

Figure 1: Two Dimensions of Measurement.

Origin of Risk: Unit-Level vs. System-Level

One dimension of software risk encompasses the domain where the problem originates.
Unit-level risks are limited to one component, a single programming language, and often
a single programmer’s work, whereas system-level risks are expansive, involving large
parts of the whole system, multiple programming languages, as well as architectural
layers, and they cross all technical stakeholders building or maintaining the system
(see Figure 2).

Struts EJB
JSP Java
Oracle
PL/SQL
Mobile Web Spring
Services
ASP.NET

Unit-level problems occur Office

COBOL
Java Hibernate

.NET
COBOL

T/SQL
DB2

SQL
Server
Sybase

in a single component
C#
APIs Messaging PL/SQL IMS
C++
NodeJS

End User Business Frameworks Data Data

User Interface Logic Logic Storage

System-level problems occur

Struts EJB
JSP Java
Oracle
PL/SQL
Mobile Web Spring
Services
ASP.NET
Java Hibernate Sybase

because of interactions
Office COBOL DB2

COBOL T/SQL
SQL
.NET
Server
C#
APIs

between components
Messaging PL/SQL IMS
C++
NodeJS

End User Business Frameworks Data Data

User Interface Logic Logic Storage

Figure 2: Unit-Level vs. System-Level Problems.

At the unit level, problems can be introduced or fixed by developers as work progresses
on a software component and its functional requirements (i.e., what that component
is supposed to do). However, this remediation is performed without concerns for the
rest of the system. This is not intentional, but since developers tend to lack visibility
or an understanding of the entire system, their decisions can only be scoped to the
individual component.

System-level problems are caused or fixed at an architectural level. They result from
the way the entire system’s components interact with each other — and with users,
databases, and external components — within the context of the software architect’s
designs and the product manager’s nonfunctional requirements (i.e., how the system is
supposed to “be”). Due to the ever-increasing size and complexity of today’s systems,

Cutter Consortium • Document prepared for CISQ 5

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

it is extremely difficult for developers, testers, or architects to see system problems

without the help of software analysis tools.

Types of Software Risk

The second dimension in the standard is a typology, which classifies problems into four
primary categories.

1. Security problems encompass flaws that expose privacy, data, and privileges.
Obviously, these can have a huge business impact. The Common Weakness
Enumeration is a standardized list of well-known types of problems, currently
tracking 1,005 types of security faults.

2. Reliability problems include system downtime, data corruption, and the ability
of operators to bring the system back from an outage.

3. Faults in performance efficiency overtax computing resources, causing slow

response times and application scalability; these problems can impact workforce
productivity and end-user satisfaction.

4. Lastly, maintainability risk occurs when the software is poorly constructed,

with the result being slow fixes to problems, an inability to respond to changing
market conditions, and high IT costs when fixes or changes need to be made.

Run, Build, Transform

In this section, let’s use a different lens with which you might use to examine your
software risk.

Your company likely has a portfolio of existing systems, systems currently under
construction as well as future plans, including requirements to move in new directions
to keep up or surpass competitors. Software risk and its mitigation work differently for
each project lifecycle bucket, so it’s worth examining how we might analyze risks and
opportunities for each type.

Run: What’s Been Built

For legacy and current systems, much of the care you’ll need to have about software risk
will be at the system level. Here, the project is entirely integrated and deployed, and
unit-level risk has pretty much been baked out.

Security will need to continue to be tested for the deployment life of the project. This
includes regular penetration testing and review of security bulletins against the known

Cutter Consortium • Document prepared for CISQ 6

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

architecture of the system since new attacks may develop against the old architecture.
“Security through antiquity,” the idea that older systems might be less vulnerable to
security breaches, has been shown to be false. For instance, a recent study of systems in
US federal agencies showed that federal agencies with more legacy systems are likely to
experience more security breaches than those with newer systems.

Reliability and performance efficiency must be monitored; however, if they continue

a nominal trend, they do not need direct attention unless they are under continuous
enhancement or modernization. Some systems, even in this lifecycle mode, must be
upgraded occasionally due to regulation changes or compliance issues. Enhancement
work such as this can be like the work of a newly built system, although more
complicated and difficult due to system age, lack of visibility into the codebase,
and lack of local expertise.

Maintainability continues to be important for the life of the project, perhaps even past
deployment, if you wish to maintain the asset. Security patches, upgrades to libraries
and operating systems, and continued employee knowledge of at least what’s in the
system and how to maintain it are important.

Lastly, a documentation of the scope and scale of the project is necessary, so you know
what you own and need to maintain. Scope includes knowing what comprises the system
as well as what other systems and processes it touches. Scale — a measure of size and
complexity of the system — includes information about what languages the system
includes, the number of components, the number of lines of code, which operating
systems and libraries are used, and overall system documentation.

If you already have accurate application blueprints for your applications, they may
contain all the information you need. If not, then you might need to use a tool that can
scan your applications, giving you detailed architectural blueprints that accurately
describe the system as it exists. If need be, you can use those blueprints as a basis for
more comprehensive discovery and description.

Build: What You’re Building Now

Projects that your company is currently working on present an interesting opportunity
as well as a challenge. Certainly, you have the most leverage over how the project is put
together at this time, but there are also many project management components to juggle
— budget, schedule, scope.

While a project is being designed, deployed, and built, all aspects of software risk
management come into play. Product management provides system requirements and
the voice of the customer. Architecture and security set up the skeleton and guardrails
for the system, describing the general engineering framework underpinning the project.
Development follows development best practices and runs continuous unit and
integration tests, while integration and deployment uses contextual software analysis

Cutter Consortium • Document prepared for CISQ 7

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

to survey the entire system as it comes together. Project management and project
sponsors work with the whole team to keep their eye on the big picture.

System-level analysis comes to the fore, allowing the project managers, architects, and
security specialists to continuously monitor the current and future health of the system.
Integrating system-level analysis into the DevOps pipeline provides the team with the
software intelligence needed to increase development speed and reduce rework, while
ensuring they drive down software risk in the final system.

To manage security risks, architectural and engineering best practices need to be

followed by developers. Application owners, architects, and security specialists need a
mechanism to verify that these best practices and standards are in fact being followed.
System-level analysis helps create this visibility, ensuring compliance as well as
providing real-time learning and feedback to developers and operations teams.

Integration engineers, QA, operations, architectural and security specialists can employ
system-level analysis to pinpoint security hotspots or weaknesses for intensive review
prior to production release.

Reliability and performance efficiency needs require the team to follow good practice
at the unit level, but actual results are theoretical until the system starts to be integrated
and can be stress-tested and scanned with system-level analysis by quality engineers.
Output from those tests can be fed back to team leads and architects to improve the
design as needed before final integration.

Maintainability is significantly less expensive and more effective if technical debt is

reduced while the code is fresh and the original teams are still working on it. Both unit-
level and system-level analysis should be used to assiduously improve readability of
the code, create documentation of methodologies and architecture, and reduce code
duplication.

Transform: What You’ll Need to Build Next

Because business realities, markets, and technology change so quickly, it’s not enough to
manage yesterday’s and today’s risk; rather, it’s imperative to run your company so that
it can meet and thrive during tomorrow’s challenges.

The writing is certainly on the wall. New technologies and areas of growth include the
Internet of Things (IoT), blockchain, and cognitive computing. Whether you’re ready or
not, these components of technology will be mainstream within a few years, along with
big, new markets and competitive market and time pressure. You’ll need to move into
these areas while managing your business risk.

Along with a framework for software risk management, you’ll also need to deploy
significant human intelligence to understand new technologies and to grow into them

Cutter Consortium • Document prepared for CISQ 8

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

gracefully. You will need system-level understanding of how your old technologies
and new technologies work together. IT leaders are taking the opportunity now to
industrialize software development and deployment, so that they can scale and
automate testing and deployment of existing software assets to free up resources for
innovation in the realm of upcoming technologies.

The Software Risk Management

Tool Landscape
Now that we have examined what software risk is and how to map it, let’s review the
tools and processes that can reduce the incidence and severity of risk, especially during
active project development when you have the most leverage. We will cover each tool in
a short summary below, although each topic could be detailed out in a report of its own.

Product Management
Product management organizes and prioritizes functional and nonfunctional
requirements so the development team can focus on delivering business value with
its work. Significant business risk can be avoided through careful planning of what to
develop and, just as crucially, what not to develop. Careful requirements analysis by
product management reduces the risk that users will have trouble using the software,
and thoughtful specification of nonfunctional requirements — including security,
privacy, reliability, performance efficiency, and maintainability — sets the project up
for both short-term and long-term success.

Development Processes
Development methodologies like Agile or waterfall provide cadences and ceremonies
that reduce the incidence of software risk elements and provide feedback processes that
move information about problems back to the developers. When the processes work in
a timely manner, developers can create a better system and thus continuously reduce
software risk as they build and integrate smaller pieces to create a larger system.

Of special note in these times of increasingly rapid change in software technologies:

your engineering team needs to have processes in place that unlock innovation and
keep developers up to date with the latest tools and technologies. Continuing education
(informal or formal) and some allocation of team time to experimentation (out of direct
line of production) will be increasingly important to keep your company competitive in
both technology and hiring and retention.

Cutter Consortium • Document prepared for CISQ 9

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

Development Tools
Developer tools such as source code management (SCM) and integrated development
environments (IDEs) have a long history, but modern versions have been turbocharged
with both features and speed, which increases developer productivity and reduces the
risk of introducing errors. They are also integrated with the DevOps toolchain, which
ensures delivery and deployment are in the scope of feedback to developers, ensuring
that the application works correctly in deployment, not just on the developer’s machine.

SCM keeps track of each revision in hundreds or thousands of files, allowing developers
to make changes with the certainty that they can roll back the changes, compare the
changes if there are problems, and work together with a team of other developers who
are making separate changes to the same files.

IDEs bundle together file management, editing, compiling, and testing tools. They offer
a dashboard-like way of combining all the tasks a developer needs to do in one place
as well as integration of software platforms like FindBugs or SonarQube, open source
tools that provide for continuous inspection of code quality. Software risk at the unit
level is made obvious to the developer as it’s instantiated, allowing the developer to
handle the problems right there and then.

Local Code Analysis

Syntactic software analysis performs static code analysis (performed without executing
the code) locally, or at the component level, with the goal of looking for duplicated code,
overly complex code, poor readability, insufficient test coverage, or violations of coding
standards. Most programming editors and compilers have these types of tools built into
them. Syntactic software analyzers can be used in many contexts: within the IDE, for
embedded software development, within one language or multiple languages, or with
more or less awareness of the target computing platform.

There are many free and open source tools available for local static code analysis; for a
detailed list, see the Wikipedia’s List of tools for static code analysis. We will specifically
mention two of the most commonly used tools.

FindBugs is an example of an open source local code analyzer. It scans for possible
defects in Java code and classifies potential errors as scariest, scary, troubling, and of
concern to help developers understand their possible severity.

SonarQube is an example of a commercialized open source tool suite that performs

syntactic software analysis. Similar to the IDE tools and plugins, SonarQube’s
capabilities are provided by a large library of commercial plugins. Moreover, it has
multiple language analyzers supporting each major programming language — C/C++,
JavaScript, C#, Java, Cobol, PL/SQL, PHP, ABAP, etc. Multiple language support means

Cutter Consortium • Document prepared for CISQ 10

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

that developers can use it universally, protecting more projects and amortizing the cost
of learning how to use it across more projects.

For all its power, syntactic code analysis is limited to locally analyzing components one
at a time, either as it runs over the changes a developer just made in an IDE or as it
moves across the project. It can see problems that are self-contained within a component,
but it doesn’t have purview over system-level problems.

It’s also unaware of extra context around problems it detects within a component; in
some cases, extra system-level context would show that it has found a false positive.
Without that context, false positives can generate noisy analysis, which the developer
or testing team must examine more carefully.

As Ayewah et al. concluded, “one of the reasons that static analysis sometimes reports
true but trivial issues is that static analysis tools, in general, don’t know what the code
is supposed to do. Thus, it can’t check that the code correctly implements what it is
supposed to do.”

You can think of syntactic code analysis as a dashboard gauge for tactical situational
awareness, allowing the developer to see in real time if software risk is starting to
accumulate in the components the developer is working on.

Global System Analysis

The limitations of syntactic software analysis have motivated industry to develop
“contextual software analysis” tools that understand the system at a global level, with
the added context of intercomponent dependencies and data flows.

The most sophisticated of these use semantic language analyzers to decompose the
software component by component, and then use the results to develop an internal
model of the entire system, with all its components, dependencies, and database
accesses. Two examples of such tools are Coverity Architecture Analysis and CAST
Application Intelligence Platform (CAST AIP). Coverity focuses on a single technology,
either Java or C++, while CAST can understand multiple technologies within the same
system, covering a range of languages and all the layers of the system.

Once a contextual software analysis tool creates its model of the system, it can examine
the broader system for areas of risk, at both the unit and system levels. Contextual
software analysis will find critical problems that analysis at the unit level cannot. For
instance, results from a study by Zude Li et al. suggests that defects involving several
components “account for approximately 30% of all defects but require 60% of corrective
changes and over 80% of the correction effort.” These defects are undetectable by local
syntactic software analysis alone. Moreover, with the added context of interactions
between components, contextual software analysis can also avoid a high degree of false
positives, which component-only analysis would trigger.

Cutter Consortium • Document prepared for CISQ 11

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

A potential drawback of contextual and system-level analysis is that because there are
orders of magnitude more files and pathways to examine, the analysis will take much
more time. Running contextual software analysis can’t be done instantly; instead, it runs
during the system integration phases of the software development lifecycle. However,
since contextual software analysis — particularly at the whole system level — can find
system stability, reliability, and security problems, it is a crucial tool in reducing
software risk.

Business Benefits of Managing

Software Risk
In this report, we have seen how we can systemize software risk management, we have
examined the kinds of problems to survey, and we have reviewed an array of powerful
tools to help measure and reduce software risk. These tools can help you industrialize
software risk management, offloading automatable work from your engineers, so they
can focus on the knowledge work that’s in your company’s future. With all these tools
in hand, your organization can improve development velocity, reduce the chance of
outages or security breaches, and compete for current and future business more
effectively.

Software risk is a critical type of business risk. Mitigating software risk directly reduces
business risk, including loss of revenue as well as any negative effects on public image,
customer happiness, and employee happiness. Moreover, it helps:

• Improve organizational agility and throughput — follow customer and user

needs more effectively and complete projects sooner and more efficiently.

• Get more for less — cut wasted resources, underperforming projects, and under-
utilized applications.

• Enhance your reputation as an analytic-driven leader.

• Place yourself in better negotiating positions with your senior team and
executive peers, with quantifiable reduced risk and improved quality.

The Cutter report “Mitigate Business Risk and Unlock Software Potential with
Contextual Software Analysis” offers more on this topic.

Cutter Consortium • Document prepared for CISQ 12

Mitigating Software-Related Business Risk Requires a Systems Perspective April 2017

About Peter Kaminski

Peter Kaminski is a Senior Consultant with Cutter’s Agile Product Management &
Software Engineering Excellence practice. He has served as technical cofounder for
six Internet startups, including Socialtext, which introduced wiki-based enterprise
collaboration, and Yipes Communications, which introduced carrier-Ethernet enterprise
metropolitan area network services. As a veteran Silicon Valley entrepreneur, Mr.
Kaminski provides intelligent and experienced insight into discussions of product and
service development and operations. He also has 30+ years’ experience in delivering
interactive consumer and enterprise software applications. With his startup background,
Mr. Kaminski has an eclectic and broad range of knowledge and skills, from the detailed
minutiae of low-level code, networking, and data representation to team dynamics,
Agile development, and business/technology collaboration to overall technology
architectures, strategic alignment, and high-level business and market strategy. He can
be reached at [email protected].

Cutter Consortium • Document prepared for CISQ 13