See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/306285779

An Exploratory Study of DevOps: Extending the Dimensions of DevOps with

Practices

Conference Paper · August 2016

CITATIONS READS

67 3,870

3 authors:

Lucy Ellen Lwakatare Pasi Kuvaja

University of Helsinki University of Oulu
29 PUBLICATIONS 1,703 CITATIONS 109 PUBLICATIONS 2,238 CITATIONS

SEE PROFILE SEE PROFILE

Markku Oivo
University of Oulu
220 PUBLICATIONS 4,941 CITATIONS

SEE PROFILE

All content following this page was uploaded by Lucy Ellen Lwakatare on 19 August 2016.

The user has requested enhancement of the downloaded file.

 An Exploratory Study of DevOps
Extending the Dimensions of DevOps with Practices

Lucy Ellen Lwakatare, Pasi Kuvaja, Markku Oivo,

M3S, Faculty of Information and Electrical Engineering

University of Oulu,
P.O. Box 3000, 90014 Oulu, Finland
Email: [email protected]

Abstract—Software-intensive companies constantly try to improve development and IT operations. The DevOps phenomenon,
their software development process for better software quality despite its growing interest in software industry, faces several
and a faster time to market. The DevOps phenomenon emerged challenges such as the lack of a clear definition [5]. This lack
with the promise of easing the process of putting new software of clear a definition has resulted to a number of problems and
changes to production at a fast rate whilst also increasing the criticisms, including tensions as to whether DevOps is about
learning and innovation cycles of their products. However, the
DevOps phenomenon lacks clear definition and practices, and
culture, technical solution or, alternatively, an entirely new role
this makes it difficult for both researchers and practitioners within a software development organisation [6].
to understand the phenomenon. In this paper, we focus on The goal of this research is to consolidate the understanding
consolidating the understanding of DevOps and its practices of DevOps phenomenon as described by practitioners. We use
as described by practitioners using multivocal literature and an exploratory case study technique that involves a review of
interviews. The study contributes to a scientific definition of multivocal ’grey’ literature and interviews. Our work extends
DevOps and patterns of DevOps practices to help identify and other previous studies that have tried to characterise the De-
adopt the phenomenon.
vOps phenomenon. Multivocal literature review and interviews
Keywords–DevOps; Continuous Deployment; Agile. were selected as appropriate approaches for this study because
DevOps is very much driven by practitioners, and as such,
I. I NTRODUCTION contribution from non-scientific community are worthwhile.
Innovative online companies, such as Amazon, Google The contribution of this paper is twofolds. First, to validate
and Facebook, have fuelled customers expectations for great and improve the scientific definition of DevOps proposed by
services at fast speed due to their quick response times to Penners and Dyck [7]. Second, to extend our work on the
customer demands. Consequently, more companies from most dimensions of DevOps [8] with a set of examplary practices
fields are learning and emulating their capabilities in order to and patterns of DevOps. The following research questions are
cope with competition and technological changes in the field addressed in this study:
of IT [1]. Today’s technology landscape and advances, such
• RQ1: How do practitioners describe DevOps as a
as cloud computing, have changed the ways in which soft-
phenomenon?
ware products are developed and delivered to customers. For
instance, in the cloud environment, providers of Software-as-a • RQ2: What are the DevOps practices according to
Service (SaaS) applications are expected to update software software practitioners?
frequently and in much faster release cycles to customers. This paper is organized as follows: Background and related
The recent paradigm shift towards fast and frequent de- work, including a scientific definition of DevOps, are presented
livery of software updates to customers is referred to as in the next section. Section 3 presents our research methodol-
continuous deployment (CD) [2]. CD has been described as an ogy. The results of the study are presented in Section 4, which
evolutionary step after Agile and continuous integration (CI) is followed by a discussion and conclusions in Section 5 and
practices [2] [3]. CD is a practice whereby software features 7, respectively. Section 6 presents validity threats including
and updates are rapidly rolled out to production as soon as limitations of the study.
they are developed, whilst also rapidly learn from real-time
customer usage of software [2] [3]. The advantage is that II. BACKGROUND AND RELATED WORK
companies can proactively identify and validate assumptions According to Humble and Molesky [9], DevOps– a blend
of customer needs by applying practices, such as feature of two words Development and Operations– is about align-
experimentation, that tightly integrate runtime-monitored data ing incentives of everybody involved in delivering software,
fromEmphsi production into the software development activi- with particular emphasis on developers, testers and operations
ties [4]. personnel. The problems resulting from misalignment between
Responsiveness to customer needs achieved through CD development and operations are not new, though their appear-
can put a strain on functional teams within an organisation ance in the literature is scarce [10]. Prior studies investigating
[2]. Consequently, the DevOps phenomenon emerged with the cooperation between developers and operations personnel in
aim of breaking down organisational silos and encouraging real contexts have revealed that very often development and
cross-functional collaboration among stakeholders involved operational activities are not tightly integrated [10] [11]. The
in software development and delivery processes–especially latter, according to Iden, Tessem and Päivärinta [11], results
to a number of problems, including IT operations not being as culture, sharing, automation, collaboration and measurement
involved in requirements specification, poor communication have been presented in other works [8] [9] [14]. However,
and information flow between the two groups, unsatisfactory the various aspects or dimensions of DevOps seem to lack
test environments, lack of knowledge transfer between the two a consolidated overview to a set of practices and patterns
groups, systems put into production before they are complete attributed to DevOps [15] [16] [17]. The latter is largely due to
and operational routines not established prior to deployment. the limited number of scientific studies reporting the DevOps
These problems were identified from a delphi study consisting phenomenon in practice, although this is changing as there
of 42 experts grouped in three panels representing the roles of is presently an increasing number of studies of DevOps e.g.,
developers, operations personnel and systems owners. In the studies from IEEE Software special issue on DevOps [18] [19]
latter study, the authors [11] concluded that operations person- [20].
nel are to be regarded as important stakeholders throughout
system development activities, especially in systems require- III. METHODOLOGY
ment, testing and deployment. We report an exploratory case study [21] on the DevOps
The closest similar work, though different in terms of phenomenon conducted between September 2015 and April
the studied phenomenon, is by Tom, Aurum and Vidgen 2016. Exploratory case studies are useful in finding out what
[12]. Using a multivocal literature review (MLR) approach is happening on a phenomenon whilst also seek new insights
supplemented by interviews, the authors of the latter study [12] and generate ideas for new research.
examined and consolidated the understanding on an unclear
phenomenon i.e., technical debt. The approach was used by A. Data Collection
the authors to develop a theoretical framework that gives a This study uses qualitative data–both primary and sec-
holistic view of the phenomenon comprising of dimensions, ondary data– collected from practitioners in two phases. Phase
attributes, precedents and outcomes. In this study we apply 1 involved collecting primary data using semi-structured in-
similar approach used by Tom, Aurum and Vidgen [12] to terviews with software practitioners from one case company.
consolidate the understanding of DevOps phenomenon whilst Secondary data, consisting of readily accessible writings from
also compare and complement other related works of DevOps the Internet, such as blogs, white papers, trade journals and
particularly those done by Kerzazi and Adams [6] and Penners articles, were collected in phase 2 of the study using Google
and Dyck [7]. search. The documents gathered during Phase 2 are collectively
referred to as multivocal literature (ML) [22]. The multivocal
A. DevOps Definition and Practices literature review provides a method for exploring a common,
often a contemporary topic of interest [22]. ML embodies
DevOps is a phenomenon that has often been said to a
views and voices of diverse set of authors expressed in variety
lack clear and precise definition [5] [7] [8] [13]. Its ambiguity
of forms [12] [22].
and lack of clarity often hinders its adoption [5]. To address
this problem, a scientific definition of DevOps is proposed 1) Phase-1 Interviews: Three interviews with software
by Penners and Dyck [7]. The definition was derived from practitioners were conducted in one company (Company A),
comparing and contrasting various descriptions of DevOps that provides consultancy and product engineering services to
and release engineering as the two terms seemed to have its customers. In the latter company, we focused our study
a big overlap [7]. According to the authors, DevOps and on one project, that involved the development of cloud-based
release engineering share the same goal of providing high- road maintenance reporting tool. The company was devel-
quality software as fast as possible. However, DevOps tries oping the tool for a customer in the public sector. Conve-
to achieve the goal by improving the collaboration aspect, nience sampling was used to select suitable companies based
whereas release engineering addresses the goal in a holistic on their participation in the Need for Speed (N4S) project
way i.e., covers all aspects that impact or influence the delivery (http://www.n4s.fi/en/), which the present study is part of. A
process of software product to customer [7]. The authors [7] contact person from company A was asked to select a project
define DevOps as: team that was applying DevOps practices. All interviews were
recorded and later transcribed for analysis. The roles of the
”a mindset, encouraging cross-functional collabora- interviewed practitioners are summarized in Table I .
tion between teams - especially development and IT
operations - within a software development orga-
nization, in order to operate resilient systems and TABLE I. S UMMARY OF I NTERVIEWEES .
accelerate delivery of changes ”.
Company A
This definition of DevOps was developed through discus-
sion with some experts, but the feedback of its description Number of Employees < 350

from practitioners was not as consistent as that of release en- Studied Project Road maintenance tool.
gineering. Based on this, the authors recommended additional
Team Size 7 (co-located).
inquiry from more practitioners for a more precise definition of
DevOps. This study explores how different practitioners have Number of Interviews (role) 2 (senior developers) 1 (project manager)
described DevOps and in addition compare our findings of
such descriptions with the definition provided by Penners and A semi-structured interview guide was used during the
Dyck [7] [13]. interviews and had questions that inquired about the (1) current
In addition to the definition of DevOps, different aspects way of working in software development and deployment,
and dimensions characterising the DevOps phenomenon such (2) strengths and weaknesses in ways of working and (3)
definitions, practices, benefits and challenges of DevOps and data into the main categories, i.e., Definition and Practices,
CD. DevOps questions were guided by the results of our a second iteration of inductive coding was performed to form
previous work that analysed and synthesized the DevOps other subcategories. The latter proceeded in multiple iterations
phenomenon as described in academic literature [7]. until similar patterns in the practices and definition were
2) Phase-2 review of multivocal literature: In phase 2, identified and saturation reached. Interestingly, the practices
we conducted MLR in the beginning of March 2016. In could be grouped into the dimensions of DevOps identified in
this study, we selected the MLR as an appropriate method previous work [8]; however, a new dimension was also added.
because DevOps is a phenomenon that is largely driven and
discussed by practitioners in non-academic circles, and as such IV. FINDINGS
it requests the review of the largely available information in In this section, we present the findings with respect to the
non-scietific forums, as seen also in a study done by Kerzazi two research questions. Fig. 1 gives a summary of the findings
and Adams [5]. Despite the apparent issues of rigor associated from MLR. For MLR, Internet blogs made up the largest
with MLR approach [22], the contemporary nature of DevOps source with 53.2% (107) of all sources. Compared to personal
phenomenon suggests some value in reviewing ML that cannot blogs, most of the blogs were affiliated with companies. The
be overlooked. As noted by Ogawa [22], the information MLR showed a growing number of sources writing about
presented in ML cannot be assessed in the same way as DevOps over the past 5 years since its conception in 2009.
that of academic literature. We therefore considered various From the case company, the project team in company
recommended procedures by Ogawa [22] in order to minimize A uses Scrum– agile software development method, with a
bias and error that can be transferred to, and incorporated in 3-week sprint cycle. The development team has three en-
the review of ML. The latter included focusing our study on vironments to which software changes are deployed, i.e.,
presenting a thicker picture of the DevOps phenomenon that development, test and staging. The production environment
would serve as input for more sophisticated examination in was not in use at the time of the interviews because the project
future research. We carried out the MLR in the following three was phased and the deliverables as well as the schedule of
stages: each phase was determined by the customer. The development
a) Data sources and search strategy: Google web team automatically deployed software changes from the test to
search engine (http://www.google.com) was used to source ML staging environment using Ansible playbooks. The infrastruc-
from the World Wide Web. The query used to retrieve results ture team located in another city supported the development
from the search engine was what is DevOps. From the retrieved team by provisioning the virtual servers used to create the
results, we went through the links provided, page by page, environments.
saving the outputs of each link as a PDF file until the page
where job adverts started and at this point, the review was A. How do practitioners describe DevOps as a phenomenon?
stopped. A total of 230 records were collected as data sources (RQ1)
and included in subsequent steps. Most practitioners acknowledge that DevOps is a concept
b) Inclusion and exclusion: A total of 230 documents that is difficult to define with accuracy. However, many of
were imported to NVivo (www.qsrinternational.com/product) them still used different terms and descriptions to describe
for analysis. The inclusion and exclusion of the records were the DevOps phenomenon as elaborated below:
done simultaneously with the initial coding of these records.
The process also involved classifying the sources with different 1) DevOps referred to as: A large number of MLR
attributes, such as author information, e.g. name, role and place sources have referred to DevOps as a cultural and professional
of work; and source information, e.g. publication year, forum movement (52). Other terms used to describe the DevOps
and link. After reviewing all 230 documents, 201 sources were phenomenon in MLR sources include practices (41), culture
included as relevant documents and excluded 29 documents as (38), approach (38), philosophy, mindset, ideology (37), tool
they were either duplicates, video links, pointers to catalogues, (36), a set of values and principles (30), software methodology
course adverts, certification adverts or presentation slides. (24), role, team or engineer (19) and strategy (8). Fig. 2
c) Quality assessment: Quality assessment was mostly summarizes the prevalence of the terms in MLR sources. It
done when classifying the sources with metadata, e.g. author should be noted that a single source of MLR can use more
name, role, place of work/affiliation, year of publication, view than one term to describe the DevOps phenomenon.
point. This process offered minimal assessment of position,
certainty and clarity of source or alignment with research goal 2) DevOps descriptions: There is an agreement that the
i.e., to consolidate the understanding of DevOps phenomenon term is a combination of development and operations, that
as determined by the research questions. encourages collaboration between software development and
operations activities. However, when describing the concept
B. Data Analysis further, practitioners often diverge their descriptions depending
Interview transcripts and results from the MLR (list of ML on whether they place the emphasis more on the goal or more
at http://tinyurl.com/z3jpu5v) were coded in NVivo. At first, on the means of achieving collaboration. The most common
codes were assigned inductively to the following categories: goals that were described include the following: to reduce
(1) Definition (with referred as and description as subcate- response time and fast deployment of high quality and reliable
gories) and (2) Practices. Other emerging themes found within software products and services, to allow instant interactions,
and across the sources were also coded (e.g., Motivations to unify workflows for transparency and collaboration. On
for DevOps, DevOps in relation to Agile and CD, Problems the other hand, those that emphasise means include: through
addressed by DevOps). Following the inductive coding of raw advanced automation and, by evolving traditional roles of
 ML Source Type ML Blog Type Publication year of ML sources

120 70 80

Number of sources
100 60 70
Number of Sources

Number of sources
60
80 50 50
40 40
60 30
30
40 20
20 10
20 10 0
0 0
Article Blog Other White company Other Blog Personal
Posts Paper blog Blog
Source Typle Blog Type Publication Year

Figure 1. ML Sources.

opment organisation from the operational organisa-

tion...like instead of provisioning new systems with a
DevOps Referred As ticket, automation projects are modifying the servers
and developer can make or commit changes to them
Number of Sources

60
by themselves when needed. If you have a separate
40 ops team, then you need to have a very good practice
in place for documenting changes and transferring
20 knowledge about the system. A DevOps person will
know how to run their own system .
0
Terms used to refer to DevOps phenomenon B. What are the DevOps practices according to software
Movement- Cultural and professional practitioners? (RQ2)
Practices
Culture DevOps practices can be crystalized into five dimensions
Approach that characterize DevOps. The five dimensions are further
Philosophy, Mindset, ideology elaborated with examplary DevOps practices and patterns to
Tool the practices below. Table 2 summarizes some of the DevOps
A set of values and principles
Methodology, method, process practices in each of the dimensions.
Role,Team, Engineer 1) Collaboration: rethinking and reorientation of roles and
Strategy teams in development and operations activities: The issue of
role is one of the most widely discussed topics by practitioners
in ML. On the otherhand, DevOps phenomenon offers no blue
Figure 2. Terms referring to DevOps and their prevalance in ML. print of how companies can reorganize those roles. A common
pattern that was observed is that, companies are forced to
rethink and reorient roles, whether new or existing, around the
performance of the entire system or service, as opposed to the
development and operations. In company A, two out of three
performance of a specific silo of department or an individual
interviewed practitioners had an understanding of DevOps.
module. Practices in collaboration are often seen as empow-
One person, the project manager, had no prior understanding
erment to team members, especially for the developer since
of the concept and had learned about it through our study. The
in some cases they gain more control over system operability.
following descriptions are extracts from interviewed practition-
This in turn helps to broaden their skillset and knowledge.
ers in company A when asked how they understood DevOps.
According to some practitioners, such control allows a single
The first developer described DevOps as
team to be responsible for all aspects, i.e. development and
a set of practices to govern everything that is related operations of the entire software product or service. Some
to installing the software, maintaining it, making criticism to this was also observed, including coding time for
sure that all these connections work, i.e. firewalls, developers is reduced and the fact that developers need to find
version management, etc. It is kind of a little bit of ways to effectively balance the support and operations tasks
what happens, right after the actual development. alongside development tasks. Others agree that, companies will
need to pick right technology and methodologies to be able to
Another description given by the second developer was:
empower developers, as advocated by DevOps. We identified
DevOps is mostly about the organisation of work, the following two common practices in the collaborative aspect
tearing down the walls that separate typical devel- of DevOps.
 TABLE II. DEVOPS DIMENSIONS, PATTERNS AND PRACTICES

DevOps Patterns of DevOps prac- Examples of practices found in ML

Dimension tices

Collaboration Rethinking and reorientation Increasing the scope of responsibilities

of roles and teams in develop- • Developers paying closer attention to deployment scripts, configuration files, load and performance testing
ment and operations activities and other activities usually associated with operations groups
• Developers learning from operations about resilience, monitoring and diagnosing distributed systems
• Developers leverage the support enabled through virtualization, microservice architecture and automation
in deployment pipeline to do less operations work, e.g., Docker to remove the need for specifying
environments specifications
• Development has access and can make changes to critical environments, e.g. production
Intensifying cooperation and involvement in each others daily work
• Development rotates roles with operations teams, operations attend developer stand-ups and showcases
• Operations involved earlier in development to understand what project environments are required to support
the application. Also, regular meetings, e.g. weekly to discuss cross-team priorities
• In circumstances in which production incidents occur, development and operations come together to
troubleshoot and resolve problems as one team
• Setting-up shared (virtual and physical) workspace

Automation Infrastructure and deployment Infrastructure-as-code

process automation • Automate and maintain infrastructure configurations and files using tools such as Chef, Puppet
• Developers are shielded from infrastructure issues and are able to create virtual development, test and
production environments as well as deploy application using tools like Vagrant and Docker
• Scripts used to handle infrastructure are versioned, testable and repeatable
• Immutable infrastructure, i.e. artifacts, in their production environments are not updated, rather the
infrastructure is always replaced
Deployment process automation
• Production-like environments are used by development teams for development and testing
• Developers self-service environments and deployments
• Consistent, reliable and repeatable deployment mechanism across different environments
• Configuration changes across environments are automated

Culture Empathy, support and good

working environment between • Both developers and operations wear pagers as responsible persons to handle incidents
development and operations • Integrating development into blameless production post-mortems
• Making communication between development and operations non-adversarial and less formal
• Mutual respect, support and willingness to work together and share responsibilities

Monitoring Instrumenting application and

aggregating monitored data • Developers and operations are both involved in determining and implementing monitoring parameters of
into insights a system
• Set-up monitoring on the production environment for development teams visible through radiators
• Development team, including QA, use small subset of high-priority test cases to be executed in production
to actively monitor the environment
• Effective instrumentation of software by development in collaboration with operations to give information
about its health and performance. Also, developers are able to quickly recover code failures in production
using aids, such as feature flags

Measurement Useful Metrics

• Both operations team and development team are incentivized and rewarded by the same metrics
• Both development and operations focus on business value as the essential unit of measurement
• Progress in development is measured in terms of a working system in production environment
• Developers use production feedback to drive decisions, improvements, and changes to the system

a) Increasing the scope of responsibilities: This prac- within the ops teamand ask should you even have an ops
tice is particularly prominent in the developer role and is team and a development team. If something is broken on the
the most common implementation of DevOps phenomenon. production machine, why do you need an ops team to fix it?
According to practitioners, developers are responsible for other It should be possible for anyone in the dev team or any other
tasks in addition to designing, coding and testing. Increasing team to just fix things that are broken.
the scope of responsibilities involves pro-activeness of team
members in learning the new tasks or alternatively leverage b) Intensifying cooperation and involvement in each
on the automation done to the technical infrastructure of the others daily work: DevOps requires both groups to recognise
project. This was also evident from the interviewed practition- their key skills in order to share and learn from each other. This
ers as expressed by the developer from company A: Shared aspect may not necessarily mean retraining or crossskilling but
access to change things, so not just that there’s collaboration encourages providing feedback and visibility across the teams
in order to improve. Involving members from each others
teams, development and operations teams can provide valuable Jenkins and it runs unit tests. If everything goes
information outside individuals areas of expertise. As such, fine, it automatically installs the new version to the
practices such as early involvement of operations in devel- test environment. At some point, we started using
opment are emphasized. The following interview extract is a Selenium for automated end-to-end functional tests.
response given by an interviewed practitioner from company We have had quite a lot of random failures with
A when asked to elaborate how the development team interacts Selenium tests, for example, we did not know if the
with the infrastructure team: software or just the test tool was broken. Quite often,
either of these were true, and sometimes I think we
We interact with them through HipChat, which is
also run out of space in CI machines and the Jenkins
like an IRC channel. We have a project channel and
build would fail because there is no space left. So
a member of infrastructure team; [name] has been
we have used [name of the assigned person from
allocated to support our project. We are basically
Infrastructure team] to debug and have a look at
just sending a message to [name]if we need some
what’s going on when we have encountered failure
help, e.g., we want two more servers to run Java
that we shouldn’t have.
virtual machine and Jenkins, and then he waves some
magic and we get what we want. It’s pretty instant b) Infrastructure-as-code: This practice is central to au-
even if we are technically in a different team i.e., tomation dimension and entails treating infrastructure the same
virtual team element, in it. way developers treat code. This includes making the process
of infrastructure provisioning, orchestration and application
2) Automation: infrastructure and deployment process au-
deployment reproducible and programmatic. Application code
tomation: Automation underlies most of the practices that
has a defined format and syntax that follows the rules of the
constitute DevOps. Two common practices include automation
programming language in order to be created. Typically, the
of deployment process and infrastructure-as-code (IaC). The
code is stored in a version-management system that logs a
two practices help to eliminate error-prone manual processes
history of code development, changes and fixes. When code
associated with deployments and configuration settings for
is compiled (built) into an application, the expectation is that
improved traceability and repeatable workflows. For instance,
a consistent application is created. When the latter occurs, the
the use of IaC practice to standardize development, test and
build process is said to be repeatable and reliable. Practic-
production environments.
ing IaC means applying the same rigor of application code
a) Deployment process automation: as evident in de- development to infrastructure provisioning, orchestration and
ployment pipeline that views the entire development to opera- deployment. All configurations should be defined in a declara-
tions lifecycle as one coherent end-to-end process rather than tive way and stored in a version management system, just like
numerous multiple and diverse steps. The deployment pipeline application code. With IaC, it becomes possible to quickly
is an integrated flow that puts great emphasis on the automation provision application environments and deploy application
of build, test and deployment processes. It involves continuous automatically and at scale as needed. The following interview
development whereby code written and committed to version extract is a response given by an interviewed practitioner from
control is built, tested and installed (deployed) in the produc- company A about how the team handles their infrastructure:
tion environment. An automated deployment process takes into
account and ensures the management of dependencies, versions We use Ansible for all server automation, so we
and configurations of application and infrastructure, which is have Ansible tasks and playbooks for the infras-
often challenging. The following interview extract is a response tructure, i.e. we use Ansible projects for setting up
given by an interviewed practitioner from company A when the machines and installing. When a new machine
asked to give an example of a DevOps practice they have had comes, we just automatically run that and it sets
in their team: up everything. Then we have other playbooks for
deploying our software on top of the page.
The actual building of the CI pipeline has been
done collaborative with ops team, who basically just 3) Culture: empathy, support and good working evironment
provisions machines for us. They have a library of for teams especially development and operations: Many of
useful profiles that can be helpful, e.g., our servers DevOps practices also involve changing culture and mindset to
have Java 8, database servers have PostgreSQL the one that encourages empathy, support and a good working
installed, etc. So, we’ve been pretty active in building environment for those involved in software development and
and modifying our own CI pipeline, and we have had delivery processes.
within ourselves actual tangible tasks to deploy our a) Empathy: According to majority of ML authors,
stuff to production. development and operations need to empathize with each other
and more importantly also with users of software product
In addition to the development team interacting with the
or service. For operations engineers, empathy helps them to
operations team to implement the deployment pipeline, the
understand the needs as well as to appreciate the importance
teams collaboratively work together to improve the pipeline
of being able to push code quickly and frequently, without
in order to improve their processes. This is described by an
problems. To developers, it also allows them to recognise
interviewed practitioner when asked to give an example of a
the problems resulting from writing code that is erroneous,
problem that the development team encountered and needed
unstable or insecure. Generally, empathy in DevOps culture
help from the infrastructure team:
allows software developers and operators to help each other
We are using Jenkins in CI, and whenever we want deliver the best software possible. Information exchanges
to update software, we push the newest version to require (and can contribute to) mutual understanding. As an
example to this pattern includes having both developers and system levels are used, and different tools are used to monitor
operations, personnel wear pagers as responsible persons to application and infrastructure. Although modern tools try to
handle incidents. provide insight into almost every aspect of system behaviour,
b) support and a good working environment for the biggest challenge facing the tools is developing data
teams: The need for mindset change and cross-discipline analytics that predict problems before they become outages.
learning associated with DevOps necessitates the support Some practitioners also argue that it is impossible to catch all
from members within and across development and operations the issues using the out-of-the box features of such tools.
teams, as well as of those outside the two functions or roles.
A good working environment that welcomes innovation, 5) Measurement: different metrics are used to monitor
experimenting and stops finger pointing when mistakes and assess the performance of processes in development and
are done as long as they constantly improve learning, helps operations activities, e.g., developers on system quality and
to facilitate and embrace DevOps phenomenon amongst others. operations on system stability. As such, instead of using proxy
metrics, DevOps, according to practitioners, emphasizes the
4) Monitoring: Instrumenting application and aggregating use of common metrics that are often business focused to
monitored data into insights: This pattern involves having assess and give incentive to both development and opera-
a continuous feedback loop that runs from the production tions teams. Additionally, developers, operations and other
environment to the start of the development cycle, including stakeholders can use production feedback to drive decisions,
a complete timeline of development and operations events. improvements and changes to the systemnot just problem
It involves proactive detection and awareness of events in reports and user suggestions, but measurable data collected on
critical environments, such as test and production, in order how the system is working according to the conversion rate or
to expose (know the state of) issues before they cause failures. whatever metric that the business uses to determine success.
According to practitioners, the latter is important especially for
SaaS application because development teams are increasingly V. D ISCUSSION
placing more reliance on detection and recovery than standard Our findings show that often practitioners vary their de-
QA testing practices. Since the cost and time of fixing defects scription of DevOps depending on whether they put their
in production for SaaS are less than packaged software, teams emphasis on either the goal or the means for achieving collabo-
tend to reduce reliance of extensive testing and instead rely ration between development and operations. The most common
more on production monitoring to detect defects as long as goal according to practitioners was to reduce response time and
they can be quickly fixed. On the other hand, monitoring of provide fast deployment of high-quality and reliable software
advanced architectures with highly distributed systems and products and services. This finding supports the second part
nodes that appear and disappear pose challenging tasks that of the definition proposed by Penners and Dyck [7]. Evidence
should not only involve operations engineers. Two common from how practitioners referred to the DevOps phenomenon
patterns in monitoring include emphasis on instrumentation of also supports that DevOps is a mindset as stated in the pro-
application and increased use of modern tools for monitoring posed definition. However, the results showed that, in addition
purposes. to being a mindset, DevOps constitutes practices attributed
a) Instrumenting application: A well-instrumented to it. We therefore argue that both mindset and practices be
software system can provide rich data and insights about its included in the definition and that the first part of the definition
health and performance that can be used in bug reporting, should not exclude the results. By doing this, we improve the
troubleshooting, feature suggestion or general finetuning of definition to be DevOps is a mind-set substantiated with a
the system. Effective instrumentation of an application tries set of practices to encourage cross-functional collaboration
to minimize problems associated with the aggregation of between teams - especially development and IT operations
large amounts of data from a variety of sources by capturing - within a software development organization, in order to
feedback as part of the application. Reliance on monitoring, de- operate resilient systems and accelerate delivery of change.
velopers can quickly recovery from code failures by the use of
feature flags that enable or disable code functionality through On the other hand, the DevOps phenomenon has no explicit
configuration settings. Additionally, instrumentation can help one-size-fits-all set of practices that guide its adoption and
to extend features of monitoring tools with domain knowledge. implementation, but that common patterns can be identified
To implement this, a broad set of skills are required amongst from its diverse set of practices. For each pattern of DevOps,
developers that involve mainly scripting and knowledge of practitioners can choose different ways to implement it even
performance-monitoring practices, often a responsibility of though it was possible to identify similar implementation.
operations. The following interview extract is a response given Patterns to the DevOps practices are more useful than one-
by an interviewed practitioner from company A when asked fits-all set of practices that will not suffice due to contextual
to give an example of a DevOps practice experienced by their limitations in organisations, e.g., team members skill and
team: scenarios vary among companies. As an example, we can
observe the reorientation of roles and teams in development
When we do not know how to do something, the and operations activities pattern in studies reported by Balalaie,
ops team, i.e. the infrastructure team, has shared Heydarnoori, and Jamshidi [19]. In the latter study, the authors
services that I use, like log aggregation services and depict the formation of small and autonomous teams consisting
monitoring services. We have asked them to set these of multi-skilled persons of both development and operations
up for us. activities as a DevOps practice. According to the authors,
b) Aggregation of monitoring data into insights: For the practice helps to minimize development and operations
comprehensive system monitoring, several tools at different inter-team coordination, which is important for Microservices.
Callanan and Spillane give empirical evidence to infrastructure VII. C ONCLUSION AND F UTURE WORK
and deployment automation pattern [18]. Continuous monitor- The analysis of multivocal literature and interviews with
ing of both infrastructure and application, as a DevOps practice software practitioners showed software practitioners use dif-
helping to bridge development including quality assurance and ferent terms and variety of practices when describing the
operations are described as lesson learned and experiences by DevOps phenomenon. We used findings from our analysis
authors of these studies [15] [16] [17] [18] [19] [20]. to provide more evidence that supports and builds upon the
Generally, the results of our study signify the importance scientific definition of DevOps proposed by Penners and Dyck
of considering context when describing DevOps practices. In [6]. Our findings showed that DevOps as a phenomenon
our case study, even though most practitioners elaborated the is not just a mind-set but rather some patterns of DevOps
practices in the context of cloud and web development, other practices described by the practitioners can be identified. This
contextual information, such as size and maturity, influenced study presents the identified patterns in relation to the five
the ways in which some practices are implemented. This is an dimensions of DevOps, i.e. collaboration, automation, culture,
important consideration that needs to be taken into account monitoring and measurement. Our findings show that DevOps
when discussing DevOps practices, i.e., to understand the phenomenon is more prominent in organisations providing
context in which the practices are described. services over the Internet. It would be beneficial for future
research to focus on further empirical evidence of DevOps
VI. T HREATS TO VALIDITY practices and patterns in companies that claim to have imple-
This study has a number of validity threats that need to be mented it. More important will be a research that not only
taken into account. We considered three categories of validity identifies the practices but also for which kinds of system,
threats– construct validity, reliability, and external validity organisations and domains are DevOps practices applicable.
[21]– and used different countermeasures to minimize the
threats. The countermeasures included: (a) consulting multiple ACKNOWLEDGMENT
sources in MLR (b) maintaining chain of evidence, and (c) This work was supported by TEKES (Finnish Funding
incorporating practitioners reviews. In addition, particulary to Agency for Innovation) as part of the Need for Speed project
MLR approach, three minimal standards for enhancing rigor (http://www.n4s.fi/) of DIGILE (Finnish Strategic Centre for
in ML suggested by Ogawa [22] were considered. Science, Technology and Innovation in the field of ICT and
To ensure construct validity, we only included one out of digital business).
three contacted and interviewed case companies. The latter is
due to not having the two other companies implement DevOps R EFERENCES
as well as the lack of DevOps understanding thereof. As a [1] M. Leppanen et al., “The Highways and Country Roads to Continuous
result of the selection process, our study faced some limitation Deployment,” IEEE Software, vol. 32, no. 2, mar 2015, pp. 64–72.
with regards to the few number of interviews included in [2] P. Rodrı́guez et al., “Continuous Deployment of Software
this study. This limitation serves as an opportunity for further Intensive Products and Services: A Systematic Mapping Study,”
Journal of Systems and Software, jan 2016. [Online]. Available:
inquiry in future works. On the otherhand, the interviews were http://www.sciencedirect.com/science/article/pii/S0164121215002812
conducted with at least two researchers to minimize researcher [3] H. H. Olsson, H. Alahyari, and J. Bosch, “Climbing the ”Stairway to
bias. A document describing the background and objective of Heaven” – A Mulitiple-Case Study Exploring Barriers in the Transition
the study was sent to practitioners prior to the interviews. With from Agile Development towards Continuous Deployment of Software,”
regards to the MLR, a threat to construct validity comes from in 38th Euromicro Conference on Software Engineering and Advanced
Applications. IEEE, sep 2012, pp. 392–399.
different constraints, e.g., the selected search term as well as
the inclusion and exclusion criteria of ML. To ensure construct [4] T. Karvonen, L. E. Lwakatare, T. Sauvola, J. Bosch, H. H. Olsson,
P. Kuvaja, and M. Oivo, “Hitting the Target: Practices for Moving To-
validity and minimal rigor, prior to MLR, the objective of the ward Innovation Experiment Systems,” in 6th International Conference
study was clearly defined which served as the primary criterion on Software Business. Springer International Publishing, 2015, pp.
for seeking and selecting ML. 117–131.
To ensure the reliability of results, initial results were made [5] J. Smeds, K. Nybom, and I. Porres, “DevOps: A Definition and
Perceived Adoption Impediments,” in 16th International Conference on
available for discussion to practitioners and other researchers Agile Software Development (XP). Springer International Publishing,
prior to the writing of this report. One meeting (with company 2015, pp. 166–177.
representatives of the interviewed company) and a workshop [6] N. Kerzazi and B. Adams, “Who Needs Release and DevOps Engi-
with practitioners of the N4S research program were also neers, and Why?” in International Workshop on Continuous Software
conducted to solicit feedback from practitioners. Both events Evolution and Delivery. ACM Press, 2016, pp. 77–83.
were useful for collecting feedback. We also ensured an audit [7] R. Penners and A. Dyck, “Release Engineering vs. DevOps-
trail, i.e., maintain all records and analysis in NVivo. With An Approach to Define Both Terms,” Full-scale Software
Engineering, 2015. [Online]. Available: https://www2.swc.rwth-
regards to ML, the reliability of the study would have improved aachen.de/docs/teaching/seminar2015/FsSE2015papers.pdf#page=53
further with contact and additional discussion with the authors [8] L. E. Lwakatare, P. Kuvaja, and M. Oivo, “Dimensions of DevOps,”
of the ML. However, due to the large number of ML sources in 16th International Conference on Agile Software Development (XP).
and diverse set of authors, at the time of writing the report, Springer International Publishing, 2015, pp. 212–217.
this was not feasible due to limited time. [9] J. Humble and J. Molesky, “Why enterprises must adopt DevOps to
External validity and the generalization of the findings is enable continuous delivery,” Cutter IT Journal, vol. 24, no. 8, 2011, pp.
6–12.
threatened by the small number of interviewees as described
[10] B. Tessem and J. Iden, “Cooperation between developers and operations
earlier as well as our reliance on ML sources as data sources in software engineering projects,” in In Proceedings of the 2008
for analysis. For this reason we acknowlede this limitation and international workshop on Cooperative and human aspects of software
, the results serve as a basis for empirical evaluations. engineering. ACM, 2008, pp. 105–108.
[11] J. Iden, B. Tessem, and T. Päivärinta, “Problems in the interplay of
development and IT operations in system development projects: A
Delphi study of Norwegian IT experts,” Information and Software
Technology, vol. 53, no. 4, apr 2011, pp. 394–406.
[12] E. Tom, A. Aurum, and R. Vidgen, “An exploration of technical debt,”
Journal of Systems and Software, vol. 86, no. 6, jun 2013, pp. 1498–
1516.
[13] A. Dyck, R. Penners, and H. Lichter, “Towards definitions for release
engineering and DevOps,” 2015.
[14] F. Erich, C. Amrit, and M. Daneva, “Cooperation between information
system development and operations: a literature review,” p. Article
No.69, 2014.
[15] J. Roche, “Adopting DevOps practices in quality assurance,” Commu-
nications of the ACM, 2013, pp. 1–8.
[16] D. Cukier, “DevOps patterns to scale web applications using cloud
services,” in In Proceedings of the 2013 conference on Systems,
programming, & applications: software for humanity. ACM, 2013,
pp. 143–152.
[17] L. Bass, I. Weber, and L. Zhu, DevOps: A Software Architect’s
Perspective. Addison-Wesley Professional, 2015.
[18] C. Ebert, G. Gallardo, J. Hernantes, and N. Serrano, “DevOps,” IEEE
Software, vol. 33, no. 3, may 2016, pp. 94–100.
[19] M. Callanan and A. Spillane, “DevOps: Making It Easy to Do the Right
Thing,” IEEE Software, vol. 33, no. 3, may 2016, pp. 53–59.
[20] A. Balalaie, A. Heydarnoori, and P. Jamshidi, “Microservices Architec-
ture Enables DevOps: Migration to a Cloud-Native Architecture,” IEEE
Software, vol. 33, no. 3, may 2016, pp. 42–52.
[21] P. Runeson and M. Höst, “Guidelines for conducting and reporting case
study research in software engineering,” Empirical software engineer-
ing, vol. 14, no. 131, 2009, pp. 131–164.
[22] R. Ogawa and B. Malen, “Towards rigor in reviews of multivocal
literatures: Applying the exploratory case study method,” Review of
Educational Research, vol. 61, no. 3, 1991, pp. 299–305.

View publication stats