On the Use of GitHub Actions in Software

Development Repositories
Alexandre Decan Tom Mens Pooya Rostami Mazrae Mehdi Golzadeh
Software Engineering Lab Software Engineering Lab Software Engineering Lab Software Engineering Lab
University of Mons University of Mons University of Mons University of Mons
Mons, Belgium Mons, Belgium Mons, Belgium Mons, Belgium
[email protected] [email protected] [email protected] [email protected]

Abstract—GitHub Actions was introduced in 2019 and con- response to GitLab’s integrated CI/CD support [6]. In August
stitutes an integrated alternative to CI/CD services for GitHub 2019, GitHub officially began supporting CI through GHA,
repositories. The deep integration with GitHub allows reposi- and the product was released publicly in November 2019.
tories to easily automate software development workflows. This
paper empirically studies the use of GitHub Actions on a dataset GHA [7] allows the automation of a wide range of tasks
comprising 68K repositories on GitHub, of which 43.9% are using based on a variety of triggers such as commits, issues, pull
GitHub Actions workflows. We analyse which workflows are requests, comments, schedules, and many more. Its deep
automated and identify the most frequent automation practices. integration into GitHub implies that GHA can be used not
We show that reuse of actions is a common practice, even if only for executing test suites or deploying new releases
this reuse is concentrated in a limited number of actions. We
study which actions are most frequently used and how workflows as in traditional CI/CD services, but also to facilitate code
refer to them. Furthermore, we discuss the related security reviews, communication, dependency and security monitoring
and versioning aspects. As such, we provide an overview of and management, etc. GHA also promotes the use and sharing
the use of GitHub Actions, constituting a necessary first step of reusable components, called actions, in workflows. These
towards a better understanding of this emerging ecosystem and actions are distributed in public repositories and on the GitHub
its implications on collaborative software development in the
GitHub social coding platform. Marketplace. They allow workflow developers to easily in-
Index Terms—GitHub Actions, continuous integration, collab- tegrate specific tasks (e.g., set up a specific programming
orative software development, workflow automation language environment, publish a release on a package registry,
run tests and check code quality) without having to write the
I. I NTRODUCTION corresponding code.
Open source software (OSS) development is a continuous, Since its public release in November 2019, GHA has
highly distributed and collaborative endeavour [1]. Develop- become the most dominant CI/CD service, only 18 months
ment of OSS projects faces many socio-technical challenges after its introduction [8]. Its Marketplace of reusable actions
[2]–[4]. The multitude of tools (e.g., version control systems, has been growing exponentially ever since, reaching 12K
software distribution managers, bug and issue trackers) and reusable actions in February 2022. It is therefore fair to
development-related activities makes it very challenging for say that GHA has become a software ecosystem of its own,
contributor communities to keep up with the rapid pace of comparable to ecosystems of reusable software libraries (such
producing and maintaining high-quality software releases. as npm, RubyGems, CRAN, Maven, and PyPI) that have been
Automated workflows were introduced to automate numer- empirically studied by many researchers in recent years (e.g.,
ous repetitive social or technical activities that are inherently [9]–[14]).
part of the collaborative software development process. Con- The emerging GHA ecosystem is worthy of being empiri-
tinuous integration, deployment and delivery (CI/CD) have cally studied in its own right since it is likely to suffer from
become the cornerstone of collaborative software develop- the same issues related to dependency management, security
ment and DevOps practices. Well-known examples of CI/CD vulnerabilities, outdated or obsolete components, backward
services are Travis, Jenkins, CircleCI and TeamCity. They compatibility, and so on. This article therefore quantitatively
automate the integration of code changes from multiple con- studies the use of GHA in 68K repositories on GitHub. We
tributors into a central repository where automated builds, tests analyse which workflows are automated and identify the most
and code quality checks run. frequent automation practices. We show that reuse of actions
GitHub is by far the largest social coding platform, hosting is a common practice and identify which actions are reused
the development history of millions of collaborative software and how. As such, we provide an overview of the use of
repositories, and accommodating over 73 million users in GHA, a necessary first step towards a better understanding
2021 [5]. GitHub publicly announced the beta version of of the emerging GHA ecosystem and its implications on
GitHub Actions (abbreviated to GHA in the remainder of software development in GitHub repositories. More concretely,
this paper) in October 2018 based on popular demand, and in we answer the following research questions:
 • What are the characteristics of GitHub repositories using merged, and rejected without a decrease in quality measured
GHA workflows? in terms of reported bugs. Cassee et al. [23] investigated the
• Which kinds of workflows are automated? impact of Travis on the social aspects of software develop-
• What are the most frequent jobs in workflows? ment, focusing on the code review practices in 685 GitHub
• What are the automation practices? projects. Their results show that projects using Travis tend
• Which actions are reused and how? to have fewer discussions in their pull requests, suggesting
• Which versioning practices are being used? that developers perform the same amount of work with less
The remainder of this paper is structured as follows. Sec- communication after the adoption of Travis. Beller et al. [26]
tion II presents related work. Section III introduces some of analysed failures in over 2,5 million code builds on Travis.
the core concepts of GHA and the data extraction process. They found that testing is the single most important reason
Sections IV to IX address the research questions. Section X why CI/CD builds fail. They also found that the use of multiple
discusses the findings and their implications. Section XI integration environments leads to 10% more failures being
presents the threats to validity of the research, and Section XII caught at build time. Widder et al. [29] conducted a mixed-
concludes. methods study to identify the reasons why projects decided to
abandon the use of Travis. By analysing a thousand projects
II. R ELATED W ORK that stopped using Travis between 2011 and 2017, they found
Continuous integration has been introduced by Fowler and that long build times, CI/CD consistency across projects, lack
Foemmel in their seminal blog in 2000 [15]. They outlined 10 of tests and difficulty to troubleshoot a build failure are among
core CI practices aiming at increasing the speed of software the most frequent reasons to abandon Travis. Zampetti et
development and improving software quality. Among others, al. [22] performed a fine-grained mixed-methods study on
they stressed fully automated and reproducible builds and tests the evolution of specific Travis pipelines. They analysed 615
that run several times a day. Elaszhary et al. [16] discuss the CI/CD pipeline configuration change commits, and proposed
benefits and challenges of these practices in three software- 16 different metrics to capture how Travis pipelines evolve
producing companies. They found that these practices are and get restructured over time.
broadly implemented but how they are implemented varies Golzadeh et al. [8] conducted a quantitative study aiming
depending on their perceived benefits, the context of the to better understand the rapidly evolving CI/CD landscape on
project, and the CI/CD tools used by the organization. They GitHub. By analysing the use of 20 different CI/CD services
call for more research to understand these differences and from 2012 to 2021 in 91K+ active npm repositories, they
how they impact software development and quality. Hilton observed that GHA has become the most dominant CI/CD
et al. [17] report on a mixed-methods study to analyse the service only 18 months after its introduction. They also found
usage, costs and benefits of continuous integration in open- that the introduction of GHA coincides with a decreasing
source projects. They found that CI/CD is widely adopted by adoption rate and an increasing discontinuation rate for other
the most popular projects and that CI/CD usage continues to CI/CD services, especially for Travis. While this study showed
grow. They also provide evidence that CI/CD helps projects that traditional CI/CD services on GitHub are getting replaced
release more often. Vasilescu et al. [18] found quantitative rapidly by GHA, only very few research articles have studied
evidence that usage of certain CI/CD services improves team GHA itself. Kinsman et al. [30] quantitatively analysed the
productivity in GitHub projects, allowing teams to integrate impact of adopting GHA in 3,190 repositories. Their results
more external contributions without a decrease in code quality. indicate that the adoption of GHA increases the number of
Lamba et al. [19] studied the spread of CI/CD and quality rejected pull requests and decreases the number of commits
assurance tools in npm package repositories. They notably in merged pull requests. By manually inspecting 209 issues
show that repositories tend to stick to a given CI/CD tool related to GHA, they concluded that developers have a pos-
once they adopt it. In addition, by investigating differences in itive perception of GHA. Valenzuela-Toledo and Bergel [31]
characteristics between early and late adopters, they found that investigated the use and maintenance of GHA workflows in
social factors play a significant role in tool adoption. Recently, 10 popular GitHub repositories. They manually inspected 222
Soares et al. [20] conducted a systematic literature review on commits related to workflow changes and determined 11 dif-
the impact of CI/CD on software development. By analysing ferent types of workflow modifications. They also uncovered
empirical evidence from 101 papers ranging from 2003 to a number of deficiencies in GHA workflow production and
2019, they found that existing research mostly revealed the maintenance, and call for adequate tooling to support creating,
positive effects of CI/CD on software development, and that editing, refactoring, and debugging GHA workflow files.
further studies are necessary to better understand the trade-
offs between adopting CI/CD and overcoming its inherent III. M ETHODOLOGY
challenges.
A. About GHA
There is a plethora of studies focusing on the use of Travis
in GitHub projects [21]–[29]. Vasilescu et al. [21] empirically To enable GHA on a repository, one has to create one or
explored the use of Travis in 918 GitHub projects. They found more YAML files, each describing a single GHA workflow,
that projects using Travis have more pull requests accepted, and store them in the .github/workflows folder. Fig. 1
 name : Example o f a w o r k f l o w f i l e 100 commits and 100 stars. We obtained 67,870 repositories
on : satisfying these criteria.
push : On 2022-01-24 we locally cloned these repositories
pull request :
schedule : to look for the presence of YAML files in the
− cron : ”0 6 * * 1” .github/workflows folder of their default branch.
jobs : We parsed these files to check whether they define a GHA
test :
name : T e s t p r o j e c t workflow, and if applicable, we extracted the relevant
runs −on : u b u n t u − l a t e s t data about the workflow (e.g., name, events), about the
steps : jobs configured in the workflow (e.g., name, uses: key,
− u s e s : a c t i o n s / checkout@v2
− name : S e t up P y t h o n steps) and about the steps defined in these jobs (e.g.,
u s e s : a c t i o n s / s e t u p −python@v2 name, commands, uses: key). At the end of this process,
with : our dataset covers 67,870 repositories containing 70,278
python − v e r s i o n : 3 . 9
− name : I n s t a l l d e p e n d e n c y workflows, 108,500 jobs, and 567,352 steps. The remainder
run : p i p i n s t a l l p y t e s t of this article presents what we found in these repositories,
− name : E x e c u t e t e s t s workflows, jobs, and steps (including their actions). The
run : p y t e s t
data and code to replicate the analysis are available on
doi.org/10.5281/zenodo.6634682.
Fig. 1. Example of a GHA workflow file.
IV. W HAT A RE THE C HARACTERISTICS OF G IT H UB
R EPOSITORIES U SING GHA W ORKFLOWS ?
shows an example of a GHA workflow file.1 A workflow
defines the set of events (e.g., a push, a pull request, or Not all GitHub repositories make use of GHA. Out of
a scheduled event) that trigger the execution of a set of the 67,870 repositories contained in the dataset, 29,778 of
jobs. A job may reuse an existing workflow through the them (i.e., 43.9%) contain at least one workflow file. Table I
uses: key. Alternatively, a job defines a list of steps that reports on the number and proportion of repositories having a
will be sequentially executed. Steps are the smallest units workflow file, distinguishing repositories based on their main
of work in a workflow. A step can specify through the programming language (as specified in the repository metadata
run: key the commands that will be executed (e.g., pip on GitHub).
install pytest) or it can delegate its task by calling
TABLE I
a predefined action or Dockerfile through the uses: key N UMBER AND PROPORTION OF REPOSITORIES USING GHA WORKFLOWS ,
(e.g., uses: actions/setup-python@v2 or uses: GROUPED BY MAIN PROGRAMMING LANGUAGE .
docker://alpine:3.8). An action is an individual task
repositories using GHA workflows
that can be shared for reuse on a public GitHub repository and language # % % language % repo.
on the GitHub Marketplace.2 Developers can use actions in JavaScript 13,542 19.6% 34.9% 15.9%
steps to avoid having to write explicitly the various commands Python 12,319 17.8% 45.9% 19.0%
TypeScript 6,362 9.2% 58.5% 12.5%
that need to be executed. Actions can access the GitHub API Java 6,105 8.8% 39.2% 8.0%
to interact with repositories (e.g., to create a comment in a C++ 5,701 8.2% 40.9% 7.8%
pull request for test reports), or any third-party API (e.g., to Go 4,988 7.2% 57.2% 9.6%
C 4,314 6.2% 36.1% 5.2%
deploy a new release on PyPI). To obtain more information PHP 4,005 5.8% 48.2% 6.5%
about GHA, the interested reader is invited to consult the C# 3,630 5.3% 34.6% 4.2%
official GitHub documentation3 or Chandrasekara and Herath’s Ruby 2,599 3.8% 50.8% 4.4%
Shell 2,327 3.4% 33.2% 2.6%
book [7]. Swift 1,411 2.4% 34.4% 1.6%
Kotlin 1,150 1.7% 56.9% 2.2%
B. Data Extraction other 694 1.0% 17.7% 0.4%
To conduct an empirical study on the use of GHA in
software development repositories, we need a large collection We observe from the fourth column that the proportion of
of GitHub repositories. The dataset should exclude repositories repositories with GHA workflows varies from one language to
that are used only for experimental or personal reasons, or that another. It ranges from 33.2% (for Shell) to 58.5% (for Type-
show no or little traces of actual software development activ- Script). For more recent languages such as TypeScript, Go,
ity [32]. We relied on the SEART GitHub search engine [33] to Kotlin, and Ruby, the majority of repositories are using GHA
obtain a list of candidate repositories. We queried the tool on workflows. The top three languages (JavaScript, Python and
2022-01-24 to get all non-fork repositories that were created TypeScript) together account for nearly half of all repositories
before 2021, which were still active in 2021, and had at least in the dataset (46.6%, third column) and nearly half of the
1 See
repositories defining a workflow (47.4%, last column).
https://github.com/pandas-dev/pandas/blob/68f763e7/.github/
workflows/code-checks.yml for a more elaborate example of a workflow file.
Since workflows help developers to automate some of
2 https://github.com/marketplace?type=actions the repetitive tasks that are inherently part of the software
3 https://docs.github.com/en/actions development process, we expect larger repositories (e.g., those
 TABLE II flows on a regular basis or workflow_dispatch and
C OMPARISON OF CHARACTERISTICS FOR G IT H UB REPOSITORIES WITH repository_dispatch to manually trigger a workflow.4
AND WITHOUT GHA WORKFLOWS .
Table III reports the 10 most frequent events occurring in
median effect size workflows and the proportion of workflows using them. Since
characteristic with without Cliff’s δ interpretation
repositories can have more than one workflow, we also report
pull requests 124 41 0.384 medium
contributors 20 11 0.277 small on the proportion of repositories having one of their workflows
commits 598 344 0.229 small triggered by these events.
issues 105 59 0.227 small
branches 5 4 0.139 negligible
age (months) 71 77 −0.082 negligible TABLE III
stars 398 334 0.078 negligible T OP 10 EVENT TYPES IN G IT H UB WORKFLOWS AND REPOSITORIES .
size (MB) 5,878 5,099 0.025 negligible
forks 84 80 0.018 negligible event % workflows % repositories
watchers 24 25 −0.013 negligible push 41.8% 63.4%
pull_request 34.1% 56.3%
workflow_dispatch 8.3% 15.4%
schedule 8.1% 16.1%
having more contributors or pull requests) to rely more release 3.0% 6.2%
pull_request_target 1.3% 2.6%
frequently on workflows. We compared the distributions of issues 1.0% 2.0%
several characteristics between GitHub repositories with and repository_dispatch 0.7% 2.0%
without workflows: size in MB, number of pull requests, com- issue_comment 0.6% 1.2%
workflow_run 0.4% 0.8%
mits, issues, contributors, branches, stars, forks and watchers.
We also considered the age of the repositories in months.
The most frequent events, push and pull_request,
A statistical difference was consistently confirmed for all
are used by more than half of the repositories. These events
characteristics (p < 0.004) by Mann-Whitney-U tests [34]
correspond to what is typically monitored by traditional CI/CD
after controlling for family-wise error rate with the Bonferroni-
services and are tightly related to the technical aspects of
Holm method [35].
software development (e.g., test, build and deploy code). The
Table II reports the median value for each considered
next most frequent events are workflow_dispatch and
characteristic, as well as the effect size of the observed
schedule. The former allows to manually trigger a workflow
difference using Cliff’s δ [36] and its interpretation [37]. We
using the GitHub API or user interface, while the latter allows
observe with medium or small effect size that repositories
triggering a workflow at a scheduled time.
with workflows exhibit a higher number of pull requests,
The most frequent events that can be associated with more
contributors, commits, and issues. We also observe, though
social activities are issues and issue_comments. These
with negligible effect size, that repositories with workflows
events, used by 2.0% and 1.2% of the repositories respectively,
tend to have more branches, stars, forks, and a larger size.
react to the creation of an issue or a comment. They can
Repositories with workflows are also younger and have fewer
be used, for example, to welcome newcomers, triage issues
watchers.
or ensuring adherence to the Contributor Licence Agreement
More than 4 out of 10 GitHub repositories use work- (CLA).
flows. Projects mostly written in JavaScript, Python, or In order to gain some insight into the purpose of these
TypeScript account for nearly half of the repositories and workflows, we analysed the most common workflow names.
one third of the repositories using workflows. Repositories Although the name: key of a workflow is optional, nearly all
with GHA workflows tend to have more contributors, pull workflows in our dataset define it (99.1%). We found 23,056
requests, commits, and issues. distinct names out of the 69,618 workflows defining one. There
are 554 names used by at least 10 repositories, and some of
them are frequently used by a large number of repositories,
V. W HICH K INDS OF W ORKFLOWS A RE AUTOMATED ? such as ci (8.2% of the repositories), test(s) (5.1%), build
(3.9%), codeql (3.5%), release (2.9%), and lint (1.0%). The
We found a total of 70,278 workflows in 29,778 repositories,
first name explicitly referring to a “social” purpose is mark
hence an average of 2.4 workflows per repository. Nearly
stale issues and pull requests and is used by 0.5% of the
half of these repositories (49.3%) defined a single workflow.
repositories.
The remaining 50.7% repositories define two (22.6%), three
(11.7%), or four workflows (6.3%), even if we found dozens
Half of the repositories using GHA define two or more
of repositories with 20 or more workflows.
workflows. Workflows are mostly triggered by push and
A workflow can be triggered by one or more events.
pull request events. The large majority of workflows appear
The events that trigger a workflow can be chosen from a
to be used for technical development-related purposes.
large list of events corresponding to the different ongoing
activities within a repository (e.g., commits pushed, pull
requests created or updated, comments created). GHA also 4 The complete list of supported events can be found on https://docs.github.
proposes special events such as schedule to execute work- com/en/actions/using-workflows/events-that-trigger-workflows.
 VI. W HAT A RE THE M OST F REQUENT J OBS IN A step either explicitly lists the commands to be executed
W ORKFLOWS ? (through the run: key) or delegates this task (through the
uses: key) to an action or a Docker image. Table IV reports
A workflow defines one or more jobs that will be executed
on the proportion of steps and repositories in function of their
in parallel (by default). We found 108,500 jobs in the 70,278
step type. For steps relying on the uses: key, we distinguish
workflows in the dataset, hence an average of 1.5 jobs per
between steps referring to an action by means of a local path,
workflow. The large majority of workflows define a single job
a reference to a Docker image, or by specifying the name
(77.8%) or two jobs (10.5%). However, since a repository can
of a repository containing the action. For the latter category,
define multiple workflows, the number of jobs per repository
we distinguish between references to the same repository, to a
is higher (3.6 on average). For instance, 65.5% of the repos-
repository of the same owner, or to another public repository.
itories have two or more jobs defined in the totality of their
workflows, and 21.3% even have 5 or more jobs. TABLE IV
As mentioned in Section III, a job either defines a list of P ROPORTION OF STEPS AND REPOSITORIES W. R . T. STEP TYPE AND
steps that will be executed to achieve its purpose, or can use ACTION TARGET.

(through the uses: key) another workflow from any public step type action target % steps % repositories
repository. In the latter case, the jobs defined in the other work- run: 49.9% 93.5%
flow will be executed. The ability to reuse other workflows in uses: local path 0.8% 2.0%
Docker image 0.1% 1.8%
a job is not commonly exploited by the repositories in our same repository 0.2% 0.4%
dataset: only 823 jobs (0.8%) from 254 distinct repositories same owner 0.7% 4.3%
(0.9%) refer to another workflow. Analysing the origin of the other repository 48.3% 99.3%
reused workflows, we found that the large majority (84.4%)
Around half of the steps (51.1%) use an action. Nearly
have the same owner as the calling job. More specifically, 279
all repositories have at least one step referring to an action.
jobs (33.9%) use a workflow within the same repository and
We observe that these steps mostly refer to another public
416 jobs (50.5%) use a workflow located in another repository
repository (48.3% of the steps and 99.3% of the repositories)
of the same owner. Only 128 jobs (15.5%) use a workflow
and, to a much lower extent, to a repository belonging to the
located in another public repository.
same owner (0.7% of the steps and 4.3% of the repositories).
Similar to Section V, we analysed the names of the jobs
There is little to no use of Docker images (0.1% of the steps).
to gain insight into their purpose. However, only 40.7% of
The other half of the steps (49.9%) run their own local
the jobs define the optional job name field. For those that do
commands. Analysing the content of run: keys, we found
not, we considered their identifier instead (i.e., the key used to
139,501 distinct commands among the 287,868 steps that
define the job). We found 32,265 distinct names, of which 851
define one. There are 134 commands being duplicated in 100+
are used by 10 or more jobs. The most frequent names (> 1%)
steps, of which 10 are duplicated in more than 1,000 steps
were build (15.5%), test (4.2%), analyse (2.2%), lint (2%),
(e.g., npm ci, yarn build, npm test). On average,
release (1.7%), and deploy (1.5%). The first most frequent
steps execute 2.9 command lines (median is 1) for a total
name for a social activity was stale (0.9%), corresponding
of 29.8 command lines per repository (median is 4).
to the task of closing issues or pull requests that have not
Looking at the step names, we found 92,853 distinct names
exhibited any recent activity.
of which 360 are used by at least 100 steps. The most
frequent (> 1%) names are install dependencies (3.7%),
The large majority of workflows define a single job, but
checkout (2.9%), build (2%), run tests (1.4%), test (1.3%),
most repositories have two or more jobs defined in the to-
checkout repository (1.2%), and checkout code (1.1%). When
tality of their workflows. Jobs mostly implement technical
we distinguish names based on the type of the steps using
activities. It is not common practice to reuse workflows in
them, we find that the use of some step names is strongly
jobs.
related to the presence of commands (i.e., runs:) or to the
reuse of an existing action (i.e., uses:). For example, we
found that install dependencies, build, and tests (including
VII. W HAT A RE THE AUTOMATION P RACTICES ?
their variants) are mostly used for steps executing commands
Steps represent the smallest unit of work in a workflow. (≥ 97% of the steps with these names) while checkout (and
They correspond to individual tasks in a job that are sequen- its variants) and setup python|node|php|etc. are mostly used
tially executed to achieve the job goal. For example, in order for steps relying on a reusable action (≥ 99%).
to publish a new release on a package registry such as PyPI, Overall, considering the names being used in at least 100
a job will define steps to (1) checkout the code, (2) setup steps, we found that 47.2% of them are specialized (i.e.,
Python, (3) install dependencies, (4) execute tests, (5) build ≥ 75% steps having that name) towards steps executing
the package, and (6) upload it on PyPI. We found 567,352 commands, and 45.3% towards steps using an action. The re-
steps for 108,500 jobs. On average, there are 5.2 steps per job maining 7.5% names are used indiscriminately for commands
(median is 5), 8.2 steps per workflow (median is 5), and 19.4 and actions, and include terms like deploy, release, and publish
steps per repository (median is 10). among others.
 A job defines five steps on average. Half of the steps are an action on a public repository do not include anything
running their own local commands while the other half use about the action’s purpose or category.5 The only place where
a reusable action. Nearly all repositories rely on actions, this information is (partially) available is on the GitHub
mostly originating from other public repositories. A large Marketplace.6 To publish an action on the Marketplace, one
majority of the most frequent tasks are implemented either has to provide additional metadata, including the primary and
almost exclusively by steps executing commands or almost secondary categories the action belongs to. Even if GitHub
exclusively by an action. allows anyone to upload to the Marketplace an action available
from a public repository, so as to increase its visibility and
VIII. W HICH ACTIONS A RE R EUSED AND H OW ? reuse, not all actions are published on the Marketplace. We
managed to find 917 of the 2,964 actions of our dataset on
Section VII revealed the common practice of steps using
the Marketplace, based on the assumption that the name of
reusable actions. From the 278,122 steps relying on a reusable
the repository where the action is developed corresponds to
action from a public repository, we found 2,964 distinct actions
its unique identifier on the Marketplace. We will discuss the
of which 724 are used at least 10 times. Table V lists the 10
threats related to this assumption in Section XI. Out of these
most frequent actions observed in steps and repositories.
917 actions we found, 752 were correctly mapped, in the sense
that the repository mentioned on the Marketplace for an action
TABLE V
T HE 10 MOST FREQUENT ACTIONS IN STEPS AND REPOSITORIES . does indeed correspond to the repository that is called by the
step. These 752 actions are used in 158,441 steps (i.e., 57%
action % steps % repositories of the steps relying on a public action).
actions/checkout 35.5% 97.8%
actions/cache 7.2% 21.6% Table VI reports on the most frequent (> 1%) categories
actions/setup-node 6.6% 26.3% for these actions, as well as on the proportion of steps and
actions/upload-artifact 5.9% 18.7% repositories using them. Since we did not observe major
actions/setup-python 5.8% 21.0%
actions/setup-java 2.6% 10.0% differences between primary and secondary categories, we
actions/setup-go 2.5% 9.1% report exclusively on the former.
actions/download-artifact 2.1% 6.4%
shivammathur/setup-php 1.3% 5.7%
codecov/codecov-action 1.3% 9.4% TABLE VI
M OST FREQUENT (> 1%) PRIMARY CATEGORIES AND THE PROPORTION
OF ACTIONS , STEPS AND REPOSITORIES USING THEM .
The most frequently used action is actions/checkout, used
by 35.5% of the steps and 97.8% of the repositories. The high action category % actions % steps % repo.
proportion of repositories using it should not be surprising Utilities 23.9% 88.2% 99.4%
Continuous integration 17.3% 4.9% 12.9%
since actions/checkout aims to ease checking out a reposi- Publishing 7.2% 0.7% 2.9%
tory, a necessary first step for executing most of the CI/CD Deployment 6.9% 0.2% 0.6%
tasks. Other frequently used actions are mainly related to the Code quality 6.1% 0.3% 1.0%
Project management 5.2% 0.4% 1.6%
deployment of a specific programming language environment Dependency management 4.4% 1.0% 2.5%
(e.g., setup-node or setup-python). Overall, 24.2% of the Code review 4.1% 0.1% 0.6%
steps use an action of the form setup-*. Testing 3.3% 0.7% 1.5%
Open Source management 3.3% 0.1% 0.5%
For each action in the dataset, we identified its provider (i.e., Container CI 2.3% 2.2% 5.5%
the name of the user or organization owning the repository). Chat 1.9% 0.3% 0.5%
Considering the top 10 of Table V, we observe that 8 of Reporting 1.7% 0.1% 0.4%
Community 1.6% 0.0% 0.1%
the most frequent actions are officially proposed by GitHub Security 1.6% 0.1% 0.3%
(distributed by the actions provider) and used by 71.2% of unspecified 4.9% 0.2% 0.6%
the steps. The first third-party provider is shivammathur with
the setup-php action. Table VI reveals that most categories correspond to technical
In the entire dataset we found 2,037 different providers, but aspects of software development. The primary categories con-
the majority of the actions used by steps come from just a taining the highest proportion of actions (as well as steps and
few providers. There are only 8 providers whose actions are repositories) are Utilities (23.9% of actions) and Continuous
called by 1,000+ steps, and 103 providers whose actions are integration (13.3% of actions). These two “catch-all” cate-
called by 100+ steps. The actions provider alone distributes gories include very diverse actions to check out repositories,
24 actions that account for 71.7% (i.e., 199,549) of the steps set up environments, create releases, etc. The third most widely
calling an action. The second most frequent called provider used category is Container CI even if only 2.3% of the actions
is docker (7 actions and 3.8% steps), followed by github (9 are part of this category. It includes actions to log in to a
actions and 3.1% steps) mostly for their CodeQL actions, and Docker registry, to run a Dockerfile, or to set up specific
by shivammathur (2 actions and 1.4% steps) for its setup- services (e.g., a MySQL database or a Redis instance). A
php action. 5 https://docs.github.com/en/actions/creating-actions/
We sought to find out more about the purpose of these metadata-syntax-for-github-actions
actions. However, the metadata that are required to distribute 6 https://github.com/marketplace?type=actions
few categories include socially related actions, such as Project full name of the repository) are referred to with a branch or
management and Code review. These two categories notably a tag name or, to a much lower extent, with a version tag.
propose actions to create or triage issues, to detect and lock The opposite can be observed for actions from repositories of
stale issues, or to add specific comments in existing issues or the same owner: they are mostly referred to using a version
pull requests. Other social categories include Chat, Reporting tag and, to a lower extent, with a branch or a tag name. The
and Community. The two former ones propose actions to notify situation is even more marked for actions coming from other
on Slack, Discord, IRC, etc. while the latter includes actions public repositories: 9 out of 10 steps in this large subset use
to ensure CLA adherence or to welcome newcomers. a version tag to refer to an action.
Assuming adherence to semantic versioning, GitHub recom-
Using actions in steps is a common practice. A few mends specifying the version tag by including only its major
actions concentrate most of the reuse, and most of them component (e.g., @v2 instead of @v2.1.3) in order to receive
are distributed by GitHub and belong to the Utilities or critical fixes and security patches while still maintaining com-
Continuous integration categories. patibility. This recommendation seems to be widely followed,
since 89.9% of the version tags used to refer to an action
IX. W HICH VERSIONING PRACTICES ARE BEING USED ? include only a major component (e.g., @v2), 0.9% a minor
component (e.g., @v2.1) and 9.2% a patch component (e.g.,
When a step uses a predefined action, in addition to
@v2.1.3).
specifying the name of the repository hosting the action it
Referring to a version using only its major component
can optionally specify which version of the action should be
has clear advantages if we assume adherence to semantic
executed, by means of a git reference. This reference can
versioning [10]. However, since versions of an action are
be a commit SHA (e.g., @753c60e0), a branch name (e.g.,
identified using git tags, this means that the action maintainer
@main), or a git tag (e.g., @v2.1.3). If no reference is
must move some of these tags each time a new version of
specified, the latest version of the action is executed. For
the action is released (e.g., moving @v2 and @v2.1 from
security and stability reasons, GitHub recommends pinning
@v2.1.3 to @v2.1.4 when version 2.1.4 is released).
an action to a full-length commit SHA. While this is the
Unless automated, this introduces an additional burden on the
most secure option, specifying a tag is more convenient since
maintainers. Forgetting to update these tags when a minor
GitHub’s release management support advocates the creation
or a patch update is released for an action implies that the
of a tag corresponding to the version number of a new release.
workflows that depend on it do not automatically benefit from
This makes it easy for a step using an action to specify which
the bug and security fixes provided by the update.
version of the action should be executed.
Moreover, we found that 16.4% of the major components
Table VII shows the number and proportion of steps in func-
used in version tags to refer to a reusable action do not target
tion of the git reference type used for actions, and the origin of
the highest major release of the corresponding action, i.e.,
these actions. We distinguish between no git reference (none),
they are relying on a lower major train. As such, dependent
a reference to a specific commit SHA, a reference to a version
workflows do not benefit from the latest bug and security fixes
tag and a reference to a branch or another tag (branch/tag).
of the action unless these changes are backported to lower
major trains as well [11].
TABLE VII
N UMBER AND PROPORTION OF STEPS W. R . T. ACTION TARGET AND GIT
REFERENCE TYPE . Around 9 out of 10 steps use a version tag when referring
to an action. Most of the version tags only specify the
steps
action origin reference type # % target % all major component of the targeted version, and one sixth of
local path none 4,397 100.0% 1.5% them refer to a lower major train.
branch/tag 2 0.0% 0.0%
same repository commit SHA 2 0.2% 0.0%
version tag 123 12.8% 0.0% X. D ISCUSSION
branch/tag 833 87.0% 0.3%
same owner none 1 0.0% 0.0% On the Popularity of GHA
commit SHA 123 3.0% 0.0%
version tag 2,641 63.9% 0.9% The growing popularity of GHA is undeniable. The findings
branch/tag 1,368 33.1% 0.5% from Section IV revealed that a significant proportion of repos-
other repository none 2 0.0% 0.0% itories rely on GHA, regardless of their main programming
commit SHA 4,601 1.7% 1.6%
version tag 258,647 93.0% 89.9%
language. These findings are confirmed by Golzadeh et al. [8]
branch/tag 14,872 5.3% 5.2% who showed that in 18 months, GHA has become the dominant
CI/CD service on GitHub. They attributed this popularity to
We observe major differences in the git references used to a combination of factors, including the deep integration of
refer to an action in function of the origin of the action. For GHA with GitHub, the ease of use, the speed, its free tier for
instance, actions on a local path are nearly exclusively referred open source projects, the availability of a large marketplace of
to without any specific reference, while the vast majority of reusable actions, and the support for many different operating
actions within the same repository (but referred to with the systems.
 We contacted five developers using GHA, who confirmed are tightly coupled to GitHub and migrating away from GHA
that these aspects played a major role in their decision to adopt could take a lot of effort. Only time will tell how this situation
GHA: “we migrated to GHA because it was directly integrated will affect the future of OSS development.
with GitHub and had all the same support across Linux, Mac,
Windows that we were looking for”, “the integration of GHA The GHA Ecosystem
into GitHub itself would be the reason of its popularity”, “you The findings reported in Section VIII have revealed that the
have this marketplace of GHA and you can reuse components ability to reuse actions in GHA has lead to a new emerging
from there, and it’s pretty straightforward and easy”, “People ecosystem that is worthy of being studied in its own right.
are using GHA for the simplicity, the speed, because it’s free, The reuse of actions in steps is a common practice that
because you have access to macOS and Windows runners [...]” allows developers to easily integrate (sometimes complex)
and “I’m just looking for something that I can set up with the tasks without having to code them. The ability for a CI/CD
absolute least friction, and in 99 times out of 100 that’s going tool to provide reusable components is not something specific
to be GitHub Actions.” to GHA, since many competing CI/CD tools are providing
Travis, the previous dominant CI/CD tool on GitHub, has similar mechanisms. For instance, CircleCI introduced orbs
been encountering many issues in recent years, such as re- in 2018, one year before GHA7 ; and Jenkins has been
strictions on their free plan, decreased reliability, poor Docker providing community-contributed plugins for years through
support, long build times and lack of innovation [29]. This plugins.jenkins.io. However, at the time of writing, GHA offers
has further contributed to the success of GHA, as shown by more than 12,000 reusable components on its Marketplace,
Golzadeh et al. [8] who observed that most migrations between about 4 times as many as CircleCI orbs and more than 6 times
CI services go from Travis towards GHA. This is confirmed higher than Jenkins plugins, and there are likely thousands
by developers that we have talked to: “Travis was really a more available actions in public GitHub repositories.
game changer 10 years ago, [but] did not really evolve [or] Given this wide availability of reusable actions, GHA should
bring any innovation, [...] so we moved away from Travis” be studied as a “software ecosystem” that bears many similari-
and “since Travis has been bought by a bigger company, it ties to ecosystems of reusable software libraries distributed by
pretty much became useless and we moved away from Travis package managers, e.g., npm, Cargo, RubyGems, Maven, PyPI
in all our projects to GHA”. and the like. The parallel with such packaging ecosystems
Another important milestone for GHA relates to GitHub is quite obvious: automated workflows, as software clients,
being acquired by Microsoft in 2018. A developer pointed out frequently express dependencies towards reusable actions that
that “because Microsoft is now the mother company of GitHub, can exist in different versions or releases. Packaging ecosys-
you see a lot of investment taking place in GitHub. The tems are known to suffer from a large number of issues in
features being added are actually coming from a lot of areas the reusable artefacts they distribute, and each of these has
from within Microsoft”. Another developer signals that GHA been an active topic of investigation. Well-known challenges
is one of these features: “Just look at the free GitHub Actions include obsolescence or outdatedness [38], [39], dependency
nowadays. That’s a lot of money they are spending for free so issues [9], [40], [41], breaking changes [10], [14], security
everybody can build stuff.” Most developers we talked to are vulnerabilities [2], [12], and so on.
positively surprised by Microsoft’s changed attitude regarding The GHA ecosystem is likely to suffer from very similar
open source, for example: “These last years, Microsoft is really issues, and these issues will continue to become more impor-
doing huge changes internally to make their reputation change tant and more impactful, as the number of reusable actions
about open source. I think Microsoft is changing its point of continues to grow at a rapid pace. Therefore, there is an
view on open source and I think it’s for the greater good of urgent need for further research as well as appropriate tooling
open source developers.” Despite these positive signs, having a to support developers of reusable actions and workflows,
private company owning the dominant platform for distributing especially since these issues may not only affect the workflows
open source software remains risky since it implies a de but also wide ranges of projects that use them. A first step
facto monopoly. At some point in the future, Microsoft might in this direction is GitHub’s built-in Dependabot dependency
change its strategy to try to make profit out of the situation: monitoring service that has recently (January 2022) started to
“There are intangible benefits that Microsoft gets and we’ll support actions.
see if changes happen in the next few years to where GitHub
makes changes to be more profitable that don’t necessarily Security Concerns
serve the free software folks.” and “I have mixed feelings about While security concerns are important to deal with for any
it, on the one hand, it really is convenient having everything software project, it is known that the attack surface of security
integrated at one place. On the other hand, how much do we issues has become several orders of magnitude higher due to
really want to invest all of open source in a single company?” the widespread dependence on reusable software libraries that
This may pose difficulties to many OSS projects hosted on can have deep transitive dependency chains [2], [12], [42],
GitHub, as they will be facing some kind of vendor lock-in. [43]. The reliance on CI/CD tools to automate development
Although GHA supports self-hosted runners, many aspects of
GHA (including the workflow syntax or the reusable actions) 7 https://circleci.com/blog/announcing-orbs-technology-partner-program/
activities in software projects continues to increase this attack collision for a valid Git object payload.”12 Unfortunately, we
surface considerably. Two developers that we talked to confirm observed in Section IX that this recommendation is not really
that this constitutes a major security concern: “In house we followed in practice. The very large majority of steps relying
have two GitHub and GitLab CI systems that we need to on a reusable action use a version tag (rather than a commit
maintain. It’s a major security concern because, from that SHA) when referring to an action, and most of the version
automation you can basically run any code on it.” and “CI/CD tags only specify the major component of the targeted version,
are very important to secure the build chain, that we should implying that any new compromised version of an action will
focus in the future into the aspect of securing the toolchain.” be automatically adopted by most dependent workflows.
For GHA in particular, multiple examples of security issues We are not aware of any publicly available quantitative
with potentially disastrous consequences have been reported, analysis having reported on the impact of reusable actions on
such as manipulating pull requests to steal arbitrary secrets,8 security vulnerabilities in software projects. Such empirical
injecting arbitrary code with workflow commands9 or bypass- research is urgently needed, in order to quantify the extent of
ing code reviews to push unreviewed code.10 A developer we the problem. That would constitute a first step towards trying
talked to specifically mentioned “You can open a pull request, to reduce the attack surface of vulnerabilities related to the
build the package, and then we will deliver it. And when you use of GHA.
do that from a pull request, there are issues with the security
considerations about the credentials, because anyone could XI. T HREATS TO VALIDITY
modify workflows or inject code, get access to the credentials We follow the structure recommended by Wohlin et al. [44]
and then access to the upload process. [...] When it’s open to discuss the main threats to validity of our research.
to everyone, you need to be careful. [...] What we do with Threats to construct validity concern the relation between
GitHub Actions, we build in one workflow, and we have a the theory behind the experiment and the observed findings.
second workflow which does the upload or the shipping or the They can be mainly due to imprecisions in the measurements
delivery with credentials which are not exposed in the build we performed. We detected the use of automated workflows in
pipeline for the pull request.” GitHub repositories on the basis of the presence of a YAML
Relying on reusable actions from third-party repositories or file in the .github/workflows folder. This approach leads
even from the Marketplace further increases the vulnerability to an overestimation since the presence of a YAML file
attack surface. Since a job executes its commands within does not necessarily imply that the corresponding workflow is
a runner shared with other jobs from the same workflow, actually being triggered and used. However, we are confident
individual jobs in a workflow can compromise other jobs they that such workflows are indeed used in the vast majority of
interact with. For example, a job could query the environment cases as there is little to no practical reason to keep workflows
variables used by a later job, write files to a shared directory in .github/workflows without using them.
that a later job processes, or even more directly interact with Another threat to construct validity stems from how we
the Docker socket and inspect other running containers and identified the git reference type used to reference public
execute commands in them.11 actions in steps. We relied on a heuristic to detect whether
Despite these risks, Section VIII revealed that it is common the git references correspond to a version number (via a
practice to rely on reusable actions. As a general rule of thumb, regular expression), to a commit SHA (based on the git
GitHub recommends to only use actions whose creator can be reference length) or to a tag or branch name (all remaining
trusted. However, even actions from trusted creators can be git references). This naive heuristic seemed to be effective
compromised. For example, an attacker having gained write since we did not find any false positives after having manually
access to the repository of a trusted action can change its checked 104 distinct randomly selected cases.
code and commands in order to compromise the repositories Another threat to validity stems from how we interpreted
depending on this action. the identifier labels for workflows and jobs. We relied on these
To further reduce the risks of using compromised actions, labels in Sections V and VI to understand the purpose of the
GitHub suggests referring to reusable actions through their workflows and jobs being defined. While some workflows and
unique commit SHA, to avoid unintentionally using a com- jobs may have a label that does not reflect their purpose, we
prised action that may have its code changed and may be believe these cases to be rare, as the goal of a label is to
able to steal secrets: “Pinning to a particular SHA helps provide an indication of the nature of the workflow or job,
mitigate the risk of a bad actor adding a backdoor to the and not to mislead practitioners or researchers.
action’s repository, as they would need to generate a SHA-1 Threats to internal validity concern choices and factors
internal to the study that could influence the observations
8 https://blog.teddykatz.com/2021/03/17/github-actions-write-access.html
we made. One of such choices relates to how we mapped
9 https://packetstormsecurity.com/files/159794/ actions used in steps with the ones distributed on the GitHub
GitHub-Widespread-Injection.html Marketplace. We relied on the assumption that the name of
10 https://medium.com/cider-sec/bypassing-required-reviews-6e1b29135cc7
11 https://docs.github.com/en/actions/security-guides/ 12 https://docs.github.com/en/actions/security-guides/
security-hardening-for-github-actions#using-third-party-actions security-hardening-for-github-actions
the repository where an action is developed corresponds to GHA workflows and reusable actions in GitHub repositories.
its unique identifier on the Marketplace. This led both to Nevertheless, we only scratched the surface of the emerging
false positives (e.g., action myci-actions/checkout is iden- GHA ecosystem. More in-depth empirical studies remain
tified as checkout while checkout on the Marketplace is required to provide a comprehensive understanding of the
provided by actions/checkout) and false negatives (e.g., GHA ecosystem. Such studies should focus on assessing, in
action actions/setup-node is identified as setup-node-js- the context of the GHA ecosystem, the prevalence and impact
environment on the Marketplace). False positives were ad- of the well-known challenges and issues encountered by more
dressed by comparing the repository referred from the calling traditional software ecosystems. Indeed, in this new context,
step with the repository listed on the action page on the Mar- those issues may not only affect the GHA workflows but also
ketplace. We were unable to address the false negatives, as this a wide range of projects using them. Additional work is also
would require extracting all actions from the Marketplace in required to study the security concerns related to the reuse of
order to obtain their development repositories. Unfortunately, actions, as well as how this ecosystem is evolving over time,
GitHub does not provide a complete list of actions in the and how it compares to similar and related CI/CD ecosystems
Marketplace, nor an API to obtain them. We remain confident (such as the ecosystem of orbs on CircleCI, of plugins on
that the findings of Section VIII are representative, since we Jenkins, or the Infrastructure as Code ecosystem of roles in
managed to map correctly 727 actions, accounting for 57% of RedHat Ansible [45]).
the steps relying on an action. As future work, we plan to conduct a complementary
Threats to conclusion validity concern the degree to which qualitative analysis by interviewing experienced practitioners
the conclusions derived from our analysis are reasonable. to identify and understand the main motivations for adopting
Since our conclusions are mostly based on quantitative ob- and using GHA and the perceived benefits and drawbacks of
servations, they are unlikely to be affected by such threats. doing so. Such qualitative insights aim to complement the
quantitative results of this paper in order to further increase
Threats to external validity concern whether the results can
the understanding of the emerging GHA ecosystem and its
be generalized outside the scope of this study. One such threat
implications on collaborative OSS development in GitHub.
was our decision to study active repositories having at least
Specifically related to the research question of which actions
100 stars and 100 commits, aiming at excluding abandoned,
are reused and how, we plan to study the versioning practices
personal or experimental repositories that do not necessarily
followed by actions and by the workflows relying on them,
correspond to software development [32]. This implies that
considering how frequently actions and workflows are updated,
we have no insight into the use of GHA in smaller or less
and to which extent workflows stay up-to-date with respect to
active repositories, and it could be the case that GHA is used
the actions they are using.
for different purposes in those repositories (e.g., to publish
GitHub pages or to compile LATEX files).
ACKNOWLEDGMENT
XII. C ONCLUSION
This research is supported by the ARC-21/25 UMONS3
GHA has become the dominant CI/CD on GitHub, only Action de Recherche Concertée financée par le Ministère
18 months after its introduction. In order to get a deeper de la Communauté française – Direction générale de
insight into the GHA ecosystem, we have conducted a quanti- l’Enseignement non obligatoire et de la Recherche scientifique,
tative study of 29,778 GitHub repositories containing 70,278 as well as by the Fonds de la Recherche Scientifique -
GHA workflows. We characterised these repositories and their FNRS under grant numbers O.0157.18F-RG43, T.0017.18 and
workflows, in terms of which jobs, steps and reusable actions T.0022.13.
were used and how. We observed that workflows tend to be
used in the more active GitHub projects (more contributors, R EFERENCES
pull requests, commits and issues). These workflows and their
jobs are primarily used for development purposes, and mostly [1] J. M. Costa, M. Cataldo, and C. R. de Souza, “The scale and evolution
of coordination needs in large-scale distributed projects: implications for
triggered by push or pull request events. About half of all the future generation of collaborative tools,” in SIGCHI Conference on
steps in jobs rely on reusable actions, mostly originating Human Factors in Computing Systems, 2011, pp. 3151–3160.
from public repositories. Most of the actions being reused [2] A. Decan, T. Mens, and E. Constantinou, “On the impact of security
vulnerabilities in the npm package dependency network,” in 15th inter-
are provided by GitHub itself, and their primary purpose is national conference on mining software repositories, 2018, pp. 181–191.
for continuous integration and other utilities. Actions appear [3] A. Decan, T. Mens, and M. Claes, “An empirical comparison of depen-
to be reused by adhering to some sort of semantic versioning. dency issues in OSS packaging ecosystems,” in International Conference
The reuse of actions can be problematic, as it has the potential on Software Analysis, Evolution and Reengineering (SANER). IEEE,
2017.
of increasing the attack surface of security issues by several [4] G. Avelino, E. Constantinou, M. T. Valente, and A. Serebrenik, “On the
orders of magnitude. abandonment and survival of open source projects: An empirical investi-
This article is the first to have provided a large-scale quan- gation,” in International Symposium on Empirical Software Engineering
and Measurement (ESEM). IEEE, 2019, pp. 1–12.
titative assessment of GitHub repository workflows relying on [5] GitHub, “The 2021 state of the octoverse - community report,” 2021.
GitHub Actions. We have quantified the widespread use of [Online]. Available: octoverse.github.com
 [6] L. Dabbish, C. Stuart, J. Tsay, and J. Herbsleb, “Social coding in Conference on Software Maintenance and Evolution (ICSME), 2019, pp.
GitHub: Transparency and collaboration in an open software repository,” 291–295.
in International Conference on Computer Supported Cooperative Work [26] M. Beller, G. Gousios, and A. Zaidman, “Oops, my tests broke the build:
(CSCW). ACM, 2012, pp. 1277–1286. An explorative analysis of Travis CI with GitHub,” in International
[7] C. Chandrasekara and P. Herath, Hands-on GitHub Actions: Implement Conference on Mining Software Repositories (MSR), 2017, pp. 356–367.
CI/CD with GitHub Action Workflows for Your Applications. Apress, [27] Y. Zhao, A. Serebrenik, Y. Zhou, V. Filkov, and B. Vasilescu, “The
2021. impact of continuous integration on other software development prac-
[8] M. Golzadeh, A. Decan, and T. Mens, “On the rise and fall of CI services tices: A large-scale empirical study,” in International Conference on
in GitHub,” in 29th IEEE International Conference on Software Analysis, Automated Software Engineering (ASE), 2017, pp. 60–71.
Evolution and Reengineering (SANER). IEEE, 2021. [28] Y. Yu, H. Wang, V. Filkov, P. Devanbu, and B. Vasilescu, “Wait for it:
[9] A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of Determinants of pull request evaluation latency on GitHub,” in Working
dependency network evolution in seven software packaging ecosystems,” Conference on Mining Software Repositories (MSR), 2015, pp. 367–371.
Empirical Software Engineering, vol. 24, no. 1, pp. 381–416, 2019. [29] D. G. Widder, M. Hilton, C. Kästner, and B. Vasilescu, “A conceptual
[10] A. Decan and T. Mens, “What do package dependencies tell us about replication of continuous integration pain points in the context of Travis
semantic versioning?” IEEE Transactions on Software Engineering, CI,” in Joint Meeting on European Software Engineering Conference and
vol. 47, no. 6, pp. 1226–1240, 2019. Symposium on the Foundations of Software Engineering (ESEC/FSE).
[11] A. Decan, T. Mens, A. Zerouali, and C. De Roover, “Back to the past– ACM, 2019, pp. 647–658.
analysing backporting practices in package dependency networks,” IEEE [30] T. Kinsman, M. Wessel, M. A. Gerosa, and C. Treude, “How do
Transactions on Software Engineering, 2021. software developers use GitHub Actions to automate their workflows?”
[12] M. Zimmermann, C.-A. Staicu, C. Tenny, and M. Pradel, “Small world in International Conference on Mining Software Repositories (MSR),
with high risks: A study of security threats in the npm ecosystem,” in 2021.
28th USENIX Security Symposium, 2019, pp. 995–1010. [31] P. Valenzuela-Toledo and A. Bergel, “Evolution of GitHub Action work-
[13] M. Valiev, B. Vasilescu, and J. Herbsleb, “Ecosystem-level determinants flows,” in 29th IEEE International Conference on Software Analysis,
of sustained activity in open-source projects: A case study of the Evolution and Reengineering (SANER). IEEE, 2022.
PyPI ecosystem,” in Joint Meeting on European Software Engineering [32] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German,
Conference and Symposium on the Foundations of Software Engineering and D. Damian, “The promises and perils of mining GitHub,” in Inter-
(ESEC/FSE). ACM, 2018, pp. 644–655. national Conference on Mining Software Repositories (MSR). ACM,
[14] J. Dietrich, D. Pearce, J. Stringer, A. Tahir, and K. Blincoe, “Dependency 2014, pp. 92–101.
versioning in the wild,” in 16th International Conference on Mining [33] O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in GitHub for
Software Repositories (MSR). IEEE, 2019, pp. 349–359. MSR studies,” in 18th IEEE/ACM International Conference on Mining
[15] M. Fowler and M. Foemmel, “Continuous Integration,” https: Software Repositories, MSR 2021. IEEE, 2021, pp. 560–564.
//martinfowler.com/articles/originalContinuousIntegration.html, Septem- [34] H. B. Mann and D. R. Whitney, “On a test of whether one of two
ber 2000, [Online; accessed 3 January 2022]. random variables is stochastically larger than the other,” Ann. Math.
[16] O. Elazhary, C. Werner, Z. S. Li, D. Lowlind, N. A. Ernst, and Statist., vol. 18, no. 1, pp. 50–60, 03 1947.
M.-A. Storey, “Uncovering the benefits and challenges of continuous [35] S. Holm, “A simple sequentially rejective multiple test procedure,”
integration practices,” IEEE Transactions on Software Engineering, pp. Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979.
1–1, 2021. [Online]. Available: http://www.jstor.org/stable/4615733
[17] M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, [36] N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal
costs, and benefits of continuous integration in open-source projects,” questions,” Psychological bulletin, vol. 114, no. 3, p. 494, 1993.
in International Conference on Automated Software Engineering (ASE). [37] J. Romano, J. D. Kromrey, J. Coraggio, J. Skowronek, and L. Devine,
IEEE, 2016, pp. 426–437. “Exploring methods for evaluating group differences on the NSSE
[18] B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov, “Quality and and other surveys: Are the t-test and Cohen’s d indices the most
productivity outcomes relating to continuous integration in GitHub,” in appropriate choices?” in Annual Meeting of the Southern Association
Joint Meeting on Foundations of Software Engineering (FSE), 2015, pp. for Institutional Research, 2006.
805–816. [38] A. Decan, T. Mens, and E. Constantinou, “On the evolution of technical
[19] H. Lamba, A. Trockman, D. Armanios, C. Kästner, H. Miller, and lag in the npm package dependency network,” in 2018 IEEE Interna-
B. Vasilescu, “Heard it through the Gitvine: An empirical study of tool tional Conference on Software Maintenance and Evolution (ICSME).
diffusion across the npm ecosystem,” in Joint Meeting on European IEEE, 2018, pp. 404–414.
Software Engineering Conference and Symposium on the Foundations [39] F. Cogo, G. Oliva, and A. E. Hassan, “Deprecation of packages and re-
of Software Engineering (ESEC/FSE). ACM, 2020, pp. 505–517. leases in software ecosystems: A case study on npm,” IEEE Transactions
[20] E. Soares, G. Sizilio, J. Santos, D. Alencar, and U. Kulesza, “The on Software Engineering, 2021.
effects of continuous integration on software development: a systematic [40] R. G. Kula, D. M. German, A. Ouni, T. Ishio, and K. Inoue, “Do develop-
literature review,” Empirical Software Engineering, 2022. ers update their library dependencies?” Empirical Software Engineering,
[21] B. Vasilescu, S. van Schuylenburg, J. Wulms, A. Serebrenik, and M. G. vol. 23, no. 1, pp. 384–417, 2018.
van den Brand, “Continuous integration in a social-coding world: Em- [41] C. Soto-Valero, N. Harrand, M. Monperrus, and B. Baudry, “A
pirical evidence from GitHub,” in International Conference on Software comprehensive study of bloated dependencies in the Maven ecosystem,”
Maintenance and Evolution (ICSME). IEEE, 2014, pp. 401–405. Empirical Software Engineering, vol. 26, no. 3, p. 45, 2021. [Online].
[22] F. Zampetti, S. Geremia, G. Bavota, and M. Di Penta, “CI/CD pipelines Available: https://doi.org/10.1007/s10664-020-09914-8
evolution and restructuring: A qualitative and quantitative study,” in [42] M. Alfadel, D. E. Costa, E. Shihab, and E. Shihab, “Empirical analysis of
International Conference on Software Maintenance and Evolution (IC- security vulnerabilities in Python packages,” in International Conference
SME), 2021. on Software Analysis, Evolution and Reengineering (SANER), 2021.
[23] N. Cassee, B. Vasilescu, and A. Serebrenik, “The silent helper: The [43] J. Düsing and B. Hermann, “Analyzing the direct and transitive impact
impact of continuous integration on code reviews,” in International Con- of vulnerabilities onto different artifact repositories,” 2021.
ference on Software Analysis, Evolution and Reengineering (SANER), [44] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and
2020, pp. 423–434. A. Wesslén, Experimentation in Software Engineering. Springer, 2012.
[24] K. Gallaba and S. McIntosh, “Use and misuse of continuous integration [45] R. Opdebeeck, A. Zerouali, C. Velázquez-Rodrı́guez, and C. De Roover,
features: An empirical study of projects that (mis)use Travis CI,” IEEE “On the practice of semantic versioning for Ansible Galaxy roles: An
Transactions on Software Engineering, vol. 46, no. 1, pp. 33–50, 2020. empirical study and a change classification model,” Journal of Systems
[25] T. Durieux, R. Abreu, M. Monperrus, T. F. Bissyandé, and L. Cruz, “An and Software, vol. 182, 2021.
analysis of 35+ million jobs of Travis CI,” in 2019 IEEE International