The Developers’ Design Thinking Toolbox in Hackathons: A Study

on the Recurring Design Methods in Software Development

Marathons

Kiev Gamaa , George Valençab , Pedro Alessioa , Rafael Formigaa , André Nevesa ,
Nycolas Lacerdab
arXiv:2206.04744v1 [cs.SE] 9 Jun 2022

a b
Universidade Federal de Pernambuco (UFPE), Recife, Brazil; Universidade Federal Rural
de Pernambuco (UFRPE), Recife, Brazil

Published in the International Journal of Human-Computer Interaction

This is the author’s pre-print version of the article. Please find the final version at:
https://doi.org/10.1080/10447318.2022.2075601

ABSTRACT
Hackathons are time-bounded collaborative events that typically take between 1 to
3 days of intense teamwork to build prototypes usually in the form of software,
aiming to specific challenges proposed by the organizers. These events became a
widespread practice in the IT industry, universities and many other scenarios, as a
result of a growing open-innovation trend in the last decade. Since the main deliv-
erable of these events is a demonstrable version of an idea, such as early hardware
or software prototypes, the short time frame requires participants to quickly under-
stand the proposed challenge or even identify issues related to a given domain. To
create solutions, teams follow an ad-hoc but effective design approach, that many
times seems informal since the background of the participants is rather centered
on technical aspects (e.g., web and mobile programming) and does not involve any
training in Design Thinking.
To understand this creative process, we conducted a set of 37 interviews with
people from 16 countries, consisting of 32 hackathons winners and 5 hackathon
organizers. We aimed to identify the design practices (i.e., processes and recurring
design methods) that are applied by winners in these types of events. Also, we
conducted a focus group with 8 people experienced in hackathons (participants and
organizers) to discuss our findings. Our analysis revealed that although hackathon
winners with IT background have no formal training on Design Thinking, they
are aware of many design methods and typically follow a sequence of phases that
involve divergent and convergent thinking to explore the problem space and propose
alternatives in a solution space, which is the rationale behind Design Thinking. We
derived a set of recommendations based on design strategies that seem to lead to
successful hackathon participation. These recommendations can also be useful to
organizers who intend to enhance the experience of newcomers in hackathons.

KEYWORDS
hackathons; design thinking; design process; computer supported cooperative work

1. Introduction

Hackathons are collaborative events that engage people in small teams to produce
software in a limited amount of time, typically in a non-stop format lasting from 1

CONTACT Kiev Gama. Email: [email protected] ORCID https://orcid.org/0000-0003-1508-6196

to 3 days (Komssi et al., 2014). This type of event is a growing phenomenon that
emerged as a way to foster innovation and became part of the routine in tech com-
panies and universities (Pe-Than et al., 2018; Warner and Guo, 2017). Hackathons
became popular worldwide in multiple contexts, being adopted for diverse purposes,
such as product development (Pe-Than et al., 2018; Saravi et al., 2018), innovation
(Angelidis et al., 2016; Rosell et al., 2014), recruitment (Richard et al., 2015; Tapia-
González, 2020), civic issues (i.e., civic hacking) (Johnson and Robinson, 2014; Lodato
and Disalvo, 2015), informal learning (Nandi and Mandernach, 2016; Warner and Guo,
2017), higher education (Gama et al., 2018; Porras et al., 2018) and domain-specific
skills training (Silver et al., 2016; Wyngaard et al., 2017).
The main deliverable in a hackathon is a demonstrable version of an idea (Komssi
et al., 2014), often in the form of a fully functioning software prototype (Pe-Than
et al., 2018). Hackathons can be classified as tech-centric, much focused on exploring
technologies and APIs, or focus-centric, when they are more oriented to a social issue
or business objective (Briscoe, 2014). In this latter format, which is the preferred one
in many disciplines (Angelidis et al., 2016; Wyngaard et al., 2017; Zapico et al., 2013)
and in innovation circles (Avalos et al., 2017; Flores et al., 2018; Rosell et al., 2014),
participants have to rapidly understand the context of the specific issue or challenge
to be tackled. There is a very short amount of time to speculate about the market
or user needs and propose innovative solutions addressing the event goals. However,
developers who participate in these events do not arrive having any previous training
on the design skills necessary to conceive such solutions.
The prevalent target audience of hackathons, although many times multidisciplinary,
is typically formed by a vast majority of people with technical backgrounds like soft-
ware developers and STEM students (Briscoe, 2014; Warner and Guo, 2017; Zapico
et al., 2013). The typical deductive-rationalist thinking style of IT engineers leads to
an approach of having a given problem and – based on logic – deducing the right solu-
tion. It differs from a Design thinking style, which explores a problem-solution space
more openly and brings a multi-perspective comprehension that helps to deal with the
ambiguity of problems (Lindberg et al., 2011). In that sense, hackathons fit the lat-
ter style by employing an ad-hoc design approach, being more organic and oscillating
from improvisation to specificity in the creation process of technical design artifacts
(Lodato and Disalvo, 2015). Such an approach is aligned with the Design discipline
as a whole, in which much design activity (especially what concerns conceptual de-
sign) is unplanned, intuitive and ad-hoc (Cross, 2011). Hackathons are a venue where
participants can explore this creative approach to develop or enhance problem-solving
skills (Nandi and Mandernach, 2016; Taylor and Clarke, 2018).
Our experience in the organization of over 30 hackathons led us to observe a pattern
in the way successful teams (i.e., winning teams) tackle the challenges. It fits the Design
Thinking style, which contrasts to the deductive-rationalist thinking style of average
teams that do not think as creatively (i.e., ”outside of the box”) and does not have
such high profile results. In these hackathons, although design techniques or methods
(e.g., brainstorming, personas) were rarely presented to participants, their strategy
rather followed many Design Thinking (Brown, 2009; Cross, 2011; Pe-Than et al.,
2018) principles towards a more user-centered and innovative approach to problem-
solving, even though their educational background on STEM (Science, Technology,
Engineering and Math) does not involve the learning of such skills. In hackathons, the
radical collocation of participating teams helps quickly advance technical work (Trainer
et al., 2016). The way of thinking design might be affected by group behavior and it can
also influence the thinking of other members of that group (Lawson, 2006). Therefore,

2
we believe that the way those successful teams intensely collaborate and think design
as a group helps to spread knowledge on Design Thinking. These developers share a
sort of Design Thinking toolbox that was not formally taught to them, but rather
organically acquired through a learn-by-doing approach.
Very little has been reported about details on the usage of design methods or tech-
niques put in practice by participants during hackathons. Many websites and blogs
report on hackathons organization1 provide general-purpose guidelines2 for properly
planning such events but without much detail on the design processes or methods in-
volved. A recent literature review (Flus and Hurst, 2021) sheds light on opportunities
for design research in hackathons, by introducing a broad perspective indicating that
hackathon participants follow the typical divergence–convergence patterns in their de-
sign process, thus confirming our field observations of an inherent Design Thinking
approach in these events. However, in general, details on the ideation process used
in these events are still missing in literature (Angarita and Nolte, 2020). We can
find isolated contributions about the hackathon process, under a broader view of its
phases (Komssi et al., 2014; Pe-Than et al., 2018; Valença et al., 2019). Few of these
works slightly approach the design perspective of individual hackathons, such as a
self-reported process in one hackathon (Olesen et al., 2018) and a detailed study that
focuses on a specific ideation method typically used in hackathons that used two events
as sample (Filippova et al., 2017).
As a way to bring a contribution to literature, by going deeper into the details of the
design processes and methods used in hackathons, we tried to better understand the
step-by-step performed by participants of hackathons, from problem to solution. We
conducted a case study of focus-centric hackathons that have competitive focus (i.e.,
awards prizes) and collected data via semi-structured interviews with 32 winners and 5
organizers of hackathons from 16 different countries. In this paper, we used the phases
of the Double Diamond design process model to frame the different design methods
we identified as being recurring in hackathon winning teams. We believe this set of
practices help to understand the proposed challenges, create software prototypes and
succeed at the event. Based on such findings, we derived a group of recommendations,
also reflecting current discussions in the maturing hackathons literature.
In Section 2, we present the conceptual background that provided the foundation for
this research, as well as related work. Section 3 describes our research methodology,
based on a qualitative paradigm. Section 4 describes our main findings around the
typical design methods adopted by hackathon winnners. In Section 5, we bring a
discussion on the findings and derive recommendations based on a critical analysis of
the design practices employed by the teams. Finally, in Section 6, we conclude this
paper by highlighting its main contributions to academia and industry. Besides, we
detail threats to its validity and future studies.

2. Conceptual Background

2.1. Design Methods and Design Thinking Process Models

In the Human-computer Interaction (HCI) field, several techniques and methods have
been developed, aiming to be applied to achieve the development of usable systems for
people (Ogunyemi et al., 2019). There is a multitude of design methods that have been

1 Major League Hacking’s guide - https://static.mlh.io/docs/mlh-member-event-guidelines.pdf

2 The Hack Day Manifesto - http://hackdaymanifesto.com/

3
developed and documented (Hanington and Martin, 2012; Mueller-Roterberg, 2018;
Ogunyemi et al., 2019), such as personas, mind mapping, Wizard of Oz, among many
others. The terms “techniques” and “methods” are used interchangeably by HCI re-
searchers in the literature because of the similar meanings of the two terms (Ogunyemi
et al., 2019).
Going beyond universal design methods, design thinking research started with the
interest of identifying mental strategies of designers, but later its concept was stretched
to a thinking process for conceiving products/services and introducing design meth-
ods into fields such as business innovation (Tschimmel, 2012). Design thinking brings
design practice among people without a scholarly background in design, and im-
portant process models have emerged in that field, with well-defined phases that
have specific objectives (Grönman and Lindfors, 2021). Proposers of Design Thinking
process models such as IDEO’s 3I model (Inspiration-Ideation-Implementation), the
Hasso-Plattner Institute model (Understand, Observe, Point-of-view, Ideate, Proto-
type, Test) and the British Council’s Double Diamond Model defend their approaches
as the most appropriate ones for innovation (Tschimmel, 2012). Synthesizing such a
process into a fixed number of steps can be very restrictive, as design thinking is an
inherently iterative and usually non-linear approach. However, models help to give a
comprehensible view and act as a support tool on the general steps that can be taken
in a project.
Behind these models, there are key shared concepts like divergent and convergent
thinking (Brenner et al., 2016), and the notion of problem and solution space, which
form a general perspective of the design process, as presented by Lindberg et al. (Lind-
berg et al., 2011) (Figure 1). The purpose of divergent thinking is to create choices,
while convergent thinking focuses on eliminating options and making choices (Brown,
2009). The problem space and solution space, illustrated on Figure 1, are also com-
monplace in design. The goal is exploring the problem space to build a shared under-
standing of the problem before the actual development process starts, and exploring
the solution space to promote a creative ideation process to choose the most viable
path among the many alternatives generated (Lindberg et al., 2011).
Mueller-Roterberg (Mueller-Roterberg, 2018) provides a perspective (Figure 2) en-
compassing both Lindberg’s perspective and the Hasso-Plattner Model, as well as
the Double Diamond framework for innovation (British Design Council, 2019). The
author highlights that, even though some Design Thinking processes are represented
sequentially, they are inherently iterative, allowing feedback to previous phases.
The Double Diamond Model (British Design Council, 2019) indicates this
divergence-convergence happening twice (the two diamonds on Figure 2), which is
in compliance with Lindberg et al. (Lindberg et al., 2011). Each of the two diamonds
brings the notion of exploring an issue more widely or deeply (divergent thinking)
followed by focused action (convergent thinking). The first diamond exploring stake-
holder needs and finding a problem definition, while the second one aims to generate
multiple potential solutions and implement one of them. The four phases are depicted
as follows:

• Discover. This stage consists of divergent thinking. The focus is on talking to

stakeholders and understanding, rather than simply assuming, what the problem
is.
• Define. This step converges the insights and information from the discovery
phase and helps to define the problem or challenge.
• Develop. This is the first stage of the second diamond, where the focus is again

4
Figure 1. Exploration of problem space and solution space follow a pattern of divergent thinking followed
by convergent thinking (adapted from the work of Lindberg et al. (Lindberg et al., 2011)).

on divergent thinking, generating different answers or solutions to the clearly

defined problem are sought.
• Deliver. This stage is convergent thinking and involve testing different solutions
at a small-scale, eliminating what would not work and improving and choosing
what would work.

2.2. Hackathons
In a nutshell, we can describe hackathons as time-bounded collaborative events in the
format of ”programming marathons” focused on solving a problem in a short time
frame (1 to 3 days) usually delivering a software prototype as a result (Briscoe, 2014;
Komssi et al., 2014). These events started in the 2000s as a method that focuses to
create solutions to tackle specific challenges (Pe-Than et al., 2018). There are different
categorizations and taxonomies on the types and formats of hackathons (Briscoe, 2014;
DiSalvo et al., 2014; Kollwitz and Dinter, 2019). We will refer to the one from Briscoe
and Mulligan (Briscoe, 2014) which loosely groups hackathons as tech-centric or focus-
centric. The former comprises events focused on developing software using a specific
technology (e.g., a hackathon aiming to promote the usage of an API). The latter
involves creating software prototypes to address a specific social issue or business
objective; for instance improving city transit systems. For a contextualization on our
perspective on the topic, throughout this section we situate Briscoe and Mulligan’s
classification in regards to three main categories of hackathons we highlight: corporate,
educational, and civic hackathons (Valença et al., 2019).
Corporate hackathons aim at broadening participation in the corporate innova-
tion network, with participants typically motivated by learning and networking (Flores
et al., 2018; Pe-Than et al., 2020). They are commonly adopted by IT companies of all
sizes, which integrate these events into their research and development activities. The
resulting outcome of such collaborative events are new ideas, early prototypes, and
even business plans (Pe-Than et al., 2018). From a perspective of the participating

5
Figure 2. Merged vision of different design thinking perspectives encompassing the Double Diamond (British
Design Council, 2019), the Hasso Plattner Model (Hasso-Plattner Institute, 2020) adapted from (Mueller-
Roterberg, 2018)

audience (i.e., the ”crowd”), corporate hackathons can be either internal or external
to an organization. Once internal, the hackathon stimulates the creative thinking
of the organization staff to reflect on those challenges and raise new ideas. For in-
stance, Facebook systematically runs hackathons and reports on further developing
and shipping 60% of the topics were proposed in these events (Raatikainen et al.,
2013). Many of these internal hackathons are tech-centric. On the other hand, ex-
ternal hackathons are open to participants outside the organization. This approach
follows the open-innovation paradigm by introducing external resources in the pro-
cess of crafting new solutions, which would tend to be focus-centric hackathons. In
the particular case of IT companies, external hackathons can alternate between a
tech-centric or focus-centric approach, or even combine the two, since in that context
these events represent a strategy to support ecosystem evolution by offering a software

6
platform for third parties to develop new products or services, and encouraging such
outsiders to become complementors in the network. This is the case of well-known
ecosystems such as Google’s Android and Apple’s iOS (Valença et al., 2019). In the
corporate world, besides promoting platforms and technology (Smolander et al., 2020)
hackathons also are a way to attract and build a community of experts (Granados and
Pareja-Eastaway, 2019), which help to foster a broader innovation ecosystem. There
are many hackathons focused on innovation, such as TechCrunch Disrupt3 , Junction4 ,
and many others that are usually listed in hackathon portals like DevPost5 . These
events have multiple corporate sponsors promoting their brands and products, and
sometimes venture capital investors attracted by the potential of the outcomes. Thes
are also under the umbrella of corporate hackathons, with events being many times a
combination of tech-centric (e.g., highly focused on innovation around products and
APIs) and focus-centric (e.g., specific problems to be addressed with technology, focus
on business models) characteristics.
Educational hackathons (Gama et al., 2018; Nandi and Mandernach, 2016; Por-
ras et al., 2018) are performed in association with teaching and learning activities,
either as an exclusive initiative of a professor or as cooperation between academia and
industry – which is sponsoring the event. These are typically tech-centric events. In IT
or Design courses, for instance, the hackathons become a contest for graduating stu-
dents to address real-life issues – thus bringing also a focus-centric approach – in a fun
scenario that enables them to intensively collaborate and enhance their abilities (Por-
ras et al., 2018). The hackathons to address civic issues, organized by the public sector
or by non-governmental organizations, can be classified as civic hackathons (John-
son and Robinson, 2014), which focus on more socially-oriented innovation (Briscoe,
2014; DiSalvo et al., 2014). These are normally focus-centric hackathons, although
Government institutions also have been using such events to generate value from open
data and APIs (a more tech-centric perspective), which are explored by different play-
ers (e.g., citizens, different types of companies, universities, etc.). In general, these
contests leverage the idea of government as a platform (Safarov et al., 2017).

2.3. Related Work

The rapid pace of hackathons leaves no space for use case definition and architec-
tural design (Thomer et al., 2016), but these events hold significant potential to ex-
plore users’ or customers’ interactions and rapid identification of their needs. Hence,
hackathons can benefit from design thinking frameworks to produce better and more
feasible customer-oriented solutions (Saravi et al., 2018). Some organizations per-
ceive the importance of design methods, including a specific preliminary phase in the
hackathon to foster participants’ ideation. For instance, to promote innovation some
hackathons can start with activities outlining the basics of Design Thinking method-
ologies and skills to participants (Avalos et al., 2017). The basic training enables groups
to focus on key suggested methods to address the challenges, from artifacts to identify
users and sponsors to artifacts to validate the ideas.
Overall, hackathons can have important learning outcomes for participants in re-
gards to design thinking. In a project described by Lyndon et al. (2018), high-school
students participated in a hackathon where they had to create an effective user in-

3 techcrunch.com/tag/disrupt/
4 www.hackjunction.com
5 www.devpost.com

7
terface for a mobile application in the medical domain, to support surgical capacity
inventory. Hands-on activities in the first day helped to build the necessary skills for
the prototyping phase that took place in the second day. According to what students
positively reported, they developed more empathy toward users and got acquainted
with a design process to develop a product. The work from Suominen et al. (Suominen
et al., 2018) describes an educational hackathon in the context of university-industry
collaboration in which teachers introduced students to design thinking methods. Page
et al. (Page et al., 2016) report on a corporate-sponsored 4-day hackathon focused on
design education that was performed at a university.
Although there is evidence of design thinking being addressed in hackathon lit-
erature, empirical studies on such design practices in the context of those events are
considered scarce (Angarita and Nolte, 2020). The design thinking process or processes
inherent to hackathons – either via formal training or as practices typically adopted
by participants – are rarely reported in detail. In fact, the literature often focuses on
broader views with proposals for organizing the whole event, such as the adaptation of
a Lean Innovation Model for a 2-day hackathon (Flores et al., 2018). Another example
is the work from Frey and Luks (Frey and Luks, 2016), who depict different phases
of a hackathon and enumerate activities expected for each phase on what they called
the innovation-driven hackathon pattern. In the first phase (Problem), the user needs
are the priority. The second phase (Solutions Alternatives) aims at thinking about
solutions for the identified problems; followed by a third phase (Prototypes) showing
a prototype with the key features (i.e., an MVP) of the alternatives. However, that
work does not explicitly mention any specific design method. Poncette et al. (2020)
briefly details what the called “hackathon flow”: brainstorming sessions using post-its
and diagramming; prototyping (software mockup); and a pitch to present the working
prototype. (Suominen et al., 2018) lists the design techniques or tools (brainstorming,
mindmapping, etc) that were taught in the studied hackathon, but the authors did
not offer additional details of their use nor interpreted such practices. (Page et al.,
2016) presents a 4-day event was structured upon the Double Diamond process model,
the descriptive analysis of the authors was not focused on design issues, but rather
on general aspects or characteristics of a hackathon such as icebreakers, leadership
and team roles, and workplace environments. McGowan (McGowan, 2019) presents a
model for supporting the construction of services by librarians in hackathons, showing
clear phases based on design thinking: Assess, Define Objectives, Develop, Deliver,
and Measure. However, the report does not focus on what design methods are applied
in each phase.
We associate the limitations of details around design methods in hackathons due to
the informal and unstructured nature of the design steps, which are hard to identify in
practice. Another assumption for such lack of studies is the typical focus of hackathon
organizers on technical concerns, as highlighted in Frey and Luks (2016), instead of
focusing on design activities. The recent literature analysis by Flus and Hurst (Flus
and Hurst, 2021) brings a broad perspective on overall design practices by hackathon
participants, highlighting the Double Diamond design process as a model that encom-
passes the typical divergence–convergence patterns that hackathon participants use.
However, the article does not provide any details on which design methods or creativ-
ity techniques can be typically followed through the phases of the double diamond.
Our main contribution in this work lies in not just identifying and listing such meth-
ods but also in analyzing how these design methods are being adopted and applied in
hackathons. By discussing the hackathon design process with experienced participants
and organizers, identifying the recurring methods and representing a perspective of

8
an overall process through phases, we generated recommendations to improve such a
roadmap.

3. Methodology

This research followed a qualitative approach to answer the following research question
(RQ): how do developers apply design processes and design methods hackathons? These
design methods inherent to competitive and focus-centric hackathons were the unit
of analysis of a descriptive case study that we conducted with participants and
organizers of these events. This type of case study sets to describe a phenomenon
within the data collected (Zainal, 2007).
To understand our unit of analysis, we performed a wide set of interviews as
well as a focus groups. Such empirical data paved the way for a inductive research
approach when we described the phenomenon in its particular context and obtained
insights. After identifying patterns within data, we adopted an abductive approach
to transcend collected data and introduce a creative aspect in data analysis (Sjøberg
et al., 2008). Such abductive reasoning presents a conceptual explanation by analogy
with relevant ideas in domains that are well-understood (Haig, 2018). Hence, to draw
one of the possible representations of the design processes followed in hackathons, we
combined established concepts from Design with concepts extracted from the state of
the practice in hackathons.
The following sections explain our data collection procedure, together with the
associated analysis and results, as represented in Figure 3.

3.1. Interviews
In June 2019, we structured the interview guide based on discussions involving the
authors with a design background, about a typical Design Thinking cycle. The au-
thor with the largest experience in conducting hackathons has a Computer Science

Figure 3. Two main research phases.

9
background – with some knowledge on Design Thinking although no formal training
on the theme – and presented the study hypothesis to the authors from the Design
field, who had a formal background on the topic but little experience in conduct-
ing hackathons. The hypothesis on the hackathon participants’ informal knowledge of
design methods was based on non-systematic observations made as part of the ac-
cumulated experience in the organization and conduction of hackathons of different
classifications (focus-centric, tech-centric and a combination of these two) and types
(corporate, civic and educational). This led to rounds of discussions that helped reduce
the initial bias in questions asking specific design techniques, which were exchanged
to more general questions, such as trying to describe the step-by-step process, and
broader questions about higher-level design activities (e.g., generation and selection
of alternatives, prototyping, testing).
This process resulted in the interview guide that was divided into two main parts:
the first part consisted of demographic questions about the interviewee (e.g., age, edu-
cational degree), while the second one involved specific questions about the hackathons
dynamics, involving teamwork and the design process. We started the second part by
asking interviewees about their groups in the hackathon, addressing topics such as
the process of team formation and team composition (e.g., the roles played by each
participant). The subsequent questions explored the steps of the design process exe-
cuted during the hackathon. In total, we included 21 questions in our protocol, which
is available in the appendix. We must highlight our adoption of semi-structured in-
terviews to ensure that a single basic structure was followed, but new questions could
be brought up based on interviewees discourse as a means to obtain a more in-depth
understanding of the topics. In the case of organizers, we adapted the way questions
were asked by bringing ”do you propose...” or ”do you see any occurrence of...” types
of inquiry.
We selected the interview participants considering their experience in competitive
hackathons that were corporate-sponsored – but not necessarily events with a single
sponsor – and more centered on problems (i.e., focus-centric), with specific challenges
where innovative solutions are mostly appreciated. This would allow us to understand
what is the participant’s path to find such innovations. Another fundamental selection
criteria concerned the participant’s experience in winning a prize (1st-3rd place) in a
hackathon at least once. We chose to contact winners for two reasons: (1) from our
experience in organizing these events we notice the combination of design methods as a
common pattern in winning teams while other participants were very tech-centric and
less worried about the design process; and (2) it is difficult to have access to the list of
participants in hackathons websites, where typically only the winners are announced.
All hackathon winners we contacted were evaluated by a jury of experts in the events
they won. Interviewees were contacted through professional social networks and re-
ferrals. In the former strategy we found participants by contacting people through
their LinkedIn or GitHub profiles after finding them in searches for event winners on
hackathon portals such as DevPost and Web sites of large hackathons such as Junc-
tion. The latter one consisted of asking peers from the authors’ professional networks
to indicate experienced hackathon participants who have already won prizes in such
events. In both cases, to avoid participant bias, only people unknown to the authors
have been contacted. Thus, we avoided reaching participants from events we previ-
ously organized. In July 2019, we performed a total of 37 interviews through video call
software with people experienced with hackathons: 5 organizers from varied countries
(Australia, Canada, Brazil, France and Portugal); and 32 winners of hackathons from
16 countries (including winners of the largest hackathons in Europe, Singapore and

10
Silicon Valley). They all have a software development background, including intervie-
wee P22, who has a Bachelor of Arts in Architecture but is a front end developer. The
complete profile of the hackathon organizers is presented in Table 1, while the profile
of the participants is shown in Table 2.

Participant Country Gender Company’s main activity

O1 Canada Male Banking
O2 Australia Male Hackathons Organization
O3 Portugal Male Hackathons Organization
O4 Brazil Male Hackathons Organization
O5 France Male Hackathons Organization
Table 1. Profile of the interviewed hackathon organizers. The first one is responsible for organizing hackathons
in a banking institution while the others are part of institutions focused on organizing hackathons.

3.2. Data Analysis

The duration of the interviews ranged from 8 to 52 minutes, with an average dura-
tion of 25 minutes. In total, we obtained 15 hours and 12 minutes of recordings. To
analyze this rich dataset, we adopted a live coding procedure, which means ”not tran-
scribing everything” (Parameswaran et al., 2019). Based on this approach, we could
code the interviews ”on the fly”, writing down memos and extracting relevant quotes
that were tagged with the respective timestamps where they show up in the digital
audio file. Researchers from a design background and experience in applying design
methodologies analyzed the whole dataset to identify occurrences of design methods
in interview transcripts. Each audio file was listened to separately by two researchers:
one who generated analytical memos (Saldaña, 2013) and registering relevant quotes
when evidence of a design method was found on the interviewee’s speech and another
who listened to the audio to confirm the identification of the techniques and eventu-
ally other techniques. The second researcher also complemented the data generating
additional memos or extracting more quotes. Such selective transcription procedure
was supported by notes taken in an interview diary created by the researchers while
performing the interviews. We also created a spreadsheet to describe the profile of each
participant, which were also based on analytical memos that detailed the researcher
perspective about the interviewee and highlighted some characteristics perceived (e.g.,
tech-focused, design-focused). In Table 2, we present the main characteristics of the
interviewees (i.e., country, age, gender, experience with software development, back-
ground, total of participation in hackathons, and number of prizes obtained).
As a complementary step to examining the interviewee’s discourse, the group of
researchers analyzing the interviews did two face-to-face meetings to reach a consensus
on the common characteristics of the design processes followed by interviewees and
the individual techniques that were mapped in the analysis. We could also extract and
map a group of best practices on the use of such design and creativity methods. This
analysis process was performed in August 2019 and revealed we had already reached
saturation close to the 30th interview, with no new occurrence of design methods
on the subsequent interviews that were analyzed. This ensured us that no additional
interviews would be necessary. As a main result of the analysis, we enumerated the
recurring design methods that hackathon winners apply, as presented in Section 4.
We framed those methods under the phases of a Design Thinking process model that
represents well the short cycle of a hackathon (Flus and Hurst, 2021), although we do

11
not intend to limit a hackathon design process to be restricted to such model.
Hackathon Experience
Participant Country Age Gender Background* Participation Prizes
P1 Angola 25 Male CS student 1 1
P2 Argentina 28 Male BSc. CS 30 5
P3 Australia 23 Male BSc. CS 5 2
P4 Australia 26 Male BSc. SE 4 1
P5 Australia 35 Male BSc. CS 5 3
P6 Brazil 21 Male CE student 6 4
P7 Brazil 19 Male CE student 4 1
P8 Brazil 23 Female CS student 2 1
P9 Brazil 22 Male CS dropout 22 16
P10 Brazil 24 Male BSc. IT 6 2
P11 Brazil 31 Male BSc. CS 4 1
P12 Canada 18 Male CS student 20 3
P13 Egypt 21 Male BSc. CE 4 4
P14 England 24 Male BSc. IT 4 1
P15 Finland 25 Male BSc. IT 10 6
P16 France 22 Male CE student 2 2
P17 France 23 Male CE student 1 1
P18 France 22 Male CE student 4 3
P19 France 21 Male CE student 2 1
P20 France 21 Male IS student 2 1
P21 India 19 Male CE student 3 1
P22 Japan 25 Male BA Arch 3 2
P23 Mexico 23 Male CE student 4 2
P24 Mexico 25 Male BSc. ME 2 2
P25 Peru 45 Male BSc. CS & Math 100+ 30+
P26 Peru 38 Male BSc. IT 5 1
P27 Peru 23 Male BSc. IE 14 6
P28 Portugal 28 Male BSc. CE 3 1
P29 Singapore 18 Male CS student 7 3
P30 USA 21 Male CS student 4 1
P31 USA 21 Male CS student 6 5
P32 USA 33 Male BSc. CS 4 1
Table 2. Profile and distinct backgrounds of interviewed hackathon participants who won different events
evaluated by expert juries.
* Arch = Architecture, CE = Computer Engineering, CS = Computer Science, IE = Industrial Engineering,
IS = Information Systems, IT = Information Technology, ME = Mechatronic Engineering

3.3. Focus Group

To enrich the resultant analysis with additional perspectives and new insights, we de-
cided to perform a focus group with people experienced with hackathons. We planned
this step between October and November 2019. In total, we selected 8 people, with a
background on Design and IT-related courses, as described in Table 3. Most of them
(5) had previously acted as organizers and participants of hackathons, which enabled
them to present opinions from both perspectives. These people were participants of
previous hackathons we organized and people from the authors’ professional networks
who have experience in participating or organizing that sort of event.
We started the session by presenting key concepts (i.e., focus-centric hackathons,
and design thinking’s goals and models) to form a common conceptual framework.
Then, we described the goals and rules of the focus group, which enabled participants
to understand the dynamics envisaged for the session. Initially, we presented a rep-
resentation of the design process that we mapped in hackathons, as a result of the
interviews previously performed. The participants were asked to critically analyze the

12
 Experience in hackathons
Age Gender Background Roles Participation Mentoring
21 Male CE student Dev/Business/Design 8 1
41 Female PhD Design Hackathon organization 30+ 20+
22 Male CE student Dev/Business 12 4
23 Female BA Design Design/Business 3 0
25 Male EE student Dev 9 2
23 Male EE student Dev 5 0
22 Male CE student Dev/Business 12 3
28 Male Design student Design/Management 3 0
Table 3. Participants of the focus group and their background (CE = Computer Engineering; EE = Electronic
Engineering)

process considering their experience with hackathons. For instance, assessing whether
they would make changes such as adding, removing or moving methods to other phases.
Finally, we promoted a discussion about (i) benefits brought by the design methods
typically applied in the design process that we represented as the Double Diamond and
(ii) overall recommendations for hackathon participants from a design perspective.
The audio recording of about an hour was also transcribed via live coding, with
codes such as how to use the design methods, and best practices for a hackathon. The
transcript was discussed among the researchers to interpret quotes from participants
and clarify specific aspects raised during the session. The resultant data interpretation
provided us with fine-grained suggestions on how to apply the design methods as well
as practices to reinforce or avoid in a hackathon. Such analysis generated a set of
recommendations, which we present in Section 4.

4. Design Methods used in Hackathons

In this section we start with a subsection focused on the perspective from organizers,
followed by a subsection with the participants’ perspective on the design process. In
the last subsection we take into account the opinions of the focus group participants,
suggesting complementary design methods throughout the phases of the process.

4.1. Organizers Perspective

Before bringing a perspective of participants, we wanted to understand how organizers
address the notion of activities that would comprise a design process in hackathons
and how they see some practices of the participant. The main results we highlight in
this section concern perceived practices on understanding user needs and prototyping;
and attempts of organizers suggesting a process for participants.
Concerning attempts to address user needs (or, an audience’s needs since there
are no users sometimes), O3 perceived that short hackathons (12h to 24h) solutions
were self-biased, without a strong foundation to justify decisions on the choice of a
problem or feature, while in longer hackathons (48h) there are clear attempts to target
a problem of a specific audience. In the case of organizer O1, who is from a banking
institution, the hackathons were short events on Saturdays (from 9 AM to 6 PM),
with the advantage of having stakeholders from the bank available during the whole
day. The problems and solutions were clear, thanks to the stakeholders, but they did
not interfere in the choices of participants. Due to the limited time, the focus was on
ideas or innovative approaches rather than technologically appealing solutions.

13
 Organizer O2 reported his institution being focused on 24-hour corporate-sponsored
hackathons targeting university students. He mentioned tech groups being focused on
the product/prototype itself, while multidisciplinary groups tend to go beyond that
to explore business models and user research. Despite the lack of recommendation
of design processes or design methods, organizer O2 observed common practices in
the first hours of the hackathon, such as performing (i) a brainstorming session and
designing wireframes to choose project ideas and (ii) quick surveys through online
questionnaires on tools like Typeform or Google Forms to validate ideas and gather
some user information. Design practices seemed more focused on UI aspects, rather
than conception aspects.
We found many hackathon organizers proposing overall dynamics at the beginning
of the event, which includes idea pitching to support team formation – although often
times groups come already formed. Concerning a project conception process, organizers
typically do not propose any specific model or set of design methods to conduct the
activities to develop the solution. The context of organizer O5 was very particular.
He is in charge of a university department that takes care of innovation initiatives,
so the mission is to foster a creative mindset. The hackathons they organize are more
mostly oriented to idea generation and value proposition, rather than creating fully
functional application prototypes or proof of concept. So, this was the only case of
specific creativity techniques being put into practice.
The closest to a product-focused process we found consisted on what was reported by
organizer O4. The company where he works focuses on conducting corporate-sponsored
hackathons. They have general guidelines for participants, mentors and clients. Their
purpose to systematize hackathons organization resulted from observations of groups
failing, as illustrated below. Hence, they started promoting a Design Thinking mindset
to participants.
”There were groups that did not know what problem they are tackling. They were
proposing a solution without a problem. It was a sort of self-sabotage. They would stick
to an idea because of the technological solution” - O4
This company started framing the hackathons using the Double Diamond as a frame-
work, but not stressing the usage of too many design methods or creativity techniques
as it was the case of organizer O5. Basically, the usage of brainstorming and persona in
the problem space, and the recommendation of presenting an MVP (Minimum Viable
Product) using storytelling in the solution space. The reported hackathon participants
came from different backgrounds (IT, design, business) and, once they knew any design
technique or method (e.g., user journey), they employed it as they want.

4.2. Participants Perspective

The majority of the winners we interviewed never had formal training in any of the
identified design methods. In one of the questions, we wanted to speculate about the
influence of designers in the group. According to interviewees, who were all developers,
the eventual designers who would occasionally be part of the groups did not bring any
creativity techniques. They were mostly focused on UI Graphical Design. User research
and ideation was an activity shared by all team and sometimes led by the interviewees
(IT background).
We found much evidence of design methods being learned from other peers during
hackathons or applied in a simplified or adapted manner. More important than using
individual methods is having a design thinking mindset. We noticed that participants

14
hold strong critical thinking, e.g. they generally reflect and investigate a given problem
before focusing on the possible solutions. With hackathons being a sort of venue where
the freedom of thinking, the ad-hoc nature of working seems to be inherent of the
prevalent culture. Some people might see a process as a restraining device for creativity,
as stated by the two participants who reported a bad experience with processes being
imposed:
”Some hackathons have this workshop style hackathon where they actually ask you to
follow their process. It kind of feels a bit forced. (...) The way I usually like to do it’s
very informal.” - P15
”It takes away our freedom. Each team has its own way of organizing its work. Having
(to follow) very defined stages is like imposing a project structure. That doesn’t leave
a good feeling. Of course, in this sense, I would value if there was someone who knew
or would suggest or support teams that have no idea how to organize, but should not be
imposed.” - P26
Interviewees mentioned hackathons as not having a structured approach. However,
our analysis showed that the approach these hackathon winners described follow a
convergent-divergent approach that is clearly a design thinking process, where many
design methods are employed typically in an informal or adapted way. In particular,
such a process was similar and consistent among interviews. As contradictory as the
above statements from P15 and P26 might sound in contrast to what the upcoming
sections show, we found a general design process taking place after analyzing what
the interviewees described. There are clear convergence-divergence phases followed by
the hackathon winners we interviewed, characterizing a Design Thinking process in
place. In Figure 4, we present a perspective showing the recurrent design methods
placed under the different phases model of the Double Diamond design process model.
Although we cannot claim that there is just one design process model for hackathons,
we found the Double Diamond as a model that easily encompasses what was described
by interviewees. This choice is also reinforced by recent hackathon literature (Flus
and Hurst, 2021). The Double Diamond moulds a very short design sprint, where
the compact timeframe makes it less iterative and more linear. The Design Council
proposes a 4-phase framework (discover-define-develop-deliver) with a set of design
methods for each phase, however we only take the 4 phases and enumerate the design
methods in the corresponding phases based on our analysis. In particular, a large
range of methods (Hanington and Martin, 2012; Mueller-Roterberg, 2018) is suited
for different phases of the design process. Hence, some of them may be applicable to
more than one phase or be usable in other contexts.
The next sections describe our mapping and understanding of the methods being
used by participants of hackathons, according to our interviews. What we bring here is
not a prescription of methods or imposition of the Double Diamond as the sole design
process model that is used in hackathons, but rather a perspective resulting from an
analysis of the state of the practice in hackathons.

4.2.1. Discover
According to the majority of interviewees, presenting a solution to a well-grounded
problem is more important than showing a complex or hi-tech application that solves
no one’s problem. First, they need to think about the problem, not the solution. They
shared the understanding that this fact was what put them ahead of the competition
in the hackathons they won when compared to other groups that did not win prizes.

15
Figure 4. The Double Diamond representating a Design Thinking process in hackathons, with the main
recurring design methods that are used by participants.

However, finding a problem requires an effort concerning requirements understand-

ing. In hackathons, a key challenge is overcoming the confinement and the limited
timeframe of the event in order to find a way to access potential users and try to un-
derstand their needs. Team members usually end up speculating based on their own
assumptions. They also ask mentors and other participants to play the role of users.
Hence, part of the understanding of the domain may come from informal discussions
with domain specialist mentors. Besides, Web searches (e.g., Google, Bing) are the
typical mechanism for gaining understanding in the target domain.
When the hackathon theme is previously made available, some participants reported
doing an initial study or defining a hot topic within the big problem presented. Such
research enables groups to gain time before the event. Otherwise, when the theme
or challenge is revealed in the first hours of the event, participants have to rush to
have a better understanding of the problem trying to contact external people or
performing general Google searches as well as accessing resources such as discussion
forums, Reditt, Quora, Facebook groups, etc. For instance, one of the interviewees
mentioned a hackathon targeting solutions for the medical domain and chose to
address Parkinson’s disease. His team followed prompts to help to guide their research
on that domain: ”What kind of problems do Parkinson’s disease patients have? We
saw there are a lot of motor skills problems” (P30).

Exploratory research. We also noticed that many of the experienced participants

we interviewed (P3, P6, P5, P13, P21, P25, P30, and. P31) performed more struc-
tured research to discover problems and user requirements, being very close to the
format of exploratory research. In design literature, this is described as a method
typically conducted in the earliest stages of a design process, acting as an immersive
experience to develop knowledge and empathy about the target audience, encompass-
ing other design methods, such as surveys, questionnaires, participants observation,
among others (Hanington and Martin, 2012). Interviewee P31 performed exploratory
research in a very consistent way, combining observations and surveys via interviews
and questionnaires.

16
 Phase Technique Occurrence
Brainstorming P3, P4, P5, P7, P8, P9, P10, P13, P15, P16, P17,
P18, P19, P20, P21, P22, P25, P27, P29
Brainwriting P7, P8, P12, P13, P28
Empathy map P23, P27
Discover
Exploratory research P3, P6, P5, P13, P21, P25, P30, P31
Interviews P3, P11, P16, P20, P31
Persona P1, P7, P18, P20, P21, P23, P26, P27, P28
Scenarios P1, P20, P21, P28
Surveys P13, P31
Questionnaires P4, P14, P31
Affinity diagramming P15, P17, P18, P21
Card sorting P5, P13, P31
Define

Concept mapping P9, P14, P20, P21, P25

Mind mapping P10, P16, P19, P29
Brainstorming P14, P21, P23, P24, P26, P29, P32
Competitive testing P2, P3, P4, P6, P7, P8, P10, P11, P12, P13, P14,
(benchmarking) P15, P16, P17, P18, P19, P20, P21, P22, P23, P25,
Develop

P26, P27, P29, P30, P31, P32

Lo-fi & Hi-fi prototyping P3, P5, P7, P8, P9, P10, P11, P12, P13, P14, P15,
P16, P17, P18, P19, P20, P21, P22, P23, P24, P25,
P26, P27, P28, P29, P30, P31, P32
Minimum Viable Product P5, P4, P7, P11, P20, P24, P30
(MVP)
SCAMPER P5, P6, P7, P10, P13, P15, P16, P18, P19, P20, P21,
P25, P26, P27, P30, P31
Storyboards P1, P2, P3, P9, P11, P13, P29
Wizard of Oz P18, P19, P22
Pitch P3, P4, P6, P7, P9, P12, P13, P14, P15, P16, P17,
P18, P19, P20, P21, P22, P23, P24, P26, P27, P28,
Deliver

P29, P30, P31, P32

Storytelling P3, P15, P16, P17, P19, P21, P22, P31
User Testing P2, P3, P5, P6, P8, P9, P10, P11, P13, P14, P16,
P18, P21, P25, P27, P30, P31
Table 4. Occurrence of the design methods identified in each interview (Participants P1-P32)

“When going to the venue, we used to see what is going wrong around us. In Los
Angeles, we saw there were many blind people on the streets. So, we made a cool object
recognition glasses project (. . . ) if it is a 48h hackathon, you have time (to contact
stakeholders). One or two people from the group ask a very selected number of questions
and that would give you a clear-cut view. We had a lot of contacts, ranging from shop
owners to people who are visually impaired. If we don’t have fruitful answers, we call
them up. (. . . ) Questionnaires on Survey Monkey or Google Forms were made available
only in the first 6 hours of the hackathon.” - P31

Surveys, questionnaires and interviews. However, other participants with no for-

mal training also did surveys, interviews and questionnaires. According to the many
methods described by Hanington and Martin (Hanington and Martin, 2012), surveys
are composed of questionnaires and interviews, but these two can also be done as
isolated practices. Questionnaires were mentioned by three participants (P4, P14, and
P31), but sometimes the term survey was used to describe a questionnaire, as P14
mentioned: ”... we survey the required data. If we are dealing with a community, we’ll
put questions on the community channel”. The survey method combining question-
naires and interviews was performed by P13 and P13. In the case of P13, for instance,

17
he highlighted: ”we do open or public surveys and ask family and friends for their
opinions”. Others relied mostly on interviews (P3, P11, P13, P16, P20, and P31).
Participant P16, for instance, used data collected during this phase in the project
pitch to show evidence of a well-founded problem.
”In the first hackathon, we called quite a few people. We spent a good hour calling our
acquaintances. We called doctors and nurses around us. This was very important for us
to develop our specifications and subsequently sell our product to the jury, supported by
quotes from users.” - P16
From the interviews we understood that when the identified target audience is a
specific group that may be difficult to reach (e.g., visually impaired), Internet forums
become a useful resource. In 48h hackathons, participants are more likely to contact
people through social networks by adopting online questionnaires or to perform quick
interviews with a small and improvised set of questions. The studied participants
reinforced the need to avoid long and biased questionnaires or interviews, which may
take too much time and generate poor results.

Brainstorming. We observed brainstorming being applied as one of the main diver-

gent thinking techniques during the Discover phase, as evidenced in Table 4. Brain-
storming helps quickly enumerating relevant problems or challenges that could be po-
tential candidates to be addressed during the hackathon. Although being commonly
used for inspiration and idea generation during an ideation phase (Ambrose and Har-
ris, 2009), this method is essentially diverging – which is the focus of the Discover
phase. The brainstorming sessions described by interviewees seemed to be very infor-
mal, not necessarily having the role of a mediator as proposed in the original method
(Osborn, 2012). These sessions follow the principles of free expression of ideas, avoid-
ing criticism so that group members are not inhibited to contribute. As proposed in
the technique, ideas will be judged after the brainstorming session, as illustrated by
P8 who described the process her group did together:
”We would see the theme (of the hackathon) and tried to brainstorm the idea that we
were planning, then we select the best ones until reaching a consensus on what everyone
would like to work on. It was more decentralized; everyone would write it down. In the
end, we chose one (idea)” - P8

Brainwriting. We identified brainwriting as an alternative to brainstorming sessions,

although less frequent in interviews (P7, P8, P12, P13, P28). It had variations that can
be more systematic, such as the 6-3-5 technique (Rhorbach, 1969; Schroeer et al., 2010).
Essentially, in brainwriting instead of the group writing ideas together, each participant
writes their ideas alone and then the selection is done collectively. Generally, it follows
very closely what is called nominal group process (Van De and Delbecq, 1971), which
include steps such as the silent generation of ideas, group discussion and voting. It
is especially useful in groups whose members do not know each other, avoiding the
imposition of dominant members that may take place in brainstorming (Litcanu et al.,
2015). In practice, some interviews showed that different adaptations of brainwriting
in hackathons can be close to a collective brainstorming session: ”we just have a Google
Doc, so we can all just see and write at the same time” (P12). The individual approach
was better illustrated by participant P28, when his group was trying to enumerate
possible topics (which he referred to as ”ideas”) to address in their project:
”We first did an individual generation of ideas. Each person wrote on a paper sheet

18
 ideas they wanted to see and then we talked and chose one of them.” - P28

Persona. Hackathon winners reinforced the relevance of understanding users’ moti-

vations. Many participants (P1, P7, P18, P20, P21, P25, P30, and P31) cited the
creation of personas (Cooper et al., 2004) to understand the problems and needs that
potential users may have. A persona is a general user archetype that can be used at
the beginning of an ethnography process or in the end, as a result of user research.
This method is key to avoid any tendency the developer may have in distorting the
user’s role when building the solution. In design literature, drawing a persona would
be rather related to the Define phase (Mueller-Roterberg, 2018), but we noticed an
anticipated use of this method to explore user needs at the beginning of the hackathon.
We understood this as a way to compensate for the absence or limited access to stake-
holders, and create hypotheses of the target users and their needs. Eventually, some
participants would consider contacts within the event (participants and mentors) or
outside (family, acquaintances, real stakeholders) to validate those hypotheses docu-
mented as a persona. Many interviewees already knew the method without any formal
training on it, while others used it without that formal background. In the testimony
from P7, we observed a group that never had any training to use that method and
was not sure they were using it correctly:
”sometimes, we didn’t see things so well, then the mentor came and said: ’no, you think
you don’t have a persona, but in fact, you do (have). This person here, you can draw a
profile to solve the problem better’” - P7

Scenario. Also in an anticipated manner in the Discover phase to map user needs, we
noticed some occurrences (P1, P20, P21, and P28) of what characterized the usage of
scenarios – although interviewees did not explicitly mentioned that term. In design, ”a
scenario is a concise description of a persona using a software-based product to achieve
a goal” (Cooper et al., 2004). During interviews and direct observation of users, it is
possible to learn about their tasks. It gives the opportunity to exercise empathy with
users, which is a key concept for good design. Participant P20 reported on the usage
of this method to build a narrative on usage scenarios of an imaginary system which
was later validated in an interview over the phone. It helped them understand their
target users would not utilize a mobile device of inappropriate size when using gloves:
”this allowed us to learn things about the professional activity and gave us direction to
certain points of the project (...) We noticed the type of tablet to use in the application
due to the use of gloves (for cleaning). These are types of details that we couldn’t have
known if we hadn’t called them.” - P20

Empathy map. Two participants (P23 and P27) mentioned empathy maps as a way
to identify users pains and gains, which can also be used to describe a persona (Ferreira
et al., 2015). In the interviews they indirectly mention how they try to speculate what
these users see, hear or want (part of the typical guideline for an empathy map),
characterizing the usage of the technique, although stating that they never had any
formal training in design methods. Participant P27, for instance, explicitly mentioned
the technique by its name and correctly summarized its concept, thus indicating he
knows the method:
“in social hackathons, we use the empathy map a lot, to find out what they (users) see,
how they feel, what they think, what they do... and give them a solution” - P27

19
4.2.2. Define
During this phase, there are practices that support participants finding the problem
they will address. The main output of this stage is the problem statement, which will
guide all the solution. The previous phase generates many inputs that have to be
filtered and converged to a bold problem statement, which is one of the key aspects
mentioned by winners of large and important hackathons. A well-defined problem will
help to convince the jury in the end. Therefore, adequately filtering and choosing
among the inputs of the Discover phase is paramount.
In Design Thinking, three constraints are frequently used to define an overlapping
criteria for choosing successful ideas: feasibility; desirability and viability (Brown,
2009). They were all naturally mentioned in the interviewees’ argumentation.
Feasibility and desirability were the standard choices for them. They correspond to
what is technically possible with the available resources (e.g., time, human resources)
and what makes sense to users, respectively, whereas viability is a business-oriented
criterion related to the commercial viability of the product. Although not always
mentioning those constraints explicitly by name, the interviewees reasoning in the
hackathons was usually guided by such criteria, showing evidence of a Design Thinking
mindset. In the Define phase, the focus in on desirability, since participants are still
framing the problem. Feasibility and viability would be considered later in phases
where the candidate solutions emerge.

Affinity diagramming and card sorting. The first phase (Discover) gener-
ates many inputs from the different methods that may have been applied. For in-
stance, brainstorming or brainwriting generates many notes – often physically de-
tached as sticky notes. Therefore, it is important to apply other methods that
help to converge those inputs from the previous phase. So, some participants (P5,
P13,P15,P17,P18,P21, and P31) spontaneously used a range of clustering methods,
applying mechanisms that closely resemble to affinity diagramming and card sorting
(Hanington and Martin, 2012). The former is a rather inductive process, where work is
done from the bottom up, initially clustering specific, small details into groups, which
will generate broader themes. The latter is a participatory design technique that al-
lows exploring how participants group items into categories and relate concepts to one
another. Their purpose is similar, and we find traces in some of the interviews, partly
illustrated in the quote from P17.
”in the brainstorming, we wrote on post-it notes stuck on the wall. We removed
the notes that were bad and put the good ones in another corner. In the end, we had
several categories of post-its for technical problems, post-its about different themes” - P17

Mind map and concept mapping. In case the results of brainstorming are written
on a whiteboard, paper or flip chart, the common approach is creating associograms
using mind maps, which we identified being applied by participants P10, P16, P19,
and P29. More elaborate mind maps resemble concept mapping method, which is in
part illustrated in the following quotes from P19 and P21, respectively. Mind mapping
helps to visually organize a problem space to better understand it and find creative and
spontaneous association between ideas (Davies, 2011; Hanington and Martin, 2012). In
its turn, concept mapping, which was used by P9, P14, P20, P21, and P25, is slightly

20
different and helps to outline relationships between ideas (Davies, 2011), which we
identified in the interviews as flow charts organizing the concepts. In the quotes from
P19 it is worth mentioning a recurring democratic practice of common agreement
based on majority voting, highlighted by many interviewees.
”a mind map on the whiteboard to connect what different people had provided in the
brainstorming, collect the ideas on that board and converge them. For the selection (step),
we tried to get everyone’s opinion” - P19
”we made some stick notes over there, we took some markers and started writing over
the table, over the board around us (...) So, the things we were facing, the issues, like
multiple problems. We grouped the ideas. How can we do that, how can we do this. We
did some mind maps and also developed some flowcharts” - P21

It is worth noting that many participants mentioned a sort of ”hack” in this process
of choosing or converging ideas. They suggested (i) to use an explicit bias towards
the evaluation criteria defined in hackathon rules as well as (ii) to try to guess the
problems or domains that would please the members of the jury, as illustrated in the
quote below.
”if you want to win, you want to meet the hackathon evaluation criteria and please the
judges. If the judge is a person who came from an environmental area, they will probably
want to see something from that area in the application, because it will touch them
directly. If the judge is from a company, they will want to see a clear business model” - P6

4.2.3. Develop
This phase focuses on divergent thinking to generate ideas and filter them. There
are many activities around prototyping here. Overall, many user experience and user
interface concerns can be validated at this phase, which acts as an ideation phase that
will result in the core concepts of the product to be developed. Having feedback on
ideas and prototypes at this stage is fundamental, since any validation of utility or
usability aspects with potential users is important to iterate on the prototypes and
improve them. It allows early ”pivoting”, which happens when the team realizes a
hypothesis is wrong, then updates the project and retests it (Müller and Thoring,
2012). An interesting example of this sort of validation interviewee mentioned calling
a hospital do discuss with doctors and nurses about features that made them see there
were certain complications in the process that made them pivoting their idea.
Although being a divergent phase, when prototyping and developing, some criteria
for filtering and selecting ideas need to be taken into account. The evaluation criteria
defined in the hackathon rules can be used to filter out their ideas. For instance, if
the hackathon demands or gives extra points for using a certain API or technology,
the solution would have to include that API, which to some participants seems to
be forced. Feasibility, from the desirability-feasibility-viability triad (Brown, 2009) is
typically taken into account for choosing ideas, as mentioned by P5:
”Do what is achievable during that timeframe with the resources you have like available
time, available software” - P5
When the hackathon is related to startups or business opportunities, the viability
of the product is also included as a criterion. Because it is not necessarily a creativity
technique, the Business Model Canvas (Osterwalder and Pigneur, 2010) was not

21
reported as one of the methods from Figure 4, but it appeared in many interviews
(P5, P7, P13, P15, P16, P26, P27) as a tool to help to consolidate important concepts,
such as target customers and value proposition.

Brainstorming. Coming back to the design methods, brainstorming may take place
again, supported by other methods that provide input for generating ideas targeting
the solution or prototype to be developed, as described in many interviews (P14, P21,
P23, P24, P26, P29, and P32) and illustrated below by P29:
”in terms of coming up with the ideas, we come up with buzzwords, like AI; we kind
of juggle around with that word and connect the dots. This feature can do this, it can
do that... By the end of that brainstorming session, we think of the different feedbacks
and inputs from everybody in terms of workload or feasibility; in the end, we have a
well-defined product that we can actually code up” - P29

Competitive testing and SCAMPER. A natural strategy unanimously adopted

is to perform competitive testing or benchmarking (Hanington and Martin, 2012), as
evidenced in Table 4 by a significant number of participants. In hackathons, the effort is
concentrated on Web searches (e.g., Google), with many mentions to websites such as
Product Hunt6 , to look for existing solutions that tackle the same target problem and
quickly evaluate them. In this activity, one can analyze the strengths and weaknesses
of products that are similar to what the group intends to develop.
By identifying the limitations or shortcomings of existing solutions, the next step
they naturally bring into a design process is the use of the SCAMPER, which was a
practice confirmed in 16 interviews. It is a design method for idea generation (Mueller-
Roterberg, 2018) that can turn an existing idea into something new and different
(Serrat, 2017). The SCAMPER method has been around for a while, is summarized as
a mnemonic that consists on a flexible checklist of idea-spurring questions (Michalko,
2006): Substitute something; Combine it with something else; Adapt something to
it; Modify or Magnify it; Put it to some other use; Eliminate something; Reverse or
Rearrange it.
We observed that the usage of SCAMPER in hackathons helps to define or improv-
ing project features based on the analysis of competition. Even without knowing the
existence of this method, the studied participants were typically guided by a subset of
the verbs comprising the SCAMPER acronym. The quote from P15 summarizes the
usage and combination of both techniques to generate project ideas:
”I usually like to Google that idea. There are some sites like Product Hunt, which are
very good for finding what other people have done, and then search if there is a startup
already doing something like that. There is a startup for everything (...) Most of the
creative process is copying what other people have done and putting it on a different
context; then, we add something on top of it or remove something. It is very difficult to
come up with an original idea in a hackathon” - P15

Minimum Viable Product (MVP). The winners had a very clear understanding
that they need to focus on an MVP, which the minimal set of features for solving the
core problem (Müller and Thoring, 2012). The MVP is what is going to be presented
to the jury. The concept is known and used by some interviewees (P5, P4, P7, P11,
P20, P24, and P30), but in majority they though it as a common hackathon practice

6 www.producthunt.com

22
and did not now this concept was borrowed from the Lean Startup approach (Ries,
2011) when asked. Participant P4 showed a clear understanding of the concept, as
illustrated below:
”Once we have some rough idea, we then plan what is the most Minimum Viable Product.
What do we need to deliver to showcase what we want to build? Often times is mainly
from a user experience perspective, a user interface perspective.” - P4
There were other interviewees completely unaware of the MVP term but very ac-
quainted with its concept and the notion of its importance, showing a clear understand-
ing of it as illustrated in the counter-example of P23, who learned that proposing too
many features and lacking focus is a bad practice:
”the errors that we had in two hackathons was not knowing what we were presenting.
We did not have the idea of what product we had made, because we had too many
functionalities and we had put many things when we were programming.” - P23

Low-fidelity and high-fidelity prototyping Before starting the development of

the product (i.e., a functional prototype), a good practice we identified in practically
all interviews (Table 4) was doing some form of prototyping (Hanington and Mar-
tin, 2012), either low-fidelity or high-fidelity prototyping, which are very helpful to
convey an idea and to quickly get user validation. Most of the interviewees said they
do at least some sketches on paper, as in the case of 24h hackathons where many
times groups have little time to prototype before development. Typically, participants
do some lo-fi prototyping with sketches on paper or whiteboards. Paper prototyping
(Snyder, 2003) helps designing, creating and testing user interfaces. These prototypes
are useful for discussing and quickly validating project ideas with mentors or other
participants before jumping into development. In longer hackathons (e.g, 36h or 48h),
groups may sometimes do lo-fidelity prototypes already using specific tools for pro-
totyping mock-ups (e.g., Balsamiq). Some create hi-fi prototypes with tools specific
for that (e.g., Figma, Sketch), simulating some interactions or providing look-and-feel
details resembling the software to be developed. We observed that depending on the
available time and performance of the team some groups would decide if both hi-fi and
lo-fi prototypes or just on type would be done. The quotes of participant P9 illustrate
that there is no rule of thumb:
”We made prototypes on the whiteboard a lot, talking to the UI designer to see what we
could do. The final prototype was on Sketch (the tool), but it is also not a rule. Some
screens we discussed on the whiteboard. In Sketch we thought of the final design.” - P9

Storyboard. As another prototyping approach confirmed in some interviews (P1, P2,

P3, P9, P11, P13, and P29), groups may also combine different wireframes or draw
storyboards. A storyboard, which can be hand sketched in paper or on a whiteboard,
helps understanding the flow of a user’s interaction and how the interface will support
each step (Snyder, 2003). Storyboards were highlighted by P2 as being important with
less experienced team members:
”You show products based on user profiles, with storyboards that I do ... If there are
junior people (in the team), a last minute change can technically be very expensive. So,
we put it down on the paper in order to know how it is organized” - P2

Wizard of Oz. In some hackathons, demonstrating a concept can be more important

23
than completing its actual implementation. Participants may deliberately choose to
fake some features – although being explicit about not having implemented it – just
to illustrate them. Going in this direction, we found some interviewees (P18, P19,
and P22) reporting on adapted usages of the Wizard of Oz method (Hanington and
Martin, 2012; Mueller-Roterberg, 2018), which consists of simulating system responses
from behind the scenes. For instance, one interviewee stated that while some action
was confirmed in the system, another group member sent a text message to the phone
being used in the demo and a fake confirmation message popped up, simulating a
push notification that was not implemented. Simulation is also useful in some Internet
of Things projects, where sometimes the smaller scale models make necessary to fake
some interactions because hardware may not be available, as in the example from P19:
”it is not because there are things that we are not able to do that we will deviate from
this aspect or find an alternative solution. Is there anything that could replace that? We
needed sensors, but we didn’t have them. We simply simulated the operation as if there
were sensors, without having them. Much of the proof of concept” - P19

4.2.4. Deliver
The last convergence consists on delivering the functional prototype. The original
Double Diamond suggests activities that make sense only if you are actually delivering
the project to customers or users. As a Test Phase (Mueller-Roterberg, 2018) where
users would evaluate and give feedback.

User testing. Doing User testing (Da Silva et al., 2011) in the context of hackathons
would actually be limited in terms of access to the actual users, unless domain special-
ists or hackathon sponsors who are stakeholders be present. Although the audience or
potential users are sometimes contacted in earlier stages (discover and define phases),
usually the case with user testing in this phase is doing simple tests with people who
are in the same venue (mentors and other groups) to get feedback and, if possible,
perform minor adjustments after that. User testing was a practice confirmed in more
than half of the interviews, as displayed in 4.
”We found the feedback from participants to be very useful. There’s a lot more partici-
pants than mentors, so it’s easier to talk to them.” - P3

Pitch and storytelling. The final moment is the pitch (Pincus, 2007), which consists
in presenting the project to a jury. It was explicitly mentioned by almost all interviews
(Table 4). A few hackathons described by interviewees had a pre-pitch moment, which
is a sort of presentation rehearsal for quick feedback before the final pitch. While some
groups would just focus on delivering a pitch focused on features, a successful pattern
for this is using a storytelling approach. Even if personas or scenarios have not explic-
itly been drawn in the first phase, they will likely appear here. The problem statement
that resulted from the second phase was also mentioned as something important to
be stated at this point, to attract the jury’s attention. Similar to the choices that
can be made in the previous phases to please the jury, we found the same pattern
here where some participants present a ”tailor-made presentation based on the judges
profile” (P4).
As stated by P30, ”The utility of the product is the most important thing”. Although
not using any more elaborate presentation technique such as using storytelling or a

24
persona to support the pitch, he mentioned the successful usage of what he called
attention getter, which would be something like impacting statistics or the impact
of the target problem in society. Other interviewees mentioned that the pitch has to
demonstrate that the prototype or product solves an important problem. One thing
many participants mentioned is that storytelling is one of the approaches that make
a difference in pitches, as in the example from P15:
”If there is something common in hackathons I’ve been is that good stories make great
pitches. They really want to hear something that they can relate to. So you have to tell
them, ’this is me, this is what I think about that concept, once I had this kind of experience
which was really bad, so I came up with this, in the hackathon, we did that and now we
have this solutions’ that’s the kind of basic thing I like to do. ” - P15
Storytelling was one common approach of the presentation of winning projects and
was complementary to the pitch, as evidenced in interviews from P3, P15, P16, P17,
P19, P21, P22, and P31. It is an approach familiar to designers, used in the conceptual
design process to present concepts to clients (Parkinson, 2016). As Brown mentioned,
”from the perspective of the design thinker, a new idea will have to tell a meaningful
story in a compelling way if it is to make itself heard” (Brown, 2009). The statement
made by P21 shows a lesson learned where he learned in practice a common design
approach:
”If you think about the storytelling before starting to develop the product you might de-
velop it much better. Because at one point at the time the story was going in a direction
and the solution was not fitting into that. So you have planned the story before you can
just align the product with that. ” - P21

4.3. Enhancing the Process with Additional Design Methods

During the focus group, participants proposed a set of additional design methods
to enhance the design process identified in the interviews. In the Discover phase,
participant P7 said groups can adopt jobs to be done (Christensen et al., 2016;
Mueller-Roterberg, 2018) to explore the origin of users’ motivations. In most cases,
this method will support speculation, as the real user is often not in place during a
hackathon. However, if it is possible to experience user situation and see the problem in
practice, P5 mentioned hackathon groups can employ more observations (Hanington
and Martin, 2012) to perform an ad-hoc ethnographic study (a lightweight version
of design ethnography approach). Alternatively, they can benefit from the 5W1H
method (Andler, 2016; Mueller-Roterberg, 2018), as pointed out by P3. Participants
may use its structure (i.e., who, what, where, when, why, how) to think about the
problem, in light of the theme or technology underlying the hackathon. Then, the
groups can identify the real causes of a problem.
The Develop phase can be enhanced by using mood boards (Hanington and Mar-
tin, 2012; Mueller-Roterberg, 2018) to organize the inspiration around the project and
keep the ideas consistent with users goals and expectations, as mentioned by P8. He
called this method “a living reference of the solution”. The User journey (Haning-
ton and Martin, 2012; Mueller-Roterberg, 2018) method was suggested by P7 as a
way to check whether the proposed solution indeed brings value to the user routine.
Additionally, the morphological box (Zwicky, 1967), cited by P2 and P4, would
enable the groups to describe the problem in the absence of a wide dataset (real data
is commonly scarce in hackathons) by using different dimensions. For instance, groups
can consider “motivation”, “relevance” and “complexity” as parameters.

25
5. Discussion

Considering our research question how do developers apply design processes and design
methods hackathons?, this section presents a discussion on expected and unexpected
findings, based on our previous experience in the organization of hackathons. Then,
based on our data collection, we describe a set of recommendations from a design per-
spective based on the common practices adopted by hackathon winners. We conclude
the section with a broader critical analysis of our findings.

5.1. Expected and Unexpected Findings

Based on our own experience in the organization of hackathons, we expected some
techniques and characteristics of the design process to be confirmed. Indeed, after the
analysis of the interviews, there were some confirmations, but we were also surprised
by some other practices that emerged for each phase, bringing aspects that were not
foreseen. We discuss here the divergence and convergence principles that were noticed,
followed by techniques that appeared as expected or not.
Design Thinking brings the notion of divergent and convergent thinking. The di-
vergence happens first, to generate and explore possibilities, followed by a converging
phase to narrow down those possibilities to something more focused. As highlighted in
Section 2.1, a general model shows this structure happening twice, as a Double Dia-
mond structure: first, diverging and converging on the problem space to understand
the problem to be tackled; then, divergence and convergence are performed again in
a phase focused on the solution ideation and prototyping. From our experience in
hackathons, we see many groups jumping straight into the solution space (i.e., the
second diamond), without much discussion on the problem to be solved, therefore
skipping the first diamond. Groups that take the lead in these competitions usually (i)
spend a good part of their time exploring the problem space, trying to understand the
target audience of the challenge (hackathons usually have a theme) and their problems;
(ii) converge to a problem statement, and only after that start (iii) speculate about
solutions and possibilities to tackle that problem; to later (iv) converge to a final proof
of concept that will be presented to the jury. We could confirm such double diverge-
converge pattern in the interviews, where this strategy to approach a problem and
think of a solution was clear. Some organizers and participants argued it is common
to jump straight to the solution space in shorter hackathons (e.g., 12h). For instance,
they can limit the problem to the event challenge (e.g., improve the quality of life of
the visually impaired) without structuring any discussion or investigation about that
problem and start the event by brainstorming an application (i.e., a solution) and its
features. However, most of the interviewed participants highlighted the importance of
even in shorter events spending some time discussing the problem, as it is fundamental
to avoid solutions to problems without an audience or that does not exist.
During the Discover phase, divergent thinking takes place to explore the problem
space and understand the audience, their needs and gather information about the
problems. However, as we have observed in app development contests, participants
usually build solutions based on their own perceptions without being based on user
needs. This is a bad assumption that frequently takes place in these events and has
been reported in literature (Lee et al., 2015). Based on our experience, we observed
that some superficial exploratory Web research about the problem domain is typi-
cally done by savvy hackathon participants, to get acquainted with the domain. For

26
instance, they may gather some statistics about the target problem (e.g., the per-
centage of the population who is overweight). Nevertheless, we were amazed by the
level of detail that some interviewees reported on their exploratory research. To our
surprise, they mentioned the usage of quick questionnaires and interviews as resources
to contact and gather data from their target audience. This was unexpected because
hackathons tend to be isolated from the target audience or users but some groups
reached out to people outside the event. We foresaw brainstorming to be mentioned
by practically all participants since it is a widespread technique used very frequently in
hackathons. Similarly, we also expected some brainwriting usage, but not as much as
it was reported. One interesting aspect is that discussions to reach a consensus on the
generated ideas was naturally merged as part of the brainstorming and brainwriting
activities. We believe this is a probable reason for not having as many design methods
in the Define phase as in the other phases, since the convergence was made together
with a method used in the discovery phase. The usage of personas was not a surprise,
as we have already seen occurrences in events we organized. However, we saw much
more mentions to what characterized the usage of scenarios as a technique in this
phase than we have observed from our experience as organizers. As a final technique
we captured from interviewees in this phase, we were impressed by how naturally
empathy maps were used to help the construction and description of personas.
In the Define phase brings a convergence of what was gathered in the discover
phase, resulting in a problem definition. The general approach for the selection of
alternative that we observed in our previous experience is around voting on topics,
regardless of where they were written. We have already seen the usage of affinity
diagramming and card sorting in events where we made sticky notes available. In
the context of this research, some of the interviewees declared a more structured
categorization approach when compared to what we had previously seen. Mind maps
were not frequently observed in our context and we did not expect so much usage of
this technique as it was reported in the interviews. The way on how concept mapping
was performed in combination with mind maps was more elaborate than in the few
times we had previously seen it. The major surprise in this phase was the interviewees
allegedly observing the professional profile or background (e.g., sustainability) of the
judges to use it as a criterion to influence the group choices aiming to take the project
toward a direction that would please this audience (i.e., the jury).
In the Develop phase, divergent thinking takes place again. This time to generate
options for the development of the project. The usage of brainstorming in this phase is
very common, which we confirmed in the interviews. In the events we have organized,
during the opening ceremony we use the term MVP (Minimum Viable Product) to
explain that the goal in a hackathon is not to make a product full of features but with
the feature or features that really matter for that problem. So, in our context, this
was always common ground. The term was also known to many of the interviewees.
Surprisingly, some participants of our interviews never heard of the term before but
precisely described its concept, highlighting the importance of adhering to those prin-
ciples in a hackathon. An important finding with interviewees concerned the natural
way of combining benchmarking and SCAMPER. The level of detail on the recurring
mention on searches on Google and Product Hunt in the pursue of apps to tackle sim-
ilar problems was impressive in some interviews. By finding the potential competitors,
participants instinctively adopted a SCAMPER-like approach – without even being
aware of that as a technique – to help to propose something different from what exists
in the market. The typical approach we were used to in our experience consisted of
participants choosing a popular and widespread app as their benchmark and propos-

27
ing their solution with a feature that app did not have. From our background, we
witnessed groups frequently doing lo-fi prototyping in the form of sketches on paper
and occasionally using specific software tools for lo-fi or hi-fi prototyping. This was
confirmed in the interviews, but with the major difference on the massive usage of
software tools for prototyping, especially in longer hackathons where they mentioned
presenting well-crafted UI that was discussed and improved through prototypes. An
interesting technique that emerged which we did not expect was that of storyboards.
Its usage was reported in interviews as it typically is used in interaction design, helping
to visualize a user’s journey or how the user would interact with the main features of
an app. Some interviews declared the simulation of features through another person
doing some action (e.g., an app confirmation message simulated as an SMS manually
sent), consisting of a clear occurrence of the Wizard of Oz technique. This was unex-
pected for us since we have never seen its occurrence in any of the hackathons were
involved.
Deliver would be the last phase on the double diamond, focusing on convergent
thinking toward the delivery of the MVP to its audience. Some steps on validation.
Since it is difficult to contact external users and there is a rush to finish the proof of
concept, we assume some testing in this phase but rather among participants, as it was
confirmed in the interviews. Although the pitch is a presentation technique expected in
this sort of event, we noticed that some participants gave a high level of importance to
it. There were many more mentions to presentation strategies than we expected, such
as the straightforward way to a bold problem statement; the quick timing to get the
jury’s attention; the usage of impacting numbers; among other strategies. We reckon
that storytelling was considered as an obvious choice for the pitch format but some
interviewees did not appreciate that format and preferred what some of them called
a more natural approach stating the problem and showing the intended benefits to
users. This was unexpected for us since storytelling seemed to be a relatively frequent
approach for pitching solutions in the hackathons we were involved. As hackathon
organizers, what we also did not expect was the recurring ”hack” of winners tai-
loring their solution or its pitch to please part of the jury, aiming at a better evaluation.

5.2. Recommendations to a Successful Participation in Hackathons

An initial recommendation is that teams must not start the hackathon with pre-
defined ideas or solutions. A naı̈ve approach we observed in many groups is trying
to start the project with a proposal in mind. Design Thinking is a user-centered ap-
proach that requires seeing the world through the user’s perspective. Hence, hackathon
teams must employ methods to enhance their empathy with the audience (i.e., po-
tential users), which is a core concept in design (Mueller-Roterberg, 2018). During
Discover phase, participants can explore divergent thinking and take a deep dive into
the problem, since a weak exploration of the problem space may lead to solving an
irrelevant problem. The importance of a clear and relevant problem statement was
repeatedly mentioned among winners, as it facilitates the transition to a potential
solution. Such immersion phase is strongly related to requirements engineering, as it
aims to get a minimum understanding of the domain and users (Hehn et al., 2019).
A second recommendation is that groups must plan for the timeframe, rules
and hackathon jury. The dedication to the problem and solution spaces will vary
according to the timeframe of the event, usually 24h to 48h but may also vary from

28
24 hours to five working days (Valença et al., 2019). For instance, a one-day event
requires quicker decisions with more focus on the solution space, while a two or three-
days hackathon allows more exploration in the problem space. Once a solution is de-
veloped, it is assessed by judges via team pitches. During the focus group, participants
highlighted the relevance of understanding the evaluation process. The criteria may
be described in the event regulation or simply resides in the judges minds, requiring
an analysis of their profiles. This strategy represents a design heuristic: aligning the
project’s desirability with the criteria that will be used by organizers.
A third recommendation concerns identifying existing solutions to propose
more innovative ones. The team should not repeat existing products, but rather
improve them in certain aspects. A combination of benchmarking (i.e., competitive
testing) with the SCAMPER (Serrat, 2017) technique can help in that direction. The
first step is identifying what exists through quick Web searches or using specific en-
gines focused on digital products (Product hunt, App Store, Google Play, etc) in order
to identify solutions that already tackle the target problem. Then, the usage of SCAM-
PER can help to suggest some addition to or modification of the identified application
or solution that already exists.
A fourth recommendation concerns prototyping to validate ideas before the
team starts constructing the solution. Prototypes help to convey ideas and are
useful to have quick validation from team members and potential stakeholders (Klotins
et al., 2019). Even paper prototypes can be very helpful to design and test user inter-
faces (Snyder, 2003). In the case of the impossibility of contacting such stakeholders,
people participating in the hackathon, especially mentors (technical or domain men-
tors), can be very useful to help to validate prototypes.
As a fifth recommendation, the pitch has to attract the jury’s attention. In
hackathons, as well as in professional life, communication skills are very important.
Conveying an idea through a story and attracting the attention of your audience is
key. In addition to storytelling and rhetoric, even body language is important in order
to do a convincing pitch (Torres and Cornelissen, 2019). There are basic guidelines to
be followed, such as the pitch being short, start with a winning logline and emphasize
on one thing the presenter wants their audience to remember in that pitch (Gallo,
2018). Regardless of a storytelling format being used or not, the team needs a bold
problem statement and engage with the target audience to show the relevance of the
product that was developed and is being presented. A good project without a good
pitch would be in disadvantage to an average project with a good pitch.
Finally, we suggest that organizers should raise participants’ awareness
about design thinking. Although organizers typically do not impose any design
thinking process, when they rarely suggest any format it follows a design thinking
philosophy. As this approach is reinforced, with proper training sessions (e.g., tutori-
als previous to the event), the process is understood by participants and becomes more
repeatable or reusable. By employing design thinking as a complementary process to
support software development, participants explore their problem-solving ability to
make their outcomes more innovative. However, any process or design method should
be a support tool instead of a strategy imposed by organizers. This is important to
avoid the feeling of losing the freedom that some participants may face when forced
to adhere to imposed practices.

29
5.3. Critical Analysis
Our investigation revealed that hackathon participants hold a clear notion of the im-
portance of a user-centered approach and use many design methods without being
aware of that. However, we found a deviation of this user-centered approach when
participants put the jury members in the central perspective as a user. It was a strat-
egy to please the jury in an attempt to get more attention and better scores in their
project evaluation.
We also observed an alignment of culture and practices to what is seen in software
startup companies that can also bring advantages to the software engineering field as a
whole. Lindberg et al. (2011) supports the idea of applying design thinking to IT devel-
opment (i.e., hardware and software development) as a complementary thinking style
that can help IT development teams to produce more innovative results. Additionally,
women can enhance the design skills and innovative capacity of a team, as disruption in
research and development tend to come from more gender diverse groups (Diaz et al.,
2013). This innovative problem-solving approach could be leveraged by broadening
the participation of female members in hackathons. But these events tend to have a
low number of women participating due to its competitive and predominantly sexist
environment (Paganini and Gama, 2020). This reality of women’s underrepresentation
was reflected in our study, as the participant winners of hackathons we interviewed
were overwhelmingly men (Table 3).
In general, the awareness of design methods and processes practiced in hackathons
can generate many benefits for IT developers:
• The startup entrepreneurial process underlying a hackathon: we iden-
tified that the conception and development of software in hackathons resemble,
on a smaller scale, the software engineering approach that takes place in soft-
ware startups. Similar to what was described by interviewees, in startups the
value proposition (target audience, promised benefits and competitive features)
of the MVP is used to identify software requirements is one of the first steps
in product engineering activities (Klotins et al., 2019). The culture of pitching
(Gallo, 2018; Pincus, 2007) ideas and products is also a key characteristic in the
startup world that is also present in hackathons. In fact, the overall approach
taken by software startups is very much related to what is advocated by design
thinking and lean startup methodologies, which are not only processes but also
include tacit knowledge in the form of practices and mindsets (Müller and Thor-
ing, 2012). Therefore, hackathons foster an entrepreneurial spirit towards new
business opportunities.
• The close relationship between the hackathons and agile requirements
engineering: the design methods identified in hackathons resemble the approach
taken in the software industry to understand requirements, especially in the con-
text of software startups where agile practices take place. Practitioners use in-
terviews, surveys, observations and demonstration of prototypes to discover new
requirements, and validate existing ones (Klotins et al., 2019). On their turn,
hackathon competitors consider people from their teams and their social net-
work as requirements sources, while product value proposition determines what
should be prioritized. In addition, such reports on startups suggest these com-
panies typically using brainstorming, mock-ups and wireframe to understand
requirements and design user interfaces of products, aiming at quicker require-
ments validation, in a similar way to what teams do in hackathons.
• The possibility to foster a new mindset via hackathons: software devel-

30
 opment companies face challenges in integrating design methods and processes
into software methodologies (Ogunyemi et al., 2019). By adopting and incorpo-
rating Design Thinking methods into their software development culture, these
hackathon participants take a step forward in regards to other professionals from
the field. These participants can shift from the traditional software engineering
mindset to cultivate a new creative mindset that is critical for software com-
panies (Müller and Thoring, 2012). Hackathons are shaping a new culture of
IT engineers that are spontaneously incorporating a design thinking mindset.
Hence, hackathons act as eye-opening events that can help companies to es-
tablish a result-driven culture that promotes innovative, solution-oriented and
self-directed thinking in developers (Valença et al., 2019).

6. Final Considerations

6.1. Contribution
We believe our work helps to complement existing literature about hackathons, which
already explored some design thinking perspectives on such events but lacked details
about the ideation practices used in hackathons. Considering the context of competi-
tive focus-centric hackathons and based on common practices followed by worldwide
participants who are frequent winners, we identified and enumerated the recurring de-
sign methods and strategies that seem to lead to successful hackathon participation.
We positioned those methods under the phases of the Double Diamond design process
model. By bringing a Design Thinking perspective, we presented how these design
methods were applied following divergent and convergent thinking in different mo-
ments of this process. We also provided recommendations for successful participation
in hackathons as well as additional design and creativity methods that can be put into
practice. In addition, we draw a parallel between the dynamics of the design thinking
approach in hackathons with the software engineering practices conducted in software
startups. Our analysis revealed a shared mindset of professionals who understand the
importance of design, which is not based on random inspiration moments but rather
in the usage of methods and hard work.
By shedding light on the methods adopted throughout this structure, we provide
relevant inputs to approach the topic in future studies. We enable other researchers
to apply our process perspective and verify its results in a given scenario, via an
improvement of the case study or through other methods such as action research.

6.2. Limitations and Threats to Validity

A possible threat to internal validity lies in our restriction to the perspective of
hackathon winners. Our experience in the field made us notice the strong usage of de-
sign methods as a recurring pattern in winning groups, however only selecting winners
for interviews generated a bias in the sample. Therefore, we cannot claim that such a
design thinking approach is only followed by winners. We did not collect data to verify
that other groups not necessarily having a winning history could share such design
practices to the same or to a lesser extent than winning groups; or if perhaps there
are typical design methods (e.g., brainstorming) normally used by all groups, but not
necessarily situated in an informal Design Thinking process. By interviewing the other
set of participants of these events, we could speculate about possible uses of design

31
methods and their impact of the results presented by the teams. Such correlation,
with a cause-consequence clause (e.g., a given design process generates a given result),
is rather associated with a quantitative analysis of the phenomenon. Another limita-
tion of our work is that we investigated only the context of competitive focus-centric
hackathons. Other settings, such as non-competitive events or hackathons that are less
innovation-oriented and more social-focused, which are mostly multidisciplinary, may
lead to different sets of practices or design methods employed by participants.
We also acknowledge a possible threat to construct validity. Our research did
not follow an exploratory stance when we would find out what was going on and seek
new insights into a specific emergent phenomenon. Instead, we aimed at depicting
a design process we perceived as being applicable to many hackathons. Hence, we
may have translated some of our initial hypotheses into interview questions, but the
rounds of discussions among authors during the construction of the interview guide
helped to avoid leading questions or questions that were very specific to what we have
experienced in hackathons. We tried to keep them neutral (e.g., participants were not
supposed to choose among options) and rather coarse-grained. This was a means not
to restrict or direct participants.
The conclusion validity of our research may have been threatened by two main
factors. First, the live coding procedure provided us with a non-exhaustive transcrip-
tion, i.e. we may have neglected important data while hearing the audio records from
interviews. To reduce such threat, we guaranteed that the researchers in charge of
conducting the interviews were also responsible for listening to the audio files. Their
familiarity with the data enabled us to guarantee a proper selection of important ex-
cerpts. Second, part of the authors had a large experience in organizing hackathons
and a strong background in Design Thinking. This shared knowledge on that topic
among the authors had a risk of confirmation bias to support the initial hypothesis.
Hence, we faced the risk of introducing such experience in the results and general in-
sights, i.e. our interpretation of the phenomenon would also stem from non-systematic
observations naturally performed during other hackathons. To reduce the chances of
having different types of researcher bias (e.g., confirmation bias, leading questions) on
top of the actual data, we promoted discussions among researchers during the analysis
or right after deriving a set of conclusions. This approach allowed us to have a less
biased understanding of interviewees discourse, with an analysis based on objective
data, although we acknowledge that it may have not completely eliminated researcher
bias since all authors shared a Design Thinking perspective that may have persisted
despite the discussions. Moreover, it mitigated translation issues, since we had four
different idioms in our dataset. We decided to present our results to a group of people
experienced with hackathons via a focus group as a means to verify the process gen-
erated from collected evidence. This whole procedure strengthened our data analysis.
The fact of this focus group be composed of people from the professional network of
the authors may have introduced personal bias with a tendency to agreement on the
proposed structure of phases and respective techniques situated on each phase. Never-
theless, they were asked to criticize the model which resulted in suggesting additional
techniques.

6.3. Future Work

We plan to conduct the following studies in future research:

• Performing an ethnographic study: we plan to perform ethnographic studies

32
 to observe in loco how hackathon participants interact with design methods. This
method is essential to address the common lack of details in an interview, mainly
because participants may forget or simply omit (considering a given information
as irrelevant) practices that could enrich our analysis. Hence, ethnography will
allow us to observe teams dynamics in a hackathon. In particular, it can raise
our understanding of whether methods were used in a correct manner or with
needed changes due to constraints of the hackathon. By suggesting the overar-
ching double diamond design process model as a starting point to different and
specific forms of hackathons (e.g., educational hackathons conducted by univer-
sity professors, or civic hackathons held by public organizations), we can analyze
the relationship between specific demographic factors (e.g., duration and loca-
tion of the event, background of participants, and even gender of participants) –
not approached by this paper – and the design methods used by hackathon par-
ticipants. Such a sociological view of the phenomenon will contribute to studies
on human factors in software engineering.
• Extend the study introducing a gender perspective: in the first round
of data collection, our pool of interviewees was majorly formed by male par-
ticipants (only one woman among the thirty-seven participants). The same for
the focus group, when we had six men in a group of eight participants. The
literature denotes that teams with a higher proportion of female members have
better results on a collective intelligence factor than other teams. (Woolley et al.,
2010). Hence, it is important to have different profiles properly represented in
the dataset. A similar study that brings the gender dimension could explore the
creativity factor that is involved on gender-diverse teams and what role it plays
on the design methods that are used by those teams in comparison to male-only
groups.

References

Ambrose, Gavin; and Paul Harris (2009). Basics design 08: design thinking. Bloomsbury
Publishing.
Andler, Nicolai (2016). Tools for project management, workshops and consulting: a must-have
compendium of essential tools and techniques. John Wiley & Sons.
Angarita, Maria Angelica Medina; and Alexander Nolte (2020). What do we know about
hackathon outcomes and how to support them?–A systematic literature review. , 2020. pp.
50–64.
Angelidis, Pantelis; Leslie Berman; Maria de la Luz Casas-Perez; Leo Anthony Celi; George E
Dafoulas; Alon Dagan; Braiam Escobar; Diego M Lopez; Julieta Noguez; Juan Sebastian
Osorio-Valencia; et al. (2016). The hackathon model to spur innovation around global
mHealth. Journal of medical engineering & technology, vol. 40, no. 7-8, pp. 392–399.
Avalos, Manuel; Victor M Larios; Pablo Salazar; and Rocio Maciel (2017). Hackathons,
semesterathons, and summerathons as vehicles to develop smart city local talent that via
their innovations promote synergy between industry, academia, government and citizens. ,
2017. pp. 1–6.
Brenner, Walter; Falk Uebernickel; and Thomas Abrell (2016). Design thinking as mindset,
process, and toolbox. Design thinking for innovation. Springer, pp. 3–21.
Briscoe, Gerard (2014). : . Digital innovation: The hackathon phenomenon.
Brown, Tim (2009). Change by Design: How Design Thinking Transforms Organizations and
Inspires Innovation. HarperCollins.
Christensen, Clayton M; Taddy Hall; Karen Dillon; and David S Duncan (2016). Know your
customers’ jobs to be done. Harvard Business Review, vol. 94, no. 9, pp. 54–62.

33
Cooper, Alan; et al. (2004). The inmates are running the asylum:[Why high-tech products drive
us crazy and how to restore the sanity], Vol. 2. Sams Indianapolis.
Cross, Nigel (2011). Design thinking: Understanding how designers think and work. Berg.
Davies, Martin (2011). Concept mapping, mind mapping and argument mapping: what are
the differences and do they matter? Higher education, vol. 62, no. 3, pp. 279–301.
Da Silva, Tiago Silva; Angela Martin; Frank Maurer; and Milene Silveira (2011). User-centered
design and agile methods: a systematic review. , 2011. pp. 77–86.
Dı́az-Garcı́a, Cristina; González-Moreno, Angela; and Jose Sáez-Martı́nez, Francisco (2013).
Gender diversity within R&D teams: Its impact on radicalness of innovation. Innovation.
Taylor & Francis, vol. 15. no. 2 pp. 149–160
DiSalvo, Carl; Melissa Gregg; and Thomas Lodato (2014). Building belonging. interactions,
vol. 21, no. 4, pp. 58–61.
Ferreira, Bruna; Williamson Silva; Edson A Oliveira Jr; and Tayana Conte (2015). Designing
Personas with Empathy Map. , 2015, Vol. 152.
Filippova, Anna; Erik Trainer; and James D Herbsleb (2017). From diversity by numbers to
diversity as process: supporting inclusiveness in software development teams with brain-
storming. , 2017. pp. 152–163.
Flores, Myrna; Matic Golob; Doroteja Maklin; Martin Herrera; Christopher Tucci; Ahmed Al-
Ashaab; Leon Williams; Adriana Encinas; Veronica Martinez; Mohamed Zaki; et al. (2018).
How can hackathons accelerate corporate innovation? , 2018. pp. 167–175.
Flus, Meagan; and Ada Hurst (2021). Design at hackathons: new opportunities for design
research. Design Science, vol. 7.
Frey, Frank J; and Michael Luks (2016). The innovation-driven hackathon: one means for
accelerating innovation. , 2016. pp. 1–11.
Gallo, Carmine (2018), The Art of the Elevator Pitch.
Gama, Kiev; Breno Alencar Gonçalves; and Pedro Alessio (2018). Hackathons in the formal
learning process. , 2018. pp. 248–253.
Granados, Cristian; and Montserrat Pareja-Eastaway (2019). How do collaborative practices
contribute to innovation in large organisations? The case of hackathons. Innovation, vol.
21, no. 4, pp. 487–505.
Grönman, Satu; Lindfors, Eila (2021). The Process Models of Design Thinking: A Literature
Review and Consideration from the Perspective of Craft, Design and Technology Education.
Techne serien-Forskning i slöjdpedagogik och slöjdvetenskap, vol. 28, no. 2, pp. 110–118.
Haig, Brian D (2018). An abductive theory of scientific method. Method Matters in Psychology.
Springer, pp. 35–64.
Hanington, Bruce; and Bella Martin (2012). Universal methods of design: 100 ways to re-
search complex problems, develop innovative ideas, and design effective solutions. Rockport
Publishers.
Hehn, Jennifer; Daniel Mendez; Falk Uebernickel; Walter Brenner; and Manfred Broy (2019).
On Integrating Design Thinking for a Human-centered Requirements Engineering. : , IEEE
Software.
Johnson, Peter; and Pamela Robinson (2014). Civic hackathons: Innovation, procurement, or
civic engagement? Review of policy research, vol. 31, no. 4, pp. 349–357.
Klotins, Eriks; Michael Unterkalmsteiner; and Tony Gorschek (2019). Software engineering in
start-up companies: An analysis of 88 experience reports. Empirical Software Engineering,
vol. 24, no. 1, pp. 68–102.
Kollwitz, Christoph; and Barbara Dinter (2019). What the hack?–towards a taxonomy of
hackathons. , 2019. pp. 354–369.
Komssi, Marko; Danielle Pichlis; Mikko Raatikainen; Klas Kindström; and Janne Järvinen
(2014). What are hackathons for? IEEE Software, vol. 32, no. 5, pp. 60–67.
Lawson, Bryan (2006). How designers think. Routledge.
Lee, Melissa; Esteve Almirall; and Jonathan Wareham (2015). Open data and civic apps: first-
generation failures, second-generation improvements. Communications of the ACM, vol. 59,
no. 1, pp. 82–89.

34
Lindberg, Tilmann; Christoph Meinel; and Ralf Wagner (2011). Design thinking: A fruitful
concept for IT development? Design thinking. Springer, pp. 3–18.
Litcanu, Marcela; Octavian Prostean; Cosmin Oros; and Alin Vasile Mnerie (2015). Brain-
writing vs. Brainstorming case study for power engineering education. Procedia-Social and
Behavioral Sciences, vol. 191 pp. 387–390.
Lodato, Thomas James; and Carl Disalvo (2015). Issueoriented hackathons as ad-hoc design
events. , 2015. pp. 328–336.
Lyndon, Mataroria P; Cassidy, Michael P; Celi, Leo Anthony; Hendrik, Luk; Kim, Yoon Jeon;
Gomez, Nicholas; Baum, Nathaniel; Bulgarelli, Lucas; Paik, Kenneth E; Dagan, Alon (2018).
Hacking hackathons: preparing the next generation for the multidisciplinary world of health-
care technology. International Journal of Medical Informatics, vol. 112 pp. 1–5 Elsevier.
McGowan, Bethany (2019). The role of the university library in creating inclusive healthcare
hackathons: A case study with design-thinking processes. IFLA journal, vol. 45 no. 3. pp.
246–253 Sage Publications Sage UK: London, England.
Michalko, Michael (2006). Thinkertoys: A handbook of creative-thinking techniques. Random
House Digital, Inc.
Mueller-Roterberg, Christian (2018), Handbook of Design Thinking.
Müller, Roland M; and Katja Thoring (2012). Design thinking vs. lean startup: A comparison
of two user-driven innovation strategies. Leading through design, vol. 151 pp. 91–106.
Nandi, Arnab; and Meris Mandernach (2016). Hackathons as an informal learning platform. ,
2016. pp. 346–351.
Ogunyemi, Abiodun Afolayan; Lamas, David; Lárusdóttir, Marta Kristin; Loizides, Fernando
(2019). A systematic mapping study of HCI practice research. : , International Journal of
Human–Computer Interaction v. 35 no. 16 pp. 1461–1486.
Olesen, Jeanette Falk; Nicolai Brodersen Hansen; and Kim Halskov (2018). Four factors in-
forming design judgement at a hackathon. , 2018. pp. 473–483.
Osborn, Alex (2012). Applied Imagination-Principles and Procedures of Creative Writing.
Read Books Ltd.
Osterwalder, Alexander; and Yves Pigneur (2010). Business model canvas. : , Self published.
Last.
Paganini, Lavinia; and Gama, Kiev (2020). Engaging women’s participation in hackathons: A
qualitative study with participants of a female-focused hackathon International Conference
on Game Jams, Hackathons and Game Creation Events, 2020. pp. 8–15.
Page, Finlay; Sylvester Sweeney; Fraser Bruce; Seaton Baxter; et al. (2016). The Use of the
“Hackathon” in Design Education: An Opportunistic Exploration. , 2016. pp. 246–251.
Parameswaran, Uma D; Jade L Ozawa-Kirk; and Gwen Latendresse (2019). To live (code) or
to not: A new method for coding in qualitative research. : , Qualitative Social Work.
Parkinson, David (2016). : . Engaging Design Pitches: Storytelling Approaches and their
Impacts.
Pe-Than, Ei Pa Pa; Alexander Nolte; Anna Filippova; Christian Bird; Steve Scallen; and
James Herbsleb (2020). Corporate hackathons, how and why? A multiple case study of
motivation, projects proposal and selection, goal setting, coordination, and outcomes. : ,
Human–Computer Interaction pp. 1–33.
Pe-Than, Ei Pa Pa; Alexander Nolte; Anna Filippova; Christian Bird; Steve Scallen; and
James D Herbsleb (2018). Designing Corporate Hackathons With a Purpose: The Future of
Software Development. IEEE Software, vol. 36, no. 1, pp. 15–22.
Pincus, Aileen (2007). The perfect (elevator) pitch. Bloomberg Business Week, vol. 18.
Poncette, Akira-Sebastian; Rojas, Pablo-David; Hofferbert, Joscha; Sosa, Alvaro Valera;
Balzer, Felix; and Braune, Katarina (2020). Hackathons as stepping stones in health care
innovation: case study with systematic recommendations. Journal of Medical Internet Re-
search, vol. 22 no. 3.
Porras, Jari; Jayden Khakurel; Jouni Ikonen; Ari Happonen; Antti Knutas; Antti Herala; and
Olaf Drögehorn (2018). Hackathons in software engineering education: lessons learned from
a decade of events. , 2018. pp. 40–47.

35
Raatikainen, Mikko; Marko Komssi; Vittorio Dal Bianco; Klas Kindstöm; and Janne Järvinen
(2013). Industrial experiences of organizing a hackathon to assess a device-centric cloud
ecosystem. , 2013. pp. 790–799.
Rhorbach, B (1969). Kreative nach regeln: Methode 635, eine neue technik zum losen von
problemen. Absatzwirtschaft, vol. 12 pp. 73–75.
Richard, Gabriela T; Yasmin B Kafai; Barrie Adleberg; and Orkan Telhan (2015). StitchFest:
Diversifying a College Hackathon to broaden participation and perceptions in computing. ,
2015. pp. 114–119.
Ries, Eric (2011). The lean startup: How today’s entrepreneurs use continuous innovation to
create radically successful businesses. Currency.
Rosell, Bard; Shiven Kumar; and John Shepherd (2014). Unleashing innovation through in-
ternal hackathons. , 2014. pp. 1–8.
Safarov, Igbal; Albert Meijer; and Stephan Grimmelikhuijsen (2017). Utilization of open
government data: A systematic literature review of types, conditions, effects and users.
Information Polity, vol. 22, no. 1, pp. 1–24.
Saldaña, Johnny (2013). The coding manual for qualitative researchers. SAGE Publications
Limited.
Saravi, Sara; Demetrios Joannou; Roy S Kalawsky; MRN King; I Marr; M Hall; PCJ Wright;
R Ravindranath; and A Hill (2018). A Systems Engineering Hackathon–A Methodology
Involving Multiple Stakeholders to Progress Conceptual Design of a Complex Engineered
Product. IEEE Access, vol. 6 pp. 38399–38410.
Schroeer, Bernd; Andreas Kain; and Udo Lindemann (2010). Supporting creativity in concep-
tual design: Method 635-extended. , 2010.
Serrat, Olivier (2017). The SCAMPER technique. Knowledge Solutions. Springer, pp. 311–314.
Silver, Julie K; David S Binder; Nevena Zubcevik; and Ross D Zafonte (2016). Healthcare
hackathons provide educational and innovation opportunities: a case study and best practice
recommendations. Journal of medical systems, vol. 40, no. 7, pp. 1–7.
Sjøberg, Dag IK; Tore Dybå; Bente CD Anda; and Jo E Hannay (2008). Building theories
in software engineering. Guide to advanced empirical software engineering. Springer, pp.
312–336.
Smolander, Kari; Paul Grünbacher; Sami Hyrynsalmi; and Slinger Jansen (2020). ACM SIG-
SOFT International Workshop on Software-Intensive Business: Start-ups, Platforms, and
Ecosystems. ACM SIGSOFT Software Engineering Notes, vol. 45, no. 1, pp. 18–20.
Snyder, Carolyn (2003). Paper prototyping: The fast and easy way to design and refine user
interfaces. Morgan Kaufmann.
Suominen, Anu Helena; Jari Jussila; Teppo Lundell; Marko Mikkola; and Heli Aramo-Immonen
(2018). Educational Hackathon: Innovation Contest for Innovation Pedagogy. , 2018.
Tapia-González, Gabriela (2020). Educational Marketing and Hackathon for Candidate Stu-
dent Recruitment. , 2020. pp. 431–437.
Taylor, Nick; and Loraine Clarke (2018). Everybody’s hacking: participation and the main-
streaming of hackathons. , 2018. pp. 1–2.
Thomer, Andrea K; Michael B Twidale; Jinlong Guo; and Matthew J Yoder (2016). Co-
designing scientific software: Hackathons for participatory interface design. , 2016. pp.
3219–3226.
Torres, Nicole; and Joep Cornelissen (2019), When You Pitch an Idea, Gestures Matter More
Than Words.
Trainer, Erik H; Arun Kalyanasundaram; Chalalai Chaihirunkarn; and James D Herbsleb
(2016). How to hackathon: Socio-technical tradeoffs in brief, intensive collocation. , 2016.
pp. 1118–1130.
Tschimmel, Katja (2012). Design Thinking as an effective Toolkit for Innovation. , 2012. p. 1.
Valença, George; Nycolas Lacerda; Maria Eduarda Rebelo; Carina Alves; and Cleidson RB
de Souza (2019). On the Benefits of Corporate Hackathons for Software Ecosystems–A
Systematic Mapping Study. , 2019. pp. 367–382.
Van De, Andrew; and Andre L Delbecq (1971). Nominal versus interacting group processes

36
 for committee decision-making effectiveness. Academy of Management Journal, vol. 14, no.
2, pp. 203–212.
Warner, Jeremy; and Philip J. GuoIn: J. Tenenberg, D. Chinn, J. Sheard, and L. Malmi (eds.).
, 2017. pp. 254–262.
Woolley, Anita Williams; Christopher F Chabris; Alex Pentland; Nada Hashmi; and Thomas W
Malone (2010). Evidence for a collective intelligence factor in the performance of human
groups. science, vol. 330, no. 6004, pp. 686–688.
Wyngaard, Jane; Heather Lynch; Jaroslaw Nabrzyski; Allen Pope; and Shantenu Jha (2017).
Hacking at the divide between polar science and HPC: using hackathons as training tools.
, 2017. pp. 352–359.
Zainal, Zaidah (2007). Case study as a research method. Jurnal Kemanusiaan, vol. 5, no. 1,.
Zapico, Jorge Luis; Daniel Pargman; Hannes Ebner; and Elina Eriksson (2013). Hacking
sustainability: Broadening participation through green hackathons. , 2013.
Zwicky, Fritz (1967). The morphological approach to discovery, invention, research and con-
struction. New methods of thought and procedure. Springer, pp. 273–297.
British Design Council (2019), Double Diamond Design Process.
Hasso-Plattner Institute (2020), Hasso-Plattner Institute.

About the authors

Kiev Gama is an associate professor of computer science at the Universidade Federal

de Pernambuco (UFPE), Brazil. He received his PhD in computer science from the
University of Grenoble, France. His research interests include several aspects of time-
bounded collaborative events (e.g., hackathons, game jams) within contexts such as
open innovation and education.
George Valença is a professor of software engineering at Universidade Federal Ru-
ral de Pernambuco (UFRPE), Brazil. He received an MSc and a PhD in Computer
Science from Universidade Federal de Pernambuco (UFPE), Brazil. He researches on
open innovation and business process management approaching topics such as power
relationships, data protection and corporate hackathons.
Pedro Alessio is a professor at the Department of Graphic Expression at Universi-
dade Federal de Pernambuco (UFPE), Brazil. He holds a bachelor’s degree in Industrial
Design from the Federal University of Pernambuco, a master’s degree and a PhD in
Informatics from the Conservatoire National des arts et métiers (CNAM), France.
Rafael Formiga holds Master’s degree in Design from the Universidade Federal de
Pernambuco (UFPE), Brazil. He is a product designer at VTEX. His main research
focuses on design methods, product design and UX research. He works with digital
design education, building bridges between academia and industry.
André Neves is an associate professor at the Universidade Federal de Pernambuco
(UFPE), Brazil. He has a bachelor’s degree in Industrial Design from the Universidade
Federal da Paraı́ba (UFPB), and a Master’s and PhD in Computer Science from
UFPE. He is experienced with the Design of computer systems, working mainly in the
investigation, development and application of methods and design techniques as an
instrument of innovation.
Nycolas Lacerda graduated in computer science from the Universidade Federal Rural
de Pernambuco (UFRPE), Brazil. His main research focus is on corporate hackathon
organizations, having participated in a research group ”Innovation for Business and
Software Evolution - IBSE” as a student researcher. He currently works as an IT
consultant.

37
Appendix A. Interview Guide

(1) Demographic Information

• Age
• Educational background
• Time of experience in the area
• Experience with hackathons (total participations, prizes won)
(2) Hackathon Dynamics
(a) Groups
• How groups were formed?
• Was there interdisciplinaraity?
• What was you role in the group?
(b) Creative process
• Did the organizers / mentors suggest the use of any creative technique
for idea generation?
• Did you know / use any specific Design Thinking methods?
• What was the step-by-step process that your team followed to design
and develop applications in hackathons?
• Did you work with someone from design domain? How did he/she in-
fluence the project?
• How were the tasks managed?
• Did you already have any idea for the hackathon project? If yes, how
was it generated?
• Could you identify or have access to the target audience or domain
experts?
• Were there any guidelines to generate and select the application solu-
tion alternatives?
• Did you research about potential competitors of your solution?
• Did you think of a differentiation strategy in relation to competitors?
• Did you do any low fidelity (lo-fi) prototype?
• Could you do any validation of the prototype or application with users
or experts?
• How tests were handled?
• How the pitch was made?

38