Download to read offline
![Learning from the Past:
a Process Recommendation System for Video Game Projects
using Postmortems Experiences
C. Politowski, L. Fontoura, F. Petrillo and Y. Gu´eh´eneuc
Departamento de Computac¸˜ao Aplicada (DCOM), Universidade Federal de Santa Maria, Santa Maria, RS, Brazil
Department of Computer Science & Software Engineering Concordia University, Montr´eal Quebec H3G 1M8, Canada
{c polito, yann-gael.gueheneuc}@encs.concordia.ca, lisandra@inf.ufsm.br, fabio@petrillo.com
Introduction
The video game industry is a multi-billionaire market,
which revenues have been growing up through the
years (US$118 billions in 2019). Video game devel-
opment is a highly complex activity and face manage-
ment and processes problems, like “unrealistic scope”
and “feature creep”.
Our work consisted in building a recommendation
system to suggest software processes, using the con-
text and the similarity degree of previous video
game projects, that is:
Figure 1: Summary of the approach.
• The construction of the processes database with
characteristics extracted from postmortems;
• The definition of the video game context to char-
acterize video game projects;
• The building of a recommendation system that
suggests the software process using the defined
context;
• Finally, the evaluation of the recommendation sys-
tem – quantitatively, qualitatively and by perform-
ing a case study with a video game development
team.
Method
Our approach is inspired by the “design concerns” re-
lated to build recommendation systems described by
Robillard et al. (2014). It consists in four challenges:
1. Data preprocessing. We analyzed the data gath-
ered from video game projects by reading 100 post-
mortems, collecting information about the develop-
ment processes, activities, and team characteristics.
2. Capturing context. We encoded the data gathered
from postmortems in JSON format (Listing 1) and
put it in a database to perform queries. The JSON
structure consists of five key:value for each process
element. A process is thus a collection of N ele-
ments.
3. Producing recommendations. The project con-
texts obtained from different postmortems are input
for Principal Component Analysis (PCA) to iden-
tify and quantify the relations in large sets of vari-
ables (Figure 2). The result of applying PCA on the
project context is an array of similar projects. With
this list, we query the database for all elements re-
lated to the projects and perform a merge, respect-
ing the elements’ phase.
4. Presenting the recommendations. We choose to
display recommended processes graphically (Fig-
ure 3).
Results
We analyzed and extracted the data from 55 post-
mortems to populate our JSON database. Every infor-
mation regarding the development process was high-
lighted and tagged with an index.
Listing 1: JSON structure used to storage process’ elements.
{
"game" : "Slow Down, Bull",
"phase" : "activities",
"element" : "exploration phase",
"desc" : "We were able to iterate
through a ton of different experiments,
many of which were discarded failures,
but which paved the path for the
strongest mechanics in the game",
"prob" : false
}
With the projects contexts, we ran a script that uses
PCA biplot to show the samples’ variability through
a graph, that is, a graph showing the similarity of the
55 projects analyzed.
Figure 2: Biplot PCA, which is a representation of the correla-
tion between the samples. The closest the dots, the more similar
the projects.
With a list of similar projects, we queried the
database for their elements. We then joined the el-
ements resulting in a new process. We assembled the
elements in a process manually, because we did not
stored the processes flows.
We used Graphiz with the visualization template to
display the new recommended process. We converted
the new recommended process in a textual DOT file,
which can be compiled into an image file.
PRE-PRODUCTIONPRE-PRODUCTION
[Vanishing Point] This slowed
development greatly, and entire weeks
were spent bumbling around the engine
trying to figure out how to make the
game with these tools. That delayed that
early, exploratory development phase
where you are figuring out how to bring
out the fun in what you have,
prototyping new mechanics and
discovering what works lasted all the
way to the end of the first semester.
[Vanishing Point] When I had it in
my hands at start of fall and was asked
to build levels and new mechanics out of
it, I never really felt the mechanic
granted me enough room to be creative
with design. It was a bit of a shallow
foundation to build off of for such a
substantial project, but we ended up
doing the best we could.exploration
phase
[Vanishing Point] The creative
direction for the project was unclear
throughout its first half, aside from a
high?level social goal; questions of
tone, mood, theme, and purpose were left
very ambiguous.
prototypingrequirements
ans constraints
[Jetpack High] In the first prototype
of the game Jetpack High, I imagined a
game in top view in which you must move
a flying saucer and avoid obstacles to
go from one point to another. In the
manner of ball games and mazes. I
prototyped the gameplay with graphics so
ugly that Jeremy did not understand the
intent and produced me a first stage
side view of a character who moves
jetpack. A kind of puzzle game
plateformer and nested.
[Vanishing Point] Task lists and
effective project management does not
only make things operate smoothly and
reduce blocks, but it also serves to
motivate people by making progress
visibly quantifiable and boost team
morale. Just because you are intimately
familiar with what needs to be done with
a game and the priority of each
individual task does not entail that
everyone else is.
milestones
planning
[Catlateral Damage] Catlateral
Damage was originally created for the
7DFPS game jam in August of 2013, along
with games like Probably Archery,
SUPERHOT, and Notch's Shambles.
planning
documentation
Figure 3: Preproduction phase of the recommended process for
the game Slow down, Bull.
Evaluation
To quantitatively assess the recommendation system
we chose two metrics to evaluate our recommenda-
tion system: Correctness and Coverage.
Table 1: Correctness and Coverage results.
Precision Recall Accuracy Catalog W. Catalog
#1 42,11% 10,09% 63,49% 8,32% 10,09%
#2 17,97% 13,07% 71,37% 14,02% 13,07%
#3 23,81% 5,52% 77,57% 4,60% 5,52%
#4 42,42% 3,76% 57,92% 3,61% 3,76%
We performed the qualitative validation using a set
of questions in a on-line form that we sent to project
authors and–or developers.
“I think the tool that you describe has some really
interesting implications from an analytical stand-
point (...)”
We performed a case study by interviewing a video
game developer team.
TRU
NOV
SER
UTI
RIS
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
3
0
1
1
1
6
8
7
3
14
7
11
12
9
7
8
11
9
13
9
7
1
2
5
0
1 - Strongly disagree 2 - Disagree 3 - Neutral 4 - Agree 5 - Strongly agree
Figure 4: Case study results. Metrics are: Trustworthiness,
Novelty, Serendipity, Utility and Risk
Conclusions
This work thus presents three main contributions. It
first describes the creation of a database of game de-
velopment processes from the analysis of 55 post-
mortems. The second identifies video game project
characteristics, like team attributes and technical de-
tails, to model projects contexts that work as input
for the recommendation system. The third and final
one describes and validates a recommendation sys-
tem capable of generating processes based on previ-
ous projects with similar contexts.
Acknowledgements
We give our thanks to the authors of the postmortems for sharing
their cases as well for their feedbacks. This work has been sup-
ported by the National Council for Scientific and Technological
Development of Brazil.
References
[1] Iman Avazpour, Teerat Pitakrat, Lars Grunske, and John
Grundy. Dimensions and metrics for evaluating recommen-
dation systems. In Recommendation systems in software en-
gineering. Springer, 2014.
[2] F´abio Petrillo, Marcelo Pimenta, Francisco Trindade, and
Carlos Dietrich. What went wrong? a survey of problems
in game development. 2009.
[3] Martin P Robillard, Walid Maalej, Robert J Walker, and
Thomas Zimmermann. Recommendation systems in software
engineering. Springer Science & Business, 2014.](https://image.slidesharecdn.com/cristiano-181208031255/75/Cristiano-Politowski-1-2048.jpg)
Uploaded byPtidej Team
Cristiano Politowski
The document summarizes a recommendation system that suggests software processes for video game projects based on similarities to past projects. The system analyzes over 100 postmortems from previous games to build a database of development processes and project contexts. It uses principal component analysis to identify similar past projects and recommends a process by combining elements from similar projects' processes. The system was evaluated both quantitatively based on correctness and coverage metrics and qualitatively through surveys and a case study with a developer team.
More Related Content
Similar to Cristiano Politowski
Cristiano Politowski
- 1. Learning from thePast: a Process Recommendation System for Video Game Projects using Postmortems Experiences C. Politowski, L. Fontoura, F. Petrillo and Y. Gu´eh´eneuc Departamento de Computac¸˜ao Aplicada (DCOM), Universidade Federal de Santa Maria, Santa Maria, RS, Brazil Department of Computer Science & Software Engineering Concordia University, Montr´eal Quebec H3G 1M8, Canada {c polito, yann-gael.gueheneuc}@encs.concordia.ca, [email protected], [email protected] Introduction The video game industry is a multi-billionaire market, which revenues have been growing up through the years (US$118 billions in 2019). Video game devel- opment is a highly complex activity and face manage- ment and processes problems, like “unrealistic scope” and “feature creep”. Our work consisted in building a recommendation system to suggest software processes, using the con- text and the similarity degree of previous video game projects, that is: Figure 1: Summary of the approach. • The construction of the processes database with characteristics extracted from postmortems; • The definition of the video game context to char- acterize video game projects; • The building of a recommendation system that suggests the software process using the defined context; • Finally, the evaluation of the recommendation sys- tem – quantitatively, qualitatively and by perform- ing a case study with a video game development team. Method Our approach is inspired by the “design concerns” re- lated to build recommendation systems described by Robillard et al. (2014). It consists in four challenges: 1. Data preprocessing. We analyzed the data gath- ered from video game projects by reading 100 post- mortems, collecting information about the develop- ment processes, activities, and team characteristics. 2. Capturing context. We encoded the data gathered from postmortems in JSON format (Listing 1) and put it in a database to perform queries. The JSON structure consists of five key:value for each process element. A process is thus a collection of N ele- ments. 3. Producing recommendations. The project con- texts obtained from different postmortems are input for Principal Component Analysis (PCA) to iden- tify and quantify the relations in large sets of vari- ables (Figure 2). The result of applying PCA on the project context is an array of similar projects. With this list, we query the database for all elements re- lated to the projects and perform a merge, respect- ing the elements’ phase. 4. Presenting the recommendations. We choose to display recommended processes graphically (Fig- ure 3). Results We analyzed and extracted the data from 55 post- mortems to populate our JSON database. Every infor- mation regarding the development process was high- lighted and tagged with an index. Listing 1: JSON structure used to storage process’ elements. { "game" : "Slow Down, Bull", "phase" : "activities", "element" : "exploration phase", "desc" : "We were able to iterate through a ton of different experiments, many of which were discarded failures, but which paved the path for the strongest mechanics in the game", "prob" : false } With the projects contexts, we ran a script that uses PCA biplot to show the samples’ variability through a graph, that is, a graph showing the similarity of the 55 projects analyzed. Figure 2: Biplot PCA, which is a representation of the correla- tion between the samples. The closest the dots, the more similar the projects. With a list of similar projects, we queried the database for their elements. We then joined the el- ements resulting in a new process. We assembled the elements in a process manually, because we did not stored the processes flows. We used Graphiz with the visualization template to display the new recommended process. We converted the new recommended process in a textual DOT file, which can be compiled into an image file. PRE-PRODUCTIONPRE-PRODUCTION [Vanishing Point] This slowed development greatly, and entire weeks were spent bumbling around the engine trying to figure out how to make the game with these tools. That delayed that early, exploratory development phase where you are figuring out how to bring out the fun in what you have, prototyping new mechanics and discovering what works lasted all the way to the end of the first semester. [Vanishing Point] When I had it in my hands at start of fall and was asked to build levels and new mechanics out of it, I never really felt the mechanic granted me enough room to be creative with design. It was a bit of a shallow foundation to build off of for such a substantial project, but we ended up doing the best we could.exploration phase [Vanishing Point] The creative direction for the project was unclear throughout its first half, aside from a high?level social goal; questions of tone, mood, theme, and purpose were left very ambiguous. prototypingrequirements ans constraints [Jetpack High] In the first prototype of the game Jetpack High, I imagined a game in top view in which you must move a flying saucer and avoid obstacles to go from one point to another. In the manner of ball games and mazes. I prototyped the gameplay with graphics so ugly that Jeremy did not understand the intent and produced me a first stage side view of a character who moves jetpack. A kind of puzzle game plateformer and nested. [Vanishing Point] Task lists and effective project management does not only make things operate smoothly and reduce blocks, but it also serves to motivate people by making progress visibly quantifiable and boost team morale. Just because you are intimately familiar with what needs to be done with a game and the priority of each individual task does not entail that everyone else is. milestones planning [Catlateral Damage] Catlateral Damage was originally created for the 7DFPS game jam in August of 2013, along with games like Probably Archery, SUPERHOT, and Notch's Shambles. planning documentation Figure 3: Preproduction phase of the recommended process for the game Slow down, Bull. Evaluation To quantitatively assess the recommendation system we chose two metrics to evaluate our recommenda- tion system: Correctness and Coverage. Table 1: Correctness and Coverage results. Precision Recall Accuracy Catalog W. Catalog #1 42,11% 10,09% 63,49% 8,32% 10,09% #2 17,97% 13,07% 71,37% 14,02% 13,07% #3 23,81% 5,52% 77,57% 4,60% 5,52% #4 42,42% 3,76% 57,92% 3,61% 3,76% We performed the qualitative validation using a set of questions in a on-line form that we sent to project authors and–or developers. “I think the tool that you describe has some really interesting implications from an analytical stand- point (...)” We performed a case study by interviewing a video game developer team. TRU NOV SER UTI RIS 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 3 0 1 1 1 6 8 7 3 14 7 11 12 9 7 8 11 9 13 9 7 1 2 5 0 1 - Strongly disagree 2 - Disagree 3 - Neutral 4 - Agree 5 - Strongly agree Figure 4: Case study results. Metrics are: Trustworthiness, Novelty, Serendipity, Utility and Risk Conclusions This work thus presents three main contributions. It first describes the creation of a database of game de- velopment processes from the analysis of 55 post- mortems. The second identifies video game project characteristics, like team attributes and technical de- tails, to model projects contexts that work as input for the recommendation system. The third and final one describes and validates a recommendation sys- tem capable of generating processes based on previ- ous projects with similar contexts. Acknowledgements We give our thanks to the authors of the postmortems for sharing their cases as well for their feedbacks. This work has been sup- ported by the National Council for Scientific and Technological Development of Brazil. References [1] Iman Avazpour, Teerat Pitakrat, Lars Grunske, and John Grundy. Dimensions and metrics for evaluating recommen- dation systems. In Recommendation systems in software en- gineering. Springer, 2014. [2] F´abio Petrillo, Marcelo Pimenta, Francisco Trindade, and Carlos Dietrich. What went wrong? a survey of problems in game development. 2009. [3] Martin P Robillard, Walid Maalej, Robert J Walker, and Thomas Zimmermann. Recommendation systems in software engineering. Springer Science & Business, 2014.