0% found this document useful (0 votes)

32 views17 pages

Ferro Postprint Gea2 2021 PDF

Uploaded by

beni marce

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

32 views17 pages

Ferro Postprint Gea2 2021 PDF

Uploaded by

beni marce

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 17

This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TLT.2022.3143519, IEEE
Transactions on Learning Technologies
IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. XX, NO. X, XXXXXX 20XX 1

Gea2: A Serious Game for Technology-Enhanced

Learning in STEM
Lauren S. Ferro, Francesco Sapio, Massimo Mecella, Marco Temperini, and Annalisa Terracina

Abstract—Education and training are among the fields taking such challenges/opportunities is the use of gamification, game-
advantage of serious games (SGs). In this paper we present based learning (GBL) [4], [5], and digital educational serious
Gea 2: A New Earth, a digital SG developed as an immersive games. They all surged ahead in the last decades as means
three-dimensional virtual learning environment, integrating sev-
eral educational resources, and including multimedia learning to face the evolution of learners, and to foster motivation and
material, communication tools, and intelligent tutoring support. engagement in them.
The game aims to complement traditional classroom activities in Serious games (SGs) are in general games used to train
science, technology, engineering, and mathematics (STEM), for skills connected to real life tasks, with a negligible interest
high school students and teachers. It incorporates an intelligent in entertainment. Their digital implementations are known
pedagogical agent that can converse in natural language with
the student and provides unsolicited hints during gameplay. The to have wider educational aims (such as applying a given
paper presents the game and its evaluation based on experiments pedagogic approach), and providing an opportunity of studying
involving about 100 participants. We think that the results instructional content, beside the general aim of training more
presented here add to the research on game-based learning for practical skills [6]. They also confront educators with signifi-
STEM, by proposing a complex game system, where artificial cant challenges, spanning from teacher’s training, to the need
intelligence techniques are integrated to support students’ learn-
ing, and by confirming that the game experience can be attractive of varied professional expertise, and several organizational
for the learners, also in very constrained classroom environments issues [7], [8], so they are not so frequently used in schools.
such as those we operated in. In this paper, we present the game Gea 2: A New Earth,
Index Terms—Serious games, game-based learning, Gea2 henceforth, which we designed and implemented with
technology-enhanced learning, STEM education. the intent to cope with the technological and pedagogical chal-
lenges we briefly mentioned earlier. The game aims to support
I. I NTRODUCTION education on topics in science, technology, engineering, and
In the last two decades the world has become signifi- mathematics (STEM), for students of the initial year in high
cantly more digital. information and communication technol- school.
ogy (ICT) has shown to have actual, and potential, application In particular, we designed the game to be a complement
in several fields of human activity. Research in technology- and enhancement of the usual and more traditional frontal
enhanced learning (TEL) has fostered the design and im- lecture, rather than supplanting, and substituting it. In Gea2,
plementation of computer-based/network-based instructional students move within a three-dimensional (3-D) environment,
tools, supporting a higher degree of individualization and ultimately reaching three planets, by space travel, to explore
custom-tailored content delivery, hardly affordable in tradi- them while solving quizzes and looking for objects to un-
tional classrooms. Intelligent educational software [1], such as lock interactive experiments. The interactive experiments are
intelligent tutoring systems (ITSs), focuses on modeling and conducted in simulation rooms where evaluation activities are
applying, through a digital system, the processes used in the practiced on each planet, with the aim to select the best one
educational interactions happening in a real world class [2]. for human life. To positively affect the learning experience,
A virtual learning environment (VLE) further extends ITSs points and badges can be obtained through player actions.
capabilities by providing the learner with additional educa- Individual and team tasks are solved by the players, who
tional resources, such as multimedia material, communication are also assisted by an interactive virtual tutor capable of
tools, recommendation systems, and more. The possibility of interacting in natural language. Underneath the game, an
interaction with other learners paves the way to new scenarios intelligent pedagogical agent (IPA) monitors the progress of
of pedagogy on the Internet [3]. the player, and is capable of autonomously deciding whether
The attractive bouquet of learning material and tools offers and when to provide unsolicited help. Players can also share
support to educational methods on the network, while present- knowledge in their team’s “Knowledge Cloud,” and consult
ing teachers with new challenges and opportunities. Among insights on a didactic cloud platform and chat with other
teammates. Lastly, the teacher can follow the game’s progress,
Manuscript received June 10, 2020; revised April 30, 2021 and August and coordinate the session from a multimedia whiteboard.
17, 2021; accepted November 14, 2021. Date of publication TBD; date of
current version November 14, 2021. This work was partly supported by the We intended to experiment with SG technology, in a high
European Space Education Resource Office, Italy (ESERO Italia) through school setting; we also aimed to add to the research work on
project SPATIO. (Corresponding author: Francesco Sapio.) artificial intelligence (AI) in education and GBL, in particular
The authors are with the Department of Computer, Control, and Manage-
ment Engineering, Sapienza University of Rome, 00185 Rome, Italy (e-mail: with the use of the IPA and the system to deliver unsolicited
lsferro, sapio, mecella, marte, [email protected]). hints operating in Gea2. In addition, from a game-production-

2372-0050 (c) 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Universita degli Studi di Roma La Sapienza. Downloaded on January 31,2022 at 12:17:54 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TLT.2022.3143519, IEEE
Transactions on Learning Technologies
IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, VOL. XX, NO. X, XXXXXX 20XX 2

process viewpoint, we wanted to apply a methodology of pro- Research in GBL, and SG, provides evidences of success
gressive development of Gea2, based also on the collaboration [9], however, there are also studies recommending prudence
with teachers involved in the experimentation. This last aspect about the actual effectiveness [10], [11].
of “co-design” is out of the scope of this paper, basically for In [12] a study is presented showing the current status
reasons of space, and yet was one of the initial drives for our of empirical research on mobile GBL in STEM education.
work. The conclusions of this survey are that, on the one hand a
Beside presenting Gea2 and its experimentation, in this great majority of the experiments provided positive results
paper we aim to analyze the experimental data, to verify from using GBL, and, on the other hand, “the fundamental
how the Gea2 learning experience was 1) appreciated by the question of when mobile GBL is an appropriate approach for
students, from the point of view of perceived usefulness and learning in STEM education and when it is not remains to be
engagement; and 2) effective as an instructional means. answered,” suggesting the need for further empirical studies
We will see that the learners have been very appreciative and “to identify how and why certain designs work in particular
engaged, while the summative assessment of their proficiency circumstances.” Another example, conducive to similar
was less than satisfactory. With regard to the just mentioned conclusions, is in [13], which describes a use of game based
negative result we will propose interpretations, relating it science learning, in middle school. The study focused on both
to some hard constraints, partially unexpected, we had to learning outcomes (using pre–post tests on the digital game
confront during the experimentation (in particular hard time and non-digital groups) and students’ self-efficacy. Results
slots, and the use of the game detached by actual lectures). were mixed, whereas self-efficacy was reported as better in
Summarizing the whole contribution of the paper, we think the digital group than in the non-digital, while no significant
that our experience added to the current research area on GBL differences were detected in relation to learning outcomes.
for STEM, by 1) proposing a complex game system, where AI There are concrete obstacles to a wider use of games:
techniques are integrated to support students’ learning (with for instance, adopting them could be challenging due to
the use of the IPA and of the unsolicited hints subsystem; existing stigmas around videogames, that can still influence
2) experimenting it in a high school setting; 3) obtaining a the educational environment; moreover, integrating the game
confirmation that the game experience can be attractive and medium within the existing syllabus may be difficult. Against
engaging for the learners, also in very constrained classroom the adoption of a SG, there may also be issues of cost, as
environments such as those we operated in; and 4) unveiling the development of a SG may involve the collaboration of
some threats to the use of this kind of systems, represented, different professional roles and stakeholders. In addition, other
in our case, by unforeseen difficulties offered by the above challenges can come from the difficulty to find usable off-
mentioned constraints: in this respect, we think that our the-shelf games, to suit the learning needs of the class. Also
experience can be useful, to us and others, in the organization other factors can propose troubles, such as the quality of the
of other experimental activities. adopted games, the need for specific teacher training to use
After introducing current research in GBL (Section II), we them, the problem of fitting activities (possibly long ones) into
give an account of the design (Section III), development (Sec- the time span of the lecture, and eventually the difficulties with
tion IV), and evaluation (Sections V, VI, and VII) of Gea2. equipment (or lack thereof) [7], [8].
Then, Section VIII proposes a discussion about the findings, In fact there is still a limited use of SG [7], [8], and a need
and a description of the limits of this study. Conclusions, and for further study [6]. Also the use of intelligent pedagogical
plans for future research are drawn in Section IX. agents (IPAs) in immersive virtual learning environments is a
promising research topic [14], but there are not very many SGs
II. R ELATED W ORK AND M OTIVATIONS implementing such combination. Of the several prior studies
Digital GBL draws from the constructivist educational the- in this field, many focus on motivation and engagement, and a
ory, and from the fields of computer games, visualization, and need is felt in recent years to study how games are eventually
human–computer interaction, to administer learning experi- able to increase the cognitive or achievement outcomes in
ences and content, through videogames [3]. It aims to provide students [6], especially with respect to how such outcomes are
and motivate the learners by means of engaging interactive in agreement with the curricular content the school expects to
instructional activities, ranging from simple tasks to accom- be acquired by the student. An additional aspect of interest,
plish, to the development of skills and solutions for complex about SG in education, is in the target of the experimental
tasks. This educational approach is expected to have success activity: Although research and application of educational
with current generations of learners, and guide them towards games in the high school context are present [15], this level of
the challenges of the technological society in the 21st-century. the educational system seems to be less frequently met, and
Games are structured contexts, with clearly defined rules, in need of further efforts [16]–[18]. Furthermore, an aspect
where players need to overcome challenges and face opponents of interest in the use of GBL is its connection with the other
to achieve victory. Games can offer incredibly immersive and activities planned in the course: The use of a game remains
engaging environments where users can learn by doing and often as a single albeit removed experience; what happens
improve skills and competences related to decision making, in the game and what flows in the classroom often does not
strategy, teamwork, as well as social skills, leadership, and connect/crossover. In some instances though, as in Ludwig
collaboration. SGs focus on the design and development of [19], classroom materials are also provided so that the learning
games with not only entertainment purposes. component of the game can extend beyond the interactive

entertainment. (like in [22]) by assimilating with an unfamiliar role (i.e., one

The summary proposed above allows to list some topics, never assumed in reality) in the team.
motivating the usefulness of the study presented in this paper. The design of Gea2 game environment utilized an iterative
Firstly, further investigation seems needed on the impact of process. We first began with a survey of existing SGs that
GBL on students, in terms of how the learning outcomes are were situated within the STEM context. After narrowing the
able to cover the curricular aims of the courses in which they scope, we focused specifically on those that teach concepts of
are applied. In particular, such curricular aims are usually physics. We also considered the accessibility of the tool via
quite strictly and formally specified by the general national mobile devices. This was an important consideration as we
educational system, and can provide quite narrow constraints. wanted that the final game could function on tablet devices.
Secondly, while previous studies focused mainly on children Next, we created a mood board to refine the aesthetic,
and higher education, the impact of GBL during the critical with the aim for it to run efficiently on a mobile device.
time of adolescence has been less explored. Hence, additional In other words, we wanted to select an aesthetic style that
research may be useful at high school level, where keeping would not be graphically demanding in terms of computational
students motivated to sharpen their competences can be harder resources (e.g., high poly/tris count). As a result, we settled
a task than elsewhere [20]. Finally, we observed that, in most on a low poly aesthetic that was simplistic enough to require
studies, games are meant to supplant, rather than complement, low computational resources, while still allowing to create
the usual classroom lecture, delivered by the teacher: The interesting, large, and explorable environments for the player.
study of the uses of GBL to flank traditional lectures can then In addition, we provided each one of the planets featured
be an interesting topic for additional research [21]. in the game, and the spaceship, with an individual color
In Gea2 we tried to meet the above mentioned lines of work, scheme and context. So, planet Maya features trees with a
aiming at developing an SG that would 1) complement the somewhat “Earth”-style aesthetic, planet Violet has a more
classroom activity of learners and teacher, 2) have learning isolated crystal aesthetic, with purple predominant, and planet
outcomes strictly connected to the ones planned in the first Melissa shows quite an imaginary martian environment. Lastly,
year of a high school, and 3) allow to deepen topics met in the the spaceship presents the player with a futuristic aesthetic,
other classroom activities, such as implementing collaborative with illuminated corridors and minimalistic décor, agreeing
learning among the students and with the teacher. with the general low-poly aesthetic. The design of the user
interface (UI) followed the same lines of simplification, with
III. D ESIGNING Gea2 the additional requirement of adapting the interface to the
size of the tablet screen, and managing the visual hierarchy
In this section we firstly present a description of the design of information, so to make important information readily
process of the game, discussing also technical and aesthetic available for the players, and to ease their navigation of the
aspects. Then we focus on two important pedagogical aspects environment.
related to the design of Gea2, namely the narrative setting Another important aspect that we have taken into account
and the nature of Gea2 as a role-playing game (RPG). These during the design of Gea2, is the experience for both the
aspects are meant to provide the player with an attractive students and teachers. Unlike most SGs, we believe that
immersive experience, and foster a cooperative approach to teachers should not be reduced to an assisting role, or excluded
her/his learning. Then we present shape and significance of from the gameplay process. Instead, teachers should play an
the game components and elements. active role, to help and guide students within the SG, much
Creating Gea2 required a series of steps that incorporated the same way they would within a classroom environment.
different considerations from both a game and educational Therefore, we also made the same considerations for what the
perspective. Gea2 consisted of three main parts 1) Serious teacher required on their end in terms of UI and information.
Game, Interactive Board, and Professor App. We described in full detail the game design in [23].
In order to ensure that during gameplay the players can
achieve the planned learning objectives, we adopted the fol- A. Narrative
lowing criteria during the design phase of Gea2: A decision taken in early stages of the project was to set
• Gea2 has to be a supporting educational tool integrated the plot around the hottest topic in astronomy: Exoplanets and
within a classroom environment. The idea is that it can the search for extraterrestrial life. Astronomy and astrobiology
be used by teachers and students in class, as a complet- perfectly support interdisciplinary and cross topics discussion,
ing/complementing part of the lecture. so they are a good match for our interest in STEM education.
• It is an RPG that addresses 21st-century skills, such as meta- Exoplanets are planets similar to Earth, orbiting a star other
competences, soft skills, communication, and collaboration then the Sun. Both adults and young persons are since long
skills. passionately interested in this topic, with some further interest
• It supports students’ learning of how to inhabit the arisen in the very last years, after the February of 2017,
headspace of someone other than their primary ego identity, when the NASA’s Spitzer Space Telescope unveiled the first-
thus affording the student an opportunity to develop a known star system with Earth-sized planets, some placed in
stronger sense of empathy. the habitable zone of the system, where a rocky planet is most
As a result, Gea2 encourages the development of a sense likely to have liquid water (the key to life as we know it) [24].
of community, by training individuals to function as a group Gea2’s plot is given in the game as follows:

A meteorite is approaching the Earth and very soon there that students should engage in learning that allows them to
will be an impact provoking earthquakes and tsunamis that explore, inquire, solve problems, and think critically. To this
will devastate continents. Students are part of a space mission end, reform efforts within each of the STEM disciplines have
and each one of them has a specific role with assigned tasks. focused on such strategies as inquiry learning [26], project-
The team’s goal is to identify, among three, an exoplanet based learning, constructivist learning [27], problem-based
that is suitable for human life. The single objectives are learning [28], and the integration of technology across all
related to the specific role. For example, physicists have to STEM disciplines. Although these efforts have fostered im-
calculate the gravity on the three planets and compare it proved learning outcomes within each of the STEM disciplines
to the Earth’s gravity. Chemists have to derive the planet’s [29], many researchers argue that in order for students to
atmosphere composition and so on. Once the student/player be fully prepared for careers in the new millennium, they
has accomplished his/her tasks, the team should compare the must be capable of thinking across disciplinary boundaries
obtained individual results and discuss together to infer which [30]. Frykholm and Glasson suggest that schools must begin
is the best planet among the three available ones. to veer away from treating each STEM discipline as a silo
and embrace an approach that blurs the boundaries of these
disciplines. They argue that students who engage in rich
B. Role-Playing Game
cross-disciplinary experiences will have a deeper conceptual
Another important decision that we have made, from a understanding of science and mathematics content [31], which
pedagogical perspective, is to implement Gea2 as an RPG. will improve their achievement in each of the disciplines.
RPGs exist in many forms, from virtual role-playing, to Further, interdisciplinary learning can foster an understanding
tabletop, to live action. While each type of role-playing of STEM concepts in their application to real world problems,
offers a unique experience, these games provide a compelling which are interdisciplinary by nature. In traditional school
escape from the mundane reality, attracting millions of players settings, the compartmentalization of scientific knowledge
worldwide. Unlike the passive experience of watching a film creates boundaries so rigid that they often serve as barriers
or reading a book, these games encourage players to actively to any efforts to develop integrative science and mathematics
take part in the adventure, sometimes even developing their programs [32].
own stories and characters. RPGs also offer a safe, relatively Gea2 embraces interdisciplinary and integrated science
consequence-free space where players can develop certain concepts. Table I reports the description and example tasks
aspects of themselves. Through role-playing, players learn how for each of the four roles appearing in the game: physicist,
to inhabit the headspace of someone other than their primary geologist, chemist, and astrobiologist. Each role has three tasks
ego identity, offering them the chance to develop a stronger (one on each planet) to accomplish.
sense of empathy. The shared, formative experience of RPGs
provides a ritual atmosphere for players to enact compelling
C. Game Components and Elements
stories or perform unusual, extraordinary deeds. In this way,
RPGs help encourage a sense of community, by teaching Here we describe the components that players can use
individuals to function as a group. Experiences transpiring in during gameplay.
RPGs allow players to develop a deeper understanding about a) Registration: At game start the learner is greeted by
themselves and one another during the adventure [22]. an introductory video, then replaced by the login/registration
Another important reason for implementing Gea2 as an screen. The registration operation collects data about the
RPG is the collaborative/cooperative aspect of learning, that is, learner (name, email, chosen password), and about the school
one of the 21st-century skills. Borich [25] asked, “What good and class (which are selected by a drop down menu, as only the
are critical-thinking, reasoning, and problem-solving skills if schools where the game was deployed appear). Each learner
your learners cannot apply them in interaction with others?”. will be operating in one of the teams defined by the teacher.
Cooperative learning activities train learners to reason and From here, the data is sent to a dedicated web service, which
perform in an adult world. Our attitudes and values are among sends back an authorization token for the player to use to gain
the most important outcomes of schooling, and are formed access to the game.
through social interaction. They provide the framework for b) Team creation: To achieve their own and teams’
thinking independently inside and outside of the classroom, objectives within Gea2, players have to explore the 3-D
and for guiding our actions outside the classroom. We believe environment. The environment consists of a spaceship and
that RPG can strongly support collaborative learning, exactly three simulation rooms—one for each planet. By exploring
for their intrinsic nature to develop a deeper understanding of these environments, players can:
other players. • collect specific (task-related) objects around the ship.
A crucial pedagogical aspect, that found easy application • gain access to the planets’ simulation rooms, by correctly
in RPG, is bound to interdisciplinary issues. In the last answering questions.
few decades, many reform initiatives have shaped teaching • solve their assigned task using interactive panels (within the
and learning in science. These reform efforts include a shift planets’ simulation rooms).
from teaching students to remember and execute isolated • seek for help in any moment, namely asking questions to
facts/skills, to having students experience learning as scien- the IPA, or consulting some ad-hoc materials external to the
tists, engineers, and mathematicians do. Pedagogues argue game (e.g., video, articles, insight notes, simulations, etc.).

TABLE I
P LAYERS ’ ROLES IN Gea2

Role Description Example Tasks

Physicist The Physicist studies a wide range of phenomena, span- Calculate the value of gravity on the three exo-
ning from subatomic particles, which all ordinary matter planets and make a comparison with the terres-
is made of, to molecular length scales of chemical trial gravity. Discuss with other team members,
and biological interest, to cosmological length scales evaluate if the discovered values are suitable for
encompassing the universe as a whole. human life.
Geologist The Geologist works to understand the history and Calculate the axial tilt and magnetic field and
future of our planet. The better they understand Earth’s understand the characteristics of the water cycle
history the better they can foresee how events and on the different planets. Discuss with other
processes of the past might influence the future. team members whether the discovered values
are compatible with human life.
Chemist The Chemist masters composition of matter and its Calculate at which temperature water boils, the
properties, dealing with molecules and atoms. They atmospheric composition and the volume of
carefully measure substance proportions, reaction rates, the three planets. Discuss with team members
and other chemical properties. whether humans can breathe in these different
atmospheres.
Astrobiologist The Astrobiologist is interested in the study of the Calculate the atmospheric temperature and pres-
origin, evolution, distribution, and future of life in the sure on the three planets. Discuss human sur-
universe. vival conditions with team members.

Therefore, the players are encouraged to explore and interact f) Unsolicited hints system: Based on learner’s in-game
with the whole virtual environment (classroom, team, teacher, behavior, the IPA provides a pedagogical intervention through
and game), which mimics the ways they would use to obtain unsolicited hints (see Section IV-E). To compute such hints
information in the real world, in order to solve problems the IPA monitors the learner’s advancement within the game,
and complete their tasks. Once the players start solving their and her/his emotions:
own objectives, the team can share information to form a • Player’s advancement: To analyze the progress in gaming
complete vision of the overall team task for selecting the best sessions, we decided to check two other important values.
planet suitable for life. As a result, Gea2 supports cooperation The first value is the overall score obtained by the player,
and collaboration (requirement of a successful RPG) via chat representative of the actions undertaken during gameplay.
systems and in-class discussions. The other important value is how many questions the player
c) Inventory system: The inventory is the repository has already asked to the IPA.
where the items collected by the player are stored, to be • Player’s emotions: We try to infer player’s feelings based
removed when they are used during an experiment in the game on the conversation with the IPA, and the participation in
(see letter i below). An item is collected by collision with it by chats with teammates. In chat posts we monitor emoticons
the player’s avatar. The inventory items are displayed in the top usage. Emoticons are visual representations of emotions
left hand corner of the user interface. Items are specified in the such as a smiley face, which are commonly found on
system through fields for the item id, and for a sound effect. social networking and messaging platforms. To facilitate
The component also implements a 3-D animation function that emoticons usage, we decided to use a limited numbers of
makes the 3-D model float in the air to make it more visible them: eight, representative of the basic emotions.
to the player.
d) Localization system: The localization system of the The system can send the monitored data to the web service
game tracks the players’ (avatars) movement and identify their delegated to analyze it and provide the unsolicited hint if
location within the game’s space. It is implemented through a needed. This function is called periodically, with an interval
script, replicated in all the game locations, that intercepts the we have empirically set to three minutes. The unsolicited hints
presence of a player and interact with a web service listening system hosts a panel for displaying the hint to the player. To
on the server. The web service feeds the Professor Virtual assess the validity of the provided unsolicited hints, that is,
Board (see later), so to make an updated live map continuously the correctness of our algorithm, we added the possibility for
available. the user to provide feedback on the hint they receive, marking
e) Note system: Sharing notes is an important part of it as helpful or not.
the students gameplay. This subsystem gives access to a note g) Chat system: To encourage communication between
composing and consulting panel, where it is possible to to teammates, we gave players a way to exchange text messages
1) write a comment, selecting the subject from a drop-down and emoticons in a dedicated chat group for each team. The UI
panel; 2) publish the composed note; 3) list the notes in features an interactive panel that hosts an input field, a text
the team’s Knowledge Cloud; and 4) see feedback received container for displaying messages, a smaller panel allowing
about the note. The note is sent to the server, which makes the selection of emoticons, and a send button. The chat is
it available to users, through the general Knowledge Cloud of implemented through the use of the Photon Chat engine [33],
the game. It also applies validation checks, and administers which provides several functions used as callback. Photon
related score and badges. Chat provides “channels” to make separate chats; we used

this feature to allow for intra-team communication (with one • Add new teams
channel for each team). Information about teams is permanently stored, and retriev-
h) Quiz system: Whenever the player accesses a quiz able during the successive gaming sessions. A student can
panel, a script is called that loads the appropriate questions, join a team during registration. After teams have been created
and answer options, dynamically from a text file: The teacher teachers can: access information about the players (such as
manages questions and answers by editing this text file. If role, assigned task, score), check the position of the players on
the player answers the questions correctly, the script plays a the map, and validate the notes of the players in the Knowledge
success sound and shows a panel with the objects that (s)he Cloud.
will need in the simulation room of the related planet. If the
answer is wrong, a “wrong answer panel” is shown. The score
of an answer depends also on the number of attempts the
player made before providing the correct answer.
i) Animated experiment panels: The 3-D game is multi-
component and offers many player–game and player–player
interaction possibilities, however a core aspect is the possi-
bility to perform interactive experiments. When the player
first interacts with an experiment monitor, if their inventory
contains all the items needed to unlock that specific ex-
periment, a panel is displayed, showing the inventory. The
panel allows the player to select the objects (s)he think are
needed for the experiment. If the selection is correct, the
experiment is unlocked: the selected objects are removed from Fig. 1. PVB visualization.
the inventory, and the interactive experiment panel is opened,
with a “success” sound. Otherwise, an on-screen notification, To enable a better cooperation among students, we have
with sound is issued. added the possibility for them to share ideas, thoughts, and
in-game discoveries they make, with the team they belong to.
IV. D EVELOPING Gea2 Published notes are at first visible only by the teacher, who can
validate them (accept/reject). In the PVB, the live 2-D map of
For this project we developed a three-tier architecture, with
the game is visible. The position of the players in the current
a presentation, a logic, and a data tier.
gaming session is continuously updated and displayed by icons
with each player’s name and role (the latter represented by a
A. Presentation Tier (Teacher and Student Workspaces) colour). The PVB also allows to scroll through the list of
The teacher workspace consists of three different appli- teams for the current class. For each team the teacher can
cations, named Professor App, Professor Virtual Board, and consult the list of its students/players, and is allowed to focus
Opedia R . on each student’s information. In particular the teacher can
check the student’s detailed score, or read her/his assigned
Professor App (PA): a web application developed to task. The teacher can also access, through the PVB, the team’s
manage gaming sessions and classrooms, and to view Knowledge Cloud and moderate the notes published by each
the collected data about the students’ in-game activities, player of the selected team.
in both past and ongoing sessions
Opedia R : a cloud platform allowing the sharing of mul-
In the PA, a teacher can: register/login, select her/his school, timedia didactic material between teachers and students
add/Select a class, and, more importantly, start/stop a gaming
session for the selected class and monitor students’ results in Opedia R was integrated into our VLE thanks to an agreement
ongoing or previous sessions. we made with its owner. Its use can be also supported by
the teacher, who could help understand the retrieved material,
Professor Virtual Board (PVB): the teacher’s tool or suggest further search queries. We could not involve the
used in the classroom to manage the gaming session, to teachers in the construction of a whole scaffolding system in
interact with the students, and to control their progresses Gea2, so we used Opedia R as an external means to offer some
of the functionalities of a dedicated scaffolding system.
A gaming session can last up to four hours. When a teacher The Student Workspace consists of two applications: a 3-D
starts a gaming session, the PA generates a unique code (PIN) virtual game (3DVG), and Opedia R .
for the session, That PIN can be used to start the PVB
application and link it to that gaming session in the classroom 3DVG: the core of the student workspace. Students
(see below). The PVB is designed to run on an interactive become players when they first register to the game, join
multimedia whiteboard. As a result, teachers can: an existing team for their class, and choose a role
• Launch the PVB and link it to the current gaming session
using the PIN already generated using the PA Once in an active game session, students can :

• Login to the 3DVG and explore the 3-D Environment the students that have joined that team. The professor services
• Recover 3-D objects they need to solve the assigned task are used to manage teachers and the classes they teach to. They
• Gain access to the simulation rooms answering quizzes allow the creation, deletion, update or retrieval of teachers, and
• Enter the simulation rooms and find the interactive panels the addition or deletion of a class from their personal list of
• Solve assigned tasks using interactive panels classes. The note services are related to the management of
• Ask help in natural language to the IPA notes in a team’s Knowledge Cloud, providing their deletion,
R
• Access insights on Opedia retrieval, and validation by the teacher, who can accept or
• Add notes and chat with teammates reject them as previously mentioned. The school services are
All the actions listed above can be performed by players in used to recover information about available schools and classes
any sequence they like. In other words, they can create and for each school. In order to provide an identification number
follow their own strategies subject to the game’s mechanics. To for schools we have used the unique key generated by the
further push the students’ problem solving skills, we developed government to identify Italian schools. Finally, the student
a quiz mechanics. Throughout the game, students are asked services deal with recovering and updating information about
to correctly answer various task-related questions, in order to players during gaming sessions. In particular they allow the
progress in the game and gain access to the simulation rooms. creation, deletion, and retrieval of a student, the creation and
Each simulation room can be accessed by a student when he or update of the game progress for each student, the deletion and
she has identified and collected those objects that are defined addition of a team for each student, the creation and retrieval
as necessary, according to her/his task. of notes, the creation and update of a player’s position, the
Most of the actions in the game are driven via a point-based addition of the question asked by the player to the IPA, the
system, which operates in the background. Students are not creation and update of the score for each player, and the
aware of this point-based system, and it is an important part provision of unsolicited hints.
of the 3DVG because it contributes to the students evaluation.
However, students can see their score by asking the teacher to C. Data Tier
view it in the PVB. Beside points, players can obtain badges The data tier consists of a complex database where all
by completing certain tasks. These badges can be accessed in data related to the different actors of the system are stored.
the spaceships’ trophy room. In particular, the database also stores the knowledge of the
The IPA is part of the 3DVG and guides the player trough IPA. We refer to the knowledge about a specific subject as
the game. The IPA that we have designed has two main being held by a non-player character (NPC). Specifically, we
functionalities. The first is to reply to questions asked in have four NPCs, one for each role that a player can have. In
natural language: it can be thus classified as a Dialogue particular, the NPC knowledge templates consists of a simple
Management System (DMS). The second is to evaluate game sequence of questions, answers and possible suggestions.
progress and emotions expressed by the players during the The data tier is implemented as a relational database con-
game, in order to infer if the player needs help and, if this is taining information about all the entities discussed in the
the case, provide unsolicited hints. The latter ability provides previous sections. Logic and data tier are strictly connected
pedagogical intelligence to our IPA. since many of the web services in the logic tier can access the
database both for reading and writing data.
B. Logic Tier
The logic tier offers the web services managing players and D. Playing the Game
teachers, teams, gaming sessions (save/resume), and all the Once the student is logged in, they begin the game from
functionalities described in Section III-C. It communicates, the spaceship (see Fig. 2). The heads-up display contains the
via network, with the presentation and the data tiers. In-game
progress is automatically saved, by uploading to the server and
storing in the database (e.g., at each player’s achievement, such
as unlocking a badge, collecting an object, or scoring points).
Progresses are reloaded in the game at log in, so the game
can be played seamlessly across different gaming sessions in
class.
To implement the web services we took advantage of
Jersey, an open source Java framework created by Oracle for
developing RESTful web services according to JavaTM API for
RESTful Web Services standard. The authentication web ser-
vices use a security token mechanism. The class services are
related to all the operations that treat the class as a whole, and
allow the creation and deletion of classes, starting or stopping Fig. 2. View of the command centre in the spaceship.
a gaming session for a class, and listing the teams in a class.
The team services allow the creation and deletion of teams, controllers that the player can use to play the game. At the
listing the notes in that team’s Knowledge Cloud, and listing bottom there are the two joypads, one to navigate the other one

to look at the virtual environment. Next to those, we find the The core of our system is an implementation of Naı̈ve Bayes
jump and interact buttons. On the right side of the screen the text classification [37], which is a probabilistic classification
player can access the chat. On the top left corner of the screen method based on language modelling under the hypothesis
is housed the inventory, which shows all the objects the player of words’ conditional independence. This algorithm is an
currently carries. On the top right corner of the screen there application of the Bayes theorem, which allows the estimation
are four buttons, to access 1) a panel for inserting answers to of the probability that a collection’s document is relevant for
the player’s tasks (assigned based on the role), 2) Opedia R , 3) a query, given the sequence of words that make up the query
the interactive virtual tutor interface, and 4) the note system itself. We then validate the answer chosen by the classifier by
panel. applying a threshold criterion.
The player can explore the 3-D environment, solving The implementation of the IPA in Gea2 is done through
quizzes and looking for any information to help her/him a script providing the user with a panel allowing to submit a
solve the assigned task. To ensure a positive reinforcement question and visualize the IPA’s textual answer. The IPA builds
of learning, we provided the possibility to unlock badges as a the answer by processing the question and querying the NPC
result of some actions. When the player approaches a badge, database for the most pertinent answer.
an on-screen message describes the actions to be completed
in order to unlock that badge. When a badge is unlocked, the V. E VALUATING Gea2: M ETHODOLOGICAL F RAMEWORK
player is notified and the badge will be displayed in the trophy In this section, and in the following two, we describe the
room. To progress in the game, the player has to enter the evaluation of Gea2. , which occurred during the two phases of
simulation rooms. The player has first to find the interactive development and experimentation of the game. The evaluation
monitors and answer some quizzes, then the entrance to the regarded two main aspects: On the one hand, we wanted to
related simulation room is unlocked. Once a simulation room measure how the students liked the experience, and perceived
is accessed, the player finds another interactive monitor, so to the game as useful to support learning. On the other hand
perform an experiment: Experiments are the key to solve the we considered the effectiveness of the game, namely how it
players’ tasks. allowed to learn STEM topics in a playful way, and to acquire
21st-century skills.
Assessment describes the process of using data to demon-
E. Intelligent Pedagogical Agent strate that stated learning goals and objectives are actually
In Gea2, the communication is implemented via a natural being met. Michael and Chen [38] state that “Serious games,
language processing (NLP) algorithm, based on an ad-hoc text like every other tool of education, must be able to show that the
retrieval problem solver and on a Naı̈ve Bayes text classifier necessary learning has occurred.” Thus, SGs, to be considered
equipped with an inner product-based threshold criterion (see a viable educational tool, must provide some means of testing
later) [34]. The problem of selecting the right answer from and progress tracking; testing must be recognizable within the
a knowledge database can be expressed similarly to an infor- context of the education or training they are attempting to
mation retrieval problem [35]. Following Conati in [36], there impart. However, learning is a complex construct, difficult to
are two main challenges: measure, and determining whether a simulation or a SG is
effective at achieving the intended learning goals is a complex,
1) Assessing students’ knowledge and learning from the in-
time-consuming, expensive, and difficult process [39], [40].
teraction with the game (whereas the connection between
Generally speaking, an assessment can be described as
learner’s game action and understanding of the underlying
either (i) summative whereby it is conducted at the end of
domain might be not clear).
a learning process and tests the overall achievements, or (ii)
2) Providing individualized interventions without interfering
formative whereby it is implemented and presented throughout
with the experience (as, often, to get success, the system
the entire learning process and continuously monitors progress
ought to provide help in ways that will not resemble, or
and failures [41]).
remind the learner of, traditional educational activities).
Considering the specific SG domain, Michael and Chen
In [34], we describe how did we try to tackle these describe three primary types of assessment: (i) completion
challenges: The IPA was developed, and integrated in Gea2, assessment, (ii) in-process assessment, and (iii) teacher evalu-
to provide the learner with unsolicited hints based on the ation [38]. Basically the first type corresponds to an automated
learner’s state of interaction in the game. The IPA is basically summative assessment, concerned with whether the player is
characterized by two abilities: able to successfully complete the game. In-process assessment
• It is an intelligent virtual tutor (IVT), that is, a virtual agent is related to observing the player behavior (progressively
capable of a conversation in natural language (NL). The stored in logs) by the teacher or automatically. If an inter-
IPA can reply in NL to questions written in NL, and is vention is managed, based on such data, this method could
implemented as a DMS [34]. be functioning like a continuous formative assessment. The
• It provides unsolicited hints to the learner, by evaluating the teacher evaluation is based on the her/his direct observations
game progression and the player’s emotions (through senti- and judgements of the students, while they are playing; it
mental analysis and emoticons usage) [34]. This makes of can function as a combination of the other two methods, but
it an IPA, that is, an IVT capable of actuating interventions typically aims at evaluating those factors that the functionali-
in the game, to support specific user needs. ties/logic of the game are not able to capture.

The most common method of post assessment currently teams. During this meeting, the students started using some
consists in testing players’ knowledge about what they learned functionalities for the first time, such as notes publishing, or
by way of a test or teacher evaluation. This method is asking questions to the IPA, or getting useful insights from
frequently employed because it is the simplest to implement. Opedia R . This is probably due to the fact that they were then
Coming to the experimentation, the tests were administered already familiar with the basic game mechanics, so they felt
in two different schools. Each test involved playing the game confident to explore further game features.
along four lectures, in class, with no extra time allowed In the last lecture (held four weeks later because of Christ-
beyond the normal time span of lectures (hence with the mas holiday) the students had to finalize their answers to the
limits imposed by a real school environment). Teachers were tasks they had been assigned, and each team had to produce
present, monitoring the activity and active whenever help was a “final decision” about what was the most suitable planet for
needed. Summative evaluations were conducted after the tests. human life. Then the players’ scores were compared on the
Moreover, we evaluated the likeability, learnability, usability, PA. Unfortunately, due to the unavoidable time constraints of
and preferences, as stated/perceived by the learners, through a the school, students were not able to discuss their findings
questionnaire. during the game, so they could not finalize a team answer to
The tests are described in the following two sections. the overall game task.
In the next two subsections we describe the questionnaires
VI. E VALUATING Gea2: T EST N O . 1 we used for the evaluation.
We had the collaboration of an Italian High School, that
provided two classes and a room hosting our tests, thus the A. Students’ Perception of the Experience
testing happened in a true context of education. The school
offers different five-years formation tracks, oriented towards We used a Likert scale, in which the format of the five-level
scientific or technical/industrial subjects. The two classes items ranged from Yes a lot to Not at all. The internal con-
had in fact different “specialization topic”. The test had 42 sistency of the questionnaire has been evaluated by Split-Half
participants, as each class was with 21 members (details are Reliability [42]. We acknowledge the low number of questions,
in Table II. The lectures schedule associated to the test is however, considering the difficulties in administering different
shown in Table III. questionnaires to a whole class, we had to limit the number of
questions to 18, in order to make the process manageable. For
TABLE II each student, we partitioned the questions in odd and even,
T EST N O . 1—D ETAILS ON C LASSES and summed the score for each half. Then, we computed the
correlation between the two halves. Finally, in order to get a
School name Classes Students better estimate of the reliability of the full test, we applied
ITIS Pascal 1F (Scientific institute) 21 (9 ♀, 12 ♂) the Spearman–Brown correction (see Table V), thus leading
ITIS Pascal 1N (Industrial/Technical institute) 21 (6 ♀, 15 ♂)
to the final value of the Cronbach’s Alpha at .92, which is an
excellent result.
The first three questions, showed in Table IV (Part 1) aimed
TABLE III to establish if students liked the game and would recommend it
T EST N O . 1—L ECTURES S CHEDULE
to other students (likeability). 64% of students enjoyed playing
the game while 15% disliked it. 81% of the students think that
Date Classes Method
Play Meeting 1 1F and 1N Game introduction, pretest Gea2 is more engaging then a traditional lecture, but jut 54%
Play Meeting 2 1F and 1N Play game would recommend the game to other students while 27% are
Play Meeting 3 1F and 1N Play game indifferent.
Play Meeting 4 1F and 1N Game conclusions, posttest
The second set of questions in Table IV (Part 2) aimed at
verifying students perception of learning outcomes (learnabil-
The test spanned over four lectures, described in the follow- ity). First, we asked the students if they learned by playing the
ing. During the first lecture the teacher introduced the game game: just 36% replied in a positive way. We then asked if
to the class and briefly explained roles, tasks and objectives. Gea2 is better than a face-to-face lecture, and if they think they
Furthermore, the pretest was administered. Students formed learned in a playful way. 67% of the students prefer this type
the teams and chose their roles. We valued the importance of learning, but just 33% think to have learned in a pleasant
of letting students self-organize and, most importantly, decide way.
their team name, in order to ensure a stronger sense of We believe that the above result, about learnability, can be
cooperation and individuality in the learning process. interpreted based on the two following observations. First, we
During the second lecture the students had credentials to had many positive feedback from students “about the game”,
access the game via premade player accounts. They grouped but they were very upset by the language used in the game.
by team and played the game for about 40 minutes, while The game was fully in English, and the students revealed great
the teacher followed the game progress using the Interactive difficulties in understanding the wording of quizzes and tasks.
Board. The language factor was also explained by the students as a
During the third lesson we divided students per role, in order reason for a reduced interaction with the IPA. These feedback
to encourage collaboration and knowledge sharing among let us decide to rewrite a good deal of the game (quizzes and

TABLE IV
T EST N O . 1—Q UESTIONNAIRE A BOUT S TUDENTS ’ P ERCEPTION OF THE E XPERIENCE

Part 1: Likeability
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q1 Did you enjoy playing the game? 2 (6%) 19 (58%) 7 (21%) 2 (6%) 3 (9%)
Q2 Do you think that playing the game is more engaging 11 (33%) 16 (48%) 2 (6%) 2 (6%) 2 (6%)
than a traditional lesson?
Q3 Would you recommend to students from other classes to 7 (21%) 11 (33%) 9 (27%) 2 (6%) 4 (12%)
try the game?

Part 2: Learnability
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q4 Do you think you have learned by playing the game? 1 (3%) 11 (33%) 7 (21%) 6 (18%) 8 (24%)
Q5 Do you think that playing the game has allowed you to 1 (3%) 10 (30%) 12 (36%) 3 (9%) 7 (21%)
learn in a more interesting way?
Q6 Do you think that playing the game has allowed you to 1 (3%) 17 (55%) 2 (6%) 7 (23%) 4 (13%)
learn faster?

Part 3: Usability
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q7 Were the user interfaces clear enough? 1 (3%) 16 (53%) 2 (7%) 7 (23%) 4 (13%)
Q8 Do you think the interactive board is useful? 7 (23%) 8 (27%) 8 (27%) 5 (17%) 2 (7%)
Q9 Do you like seeing team positions on the interactive 10 (31%) 6 (19%) 11 (34%) 3 (9%) 2 (6%)
board?
Q10 Do you like seeing scores on the interactive board? 11 (34%) 12 (38%) 7 (22%) 0 (0%) 2 (6%)
Q11 Did you use Opedia? 1 (3%) 14 (44%) 1 (3%) 6 (19%) 10 (31%)
Q12 Were the insights of Opedia useful? 4 (13%) 8 (25%) 7 (22%) 3 (9%) 10 (31%)
Q13 Was the Intelligent Pedagogical Agent useful? 0 (0%) 10 (31%) 9 (28%) 6 (19%) 7 (22%)
Q14 Was creating and sharing personal notes with team 7 (19%) 8 (22%) 19 (51%) 2 (5%) 1 (3%)
members useful?
Q15 Do you like that the teacher is responsible for note 2 (6%) 4 (13%) 17 (53%) 4 (13%) 5 (16%)
validation?
Q16 Would you like to have a personal customizable avatar? 14 (50%) 8 (29%) 5 (18%) 1 (4%) 0 (0%)

Part 4: Preferences
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q17 Do you like videogames? 23 (74%) 0 (0%) 0 (0%) 6 (19%) 2 (6%)
Q18 Do you think Gea 2: A New Earth can be considered a 13 (41%) 0 (0%) 14 (44%) 5 (16%) 0 (0%)
videogame?

TABLE V any correlation of these two answers with the likeability or

T EST N O . 1—Q UESTIONNAIRE R ELIABILITY learnability parts of the questionnaire.

Correlation Coefficient .859435333

B. Summative Assessment: Pre–Post Tests
Spearman–Brown correction .9244046488
The summative evaluation was performed by pre–post com-
petence questionnaires: 22 questions tried to measure STEM
knowledge. They were scored 1 point for each correct answer,
tasks) in Italian, for the next test. (We, though, had to leave the 0 otherwise. We grouped the questions in six categories,
IPA in English, as we had no availability for a reliable version five of them representing specific subject matters, and one
of the NLP software to use for Italian language). Secondly, collecting general knowledge (see Table VI). Some questions
another important remark coming from students was that there were suitable to be part of more than one category. The
were many bugs in the game and sometimes they wasted time questions and their partition in categories are reported in the
while trapped in a collision box, or jumping in places with no supplementary material for this article.
exit, or just having fun. In the final version of the game we The results of the summative assessment suggest that stu-
spent a lot of time fixing malfunctions and bugs. dents feebly learned. However, as explained by the teachers,
Finally, we asked other questions aimed at evaluating single both classes were showing already a low level of proficiency.
components of the game (usability), as reported in Table IV In particular, about the topics of interest for the test, only 30%
(Part 3). We used these feedback from Table IV to iterate the of the students was reaching sufficiency at the time of the
game interface, and Opedia R contents, in view of test no. 2. test, so that another test, given after only face-to-face lectures,
The last part of the questionnaire regarded the attitude of would have been likely to show very poor results. Another
the participants toward gaming, however we did not unveil important aspect, for the test evaluation, is in that students

TABLE VI • Quizzes and tasks were proposed in Italian;

T EST N O . 1—AVERAGE G ROWTH PER C ATEGORY • The amount of objects to collect was reduced to 6 per player;
• The use of IPA, notes and chat let now gain less points than
Category All Males Females in test no. 1;
General 14,42% 14,28% 14,06% • The IPA was now providing unsolicited hints.
Biology 17,53% 15,65% 17,26%
• Added a further step necessary to unlock the experiment
Environment 16,27% 23,81% -2,08%
Astronomy 13,23% 14,88% 8,33% panel in the simulation rooms.
Chemistry 23,48% 20,24% 19,45% In test no. 1, in order to access the experiment panel it was
Physics 5,25% 0,24% 13,33%
enough to have the correct objects in the inventory, which are
All Categories 13,91% 12,97% 13,38%
automatically selected by the game. Now, the player had to
select the object(s) explicitly, to show more comprehension of
the process. As mentioned above, the unsolicited hints system
had also a very poor preparation in general, as again put by
was a very important improvement coming for test no. 2. With
the teachers. The game’s contents have been prepared having
this upgrade we enriched the game with the IPA, able to decide
in mind an audience more familiar with the prescribed syllabus
on its own whether to help the player or not [34].
of the target school grade, and this played against better results
In the following subsections we analyze the results of test
in the majority of the game’s tasks.
no. 2, following the same lines adopted for the previous test.
VII. E VALUATING Gea2: T EST N O . 2 On the other hand, during this test, we added an analysis
related to 1) the perception of the students about the soft skills
The second test followed the same organization of the
related to the pedagogical questionnaire in Table XII, and 2)
previous one, and was performed with two classes in another
the students’ reactions to the unsolicited hints. Two additional
school. The classes had the same disciplinary specializations as
subsections consider these last aspects.
the two classes in Test 1. We had an overall of 53 participants
(the classes were of 24 and 29 students) (Table VII shows the
details). Table VIII shows the lecture schedule and methods A. Students’ Perception of the Experience
of work. We used the same questionnaire used in test no. 1. Table
IX reports the overall results. The internal consistency for this
TABLE VII questionnaire was evaluated by Split-Half reliability [42], as
T EST N O . 2—D ETAILS ON CLASSES
seen for test no. 1. Table X) shows the results. We had a final
Cronbach’s Alpha at .91, which is still an excellent result.
School Name Classes Students
We found that the percentage of likeability was more or
ITIS Pascal 1H (Scientific institute) 29 (9 ♀, 20 ♂)
ITIS Pascal 1D (Industrial/Technical institute) 24 (12 ♀, 12 ♂)
less the same as for the first run. However, the learnability
perception had improved a lot (from 37% to 64%). That
is probably due to the use of Italian language, and to the
improved quality of the game and contents. Unfortunately
TABLE VIII
T EST N O . 2—L ECTURES SCHEDULE
the student’s perceptions had not a confirmation from the
summative assessment, as we will see later. Students found the
Date Classes Methods interface more clear, and that is due to the effort we have made
Play Meeting 1 1H and 1D Game introduction, pretest
in correcting errors in the game. We believe that during test
Play Meeting 2 1H and 1D Play game no. 2 students used the interactive board more successfully,
Play Meeting 3 1H and 1D Play game participating and exchanging with the teacher to a greater
Play Meeting 4.1 1H Play game, game conclusions extent. We also saw that Opedia R was not used very much:
Play Meeting 4.2 1D Play game, game conclusions
Conclusion.1 1H and 1D Posttest, questionnaire even if its content is targeted to the game, students preferred
Conclusion.2 1H Posttest, questionnaire going on the Internet. The IPA was under-used, as expected,
considering that it remained in English: many students felt,
again, not confident enough to ask questions in English. This
In test no. 2, we collaborated with two researchers from the
could of course hinder also the appreciation for the unsolicited
pedagogy department of our university: One of them followed
hints: we see later that, actually, the majority of the hints
all the lectures, taking notes about the learning aspects of the
were positively appreciated, although the language problem
game and about social dynamics among students, as related to
remains. Finally, we saw that a strong majority wished to have
the game. With their help we prepared a questionnaire, deemed
a personal avatar to customise in the game. Also in test no.
to support analysis about the acquisition of side skills, such
2, the final two questions did not appear to be particularly
as the ability to apply problem decomposition, to perform
revealing, or connected to the other results.
progressive solution of a problem, and to put in practice
various aspects of a collaborative approach (see Table XII).
According to test no. 1’s feedback we got into a second B. Summative assessment: Pre–post test
development phase, where several improvements were intro- The summative evaluation in test no. 2 proceeded by the
duced. We produced several minor usability enhancements, but same protocol of test no. 1. We administered pre–post tests:
the main changes were as follows. Table XI shows the average knowledge growth per category.

TABLE IX
T EST N O . 2—Q UESTIONNAIRE A BOUT S TUDENTS ’ P ERCEPTION OF THE E XPERIENCE

Part 1: Likeability
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q1 Did you enjoy playing the game? 5 (24%) 8 (38%) 5 (24%) 1 (5%) 2 (10%)
Q2 Do you think that playing the game is more engaging 8 (38%) 7 (33%) 3 (14%) 2 (10%) 1 (5%)
than a traditional lesson?
Q3 Would you recommend to students from other classes to 3 (14%) 8 (38%) 4 (19%) 3 (14%) 3 (14%)
try the game?

Part 2: Learnability
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q4 Do you think you have learned by playing the game? 4 (9%) 26 (55%) 0 (0%) 8 (17%) 9 (19%)
Q5 Do you think that playing the game has allowed you to 4 (9%) 26 (55%) 0 (0%) 8 (17%) 9 (19%)
learn in a more interesting way?
Q6 Do you think that playing the game has allowed you to 13 (28%) 19 (40%) 2 (4%) 5 (11%) 8 (17%)
learn faster?

Part 3: Usability
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q7 Were the user interfaces clear enough? 6 (13%) 23 (49%) 6 (13%) 6 (13%) 6 (13%)
Q8 Do you think the interactive board is useful? 12 (25%) 18 (38%) 6 (13%) 6 (13%) 6 (13%)
Q9 Do you like seeing team positions on the interactive 12 (26%) 16 (34%) 12 (26%) 3 (6%) 4 (9%)
board?
Q10 Do you like seeing scores on the interactive board? 12 (26%) 16 (34%) 12 (26%) 4 (9%) 3 (6%)
Q11 Did you use Opedia? 6 (12%) 14 (28%) 8 (16%) 16 (32%) 6 (12%)
Q12 Were the insights of Opedia useful? 4 (9%) 11 (23%) 16 (34%) 8 (17%) 8 (17%)
Q13 Was the Intelligent Pedagogical Agent useful? 4 (9%) 13 (28%) 6 (13%) 12 (26%) 12 (26%)
Q14 Was creating and sharing personal notes with team 10 (21%) 15 (32%) 12 (26%) 5 (11%) 5 (11%)
members useful?
Q15 Do you like that the teacher is responsible for note 8 (17%) 13 (28%) 10 (21%) 6 (13%) 10 (21%)
validation?
Q16 Would you like to have a personal customizable avatar? 26 (55%) 12 (26%) 5 (11%) 2 (4%) 2 (4%)

Part 4: Preferences
Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q17 Do you like videogames? 33 (70%) 0 (0%) 0 (0%) 10 (21%) 4 (9%)
Q18 Do you think that Gea 2: A New Earth can be consid- 20 (43%) 0 (0%) 20 (43%) 7 (15%) 0 (0%)
ered a videogame?

TABLE X expected the game would improve the students understanding

T EST N O . 2—Q UESTIONNAIRE R ELIABILITY of the topics that had been already explained in class, while
the game ended up being used as a replacement of face-to-face
Correlation Coefficient .8361431978 lectures on the related topics. On the contrary, according to
Spearman–Brown correction .9107603359
teachers’ information, the students were not at all prepared in
advance on the topics of the game, and they were showing a
TABLE XI
low proficiency in general (more than half of the students, 16
T EST N O . 2—AVERAGE G ROWTH PER C ATEGORY in 1H and 14 in 1D, were with insufficient marks in STEM
subject matters).
Category All Males Females We can, however, report a positive result here, with the
General 25,02% 26,39% 22,17% substantial improvement of the knowledge growth with respect
Biology 32,57% 33,61% 28,57% to test no. 1 (average growth is 25.4% in test no. 2 and was
Environment 35,14% 37,86% 27,08% 13.9% in test no. 1).
Astronomy 16,55% 22,48% 9,03%
Chemistry 38,23% 37,77% 36,11%
Physics 21,36% 26,86% 16,67% C. Pedagogical Questionnaire
All Categories 25,55% 28,34% 20,96%
As mentioned above, we worked with two pedagogues and,
beside the annotations during game sessions, they helped
preparing a questionnaire (using a Likert scale) aimed at
The general result is not so good, from the point of view of verifying how certain side skills were possessed, such as
effectiveness of the game as a learning tool. In fact, the main approaches to problem solving (problem decomposition and
problem was in the conditions in which the game was used. We progressive solution), and orientation toward a collaborative

approach. Table XII shows the results (all statements started We think that, after the experience, we can present some
with “I learned to:”). positive results, some controversial findings, and a description
Plain positive responses (“Yes . . . ”) cover an average of of the difficulties to overcome in our educational system,
32.7% of the answers (SD 7.58). Non negative answers (“Yes and maybe others’, while undertaking experiments with GBL
. . . ” and “Indif.”) have a mean of 65.2% (SD 7.65). Two in- technology.
teresting aspects in the questionnaire are about the perception
of the importance of Collaboration and Interdisciplinary. We A. Discussion
think that questions Q2, Q5, Q6, Q7, and Q8 can better reveal Sections V, VI, and VII have shown positive results with
the perception about Collaboration: in this case, we have for respect to the students’ perception of Gea2 usefulness.
non negative answers, a mean of 67.93 (SD 4.82). Similarly, A large majority of participants liked using the game, with
we think that questions Q4 and Q10 can be representative of an average of positive or very positive likeability staying above
the Interdisciplinary perception: in this case, we have a mean 60% in both the test phases. There is, in fact, a decrease from
of 66.67. We conclude that these aspects seem to be felt by test no. 1 to test no. 2, mainly related to answers to Q2, which
the students as the more positive outcome of the game. we cannot explain, unless referring to the the fact that we had
to work with different classes, in test no. 2, than in test no. 1
D. Assessment of Unsolicited Hints (see a discussion of limits of this research in VIII-B).
Regarding learnability, we have witnessed a growth in the
The hints mechanism has been just tested jointly with the perception of usefulness shown by the students, and, in the
game itself, so we could not part its specific achievements second phase, almost two out of three participants found Gea2
from those of the game. From a formal point of view we can an effective support to learning: This perception grew from
only see the perception of the students, as provided by the 42% of test no.1 till 65% of test no.2.
answers to the questionnaires (Tables IV and IX) and from We think that the analysis of usability did not provide
the feedback during the experimentation, reported in the next great insight: in general the average positive or very positive
paragraph. From these data we inferred that this would be a feedback rise from 55% to 63% along the test phases, but we
core feature for the success of the game. saw issues with some functionalities and especially with the
In order to evaluate it, instead of asking players if they system’s language, on which we worked in between phases,
found the hints useful, we gave them the opportunity to vote and that needs additional work before before being ready for
thumb up or down. further experiments. Notably there are three questions where
In one game session, for all users (14 because they played a decrease of feedback is witnessed: Q11 and Q12 are related
in pairs when possible), we provided 72 hints, 47 were judged to the use of Opedia R , which prompts us to make changes
positive (65%) and 25 negative (35%). Interestingly, we saw in future; on the other hand, Q10 (concerning the reaction at
that when players’ emotion was negative (so, some frustration seeing the team score made public) is conceivably related to
was felt, and help was much needed) they would always reply different availability to competition of the involved classes,
the hint with thumb up, whereas when emotion was positive but we don’t think we can offer a final explanation.
the reply would be always thumb down. Regarding the effectiveness of Gea2 as a learning enhancer,
To have a better understanding of unsolicited hints provi- we proposed pre–post questionnaires. As seen in Sections VI
sioning we have to make the following annotations: and VII we partitioned the questions in categories, and in each
• All the provided hints were in English and students com- test we computed the average growth of students’ proficiency
plained about that on the categories. The result is not rich, but not negative at all
• Emotion detection was limited because of the limited usage in absolute terms: the average growth over all the categories
of IPA and chat. was 14% in test no. 1, and raised to 25% in test no. 2.
With all the above limits, we have an indication that One aspect of Gea2 is in the implicit interdisciplinary
emotions’ detection can be a valuable mechanism to provide learning experience it fosters. In the game, the players face
help during game sessions. the need to use knowledge coming from different fields, such
as chemistry, physics, astrobiology, or geology, not to mention
the concepts connected to life sustainability. We think that
VIII. D ISCUSSION OF R ESULTS AND L IMITS OF THE Gea2 was able to foster such an interdisciplinary learning,
R ESEARCH considering the different roles that the students in the same
Beside presenting the design and implementation of Gea2, team impersonates, and also in view of the results described
the two main motivations of this paper were in 1) evaluation in the previous paragraph, coming from questionnaires where
of how high school students would appreciate a game activity questions related to the various fields of interest were com-
integrated in their normal study program, and 2) analysis of prised.
the effectiveness of such an activity on learning outcomes. We The use of English language had been a continuing problem
had the possibility to perform a two-phases development and in our tests. This note is supported by the growth of learnability
experimentation, also enjoying the collaboration of teachers. (as perceived in the tests) and by the fact that the interactive
On the other hand, we had also to deal with rigidity and board was used more and more successfully, during the second
constraints during the experimentation, derived by the admin- test, while the part of the game that remained in English was
istrative organization of the schools. less used.

TABLE XII
T EST N O . 2—P EDAGOGICAL Q UESTIONNAIRE (A NSWERS TO “I LEARNED TO ...”)

Qn Questions Yes, a lot Yes, enough Indifferent Not much Not at all
Q1 Break-down tasks into small problems 3 (6%) 9 (19%) 16 (33%) 13 (27%) 7 (15%)
Q2 Appreciate collaborative work 6 (13%) 15 (31%) 10 (21%) 12 (25%) 5 (10%)
Q3 Appreciate technology potential 2 (4%) 10 (21%) 13 (27%) 14 (29%) 9 (19%)
Q4 View topics from a different perspective 1 (2%) 12 (25%) 14 (29%) 14 (29%) 7 (15%)
Q5 Achieve complex tasks in a collaborative way 5 (10%) 13 (27%) 15 (31%) 8 (17%) 7 (15%)
Q6 Plan collaborative work 5 (10%) 10 (21%) 21 (44%) 9 (19%) 3 (6%)
Q7 Understand requirements coming from different people 7 (16%) 8 (18%) 14 (31%) 12 (27%) 4 (9%)
Q8 Develop an idea in a collaborative way 4 (8%) 13 (27%) 15 (31%) 9 (19%) 7 (15%)
Q9 Understand the importance of work in progress 1 (2%) 10 (21%) 22 (46%) 11 (23%) 4 (8%)
Q10 Understand the importance of interdisciplinarity 6 (13%) 16 (33%) 15 (31%) 7 (15%) 4 (8%)

This subsection presented the quantifiable results of the learners, on average, ill equipped to deal with Gea2’s
proposed research. They can be summarized as being 1) topics;
related to a part of the students population which occurs 2) The use of the game as a substitute, rather than a comple-
less frequently in studies of GBL, 2) confirmatory of other mentary tool, for the lectures hosting the game sessions.
results in literature about the attractiveness and appreciation The game could not have been an additional activity, as a
of students for SGs integrated in their study activity, and 3) consequence of the tight schedule a class have during the
not firmly concluding on the effectiveness of the game as a school year;
learning tool. 3) The presence of some implementation issues in the cur-
In the next subsection we explore the limits of this research rent software version of the game, and in particular the
(that produced, for instance, the above point 3). Among such extensive use of English language in the user interfaces:
limits, we analyze also some factors of disturbance, acting on average, the students had greater difficulties with this
on the experiment, trying to give a description of experience language issue than expected.
that could help (hopefully not only us) in future experimental The first two issues as listed above were also true in the
activities. second test, so they represent a continuing limitation of this
research. In regards to the third issue above, we could learn
B. Limits from the feedback and let the game undergo an implementation
upgrade. We solved several software and interface issues, and
The first aspect worth noticing in this subsection, is in that
localized the interface language, except than for the IPA since
Gea2 effectiveness, as a learning tool, was measured only by
we had no reliable NLP software to be used in place of the
a non in-depth analysis of pre–post tests on an overall limited
current one.
number of students. This of course means that we cannot claim
Two additional aspects could be pointed out as limiting the
a whole result on this side, neither positive nor negative. As an
overall effectiveness of Gea2 in the experiments. One aspect is
explanation we can submit that, during the experimentation,
the persistent need for interaction in English with the IPA. The
we have been given limited allowance by the teachers, who
other aspect is related to the lack of a complete scaffolding
were in turn subject to administrative and formal constraints.
system in Gea2: we couldn’t involve the teachers in the
For instance, we could experiment only on a few classes and
implementation of such a sub-system (especially with regard
only by exposing the whole of each class to Gea2, and we
to the development of adequate contents), so we opted for
could not use data from other classes, or from previous years.
the inclusion in Gea2 of Opedia R . Eventually Opedia R was
Other constraints were in the timing of the activities, that could
not used (nor appreciated—see answers to Q11) as much as
be performed only during official lecture hours allotted for the
we hoped, so future versions of Gea2 will have to deal with
teacher’s subject matter, according to an inflexible time plan.
a proper solution. Although still suffering from the issues
And extra school activities were excluded by the nature of the
mentioned above in point 1) and 2), the second test produced
experiment, that was supposed to be conducted in classroom.
better results than the first, as seen in Section VIII-A. So we
This also brought the consequence that the time the teacher
hope that, despite the limitation described above, this paper
was giving us (i.e., to the game playing) was subtracted by
can provide the reader with useful results, and with useful
the normal lecture time, which implied that the game ended
information about threats to avoid during an experimentation
up being a substitute of the lecture on the topics of interest,
with SG, and GBL in general.
and not, as planned in our design, as a supporting resource for
the lecture.
Regarding the effectiveness issue, we collaborated with IX. C ONCLUSION
the teachers to examine the results and concluded that the Digital games are going to be used more and more exten-
unsatisfactory results mostly depended on three factors: sively for educational purposes. Research in AI is giving useful
1) The objectively low level of students’ previous proficiency insights for the development of VLEs, allowing for intelligent,
in STEM subject matters, which might have made the adaptive, and effective behavior of tutoring systems. Following

these premises, we have developed Gea2, an SG to be used [7] V. M. Perez-Colado, I. J. Perez-Colado, M. Freire-Moran, I. Martinez-
in class for teaching STEM in high school, as a supporting Ortiz, and B. Fernandez-Manjon, “Simplifying the creation of adventure
serious games with educational oriented features,” Educ. Technol. &
and completing part of the lecture. The pedagogical aims of Soc., vol. 22, no. 3, pp. 32–46, 2019.
Gea2 are to encourage the acquisition of meta competences [8] D. Tuparova, V. Veleva, and G. Tuparov, “About some barriers in usage
and 21st-century skills, beside learning STEM in a playful of educational computer games by teachers in STEM,” in Proc. 42nd
Int. Conv. Information and Communication Technology, Electronics and
way. Students can explore a 3-D environment that includes Microelectronics (MIPRO’19). IEEE, Opatija, Croatia, May 20–24
a spaceship and three simulation rooms, one for each planet 2019, pp. 727–730, doi: 10.23919/MIPRO.2019.8756999.
that they have to evaluate, looking for objects that unlock [9] S. de Freitas, “Are games effective learning tools? A review of educa-
tional games,” Educ. Technol. & Soc., vol. 21, no. 2, pp. 74–84, 2018.
interactive experiments in the simulation rooms, and solving [10] J. Kirriemuir and A. McFarlane. “Literature review in games
quizzes. and learning,”. NESTA Futurelab, Bristol, UK, Rep. No. 8,
Overall, we can conclude that the game proved to be 2004. [Online]. Available: https://www.researchgate.net/publication/
32231341 Literature Review in Games and Learning
enjoyable and students would recommend it to others. As of [11] R. van Roy and B. Zaman, “Need-supporting gamification in education:
the AI algorithm performance underneath the IPA, students An assessment of motivational effects over time,” Comput. & Educ., vol.
assessed it to be effective. The students in the sample found 127, pp. 283–297, 2018, doi: 10.1016/j.compedu.2018.08.018.
[12] F. Gao, L. Li, and Y. Sun, “A systematic review of mobile game-based
the game more engaging than traditional lectures, and felt that learning in stem education,” Educ. Technol. Res. Develop., vol. 68, pp.
they were learning while playing, although not necessarily 1791–1827, 2020, doi: 10.1007/s11423-020-09787-0.
in a faster way. In addition, the game interfaces received [13] M. Wang and X. Zheng, “Using game-based learning to support learning
science: A study with middle school students,” The Asia-Pacific Educ.
positive feedback, as students found them easy to understand Researcher, vol. 30, pp. 167–176, 2021, doi: 10.1007/s40299-020-
and useful. 00523-z.
An educational game such as Gea2 has the development of [14] M. Soliman and C. Guetl, “Intelligent pedagogical agents in immersive
virtual learning environments: A review,” in Proc. 33rd Int. Conv.
problem solving and critical-thinking skills among its natural Information and Communication Technology, Electronics and Micro-
aims; we did not measure the effectiveness of Gea2, out of electronics (MIPRO’10). IEEE, Opatija, Croatia, May 24–28 2010,
the experiments presented in this paper, and we regard further pp. 827–832.
[15] A. Serrano-Laguna, B. Manero, M. Freire, and B. Fernández-Manjón,
investigation in this direction as a topic for future work. On “A methodology for assessing the effectiveness of serious games and
the other hand, the first interventions on the game will be those for inferring player learning outcomes,” Multimedia Tools and Appl.,
directly based on the students’ feedback we had after the tests. vol. 77, pp. 2849–2871, 2018, doi: 10.1007/s11042-017-4467-6.
[16] P. Backlund and M. Hendrix, “Educational games — Are they worth
The most requested feature was to make an avatar available the effort? A literature survey of the effectiveness of serious games,” in
in the game. It would represent the player in the virtual Proc. 5th Int. Conf. Games and Virtual Worlds for Serious Applications
environment, and should be customizable by them, to reflect (VS-GAMES’13). IEEE, Poole, UK, Sept. 11–13 2013, pp. 1–8, doi:
10.1109/VS-GAMES.2013.6624226.
their personal traits. We believe that this could strengthen the [17] M. Liu, J. Kang, S. Liu, W. Zou, and J. Hodson, “Learning analytics as
gaming experience and students’ engagement. an assessment tool in serious games: A review of literature,” in Serious
Lastly, another future development that we see useful for the Games and Edutainment Applications, M. Ma and A. Oikonomou,
Eds. London, UK: Springer-Verlag, 2017, vol. II, pp. 537–563, doi:
game will be a text-to-speech engine, which would improve 10.1007/978-1-4471-2161-9.
the modularity and the chance to extend the use of the IPA. [18] M. A. Miljanovic and J. S. Bradbury, “A review of serious games for
programming,” in Proc. 4th Joint Conf. Serious Games (JCSG’18), ser.
LNCS, vol. 11243. Springer, Cham, Darmstadt, Germany, Nov. 7–8
ACKNOWLEDGMENT 2018, pp. 204–216, doi: 10.1007/978-3-030-02762-9 21.
[19] Ludwig. (2013) (Last accessed on 2021-08-15). [Online]. Available:
The authors thank the instructors and students of the schools http://www.playludwig.com/en/info/info-for-teachers/
ITI Pascal and ITI Bernini, for their support in testing and [20] W. Hagborg, “Grades and motivational orientation among high-school
validating Gea2. students,” J. Psychoeducational Assessment, vol. 10, no. 4, pp. 355–361,
1992, doi: 10.1177/073428299201000405.
[21] M. Virvou, G. Katsionis, and K. Manos, “Combining software games
R EFERENCES with education: Evaluation of its educational effectiveness,” Educ.
Technol. & Soc., vol. 8, no. 2, pp. 54–65, 2005.
[1] N. Pinkwart, “Another 25 years of AIED? Challenges and opportunities [22] S. L. Bowman, The Functions of Role-Playing Games: How Participants
for intelligent educational technologies of the future,” Int. J. Artif. Intell. Create Community, Solve Problems and Explore Identity. Jefferson, NC,
Educ., vol. 26, pp. 771–783, 2016, doi: 10.1007/s40593-016-0099-7. USA: McFarland & Co., 2010.
[2] E. Wenger, Artificial Intelligence and Tutoring Systems: Computational [23] A. Terracina et al., “Conquering an exo-planet through the use of
and Cognitive Approaches to the Communication of Knowledge. Mor- a virtual role playing game assisted by an emotionally intelligent
gan Kaufmann, 2014. pedagogical agent,” in Proc. 10th Eur. Conf. Games Based Learning
[3] C. Aldrich, Learning Online with Games, Simulations, and Virtual (ECGBL’16), Paisley, Scotland, UK, Oct. 6–7 2016, pp. 666–675.
Worlds: Strategies for Online Instruction. Hoboken, NJ, USA: Wiley, [24] NASA. (2017) Last accessed on 2021-08-15. [Online]. Avail-
2009, vol. 23. able: https://www.nasa.gov/press-release/nasa-telescope-reveals-largest-
[4] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game design batch-of-earth-size-habitable-zone-planets-around
elements to gamefulness: Defining ‘gamification’,” in Proc. 15th Int. [25] G. D. Borich, Effective Teaching Methods: Research-Based Practice,
Academic MindTrek Conf. (MindTrek’11). ACM, Tampere, Finland, 7th ed. Pearson Education India, 2011.
Sept. 28–30 2011, pp. 9–15, doi: 10.1145/2181037.2181040. [26] R. Bybee, “Teaching science as inquiry,” in Inquiring into Inquiry
[5] K. M. Kapp, The Gamification of Learning and Instruction: Game-Based Learning and Teaching in Science, J. Minstrell and E. H. van Zee, Eds.
Methods and Strategies for Training and Education. Hoboken, NJ, Washington, DC, USA: American Association for the Advancement of
USA: Wiley, 2012. Science, 2000, pp. 20–46.
[6] R. L. Lamb, L. Annetta, J. Firestone, and E. Etopio, “A meta-analysis [27] R. E. Mayer, “Should there be a three-strikes rule against pure discovery
with examination of moderators of student cognition, affect, and learning learning?” Amer. Psychologist, vol. 59, no. 1, pp. 14–19, 2004, doi:
outcomes while using serious educational games, serious games, and 10.1037/0003-066X.59.1.14.
simulations,” Comput. in Human Behav., vol. 80, pp. 158–167, 2018, [28] K. Goodnough and M. Cashion, “Exploring problem-based learning
doi: 10.1016/j.chb.2017.10.040. in the context of high school science: Design and implementation

issues,” School Sci. & Math., vol. 106, no. 7, pp. 280–295, 2006, doi: Massimo Mecella obtained the Ph.D. degree in
10.1111/j.1949-8594.2006.tb17919.x. engineering in computer science from Sapienza
[29] D. Cichon and J. Ellis, “The effects of MATH Connections on student University of Rome, Rome, Italy, in 2002. He is
achievement, confidence, and perception,” in Standards-based school currently a full professor with the Department of
mathematics curricula. Evanston, IL, USA: Routledge, 2003, pp. 345– Computer, Control, and Management Engineering,
374, doi: 10.4324/9781003064275. Sapienza University of Rome, Rome, Italy. His
[30] R. Q. Berry III, P. A. Reed, J. M. Ritz, C. Y. Lin, S. Hsiung, and research focuses on business process management,
W. Frazier, “STEM initiatives: Stimulating students to improve science cyber-physical systems and the Internet of Things,
and mathematics achievement,” The Technology Teacher, vol. 64, no. 4, advanced human–computer interfaces, software ar-
pp. 23–29, 2004. chitectures, and service-oriented computing, with
[31] J. Frykholm and G. Glasson, “Connecting science and mathematics applications in multiple fields including digital gov-
instruction: Pedagogical context knowledge for teachers,” School Sci. ernment, smart spaces, Industry 4.0, healthcare, disaster/crisis response and
& Math., vol. 105, no. 3, pp. 127–141, 2005, doi: 10.1111/j.1949- management, cybersecurity, and the digital humanities. He has published over
8594.2005.tb18047.x. 180 research papers and chaired various conferences in the above-mentioned
[32] S. Nikitina and V. B. Mansilla. “Three strategies for interdisciplinary areas.
math and science teaching: A case of the Illinois Mathematics and
Science Academy,”. Harvard Graduate School of Education, Cambridge,
MA, USA, Goodwork Project Rep. Ser. no. 21, Mar. 2003. [Online].
Available: https://www.researchgate.net/publication/228379792 Three
Strategies for Interdisciplinary Math and Science Teaching A Case Francesco Sapio obtained the PhD degree in Engi-
of the Illinois Mathematics and Science Academy neering in Computer Science from Sapienza Univer-
[33] Photon. (2017) Photon chat engine. Last accessed on 2021-08-15. sity of Rome, Rome, Italy, in 2021. He is currently a
[Online]. Available: https://www.photonengine.com/en/chat researcher with the Department of Computer, Con-
[34] L. Ferro, F. Sapio, M. Mecella, M. Temperini, and A. Terracina, trol and Management Engineering, Sapienza Univer-
“Intelligent pedagogic agents (IPAs) in GEA2, an educational game sity of Rome, Rome, Italy. His research focuses on
to teach STEM topics,” in Proc. 10th Int. Conf. Methodologies and Serious Games, Cultural Heritage, Social and Col-
Intelligent Systems for Technology Enhanced Learning (MIS4TEL’20), laborative Learning, Game Based Learning, Technol-
ser. AISC, vol. 1236. Springer-Verlag, Online, Oct. 6–8 2020, pp. ogy Enhanced Learning, AI applied to videogames,
226–236, doi: 10.1007/978-3-030-52287-2 23. Gaming Software Architectures, Human-Computer
[35] M. Lease, “Natural language processing for information retrieval: The Interaction, Industry 4.0, with applications in 3-D
time is ripe (again),” in Proc. Conf. Information and Knowledge Man- Visualization, Architecture, Gaming, Healthcare and Entertainment.
agement (CIKM’07), 1st Ph.D. Workshop in CIKM (PIKM’07). ACM,
Lisbon, Portugal, Nov. 6–10 2007, pp. 1–8, 10.1145/1316874.1316876.
[36] C. Conati, “Probabilistic Assessment of User’s Emotions in Edu-
cational Games,” J. Appl. Artif. Intell., pp. 555–575, 2002, doi:
10.1080/08839510290030390. Marco Temperini obtained the Ph.D. degree in
[37] C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Informa- computer science from Sapienza University of
tion Retrieval. Cambridge University Press, 2008. Rome, Rome, Italy, in 1992. He is currently an as-
[38] D. R. Michael and S. L. Chen, Serious Games: Games that Educate, sociate professor with the Department of Computer,
Train, and Inform. Boston, MA, USA: Thomson Course Technology, Control, and Management Engineering at the same
2005. university. His research activity in the education
[39] R. T. Hays. “The effectiveness of instructional games: A literature arena is on adaptive e-learning, social and collab-
review and discussion,”. Naval Air Warfare Center Training Systems orative learning, game-based learning, automated
Division, Orlando, FL, USA, Rep. no. 2005-004, Nov. 2005. [Online]. assessment, and learning analytics. He has orga-
Available: https://apps.dtic.mil/sti/pdfs/ADA441935.pdf nized several international workshops and chaired
[40] J. Enfield, R. D. Myers, M. Lara, and T. W. Frick, “Innovation a number of international conferences in the field
diffusion: Assessment of strategies within the diffusion simulation of technology-enhanced learning. He has been a work-package leader and
game,” Simul. & Gaming, vol. 43, no. 2, pp. 188–214, 2012, doi: research unit coordinator for several European Union research projects.
10.1177/1046878111408024.
[41] C. Boston, “The concept of formative assessment,” Practical Assess-
ment, Res., & Eval., vol. 8, Article 9, 2002, doi: 10.7275/kmcq-dj31.
[42] N. J. Salkind, Encyclopedia of research design. Newbury Park, CA,
USA: SAGE, 2010, vol. 1, doi: 10.4135/9781412961288. Annalisa Terracina obtained the Ph.D. degree
in computer science from Sapienza University of
Rome, Rome, Italy, in 2017. She has been working
in the field of Earth observation and space missions
for many years, previously serving as a consultant
for the European Space Agency (ESA) and the
Italian Space Agency (ASI) as well as a technical
manager for several European Union projects. Since
2018, she has been a project manager for ESERO
Lauren S. Ferro obtained the Ph.D. degree in Media Italia, a collaboration between ESA and ASI focus-
and Communication, specializing in player and user ing on space-related STEM education.
modeling and profiling, from the Royal Melbourne
Institute of Technology, Melbourne, Australia, in
2017. She is currently pursuing a second Ph.D. from
the Department of Computer, Control, and Man-
agement Engineering, Sapienza University, Rome,
Italy, focusing on human factors in cybersecurity.
Her research activity centers primarily around un-
derstanding human and player behavior in various
interactive experiences in the areas of gamification,
game-based learning, cultural heritage, cybersecurity, and psychology.

S UPPLEMENTARY M ATERIAL • Q15: What do all the solar system planets have in
The list of questions used for the Summative assessments common?
(translated from local language) were: • Q16: What is an astro-biologist?
• Q17: Are volcanoes important for the origin of life?
• Q1: What is an ESO-Planet.
• Q18: Who should be part of a team of scientists studying
• Q2: What does integrated science in school means?
an exoplanet?
• Q3: What is it a NEO?
• Q19: What is the acceleration of gravity on Earth worth?
• Q4: What does ASI stand for?
• Q20: What is a conceptual map useful for?
• Q5: With which experiment is it possible to calculate the
• Q21: What the water cycle defines?
gravity acceleration?
• Q22: What is the difference between a virus and a
• Q6: The density of an object is defined as ...
bacterium?
• Q7: What are the 3 main gases that make up the earth’s
atmosphere? Table XIII shows the partition of the questions in categories,
• Q8: What instrument can I use to study the composition used for the analysis of effectiveness of Gea2. The names of
of a gas? the categories are self-explanatory. Notice that each question
• Q9: The inclination of the Earth’s axis is important for can pertain to, and so being included in, different categories.
which reason?
• Q10: How long does it take for humans to die of
hypothermia if exposed to a temperature of -20◦ ? TABLE XIII
• Q11: What is the atmospheric pressure of the earth’s
Q UESTIONS IN EACH CATEGORY
atmosphere at sea level? Category Questions
• Q12: Why is the Earth’s magnetic field important? General Q1, Q2, Q3, Q4, Q13, Q16, Q17, Q18, Q20
• Q13: Have we discovered liquid water on other planets Biology Q21, Q7, Q8, Q10, Q22, Q16, Q17
Environment Q21, Q7
in the solar system? Astronomy Q3, Q9, Q13, Q12, Q15, Q16
• Q14: Does the inclination of the Earth’s axis affect the Chemistry Q6, Q7, Q8, Q22
length of the day? Physics Q5, Q6 Q11, Q12, Q14, Q15, Q19

Lesson Plan Grade 5 Math-Demo Multiplying Decimals
100% (13)
Lesson Plan Grade 5 Math-Demo Multiplying Decimals
3 pages
Franz Wa 2014
No ratings yet
Franz Wa 2014
21 pages
Experiencia de Gamificacion en Secundaria en El Ap
No ratings yet
Experiencia de Gamificacion en Secundaria en El Ap
22 pages
Development of An Educational Game Based On Iot
No ratings yet
Development of An Educational Game Based On Iot
10 pages
Sistema Inteligente de Enseñanza Interactiva A Través de Videojuegos
No ratings yet
Sistema Inteligente de Enseñanza Interactiva A Través de Videojuegos
14 pages
Journal of Computer Science Research - Vol.5, Iss.4 October 2023
No ratings yet
Journal of Computer Science Research - Vol.5, Iss.4 October 2023
62 pages
Digital Games in Engineering Education Systematic Review and Future Trends
No ratings yet
Digital Games in Engineering Education Systematic Review and Future Trends
20 pages
Designing Effective Serious Games
No ratings yet
Designing Effective Serious Games
21 pages
Electronics 13 00890
No ratings yet
Electronics 13 00890
24 pages
Towards The Design of Intelligent Educational Gaming Systems
No ratings yet
Towards The Design of Intelligent Educational Gaming Systems
4 pages
Serious Games
No ratings yet
Serious Games
21 pages
Teaching Digital Circuit Design With A 3-D Video Game The Impact of Using In-Game Tools On Students Performance
No ratings yet
Teaching Digital Circuit Design With A 3-D Video Game The Impact of Using In-Game Tools On Students Performance
8 pages
Evaluation of an Ai Assisted Adaptive Educational Game System
No ratings yet
Evaluation of an Ai Assisted Adaptive Educational Game System
10 pages
Trends in Technology: Gaming Presentation
No ratings yet
Trends in Technology: Gaming Presentation
11 pages
Elzaba, STEM19-2 #2245F
No ratings yet
Elzaba, STEM19-2 #2245F
7 pages
Circuit Craft 2D App Final Draft
No ratings yet
Circuit Craft 2D App Final Draft
9 pages
Comparing Traditional Teaching and Game-Based Learning Using Teacher-Authored Games On Computer Science Education
No ratings yet
Comparing Traditional Teaching and Game-Based Learning Using Teacher-Authored Games On Computer Science Education
7 pages
Open-Ended Video Games A Model For Devel
No ratings yet
Open-Ended Video Games A Model For Devel
32 pages
Learning Physics With Digital Game Simul
No ratings yet
Learning Physics With Digital Game Simul
13 pages
Digital Game Electrical Enginnering Motivation
No ratings yet
Digital Game Electrical Enginnering Motivation
6 pages
Digital Game-Based Learning: An Overview
100% (1)
Digital Game-Based Learning: An Overview
13 pages
An Exploratory Study of Blending The Virtual World and The L - 2018 - Computers
No ratings yet
An Exploratory Study of Blending The Virtual World and The L - 2018 - Computers
15 pages
Lamb 2015
No ratings yet
Lamb 2015
15 pages
Fostering Science Teachers' Design For Inquiry-Based Learning by Using A Serious Game
No ratings yet
Fostering Science Teachers' Design For Inquiry-Based Learning by Using A Serious Game
5 pages
Chapter Ii: Review of Related Literature
No ratings yet
Chapter Ii: Review of Related Literature
19 pages
Game-Based Learning: Increasing The Logical-Mathematical, Naturalistic, and Linguistic Learning Levels of Primary School Students
No ratings yet
Game-Based Learning: Increasing The Logical-Mathematical, Naturalistic, and Linguistic Learning Levels of Primary School Students
9 pages
Investigating Learners' Technology Engagement - A Perspective From Ubiquitous Game-Based Learning in Smart Campus
No ratings yet
Investigating Learners' Technology Engagement - A Perspective From Ubiquitous Game-Based Learning in Smart Campus
9 pages
Juego y Curriculo
No ratings yet
Juego y Curriculo
3 pages
1 s2.0 S2949678024000205 Main
No ratings yet
1 s2.0 S2949678024000205 Main
10 pages
5 Ceorad - Gamification As Support in Teaching Physics and Mathematics For Developing Digital and Sustainable Competences
No ratings yet
5 Ceorad - Gamification As Support in Teaching Physics and Mathematics For Developing Digital and Sustainable Competences
16 pages
Current Trends in GameBased Learning
No ratings yet
Current Trends in GameBased Learning
129 pages
2
No ratings yet
2
17 pages
To Learn Scientifically, Effectively, and Enjoyably-A Review of Educational Games
No ratings yet
To Learn Scientifically, Effectively, and Enjoyably-A Review of Educational Games
10 pages
Preproduction For Games Module 3
No ratings yet
Preproduction For Games Module 3
61 pages
Game Based Learning Vs Gamification From The Higher Education Students Perspective
No ratings yet
Game Based Learning Vs Gamification From The Higher Education Students Perspective
7 pages
Thesis About Educational Computer Games
100% (3)
Thesis About Educational Computer Games
5 pages
Designing Game Based Learning For High School Artificial Intelligence Education
No ratings yet
Designing Game Based Learning For High School Artificial Intelligence Education
15 pages
AI in Board Game Based Learning
No ratings yet
AI in Board Game Based Learning
15 pages
Game Immersion Experience
No ratings yet
Game Immersion Experience
22 pages
A Critique Digital Game Based Learning in The Light of Marshall Machluhan 1
No ratings yet
A Critique Digital Game Based Learning in The Light of Marshall Machluhan 1
7 pages
Current Emerging Technologies in Today's Curriculum and Their Effects On Student Education Brieann Robles College of Southern Nevada
No ratings yet
Current Emerging Technologies in Today's Curriculum and Their Effects On Student Education Brieann Robles College of Southern Nevada
9 pages
Design, Motivation, and Frameworks in Game-Based Learning (Wee Hoe Tan (Editor) )
No ratings yet
Design, Motivation, and Frameworks in Game-Based Learning (Wee Hoe Tan (Editor) )
333 pages
Educational Computer Games Thesis
100% (3)
Educational Computer Games Thesis
8 pages
Urn NBN Fi Uef-20150837
No ratings yet
Urn NBN Fi Uef-20150837
101 pages
International Journal of Educational Technology in Higher Education 2365-9440
No ratings yet
International Journal of Educational Technology in Higher Education 2365-9440
4 pages
A Systematic Review of Serious Games As Tools For
No ratings yet
A Systematic Review of Serious Games As Tools For
29 pages
GameBased Learning Gamification in Education and Serious Games
No ratings yet
GameBased Learning Gamification in Education and Serious Games
184 pages
Revolutionising Role-Playing Games With ChatGPT
No ratings yet
Revolutionising Role-Playing Games With ChatGPT
14 pages
Students ' Perception of Kahoot!'s Influence On Teaching and Learning
No ratings yet
Students ' Perception of Kahoot!'s Influence On Teaching and Learning
23 pages
Virtual Environment For Teaching and Learning in Residential Electrical Installations Projects
No ratings yet
Virtual Environment For Teaching and Learning in Residential Electrical Installations Projects
6 pages
R2L - SG and Gami Cation - A Systematic Literature Review
No ratings yet
R2L - SG and Gami Cation - A Systematic Literature Review
13 pages
(Richard T. Beckwith, Harold Chaput, Brian M. Slat
No ratings yet
(Richard T. Beckwith, Harold Chaput, Brian M. Slat
191 pages
Intelligent Game Recommendation System
No ratings yet
Intelligent Game Recommendation System
9 pages
Game-Based Learning in Computer Science Education: A Scoping Literature Review
No ratings yet
Game-Based Learning in Computer Science Education: A Scoping Literature Review
23 pages
Concept Paper Sir Kirk
No ratings yet
Concept Paper Sir Kirk
6 pages
A Model For The Design of Puzzle-Based Games Including Virtual and Physical Objects
No ratings yet
A Model For The Design of Puzzle-Based Games Including Virtual and Physical Objects
17 pages
Laptop Programs For Students: Education & Technology
No ratings yet
Laptop Programs For Students: Education & Technology
4 pages
Approaching Sustainability Learning Via Digital Serious Games
No ratings yet
Approaching Sustainability Learning Via Digital Serious Games
18 pages
Borad Games and Board Design As Learning Tools For Complex Scientific Concepts: Some Experiences
No ratings yet
Borad Games and Board Design As Learning Tools For Complex Scientific Concepts: Some Experiences
14 pages
A Theoretical Framework For Serious Game Design - Exploring Pedago
No ratings yet
A Theoretical Framework For Serious Game Design - Exploring Pedago
23 pages
Final Activity 1
No ratings yet
Final Activity 1
3 pages
Lesson Plan For One Fish Two Fish
No ratings yet
Lesson Plan For One Fish Two Fish
1 page
World Art Unit Plan
No ratings yet
World Art Unit Plan
10 pages
IEP at A Glance - Clean PDF
No ratings yet
IEP at A Glance - Clean PDF
1 page
Doctor of Theology: Silliman University Divinity School
No ratings yet
Doctor of Theology: Silliman University Divinity School
10 pages
Narrative Report: Module: Commercial Cooking
No ratings yet
Narrative Report: Module: Commercial Cooking
3 pages
Knowledge Power and Educational Reform Applying The Sociology of Basil Bernstein 1st Edition Rob Moore Instant Download
No ratings yet
Knowledge Power and Educational Reform Applying The Sociology of Basil Bernstein 1st Edition Rob Moore Instant Download
52 pages
Module 4. Lesson 2
No ratings yet
Module 4. Lesson 2
5 pages
Latest CBSE Guidelines For Primary
No ratings yet
Latest CBSE Guidelines For Primary
2 pages
PPST PDF
No ratings yet
PPST PDF
40 pages
Analysis of Rhetorical Moves of Journal Articles and Its Implication To The Teaching of Academic Writing
No ratings yet
Analysis of Rhetorical Moves of Journal Articles and Its Implication To The Teaching of Academic Writing
11 pages
Language Art Example Lesson Plan Action Song
0% (1)
Language Art Example Lesson Plan Action Song
2 pages
Unit 4 Prof Ed 7
No ratings yet
Unit 4 Prof Ed 7
5 pages
Aip-Mtpa-Smea 2017-2018
No ratings yet
Aip-Mtpa-Smea 2017-2018
27 pages
Clouds Lesson Plan
0% (1)
Clouds Lesson Plan
3 pages
Assignment 6
No ratings yet
Assignment 6
2 pages
PED F2F Brochure
No ratings yet
PED F2F Brochure
2 pages
Tracking Form PD - School Heads and Supervisors
No ratings yet
Tracking Form PD - School Heads and Supervisors
5 pages
10 Plants That Shook The World 2
No ratings yet
10 Plants That Shook The World 2
7 pages
Continuous Assessment Specification
No ratings yet
Continuous Assessment Specification
2 pages
Practicum Innovation Project
100% (2)
Practicum Innovation Project
4 pages
Effects of Educational Games Towards Senior High School Students Learning
No ratings yet
Effects of Educational Games Towards Senior High School Students Learning
9 pages
Educ 301 The Teacher and The School Curriculum First Semester, AY 2020-2021 3 None 3 None
No ratings yet
Educ 301 The Teacher and The School Curriculum First Semester, AY 2020-2021 3 None 3 None
8 pages
Key Assessment - Instructional Design
No ratings yet
Key Assessment - Instructional Design
9 pages
ENGL 5010 Syllabus (FA23)
No ratings yet
ENGL 5010 Syllabus (FA23)
7 pages
Week 4 Day 1
100% (1)
Week 4 Day 1
4 pages
Internship Final Report
No ratings yet
Internship Final Report
11 pages
Final Assessment Report - : B. Ed. Primary Year 2, Semester 4
No ratings yet
Final Assessment Report - : B. Ed. Primary Year 2, Semester 4
7 pages
Lesson Plan For Counteries, Nationality and Languages
No ratings yet
Lesson Plan For Counteries, Nationality and Languages
1 page

Ferro Postprint Gea2 2021 PDF

Uploaded by

Ferro Postprint Gea2 2021 PDF

Uploaded by

This article has been accepted for publication in a future issue of this journal, but has not been

Gea2: A Serious Game for Technology-Enhanced

entertainment. (like in [22]) by assimilating with an unfamiliar role (i.e., one

Role Description Example Tasks

TABLE V any correlation of these two answers with the likeability or

Correlation Coefficient .859435333

TABLE VI • Quizzes and tasks were proposed in Italian;

TABLE X expected the game would improve the students understanding

You might also like