Current Trends

in Game-Based
Learning
Edited by
Margarida M. Marques and Lúcia Pombo
Printed Edition of the Special Issue Published in Education Sciences

www.mdpi.com/journal/education
Current Trends in Game-Based
Learning
Current Trends in Game-Based
Learning

Editors
Margarida M. Marques
Lúcia Pombo

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin
Editors
Margarida M. Marques Lúcia Pombo
Department of Education Department of Education
and Psychology and Psychology
University of Aveiro University of Aveiro
Aveiro Aveiro
Portugal Portugal

Editorial Office
MDPI
St. Alban-Anlage 66
4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal
Education Sciences (ISSN 2227-7102) (available at: www.mdpi.com/journal/education/special
issues/current trends game based learning supported mobile devices).

For citation purposes, cite each article independently as indicated on the article page online and as
indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. Journal Name Year, Volume Number,
Page Range.

ISBN 978-3-0365-2316-3 (Hbk)

ISBN 978-3-0365-2315-6 (PDF)

© 2021 by the authors. Articles in this book are Open Access and distributed under the Creative
Commons Attribution (CC BY) license, which allows users to download, copy and build upon
published articles, as long as the author and publisher are properly credited, which ensures maximum
dissemination and a wider impact of our publications.
The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons
license CC BY-NC-ND.
Contents

About the Editors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Margarida M. Marques and Lúcia Pombo

Current Trends in Game-Based Learning—Introduction to a Special Collection of Research
Reprinted from: Educ. Sci. 2021, 11, 622, doi:10.3390/educsci11100622 . . . . . . . . . . . . . . . . 1

Lúcia Pombo and Margarida M. Marques

The Potential Educational Value of Mobile Augmented Reality Games: The Case of EduPARK
App
Reprinted from: Educ. Sci. 2020, 10, 287, doi:10.3390/educsci10100287 . . . . . . . . . . . . . . . . 3

Margarida M. Marques and Lúcia Pombo

The Impact of Teacher Training Using Mobile Augmented Reality Games on Their Professional
Development
Reprinted from: Educ. Sci. 2021, 11, 404, doi:10.3390/educsci11080404 . . . . . . . . . . . . . . . . 23

Dionı́sia Laranjeiro
Development of Game-Based M-Learning Apps for Preschoolers
Reprinted from: Educ. Sci. 2021, 11, 229, doi:10.3390/educsci11050229 . . . . . . . . . . . . . . . . 45

Rita Tavares, Rui Marques Vieira and Luı́s Pedro

Mobile App for Science Education: Designing the Learning Approach
Reprinted from: Educ. Sci. 2021, 11, 79, doi:10.3390/educsci11020079 . . . . . . . . . . . . . . . . 71

Friday Joseph Agbo, Ismaila Temitayo Sanusi, Solomon Sunday Oyelere and Jarkko
Suhonen
Application of Virtual Reality in Computer Science Education: A Systemic Review Based on
Bibliometric and Content Analysis Methods
Reprinted from: Educ. Sci. 2021, 11, 142, doi:10.3390/educsci11030142 . . . . . . . . . . . . . . . . 95

v
About the Editors
Margarida M. Marques
Researcher at the CIDTFF - Research Centre on Didactics and Technology in the Education of
Trainers, in University of Aveiro (Portugal). She has expertise in technology-supported game-based
learning in science education and her research interests include scientific literacy and teacher
professional development. As a member of the EduPARK project, she won the 2018 ECIU Team
Award for Innovation in Teaching and Learning of the European Consortium of Innovative
Universities (ECIU). Additionally, she has been teaching curricular units of High Degrees and Master
Courses and supervising PhD and post-doc students in the Department of Education and Psychology.
She holds a PhD in Didactics and Training, branch of Didactics and Curricular Development.

Lúcia Pombo
Auxiliary Researcher and Vice-Coordinator of the CIDTFF - Research Centre on Didactics and
Technology in the Education of Trainers, in University of Aveiro (Portugal). She holds two PhDs:
in Biology and in Education. Her expertise is on Technology Enhanced learning, Science Education,
mobile Learning, gamification and quality in Education. She is the Coordinator of the EduPARK
project funded by FEDER-FCT. The EduPARK seeks to use outdoor learning strategies, by means of an
interactive mobile Augmented Reality app that supports geocaching activities in formal, non-formal
and informal contexts, in Aveiro city park, in Portugal. She was awarded by the prestigious European
Consortium of Innovative Universities (ECIU) Team Award for Innovation in Teaching and Learning,
in 2018. She has been lecturing curricular units, in the Department of Education and Psychology, such
as Natural Sciences and Educational Intervention Projects, she supervises PhD and MA students in
ICT in Education and she has been also involved in teachers training.

vii
 education
sciences
Editorial
Current Trends in Game-Based Learning—Introduction to a
Special Collection of Research
Margarida M. Marques * and Lúcia Pombo *

CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers, Department of Education
and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal
* Correspondence: [email protected] (M.M.M.); [email protected] (L.P.)

The potential of digital games to promote learning is a growing field of research.

Researchers have been extensively analyzing the impact of games on learner motivation
and engagement in educational settings [1] and even reporting the contribution of this
approach to a variety of learning outcomes, such as concept understanding [2,3] and soft
skills development [4].
A myriad of technological options can be used to support digital game-based learning.
One popular technology in this context is the mobile device, considering its high penetra-
tion rate in our societies, even among young people. These can be combined with other
technologies, such as Augmented Reality (AR) or Virtual Reality (VR), to increase students’
motivation and engagement in learning processes [5,6].
Because of this, there is an emergent need to know and promote good practices in
the development and implementation of game-based learning approaches in educational
settings. This was the motto for the proposal of the Education Sciences (ISSN: 2227-7102)



Special Issue “Current Trends in Game-Based Learning”. This book is a reprint of this


Special Issue, collecting a set of five papers that illustrate the contribution of innovative
Citation: Marques, M.M.; Pombo, L. approaches to education, specifically the ones exploring the motivational factors associated
Current Trends in Game-Based with playing games and the technology that may support them.
Learning—Introduction to a Special Considering the above, the first work in this book, by Lúcia Pombo and Margarida M.
Collection of Research. Educ. Sci. Marques, presents a study where the “Educational Value Scale of Mobile AR Games” was
2021, 11, 622. https://doi.org/ used in an illustrative case: The EduPARK app. The study revealed that games sustained
10.3390/educsci11100622 by mobile devices and integrating AR can have high educational value, particularly with
students aged 10 to 15 years old.
Received: 22 September 2021
Second is a piece from the same authors focusing on the need for teacher training in
Accepted: 30 September 2021
these innovative approaches. Hence, the authors conducted a case study of the impact of a
Published: 11 October 2021
teacher-training initiative on trainees’ professional development. It revealed improvements
in teachers’ knowledge and experience with game-based learning, mobile learning, and
Publisher’s Note: MDPI stays neutral
AR, as well as their ability to identify both benefits and barriers to these approaches.
with regard to jurisdictional claims in
The third work, by Dionísia Laranjeiro, discusses the process of designing, developing,
published maps and institutional affil-
and evaluating a set of four thematic educational apps composed of a set of games suitable
iations.
for preschoolers in autonomous or guided activities. This work was developed under the
“Aprender XXI” (Learn XXI) project by a multidisciplinary team of educational researchers,
technology developers, and end-users, in this case, children and kindergarten educators.
The fourth chapter, by Rita Tavares, Rui Marques Vieira, and Luís Pedro, focuses on the
Copyright: © 2021 by the authors.
promotion of primary students’ scientific competences and self-regulated learning through
Licensee MDPI, Basel, Switzerland.
their interaction with a mobile app. More specifically, it presents the conception process of
This article is an open access article
the interaction design with a strong theoretical base, as it considers a learning approach
distributed under the terms and
proposal that combines the Universal Design for Learning principles, the Inquiry-Based
conditions of the Creative Commons
Attribution (CC BY) license (https://
Science Education and the BSCS 5E Instructional Model.
creativecommons.org/licenses/by/
Finally, the work of Friday Joseph Agbo, Ismaila Temitayo Sanusi, Solomon Sunday
4.0/).
Oyelere, and Jarkko Suhonen presents a literature review on the use of VR in computer sci-

1
Educ. Sci. 2021, 11, 622

ence education. Among other relevant findings, this study reveals that game-based learning
and gamification have been leveraged for computer science education integrating VR.

References
1. Shu, L.; Liu, M. Student Engagement in Game-Based Learning: A Literature Review from 2008 to 2018. J. Educ. Multimed.
Hypermedia 2019, 28, 193–215. Available online: https://www.learntechlib.org/primary/p/183934/ (accessed on 13 April 2020).
2. Huang, Y.-L.; Chang, D.-F.; Wu, B. Mobile Game-Based Learning with a Mobile App: Motivational Effects and Learning
Performance. J. Adv. Comput. Intell. Intell. Inform. 2017, 21, 963–970. [CrossRef]
3. Pombo, L.; Marques, M.M. Improving students’ learning with a mobile augmented reality approach—the EduPARK game.
Interact. Technol. Smart Educ. 2019, 16, 392–406. [CrossRef]
4. Kailani, S.; Newton, R.; Pedersen, S. Game-Based learning and problem-solving skills: A systematic review of the literature. In
Proceedings of the EdMedia + Innovate Learning, Amsterdam, The Netherlands, 24–28 June 2019; pp. 1127–1137. Available
online: https://www.learntechlib.org/primary/p/210119/ (accessed on 13 April 2020).
5. Pombo, L.; Marques, M.M. An App that Changes Mentalities about Mobile Learning—The EduPARK Augmented Reality Activity.
Computers 2019, 8, 37. [CrossRef]
6. Bodzin, A.; Junior, R.A.; Hammond, T.; Anastasio, D. Investigating Engagement and Flow with a Placed-Based Immersive Virtual
Reality Game. J. Sci. Educ. Technol. 2020, 30, 347–360. [CrossRef]

2
 education
sciences
Article
The Potential Educational Value of Mobile
Augmented Reality Games: The Case of
EduPARK App
Lúcia Pombo * and Margarida M. Marques *
CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers, Department of Education
and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal
* Correspondence: [email protected] (L.P.); [email protected] (M.M.M.)




Received: 22 September 2020; Accepted: 14 October 2020; Published: 16 October 2020


Abstract: New teaching methodologies are nowadays integrating mobile devices, augmented reality
(AR), and game-based learning in educational contexts. The combination of these three elements is
considered highly innovative, and it allows learning to move beyond traditional classroom environments
to nature spaces that students can physically explore. The literature does not present many studies
of this approach’s educational value. The purpose of the study is to present an illustrative case of a
mobile AR game in order to analyse its educational value based on the users’ opinion, both teachers
and students, and on logs of game results. Through a mixed method approach, the educational value
scale was applied to 924 users after playing the EduPARK app in a Green City Park. Results revealed
high educational value scores, especially among teachers and students of 2nd and 3rd Cycles of Basic
Education (83.0 for both). Hence, this particular software seems to be more suitable for 10–15 years-old
students who highlighted motivational features, such as treasure hunting, points gathering, the use of
mobile devices in nature settings, and AR features to learn. This study empirically revealed that mobile
AR games have educational value, so these specific game features might be useful for those who are
interested in creating or using games supported by apps for educational purposes.

Keywords: educational value; mobile learning; game-based learning; augmented reality; mixed methods

1. Introduction
Game strategies are used in Education with the aim of improving the learning process by making
use of the motivating effects of game elements and techniques. Both strategies of game-based learning
and gamification intend to motivate students for learning. Engagement and motivation are key factors
that influence the students’ performance during a learning process [1]. However, game-based learning is
not the same as gamification, as the first makes use of a game for learning, and the latter integrates a set of
technical concepts (such as points, badges, and leaderboards) in the learning process. Games have been
used for a long time in teaching and learning; however, they have not always been properly investigated.
The increasing use of mobile phones makes it possible to explore digital educational games in
outside environments, and when combined with augmented reality (AR) content, the educative effect
may be exponential and result in best practices [2]. With the use of AR, the real-world environment can
be augmented by providing users with accurate digital overlays. AR is a promising technology that
has the potential to encourage learners to explore learning materials from a totally new perspective.
Additionally, technological advancements along with the proliferation of wireless mobile devices, such
as smartphones and tablets, allow for widening the scope of educational AR applications [3].
The integration of each of the above-mentioned elements (games, mobile devices, and AR) in
educational contexts is considered innovative [3–5]. Moreover, their combination has revealed diverse

3
Educ. Sci. 2020, 10, 287

benefits for learning, both at the cognitive and emotional levels [6–8], and supports learning to move
beyond traditional classroom environments to nature spaces that students can physically explore [9].
Regarding off the shelf games, Lin, Huang, and Lin [10] analyse the game design of “Pokémon
GO” in contexts where parents play together with their children, and found that some of the most
valued features are its user-friendly interface, going outdoors, visiting places of interest, and combining
games with exercise.
In what concerns games developed for educational purposes, Preka and Rangoussi [4] combine
quick response (QR) codes with AR to create a collaborative game for music learning in Early Childhood
Education in indoors and outdoors activities. Evaluation results indicate that the AR–QR technology
is a powerful tool that triggers and sustains children’s interest during the learning process and can
enhance their cognitive and collaborative skills, as well as their social interaction.
Another study [11] presents a technical framework, Mobile Augmented-Reality Games for Instructional
Support (MAGIS), for the development of this type of game. The authors illustrated the usefulness of
the framework for implementing outdoor location-based educational games through the analysis of the
game “Igpaw: Intramuros” for History learning. The authors highlighted that players’ enjoyment of the
game tended to be adversely affected by weather conditions, long walking distances between markers,
and devices’ battery life.
The EduPARK app is an example of a successful mobile AR game for learning [12]. It was developed
to foster collaboration [13], and authentic [14] and situated learning [15] outside the classroom, under the
umbrella of constructivist learning [16], as the learner or app user assumes an active role by constructing
new knowledge within the articulation between the learning experience and previous knowledge.
The EduPARK app aims to be explored in situ, an Urban Green Park, which is a context that can be
used to promote new modes of learning in science education, since the ability to understand ecosystems
is enhanced by experiences in real environments [17]. The app gives access to excellent cross subjects’
educational materials, both in Portuguese and in English. It comprises a very useful tool for Portuguese
teachers and students to explore scientific knowledge by accessing contextualised and appealing
information on biological and historical references that augment the experience of exploring a local
Urban Green Park. This type of innovative educational resource is not common in Portuguese speaking
countries, hence, adding to the relevance of the EduPARK app.
Users’ subjective perceptions about educational software can be evaluated with different tools,
for example, System Usability Scale (SUS) and Educational Value Scale (EVS). The SUS is a robust,
effective, and inexpensive tool, developed by Brooke in 1986, to quickly measure users’ subjective
perceptions of computer systems usability [18]. With only 10 questions on a 5-point scale (ranging from
“strongly agree” to “strongly disagree”), it is one of the most used tools for measuring perceptions of
usability, with a scale varying from 0 to 100 [19]. Sauro [20] reviewed 500 studies and found out that a
SUS score of 68 could be considered average.
The tool used in this study, the EVS, was developed based on the SUS to quickly and easily collect
users’ subjective rating of the educational value of an app for outdoor green settings, considering
the following dimensions: (a) Learning value; (b) intrinsic motivation; (c) engagement; (d) authentic
learning; (e) lifelong learning; and (f) conservation and sustainability habits. Taking this into account,
two items from each dimension were included in the scale, as described in [9,21]. As this is a new tool,
there is a need of further studies to establish standards to support values interpretation.
Previous studies 9 showed that the EduPARK game achieved an average Educational Value Scale
(EVS) of 83.8 and an average System Usability Scale (SUS) of 80.2, according to 244 students attending
the 2nd or 3rd Cycles of Basic Education. This demonstrates its high educational value and usability
for students of these school levels, indicating that the app can be used, as reference, by the international
community, namely those who are interested in designing educational apps based on previous good
experiences. As the EduPARK app is intended to be used by several school levels (as explained in
the next section), there is a need to aggregate data in order to yield a wider picture of its educational
value and its adequacy for different target publics. Hence, the purpose of this study is to present the

4
Educ. Sci. 2020, 10, 287

EduPARK app, an illustrative case of a mobile AR game, in order to analyse its educational value
based on the users’ opinion and on logs of game results. Aggregated data includes 924 questionnaires,
filled in by teachers and students of non-higher education contexts.
The analysis sustains a reflection on the enhancement of the educational value of mobile AR
games by presenting the specificities of the EduPARK game, including its educational resources, such
as 2D and 3D models that mix real and virtual worlds, combining familiar technology with outdoor
learning strategies. Finally, some guidelines arise that might be useful to inspire other educational
game producers by providing theoretical and practical frameworks that can be useful in other natural
environments to open horizons and opportunities for Education.

2. The EduPARK Game

As the main purpose of this paper is to analyse the educational value of a mobile AR game
accessed through the EduPARK app, there is a need to present the educational principles that guided
its development and the app’s features that may enhance learning.
The EduPARK app is the main product of the project with the same name. It was developed with
the aim of supporting social constructivism approaches to teaching [16] in a game-based approach for
student engagement and motivation. These are well known factors influencing learning [22]. Hence,
the app was created to support the users’ construction of meaning through experiences in an Urban
Green Park, being meaning influenced by the interaction of the learners’ prior knowledge with the new
experiences, as well as by their interactions with others. When designing the users’ interaction with the
app, the followed principles were: (i) To stimulate the app users to become active participants in their
own learning, in a student-centred learning process with hands-on activities; (ii) to foster collaboration
among app users from the same work group, through debate of ideas before answering questions,
instead of making use of only competitive approaches; (iii) to embed learning in a real context, a park,
which is rich in biology, history, and mathematics learning opportunities, thus providing authentic
and situated learning experiences; (iv) to offer multiple modes of representation, namely through AR
contents with video, audio, text, and 3D models; (v) to allow users to progress at their own pace,
not establishing limited time to answering the challenges posed through the app; and (vi) to provide
feedback with a scientific explanation after answering the proposed challenges.
Besides the social constructivism approach, game features, such as the treasure hunt format with a
friendly mascot giving hints and feedback, accumulating points by correctly answering the challenges,
and the leaderboard to show the best performance, can promote motivation by making boring
content more enjoyable [22,23]. All these features were integrated in the EduPARK teaching approach.
Moreover, allying these game features with the use of mobile devices (still widely forbidden in formal
education contexts) and AR technologies is another factor increasing motivation and engagement with
learning [3].
As mentioned before, the real outdoor context where learning is promoted by the app is a park.
These areas have high ecological and environmental value, so they should be preserved. The use of
this educational mobile AR game in the selected park promotes positive attitudes towards nature
conservation and sustainability in the community [22], thus adding to the app’s educational value.
For educational relevance of the app and game approach, it was important to carefully analyse
the National Curriculum to identify multidisciplinary issues (e.g., integrating Biology and History) to
integrate in the educational guides (games), so that students may correlate the experiences promoted
by the app with the aimed curriculum learning. The fact that the educational guides were designed to
be explored in the park provides an example of a truly authentic context for situated learning, where
the location is essential for learning [23].
The EduPARK app can be installed from the project site (edupark.web.ua.pt/app). It requires the
update of quizzes after downloading; dismissing the use of mobile web in the park. The app was
developed for Android using Unity 5 and Vuforia SDK, integrating AR and quiz games, in the logic

5
Educ. Sci. 2020, 10, x FOR PEER REVIEW 4 of 20

The EduPARK app can be installed from the project site (edupark.web.ua.pt/app). It requires
the update of quizzes after downloading; dismissing the use of mobile web in the park. The app was
Educ. Sci. 2020, 10, 287
developed for Android using Unity 5 and Vuforia SDK, integrating AR and quiz games, in the logic
of a treasure hunt [24]. Therefore, it was designed to provide a learning experience that requires the
of a treasure of
exploration hunt [24]. Therefore,
a Green it was D.
Park, the Infante designed
Pedro Parkto provide
(Aveiro, a learning
Portugal). experience that requires the
exploration of a Green Park, the Infante D. Pedro Park (Aveiro, Portugal).
The app is intuitive and can be used autonomously, either individually or in a group, at any time
usingThetheapp
game is intuitive
mode orand can be
explore usedmode.
freely autonomously,
The last one either individually
allows accessingorAR in acontent,
group, at any time
without the
using the game
requirement ofmode or explore
following freely mode. path.
a predetermined The last one game
In the allowsmode,
accessing ARare
users content, without
welcomed by the
the
requirement
EduPARK mascot, of following a predetermined
a female monkey who path. In the the
explains game mode,
rules users
to the are welcomed
players. The game by the EduPARK
objective is to
mascot, a female
gather points bymonkey
correctly who explains quiz
answering the rules to the players.
questions. The user, Theor game
groupobjective
of users,iswho
to gather points
gets the by
higher
correctly answering
score is the winner.quiz questions.
So, the game isThe user,
more or group when
interesting of users, who gets
several the play
groups higherat score is thetime
the same winner.
in a
So, the game
friendly is more interesting
competition when several groups play at the same time in a friendly competition
climate [12,25].
climateFor[12,25].
motivational and engagement purposes, the app allows the creation of several profiles that
Forthe
record motivational
progress of andtheengagement purposes,
explored games. For the
eachapp allowsit the
profile, creation to
is possible of know
severalall profiles that
completed
record
games,the progress ofofthe
percentage explored
correct games.
answers, For each
number of profile,
visited itmarkers,
is possible andto number
know allof completed games,
found treasures
percentage
[12]. of correct answers, number of visited markers, and number of found treasures [12].
For
Forthethefirst-time
first-timeexperience
experienceof ofplaying,
playing,the theuser
userisisguided
guidedby byaatutorial
tutorialthat
thatintroduces
introducesthe theapp’s
app’s
features
features(Figure
(Figure1). 1).The
Thegame
gameincludes:
includes: (i)
(i) Instructions
Instructions for for users
users to to find
findlocations
locationsin inthe
thepark,
park,in inorder
order
to
tofollow
followaapredetermined
predeterminedpath; path;(ii)
(ii)questions
questionswhosewhoseanswer
answerrequires
requiresobserving
observingthe thesurroundings
surroundingsor or
analysing
analysing multimedia
multimedia resources,
resources, sometimes
sometimes in in AR
AR format;
format; (iii)
(iii)feedback
feedback providing
providingan anexplanation
explanation
about
aboutthe thecorrect
correct answer;
answer; (iv)(iv)
the number of accumulated
the number of accumulated points;points;
(v) challenges to find virtual
(v) challenges to findtreasures
virtual
(caches), enclosing extra credits—bananas—the number of assigned
treasures (caches), enclosing extra credits—bananas—the number of assigned bananas decreases bananas decreases with the time
needed
with thetotime
find needed
the treasure;
to find (vi) the
the accumulated
treasure; (vi) the bananas can be bananas
accumulated exchanged canfor
beclues to helpfor
exchanged players
clues
answer
to help later
playersquestions
answeror are questions
later convertedor into
arepoints
convertedat theinto
endpoints
of the at game [12].of the game [12].
the end

Figure 1. Several displays of the EduPARK app showing some of its main features.
Figure 1. Several displays of the EduPARK app showing some of its main features.
The logic of searching for virtual treasures/caches is based on Geocaching principles, integrated
Thegame
into the logicto
ofincrease
searching forinvolvement
user virtual treasures/caches is based
and motivation. AnonARGeocaching principles,
icon is always availableintegrated
so that,
into the game to increase user involvement and motivation. An AR icon is always
at any time, users can point the camera on their mobile device to specific images, called AR markers, available so that,
at access
to any time,
the users can point
AR content. Thethecompass
camera on cantheir mobile
be used device
at any to during
time specificthe
images,
game,called AR markers,
to support users’
orientation through the park. It is also possible to access an interactive map to view the four game users’
to access the AR content. The compass can be used at any time during the game, to support areas
orientation
and various through
locationsthe park.during
to visit It is also
the possible
game. Theseto access an interactive
locations are usually map to viewwith
associated the four game
botanical
areas and
species various
of the locations
park, marked to visit
with during
physical the game.
plaques with AR These locations
markers givingare usually
access to ARassociated with
content about
botanical species of the park, marked with physical plaques with AR markers giving
the species. Other locations for players to visit are historical interest points with “natural” AR markers, access to AR
content about the species. Other locations for players to visit are historical
for example pre-existing tiles or information boards to access additional AR information [12,24]. interest points with
“natural” AR markers,
For different for example
target publics, therepre-existing
are different tiles or information
educational guides thatboards to access
encourage additional
users to followAR
a
information [12,24].
path to promote learning relevant for the curriculum of Sciences, Mathematics, and History, among other
subjects. The project created three guides aimed at students and teachers of different school levels:
(i) 1st Cycle of Basic Education (CBE) comprising school years from 1 to 4 (attended by children with

6
 For different target publics, there are different educational guides that encourage users to follow
a path to promote learning relevant for the curriculum of Sciences, Mathematics, and History, among
other subjects. The project created three guides aimed at students and teachers of different school
Educ. Sci. 2020, 10, 287
levels: i) 1st Cycle of Basic Education (CBE) comprising school years from 1 to 4 (attended by children
with 6 to 9 years-old), ii) 2nd and 3rd CBE, comprising school years from 5 to 9 (attended by children
6 to
with 109toyears-old), (ii) 2nd
14 years-old), andandiii) 3rd CBE, comprising
Secondary, from 10 to school yearsyears
12 school from(attended
5 to 9 (attended
by studentsby children
15 to 17
withold),
years 10 toand 14 years-old), and (iii) A
higher education. Secondary,
fourth guide from was
10 toalso
12 school
created years
for (attended
the publicby whostudents
visits 15 thetopark,
17
years old),
including and higher
questions education.
of general A fourth
culture, guide was
for example, also created
about for the
the typical public style
aesthetic who visits
of Aveiro the park,
[25].
including
Each guide questions
consistsof general
of fourculture,
stages forthatexample,
correspondabouttothe typical aesthetic
a specific area of style of Aveiro
the park, [25]. the
in which
user is Each guide consists
challenged to searchof fourforstages thatofcorrespond
points interest, to a specific
collect area of theanswer
information, park, in which
multiple the choice
user
is challenged
questions, to search
receiving for points
immediate of interest,
feedback collector
whether information,
not they answeranswer the multiple
questionchoice questions,
correctly. The
receiving immediate feedback whether or not they answer the question
questions may have associated multimedia resources, such as audio, photography, illustration, video, correctly. The questions may
or have associated
3D objects in AR multimedia
[12]. resources, such as audio, photography, illustration, video, or 3D objects in
ARAt[12].
the end of each stage, the user has five minutes to find virtual treasures in a “treasure hunt”
inspired Atby the end of eachprinciples.
Geocaching stage, the user Afterhasthisfive minutes
period, to find
if the virtual
treasure is treasures
not found, in the
a “treasure
game proceedshunt”
inspired by Geocaching principles. After this period, if the treasure is not found, the game proceeds
normally initiating a new stage. Points are accumulated throughout the game, whenever the users
normally initiating a new stage. Points are accumulated throughout the game, whenever the users
answer the questions correctly and whenever they find an AR marker. At the end of the game, the
answer the questions correctly and whenever they find an AR marker. At the end of the game, the users
users have access to the number of correct answers and the total number of accumulated points [24].
have access to the number of correct answers and the total number of accumulated points [24].
AR combines the real world with a virtual world, which can be three-dimensional and
AR combines the real world with a virtual world, which can be three-dimensional and interactive
interactive in real time. Usually, the camera of a mobile device is used to detect a previously defined
in real time. Usually, the camera of a mobile device is used to detect a previously defined marker
marker (image) that activates associated virtual content. The physical AR plaques, which support the
(image) that activates associated virtual content. The physical AR plaques, which support the AR
ARmarkers,
markers, have have a digital
a digital laser engraving
laser engraving on laminatedon laminated
vinyl withvinyl with protection
ultraviolet ultravioletonprotection
galvanizedon
galvanized
plaques. Theseplaques.are These
fixed to are
thefixed
groundto the
by ground
means ofby meanspiles,
external of external
in order piles, in order permanent
to constitute to constitute
permanent elements
elements in the park. in the park.
TheThemarkers
markers about
aboutthe thepark’s
park’splants
plantspresent
present aa menu (Figure2)
menu (Figure 2)that
thatincludes
includesinformation
information about
about
thethe
plant, the leaf, the flower, the fruit, its origin, ecology, and curiosities. The
plant, the leaf, the flower, the fruit, its origin, ecology, and curiosities. The users can choose what users can choose what
they
they want
want toto haveaccess
have accesstotosystematized
systematizedinformation
information with aa photo photoor orimage.
image.The Thewords
wordsspecific
specific to to
Botany, which may not be known to ordinary people, are underlined in blue, and their meaning can be
Botany, which may not be known to ordinary people, are underlined in blue, and their meaning can
be known
knownby byclicking
clickingononthem,them, like
likea glossary.
a glossary. TheThe
useruser
can can
digitally interact
digitally with with
interact the 3D themodels
3D modelsof the of
plant leaves, being able to rotate them and observe the top and bottom
the plant leaves, being able to rotate them and observe the top and bottom page of the leaf, which page of the leaf, which is often is
useful
often to identify
useful species.
to identify This feature
species. is particularly
This feature advantageous
is particularly when thewhen
advantageous specimen is leafless, is
the specimen
in the case of deciduous species. All this information, associated
leafless, in the case of deciduous species. All this information, associated with AR markers, with AR markers, is available in is
Portuguese
available and in English,
in Portuguese andsointhat foreignsotourists
English, can alsotourists
that foreign use the EduPARK
can also use appthe
to know
EduPARK more aboutapp to
the Park’s species [12,24]. For greater user convenience, it is possible
know more about the Park’s species [12,24]. For greater user convenience, it is possible to freeze to freeze the AR information thatthe
appears on the mobile device’s screen. This way, the user can turn
AR information that appears on the mobile device’s screen. This way, the user can turn away from away from the marker without
thelosing
marker thewithout
associated information.
losing the associated information.

Figure 2. Augmented Reality (AR) markers triggering an example of an interactive menu with AR
Figure 2. Augmented Reality (AR) markers triggering an example of an interactive menu with AR
plant information.
plant information.

7
Educ. Sci. 2020, 10, 287
Educ. Sci. 2020, 10, x FOR PEER REVIEW 6 of 20
Educ.
Educ.Sci.
Sci.2020,
2020,10,
10,xxFOR
FORPEER
PEER REVIEW
REVIEW 66 of
of 20
20
Additionally, the app recognizes “natural” markers, pre-existing in the Park, such as tiles and
Additionally, the app recognizes “natural” markers, pre-existing in the Park, such as tiles and
information signs for
Additionally,
Additionally, themonuments.
the app
app recognizes
recognizes These were frequently
“natural”
“natural” markers, used to provide
markers, pre-existing
pre-existing in 3DPark,
in the
the contentssuchproduced
such as
as tiles by
tiles and
information signs for monuments. These were frequently used to provide 3DPark,
contents produced and
by
the EduPARKsigns
information
information project,
signs for which aim toThese
monuments.
for monuments. provide
These were
were information
frequentlythat
frequently used
usedcomplements
to provide
to provide 3D 3Dthecontents
reality observable
contents produced
produced by in
by
the EduPARK project, which aim to provide information that complements the reality observable in
those
the locations.
EduPARK
the EduPARK Next,
project, a brief
which description
aim to of
provide the produced
information contents
that is presented.
complements the
the reality observable in
reality observable
those locations.project,
Next, awhich aim to provide
brief description of the information that complements
produced contents is presented. in
those
those locations.
locations. Next,
Next, aa brief
brief description
description of of the
the produced
produced contents
contents is is presented.
presented.
•• Moliceiro
Moliceiro(typical
(typicalboatboatofofthe
thecity)
city)(Figure
(Figure3): 3): On
On the
the ancient
ancient moliceiro’s
moliceiro’s tile,
tile, the
the ARAR functionality
functionality
•• overlays
Moliceiro
Moliceiro a (typical
real
(typical boat
photograph
boat of
of the
of
the city)
acity) (Figure
current
(Figure 3):
moliceiro
3): On
On the
to ancient
highlight
the ancient moliceiro’s
the change
moliceiro’s oftile,
use
tile, the AR
from
the functionality
transportation
AR functionality
overlays a real photograph of a current moliceiro to highlight the change of use from
overlays
of aquatic
overlays aaflora
real
realto photograph
fertilize
photograph of
farmlands
of aa current
to tourist
current moliceiro to
transportation.
moliceiro to highlight
highlight the
the change
change of
of use
use from
from
transportation of aquatic flora to fertilize farmlands to tourist transportation.
transportation of aquatic flora to fertilize farmlands to tourist
transportation of aquatic flora to fertilize farmlands to tourist transportation. transportation.

Figure 3. The typical Aveiro boat AR image.

Figure
Figure 3.
Figure 3. The
3. The typical
The typical Aveiro
typical Aveiro boat
Aveiroboat AR
boatAR image.
ARimage.
image.
• Santo António (saint adored by the Portuguese population) (Figure 4): On this tile, the AR
••
• Santo
SantoAntónio
Santo António
António (saint adored
(saint
(saint by the
adored
adored byPortuguese
the population)
the Portuguese (Figure 4):
population) On this4):
(Figure tile,
On thethis
AR tile,
functionality
tile, the
the AR
functionality overlays severalbybuttons Portuguese
on parts population)
with religious(Figure 4): On
significance, this
which AR
become
overlays several
functionality
functionality buttons several
overlays
overlays on partsbuttons
several with religious
buttons on
on significance,
parts
parts with
with which significance,
religious
religious become interactive
significance, which
which and give
become
become
interactive and give access to a brief description.
access to a brief
interactive
interactive anddescription.
and give
give access
access to
to aa brief
brief description.
description.

Figure 4.
Figure 4. The
The saint
saint tile
tile AR
AR information
information and
and interactivity.
interactivity.
Figure
Figure 4.
4. The
The saint
saint tile
tile AR
AR information
information and
and interactivity.
interactivity.
•• Symmetric
Symmetrictile
tile(Figure 5): 5):
(Figure On
5): this
On tile,
thisthe ARthe
tile, functionality
the overlays an
AR functionality animated
overlays anthree-dimensional
animated three-
•• Symmetric
Symmetric tile
tile (Figure
(Figure 5): On
On this
this tile,
tile, the AR
AR functionality
functionality overlays
overlays an
an animated
animated three-
three-
tile that demonstrates
dimensional
dimensional tile that its axes of symmetry,
tile that demonstrates
demonstrates
its axes of
its axes through
of folding.
symmetry,
symmetry,
through folding.
through folding.
dimensional tile that demonstrates its axes of symmetry, through folding.

Figure 5. The symmetric tile AR model.

Figure 5. The symmetric tile AR model.
Figure
Figure 5.
5. The
The symmetric
symmetric tile
tile AR
AR model.
model.
• Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR functionality
• Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR
•• overlays
Torreão a(ancient
Torreão three-dimensional
(ancient water reconstruction
wateradeposit)
deposit) (Figure of
(Figure 6):the
6): On
Onbuilding,
the animated
the building
building by its decomposition
identification
identification plaque, into
plaque, the
the the
AR
functionality overlays three-dimensional reconstruction of the building, animated by AR
its
three main
functionalitygeometric
functionality overlays
overlays solids that compose it.
aa three-dimensional
three-dimensional reconstruction of of the building, animated by its
decomposition into the three main geometric reconstruction
solids that compose the
it. building, animated by its
decomposition
decomposition into
into the
the three
three main
main geometric
geometric solids
solids that
that compose
compose it.
it.

8
Educ.
Educ.Sci. 10,287
2020,10,
Sci.2020, x FOR PEER REVIEW 7 of 20
Educ. Sci.Sci.
Educ. 2020, 10,10,
2020, x FOR PEER
x FOR REVIEW
PEER REVIEW 7 of 2020
7 of

Figure6.6.The
Figure Thewater
waterdeposit
depositAR
ARmodel
modeland
andits
itsdecomposition
decompositioninto
intogeometric
geometricsolids.
solids.
Figure 6. 6.
Figure The water
The deposit
water AR
deposit model
AR and
model itsits
and decomposition into
decomposition geometric
into solids.
geometric solids.
•• Ducks’
Ducks’House
House (wooden
(wooden construction for thefor
construction ducks
the inducks
the parks lake)parks
in the (Figure 7): On
lake) an identification
(Figure 7): On an
• • Ducks’
Ducks’ House
House (wooden
(wooden construction
construction forforthe
theducks
ducks ininthe parks
the parkslake)
lake)(Figure
(Figure 7):7):On
Onanan
plaque, the ARplaque,
identification functionality
the ARoverlays a three-dimensional
functionality reconstruction reconstruction
overlays a three-dimensional of the ducks’ house.
of the
identification
identificationplaque,
plaque, the
theARARfunctionality
functionality overlays
overlaysa athree-dimensional
three-dimensionalreconstruction
reconstructionofofthethe
The object
ducks’ allows
house. Theitsobject
rotation on all
allows itsaxes.
rotation on all axes.
ducks’
ducks’house.
house. The
The object allows
object allowsitsits
rotation
rotationononallall
axes.
axes.

Figure7.7.The
Figure Theducks’
ducks’house
houseAR
ARmodel,
model,ininlateral
lateraland
andtop
topperspectives.
perspectives.
Figure 7. 7.
Figure The ducks’
The house
ducks’ AR
house model,
AR in in
model, lateral and
lateral top
and perspectives.
top perspectives.
•• Monument
Monumentto toDr
Dr Jaime
Jaime de Magalhães Lima
de Magalhães Lima(local
(localpersonality)
personality)(Figure
(Figure8):8):OnOn thethe monument,
monument, the
• • Monument
Monument totoDrDrJaime
JaimededeMagalhães
Magalhães Lima
Lima(local personality)
(local personality) (Figure
(Figure 8):8):
On Onthe monument,
the monument, the
the
the
ARAR functionality
functionality overlays
overlays the the three-dimensional
three-dimensional reconstruction,
reconstruction, whichwhich has interactivity
has interactivity to
to allow
AR functionality overlays the three-dimensional reconstruction, which has interactivity
AR functionality overlays the three-dimensional reconstruction, which has interactivity to allow to allow
allow the exploration
the exploration ofgeometric
of the the geometric solids
solids thatthat compose
compose it. This
it. This model
model is triggered
is triggered by by pointing
pointing the
the exploration
the exploration ofof
the geometric
the geometric solids that
solids compose
that compose it.it.
This
Thismodel
model is is
triggered
triggered bybypointing
pointing the
the
the mobile
mobile device
device cameradirectly
camera directlyatatthe
themonument,
monument,not not requiring
requiring a physical
physical AR ARplaque.
plaque.This
Thiswas
was
mobile
mobiledevice
device camera
camera directly
directlyatat
the monument,
the monument, not requiring
not requiring a physical
a physical AR ARplaque.
plaque. This
Thiswas
was
the
themarker
markerthat
that constituted greatestchallenge
constituted the greatest challengefrom
fromthe thetechnological
technological point
point of of view,
view, since
since the
the marker
the markerthat constituted
that constituted the greatest
the greatestchallenge
challenge from
from the technological
the technological point
pointofof
view,
view,since
sincethe
the
the overlap
overlap of of
thethe
3D3Dmodel
modelononthe thephysical
physicalmonument
monument requires
requires aa high precision
precision inin terms
termsofof
overlap
overlapofofthe
the3D3Dmodel
modelononthe thephysical
physicalmonument
monumentrequires
requiresa ahighhighprecision
precisionininterms
termsofof
dimension
dimensionand andpositioning.
positioning.
dimension
dimension andandpositioning.
positioning.

Figure8.8.The
Figure Themonument
monumentAR
ARmodel
modeland
andits
itsinteractivity.
interactivity.
Figure 8. 8.
Figure The monument
The AR
monument model
AR and
model itsits
and interactivity.
interactivity.
• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in the
• • EduPARK
EduPARK Mascot
Mascot (the inspiration
(the inspiration forfor
this mascot
this was
mascot the
was fact
the that
fact a female
that monkey
a female monkey lived
lived inin
the
the
park for several years, so it is commonly known as the Monkey Park) (Figure 9): On an
park
parkforforseveral
severalyears,
years,sosoit itis iscommonly
commonlyknownknownasasthetheMonkey
MonkeyPark)
Park)(Figure
(Figure9):9):OnOnanan
9
Educ. Sci. 2020, 10, 287

• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in
Educ. Sci. 2020, 10, x FOR PEER REVIEW
the park for several years, so it is commonly known as the Monkey Park) (Figure 9): 8On of 20
an
identification plaque, the AR functionality overlays an animated three-dimensional model of
identification plaque, the AR functionality overlays an animated three-dimensional model of the
the mascot and her living cage, allowing the visualization of the monkey that remains an iconic
mascot and her living cage, allowing the visualization of the monkey that remains an iconic
symbol of the park.
symbol of the park.

Figure
Figure9.9.The
TheEduPARK
EduPARKmascot
mascotAR
ARanimated
animatedmodel.
model.

In summary, the EduPARK app innovation relies on the articulation of the following
In summary, the EduPARK app innovation relies on the articulation of the following components:
components: (i) The use of new and easy to explore technologies, mobile supported AR; (ii)
(i) The use of new and easy to explore technologies, mobile supported AR; (ii) Geocaching-based
Geocaching-based learning in outdoor environments; and (iii) cross subjects educational materials
learning in outdoor environments; and (iii) cross subjects educational materials (guides specific for
(guides specific for different educational levels) [25].
different educational levels) [25].
3. Materials and Methods
3. Materials and Methods
Mixed method research approaches are known for combining qualitative and quantitative
Mixed method research approaches are known for combining qualitative and quantitative elements
elements to achieve a level of understanding and corroboration in breadth and depth, not possible
to achieve a level of understanding and corroboration in breadth and depth, not possible through
through either approach on its own [26]. So, a mixed method approach is used to analyse the effect
either approach on its own [26]. So, a mixed method approach is used to analyse the effect of using the
of using the EduPARK app into the following dimensions of educational value: (a) Learning value;
EduPARK app into the following dimensions of educational value: (a) Learning value; (b) intrinsic
(b) intrinsic motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f)
motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f) conservation and
conservation and sustainability habits by triangulation of teachers and students’ opinions, after
sustainability habits by triangulation of teachers and students’ opinions, after playing the EduPARK
playing the EduPARK game, with game results.
game, with game results.
This section comprises four sections: (i) A context introduction regarding the EduPARK activity
This section comprises four sections: (i) A context introduction regarding the EduPARK activity and
and study participants; (ii) data collection methods; (iii) data analysis strategies; and finally, (iv)
study participants; (ii) data collection methods; (iii) data analysis strategies; and finally, (iv) sample description.
sample description.
3.1. The EduPARK Activity and Study Participants
3.1. The EduPARK Activity and Study Participants
To collect data, the research team organizes activities involving schools and other educational entities
(suchToas collect data,study
after school the research team
centres) for organizes
a period activities
of about involving
one year, schools
from March 2018and other
to April educational
2019, in Infante
entities (such as after school study centres) for a period of about one year, from March
D. Pedro Park (Aveiro, Portugal). The EduPARK activity, organised under the project, comprises: 2018 to April
2019, in Infante D. Pedro Park (Aveiro, Portugal). The EduPARK activity, organised under the project,
(a) A small introduction about the activity program and some instructions on how to use the
comprises:
EduPARK app to play;
(a) A small introduction about the activity program and some instructions on how to use the
(b) the EduPARK app does not require internet connection, and the game is usually played by teams
EduPARK app to play;
of three or four students accompanied by an adult (mostly teachers, but also other school staff or
(b) the EduPARK app does not require internet connection, and the game is usually played by
EduPARK team members);
teams of three or four students accompanied by an adult (mostly teachers, but also other school staff
(c) the leaderboard construction and announcement to participants, with small prizes distribution
or EduPARK team members);
to the three groups with higher scores, such as medals of 1st, 2nd, and 3rd places or project’s
(c) the leaderboard construction and announcement to participants, with small prizes
merchandising items (for example, pen, bracelet, mobile phone holder, yo-yo, etc., depending on
distribution to the three groups with higher scores, such as medals of 1st, 2nd, and 3rd places or
the age of the players).
project’s merchandising items (for example, pen, bracelet, mobile phone holder, yo-yo, etc.,
depending
For theongame
the age of thethe
playing, players).
EduPARK project provided mobile devices (smartphones or tablets) for
participants to useplaying,
For the game during the
theactivity,
EduPARK whenever
projectneeded.
provided mobile devices (smartphones or tablets)
for participants to use during the activity, whenever needed.
A total of 44 activities for 1007 students, from the 1st Cycle of Basic Education (CBE) to
Secondary Teaching, and 122 accompanying adults, 10 usually teachers, were organised both in formal

and non-formal educational contexts. After student distribution by groups and resources checking,
the groups started playing the educational guide for their academic level, in a lagged departure
Educ. Sci. 2020, 10, 287

A total of 44 activities for 1007 students, from the 1st Cycle of Basic Education (CBE) to Secondary
Teaching, and 122 accompanying adults, usually teachers, were organised both in formal and non-formal
educational contexts. After student distribution by groups and resources checking, the groups started
playing the educational guide for their academic level, in a lagged departure organization. Five
activities (out of 44) were mixed in what concerns students’ school level, and the games played in
these sessions were either the one for the 1st CBE or the one for the 2nd/3rd CBE. Table 1 shows, per
school level, the number of activities (total of 42), students (total of 1007), and teachers and other
adults (total of 122) who accompanied the students in the activities between March 2018 and April
2019. Considering both students and teachers, the total of participants was 1129.

Table 1. The relation of number of activities, participant students, and accompanying teachers in the
EduPARK activities, per school level. CBE: Cycle of Basic Education.

1st CBE 2nd/3rd CBE Secondary Teaching Total

Number of activities 23 (5 mixed) 21 (5 mixed) 3 42
Number of students 476 396 74 1007
Number of teachers 91 24 7 122

3.2. Data Collection

Data collection included a paper questionnaire and automatic app logging mechanisms, supporting
triangulation of users’ subjective points of view with their performance during the game (for example,
number of right answers), which are objective results. At the end of the game playing activity, each
participant (both students and teachers or other accompanying adults) was invited to complete an
evaluation questionnaire. The questionnaire was similar for all the school levels and types of users
(students and teachers); however, it included adaptations to the age and educational context of the
respondents. Teachers who accompanied classes in the activity more than once filled in only one
questionnaire, in the first activity, so the number of teacher questionnaires is lower than the number
of effective participant teachers. This option supports the respondents in avoiding biases caused by
increasing familiarity, as, according to Sauro [27], having prior experience with the system increases
scores. For example, when analysing websites usability with Brooke’s System Usability Scale [28],
Sauro found that users who had previously experienced the website, tended to generate higher SUS
scores (11% higher) than first-time users.
The set of questionnaires analysed in this study were used in previous studies of the EduPARK
project [9,29]. The tool comprises four sections, with mostly closed-ended questions in a Likert scale,
where 1 corresponds to ‘strongly disagree’ and 5 to ‘strongly agree’, and a minority of open-ended
questions that complement the quantitative data and provide a level of insight not captured by
the closed questions. One section collected basic demographic data, such as age and gender, their
familiarity with mobile devices, and their opinion on mobile learning advantages and disadvantages.
Another section is about the interest regarding the activity of playing the EduPARK game in the park;
although this is not the focus of this work. Another section refers to the Educational Value Scale (EVS)
(presented in [9] and reliability demonstrated in [21]), and the last one consists of the System Usability
Scale (SUS) [18,28,30] with the minor adjustment of replacing the word “system” with “app”, which
does not appear to have an effect on the resulting scores [19]. The same authors highlighted SUS high
reliability and pointed it as the most sensitive poststudy questionnaire, designed to assess perceptions
of usability.
The questionnaire for students of the 1st CBE did not have questions on mobile learning advantages
and disadvantages nor an open answer question to justify their opinion regarding the interest of the
activity. Teachers’ questionnaires had additional questions regarding the interest of the activity for
their own practice and for their colleagues.

11
Educ. Sci. 2020, 10, 287

Only fully answered questionnaires were included in the study. All teachers’ questionnaires
were considered, however, some students’ questionnaires were discarded, as some questions were
not answered.
The app includes automatic mechanisms of game log generation. These mechanisms allowed the
collection of anonymous information from finished games during the observation period (between
March 2018 and April 2019). The information includes: (a) Final score (points gathered through
correct answers and points gained through collecting bananas); (b) game time; (c) number of questions
answered correctly and incorrectly; and (d) number of hunted Geocaching treasures.
All data collection, processing, and storage procedures respected research ethics principles. Data
were collected anonymously and did not include any personal information or set of information
allowing the identification of specific participants.

3.3. Data Analysis

The focus of this study is the educational value of mobile AR games, analysed through an illustrative
example, the EduPARK app. So, the questionnaire data analysis focuses the section comprised by the
EVS, EduPARK activity interest, including qualitative data. Game logs were also analysed to triangulate
the users’ opinions.
Data were analysed through scores computing, descriptive statistics, and content analysis of open
response questions.
The computing of EVS scores is similar to the SUS computing process described by Brooke [18,28].
However, as described by Sauro and Lewis [19], to get a 12 items scale to range from 0 to 100, the sum
of the 12 EVS items contribution is multiplied by 2.0834 (the result of the division of 100 by 48).
As questionnaires’ open questions aimed solely to collect the users’ reactions to the EduPARK
activity, content analysis was performed to present illustrative citations of participants’ answers.
Document analysis of the app game logs was performed. A table with the overall results, regarding
game scores attained, number of correct and incorrect answers, game time, etc., is presented in the
next section.
Finally, the questionnaire and app log data were triangulated to provide a more comprehensive
knowledge of users’ opinion regarding the EduPARK app educational value. This analysis is presented
in the next section.

3.4. Sample Description

The survey allowed gathering information about the users’ profile. Table 2 presents the characterization
of each type of user in terms of number of returned questionnaires (valid and fully filled in), age, gender,
school year, Android mobile device ownership, and its use to learn or to promote learning. The response
rate was high in all types of users, more specifically, it was 85.1% for 1st CBE students, 97.0% for
2nd and 3rd CBE students, 87.8% for Secondary Teaching students, and 57.4% for accompanying
teachers. It is worthwhile to note that several teachers accompanied groups of students in more
than one activity; hence, they were counted more than once in the participants section. However,
these teachers were asked to answer the questionnaire only in their first participation, capturing their
immediate perceptions. As highlighted in the Materials and Methods section, the literature reports an
increase of SUS scores with the increasing familiarity with the system under evaluation [27].
Students’ ages vary within the expected values for their respective schooling levels in Portugal:
6–11 and mean value of 8.1 for 1st CBE, 10–15 and mean of 11.0 for 2nd and 3rd CBE, and 15–20
and mean of 16.6 for secondary teaching. Teachers’ ages vary between those that are expected for
graduated workforces, between 23 and 64 years-old, with a mean of 42.1.
The proportion of female and male students is balanced (47.4%, 53.1%, and 55.4% of females);
however, the teaching class is composed mainly by female teachers (77.1%) according to the general
scenario in Portuguese schools, where the female teachers percentage was 77.9 in 2019 [31].

12
Educ. Sci. 2020, 10, 287

Table 2. A characterization of each type of participant on the EduPARK activities, from March 2018 to
April 2019.
Type of User
General 1st CBE 2nd/3rd CBE Secondary Teaching Accompanying Teachers
Information
Valid questionnaires 405 384 65 70
Range 6–11 10–15 15–20 23–64
Age
Mean 8.1 11.0 16.6 42.1
Gender (female) 192 (47.4%) 204 (53.1%) 36 (55.4%) 54 (77.1%)
year 1: 28 (6.9%)
year 5: 294 (76.6%) year 10: 22 (33.8%) Graduation: 47 (67.1%)
School year/ year 2: 177 (43.7%)
year 8: 61 (15.9%) year 11: 28 (43.1%) Masters: 15 (21.4%)
Academic degree year 3: 127 (18.0%)
others: 29 (7.6%) year 12: 15 (23.1%) others: 8 (11.4%)
year 4: 73 (18.0%)
Owns Android 287 (70.9%) 331 (86.2%) 55 (84.6%) 65 (92.9%)
Often 108 (26.7%) 77 (20.1%) 29 (44.6%) 28 (40.0%)
Mobile
Sometimes 194 (47.9%) 261 (68.0%) 33 (50.8%) 32 (45.7%)
learning
Never 108 (25.4%) 46 (12.0%) 3 (4.6%) 10 (14.3%)

In the 1st CBE, most participating students frequented year 2 (43.7%), followed by year 3 and
year 4 students (both 18.0%), and fewer students in year 1 (6.9%), which are expectable results,
as the EduPARK app requires reading skills. In the 2nd and 3rd CBE, most students frequented year
5 (76.6%), followed by year 8 students (15.9%), with a small participation of students from other
school-years (7.6% for years 6, 7, and 9). This participation disparity can be associated with the National
Directives for Natural Sciences Curriculum, as environment-related learning content is the focus of
school-years 5 and 8. In the secondary teaching context, the participation in the EduPARK activity
involves one class of each school-year. Comparing with the other school levels, this smaller level of
participation may indicate a lower educative value of the app for secondary teaching. An alternative
explicative hypothesis can be teachers’ fears of adopting new teaching methodologies with classes
of students that will be submitted to mandatory national exams. Regarding participating teachers,
all had higher education qualifications, which is a prerequisite in the Portuguese Education System.
A considerable portion (32.8% = 21.4% + 11.4%) continued studies further (Post-graduation, Master’s
degree, and Doctorate).
The majority of students and teachers owned Android mobile devices, such as smartphones or
tablets. The lower Android device penetration rate is in the 1st CBE group (70.9%), and the higher rate
is in the teachers’ group (92.9%).
Finally, most participants mentioned using mobile devices to learn or to promote learning. Most
students and teachers claimed they used mobile devices for learning purposes either sometimes (47.9%,
68.0%, 50.8%, and 45.7%) or frequently (26.7%, 20.1%, 44.6%, and 40.0%). The smallest answer proportion
was registered in the “Never uses” option for all types of participants (25.4%, 12.0%, 4.6%, and 14.3%).
According to the results, most of these students and their teachers are already quite familiar with mobile
technologies and usually employ them for learning. The results seem to support the literature, regarding
the proliferation of mobile devices [29], especially in what concerns the young population.
When questioned about advantages of the use of mobile devices to learn, students and accompanying
teachers mostly indicated a positive perspective regarding mobile learning, and Figure 10 shows the
percentage of agreement with each advantage sentence.
All sentences related with positive aspects of using mobile devices to learn achieved a frequency of
at least 244 students of basic education from a total of 384 (63.5%), 36 Secondary students from a total of 65
(55.4%), and 46 accompanying teachers from a total of 70 (65.7%). Only 3.6% of basic education students
did not recognize any advantage in mobile learning. Among the most acknowledged advantages are
‘you can learn in a fun way’ (81.8% for Basic Education students and 88.6% for accompanying teachers),
and ‘it is easy to find the information I want’ (84.6%), for Secondary students. Moreover, 9.9% Basic
Education students added new advantages, such as: ‘Learn quickly’, ‘It’s much more interesting’, ‘It’s
not boring’, ‘We can work as a team’, and ‘We can learn on other locations besides school’. On the

13
Educ. Sci. 2020, 10, 287

other hand, accompanying teachers (5.7%) reported ‘The use of images’, ‘It is fast’, and ‘It promotes
discovery learning’.
Educ. Sci. 2020, 10, x FOR PEER REVIEW 12 of 20

Figure 10. The opinions of participant students (2nd/3rd Cycles of Basic Education and secondary
Figure 10. The opinions of participant students (2nd/3rd Cycles of Basic Education and secondary
teaching), and accompanying teachers, in percentage, about advantages in using mobile devices to learn.
teaching), and accompanying teachers, in percentage, about advantages in using mobile devices to
learn.
Regarding the difficulties of mobile learning, Figure 11 shows that 141 Basic Education students
(36.7%), 29 Secondary students (44.6%), and 16 accompanying teachers (22.9%), recognized not having
All sentences related with positive aspects of using mobile devices to learn achieved a frequency
any difficulties in the use of mobile devices to learn. The most stated difficulties are the need for an
of at least 244 students of basic education from a total of 384 (63.5%), 36 Secondary students from a
internet connection (mentioned by 62.9% of accompanying teachers and 59.9% of basic education
total of 65 (55.4%), and 46 accompanying teachers from a total of 70 (65.7%). Only 3.6% of basic
students), and increased battery consumption (mentioned 55.4% of secondary students). The EduPARK
education students did not recognize any advantage in mobile learning. Among the most
project approach contributes to reduce these constrains, as: i) The game supporting app was conceived
acknowledged advantages are ‘you can learn in a fun way’ (81.8% for Basic Education students and
for offline use, not requiring internet connection, so this is not an issue; ii) in activities promoted by the
88.6% for accompanying teachers), and ‘it is easy to find the information I want’ (84.6%), for
EduPARK team, the project provides full charged mobile devices, thus, not interfering with the mobile
Secondary students. Moreover, 9.9% Basic Education students added new advantages, such as:
devices’ battery
Educ. Sci. 2020, 10, x of
FORparticipants.
PEER REVIEW 13 of 20
‘Learn quickly’, ‘It’s much more interesting’, ‘It’s not boring’, ‘We can work as a team’, and ‘We can
learn on other locations besides school’. On the other hand, accompanying teachers (5.7%) reported
‘The use of images’, ‘It is fast’, and ‘It promotes discovery learning’.
Regarding the difficulties of mobile learning, Figure 11 shows that 141 Basic Education students
(36.7%), 29 Secondary students (44.6%), and 16 accompanying teachers (22.9%), recognized not
having any difficulties in the use of mobile devices to learn. The most stated difficulties are the need
for an internet connection (mentioned by 62.9% of accompanying teachers and 59.9% of basic
education students), and increased battery consumption (mentioned 55.4% of secondary students).
The EduPARK project approach contributes to reduce these constrains, as: i) The game supporting
app was conceived for offline use, not requiring internet connection, so this is not an issue; ii) in
activities promoted by the EduPARK team, the project provides full charged mobile devices, thus,
not interfering with the mobile devices’ battery of participants.
Finally, 5.5% Basic Education students added new difficulties, such as ‘This is only for androids’,
‘It is not possible to explore the game in the classroom’, ‘running out of battery’, and ‘preference to
study in books’. The first mentioned difficulty will be considered in further similar apps, intending
to produce games for both Android and iOS operating systems.

Figure
Figure 11. The opinions
11. The opinions of
of participant students (from
participant students (from 2nd
2nd and
and 3rd
3rd Cycles
Cycles of
of Basic
Basic Education
Education and
and
Secondary
Secondary Teaching), and accompanying teachers, in percentage, about difficulties in using
Teaching), and accompanying teachers, in percentage, about difficulties in using mobile
mobile
devices
devices to
to learn.
learn.

4. Results and Discussion 14

This section reports this study’s main results and their discussion in light of the consulted
literature. The data were collected in the EduPARK activities conducted between March 2018 and
Educ. Sci. 2020, 10, 287

Finally, 5.5% Basic Education students added new difficulties, such as ‘This is only for androids’,
‘It is not possible to explore the game in the classroom’, ‘running out of battery’, and ‘preference to
study in books’. The first mentioned difficulty will be considered in further similar apps, intending to
produce games for both Android and iOS operating systems.

4. Results and Discussion

This section reports this study’s main results and their discussion in light of the consulted
literature. The data were collected in the EduPARK activities conducted between March 2018 and
April 2019. First, the authors analysed the EduPARK app EVS for the entire aggregated dataset and
for each of the considered target public: Students of 1st Cycle of Basic Education (CBE), 2nd and 3rd
CBE, and secondary teaching, and their accompanying teachers and other adults (other school staff,
parents, etc.). Data from open questions of the questionnaire about the interest of the EduPARK activity
are also analysed through content analysis. Finally, app game logs of finished games uploaded to the
EduPARK web platform were submitted to document analysis.

4.1. Users’ Perceptions: Educational Value and Interest

Figure 12 presents the participants’ general perceptions on the app educational value. The EVS
items are worded in positive and negative sentences alternatively. Overall, participants’ perceptions
are positive, as most respondents (strongly) agree with the scale positive formulated items and
(strongly) disagree with the negative formulated items. For example, 629 (68.1%) participants strongly
agree with the sentence “This app helps you/students learn more about topics we study/I teach at
school”. A similar amount, 639 (69.2%) participants, strongly disagree with the sentence “This app
does not help to realize that it is important to protect nature.” These results indicate that participants
considered that the EduPARK app comprises all the dimensions of the educational value analysed in
this study: Learning value, intrinsic motivation, engagement, authentic learning, lifelong learning,
and conservation and sustainability habits. Subsets of students and teachers from this dataset achieved
similar
Educ. Sci.results
2020, 10,in previous
x FOR studies [9,29,32].
PEER REVIEW 14 of 20

Figure 12. The

Figure12. The EduPARK
EduPARK appapp educational value according
educational value according to
to the
the project
projectparticipants
participants(students
(studentsandand
teachers):
teachers):Answers
Answerstotoeach Educational
each EducationalValue Scale
Value (EVS)
Scale item,item,
(EVS) from strongly agree to
from strongly strongly
agree disagree.
to strongly
(VL: Learning
disagree. (VL: value; IM:value;
Learning Intrinsic
IM:motivation; E: Engagement;
Intrinsic motivation; AL: Authentic
E: Engagement; learning;learning;
AL: Authentic LL: Lifelong
LL:
learning; CSH: Conservation and sustainability habits).
Lifelong learning; CSH: Conservation and sustainability habits).

EVS
EVSdata
data are
are submitted
submitted toto exploratory data analysis.
exploratory data analysis. Table
Table33presents
presentsthe
thecomputed
computedEVS EVSscores
scores
arithmetic
arithmetic means, medians, modes (measures of central tendency), standard deviation (measure ofof
means, medians, modes (measures of central tendency), standard deviation (measure
variability),
variability),as
aswell
wellas
as minimum
minimum and maximum values,
and maximum values, for
foreach
eachtype
typeof
ofuser
userand
andfor
forthe
theentire
entiredataset.
dataset.
This
This table reveals high arithmetic means and medians of EVS for all participants in this study:above
table reveals high arithmetic means and medians of EVS for all participants in this study: All All
80 for students
above in basic in
80 for students education and accompanying
basic education teachers,teachers,
and accompanying and aroundand70 for students
around 70 for in secondary
students in
secondary teaching. Additionally, the median is 85.4 for two EduPARK’s target publics and for all
users. It is worth to note that all groups of participants included several users that attributed the
15
highest value possible for EVS (100). This preliminary analysis indicates that the EduPARK game
educational value is considered high by the questionnaire respondents; however, it seems to be
considered less relevant for secondary teaching students.
Educ. Sci. 2020, 10, 287

teaching. Additionally, the median is 85.4 for two EduPARK’s target publics and for all users. It is
worth to note that all groups of participants included several users that attributed the highest value
possible for EVS (100). This preliminary analysis indicates that the EduPARK game educational value
is considered high by the questionnaire respondents; however, it seems to be considered less relevant
for secondary teaching students.

Table 3. EVS scores descriptive statistics for each type of user of the EduPARK app, in the activities
organized by the project, from March 2018 to April 2019.
Type of User

Descriptive 1st CBE 2nd/3rd CBE Secondary Teaching Accompanying Teachers All Users
Statistics
Mean 83.0 83.0 69.1 83.0 82.0
EVS Median 85.4 85.4 70.8 86.5 85.4
Mode 91.7 91.7 72.9 89.6 91.7
Standard deviation 13.9 13.6 13.4 11.9 14.0
Minimum-Maximum 16.7–100.0 20.8–100.0 39.6–100.0 54.2–100.0 16.7–100.0

Further analysis included the multiple boxplot display shown in Figure 13. The boxplots in this
figure present both summary statistics (minimum value, lower hinge, median, arithmetic mean, upper
hinge, and maximum value) and raw data. According to Theus and Urbanek [33], boxplots show
robust measures of location and spread of datasets, and they are known for providing visual aids to
compare two or more datasets. In each boxplot, the vertical lines represent ordered data (from the
minimum to the maximum values); the boxes contain the middle range data, from the 1st quartile (25th
percentile) to the 3rd quartile (75th percentile); the middle lines in the boxes indicate the median value
for each dataset; and the cross below the median indicates the arithmetic mean. Basic education datasets
present potential outliers, which are the dots located 1.5 times below the size of the box. Therefore,
the arithmetic mean of these two datasets must be treated with scepticism, as this central tendency
measure is affected by outliers. In this case, the median is a suitable central tendency measure [33].
It is possible that the presence of potential outliers in the younger students’ datasets is due to the
high level of excitement related to the timing and location where the questionnaires were applied (just
after playing the game, in the park). This was reported before 9 , and may have hindered students’
concentration during the questionnaire filling. Moreover, possible reading difficulties of young students
may also have biased the results. These hypotheses are supported by a previous study [21], where a
relatively low Cronbach’s coefficient α (0.653) was found for this dataset. According to Hair et al. [34],
this value indicates a not yet acceptable reliability (0.7), however, other authors consider the value 0.6
as the lower bound of reliability acceptance, particularly in the early stage of research [35,36], which
is the case of the EVS. To reduce the impact of reading difficulties, children were supported by the
adults who accompanied each group, to assure they understood the questions and answer options,
whenever needed.
As mentioned above, the medians of the two datasets of basic education students are 85.4, the one
of the accompanying teachers is very similar, 86.5; and the median of secondary teaching students
is 70.8, which is the lowest. Moreover, as secondary teaching students’ median lies entirely outside
the interquartile ranges of the remaining target groups, this dataset is likely to be different from the
other three.
The interquartile ranges (box lengths) are small and similar for all the datasets: 16.7 for 1st CBE,
14.6 for 2nd/3rd CBE, 19.8 for secondary teaching, and 16.7 for accompanying teachers. Hence, all the
datasets seem to concentrate near their median values.
Regarding distribution of the datasets, the box and whiskers considered together reveal the range
of each dataset. Secondary Teaching students are the participants with the most scattered data, as its
range is 60.4. Therefore, the consensus regarding the Educational Value of the EduPARK app among
Secondary Teaching students was lower than among the other types of participants.

16
Educ. Sci. 2020, 10, 287

Figure 13. EVS scores boxplots for each target public of EduPARK.

As the interquartile ranges of basic education students and accompanying teachers’ datasets
roughly overlap each other, these datasets seem similar. On the other hand, the interquartile range for
Secondary Teaching students’ dataset is located below all the other interquartile ranges, also indicating
there is a difference between them. These results support the initial analysis (presented in Table 3),
as the EVS median values for basic education students and accompanying teachers are higher than the
EVS median value for secondary teaching students. These results are supported by previous studies,
where the mean EVS value was 83.8 for 244 students in 2nd and 3rd CBE [29] and 88.2 for a cohort of
45 teachers in a workshop [32]. This indicates that the EduPARK app has a high educational value.
Moreover, in this study, teachers also assigned high EVS scores, revealing the EduPARK app is a mobile
learning resource with a high educational value for practitioners.
To understand further the case of the educational value of the EduPARK app for Secondary Teaching,
the cohort of teachers who accompanied Secondary Teaching students was analysed separately to collect
indicators of possible differences in teachers’ perspectives of the educational value of the app, according
to the school level (basic education vs secondary teaching) they accompanied. These teachers constitute a
small portion of this type of participant (7 in a total of 70), and assigned the following EVS scores: 81.3;
81.3; 89.6; 89.6; 93.8; 97.9; and 97.9. This yields a mean of 83.1 and median of 87.5, which is in line with
the overall teacher dataset. Hence, despite this small number of teachers, this result supports the idea
that, although the EduPARK app educational value for secondary teaching may not be immediately
identified by teachers, the practitioners who participated with secondary teaching students revealed a
very positive opinion regarding the app’s educational value.

4.2. Game Logs: Educational Value

App game logs of finished games uploaded to the EduPARK web platform are presented in Table 4
to triangulate with data gathered from the questionnaire for more comprehensive knowledge regarding
the EduPARK app educational value. Each log corresponds to the performance of a group of students
(usually 3 or 4) who played the game collaboratively, in teams. These values (final score, game time,
right and wrong answers, and found treasures) are generated automatically by the mobile devices and
uploaded to the project web platform, after game over. The data are anonymous, and accessed only by
the project team.

17
Educ. Sci. 2020, 10, 287

Table 4. The app game logs of finished games uploaded to the EduPARK web platform from March
2018 to April 2019.

Type of User
1st CBE 2nd/3rd CBE Secondary Teaching
General Information
Number of returned logs 125 100 28
Mean 234.4 271.1 213.0
Final score Standard deviation 42.2 44.3 39.3
Minimum-maximum 134–320 175–361 134–260
Mean 01:28 01:19 01:15
Game time Standard deviation 00:17 00:13 00:24
Minimum-maximum 00:52–02:12 00:43–01:55 00:34–01:54
Mean 21.6 24.4 18.5
Right answers Standard deviation 3.4 3.6 4.0
Minimum-maximum 11–29 17–31 11–25
Mean 5.3 6.6 11.5
Wrong answers Standard deviation 3.2 3.6 5.6
Minimum-maximum 0–16 0–14 4–23
Found treasures Mean 3.6 3.8 3.8

Table 4 presents the app game logs organized per target group. The final score is calculated by the
sum of points gathered through correct answers and points gained through collecting bananas in the
hunted treasures. The achievement of higher scores indicates higher educational value, as teams need
to observe their environment, to select information provided by the AR contents accessed through the
app, to analyse alternative solutions for the challenges, to negotiate meaning among group members,
and to provide the answer considered correct by the collective.
From the table, the final score average is higher in the 2nd/3rd CBE group of students, as well as
the minimum and maximum final score, compared to the other type of students. This is in accordance
to the EVS scores for this group, reinforcing the high educational value of the app for this school level.
Game time, presented in the format hour:minute, corresponds to the time consumed from starting
the game to its end (when the final scores are displayed in the last screen). The average game time is
decreasing with the age of students, as older students are quicker to read and to answer questions
(reaching less minimum and maximum time to finish the game), when compared to the younger ones.
The game time does not indicate better performance, because students could finish the game very
quickly, for example, in just over half an hour, without answering correctly the questions and not
properly exploring all the educational resources available in the game, for example, AR contents, 3D
models, images, or additional information. Moreover, the variance in game time can also indicate that
students progress in the activity at their own pace, providing enough time for students to interact with
each other and their prior knowledge with the new experiences, and thus, supporting learning. The fact
that all groups were able to finish the game also indicates that the educational guides supported by the
app are adequate to their respective school levels.
The average of questions answered correctly is higher in the 2nd/3rd CBE group of students
(24.4) comparing to the 1st CBE (21.6) and secondary teaching group of students (18.5). Accordingly,
the minimum and the maximum number of right answers is also higher for this group of students.
The opposite situation occurs with the wrong answers, showing that some basic education groups
answered correctly all the questions (zero wrong answers), contrasting to 4 minimum wrong answers
for secondary teaching students. Accordingly, the older students reached higher values of maximum
number of wrong answers (23), indicating that this type of student revealed more interest in finishing
the game quickly than having a good learning performance. It is worthwhile to note that the AR
contents, which are mostly provided before the proposed challenges, support the students in finding
the solutions. It is a matter of students deciding to explore them properly, in order to select and analyse
the contextualized information, to have a good performance in the game. The differences in the means

18
Educ. Sci. 2020, 10, 287

of right and wrong answers, once more, indicates that the app is more suitable for the 2nd/3rd CBE
group of students.
The averages of hunted Geocaching treasures are quite similar in the three types of students
(varying from 3.6 to 3.8), which means that almost all group of students found the four virtual
treasures available in the game. The Geocaching treasure hunt is a feature introduced in the game for
motivational purposes, so the fact that the groups attempted to find the treasures is a strong indicator
that they are engaged with the activity. It is worthwhile to note that finding the treasures is not essential
to proceed with the game, as the groups could keep playing without finding any treasure.
Finally, the above presented game logs analysis triangulated with EVS scores (presented in the
previous sub-section) reinforces the finding of the suitability of the EduPARK game as an educational
resource, particularly for 2nd/3rd CBE students. Moreover, participants’ perceptions are in consonance
with students’ performance in the game, as they achieved, overall, a good game performance.

5. Conclusions
This work addresses an identified need of research reported before [9], as the previous analysis
of the Educational Value of the EduPARK app, with a smaller sample of 1st CBE students, is now
expanded to include a higher number of respondents and other target publics.
The EduPARK game achieved an average EVS of 82.0, with higher values for the subsets of
data referent to basic education students and for teachers, who experienced this mobile AR game in
loco, that is, in a Green City Park. This applies to a sample of users who claimed: i) To have their
own Android mobile devices (the lower penetration rate was registered in the younger students,
with 70.9%); ii) to use them to learn or promote learning, at least sometimes (particularly among
secondary teaching students, with 95.4%); and iii) to have a quite positive perspective regarding
mobile learning, considering it is more advantageous than disadvantageous. The high educational
value of this mobile AR game is supported by the data collected through the app logging mechanisms,
as the groups of students achieved a good performance, overall. Educational resources that combine
this set of innovative features, as being mobile, designed for outdoor use (namely in urban parks),
with contextualized AR contents, and supporting game-based activities, may promote learning, both
at a cognitive level and at an affective one, increasing motivation for learning.
The game revealed a particularly high educational value for basic education students, as secondary
teaching students assigned it lower EVS scores. Moreover, the small number of activities with secondary
teaching students (3 out of 42) seems to support the lower adequateness of this methodology for this
school level. However, both basic education and secondary teaching teachers, who accompanied
students in the activity, revealed a very positive opinion regarding the ability of the app to promote
learning. This result may indicate that secondary teaching teachers are more reluctant in trying out
new approaches with students of a school level that comprises national exams. This issue needs further
study, as the number of teachers accompanying this school level was low (7 in a total of 70).
Secondary teachers’ reluctance in adopting this mobile AR game may be due to dominant
mentalities associating mobile devices, games, and parks to distraction, play, and leisure [12].
Nevertheless, early adopters among teachers seem quite optimist towards this approach and may
promote changes in their colleagues’ mentalities on how their students can learn. Still, more studies
involving a higher number of students and teachers of this school level are needed.
Limitations of this study are related to the young age of most participants (405 students in 1st CBE
and 384 students in 2nd and 3rd CBE). As discussed before, factors such as level of excitement and
reading difficulties may have influenced the results, for this age group. Compensation methods may
include adapted questionnaires in terms of vocabulary and supporting children in the interpretation of
the items, which were the main strategies in the EduPARK activities. In future studies, the use of face
emojis instead of the numbered Likert may be a powerful alternative for young students [19].
Another issue to consider is groups’ constitution, which may also have influenced the results,
as each student’s participation in the game is lower in bigger groups. This factor may have an impact

19
Educ. Sci. 2020, 10, 287

on how the activity is perceived by the app players. However, this variation in groups’ constitution
could not be addressed, particularly with activities with a lot of participants, as it was related with the
human resources available to accompany each group of children in each session.
Finally, as the EVS is a new data collection tool, more studies involving the rating of other educational
mobile AR games are needed, to both improve and consolidate this data collection tool, and also to
establish benchmarks and scale norms. Future research must address this issue, by implementing the EVS
with more users and by analysing the educational value of other mobile AR games.
In terms of implications for research, this paper contributes to the mobile game-based AR learning
literature, as it is an empirical study with evidence on the educational value of the integration of these
elements in teaching practices. For practitioners, this work also bears the report of an example of
excellent cross-subjects’ educational materials—the learning game—that comprises a very useful tool
for teachers and students to explore scientific knowledge by accessing appealing information on cross
subjects references (such as biological and historical) of a local Urban Green Park.

Author Contributions: Conceptualization, L.P. and M.M.M.; Data curation, L.P. and M.M.M.; Formal analysis,
M.M.M.; Investigation, L.P. and M.M.M.; Methodology, L.P. and M.M.M.; Project administration, L.P.;
Writing—original draft, L.P. and M.M.M.; Writing—review & editing, L.P. and M.M.M. All authors have
read and agreed to the published version of the manuscript.
Funding: This research was funded by FEDER-Fundo Europeu de Desenvolvimento Regional funds through the
COMPETE 2020-Operational Programme for Competitiveness and Internationalisation (POCI), and by Portuguese
funds through FC-Fundação para a Ciência e a Tecnologia, grant number POCI-01-0145-FEDER-016542, and the
work of the second author is funded by national funds (OE), through FCT–Fundação para a Ciência e a Tecnologia,
I.P., in the scope of the framework contract foreseen in the numbers 4, 5 and 6 of the article 23, of the Decree-Law
57/2016, of August 29, changed by Law 57/2017, of July 19.
Acknowledgments: The authors thank the reviewers for their insightful comments.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design of the
study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to
publish the results.

References
1. Fatih, Y.; Kumalija, J.; Sun, Y. Mobile Learning Based Gamification in a History Learning Context; International
Association for the Development of the Information Society. 2018. Available online: http://www.iadisportal.
org (accessed on 3 September 2020).
2. Aivelo, T.; Uitto, A. Digital Gaming for Evolutionary Biology Learning: The Case Study of Parasite Race, an
Augmented Reality Location-Based Game. LUMAT Int. J. Math Sci. Technol. Educ. 2016, 4, 1–26. [CrossRef]
3. Sungkur, R.K.; Panchoo, A.; Bhoyroo, N.K. Augmented Reality, the Future of Contextual Mobile Learning.
Interact. Technol. Smart Educ. 2016, 13, 123–146. [CrossRef]
4. Preka, G.; Rangoussi, M. Augmented Reality and QR Codes for Teaching Music to Preschoolers and
Kindergarteners: Educational Intervention and Evaluation. In Proceedings of the 11th International Conference
on Computer Supported Education; SCITEPRESS—Science and Technology Publications: Setubal, Portugal,
2019; Volume 1, pp. 113–123. [CrossRef]
5. Zabala-Vargas, S.A.; Ardila-Segovia, D.A.; García-Mora, L.H.; Benito-Crosetti, B.L.D. Game-Based Learning
(GBL) Applied to the Teaching of Mathematics in Higher Education. A Systematic Review of the Literature.
Form. Univ. 2020, 13, 13–26. [CrossRef]
6. Fotaris, P.; Pellas, N.; Kazanidis, I.; Smith, P. A Systematic Review of Augmented Reality Game-Based
Applications in Primary Education. In Proceedings of the 11th European Conference on Games Based Learning;
Academic Conferences and Publishing International Limited: Graz, Austria, 2017; pp. 181–190. Available
online: http://www.academic-bookshop.com/ourshop/prod_6222013-ECGBL-2017-PDF-The-11th-Europ
ean-Conference-on-GameBased-Learning.html (accessed on 16 October 2020).
7. Koutromanos, G.; Sofos, A.; Avraamidou, L. The Use of Augmented Reality Games in Education: A Review
of the Literature. EMI Educ Media Int. 2015, 52, 253–271. [CrossRef]
8. Shu, L.; Liu, M. Student Engagement in Game-Based Learning: A Literature Review from 2008 to 2018.
J. Educ. Multimed. Hypermedia 2019, 28, 193–215.

20
Educ. Sci. 2020, 10, 287

9. Pombo, L.; Marques, M.M. Learning with the Augmented Reality EduPARK Game-Like App: Its Usability and
Educational Value for Primary Education. In Intelligent Computing—Proceedings of the Computing Conference;
Arai, K., Bhatia, R., Kapoor, S., Eds.; Springer: Cham, Switzerland, 2019; pp. 113–125.
10. Lin, H.; Huang, K.L.; Lin, W. A Preliminary Study on the Game Design of Pokémon GO and Its Effect
on Parent-Child Interaction. In Cross-Cultural Design. User Experience of Products, Services, and Intelligent
Environments. HCII 2020; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2020; Volume
12192, pp. 115–127. [CrossRef]
11. Vidal, E.C.E.; Ty, J.F.; Caluya, N.R.; Rodrigo, M.M.T. MAGIS: Mobile Augmented-Reality Games for
Instructional Support. Interact. Learn. Environ. 2019, 27, 895–907. [CrossRef]
12. Pombo, L.; Marques, M.M. An App That Changes Mentalities about Mobile Learning—The EduPARK
Augmented Reality Activity. Computers 2019, 8, 37. [CrossRef]
13. Lai, E.R. Collaboration: A Literature Review Research Report. 2011. Available online: https://images.pears
onassessments.com/images/tmrs/Collaboration-Review.pdf (accessed on 15 September 2020).
14. Hill, A.M.; Smith, H.A. Research in Purpose and Value for the Study of Technology in Secondary Schools:
A Theory of Authentic Learning. Int. J. Technol. Des. Educ. 2005, 15, 19–32. [CrossRef]
15. Lave, J.; Wenger, E. Situated Learning: Legitimate Peripheral Participation; Cambridge University Press:
Cambridge, UK, 1991.
16. Burr, V. Social Constructionism; Routledge: Cambridge, UK, 2015.
17. Kamarainen, A.M.; Metcalf, S.; Grotzer, T.; Browne, A.; Mazzuca, D.; Tutwiler, M.S.; Dede, C. EcoMOBILE:
Integrating Augmented Reality and Probeware with Environmental Education Field Trips. Comput. Educ.
2013, 68, 545–556. [CrossRef]
18. Brooke, J. SUS: A Retrospective. J. Usability Stud. 2013, 8, 29–40.
19. Sauro, J.; Lewis, J.R. Standardized Usability Questionnaires; Morgan Kaufmann: Amsterdam, The Netherlands, 2012.
20. Sauro, J. MeasuringU: Measuring Usability with the System Usability Scale (SUS). Available online: http:
//measuringu.com/sus/ (accessed on 10 February 2017).
21. Marques, M.M.; Pombo, L. Internal Consistency of the Educational Value Scale for Green Outdoor
Settings—The Case of EduPARK App. EDEN Eleventh Research Workshop, 2020; in press.
22. Pombo, L. Learning with an App? It’s a Walk in the Park. Prim. Sci. 2018, 153, 12–15.
23. Laine, T. Mobile Educational Augmented Reality Games: A Systematic Literature Review and Two Case
Studies. Computers 2018, 7, 19. [CrossRef]
24. Pombo, L.; Marques, M.M.; Afonso, L.; Dias, P.; Madeira, J. Evaluation of a Mobile Augmented Reality Game
Application as an Outdoor Learning Tool. Int. J. Mob. Blended Learn. 2019, 11, 59–79. [CrossRef]
25. Pombo, L.; Marques, M.M.; Lucas, M.; Carlos, V.; Loureiro, M.J.; Guerra, C. Moving Learning into a Smart
Urban Park: Students’ Perceptions of the Augmented Reality EduPARK Mobile Game. Interact. Des. Archit.
J. 2017, 35, 117–134.
26. Schoonenboom, J.; Johnson, R.B. How to Construct a Mixed Methods Research Design. Koln. Z. Soz. Sozpsychol.
2017, 69, 107–131. [CrossRef] [PubMed]
27. Sauro, J. SUStisfied? Little-Known System Usability Scale Facts. User Exp. Mag. 2011, 10. Available online:
http://uxpamagazine.org/sustified/ (accessed on 16 October 2020).
28. Brooke, J. SUS—A Quick and Dirty Usability Scale. In Usability Evaluation in Industry; Jordan, P.W., Thomas, B.,
Weerdmeester, B.A., McClelland, I.L., Eds.; Taylor & Francis: London, UK, 1996; pp. 189–194.
29. Pombo, L.; Marques, M.M. Improving Students’ Learning with a Mobile Augmented Reality Approach—
The EduPARK Game. Interact. Technol. Smart Educ. 2019, 16, 392–406. [CrossRef]
30. Martins, A.I.; Rosa, A.F.; Queirós, A.; Silva, A.; Rocha, N.P. European Portuguese Validation of the System
Usability Scale (SUS). Procedia Comput. Sci. 2015, 67, 293–300. [CrossRef]
31. FFMS. PORDATA—Docentes Do Sexo Feminino Em % Dos Docentes Em Exercício Nos Ensinos Pré-Escolar,
Básico E Secundário: Total E Por Nível De Ensino. Available online: https://www.pordata.pt/Portugal/Doce
ntes+do+sexo+feminino+em+percentagem+dos+docentes+em+exerc{{í}}cio+nos+ensinos+pr{{é}}+esco
lar++b{{á}}sico+e+secund{{á}}rio+total+e+por+n{{í}}vel+de+ensino-782 (accessed on 15 September 2020).
32. Marques, M.M.; Pombo, L. Game-Based Mobile Learning with Augmented Reality: Are Teachers Ready to
Adopt It? Proj. Des. Lit. Cornerstones Smart Educ. 2020, 43, 207–218.
33. Theus, M.; Urbanek, S. Interactive Graphics for Data Analysis: Principles and Examples. Available
online: https://books.google.pt/books?id=xHIH1Q47FeoC&pg=PA34&dq=boxplot&hl=pt-PT&sa=X&ved

21
Educ. Sci. 2020, 10, 287

=0ahUKEwjt8YeUouTpAhWwBWMBHVpZAcwQ6AEITDAD#v=onepage&q=boxplot&f=false (accessed
on 3 June 2020).
34. Hair, J.; Black, W.C.; Babin, B.J.; Anderson, R.E.; Tatham, R.L. Multivariate Data Analysis, 7th ed.; Prentice
Hall: Upper Saddle River, NJ, USA, 2010.
35. Nunnally, J.C. Psychometric Theory; McGraw-Hill: New York, NY, USA, 1967.
36. Griethuijsen, R.A.L.F.; Eijck, M.W.; Haste, H.; Brok, P.J.; Skinner, N.C.; Mansour, N.; Gencer, A.S.; BouJaoude, S.
Global Patterns in Students’ Views of Science and Interest in Science. Res. Sci. Educ. 2015, 45, 581–603.
[CrossRef]

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional
affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).

22
 education
sciences
Article
The Impact of Teacher Training Using Mobile Augmented
Reality Games on Their Professional Development
Margarida M. Marques * and Lucia Pombo *

CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers, Department of Education
and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal
* Correspondence: [email protected] (M.M.M.); [email protected] (L.P.)

Abstract: Ongoing technology progress sustains innovative teaching approaches. Mobile devices,
augmented reality (AR), and games are a few of the new resources that teachers have at their
disposal to promote student learning. However, their effective integration into practices requires
training, so there is a need to analyze the impact of training initiatives on teacher professional
development. A case study is being conducted on the development process of mobile AR games
for Science, Technology, Engineering and Mathematics (STEM) learning by 14 Portuguese in-service
teachers in a 50 h workshop. This contribution refers to the analysis of this training’s impact on
teacher professional development through a questionnaire filled in at the beginning and end of
the workshop. This study registered a higher impact on teachers’ understanding of AR educative
use, the less-known approach, compared to mobile and game-based learning. Moreover, teachers
became more experienced with these approaches as learners, and reported having explored them
with their students during the workshop period. Teacher ability to identify benefits and barriers in



these approaches increased with the workshop, particularly the learning that could be promoted


with mobile AR games. The presented set of barriers to implementation is relevant to future teacher
Citation: Marques, M.M.; Pombo, L.
professional-development initiatives.
The Impact of Teacher Training Using
Mobile Augmented Reality Games on
Keywords: continuous teacher training; mobile learning; educative augmented reality; game-based
Their Professional Development.
learning; training impact; teacher professional development
Educ. Sci. 2021, 11, 404. https://
doi.org/10.3390/educsci11080404

Academic Editors: Greg Kessler and

James Albright 1. Introduction
Mobile technologies’ versatility and relatively low cost may be two factors concurring
Received: 31 May 2021 to their pervasiveness in modern industrialized societies. Hence, the availability of a
Accepted: 30 July 2021 resource that allows performing a variety of tasks, namely some related to information
Published: 5 August 2021 seeking and treatment, makes their integration in teaching and learning practices a logical
step to take, having given rise to the term “mobile learning”.
Publisher’s Note: MDPI stays neutral Mobile learning can be defined in different ways. Perspectives may fall into one of the
with regard to jurisdictional claims in following categories: (1) technocentric, referring to learning processes that occur through
published maps and institutional affil- the use of handheld technologies, such as mobile phones; (2) mobility of the technology,
iations.
highlighting the portability allowed by the small size of the supporting technologies;
(3) relationship to e-learning, with mobile learning being considered an extension of e-
learning; (4) augmenting formal education, as a way of extending face-to-face education;
(5) learner-centered, referring to learning when the learner is not at a fixed location, thus
Copyright: © 2021 by the authors. focusing on the learner’s mobility [1–3]. Despite the perspective adopted, the use of mobile
Licensee MDPI, Basel, Switzerland. devices for educational purposes has been a growing field of research with a history of
This article is an open access article positive empirical results [4], and their use in game-based learning approaches has also
distributed under the terms and been documented as effective [5].
conditions of the Creative Commons
Game-based learning usually refers to the use of gameplay with the purpose of
Attribution (CC BY) license (https://
promoting determined learning objectives [6]. With the motivation factor being one of
creativecommons.org/licenses/by/
the most cited arguments in the literature [7–10], other positive results include student
4.0/).

23
Educ. Sci. 2021, 11, 404

satisfaction with the learning experience [11], increased student achievement across all
schooling levels [12]; and facilitation of students’ 21st-century skill development [13]. To
achieve these positive results, some caution is required, namely incorporating learning
theories into the game design [13], taking into account learners’ personal factors, such
as learning achievement or learning styles [14], and challenging players in a way that
matches students’ abilities [15]. Additionally, the competition created by games may
increase students’ engagement in challenging learning situations and improve their overall
sense of enjoyment. When game-winning conditions require working with other players,
collaborative dynamics can also be promoted [16].
Despite being considered effective learning tools [8], in real formal education sce-
narios, accepting digital games in the classroom is one of the initial barriers to overcome,
which may be related to factors such as the availability (and reliability) of the support-
ing technology, prerequisite knowledge required of the teacher, and even financial and
licensing issues [17]. For example, a study conducted by Russo, Bragg, and Russo [18]
highlighted that although Australian mathematic primary teachers mentioned frequently
using games in their practices, digital games were virtually not among their favorite types
of games, which may be interpreted as a reluctancy to incorporate digital technologies to
support game-based learning.
Future developments in mobile and game-based learning involve evaluating and
analyzing game usage data, providing powerful tools on how to create better learning
experiences, and developing game-based learning, supported by significant data about
users’ perception and their performance while playing [19].
These educational approaches, when combined with emerging augmented reality
(AR) technologies, can enhance learning experiences, as they can enrich and contextu-
alize learning information offered to learners [20]. AR supported by mobile games can
move learning to outdoor settings, fostering authentic learning, and also personal and
collaborative learning with a lifelong learning perspective [21]. Other authors [7,22] stress
the unique affordances of AR, as an “immersive” interface that enables participants to
interact with digital information embedded within the physical environment, supporting
situated learning. Outdoor collaborative learning activities using AR become an approach
scarcely found in the educational context, although with high potential in education [7,22].
Moreover, the incorporation of AR into educational practices for effective learning, instead
of for merely beautiful scenography, requires teacher training in teaching methodologies
with AR technologies [23], as teachers are often reluctant to use or integrate them into their
science curriculum [24,25].
There is a scarcity of educational resources, such as educational mobile games, that
integrate curriculum contents. Considering this scarcity, the EduPARK project: http:
//edupark.web.ua.pt/?lang=en (accessed on 3 August 2021) developed and evaluated
an app [26] that can be used autonomously, and at any time, using the ”game” mode
or the ”explore freely” mode. It promotes authentic learning so that visitors can enjoy a
healthy walk while learning. The game includes several learning guides for different target
groups: teachers and students from basic to higher education, and also for park visitors
and the public, from a lifelong learning perspective. The tourist guide is also offered in
English. The guides integrate multidisciplinary issues under the Portuguese National
Education Curriculum and propose interdisciplinary questions articulated to educational
challenges along the park using the logic of a treasure hunt. The game enables visitors to
explore and access information about the plant species living in the city park, historical
references, different multimedia contents, and a park map, allowing interaction. The goal is
to accumulate points by answering the questions correctly, visualizing AR marker contents
that help to answer questions, and finding virtual caches/treasures (3D images) [27].
The literature has noted that teachers’ adoption of mobile technologies may be in-
fluenced by factors such as their digital literacy, ICT anxiety, teaching self-efficacy, and
perceived ease of use and usefulness [28]. Moreover, teachers need to develop the ability to
implement technology in their practices [28]. Hence, the EduPARK project has organized

24
Educ. Sci. 2021, 11, 404

several short-term workshops for teachers so they can feel confident in using them in their
practices, widening the use of mobile and game-based learning. However, the need to
involve teachers in the creation of games and of educational resources to integrate into
games encouraged the authors of this paper to propose, as trainers, a long-term accredited
course with impact on teachers’ career progression. The course was directed at Science,
Technology, Engineering and Mathematics (STEM) teachers, and allowed them time to
explore and experiment with tools for game-based learning, prompting teachers to develop
learning content, as advised by De Freitas [29]. Hence, teachers were not only users, but
also creators of educational games integrating AR contents.
The scarcity of teacher training on mobile game-based learning with AR makes it
relevant to analyze their potential for teachers’ practice changes, starting with the percep-
tions of the involved stakeholders [30]. In this paper, the authors take one step forward
by analyzing not just teachers’ perceptions, but also the impact of this workshop on the
teacher trainees’ professional development with respect to the main concepts/teaching
approaches it addresses: mobile learning, AR use in education, and game-based-learning.
This research comprises training opportunities that revolve around mobile game-based
learning with AR, which are innovative, and therefore it is important to analyze their
potential in teachers’ practice changes.
The next sections briefly present and discuss the adopted materials and methods, and
the results concerning the impact of this workshop on the teacher trainees’ professional
development around the three main concepts of this study: (i) mobile learning, (ii) AR use
in education, and (iii) game-based-learning. Finally, the Conclusions section summarizes
the main findings, some limitations, and lines of future work.

2. Materials and Methods

This research was conducted under a case study [31]. Case studies are acknowledged
in the literature as effective methodologies to investigate and understand complex issues in
real-world settings that do not aim to extrapolate probabilities through statistical general-
ization [31,32]. This research approach is adequate when researchers’ want to understand
a real-world case and assume that such an understanding is likely to involve important
contextual conditions’ [31].
The case in this study was the development process of educational resources by
14 in-service teachers during a 50 h workshop (25 h face-to-face and 25 h autonomous
work) developed in the Center Region of Portugal between October 2020 and January
2021. The workshop aimed to promote the collaborative development of open digital
educational resources that foster STEM learning based on a game approach and supported
by mobile devices.
The research question that guided the work reported in this contribution was: What
is the impact of this workshop on the teacher trainees’ professional development with
respect to: (i) mobile learning, (ii) AR use in education, and (iii) game-based-learning, in
what concerns: (a) basic knowledge, (b) teacher experience as a learner, (c) reported use in
teaching practice, and (d) opinions about benefits and barriers?
To answer the research question, an adaptative online questionnaire was applied at
the beginning of the workshop and right after its end, so the results could be compared to
analyze the workshop’s impact on the teacher trainees’ professional development.
The initial questionnaire comprised a sum of closed- and open-ended questions and
was organized into four sections:
1. Motivations and expectations, with a multiple-choice question on motivations to
attend the workshop and an open-ended question on expectations;
2. Conditions for the use of digital technologies in the teachers’ educational context,
with a few closed-ended questions on the types of digital technologies (e.g., desktop
computers, smartphones, internet connection) available for their practice, for students’
learning and school policy on mobile devices use, and also an optional open-ended
question for additional comments on this topic;

25
Educ. Sci. 2021, 11, 404

3. Workshop teaching approaches: (i) mobile learning, (ii) AR use in education, and
(iii) game-based-learning; all these topics included a mixture of open- and closed-
ended questions for data collection on teacher-trainees’ basic knowledge, previous
experience and frequency of use, perceptions about benefits and barriers, and addi-
tional comments on the topic;
4. Demographic data, such as gender, academic qualifications, years of teaching experi-
ence, subjects, schoolyears that they were teaching, and average number of students
in their classes;
The final questionnaire was quite similar to the initial one, but without the demo-
graphic data section, as the population was already characterized. Hence, it comprised
three sections:
1. Workshop assessment, with a set of multiple-choice questions on aspects such as the
methodology, level of difficulty of the proposed activities, and reported readiness to
explore the teaching approaches, as well as a few optional open-ended questions to
deepen teachers’ perspectives;
2. Differences in the conditions for the use of digital technologies in the teachers’ edu-
cational context, with a multiple-choice question, as well as an optional open-ended
question for additional comments;
3. Workshop teaching approaches, similar to the initial questionnaire.
For questionnaire content validity, both versions were analyzed by two educational re-
searchers with different experiences. One was a teacher who was undertaking her doctoral
studies. The other was an experienced researcher and methodology professor at a public
university, with a Ph.D. and postdoctoral training. After introducing changes suggested by
the educational researchers, they considered the questionnaire clear, understandable, and
suitable for the target population.
In the present study, only data regarding the research question was analyzed (Appendix A
presents Section 3 questions of the initial—Table A2—and final questionnaires—Table A2).
The analysis included descriptive statistics for answers to closed-ended questions and
content analysis for answers to open-ended questions. When relevant, initial and final
responses were compared.
For the content analysis related to teachers’ basic knowledge on the considered topics,
each answer was classified as correct, partially correct, or incorrect, in accordance with
the researchers’ own perspectives on the topics. These were in line with the workshop
activities, as the researchers were also the teacher trainers.
For the content analysis of the benefits and barriers associated with each teaching
approach, the analysis categories developed in the previous study [30] were revised. For
example, two categories (“It is easy or quick to find information” and “Supports higher
interaction”) were included in a broader category (”Supports better learning”). Moreover,
in each teacher answer, the number of different types of benefits or barriers was identified.

Study Participants
The study participants were characterized according to their: (i) Demographic data;
(ii) Motivations and expectations concerning the workshop; and (iii) Conditions for tech-
nology use in schools. These data are available in the previous study of Marques and
Pombo [30]. To comply with the General Data Protection Regulations, the questionnaire
included a closed-ended question respecting for informed consent to participate in this
study. Out of 16 teachers attending the workshop, 14 agreed and signed the informed
consent, so the following data are related only to those 14 teachers.
Concerning demographic data, in terms of gender, 12 were females and 2 were males.
Ten teachers had a high degree, mandatory by Portuguese law, one had a post-graduation
course, and three had a master’s degree. All teachers were experienced: (i) two had 11 to
20 years of experience; (ii) eight had between 21 and 30 years, and (iii) four had more than
31 years of experience.

26
Educ. Sci. 2021, 11, 404

Three teachers lectured Mathematics in the 3rd cycle of basic education (CBE, corre-
sponding to school years 7 to 9) or in secondary teaching (ST, years 10 to 12), six teachers
lectured Physics and Chemistry in the 3rd CBE or SE, one teacher lectured Nature Sciences
in the 3rd CBE, and six teachers lectured Mathematics or Nature Sciences in the 2nd CBE
(years 5 and 6). Their classes varied from 16 to 20 students (4 teachers), 21 to 25 students
(4), and 26 to 30 students (4).
With respect to motivations, in the closed-ended question regarding reasons for attend-
ing the workshop, the participant teachers selected: “Updating or acquiring knowledge”
and ”Possibility to have access to new resources” with 12 mentions each; and “Combination
of the workshop topics”, “Possibility of changing teaching practice” and “Professional
valorization” with 10 mentions each. Concerning their expectations regarding the work-
shop, teachers mentioned they expected to learn more about the workshop approaches
(9 mentions) to improve their teacher practice (7 mentions) and, consequently, to have
an impact on students’ motivation to learn (6), achievement (3), and behavior (1). Three
teachers also mentioned they were curious about the workshop topics.
Concerning conditions for technology use, results indicated that participating teachers
and their students had conditions from the technological point of view, although students
had lower access to technology than teachers. All teachers reported to have data shown in
the classroom, and seven had an interactive board. Students’ access to these resources was
lower—only two and five, respectively.
All teachers mentioned having computer access for their teaching practices, either
desktop, portable, or tablet, and the same applied to the students of 12 of the 14 respondents.
One teacher, who in a previous question reported tablet and laptop access only for teachers,
highlighted in the open-ended question that it is unreliable technology: “The school has
some laptops and some tablets, not always in the best conditions and some rooms have
a desktop computer. All classrooms have a projector, but teachers need to use their own
laptops.” (Q5).
Twelve teachers reported having either a feature phone (2) or a smartphone (10), and
one stated having both. Schools provided students’ access to mobile phones in only five
cases in this cohort. In addition, 12 teachers mentioned having an email account, whereas
nine respondents revealed their students had access to email accounts.
Although the results revealed that teachers and students mostly had access to tech-
nology to use in the school, 11 classified the technology as reasonable, one as bad and
another as good. It is worth noting that one teacher mentioned their school did not provide
students access to any type of the considered technology, providing only internet access.
All respondents reported internet access for themselves and their students, although
only classifying it as reasonable (13), and feeling some constraints in its use, such as
slowness and insecurity, among others. Only one teacher classified their school internet
connection as good.
As to the school policy on mobile devices use in classrooms, seven teachers mentioned
it was allowed, five said it was forbidden, and two acknowledged not knowing. One step
further, six respondents mentioned that the school provided guidance to students on the
proper use of mobile devices, five reported this did not happen in their school, and three
did not know.
No relevant technology-conditions changes were reported by the teachers at the end
of the workshop.

3. Results and Discussion

This section presents and discusses the results obtained through the initial and final
questionnaires, with the aim of analyzing the impact of the workshop on the teacher
trainees’ professional development. Therefore, it is organized according to the dimensions
of the research question—basic knowledge, teacher experience as a learner, reported use
in teaching practice, and opinions about benefits and barriers—for all the training topics
(mobile learning, AR use in education, and game-based learning).

27
 This section presents and discusses the results obtained through the initial and final
questionnaires, with the aim of analyzing the impact of the workshop on the teacher train-
ees’ professional development. Therefore, it is organized according to the dimensions of
the research question—basic knowledge, teacher experience as a learner, reported use in
teaching
Educ. Sci. 2021, 11, 404 practice, and opinions about benefits and barriers—for all the training topics
(mobile learning, AR use in education, and game-based learning).

3.1. Basic Knowledge

3.1. Basic Knowledge
Figure 1 presents the overall results regarding teachers’ answers to the prompt to
explain the concepts ofFigure
mobile1 presents
learning, the overall results
educational regarding
use of AR, teachers’ answers
and game-based learning.to the prompt to
explain the concepts of mobile learning, educational use of AR, and game-based learning.
Detailed analysis is presented, with the support of tables dedicated to each key concept.
Detailed analysis is presented, with the support of tables dedicated to each key concept.

At the beginning
learning
Mobile

At the end
Game-based Educational

At the beginning
use of AR

At the end

At the beginning
learning

At the end

0 2 4 6 8 10 12 14

Does not know/answer Answers incorrect Answers partially correct Answers correct

Figure 1. Frequency of teachers’ types of answers to the questions on basic knowledge about mobile learning, educational
Figure 1. Frequency of teachers’ types of answers to the questions on basic knowledge about mobile
use of AR, and game-based
learning, learning.
educational use of AR, and game-based learning.
Table 1 presents the types of answers given by teachers, their frequency, and examples
Table 1 presents the types of answers given by teachers, their frequency, and exam-
of citations that illustrate each type of answer given by teachers when asked about the
ples of citations that illustrate each type of answer given by teachers when asked about
meaning of mobile learning, at the beginning and end of the workshop. It reveals that the
the meaning of mobile learning, at the beginning and end of the workshop. It reveals that
number of incorrect answers decreased (from 3 to 1), the number of partially correct answers
the number of incorrect answers decreased (from 3 to 1), the number of partially correct
increased (from 2 to 4), and the correct answers remained the same (8); hence, small positive
answers increased (from 2 to
differences 4), andthe
between theinitial
correct answers
and the finalremained
phases ofthe
thesame (8); hence,
workshop were registered.
small positive differences between the initial and the final phases of the workshop were
registered. of teachers’ answers on their understanding of the mobile-learning concept, at the beginning and end
Table 1. Comparison
of the workshop.
Table 1. Comparison of teachers’ answers on their understanding of the mobile-learning concept,
at the beginning and end of theAtworkshop.
the Beginning At the End
Type of Answer
f1 Citation Example f1 Citation Example
Did not know/answer
At the
1
Beginning
“I have no idea what it is.” Q1 2 1
At the End Q6 2
Type of Answer
f1 Citation Example f1 Citation Example
“Mobile devices being used naturally in
id not know/answer 1 “I have no idea what itsystem
“Education is.” Q1
using2 1
mobile Q6 2
students’ daily lives, can provide valuable
Answered incorrectly 3 1
devices.” Q9 support
“Mobile devices being used to students
natu-and teachers in the
school context.” Q4
rally in students’ daily lives, can
“Education system using mobile “I understand by mobile learning the use of
nswered incorrectly 3 1 provide valuablemobile
support to in
devices stu-
learning approaches
devices.” Q9
outsidein
dents and teachers the classroom,
the school and inside, based on
“Learning that is supported by games combined with emerging technologies
Answered partially correctly 2 4 context.” Q4
new education technologies.” Q8 of Augmented Reality (AR), integrating
“Learning that is supported by “I understandprinciples
by mobile of Geocaching,
learning and use of app that
wered partially correctly 2 4 can facilitate and improve student-centered
new education technologies.” Q8 the use of mobile devices in learn-
teaching-learning experiences.” Q8
“Learning using mobile devices
Answered correct, but focused “Learning using a mobile device, such as a
8 (mobile phones, tablets, etc.).” 8
on the technology facet cell phone, smartphone.” Q9
Q4
Total 14 14
1 f: Frequency or number of teachers’ answers included in each type of answer; 2 Q: questionnaire ID.

28
Educ. Sci. 2021, 11, 404

It is noteworthy to highlight that, even at the end of the workshop, all answers
considered correct were simple and technocentric definitions of mobile learning, not
considering, e.g., the mobility of learner [1–3], despite these issues being discussed in
face-to-face sessions.
At the end of the workshop, when questioned about examples of mobile learning
projects or initiatives, 10 teachers mentioned at least one educational mobile app example,
which was an increase compared to the eight teachers in the first questionnaire [30]. Most
teachers (8) mentioned the EduPARK app, which was used during the training. The Khan
Academy was mentioned twice, the same as at the beginning of the workshop, although
this platform is not necessarily used in mobile contexts.
The results presented above indicate a small impact of the workshop on teachers’
ability to explain the concept of mobile learning. The strategies explored in the workshop
included discussion in face-to-face sessions, analyzing texts about mobile learning to
produce a reflexive text, and using mobile devices to learn in a training context. However,
their impact on teachers’ learning fell behind what was initially expected. Moreover,
practical tasks, such as naming examples of mobile learning, seemed to be easier for teachers
when compared to theoretical tasks, such as defining the concept. This was reinforced by a
teacher’s answer to the questionnaire prompt “Indicate at least one [workshop] activity
that was too difficult and explain why”: “The initial reflection was too theoretical. In
my opinion, teachers mainly need to have a practical component in training. I think that
searching a certain topic adds little to the improvement of teaching practice” (Q6). This
teacher revealed that they did not value the literature search’s potential contribution to
teaching practice.
Table 2 reveals the types of answers given by teachers when asked about the meaning
of educative use of AR, at the beginning and end of the workshop. It is possible to identify
a decrease in the number of teachers acknowledging not knowing or not answering, from
4 to 1, and of teachers answering incorrectly, from 7 to 4. The same table also shows an
increase in the number of teachers answering partially correct (usually only defining AR,
and not mentioning its potential for education), from 3 to 7, and in the correct answers,
from 0 to 2. These results revealed the trainees’ difficulty in explaining the educative use of
AR, although progress was made by teachers concerning this topic.

Table 2. Comparison of teachers’ answers on their understanding of the use of AR for educational purposes, at the beginning
and end of the workshop.

At the Beginning At the End

Type of Answer
f1 Citation Example f1 Citation Example
Did not know/answer 4 N/A 1 N/A
“I think it’s the use of cameras” “AR allows to have a more detailed and enhanced
Answered incorrectly 7 4
Q7 view of the object in question.” Q12 2
“Enrichment of a natural
Answered partially ”Integration of virtual elements in real-world
3 environment with virtual 7
correctly visualizations via mobile devices.” Q13
objects.” Q13
“AR combines the real world with the virtual
world, which can be three-dimensional and
interactive in real time. The information can . . .
Answered correctly 0 N/A 2
support the understanding of phenomena and
abstract concepts that are not possible to observe
using a traditional manual.” Q7
Total 14 14
1 f: Frequency or number of teachers’ answers included in each type of answer; 2 Q: questionnaire ID.

Again, at the end of the training, when questioned about examples of AR use in
education, 9 teachers presented at least one educational AR example; all mentioned the one
explored in the workshop. Compared with the results from the beginning of the workshop,
only 2 teachers mentioned adequate examples, and 12 did not present any example at

29
Educ. Sci. 2021, 11, 404

all [30]—there was an evident impact at this level. From these results, it is reasonable to
claim that the workshop had an impact on teachers’ knowledge of educative use of AR,
particularly in what concerns the ability to provide concrete examples.
Finally, Table 3 presents the types of answers given by teachers when asked about
their understanding of game-based learning, at the beginning and end of the workshop.
The table reveals small differences between the initial and the final phases of the workshop,
regarding ”Did not know/answer”, from 2 to 1, and ”Answered correct, but in a simple
way”, from 9 to 11.

Table 3. Comparison of teachers’ answers on their understanding of game-based learning, at the beginning and end of
the workshop.

At the Beginning At the End

Type of Answer
f1 Citation Example f1 Citation Example
Did not know/answer 2 “I do not know.” Q16 2 1 N/A
“Dynamic methodology that leads to “Learning that uses virtual reality applied
Answered incorrectly 1 1
learning in an interactive way.” Q12 to the physical space.” Q15 2
“It is a dynamic learning process. There
Answered partially “I think it’s based on playful activities”
2 1 must be a balance between fun, motivation
correctly Q7
and learning”’ Q12
Answered correctly, but “Use of games to learn the syllabus.”
9 11 “Use of the game as a learning tool.” Q4
in a simple way Q15
Total 14 14
1 f: Frequency or number of teachers’ answers included in each type of answer; 2 Q: questionnaire ID.

At the end of the training, when questioned about examples of game-based learning,
12 teachers presented at least one illustrative example, and 7 mentioned the one explored
in the workshop. At the beginning of the workshop, only 5 teachers were able to present
valid examples of game-based learning [30], so the cohort of teachers revealed progress at
this level.
In summary, the presented results indicated that the workshop had a higher impact
on the topic about which teachers knew less: the educative use of augmented reality. A
smaller impact was registered concerning mobile learning and game-based learning. The
impact was more evident, in all topics, when teachers were asked to mention examples.

3.2. Teacher Experience as A Learner

Figure 2 summarizes teachers’ experience, as learners, with mobile learning, AR
educational use, and game-based learning. At the beginning of the workshop, teachers
reported the following previous experience: (a) All (14) had experienced mobile learning,
even though only 6 presented valid examples; (b) none had learned with AR; and (c) 8 had
experienced game-based learning [30].
When asked if they had used mobile devices, AR, and/or games to learn during the
workshop, 13 teachers mentioned they experienced the three approaches; 1 teacher reported
having used only mobile devices and games to learn, but not AR. When comparing these
results with teachers’ previous experience, it is possible to claim that all became more
experienced with these approaches, particularly concerning AR. For 13 respondents, the
workshop allowed them to experience this technology to learn for the first time. Only
1 teacher mentioned not using AR to learn during the workshop. This result may be
interpreted in two different ways. Either this individual did not use AR technology, despite
being given the opportunity to do so, or this teacher used it, but considered that this
experience did not provide learning. As the questionnaires were anonymous, it was not
possible to explore further this issue.

30
 3.2. Teacher Experience as A Learner
Figure 2 summarizes teachers’ experience, as learners, with mobile learning, AR ed-
ucational use, and game-based learning. At the beginning of the workshop, teachers re-
Educ. Sci. 2021, 11, 404 ported the following previous experience: (a) All (14) had experienced mobile learning,
even though only 6 presented valid examples; (b) none had learned with AR; and (c) 8
had experienced game-based learning [30].

Mobile learning 14
14

Educational use of AR 13
0

Game-based learning 14
8

0 4 8 12 16
frequency
Teachers’ experience as learners
during the workshop before the workshop

Figure2.2.Comparison
Figure Comparisonofofteachers’
teachers’experience
experienceasaslearners
learnerswith
withmobile
mobiledevices,
devices,AR,
AR,and
andgame(s),
game(s),
before and during the workshop.
before and during the workshop.

3.3. Reported
When Use
asked in if
Teaching Practice
they had used mobile devices, AR, and/or games to learn during the
Figure 3 summarizes teachers’ exploration
workshop, 13 teachers mentioned of the workshop
they experienced the three approaches
approaches;in1their prac-
teacher re-
tices.
portedAthaving
the beginning
used onlyofmobile
the workshop,
devices and 11games
teachers reported
to learn, previous
but not experience
AR. When in
comparing
exploring mobile
these results with devices withprevious
teachers’ their students to promote
experience, learning,
it is possible to although
claim thatonly 6 pre-
all became
sented
more valid examples
experienced in these
with response to the following
approaches, question.
particularly These results
concerning AR. For seem to indicate
13 respondents,
that
the teachers
workshop areallowed
beginning
them totointegrate mobile
experience learning in to
this technology their practices
learn [30],time.
for the first as advo-
Only
cated by several
1 teacher Horizon
mentioned notReports
using AR [33–35].
to learnThe fact that
during teachers began
the workshop. Thisto promote
result may mobile
be inter-
Educ. Sci. 2021, 11, x FOR PEER REVIEW 10 of 22
learning was
preted in in line
two with aways.
different previous study
Either this[24] and withdid
individual thenot
Portuguese State of Education
use AR technology, despite
Report
being [36].
given the opportunity to do so, or this teacher used it, but considered that this ex-
perience did not provide learning. As the questionnaires were anonymous, it was not pos-
sible to explore further this issue.
Mobile learning 12
7
3.3. Reported Use in Teaching Practice
Figure 3use
Educational summarizes
of AR teachers’ exploration of the workshop
9 approaches in their prac-
0
tices. At the beginning of the workshop, 11 teachers reported previous experience in ex-
ploring mobile devices with their students to promote learning, although only 6 presented
Game-based 12
valid exampleslearning
in response to the following question. These11results seem to indicate that
teachers are beginning to integrate mobile learning in their practices [30], as advocated by
several Horizon Reports0[33–35]. The4fact that teachers
8 began12
to promote 16mobile learning
was in line with a previous study [24] and with the Portuguese State of Education Report
frequency
[36]. Teachers’ use in their practices
during the workshop before the workshop

Figure3.3.Comparison
Figure Comparisonofofteachers’
teachers’use
useofofmobile
mobiledevices,
devices,AR,
AR,and
andgame(s)
game(s)inintheir
theirpractices,
practices,before
before
and during the workshop.
and during the workshop.

Teachers
Teacherswho whoreported
reportedneverneverhaving
havingusedusedmobile
mobiledevices
devicesinintheir
theirteaching
teachingpractices
practices
before
beforethetheworkshop
workshopmentioned
mentionedthey theycould
couldchange
changethatthatdue
duetototwo
twomain
mainfactors:
factors:(a)
(a)using
using
mobile
mobiledevices
devicesmaymayincrease
increasestudents’
students’motivation
motivationtotolearn;
learn;and
and(b)(b)mobile
mobiledevices’
devices’avail-
avail-
ability
abilitytotomost
moststudents
students[30].
[30].
No
Noteacher
teacherhad
had previous
previous experience
experience in in using
using ARARto topromote
promotetheirtheirstudents’
students’learning
learn-
ing [30].
[30]. TheThe Horizon
Horizon initiative
initiative [37][37]
hashas placed
placed ARAR technology
technology in in
thethe time-to-adopt
time-to-adopt group
group for
for K-12
K-12 educational
educational contexts,
contexts, highlighting
highlighting its its potential
potential to provide
to provide powerful,
powerful, contextual,
contextual, and
and in situ
in situ visual
visual andand interactive
interactive learning
learning experiences.
experiences. However,
However, our our empirical
empirical results
results indi-
indicated
cated that about 8 years later, teachers are still not exploring this technology withhigh
that about 8 years later, teachers are still not exploring this technology with high
educational
educationalvalue.
value. Factors that could
Factors that could change
changethisthissituation,
situation,according
accordingtoto teachers,
teachers, are:
are: (a)
(a) teacher
teacher knowledge
knowledge ononhowhowto touseuse
ARAR to to promote
promote learning,
learning, developed
developed in in professional
professional de-
development initiatives;(b)
velopment initiatives; (b)knowing
knowingthat that the
the use
use of
of AR
AR technology
technology may may increase
increasestudents’
students’
motivation to learn; (c) AR may facilitate teaching and learning processes; and (d) teacher
willingness to change practice 31 [30].
Seven teachers reported previous experience in promoting game-based learning [30].
From these, 6 presented valid examples, usually mentioning a quiz format. The game ap-
proach is one with high expression in international reports. In the Horizon series, it is
Educ. Sci. 2021, 11, 404

motivation to learn; (c) AR may facilitate teaching and learning processes; and (d) teacher
willingness to change practice [30].
Seven teachers reported previous experience in promoting game-based learning [30].
From these, 6 presented valid examples, usually mentioning a quiz format. The game
approach is one with high expression in international reports. In the Horizon series, it
is presented as an effective and versatile approach with gains in student engagement,
creativity, and authentic learning [38,39]. However, this seems to still be an approach with
limited expression in teacher practices. This result contrasts with Russo, Bragg, and Russo’s
study [18], in which the majority of teachers mentioned using educational games at least
once a week, revealing this approach can be popular among Australian practitioners.
According to this study’s teacher cohort, factors that could promote higher exploration
of games in formal education are: (a) using games, which may increase students’ motivation
to learn; (b) teacher knowledge of games that can be explored to promote learning; and
(c) access to resources, either the games or their supporting technologies [30].
Regarding the use of mobile devices, AR, and games in teaching practices, during
about three months, which corresponded to the workshop period, 12 teachers mentioned
they promoted mobile and/or game-based learning, and 9 teachers mentioned using AR.
As to the reasons for not exploring these approaches, teachers mentioned: (a) lack of
opportunities or time (“It was not timely” (Q4)); (b) lack of resources (“Did not have
conditions in the classroom” (Q16)); (c) lack of teacher readiness (“Because I still don’t
feel comfortable using it” (Q9)); (d) lack of students’ skills (“Because students don’t know
how to use it for this purpose” (Q16)); and (e) COVID-related barriers (“Considering the
pandemic context that we live in” (Q15)).
Moreover, teachers had the opportunity to present additional (optional) comments
on the workshop topics, and their answers revealed teachers generally sustained positive
perceptions, as revealed by the following citations: ”It [mobile leaning] is an asset for
the knowledge acquisition in a more fun and motivating way” (Q2); “AR is a tool that
should be explored in lessons’ (Q5); and ‘It [the game] is an excellent tool for motivation”
(Q1). On the other hand, some teachers seemed to envision difficulties: “I would like to
be able to use them [mobile devices] in the classroom” (Q16); and “It [AR] is still not very
disseminated in the teaching community” (Q14).
Facing these results, it is reasonable to consider that the workshop had an impact
on teachers’ practices, according to their self-reports. However, it is worthwhile to note
that even after long-term training, a small group of teachers (2 to 5, varying with the
considered approach) remained reluctant to try out the approaches for which they were
training. These reluctancy and lack of readiness to explore mobile, AR, and game-based
learning were not evident during the face-to-face training sessions, so it seems reasonable
to assume that teachers tended to contribute less to the large-group discussion when they
did not agree or did not feel comfortable with the approaches under analysis. In the future,
teacher trainers must take this result into account in order to have a deeper impact on the
practices of teachers who are reluctant regarding the exploration of mobile learning, AR,
and game-based learning.

3.4. Opinions about Benefits and Barriers

Figures 4 and 5 and Tables 4–6 summarize teachers’ answers to the open-ended ques-
tion “What potential/advantages do you identify in using [mobile devices/AR/game(s)]
to promote learning?”.

32
 promote learning?”.
Figure 4 reveals that teachers were able to mention more types of benefits and barri-
ers at the end of the workshop, when comparing to the beginning. For example, teachers
identified 24 types of benefits of mobile learning at the beginning, and 31 benefits at the
end. Similarly, teachers identified 26 types of barriers for mobile learning at the beginning
Educ. Sci. 2021, 11, 404 and 37 barriers at the end. Thus, the workshop seems to have had an impact on teachers’
ability to acknowledge both benefits and barriers to the educative use of mobile devices,
AR, and games.

Mobile learning 24

at the beginning
26

Educational use of AR 16
16

Game-based learning 20
19

Educ. Sci. 2021, 11, x FOR PEER REVIEW 13 of 22

0 10 20 30 40
Mobile learning 31
Benefits Barriers 37
at the end

similar benefits to all the approaches, as illustrated by the following citations from Q7:
Educational use of AR 24
“[Mobile devices] promote engagement and motivation in27
students.”; ”[AR] fosters moti-
vation, commitment, and enthusiasm for learning.”; and ”Extrinsic motivation is en-
hanced by the strategies of the game.” This result21 may be related to the fact that all the
Game-based learning
approaches were focused on the same training course and exemplified with the same ed-
31
ucational resource, a mobile app that supports both gaming and AR. However, even in
the initial questionnaire,
0 teachers
5 associated
10 15 these
20 three
25 approaches
30 35 in similar
40 ways. This
was illustrated by the benefits of mobile learning, AR, and game-based learning reported
in Q4: “Learning in a playfulTypes of same sentence includedfrequency
way” (the in all the answers). There-
Benefits Barriers
fore, it seems that the association between the approaches occurred before the training.
Figure 5 summarizes the frequency for each type of benefit teachers mentioned in
Figure 4. Comparison of the frequency of types of benefits and barriers, identified by teachers,
their answers at the beginning and at the e2nd of the workshop. For the two most frequent
on mobile learning, educative use of AR, and game-based learning, at the beginning and end of
benefits, “Supports better learning” and “Motivates students to learn”, there was an in-
the workshop.
crease of frequency after the workshop.

Supports better learning (e.g., Supports5 better learning (e.g., 11

4 13
fast information) 3 fast information) 8

9 10
Motivates students to learn Motivates 7students to learn 6
10 9

2 1
Allows learning in a fun way 1Allows learning in a fun way 1
5 2

1 0
Are easy to use 0 Are easy to use 0
0 0

(Up-to-date) hardware is (Up-to-date)

4 hardware is 4
1 0
available 0 available 0

1 0
Digital resources are available 0 Digital resources are available 0
0 0

Diversifies teaching/ learning Diversifies

2 teaching/ learning 4
2 2
methodologies 0 methodologies 1

0 0
Can be done without the teacher Can
0 be done without the teacher 1
2 0

Other factors (e.g., supports 0 Other factors (e.g., supports 1

1 1
experience sharing) 0 experience sharing,) 1

0 5
15 0 10 5 10
15
frequency frequency
Benefits at the beginning Benefits at the end
Mobile learning Educational use of AR Game-based learning

Figure 5. Comparison of the frequency of each type of benefit identified by teachers on mobile
Figure 5. Comparison of the frequency of each type of benefit identified by teachers on mobile learning, educative use of
learning, educative use of AR, and game-based learning, at the beginning and end of the work-
AR, and game-based learning, at the beginning
shop. and end of the workshop.
33
In a more detailed analysis, Table 5 shows that initially, the most pointed types of
benefits were: (a) ”Motivates students to learn” (total of 26 teacher mentions); (b) “Sup-
ports better learning” (total of 12); and (c) “Allows learning in a funny way” (total of 8),
Educ. Sci. 2021, 11, 404

Table 4. Numbers of types of benefits and barriers on the educative use of mobile devices, AR, and
game(s), according to teachers at the beginning and end of the workshop.

f 1 at the Beginning f 1 at the End

Number of
Mobile AR Game Mobile AR Game
0 0 4 0 0 1 0
1 6 6 10 3 5 7
2 6 2 2 6 5 4
3 2 2 2 4 3 2
Benefits
4 0 0 0 1 0 0
5 0 0 0 0 0 0
6 0 0 0 0 0 0
7 0 0 0 0 0 0
T 24 16 20 31 24 21
A 1.7 1.1 1.4 2.2 1.7 1.5
0 0 0 2 0 1 0
1 5 6 6 4 6 4
2 6 2 5 1 3 5
3 3 2 1 8 3 4
Barriers
4 0 0 0 0 0 0
5 0 0 0 0 0 1
6 0 0 0 0 1 0
7 0 0 0 1 0 0
T 26 16 19 37 27 31
A 1.9 1.1 1.4 2.6 1.9 2.2
1The first column indicates the number of types of benefits (numerals presented in bold) and the number of types
of barriers (numerals also in bold) given in each questionnaire. f: Frequency of teachers’ whose answers included
each number of types of benefits or barriers. T: Total. A: Average.

Table 5. Benefits of exploration of mobile devices, AR, and game(s) in teaching practices, according to teachers at the
beginning and end of the workshop.

f 1 at the Beginning f 1 at the End

Benefits Theme Mobile T Mobile T
AR Game(s) AR Game(s)
Devices Devices
Supports better learning (e.g., fast
5 4 3 12 11 13 8 32
information, higher interactivity)
Motivates students to learn 9 7 10 26 10 6 9 25
Allows learning in a fun way 2 1 5 8 1 1 2 4
Are easy to use 1 0 0 1 0 0 0 0
(Up-to-date) hardware is available 4 1 0 5 4 0 0 4
Digital resources are available 1 0 0 1 0 0 0 0
Diversifies teaching/learning
2 2 0 4 4 2 1 7
methodologies
Can be done without the teacher 0 0 2 2 0 1 0 1
Other factors (e.g., supports experience
0 1 0 1 1 1 1 3
sharing, allows outdoor lessons, etc.)
T 24 16 20 60 31 24 21 76
1f: Frequency of teachers’ answers included in each type of answer. T: Total (values for all the benefits at the beginning and end of the
workshop are resented in the grey cells).

34
Educ. Sci. 2021, 11, 404

Table 6. Barriers to exploration of mobile devices, AR, and game(s) in teaching practices, according to teachers at the
beginning and end of the workshop.

f 1 at the Beginning f 1 at the End

Barriers Theme Mobile T Mobile T
AR Game(s) AR Game(s)
Devices Devices
Risk of poorer learning (e.g., unreliable
1 0 3 4 1 2 3 6
information)
Risk of student distraction 6 2 2 10 7 2 6 15
Risk of demotivation (e.g., to those who
0 0 1 1 0 0 2 2
do not like to play)
Its use is forbidden by school policy 2 0 0 2 2 0 0 2
The hardware is not available or is too
7 6 5 18 7 6 2 15
diverse
Lack of (quality) internet connection 5 3 2 10 6 3 3 12
Lack of suitable digital resources 1 0 0 1 0 3 3 6
Lack of teacher didactic competence on
0 2 0 2 1 1 2 4
these approaches
Lack of teacher and/or student digital
4 1 0 5 5 2 0 7
competence
Lack of time (to prepare, to explore in
0 2 4 6 3 2 4 9
lesson)
Risk of student addiction 0 0 1 1 0 0 2 2
Other factors (e.g., not being current
practice; teacher demotivation, battery 0 0 1 1 5 6 4 15
time)
T 26 16 19 61 37 27 31 95
1f: Frequency of teachers’ answers included in each type of answer. T: Total (values for all the benefits at the beginning and end of the
workshop are resented in the grey cells).

Figure 4 reveals that teachers were able to mention more types of benefits and barriers
at the end of the workshop, when comparing to the beginning. For example, teachers
identified 24 types of benefits of mobile learning at the beginning, and 31 benefits at the
end. Similarly, teachers identified 26 types of barriers for mobile learning at the beginning
and 37 barriers at the end. Thus, the workshop seems to have had an impact on teachers’
ability to acknowledge both benefits and barriers to the educative use of mobile devices,
AR, and games.
Taking into account that the more benefits teachers identified, the more positive
their perspectives could be considered, and that the more barriers teachers identified, the
more negative their perspectives could be considered, Figure 4 seems to indicate that
the workshop had a more intense impact regarding the barriers, which may indicate a
moderate negative view. This was a surprising result after training on these educational
approaches, but it is possible to hypothesize that knowing the approaches better made
teachers more conscious of potential barriers to their implementation.
Presenting results in more detail, Table 4 shows that at the beginning of the workshop,
teachers pointed toward 0 to 3 different types of benefits and barriers for each workshop
approach. They reported a total of 60 benefits (24 for mobile learning, 16 for AR, and 20 for
games) and 61 barriers (26, 16, and 19, respectively), which seemed to reveal teachers’
initial neutral perspective on the educative use of mobile devices, AR, and games [30].
After the workshop, teachers pointed toward 0 to 4 types of benefits and 7 types of
barriers for each workshop approach. They identified a total of 76 (31 for mobile learning,
24 for AR, and 21 for games) types of benefits and 95 (37, 27, and 31 respectively) types
of barriers.

35
Educ. Sci. 2021, 11, 404

It is notable how, at both data collection moments, teachers seemed to associate the
benefits and the barriers of mobile learning with the ones of AR, and also with those of
game-based learning. For example, even at the end of the workshop, teachers mentioned
similar benefits to all the approaches, as illustrated by the following citations from Q7:
“[Mobile devices] promote engagement and motivation in students.”; ”[AR] fosters motiva-
tion, commitment, and enthusiasm for learning.”; and ”Extrinsic motivation is enhanced
by the strategies of the game.” This result may be related to the fact that all the approaches
were focused on the same training course and exemplified with the same educational
resource, a mobile app that supports both gaming and AR. However, even in the initial
questionnaire, teachers associated these three approaches in similar ways. This was illus-
trated by the benefits of mobile learning, AR, and game-based learning reported in Q4:
“Learning in a playful way” (the same sentence included in all the answers). Therefore, it
seems that the association between the approaches occurred before the training.
Figure 5 summarizes the frequency for each type of benefit teachers mentioned in
their answers at the beginning and at the e2nd of the workshop. For the two most frequent
benefits, “Supports better learning” and “Motivates students to learn”, there was an
increase of frequency after the workshop.
In a more detailed analysis, Table 5 shows that initially, the most pointed types of
benefits were: (a) ”Motivates students to learn” (total of 26 teacher mentions); (b) “Supports
better learning” (total of 12); and (c) “Allows learning in a funny way” (total of 8), with
this last one having more expression regarding the game-based learning approach. At the
end of the workshop, motivation (total of 25 mentions) was surpassed by better learning
(total of 32), these being the most relevant benefits for the majority of teachers. These
results were in line with a previous study [10], related to a short-term teacher training
on the same topics, but focused on mobile learning, where student motivation, ease in
finding information (which in this study was included in the better learning category), and
technology availability stood out. The results indicated that teacher training supported
more teachers in identifying learning promoted by these approaches, although with less
intensity for games (an increase from 3 to 8 mentions in the present study) and higher
intensity for AR (an increase from 4 to 13 mentions). Therefore, the workshop seems to
have had a greater impact on teachers’ ability to acknowledge how a technology, unknown
to most of the cohort before the training (the AR), can support learning. Moreover, teachers’
answers were longer and more elaborated at the end of the workshop, which may be
interpreted as an impact of the workshop on teachers’ understanding of the learning
that can be potentiated by the approaches. This is illustrated by the following citations:
“[AR] improves knowledge” (Q9 at the beginning); and “[AR] shows in three dimensions
(3D) some topics covered in the classroom, so that the student can better understand the
«reality». Greater interactivity” (Q9 at the end of the workshop).
Fun learning is a theme that was present in all workshop topics; however, teachers’
focus on this feature decreased (from a total of 8 to a total of 4 mentions). This result seems
to point out that training in these approaches contributes to the transformation of teachers’
perceptions of mobile devices, AR, and games being used just for fun to perceptions that
these approaches can effectively support deeper learning. The articulation of these elements
may sustain mentality changes regarding learning, which is a claim that has been made
previously [10].
Other types of benefits pointed out by teachers included aspects related to the avail-
ability of technological hardware (usually, associated with mobile devices) and software,
its ease of use, and the fact that by using these approaches, teachers and students diversify
teaching and learning experiences, particularly in formal education contexts. Some benefits
frequently selected by teachers in the previous study [10] were not mentioned by this
study’s cohort, specifically ”The information is up-to-date”, ”Does not waste paper”, or
”Facilitates teachers work, namely in assessment”.

36
 Other types of benefits pointed out by teachers included aspects related to the avail-
ability of technological hardware (usually, associated with mobile devices) and software,
its ease of use, and the fact that by using these approaches, teachers and students diversify
teaching and learning experiences, particularly in formal education contexts. Some bene-
Educ. Sci. 2021, 11, 404 fits frequently selected by teachers in the previous study [10] were not mentioned by this
study’s cohort, specifically ”The information is up-to-date”, ”Does not waste paper”, or
”Facilitates teachers work, namely in assessment”.
Figure
Figure66summarizes
summarizesthe thefrequency
frequency of
ofeach
eachtype
typeof
ofbarrier
barrierteachers
teachersmentioned
mentioned in
intheir
their
answers at the beginning and end of the workshop. Overall, most barriers registered
answers at the beginning and end of the workshop. Overall, most barriers registered higher
higher frequencies
frequencies at the
at the end of end of the workshop.
the workshop.

Risk of poorer learning (e.g., Risk1of poorer learning (e.g., 1

0 2
unreliable information) 3 information)
unreliable 3
6 7
Risk of student distraction Risk of
2 student distraction 2
2 6
Risk of demotivation (e.g., to Risk
0 of demotivation (e.g., to 0
0 0
those who do not like to play) those 1who do not like to play) 2
Its use is forbitten by school Its use 2is forbitten by school 2
0 0
policy 0 policy 0
The hardware is not available The hardware is not available
7 7
6 6
or is too diverse 5
or is too diverse 2
Lack of (quality) internet Lack of (quality)
5 internet 6
3 3
connection 2 connection 3
Lack of suitable digital 1 Lack of suitable digital 0
0 3
resources 0 resources 3
Lack of teacher didactic 0 Lack of teacher didactic 1
2 1
competence on these… 0 competence on these… 2
Lack of teacher and/or student Lack of teacher and/or
4 student 5
1 2
digital competence 0 digital competence 0
Lack of time (to prepare, to 0Lack of time (to prepare, to 3
2 2
explore in lesson) explore 4in lesson) 4
0 0
Risk of student addiction 0 Risk of student addiction 0
1 2
Other factors (e.g., teacher 0 Other factors (e.g., teacher 5
0 6
demotivation) 1 demotivation) 4

0 5 010 5 10
frequency frequency
Barriers at the beginning Barriers at the end
Mobile learning Educational use of AR Game-based learning

Figure 6. Comparison of the frequency

Figure of each type
6. Comparison of barrier,
of the identified
frequency of eachby teachers
type on mobile
of barrier, learning,
identified educative
by teachers use of
on mobile
learning,
AR, and game-based learning, at the educative
beginninguse
andofend
AR,ofand
the game-based
workshop. learning, at the beginning and end of the work-
shop.
In a more detailed analysis, Table 6 shows that the most frequent types of barriers
were:In(a) “The hardware
a more is not available
detailed analysis, Table 6orshows
is too diverse” (fromfrequent
that the most 18 teacher mentions
types to 15);
of barriers
(b) “Lack
were: of (quality)
(a) “The internet
hardware connection”
is not available or(from 10 mentions
is too to 12); 18
diverse” (from and (c) ”Risk
teacher of student
mentions to
distraction”
15); (b) “Lack(from 10 mentions
of (quality) to 15).
internet All these barriers
connection” (from 10were found in
mentions to a12);
previous
and (c)study
”Risk[10],
of
although
student with a much
distraction” smaller
(from expression:
10 mentions to 15).(a)All1 teacher (in a total
these barriers wereoffound
26) mentioned the
in a previous
lack of access to mobile devices for some students; (b) 5 teachers selected the need for
an internet connection or its lack of quality; and (c) 9 teachers selected student access
to distractions. In fact, in the previous study, the most expressed barriers were the risk
of developing mobile-device dependence, increased battery consumption, and school
prohibition of mobile device use in classes. All these barriers emerged in the present study
as well, but with different intensities. Nevertheless, these issues need to be considered
by teachers and teacher trainers in order to effectively promote mobile learning, AR, and
game-based approaches in teacher practices.
Other types of barriers pointed out by teachers at both data-collection moments
included aspects related to lack of time (from 6 to 9 mentions), lack of teacher or student
digital competence (from 5 to 7), risk of poorer learning (from 4 to 6), and lack of suitable
digital resources (from 1 to 6). None of these barriers were identified in the previous
37
Educ. Sci. 2021, 11, 404

study [10]. However, other themes in common with the previous study, which emerged
with small expression in this study, were: (a) School prohibition of use in classes (only for
mobile devices); (b) Lack of teacher didactic competence in these approaches; and (c) Risk
of student addiction (only for mobile devices in the first study and only for games in the
present study). So, once again the same barrier themes were identified in both studies,
although with different intensities.
Finally, it is noteworthy to highlight that teachers pointed out some barriers that were
not exclusive to these approaches. For example, regarding lack of time, a teacher mentioned
at the end of the workshop: “The time spent on classroom with gaming activities may
lead to not being able to teach all the [curricular] content” (Q4). The same may be said
regarding most teaching approaches that teachers do not know yet, which require a higher
investment in terms of planning and implementation during lessons.

4. Conclusions
This work addresses the need to analyze the impact of continuous teacher training
initiatives concerning new technology supporting teaching approaches on professional
development. It WAS conducted under a case study [31] on the development process of
mobile AR games for STEM learning by 14 in-service teachers during a 50 h workshop in
Portugal. Hence, the present paper presented the analysis of the impact of this workshop
on teacher trainees’ professional development through a questionnaire filled in at the
beginning and end of the workshop.
Regarding teachers’ understanding of mobile learning, AR, and game-based learning,
this study registered a higher impact on AR educative use, which was the less-known
approach for teachers, compared to mobile and game-based learning. Teachers revealed
difficulties in explaining concepts’ definitions even at the end of the workshop; however,
they demonstrated increased understanding and increased ability to provide concrete
examples of each teaching approach.
In what concerns teachers’ experience in educational contexts, teachers became more
experienced with mobile learning, AR, and game-based learning as learners themselves,
and reported having explored them with their students during the three-month workshop
period. Hence, this study’s results support the claim that the analyzed workshop promoted
teacher practice changes, although only through self-reports. Future investigations may
include teacher practice observations to ascertain the accuracy of teachers’ claims.
Finally, teachers’ ability to identify benefits and barriers through the workshop teach-
ing approaches increased with the training, although with more intensity with respect to
barriers. Nevertheless, the most mentioned benefits pointed out by teachers were related to
the improved student learning and motivation that may be promoted by mobile AR games.
On the other hand, among the barriers to the implementation of these approaches in teach-
ing practices that stood out was the unavailability of proper hardware to support them, or
even the hardware diversity that emerges from the bring-your-own-device option, the risk
of student distraction, and lack of a quality internet connection. These barriers gain higher
relevance if the 2019 State of Education Report [36] is considered, as it mentions the wear
and tear of the Portuguese schools’ computer park, and the internet connection fragility
in the majority of schools. Therefore, the presented set of barriers to implementation is
relevant both for in-service teachers and for teacher trainees preparing future professional
development initiatives.
With only 14 participant teachers, this study did not aim to provide results generaliz-
able to the entire Portuguese teacher population. However, this teacher cohort very closely
reflected the Portuguese teacher profile [23,24] in terms of gender and experience. Hence,
this study could be a good indicator of the teacher population status on these matters.
In sum, this study presented empirical evidence that long-term teacher training
concerning the educative exploration of new technologies, which includes the creation of
educational resources, may contribute to the transformation of teachers’ perceptions. These
seemed to evolve from a perspective that mobile devices, AR, and game-based approaches

38
Educ. Sci. 2021, 11, 404

are considered just for the fun they provide, to perceptions that these approaches can
effectively support deeper learning, and hence changing mentalities on how people can
learn [40]. Consequently, a recommendation to educational researchers and teacher trainers
emerged: to build upon these workshop methodologies in order to have an impact on
teacher professional development in what concerns the integration of innovative teaching
technologies in their practice.

Author Contributions: Conceptualization, M.M.M. and L.P.; data curation and formal analysis,
M.M.M.; investigation, M.M.M. and L.P.; methodology, M.M.M. and L.P.; project administration,
M.M.M. and L.P.; writing—original draft, M.M.M. and L.P.; writing—review and editing, M.M.M.
and L.P. Both authors have read and agreed to the published version of the manuscript.
Funding: The work of the first author was funded by national funds (OE), through the University of
Aveiro, in the scope of the framework contract foreseen in the numbers 4, 5, and 6 of Article 23 of the
Decree-Law 57/29 August 2016, changed by Law 57/19 July 2017.
Institutional Review Board Statement: Ethical review and approval were waived for this study, due
to being a study involving a small number of healthy adults, participating under informed consent,
and with no sensitive data collection.
Informed Consent Statement: Written informed consent was obtained from all subjects involved in
the study.
Data Availability Statement: The data presented in this study are available on request from the
corresponding author. The data are not publicly available due to privacy issues.
Acknowledgments: The authors thank the participating teachers for accepting taking part of this
study. The authors also thank the reviewers for their insightful comments.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or
in the decision to publish the results.

Appendix A

Table A1. Questions of the Section 3 of the Initial Questionnaire.

Question ID Type Questions in English (in Portuguese in the Original Questionnaire)

Explain, in your own words or citing authors in the literature, what you mean by
G4Q01 open-ended
mobile learning.
If you know of any educational initiatives or projects involving mobile devices,
G4Q02 open-ended
briefly describe an example.
Have you ever used mobile devices to learn?
Yes
G4Q03 closed-ended, one
No
I don’t know/I don’t remember
G4Q04 open-ended Briefly describe an experience where you have used a mobile device to learn.
How often do you use mobile devices to promote learning?
Never used
closed-ended, one
G4Q05 Sometimes
option selection
Periodically (e.g., twice a month per class)
Very often (for example, almost every day)
G4Q06 open-ended What could motivate you to use mobile devices to promote learning?
Briefly describe an experience where you have used mobile devices to promote
G4Q07 open-ended
learning.
What potential/advantages do you identify in the use of mobile devices to promote
learning?
G4Q08 open-ended
(Please clearly present at least three strengths/advantages that may affect the
teaching class)

39
Educ. Sci. 2021, 11, 404

Table A1. Cont.

Question ID Type Questions in English (in Portuguese in the Original Questionnaire)

What barriers/constraints do you recognize in using mobile devices to promote
G4Q09 open-ended learning?
(Please clearly state at least three barriers/constraints that can affect teachers)
G4Q10 open-ended Do you have any comments regarding mobile learning?
Explain, in your own words or citing authors in the literature, what you mean by
G5Q01 open-ended
educational use of augmented reality.
If you know any educational initiatives or projects that involve augmented reality,
G5Q02 open-ended
briefly describe an example.
Have you ever had any experience in augmented reality (regardless of context)?
closed-ended, one Yes
G5Q03
option selection No
I don’t know/I don’t remember
Briefly describe an experience in which you have used augmented reality
G5Q04 open-ended
(regardless of context).
Have you ever used augmented reality to learn?
closed-ended, one Yes
G5Q05
option selection No
I don’t know/I don’t remember
G5Q06 open-ended Briefly describe an experience in which you have used augmented reality to learn.
How often do you use augmented reality to promote learning?
Never used
closed-ended, one
G5Q07 Sometimes
option selection
Periodically (e.g., twice a month per class)
Very often (for example, almost every day)
G5Q08 open-ended What could motivate you to use augmented reality to promote learning?
Briefly describe an experience in which you have used augmented reality to
G5Q09 open-ended
promote learning.
What potential/advantages do you identify in using augmented reality to promote
G5Q10 open-ended learning?
(Please clearly present at least three strengths/advantages that may affect teachers.)
What barriers/constraints do you recognize in the use of augmented reality to
G5Q11 open-ended promote learning?
(Please clearly state at least three barriers/constraints that can affect teachers.)
G5Q12 open-ended Do you have any comments regarding the educational use of augmented reality?
Explain, in your own words or citing authors in the literature, what you mean by
G6Q01 open-ended
game-based learning.
If you know any educational initiatives or projects that involve the use of game(s),
G6Q02 open-ended
briefly describe an example.
Have you ever used game(s) to learn?
closed-ended, one Yes
G6Q03
option selection No
I don’t know/I don’t remember
G6Q04 open-ended Briefly describe an experience in which you have used game(s) to learn.
How often do you use game(s) to promote learning?
Never used
closed-ended, one
G6Q05 Sometimes
option selection
Periodically (e.g., twice a month per class)
Very often (for example, almost every day)
G6Q06 open-ended What could motivate you to use game(s) to promote learning?
G6Q07 open-ended Briefly describe an experience where you have used game(s) to promote learning.

40
Educ. Sci. 2021, 11, 404

Table A1. Cont.

Question ID Type Questions in English (in Portuguese in the Original Questionnaire)

What potential/advantages do you identify in using game(s) to promote learning?
G6Q08 open-ended
(Please clearly present at least three strengths/advantages that may affect teachers.)
What barriers/constraints do you recognize in using game(s) to promote learning?
G6Q09 open-ended
(Please clearly state at least three barriers/constraints that can affect teachers.)
G6Q10 open-ended Do you have any comments regarding game-based learning?

Table A2. Questions of the Section 3 of the Final Questionnaire.

Question ID Type Questions—in English (in Portuguese in the Original Questionnaire)

Explain, in your own words or citing authors in the literature, what you mean by
G4Q01 open-ended
mobile learning.
If you know of any educational initiatives or projects involving mobile devices,
G4Q02 open-ended
briefly describe an example.
In this Training Workshop did you use mobile devices to learn?
closed-ended, one
G4Q03 Yes
option selection
No
During the period in which this Training Workshop took place, did you use mobile
closed-ended, one devices to promote learning?
G4Q04
option selection Yes
No
Why did you decide not to use mobile devices to promote learning during the
G4Q05 open-ended
period in which this Training Workshop took place?
What potential/advantages do you identify in the use of mobile devices to promote
G4Q06 open-ended learning?
(Please clearly present at least three strengths/advantages that may affect teachers)
What barriers/constraints do you recognize in using mobile devices to promote
G4Q07 open-ended learning?
(Please clearly state at least three barriers/constraints that can affect teachers)
G4Q08 open-ended Do you have any comments regarding mobile learning?
Explain, in your own words or citing authors in the literature, what you mean by
G5Q01 open-ended
educational use of augmented reality.
If you know any educational initiatives or projects that involve augmented reality,
G5Q02 open-ended
briefly describe an example.
In this Training Workshop did you use augmented reality to learn?
closed-ended, one
G5Q03 Yes
option selection
No
During the period in which this Training Workshop took place, did you use
closed-ended, one augmented reality to promote learning?
G5Q04
option selection Yes
No
Why did you decide not to use augmented reality to promote learning during the
G5Q05 open-ended
period in which this Training Workshop took place?
What potential/advantages do you identify in using augmented reality to promote
G5Q06 open-ended learning?
(Please clearly present at least three strengths/advantages that may affect teachers)
What barriers/constraints do you recognize in the use of augmented reality to
G5Q07 open-ended promote learning?
(Please clearly present at least three barriers/constraints that may affect teachers)

41
Educ. Sci. 2021, 11, 404

Table A2. Cont.

Question ID Type Questions in English (in Portuguese in the Original Questionnaire)

G5Q08 open-ended Do you have any comments regarding the educational use of augmented reality?
Explain, in your own words or citing authors in the literature, what you mean by
G6Q01 open-ended
game-based learning.
If you know any educational initiatives or projects that involve the use of game(s),
G6Q02 open-ended
briefly describe an example.
In this Training Workshop did you use game(s) to learn?
closed-ended, one Please select only one of the following options:
G6Q03
option selection Yes
No
During the period in which this Training Workshop took place, did you use game(s)
to promote learning?
closed-ended, one
G6Q04 Please select only one of the following options:
option selection
Yes
No
Why did you decide not to use game(s) to promote learning during the period this
G6Q05 open-ended
Training Workshop took place?
What potential/advantages do you identify in using game(s) to promote learning?
G6Q06 open-ended
(Please clearly present at least three strengths/advantages that may affect teachers)
What barriers/constraints do you recognize in using game(s) to promote learning?
G6Q07 open-ended
(Please clearly state at least three barriers/constraints that can affect teachers)
G6Q08 open-ended Do you have any comments regarding game-based learning?

References
1. Winters, N. What is mobile learning? In Big Issues in Mobile Learning: Report of a Workshop by the Kaleidoscope Network of Excellence
Mobile Learning Initiative; Sharples, M., Ed.; University of Nottingham: Nottingham, UK, 2006; pp. 4–8.
2. Farley, H.; Murphy, A.; Rees, S. Revisiting the Definition of Mobile Learning. Available online: https://www.learntechlib.org/p/
171139/ (accessed on 1 July 2021).
3. Traxler, J. Learning in a Mobile Age. Int. J. Mob. Blended Learn. 2009, 1, 1–12. [CrossRef]
4. Zydney, J.M.; Warner, Z. Mobile apps for science learning: Review of research. Comput. Educ. 2016, 94, 1–17. [CrossRef]
5. Huang, Y.-L.; Chang, D.-F.; Wu, B. Mobile Game-Based Learning with a Mobile App: Motivational Effects and Learning
Performance. J. Adv. Comput. Intell. Intell. Inform. 2017, 21, 963–970. [CrossRef]
6. Plass, J.L.; Homer, B.D.; Kinzer, C.K. Foundations of Game-Based Learning. Educ. Psychol. 2015, 50, 258–283. [CrossRef]
7. Laine, T. Mobile Educational Augmented Reality Games: A Systematic Literature Review and Two Case Studies. Computers 2018,
7, 19. [CrossRef]
8. De Freitas, S. Are Games Effective Learning Tools? A Review of Educational Games. Educ. Technol. Soc. 2018, 21, 74–84. [CrossRef]
9. Tobias, S.; Fletcher, J.D.; Wind, A.P. Game-Based Learning. In Handbook of Research on Educational Communications and Technology;
Spector, J., Merrill, M., Elen, J., Bishop, M., Eds.; Springer: New York, NY, USA, 2014; pp. 485–503. ISBN1 978-1-4614-3184-8.
ISBN2 978-1-4614-3185-5.
10. Marques, M.M.; Pombo, L. Game-Based Mobile Learning with Augmented Reality: Are Teachers Ready to Adopt It? In Smart
Innovation, Systems and Technologies; Springer: New York, NY, USA, 2014; Volume 158, ISBN 9789811396519. [CrossRef]
11. Gao, F.; Li, L.; Sun, Y. A systematic review of mobile game-based learning in STEM education. Educ. Technol. Res. Dev. 2020, 68,
1791–1827. [CrossRef]
12. Karakoç, B.; Eryılmaz, K.; Özpolat, E.T.; Yıldırım, İ. The Effect of Game-Based Learning on Student Achievement: A Meta-Analysis
Study. Technol. Knowl. Learn. 2020, 1–16. [CrossRef]
13. Qian, M.; Clark, K.R. Game-based Learning and 21st century skills: A review of recent research. Comput. Hum. Behav. 2016, 63,
50–58. [CrossRef]
14. Chang, C.-Y.; Hwang, G.-J. Trends in Digital Game-Based Learning in the Mobile Era: A Systematic Review of Journal Publications
from 2007 to 2016; Int. J. Mob. Learn. Organ. 2019, 13, 68–90. [CrossRef]
15. Abdul Jabbar, A.I.; Felicia, P. Gameplay Engagement and Learning in Game-Based Learning: A Systematic Review. Rev. Educ.
Res. 2015, 85, 740. [CrossRef]
16. Hwang, G.-J.; Wu, P.-H.; Chen, C.-C.; Tu, N.-T. Effects of an augmented reality-based educational game on students’ learning
achievements and attitudes in real-world observations. Interact. Learn. Environ. 2016, 24, 1895–1906. [CrossRef]

42
Educ. Sci. 2021, 11, 404

17. Pivec, P. Game-Based Learning or Game-Based Teaching? (Report No. 1509); British Educational Communications and Technology
Agency (BECTA). 2009. Available online: http://www.becta.org.ukhttp//www.becta.org.uk (accessed on 12 May 2021).
18. Russo, J.A.; Bragg, L.; Russo, T. How Primary Teachers Use Games to Support Their Teaching of Mathematics. Int. Electron. J.
Elem. Educ. 2021, 13, 407–419.
19. Groff, J.; Clarke-Midura, J.; Owen, V.E.; Rosenheck, L.; Beall, M. Better Learning in Games: A Balanced Design Lens for a New
Generation of Learning Games; Massachusetts Institute of Technology: Cambridge, MA, USA, 2015.
20. Markouzis, D.; Fessakis, G. Rapid Prototyping of Interactive Storytelling and Mobile Augmented Reality Applications for
Learning and Entertainment—The case of “k-Knights”. Int. J. Eng. Pedagog. 2016, 6, 30. [CrossRef]
21. Sung, M. A Study of Adults’ Perception and Needs for Smart Learning. Procedia Soc. Behav. Sci. 2015, 191, 115–120. [CrossRef]
22. Akçayır, M.; Akçayır, G. Advantages and challenges associated with augmented reality for education: A systematic review of the
literature. Educ. Res. Rev. 2017, 20, 1–11. [CrossRef]
23. Cabero, J.; Barroso, J. The educational possibilities of Augmented Reality. New Approaches Educ. Res. 2016, 5, 44–50. [CrossRef]
24. Pombo, L.; Marques, M.M.; Carlos, V. Mobile augmented reality game-based learning: Teacher training using the EduPARK app.
Da Investig. às Práticas. 2019, 9, 3–30. [CrossRef]
25. Ertmer, P.A.; Ottenbreit-Leftwich, A.T. Teacher technology change: How knowledge, confidence, beliefs, and culture intersect.
J. Res. Technol. Educ. 2010, 42, 255–284. [CrossRef]
26. Pombo, L.; Marques, M.M.; Afonso, L.; Dias, P.; Madeira, J. Evaluation of a mobile augmented reality game application as an
outdoor learning tool. Int. J. Mob. Blended Learn. 2019, 11, 59–79. [CrossRef]
27. Pombo, L. Learning with an app? It’s a walk in the park. Prim. Sci. 2018, 12–15. Available online: https://www.learntechlib.org/
p/190712/ (accessed on 18 May 2021).
28. Jeffrey, K.; Kinshuk, L. Factors Impacting Teachers’ Adoption of Mobile Learning. Available online: http://www.jite.org/
documents/Vol13/JITEv13ResearchP141-162MacCallum0455.pdf (accessed on 17 April 2021).
29. De Freitas, S. Learning in Immersive Worlds A Review of Game-Based Learning Prepared for the JISC e-Learning Programme.
Available online: http://www.jisc.ac.uk/media/documents/programmes/elearninginnovation/gamingreport_v3.pdf (accessed
on 10 May 2021).
30. Marques, M.M.; Pombo, L. Teachers’ experiences and perceptions regarding mobile augmented reality games: A case study of a
teacher training. In Proceedings of the INTED2021 Conference; Chova, L.G., Martínez, A.L., Torres, I.C., Eds.; IATED: Online, 2021;
pp. 8938–8947.
31. Yin, R.K. Case Study Research and Applications: Design and Methods, 6th ed.; SAGE Publications, Inc.: Los Angeles, CA, USA, 2017.
32. Harrison, H.; Birks, M.; Franklin, R.; Mills, J. Case study research: Foundations and methodological orientations. Forum Qual.
Sozialforsch. 2017, 18. [CrossRef]
33. Horizon Project Advisory Board. Horizon Report 2007 Edition (NMC and Educause); The New Media Consortium: Stanford, CA,
USA, 2007; ISBN 0976508745.
34. Johnson, L.; Adams, S.; Estrada, V.; Freeman, A. NMC Horizon Report: 2015 K-12 Edition; The New Media Consortium: Austin, TX,
USA, 2015; ISBN 9780991482856.
35. Johnson, L.; Smith, R.; Levine, A.; Haywood, K. 2010 Horizon Report: K-12 Edition; The New Media Consortium: Austin, TX, USA,
2010; ISBN 9780982533444.
36. Conselho Nacional de Educação. Estado da Educação 2019; Conselho Nacional de Educação: Lisboa, Portugal, 2020.
37. Johnson, L.; Adams, S.; Cummins, M. NMC Horizon Report: 2012 K-12 Edition; The New Media Consortium: Austin, TX, USA,
2012.
38. Johnson, L.; Adams, S.; Haywood, K. NMC Horizon Report: 2011 K-12 Edition; The New Media Consortium: Austin, TX, USA,
2011; ISBN 9780982829097.
39. Johnson, L.; Becker, S.A.; Estrada, V.; Freeman, A. NMC Horizon Report: 2014 K-12 Edition; The New Media Consortium: Austin,
TX, USA, 2014; ISBN 9780991482856.
40. Pombo, L.; Marques, M.M. An App that Changes Mentalities about Mobile Learning—The EduPARK Augmented Reality Activity.
Computers 2019, 8, 37. [CrossRef]

43
 education
sciences
Article
Development of Game-Based M-Learning Apps
for Preschoolers
Dionísia Laranjeiro

CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers,

Department of Education and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal;
[email protected]

Abstract: Recent studies indicate tablets as the preferred devices of preschool children, due to porta-
bility, autonomy of use and variety of apps. There is also extensive evidence of the contributions
of digital technologies in different areas of learning at these ages. The Aprender XXI project aimed
to develop game-based learning apps, with content recommended in the Curriculum Guidelines
for Pre-School Education (CGPE). The project used Design-Based Research (DBR) methodology,
which combines scientific research and technological development. It was divided into three phases:
preliminary study (literature review, search for existing apps, study of preschool curriculum), de-
velopment (specifications, scriptwriting, design and programing) and evaluation (tests with users
and conclusions). The preliminary study identified the needs to define robust apps. The evaluation
with children and educator validated the development and defined improvements in the apps. As a
result, we obtained four thematic apps—environment, health, citizenship and professions, composed
of a set of games, suitable for autonomous use for children or for educational activities guided by
educators in kindergarten. In addition, a website collects children’s play data, which is represented
with flowers in a virtual world, to illustrate their participation/collaboration for a better future.






Keywords: game-based learning; mobile learning; educational apps; kindergarten; design-based research
Citation: Laranjeiro, D. Development
of Game-Based M-Learning Apps for
Preschoolers. Educ. Sci. 2021, 11, 229.
https://doi.org/10.3390/ 1. Introduction
educsci11050229
Today’s children belong to a generation that is familiar with technologies, such as
computers, tablets and the internet. Prensky [1] called this generation “digital natives”,
Academic Editor: James Albright
affirming that digital language is part of their lives and it can even change their thinking
patterns. It is important to understand some characteristics and skills of these children
Received: 16 April 2021
Accepted: 8 May 2021
to provide interesting learning, such as recognizing that they are used to receiving large
Published: 12 May 2021
amounts of information and instant gratification, they prefer access to non-linear infor-
mation, they like to network and perform various tasks at the same time. This argument
Publisher’s Note: MDPI stays neutral
is criticized by Thomas [2], claiming that the concept of digital natives presupposes a
with regard to jurisdictional claims in
homogeneous generation, which learns differently from previous generations. Hattie and
published maps and institutional affil- Yates [3] reinforce that human capabilities are not that malleable. There is no scientific
iations. evidence that living in an age dominated by new technologies causes changes in the brain
organization or in cognitive abilities. The European Commission assumed a balanced
perspective, declaring that children up to eight years old can easily acquire digital skills but
their abilities are limited to the level of cognitive development. Additionally, some children
Copyright: © 2021 by the author.
show more interest and skills than others. Their lives are not dominated by technologies, as
Licensee MDPI, Basel, Switzerland.
they carry out many other activities. Sometimes children use technologies to complement
This article is an open access article
their offline interests, for example, to search online and photograph [4]. The use of ICT
distributed under the terms and in kindergarten is not consensual and there is still some resistance to integration in this
conditions of the Creative Commons educational context, caused by several reasons: lack of conditions and outdated devices in
Attribution (CC BY) license (https:// kindergartens, lack of adequate training for kindergarten teachers, beliefs of some parents
creativecommons.org/licenses/by/ and educators that the use of technologies at these ages is not beneficial [5]. These concerns
4.0/). are supported by researchers and child advocacy groups who argue that technologies can

45
Educ. Sci. 2021, 11, 229

impede physical, emotional, cognitive and social development in early childhood. Some
risks include obesity, eyestrain, sleep disorders and social isolation [6]. Larry Cuban [7]
claims that research on the use of technology in the learning of preschoolers is limited
and ambiguous, and it cannot be confirmed whether preschoolers should use computers,
for how long, for what tasks and under what conditions. Richard Mayer [8] agrees that
there are strong claims about the potential of technology in transforming education that
has not been scientifically tested. However, these high expectations have led to massive
implementations of technological interventions in schools that have had failed results. He
believes that the problem lies in the approach to learning with technology, which is focused
on technology while it should be focused on the learner. In a technology-centered approach
the goal is to give access to the latest technology, hoping that teachers and students will
adapt and use it. The learner-centered approach focuses on the learner, the way learning
happens and how the mind works. In this approach, it is technology that is adapted
to the needs of students and teachers. The joint position statement of NAEYC and the
Fred Rodgers Center argues that if the integration of technology in kindergarten is built
upon solid developmental foundations, and educators are aware of the challenges and
opportunities, the quality of the educational program is improved by using technologies
for the benefit of all children. Some recommendations are that educators have digital
literacy to make informed choices that can maximize learning opportunities; they should
monitor the time and use of devices by children; technology must be used to support
learning and expand access to new content. It should not replace creative games, outdoor
experiences, and interactions with children and adults [9]. The American Academy of
Pediatrics also set recommendations to help families with young children promote healthy
media use. At home, children aged two to five should use the media together with their
parents to help them understand the content. Use should be limited to one hour a day,
with a pre-selection of quality content. From the age of six, parents must define time limits,
locations and types of media that can be used, maintaining balance with physical activity,
reading, social interaction and other activities essential to health. Children should not use
screens one hour before going to bed, to avoid sleep disturbances [10]. Despite all the
concerns and risks identified, extensive literature documents the contributions of digital
technologies in the learning and skills development of pre-school children [11–13]. In lan-
guage development, digital technologies can impact the fluency and complexity of speech,
the development of verbal communication and written language, vocabulary, syntax and
word recognition [14]. As examples, the use of word processing software can enhance the
understanding of written code, and the creation of podcasts can enrich orality, expression
and communication [15]. In math, there are specific applications that stimulate mathe-
matical concepts, such as counting, classification or logical thinking. Drawing programs
develop geometric and spatial knowledge. Programing encourages creative thinking and
problem solving [14]. Internet allows contact with other natural, social and cultural realities.
A good example is the use of Google Earth® for an interactive and precise exploration of
the world, to understand space and local geography in relation to global geography [15].
Using various technologies, pre-school children acquire digital literacy, which includes the
ability to search online, use the mouse and touchscreen and create multimodal texts [12].
For the development of artistic skills, there are interactive tools and environments, which
can be used for exploring music, drawing, painting, animation, creative writing, poetry
and storytelling. These are usually open-ended applications, quite versatile and adaptable
to different contexts, and it is up to the educator to plan their use in learning activities.
With regard to develop preschoolers’ cognitive skills, carrying out learning activities based
on apps and digital games improves abstract thinking, reflection, analysis and evaluation
of information, problem solving, spatial representation, attention and memorization [13].
Social tools foster learning and teamwork, by providing communication and collaboration
features, which can be used in group projects, in the classroom or remotely. This type of
activities promotes interaction, team spirit and develops language and critical thinking.
Exposure to other perspectives and divergence of opinions stimulates dialogue and joint

46
Educ. Sci. 2021, 11, 229

analysis of possibilities, which results in higher quality decisions. All this involvement
increases the interest in the content and leads to better learning performances [16]. Despite
the rapid evolution of technologies, there are some propositions in the literature with more
than twenty years similar to the current ideas, regarding children of preschool age: they
write on the keyboard and use computers and software with confidence; they face difficul-
ties as challenges and almost always manage to overcome them; computers help children
to learn, so the educator has to know how to help them, assuming a less instructional and
more guiding role; computers encourage social interaction and communication, as children
prefer to work with friends and talk while they work; cooperative work on the computer
generates enthusiasm and interest in the activity; children with special educational needs
can benefit from digital technologies, in physical, emotional and social terms [17,18].
Recent studies point out that access to mobile devices is universal for children up
to eight years old and indicate tablets as their favorite devices. Using comparative data
from 2011 and 2017, a report on the use of screen media by North American children ages
zero to eight points out that the major change has occurred in the type of devices rather
than in screen time, which remained approximately 2 h a day. The use of mobile devices
has increased substantially (from 5 to 48 min a day) while the use of television, DVD
players, computers, and video game consoles has decreased. The study indicates that 95%
of children aged zero to eight have a smartphone at home and 42% have their own tablet.
67% of parents believe that the use of digital media helps their children in learning [19]. In
Portugal, a study on the use of technology in early childhood also indicates the tablet as
their preferred device. In terms of consumption, 50% of children play digital games and
50% of children aged three to five who access the internet, use their own tablet [20]. The
mobile market has brought a wide range of educational apps that integrate different ways
of understanding, knowing and expressing, valuing the multiple forms of intelligence—
linguistic, mathematical, creative and artistic. A learner with access to a selected set of
educational apps can perform activities in languages, arts, sciences, among others [21].
Children in pre-school education can benefit from the characteristics of tablets, due to the
appropriate size, portability and long battery life, which gives them autonomy in use and
allows them to explore apps in various locations. The possibility of multitouch interaction
also favors joint use with friends, educators and parents. There is a wide variety of apps
with multimodal content, available for different educational purposes and appropriate
for these ages [22]. Educational apps can be classified into three levels: instructional
apps, based on exercise-reward, aimed at the acquisition of specific content and skills;
manipulable apps that manage ideas and content, allowing multiple responses to a set of
variables; constructive apps, with an open structure that serves to create or communicate,
allowing learners to build a learning object from a set of available components. In terms
of contribution to cognitive development, there is a growth in the learner’s involvement
and motivation from instructional to constructive apps [23]. Most educational apps are not
directly related to specific curricular themes, but have versatility and flexibility to be used
in different contexts.
In the development of digital pedagogical applications, it is necessary to take into
account three components that influence learning: the child (to whom it is addressed), the
adult (who guides learning) and technology. Older children, with previous experience,
longer exposure and access time at home, can benefit more from technology in learning [12].
Adults have a mediating role, whether they are parents or kindergarten educators. They
can plan and guide activities with technology, prepare questions, encourage interaction
and experimentation. They must monitor the screen time and use of technological de-
vices, support and intervene when necessary, promoting gradual autonomy [24]. With
regard to technology, design mechanisms, teaching-learning approaches and content are
conditioning factors for learning [12]. In terms of design, pedagogical applications for
children must use graphics and actions that provide context; they must use simple and
clear instructions, based on images; they must have an intuitive interface and interactivity
for independent use, but also a challenging approach with multiple opportunities for

47
Educ. Sci. 2021, 11, 229

success, to maintain interest [24]. They must have clear and understandable feedback
mechanisms for the child, that make connections between actions and results, guiding
the child’s performance. The content must be appropriate and meaningful to the child,
allowing to explore different domains (cognitive, physical, emotional) and offering oppor-
tunities to complement learning with offline activities, such as drawings or playing games
outside [25]. Digital games are a very appealing type of multimedia content for children,
which can be used in the teaching-learning process. The learning of younger children is
intuitive and action-oriented. Children experience and discover reality through error and
success, observing the consequences of their actions, influenced by the context, with the
support of adults [26]. While playing games, children have an interested and cooperative
attitude. They are active in the search for information and oriented to achieve results [27].
Games present content in a fun way and give opportunities to practice, with immediate
feedback on the results. They allow children to experience and solve challenges, encourage
collaboration, teach how to respect rules, how to work individually and as a team. Games
provide variety and flexibility of learning, for example, there may be individual games,
games for small or large groups, closed or open games [28].
This paper presents the project Aprender XXI, an enterprise R&D project, involving
a research team, a technological team and the participation of users from a kindergarten.
This project aimed to develop mobile apps to promote the learning of pre-school children,
using game-based learning strategies and addressing content areas, according to the CGPE,
from the Portuguese Ministry of Education [29]. Technological development was driven by
scientific research, to generate rigorous knowledge about the development process and the
results achieved. The project lasted 24 months and adopted the DBR methodology, as it
was considered an appropriate methodology for the context and the defined objectives.

2. Materials and Methods

The term DBR encompasses a group of research methodologies based on design and
development, which have common characteristics [30], such as—design experiments [31],
development research [32], educational design research [33].
This methodology allows exploring the potential of technologies in education to solve
a real problem, bringing practical and scientific contributions [32]. It may include the
development of technological products, materials and instructional activities that use
technologies. DBR includes problem analysis, design and conception of an educational
prototype, evaluation and review activities [34]. The user must be involved in the design
and development process, namely in the use and evaluation of the prototype [35]. Scientific
knowledge influences development, which is then tested in the field, bringing empirical
data to improve the product and validate knowledge [32]. In conclusion, DBR is based, on
the one hand, on rigorous and reflective research to build knowledge and design principles
that can guide developments. On the other hand, it seeks to solve concrete problems,
involving users and acting on the context [36].
For all these reasons, DBR methodology was chosen for the development of this project.
The Aprender XXI apps were designed to explore the potential of mobile game-based
learning in kindergarten. A kindergarten classroom was involved, with the participation of
an educator and 22 children in the development process, which means that the prototypes
of the apps were used and evaluated in the context (kindergarten), to be corrected and
improved in a final version. The results achieved are the final versions of four game-based
learning apps for preschool children.
The model proposed by Plomp [33] to operationalize the DBR methodology was
adapted to the project, divided into three phases: Phase 1—preliminary study, which
included the literature review, search for solutions that already exist in the market and
acquisition of knowledge regarding learning contents and skills to be acquired at preschool
age; Phase 2—development, started with the writing of technical specifications of the apps,
the choice of learning content to be included in the apps, the adaptation and script of digital
content, design and programing of technological products; Phase 3—evaluation, which

48
Educ. Sci. 2021, 11, 229

included tests with users in kindergarten and product improvements, ending with the final
versions of the apps.

2.1. Preliminary Study

The first phase was the preliminary study necessary for the development of game-
based learning apps for children aged three to six years old. A literature review on
children’s learning with technology and games served to contextualize and substantiate the
theoretical framework and to become aware of research already performed, such as recent
cases of use of apps in learning practices in kindergarten. The platforms Scopus and Web
of Science were selected for search and constitution of the documental corpus. In order
to contemplate national projects, searches were also carried out in national journals and
repositories of the Portuguese Universities. Further, complementary search was done on the
aggregator system Google Scholar. Research equations were combinations and variations
of the terms: apps/tablets, kindergarten/preschool and children/toddlers. The research
was carried out in Portuguese and English languages. As criteria for inclusion, researchers
considered articles of scientific journals, research reports, conference proceedings and
doctoral theses with theoretical discussions and practical experiences of using apps for
learning in kindergarten. Documents centered on other levels of education, or technologies
that fell outside the scope (e.g., websites, CD-ROM) were excluded. Since it is not the
objective of this article to present the entire literature review, some studies were selected
to be presented in the “results of the preliminary study” section, due to their relevance to
the project.
Another preliminary study carried out was the search and selection of educational
apps suitable for these ages, to learn more about the “state of the art” and understand the
existing offer that can be used in an educational context, identifying the features they offer,
innovations, trends and good practices, as well as weaknesses that can be suppressed. To
this end, a search was made for apps available in online stores (App Store® and Google
Play® , 2017 versions). These stores have limitations in terms of organization, search and
filtering, making it difficult to select apps, in an immense offer with daily growth. Thus, it
was decided to select the most prominent apps in stores, by the number of comments or
downloads. At the same time, a web search was done, combining some terms (apps, chil-
dren/kids, kindergarten/preschoolers, learning, games) to find lists of apps and reviews
from experts that reflect on the most used and well positioned apps for these purposes.
This research allowed to find app sites, with information about their educational approach
and content. In the Stores, it was possible to view demonstration videos and download the
apps to test. Some apps were excluded because they were outside the scope of the project.
As a result, a solid set of apps was obtained, to be subsequently evaluated. The evaluation
considered dimensions relevant to the educational level, establishing the categories of
analysis based on the CGPE. In relation to game dynamics, the presence of rules, obstacles
and quantification of actions was verified. In terms of skills development, the possibility of
creating, building, collaborating or cooperating was considered. Regarding the content, it
was analyzed whether the focus was scientific, simulation of reality, or storytelling.
Since this project focuses on children aged three to six, who may attend kindergarten,
it was necessary to understand what the approaches were to formal learning for children
of this age group, as apps must respond to educators’ pedagogical practices in order to
be efficient. To this end, CGPE and educational guidelines from other European countries
were studied, in order to define appropriate content at national level, but with the potential
for internationalization.

2.2. Development
Based on the results of the preliminary study, the development phase began. Tech-
nological development consisted of a set of activities that were divided into smaller tasks
with defined objectives, deadlines, human resources and dependencies. It started with
the definition of specifications—detailed descriptions of what needs to be developed to

49
Educ. Sci. 2021, 11, 229

respond to the identified needs. The specifications delimit the scope of the project and
present guidelines, common to the different functions and team members: content creation,
design and programing [37]. Four types of specifications were defined:
• Concept specifications—presentation of the creative concept and general approach,
explaining what is common and different in all apps, how it applies in each app and
on the website, how they interconnect and relate all conceptual elements;
• Design specifications—instructions for the graphical approach, interface characteris-
tics, design and illustration components;
• Content specifications—indications about the content to be developed in each app;
guidelines for screenwriting;
• Programing specifications—programing languages to be used, technologies to sup-
port development.
After defining the specifications, content planning started, with the choice of themes
and subthemes for each app, based on the CGPE. Screenwriting described what appears
on each screen, in terms of content, graphics, interaction, feedback and sound. After the
specifications and content were defined, the design began, with graphic studies, creation of
characters, scenarios, objects and animations. The content was programed in four apps
for mobile devices (Android® 4.0.3 or later and iPads® OS 8.0 or later) and an integrating
website. The development also included internal tests, before the evaluation with users. In
the internal tests, errors were corrected and interaction details were changed to improve
the user experience.

2.3. Evaluation
The evaluation phase aimed to test the prototypes with users, to correct or validate
the direction of technological development. It was carried out in a kindergarten classroom,
with an educator and 22 children, aged three and six years old, as the use in the context
and the interaction between users was essential for the evaluation [38]. Data collection was
based on the observation of the use of prototypes by the children and the educator. At
the end of the observation sessions, an interview was conducted with the educator. The
specific objectives of this evaluation were:
• Obtain data from the observation of children using the apps—data related to the user
experience (check if the interface was intuitive, easy to learn and use, check if the
objectives were understood) and the importance given to the content (understanding,
interest, games played more times);
• Obtain data on the educator’s perception—usefulness of games and content, dynamics
and activities that can be developed with the apps, usage expectations;
• Analyze the data, as a way of evaluating the prototypes. Making decisions about
necessary changes, to improve the apps in the final versions.
Some kindergartens were identified in order to integrate a group of children in the
project. It should be a heterogeneous group, to test the prototypes with different ages. The
educator should have a predisposition to use technologies and a willingness to integrate
the project in pedagogical activities. A kindergarten classroom was intentionally selected
because it complied with the prerequisites. The invitation was formally made to the
institution and the project was presented to the educator, who transmitted the information
to the parents and obtained informed consent. The pilot project was approved and took
place at the kindergarten, during the first period of the 2018–19 school year, involving
the kindergarten teacher and her group with 22 children between the ages of three and
six years old.
The data collection technique chosen was participant observation, as it is useful for
studying small groups and events that last a short time. It is an opportunity to record
information as it occurs in a context, to study non-verbal behaviors and people who
have difficulty verbalizing their ideas, such as pre-school children [39]. Following the
considerations of the authors, Cohen, Manion and Morrison [40], the observation was

50
Educ. Sci. 2021, 11, 229

planned, defining when, where, how and what to observe. Four observation sessions were
held, one for each app. The researcher interacted with the users (children and educator) in
their natural environment, giving initial explanations about the apps. Then, the children
used tablets to explore the apps, freely and/or with the guidance of the educator.
It was necessary to take into account some limitations of this data collection technique.
The presence of an observer in the context can impact the behavior of those being observed
or lead to disturbances of what is intended to be observed, which may cause changes
in the outcomes of a study [41]. In a research with children, it is necessary to consider
the way they look at adults, which can lead to inhibition or the search for attention and
approval. It is difficult for children to accept an adult as an equal, although they can tolerate
their presence in the group [42]. In this investigation, the children involved were used to
receiving adults in the classroom to do specific activities, such as talking about professions,
carrying out science experiments, cooking. In this sense, the researcher chose to follow
the procedures they were used to, in order to reduce the strangeness element. She was
introduced as a person who came to do an activity with tablets. During the sessions, she
explained the games, helped the children punctually, but intervened as little as possible
to make the use as natural as possible—individually, between peers or with the educator.
This posture allowed the researcher to take notes during the observation, recording a large
number of comments immediately, to reduce the possibility of bias in the data, which is
another limitation of the observation technique [41]. An observation grid was created to fill
with data collected in the sessions. The grid had a table for each app, with games identified
by a name and separated by lines. In each line, there were fields to fill in, regarding the
use of a child: (1) liked it or not; (2) understood or not; (3) comments during the game;
(4) notes from the researcher, such as interaction with other children or with the educator.
The data collected about each child were the name, age and gender, which were then stored
separately and coded from C1 to C22.
Each session lasted the morning activity period. The observation was semi-structured,
because it started from a set of topics that were intended to be observed, in order to obtain
a rich description of the events that occurred in the context [40]. In addition, an interview
was conducted as an instrument to collect more detailed and in-depth information from the
kindergarten educator, to understand the possible use of apps in an educational context,
and also to collect opinions and suggestions. To help conduct the interview, a guide was
created with questions previously formulated to answer the objectives of the evaluation. It
used open questions related to experience and opinion, to understand thoughts, beliefs
and attitudes about certain topics [40]. The interview was conducted in the kindergarten.
Data were analyzed to obtain the educator’s perception and to assess the need to make
changes in technological development.

3. Results
The results of each phase are presented separately and sequentially, as each phase
influenced the next. The discussion and conclusions combine the results of the three phases.

3.1. Results of the Preliminary Study

The literature review focused on projects that contributed with results on the role
of tablets in pre-school education and the feasibility of integration in this context. The
complete study can be read in the preliminary study report [43]. For this article, some
projects were selected in which kindergarten children used apps for learning, covering
different areas of knowledge and skills. One study, on drawing activities with tablets, re-
ports that children enjoyed the experience, showed persistence and quality in the drawings,
developed familiarity with the tablet and used them progressively more time and more
autonomously, over six weeks [44]. In another research, the tablet was used for literacy
activities, with apps for creating narratives and acquiring vocabulary. Children understood
the apps, played an active role in the games, shared the tablet, showed a good mood and
communicated. The activity promoted language learning and increased speech skills [45].

51
Educ. Sci. 2021, 11, 229

The use of math apps in a Greek kindergarten had better results than teaching traditional
mathematics [46]. The use of apps for multimedia production, in groups, generated curios-
ity, communication and collaborative work, stimulated creativity, hand-eye coordination
and fine motor skills [47]. A dollhouse decoration game app allowed children to reproduce
traditional play, work on spatial organization skills and learn the relationship between
objects. At the same time, it stimulated cooperative work and creativity. A photo editing
app allowed the creation of graphic compositions. These experiences showed the potential
to influence the cognitive development, autonomy and creativity of children with mobile
apps [48].
In the survey of educational apps for pre-school education, an extensive number of
apps was found. The full study is available in the preliminary study report. A selection
was made of the apps that best meet the project’s goals, as they were suitable for the
ages of three to six years old, and covered content areas proposed in the CGPE. As a
summary, Table 1 presents some apps (or collections of apps), with categories relevant to
the educational level, according to CGPE [49].

Table 1. List of selected apps. Adapted from Zagalo & Laranjeiro (2018).

Scientific Reality
Open Rules Obstacles Scores Build Collaboration Storytelling
Content Simulation
Toca Boca x x x x
Dr. Panda x x x x
Sago Mini World x x x
Sesame Street x x x
Montessory
x x x x
Preschool
LumiKids x x x x x
Lego Apps x x x
Duck Duck Moose x x x
Minilab x x x

In general, all apps have sets of games and correspond to several categories. The most
prevalent categories are the presence of open rules, which allow children to act freely; the
possibility of building/construction within the game, which stimulates creativity; and the
simulation of reality, with activities that are not compulsory, but closely linked to their daily
lives. This emphasizes the agency of players who are in a phase of discovering themselves
and others. A less interesting point is the approach to scientific content in some apps,
with greater rigidity and a more instructional approach, using game logics as overcoming
obstacles to score, stating the need to quantify activities. Another category rarely found
was collaboration/cooperation, which may be associated with parents’ fears about their
children’s access to the online environment, but that narrows the potential of mobile apps.
It was also concluded that, in the national context, there were few apps aligned with the
CGPE. The learning apps in Portuguese were scarce, more oriented to the consumer market
and, tendentially, instructional. Most of the educational apps found were in English, which
may be an exclusion factor for use in Portuguese classrooms, particularly in pre-school
education, when children have little contact with foreign languages.
To understand the approach to formal learning in kindergarten, the CGPE and edu-
cational guidelines from other European countries were consulted, with the intention of
developing apps according to the content, and pedagogical practices of kindergarten teach-
ers. European countries provide official guides for pre-school education, in documents
that refer to learning objectives at an emotional, social and intellectual level. They give
indications about pedagogical approaches, learning activities and assessment methods and
define skills to be developed, such as language and communication. There are differences
in the learning goals defined by country, but most emphasize reading, numeracy and

52
Educ. Sci. 2021, 11, 229

logical thinking, as well as the importance of the arts and knowledge of the world for
children aged three to six. Pre-school institutions are free to decide the curriculum, the
organization of activities and methodologies they want to adopt. A balance between indi-
vidual and group work is recommended, and also between activities initiated by educators
and children. About half of the countries believe that free play should be encouraged as an
essential way of learning [50]. In Portugal, the CGPE follow an approach that is supported
by the three main areas of human development: socio-affective, motor and cognitive. The
document proposes that children develop through exploration and action on the world
around them, with activities planned by the educator that should enhance the discovery
of relationships with themselves, with others and with objects, and encourage reflection,
understanding, transformation and complexification of knowledge. It distinguishes three
content areas—personal and social training; expression and communication; knowledge
of the world. Personal and social training integrates citizenship, multiculturality, sense
of identity, education for values, such as tolerance, sharing and justice, seeking to make
children conscious and supportive, with the ability to solve problems. The area of expres-
sion and communication refers to the acquisition of different forms of language, for the
child to make their representations, interact with others, express thoughts and feelings.
It is divided into the scientific areas of physical education, artistic education, oral and
written language, mathematics. In the last area, knowledge of the world, the child’s natural
curiosity is stimulated, fostering new situations for discovering and exploring the world.
Three components are used: scientific methodology, sciences and technologies. The CGPE
recognize the child as the central element of learning and consider playing as the natural
and spontaneous way of learning. It advises a holistic approach to content areas, in a
process of articulated construction of knowledge [29].

3.2. Results of the Development Phase

After the preliminary study was completed, development began. Four apps were
defined. Each app refers to a main theme, with five games that address sub-themes,
accordingly with the content areas and knowledge domains defined in the CGPE.

3.2.1. Nature Kids

Nature Kids is an environmental education app, which explores concepts and chal-
lenges related to sustainability, biodiversity, recycling, forest and animal life preservation
(Figure 1). It has five games: (1) Forest—find baby animals in the forest (squirrels, rab-
bits, deer. . . ) and hand them over to their parents; (2) Feed the Farm Animals—feed the
cows, chickens, pigs and other animals on a farm, according to their requests, that are
presented with illustrations of food in speech bubbles; (3) Plantations—the whole pro-
cess of sowing or planting until harvesting fruits and vegetables—putting seeds, water,
fertilizer, watching them grow, harvesting; (4) Recycling—separate waste for the correct
Educ. Sci. 2021, 11, x FOR PEER REVIEW 10 of 24
containers—paper, glass, packaging; (5) Meteorology—dress up characters, according to
the weather conditions.

(a) (b) (c)

Figure 1. Print screens
Figure 1. screens of
of games
games from
from the
the app
app Nature
Nature Kids:
Kids: (a)
(a) Recycling;
Recycling; (b)
(b) Plantation;
Plantation; (c)
(c) Feed
Feedthe
theFarm
FarmAnimals.
Animals.

The main domains of the CGPE addressed in this app are the knowledge of the phys-
53
ical and natural world (characteristics of animals and plants, nature preservation and re-
spect for the environment, properties of objects, climate and influence in daily life). Table
2 summarizes the learning that can be promoted by the Nature Kids app, according to the
CGPE.
Educ. Sci. 2021, 11, 229

The main domains of the CGPE addressed in this app are the knowledge of the
physical and natural world (characteristics of animals and plants, nature preservation
and respect for the environment, properties of objects, climate and influence in daily life).
Table 2 summarizes the learning that can be promoted by the Nature Kids app, according
to the CGPE.

Table 2. Nature Kids games, content and learning.

Learning, Attitudes, Know-How

Game Content Area Knowledge Domain
That Can Be Explored
Characteristics of animals, identify
Knowledge of the world Science—physical/natural world
similarities and differences of animals
Democratic coexistence and Respect for the other, solidarity,
Personal and social training
Forest citizenship interventional attitude
Identify numbers, quantities, count;
Expression and
Math locate objects and group by
communication
characteristics
Characteristics of animals, identify
Knowledge of the world Science—physical/natural world
similarities and differences of animals
Feed animals
Making decisions, taking into account
Personal and social training Independence and autonomy the well-being of others; distinguish
food and its importance for health
Characteristics of plants, nature
preservation, respect for the
Plantations Knowledge of the world Science—physical/natural world
environment; phenomena and
transformations in the natural world
Respect for the environment;
properties of objects, identify
Knowledge of the world Science—physical/natural world
similarities and differences of
Recycling materials
Expression and Identify and locate objects; group by
Math
communication characteristics
Climate influence in daily life;
similarities and differences of
Meteorology Knowledge of the world Science—physical/natural world materials (clothes);
phenomena/transformations in the
natural world

3.2.2. Healthy Kids

Healthy Kids is the health education app, that addresses the subthemes of healthy
eating, hygiene, oral health, physical activity (Figure 2). It has five games: (1) Creat-
ing Healthy Snacks—using available food, children can create sandwiches and drinks;
(2) Supermarket—shopping for healthy food at the supermarket, selecting options from
the shelves. When passing the food in the cashier, it will check if the choices are healthy;
(3) Dentist—going to the dentist, clean the teeth, treat caries and put a brace; (4) Physical
Activity—choose an activity to train outdoors, such as cycling, rollerblading, skateboarding;
(5) Beach—going to the beach, put on sunscreen, put on a cap, stay in the shade, play in the
sand, see the flags.

54
Educ. Sci. 2021, 11, 229
Educ. Sci. 2021, 11, x FOR PEER REVIEW 11 of 24

(a) (b) (c)

Figure 2. Print screens
Figure 2. screens of
of games
games from
from the
the app
app Healthy
Healthy Kids:
Kids: (a)
(a) Beach;
Beach; (b)
(b) Supermarket;
Supermarket; (c)
(c) Physical
Physical Activity.
Activity.

The
The main
mainlearning
learningcontents
contentsrelate toto
relate health
healthcare andand
care safety of the
safety of body, well-being
the body, and
well-being
responsible choices,
and responsible which
choices, are included
which in theinareas
are included of world
the areas knowledge
of world and personal
knowledge and
and personal
social training
and social of theofCGPE.
training Table Table
the CGPE. 3 summarizes the learning
3 summarizes that can
the learning be can
that promoted by the
be promoted
Healthy Kids app,
by the Healthy Kidsaccording to the CGPE.
app, according to the CGPE.

Table3.3.Healthy
Table HealthyKids
Kidsgames,
games,content
contentand
andlearning.
learning.

Game Content Area Knowledge DomainKnowledge Learning, Learning, Attitudes,

Attitudes, Know-How That CanKnow-How
Be Explored
Game Content Area Domain
That Can
Knowing how to take care of their well-being;Be Explored
Healthy Personal and Knowing
understand the importance how to take
of healthy careand
habits of their
healthy
Independence/autonomy well-being; understand the
Snacks social training foods; ability to make good choices; distinguish food and
importance of healthy habits and
Healthy Snacks Personal and social training Independence/autonomy
its importance for health
healthy foods; ability to make good
Knowing how to take care of their
choices; well-being;
distinguish food and its
Personal and importance
Understand the importance of healthy for health
habits and healthy
Independence/autonomy
social training Knowing
foods; ability to make good how to
choices; take care offood
distinguish their and
Supermarket well-being; Understand the
its importance for health
importance of healthy habits and
Personal
Expression andand social training Independence/autonomy
Math Count objects, identifyhealthy
numbers foods; ability to make good
Supermarket communication
choices; distinguish food and its
Knowledge of Science—physical/natural importance for health
Health care and safety of the body
the world world
Expression and
Math Count objects, identify numbers
Dentist communication Understand the importance of rules, healthy habits and
Personal and Independence/autonomy;
Knowledge of the world personal hygiene,
Science—physical/natural worldsuchHealth
as washing
care andthe teeth
safety ofand going to
the body
social training identity and self-esteem
the dentist Understand the importance of rules,
Dentist Knowledge of Science—physical/natural
Independence/autonomy; healthy habits and personal hygiene,
Personal and social training Health care and safety of the body
the world world identity and self-esteem such as washing the teeth and going
Physical to the dentist
Understand the importance of rules, healthy and
Activity Personal and
Knowledge of the world
Independence/ Science—physical/natural
autonomy worldwhyHealth
personal hygiene; care and safety
it is important of the body
to exercise
social training
Physical Activity regularly Understand the importance of rules,
Personal
Knowledge of andScience—physical/natural
social training Independence/ autonomy
Health care and safetyhealthy and personal
of the body; hygiene; why it
understand
is important to exercise regularly
the world world phenomena and transformations in the natural world
Beach Healthfor care and well-being
safety of theandbody;
Personal and Make responsible decisions their
Knowledge of the world Science—physical/natural world
Independence/autonomy understand phenomena and
social training safety
Beach transformations in the natural world
Make responsible decisions for their
Personal and social
3.2.3. training
Citizen Kids Independence/autonomy
well-being and safety
Citizen Kids in the citizenship app, that addresses the subthemes of civility, multi-
culturalism and preservation of cultural heritage (Figure 3). It has five games: (1) Play
3.2.3. Citizen Kids children in the areas of the kindergarten classroom. In each area, there
Areas—distribute
Citizen
can only be aKids in the
certain citizenship
number app, that
of children (e.g.,addresses
dollhousethe
cansubthemes of civility,
have 4 characters, multi-
library—
culturalism and preservation
2, paintings—3). The player has of cultural heritage
to distribute (Figure 3). in
the characters It the
has areas,
five games:
taking (1) Play
into ac-
Areas—distribute
count their wisheschildren in the areas
that are shown of thebubbles.
in thought kindergarten
Whenclassroom. In many
there are too each area, there
characters
wanting to go to an area, they have to wait for their turn. They stay in the queue, until the
55
Educ. Sci. 2021, 11, 229

can only be a certain number of children (e.g., dollhouse can have 4 characters, library—2,
Educ. Sci. 2021, 11, x FOR PEER REVIEW 12 of 24
paintings—3). The player has to distribute the characters in the areas, taking into account
their wishes that are shown in thought bubbles. When there are too many characters
wanting to go to an area, they have to wait for their turn. They stay in the queue, until
the player
player removes
removes one one
thatthat
has has
beenbeen
therethere
for afor a long
long time.time. (2) Playground—distribute
(2) Playground—distribute char-
characters
acters in thein playground,
the playground, similarly
similarly to thetoprevious
the previousone, one, but outside.
but outside. The characters
The characters must
must be distributed
be distributed among among different
different spaces:
spaces: slide,
slide, sandbox,hopscotch,
sandbox, hopscotch,soccer.
soccer. (3)
(3) Behavior
Behavior
Board—a
Board—a board
board divided
divided intointo two
two sides.
sides. On
On thethe left
left side
side there
there isis the
the correct
correct symbol
symbol and and onon
the
the right
right side
side there
there isis the
the wrong
wrong symbol.
symbol. In In the
the game,
game, children
children explore
explore aa room
room where
where they
they
discover
discover notes
notes with
with correct
correct attitudes
attitudes (lift
(lift the
the finger
finger to to speak)
speak) and and wrong
wrong attitudes
attitudes (push,
(push,
throw tantrums) and they can build a board, choosing the correct
throw tantrums) and they can build a board, choosing the correct and wrong behaviors, and wrong behaviors,
with
with the
thenotes
notesthey
theyprefer.
prefer.(4) (4)Clothes
Clothesofof the World—dress
the World—dress dolls with
dolls clothes
with thatthat
clothes represent
repre-
different countries
sent different and customs
countries in the world.
and customs in the Players can choose
world. Players canachoose
countryaon the map
country onand
the
see
maptheandtypical costume.
see the typicalThen they can
costume. Then opentheythecan
closet
opento choose
the closetclothes and dress
to choose the doll
clothes and
according to theaccording
dress the doll costume or to otherwise,
the costume as or
they wish. When
otherwise, theywish.
as they open the
Whencloset
they theopen
clothes
the
change, to give them several options; (5) Preservation of Heritage—on
closet the clothes change, to give them several options; (5) Preservation of Heritage—on a a walk down the
street, the character
walk down the street,goes thethrough
characterplaces
goesthat he canplaces
through help preserve,
that he can for help
example, picking
preserve, for
up papers from the floor, washing a statue with a hose, fix a broken
example, picking up papers from the floor, washing a statue with a hose, fix a broken church-stained glass,
tearing wall posters.
church-stained glass, tearing wall posters.

(a) (b) (c)

3. Print screens of games from
Figure 3. from the
the app
app Citizen
Citizen Kids:
Kids: (a)
(a) Play
Play Areas;
Areas; (b)
(b) Clothes
Clothes of
of the
the World;
World;(c)
(c)Behavior
BehaviorBoard.
Board.

Citizen
Citizen Kids
Kidsappappaddresses
addressesthethe
CGPE,
CGPE,in the fieldfield
in the of personal and social
of personal training,
and social areas
training,
of democratic
areas coexistence
of democratic and citizenship,
coexistence exploring
and citizenship, contentscontents
exploring about civility
about and rulesand
civility of
living in living
rules of society,inmulticulturalism and diversity,
society, multiculturalism and solidarity
diversity,and preservation
solidarity of the natural
and preservation of
and culturaland
the natural heritage. Table
cultural 4 summarizes
heritage. Table 4the learning that
summarizes the can be promoted
learning that canbybethe Citizen
promoted
Kids
by theapp, according
Citizen to the
Kids app, CGPE. to the CGPE.
according

3.2.4. Busy Kids

Table 4. Citizen Kids games, content and learning.
Busy Kids is the profession awareness app that offers games about different jobs and
Game Content Arearoutines Knowledge
(Figure 4). Domain Learning,
It has five games. Attitudes, Know-How
(1) Postman—the That
character has to Can
sort Be Explored
mail, deliver
mail to addresses in a street, Respect for
following a the other,
map and solidarity,arrows
directional critical to
and
advance in the
Democratic
Personal and scenario. (2) Fashion Designer—the interventional
player attitude
chooses towards
fabric the cuts
patterns, world around
fabrics them,
according
coexistence/citizenship;
social training to shapes, decorates clothes with props;decisions
making (3) Builder—mix cement,
taking into accountlaythe
bricks, build walls
well-being of
Play Areas Independence/autonomy
according to different geometricthe shapes;
other (4) Office Worker—uses the computer, answers
Expression andthe phone, uses paper, pens, stamps and other
Identify officeidentify,
numbers, supplies.count
(5) Mechanic—changes
and locate objects; tires,
Math
paints cars, replaces headlights, among others.
communication group objects by characteristics
Respect for the other, solidarity, critical and
Democratic
Personal and interventional attitude towards the world around them,
coexistence/citizenship;
social training making decisions taking into account the well-being of
Playground Independence/autonomy
the other
Expression and Identify numbers, Identify, count and locate objects;
Math
communication group objects by characteristics
Identity/self-esteem; Awareness of belonging to a group and respect for rules
Behaviour Personal and
Democratic 56 and others members; ability to make choices and assume
Board social training
coexistence/citizenship responsibilities
Educ. Sci. 2021, 11, 229

Table 4. Citizen Kids games, content and learning.

Learning, Attitudes, Know-How

Game Content Area Knowledge Domain
That Can Be Explored
Respect for the other, solidarity,
Democratic critical and interventional attitude
Personal and social training coexistence/citizenship; towards the world around them,
Independence/autonomy making decisions taking into account
Play Areas
the well-being of the other
Identify numbers, identify, count and
Expression and
Math locate objects; group objects by
communication
characteristics
Respect for the other, solidarity,
Democratic critical and interventional attitude
Personal and social training coexistence/citizenship; towards the world around them,
Independence/autonomy making decisions taking into account
Playground the well-being of the other
Identify numbers, Identify, count and
Expression and
Math locate objects; group objects by
communication
characteristics
Awareness of belonging to a group
Identity/self-esteem;
and respect for rules and others
Personal and social training Democratic
members; ability to make choices and
Behaviour Board coexistence/citizenship
assume responsibilities
Expressions and Ability to recognize elements of
Visual Arts
Communication visual communication
Identify physical, social and cultural
aspects of the community and
Science—physical/natural
Knowledge of the world identify differences and similarities
world
with other communities; know and
respect cultural diversity
Clothes of the World
Know and value manifestations of
cultural heritage; respect diversity;
Identity/self-esteem;
accept their social and cultural
Personal and social training Democratic
identity, in relation to others;
coexistence/citizenship
recognize and value social and
cultural ties
Recognize central elements of the
community, highlighting physical,
Science—physical/natural
Knowledge of the world social and cultural aspects; identify
world; social world
cultural practices; identify objects and
Preservation of
their properties
Heritage
Know and value manifestations of
Independence/autonomy;
cultural heritage; recognize and value
Personal and social training Democratic
social and cultural ties, recognizing
coexistence/citizenship
the need for preservation.

Busy Kids app exposes children to contents in the content area of knowledge of the
world, suggested in the CGPE, such as knowing the community, life situations and cultural
practices, but also, addresses the expression and communication area. Table 5 summarizes
the learning that can be promoted by the Citizen Kids app, according to the CGPE.

57
 routines (Figure 4). It has five games. (1) Postman—the character has to sort mail, deliver
mail to addresses in a street, following a map and directional arrows to advance in the
scenario. (2) Fashion Designer—the player chooses fabric patterns, cuts fabrics according
to shapes, decorates clothes with props; (3) Builder—mix cement, lay bricks, build walls
Educ. Sci. 2021, 11, 229 according to different geometric shapes; (4) Office Worker—uses the computer, answers
the phone, uses paper, pens, stamps and other office supplies. (5) Mechanic—changes
tires, paints cars, replaces headlights, among others.

(a) (b) (c)

Figure 4. Print screens of games from the app Busy Kids: (a) Builder; (b) Office Worker;
Worker; (c)
(c) Postman.
Postman.

Busy Kids app exposes children to contents in the content area of knowledge of the
Table 5. Busy Kids games, content and learning.
world, suggested in the CGPE, such as knowing the community, life situations and cul-
tural practices, but also, addresses the expression and communication area.Know-How
Learning, Attitudes, Table 5 sum-
Game Content Area Knowledge Domain
marizes the learning that can be promoted by the Citizen KidsThat
app,Can
according to the CGPE.
Be Explored
Knowing the community; life
Science—physical/natural/ social
Knowledge of the world situations and cultural practices; jobs
world
and routines.
Postman Group objects by characteristics;
Expression and locate objects, using orientation
Math
communication concepts. Recognize locations and
maps. Follow an itinerary.
Knowing the community; life
situations and cultural practices; jobs
Knowledge of the world Science—Social world
and routines. Recognize properties of
different materials.
Fashion Designer
Recognize geometric figures; identify
Expression and patterns and symmetries. Develop
Visual Arts; Math
communication expressive/creative skills through
graphic compositions.
Knowing the community; life
Science—physical/natural/social situations and cultural practices; jobs
Knowledge of the world
world and routines; Recognize properties of
Builder different materials.
Recognize geometric shapes and
Expressions and
Math figures; identify patterns and
Communication
symmetries.
Knowing the community; life
Knowledge of the world Science—Social world situations and cultural practices; jobs
and routines.
Office Worker
Recognize and write number,
Expressions and
Math; Approach to writing; Recognize and write letters; Be aware
Communication
of text directionality.
Knowing the community; life
Knowledge of the world Science—Social world situations and cultural practices; jobs
Mechanic and routines.
Expressions and Develop expressive and creative skills
Visual Arts;
Communication through graphic compositions.

3.2.5. Common Approach, Multidisciplinary Content and Integrating Website

The apps are based on image (illustration, animation) and sound (music and sound
effects), which are suitable ways to communicate with children, regardless of their native
language and literacy level. Written text and audio narration were excluded. Scenarios are
58
Educ. Sci. 2021, 11, 229

filled with interactive and animated elements, to incite children to freely explore different
spots. Objects with small interactivity stimulate curiosity and discovery, as well as an
understanding of the world around them (e.g., touching a perched bird, the bird flies;
Touching a switch, the light turns on or off; Touch a tap, the water flows). These small
interactions allow the child to appropriate concepts and make sense of the world. In certain
games, there are concrete tasks. Other games have open rules, so children build freely and
progress according to their will and speed.
In addition to the knowledge and skills immediately associated with each game, there
are other learnings that are more subtly integrated to explore multidisciplinary, such as
mathematics. In the Forest game, the child has to count and find the baby animals that
parents had lost. In the Construction game, the player respects geometric shapes to create
a wall. In the Postman game, children interpret maps to orient the character and move
around. Other knowledge domains that are suggested in the CGPE can be explored with
the Apps. One it the introduction to scientific methodology, which proposes to carry out
activities that allow the child to question, hypothesize, predict responses, experiment,
collect, organize and analyze information and draw conclusions. This type of activity
can be promoted with the support of the educator, in the exploration of various games.
Examples: which foods are healthy? What happens if the boy does not put on sunscreen?
What do animals eat? How will the seed end up in fruit?
Additionally, the use of technologies is a learning area for CGPE—to recognize and
use technological equipment, with care and safety. This is promoted by using tablets to
play the apps, individually and autonomously, or in peer or group activities, with guidance
from the educator.
These apps do not promote competition among children, but cooperation, since they
all want to contribute to a better world, which is translated into a system of flowers in a
virtual world. When children play a game, a flower appears in a virtual world inside the
app, symbolizing their individual participation. But they can also send their contribution to
the project’s website, where all the flowers of all the children are represented. It symbolizes
their total participation in the global world (Figure 5).
This aggregator site gathers data on the use of apps and shows the contribution of all
users to a better world. Children can see the graphic representation of a world in which they
participate. Thus, children begin to learn that their actions have a personal impact and a
global impact, and they feel that they have contributed to a better future, which is extremely
important in environmental, citizenship, public health, economy and employment issues,
the main themes of the apps. On this website, parents and educators find educational
guides for each app, with the objectives of the games, how to play, conversation topics
before, during and after playing the games, and suggestions for educational activities to do
with the children.

3.3. Results of the Evaluation Phase

The results of the observation (children using the apps) and the interview (educator’s
perception) served to implement corrections and improvements in the final versions of the
apps, in order to make the products more efficient for educational activities in kindergarten
and to achieve a greater interest and involvement of the children. A pilot study report
details all the improvements made in each game, in the four apps [51]. For this article, the
results of each observation session are summarized, stating the main improvements made
in the final versions of the apps.

59
 ance from the educator.
These apps do not promote competition among children, but cooperation, since they
all want to contribute to a better world, which is translated into a system of flowers in a
virtual world. When children play a game, a flower appears in a virtual world inside the
Educ. Sci. 2021, 11, 229 app, symbolizing their individual participation. But they can also send their contribution
to the project’s website, where all the flowers of all the children are represented. It sym-
bolizes their total participation in the global world (Figure 5).

Figure
Figure 5.
5. Children’s
Children’s participation
participation in
in the
the Aprender
Aprender XXI
XXI World.
World.

This aggregator site gathers data on the use of apps and shows the contribution of all
3.3.1. Observation Session of the Nature Kids App
users to a better world. Children can see the graphic representation of a world in which
they In the ForestThus,
participate. gamechildren
(delivering
beginbaby
to animals to their
learn that their parents),
actions havesomea children
personal(C2, C7,
impact
C11, C18) found it difficult to catch the baby animals, as they were very small (squirrel
and a global impact, and they feel that they have contributed to a better future, which is
and rabbit). The animals were enlarged in the final version. C13 found two baby animals
extremely important in environmental, citizenship, public health, economy and employ-
and tried to pick them at the same time, which was not possible. It was also fixed. The
ment issues, the main themes of the apps. On this website, parents and educators find
game always started with the same animals and the children wanted to see them in another
educational guides for each app, with the objectives of the games, how to play, conversa-
order. It was necessary to make it random. In the Meteorology game (Figure 6) (dressing
tion topics before, during and after playing the games, and suggestions for educational
characters according to the weather) the gameplay had to be explained to the first children
activities to do with the children.
who played (C4, C6, C8). They did not understand how to change the climate and how
to see different clothes. This confirmed the need to implement graphic instructions and
3.3. Results of the Evaluation Phase
feedback to guide the action, for autonomous use by children. C20 found an error, which
The results
has been fixed. Aofcharacter
the observation
dressed(children
accordingusing
to thethe apps) did
weather andnot
the look
interview
happy.(educator’s
The game
perception)
Feeding the Farm Animals was not well understood. Children did not final
served to implement corrections and improvements in the versions
understand of
that
the apps, in order to make the products more efficient for educational activities
the animals were making specific requests, using speech bubble (meat, fish, water...) and in kinder-
garten
that theyand hadto to
achieve
satisfyathe
greater
needsinterest andat
requested involvement of the
that time. They children.
played A pilot
the game study
without
report
obeying details all the
the rules, improvements
feeding the animals made in each
as they game,
wanted. C11 in gave
the four appsmeat,
the dog [51]. then
For this
she
gave him
article, thewater
resultsbecause
of each he could be thirsty.
observation Thesummarized,
session are dog had made no request
stating the mainat improve-
all. After
the researcher
ments made inexplained the gameplay,
the final versions of the children
apps. found it difficult to satisfy the requests at
the right time. It was decided to remove this rule from the game. The objective changed
to feed
3.3.1. the animals,
Observation without
Session having
of the Nature to obey time or specific requests, but according to
Kids App
the animal’s diet (carnivore,
In the Forest herbivore,
game (delivering babygranivore). Graphic
animals to their feedback
parents), some was implemented,
children (C2, C7,
showing whether the animal liked it or not. The educator suggested that
C11, C18) found it difficult to catch the baby animals, as they were very small (squirrel the animals could
respond positively with a heart. The Plantations game also had to be
and rabbit). The animals were enlarged in the final version. C13 found two baby animalschanged. The game
had several actions—sowing/planting, watering, fertilizing, harvesting. Since the children
planted, the plantations started to grow and were ready to be harvested, regardless of
whether the children carried out the actions. Children took some time to do these actions.
C2 planted strawberries and harvested, without taking any intermediate action. The game
has been corrected to make some steps mandatory for plants to grow. The Recycling game
generated collaboration and communication. C5 and C6 played together, passing the tablet
to each other. Several children pointed out the right container and commented where each
type of garbage should be placed.

60
 be harvested, regardless of whether the children carried out the actions. Ch
some time to do these actions. C2 planted strawberries and harvested, withou
intermediate action. The game has been corrected to make some steps ma
plants to grow. The Recycling game generated collaboration and communicat
Educ. Sci. 2021, 11, 229
C6 played together, passing the tablet to each other. Several children pointed o
container and commented where each type of garbage should be placed.

Figure
Figure 6. Child playing
6. Child the Meteorology
playing game, educator
the Meteorology game,guiding, severalguiding,
educator classmatesseveral
watching.
classmates w
3.3.2. Observation Session of the Healthy Kids App
3.3.2. Observation
In the SupermarketSession ofwas
game, C22 thethe
Healthy Kids
first to play andApp
immediately understood the
game. InHethe
chose assorted food game,
Supermarket on the shelves
C22 was andthe
went to the
first to cashier
play andto pay. The cash
immediately und
register screen showed a sad face when the chosen foods were not healthy. C22 repeated
game.
the game,He chose
buying assorted
only food The
healthy foods. on rest
theofshelves andcommented,
the children went to the cashier
to help him to pa
register
choose thescreen showed
right food. C1 founda sad face
a bug, when
which wasthe chosen
fixed foods
in the final wereHenot
version. puthealthy.
the C
character
the game, of the game walking
buying in the opposite
only healthy foods.direction.
The restThe of
character left the screen
the children and
commented, t
did not appear again. In the Dentist game, C10 was the first to play. He did not realize
choose the right food. C1 found a bug, which was fixed in the final version.
that he had to clean all teeth in the exposed mouth. He received an explanation from the
characterand
researcher offollowed
the game the walking in the
rest of the steps opposite
to the direction.
end, without TheThe
any doubt. character
educator left the
felt that there was little distinction between step 2 (removing the caries and opening a
cavity in the tooth) to step 3 (putting mass on the tooth). The cavity and the mass looked
similar. In the final version, changes in shape and color were made in both steps. The
instruments were difficult to use on the lower teeth, even for five-year-old children. It has
been corrected by increasing the touch area on the screen. In Creating Healthy Snacks
game, C12 found it very fun to create a giant sandwich and a drink with all the elements
available. Then, the educator told her to create a healthy sandwich. Other children gave
suggestions and commented. The game was well accepted, with no doubts or need for
changes. The Beach game was well understood. C8 found an error. He was unable to give
the character water to drink. He dragged the bottle to his mouth, but nothing happened,
and the character continued to show thirst (Figure 7). It was necessary to increase the
touch area of the mouth. The Physical Activity game did not attract so much interest or
any interaction with each other. Children entered the garage, chose a prop (skateboard,
bicycle, rollerblades) for the character, used it outdoors, in the garden, and left the game.

61
 changes. The Beach game was well understood. C8 found an error. He was un
the character water to drink. He dragged the bottle to his mouth, but nothing
and the character continued to show thirst (Figure 7). It was necessary to
touch area of the mouth. The Physical Activity game did not attract so much
Educ. Sci. 2021, 11, 229
any interaction with each other. Children entered the garage, chose a prop (
bicycle, rollerblades) for the character, used it outdoors, in the garden, and le

Figure
Figure7. Detail of child
7. Detail of playing the Beachthe
child playing game, tryinggame,
Beach to givetrying
water toto
thegive
character.
water to the character.
3.3.3. Observation Session of the Citizen Kids App
3.3.3.
TheObservation Session
games Play Areas of the Citizen
and Playground (Figure 8)Kids
were App
highly requested, being played
by eightTheandgames
nine children,
Play respectively.
Areas and InPlayground
both games, children
(Figurewanted
8) were to move the reque
highly
characters to different areas and see if they were happy. The other children watched and
played by eight and nine children, respectively. In both games, children wan
made comments that indicated they understood the games. Some comments: C3 (while
the charactersare
watching)—“you toalways
different areas
moving them and see if C4
around!”; they were
(while happy. The other
playing)—“because they childr
want
andto”;madeC12 (while
commentswatching)—“Otherwise,
that indicatedthey they willunderstood
be sad!”; C3—“But they haveSome
the games. to com
wait”. In certain interactions, children dragged a character
(while watching)—“you are always moving them around!”; C4 (while pla to an area, but it did not stay
there. This was fixed in the final version, in both games. The game Board of Behaviors
cause they want to”; C12 (while watching)—“Otherwise, they will be sad!”; C3
was not well understood, even by the five-year-old children. It was necessary to explain
have to wait”.
everything, step byIn certain
step. interactions,
They did children
not realize that the board dragged
had one a character
area to an are
for the right
behaviors
not stayand another
there. Thisforwas
the wrong
fixedbehaviors.
in the finalIn the final version,
version, it was
in both necessary
games. Thetogame B
change the graphics to better delineate the different areas. C8 played the game under the
haviors was not well understood, even by the five-year-old children. It was n
guidance of the researcher and did not like it. C14 also needed guidance, but he filled the
explain
board everything,
to the step by to
end. It was necessary step. Theythedid
improve notfor
game, realize that the
autonomous use,board
insertinghad one
right behaviors
graphical explanations andandanother
feedbacks.forThethe wrong
game behaviors.
Preservation In the
of heritage wasfinal version, it wa
understood
and played six times by children aged three, four and five. The
to change the graphics to better delineate the different areas. C8 played game has a path with tasks, the g
which were carried out by children without the need for explanation. After finishing the
the guidance of the researcher and did not like it. C14 also needed guidance,
game, C19 made the course again without stopping, trying to be the fastest. The others
the board interested.
commented, to the end. It was
It was necessary
possible tothat
to notice improve the game,
they already fortasks
knew the autonomous
(e.g., u
C12—“Next,
graphical you have to cleanand
explanations the posters on the wall”).
feedbacks. The game In thePreservation
game Clothes ofof theheritage
World, was
C13 and C15 had difficulty dragging shoes and glasses. The
and played six times by children aged three, four and five. The game has need to increase the size of a path
these elements was noted and corrected in the final version. They also did not realize that
they could choose another country by touching the map, nor change the clothes inside the
closet, by closing and opening its doors. It was necessary to add visual clues.

62
 commented, interested. It was possible to notice that they already knew the tasks (e.g
C12—“Next, you have to clean the posters on the wall”). In the game Clothes of the World
C13 and C15 had difficulty dragging shoes and glasses. The need to increase the size o
these elements was noted and corrected in the final version. They also did not realize tha
Educ. Sci. 2021, 11, 229
they could choose another country by touching the map, nor change the clothes inside th
closet, by closing and opening its doors. It was necessary to add visual clues.

Figure8.8.Child
Figure Childplaying autonomously
playing the game
autonomously Playground.
the game FriendsFriends
Playground. watching and commenting.
watching and commenting.

3.3.4. Observation Session of the Busy Kids App

3.3.4. Observation Session of the Busy Kids App
The most chosen game of all apps was the Builder game, played by 15 children
(Figure The
9). most
Some chosen game ofinallcollaboration.
played together, apps was the Builder game,
Three-year-old C2 played
was the by first15tochildren
play. (Fig
ure
As he9).
hadSome
doubts,played together,explained
the researcher in collaboration.
the game and Three-year-old
all the childrenC2 was the
listened. first to play. A
However,
some difficulties have arisen. The interaction for making cement consisted of touching theHoweve
he had doubts, the researcher explained the game and all the children listened.
materials and touching
some difficulties havethe machine’s
arisen. switch to turn
The interaction it on. Seven
for making children
cement tried to
consisted ofdrag
touching th
the materials into the machine and nothing happened. In the final version,
materials and touching the machine’s switch to turn it on. Seven children tried to drag th the interaction
was changed,
materials intofrom
the touching
machineto anddragging,
nothingtohappened.
be more intuitive. Theyversion,
In the final also did thenot interaction
realize wa
that they had to press the switch to start the cement machine. The
changed, from touching to dragging, to be more intuitive. They also did not realize tha size of the switch and
the luminous graphic feedback were increased. In the second part of the game, they had
they had to press the switch to start the cement machine. The size of the switch and th
to build a wall, but five children did not take the trowel to put the cement. It has been
luminousbygraphic
corrected feedback
highlighting were in
the trowel increased.
the instrumentIn themenu.
second partplacing
After of thethe game,
cementthey had t
and brick, four children took the hammer and broke the brick, not understanding thebeen cor
build a wall, but five children did not take the trowel to put the cement. It has
rected by
function highlighting
of the hammer. It was the trowel
necessary in to
the instrument
separate menu. After
the construction placingfrom
instruments the cement
the an
destruction
brick, fourinstruments
children took in the menu.
the hammerAs theand
walls couldthe
broke have different
brick, shapes, the educator
not understanding the functio
considered
of the hammer. that thisItwas
wasa good game to
necessary to learn
separatemathematics. The gameinstruments
the construction Fashion Designer from the de
was not spontaneously
struction instruments chosen
in thebymenu.
any child. C18walls
As the volunteered
could at the end,
have to experiment.
different shapes, the Theeducato
researcher explained the first step—choosing a fabric. From there,
considered that this was a good game to learn mathematics. The game Fashion Designe he played alone, without
difficulty, understanding all the possibilities. In the Office Worker game, C15 was the first
was not spontaneously chosen by any child. C18 volunteered at the end, to experimen
to play. She thought the goal was to write on the computer that appeared in the game. It
The researcher explained the first step—choosing a fabric. From there, he played alone
was explained to her that she could explore other office supplies (phone, pad, pens, among
withoutThe
others). difficulty, understanding
game’s keyboard all the possibilities.
was simplified, In thevowels,
it only contained Office Worker
space key game,
and C15 wa
the first
emojis. The toeducator
play. She thoughtplacing
suggested the goal was tokeyboard
a virtual write onwith the all
computer that
the letters, appeared
so that the in th
children could practice writing. She also considered that the phone was useful for children
game. It was explained to her that she could explore other office supplies (phone, pad
to trainamong
pens, their parents’
others).contact numbers.
The game’s The change
keyboard wastosimplified,
the keyboard was not
it only implemented,
contained vowels, spac
due
keytoandlackemojis.
of spaceTheon the game screen.
educator The Postman
suggested placing game also keyboard
a virtual had to be explained.
with all the Theletters, s
first part, separating the types of letters and packages, was apprehended
that the children could practice writing. She also considered that the phone was immediately. Theuseful fo
second part, delivery mail following a map and arrows, was more difficult to understand
and play. However, the gameplay remained, because it trains important skills, such as
orientation, interpretation of maps and the sense of laterality. The Mechanic game was
easily understood, although there were doubts about the tools for fixing dents, filling tires

63
 immediately. The second part, delivery mail following a map and arrows, was m
ficult to understand and play. However, the gameplay remained, because it tr
Educ. Sci. 2021, 11, 229 portant skills, such as orientation, interpretation of maps and the sense of latera
Mechanic game was easily understood, although there were doubts about the
fixing dents, filling tires and painting. It was necessary to change the illustrations
items.
and Three-year-old
painting. C16tospent
It was necessary changefour minutes decorating
the illustrations theThree-year-old
of these items. car, worried about
C16 spent four minutes decorating the car, worried about the aesthetic.
thetic.

Figure
Figure Children
9. 9. collaborating,
Children playingplaying
collaborating, together the Builderthe
together game, educator
Builder helping.
game, educator helping.
3.3.5. Interview with the Kindergarten Educator
3.3.5. Interview with the Kindergarten Educator
The interview with the educator aimed to obtain information to characterize the
contextThewhere the testswith
interview took theplace and to evaluate
educator aimed to theobtain
apps, as to relevance to
information (content
characterize
validity),
text where consistency
the tests(construct
took placevalidity)
and andtoexpected
evaluate practicality,
the apps,that asis,
totorelevance
understand(content
if v
the educator would use the product and identify obstacles to its use [35].
consistency (construct validity) and expected practicality, that is, to understand i
Regarding the context, the educator already carried out technological activities with
ucator
the would
children, such use the product
as watching videos on and identify
YouTube obstacles
® , searching to its use
on Google [35]. images.
® , printing

In the Regarding
kindergartenthe context,
classroom, theythehad
educator
a computer, already
wherecarried
childrenoutcouldtechnological
write letters activi
or draw pictures. The educator also promoted digital
the children, such as watching videos on YouTube , searching on Google communication.
® Children used ®, prin
Messenger ® in kindergarten to send emojis to their parents. The educator used Skype® to
ages. In the kindergarten classroom, they had a computer, where children cou
make video calls with the children, when they were sick at home. They did not have a tablet
letters or draw pictures. The educator also promoted digital communication. C
in the kindergarten room, but the educator recognized that it was the most appropriate
used
and Messenger
preferred
® in kindergarten to send emojis to their parents. The educa
device. Regarding the expectation of use, the educator did not know many
learning
Skype® apps to makeand thought
video calls that with
the Aprender XXI apps
the children, whenweretheyin accordance
were sickwith the They
at home.
interests of the children and the thematic areas they usually
have a tablet in the kindergarten room, but the educator recognized that it waswork on. The educator liked
all the apps and gave concrete suggestions in some games. One suggestion was for games
appropriate and preferred device. Regarding the expectation of use, the educator
where the buttons are integrated into the scenario. A greater distinction should be made
know many
between buttonslearning apps andand
that have interaction thought
scenery that
propsthe thatAprender XXI apps
are only illustrative. were
It can be in acc
awith
visualthecueinterests of the
on the buttons children animation)
(brightness, and the thematic areas
or, after some they
time usually
without action,work
a on. T
help
catorcould
liked appear, such
all the as an
apps andarrowgaveindicating
concrete a path. The educator
suggestions also had
in some suggestions
games. One sugges
about the structure and content of the educational guides for
for games where the buttons are integrated into the scenario. A greater distinctionparents and educators. It
could be a guide per game. Each guide should have an introduction about the game, goals
be made between buttons that have interaction and scenery props that are only
and gameplay, so that the educator understands it first and then can explain the game to
tive.
the It can guide
children, be a visual
the usecue andon planthe buttonsRegarding
activities. (brightness, animation)
the advantages, or,
the after some tim
educator
said that this type of activity is very stimulating for children,
out action, a help could appear, such as an arrow indicating a path. The educatorpleases them and transmits
knowledge
suggestions in the waythe
about theystructure
like. It also andpromotes
content communication
of the educational and collaboration
guides forinparents
a joint activity. As for the constraints, she indicated lack of time, the size of the group
ucators. It could be a guide per game. Each guide should have an introduction a
and the lack of equipment. This is a very large group (22 children) to give personalized
game, goals
attention in thisand
typegameplay,
of activity, whichso that the educator
is something that understands
all the children it firsttoand
want do atthen can
thesame
the game to the
time. children,
Referring to the guide theshe
devices, usesaid
and plan
that dueactivities. Regarding
to the results the advant
of this activity,
educator said that this type of activity is very stimulating for children, pleases th
transmits knowledge in the way they like. It also promotes communication and
64
ration in a joint activity. As for the constraints, she indicated lack of time, the si
group and the lack of equipment. This is a very large group (22 children) to give
alized attention in this type of activity, which is something that all the children
Educ. Sci. 2021, 11, 229

she would ask the institution or even the parents, to purchase tablets for the classroom’s
educational activities.
After the evaluation with users, changes were made to the games and the final versions
of the apps were completed. The project ended with the release of the four apps on the
App Store® (iOS 8.0) and Google Play® (Android 4.0.3), the online publication of the
website with the virtual world that flourishes with the participation of children in apps,
and educational guides for parents and educators.

4. Discussion
Following the DBR methodology, the project Aprender XXI was divided into three
phases: preliminary study, development and evaluation. The preliminary study served
as a theoretical framework for contextualizing and supporting the project, and to identify
the existing needs that helped define the objectives and content of the apps. The search for
educational apps suitable for these ages allowed to know the “state of the art” and under-
stand the existing offer in the market that can be used in an educational context, as well as
to identify limitations and opportunities for development. To end this phase, Portuguese
curriculum guidelines for pre-school education were studied, along with guidelines from
other countries. The analysis of data collected in the preliminary study was essential in
the development phase—definition of specifications, selection of contents, scriptwriting,
design and programing of the games. The theoretical knowledge was integrated in a
practical way in the apps, such as opting for open games (e.g., Office Worker) or games
that allowed to manipulate several variables (e.g., Meteorology, Mechanic), because they
have a greater cognitive and creative contribution, and promote group interaction and
communication [23,25]. Games were planned to allow children to experiment different
scenarios, to solve challenges (e.g., Builder, Healthy Snacks), to respect rules (e.g., Behavior
Board, Playground, Play Areas), providing variety and flexibility of learning [28]. Design
decisions considered the literature: use of illustrations and animations that provide context;
have a simple interface and interactivity that is easy to learn and use; image-based feedback
related to actions [24]. The search for apps in the market showed a lack of apps suitable for
educational activities in kindergarten, according to CGPE and in Portuguese language. The
content chosen was based on the areas suggested in pre-school education guidelines [29],
adapted to games that could be used by educators in a multidisciplinary way, combining,
for example, mathematics, geography, citizenship and environment [21].
The kindergarten sessions were important to test the products with the public (children
and educator), to verify the need to change the course of development or to continue the
implementation according to the plan. The results of the evaluation showed that the apps
met the interests of the children, were pedagogically useful and provided a new educational
tool for the educator. In this way, the evaluation validated the options that had been taken
in the development phase. There were also evaluation results that added knowledge and
led to improvements in the games. It became clear the importance of making interactive
elements more visible, so lighting effects or animation were added to the buttons feedback,
and some objects or outlines were increased. Visual instructions were introduced in all
games to guide children in their autonomous use. Some games had specific corrections, as
a result of observing their use by the children. Minor bugs were also detected and fixed.
These corrections were made, before the release of the final versions of the apps in the
Stores, that were linked to the website, with the flower system to visualize the cooperation
and participation of children.

5. Conclusions
This project aimed to develop game-based learning apps for preschoolers, using the
DBR methodology to support technological development with scientific research in the area
of technologies in education. The information collected in the three phases (preliminary
study, development and evaluation), the involvement of the target audience (educator
and children) and the triangulation with theoretical studies and existing apps guided the

65
Educ. Sci. 2021, 11, 229

technological development, resulting in robust products and contributions to the theory.

Summarizing some principles of this study:
• CGPE is a guiding document that presents areas of content and learning for children
in kindergarten, reiterating the importance of playing as a primary form of learning.
• The areas of expression and communication, knowledge of the world and personal
and social training are divided into domains of knowledge, with learning potential.
These content areas were included in the apps, to meet the needs of pedagogical
activities in the kindergarten, respecting the CGPE.
• Extensive literature documents the benefits of using technology in children’s learning,
in different content areas and knowledge domains. In Portugal, there are limitations
in the apps available on the market, regarding the learning promoted for these ages.
• In the literature, there are recommendations for the creation of multimedia resources
for children, such as the use of graphics and actions that provide context; simple in-
structions based on images; intuitive interface and interactivity; challenging approach
with multiple opportunities for success; clear and understandable feedback. Digital
games are a very appealing type of multimedia content for children.
• These recommendations were followed in the development of apps and their relevance
was confirmed in tests with users in kindergarten. The tests also made it possible to
find and correct bugs, identify improvements to be developed in the final versions of
the apps and validate their educational value with the educator.
Some limitations to the study are related to characteristics of the DBR methodology
and to the development context, a micro company where the researcher is also a project
manager. It was necessary to involve and combine several people with different profiles
and rhythms—researchers, technological team and target audience [52]. Scientific research
requires time to collect and analyze data at each stage. The involvement of the educator
and children is conditioned by timetables, school calendars and pedagogical plans, so it
was necessary to adjust the research to these restrictions to achieve their participation. The
company’s team had reduced availability, because it had several projects with tasks and
deadlines running at the same time. This resulted in a limitation of the project, which
was the impossibility of involving the target audience in all phases, particularly in the
preliminary study, to survey the needs. It would be interesting in a later version, to
include questionnaires or focus groups involving the target audience from the beginning.
Another limitation is the difficulty in generalizing the results, since, in DBR, the sample of
participants is not representative of reality, but rather small and intentional [32]. Thus, the
products are tested with small groups of users and released on the market. Subsequently,
with continued use and new data, they evolve to improved versions. Thus, the design
principles presented in this article follow the recommendations of several authors [32,33,53],
to be contextualized, so that they can be interpreted, adapted and contribute to knowledge.
It would be interesting to repeat the evaluation in other kindergarten classrooms to increase
the degree of generalization.
In terms of impact for end users, the results of the evaluation indicate that the products
have the potential to promote the learning of pre-school children, since the games have
contents covered in kindergarten, the children showed interest in playing the games and
the educator identified possible educational activities to do with apps. A more in-depth
case study on the use of apps in kindergarten would be interesting to measure learning
outcomes and suggest an implementation plan in the classroom, with a certain number of
sessions, duration and learning objectives.
As a final note, the researcher considers that the project Aprender XXI is an example of
good practice, for a research project developed in a business context in the area of education
technologies. The project represented a business effort to reinforce internal R&D skills to
create knowledge and greater competitiveness. It stimulated technological experimentation
and scientific dissemination led by an SME. The results (four apps in two stores) were able
to generate value for the company and integrate an initial internationalization strategy.

66
Educ. Sci. 2021, 11, 229

Funding: This research was funded by the European Union through the European Regional Devel-
opment Fund, the 2020 Center Program, under the project “Aprender XXI: Game-based m-learning
for children in kindergarten”, grant number CENTRO-01-0247-FEDER-009828).
Institutional Review Board Statement: Ethical approval was not sought for the present study be-
cause no more than minimal risk was identified. This study meets the necessary ethical requirements
and does not include activities or results that pose safety problems for the participants. Data were
properly anonymized and informed consent was obtained at the time of original data collection. Data
storage meets current Data Protection regulations.
Informed Consent Statement: Informed consent was obtained from all subjects involved in the
study, with the assent of children and permission of their parents.
Data Availability Statement: Data are contained in supplementary material, available in open
access on Zenodo [43,51]. The following supplementary materials are available online: website
(http://aprenderxxi.criamagin.com) (accessed on 5 April 2021); App Nature Kids (https://itunes.
apple.com/us/app/nature-kids/id1459394172 (accessed on 5 April 2021) e https://play.google.
com/store/apps/details?id=com.criamagin.naturakids (accessed on 5 April 2021)); App Healthy
Kids (https://itunes.apple.com/us/app/healthy-kids/id1459395328 (accessed on 5 April 2021) e
https://play.google.com/store/apps/details?id=com.criamagin.healthykids (accessed on 5 April
2021)), App Citizen Kids (https://itunes.apple.com/us/app/citizen-kids/id1459383445 (accessed on
5 April 2021) e https://play.google.com/store/apps/details?id=com.criamagin.citizenkids (accessed
on 5 April 2021)); App Busy Kids (https://itunes.apple.com/us/app/busy-kids/id1459400995 (ac-
cessed on 5 April 2021) e https://play.google.com/store/apps/details?id=com.criamagin.busykids
(accessed on 5 April 2021)).
Acknowledgments: The author thank the reviewers for their insightful comments.
Conflicts of Interest: The author declares no conflict of interest. The funders had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or
in the decision to publish the results.

References
1. Prensky, M. Digital Natives, Digital Immigrants Part 1. Horizon 2001, 9, 1–6. [CrossRef]
2. Thomas, M. Deconstructing Digital Natives: Young People, Technology, and the New Literacies; Routledge: New York, NY, USA, 2011.
3. Hattie, J.; Yates, G. Your Students Are Digital Natives. Or Are They? In Visible Learning and the Science of How We Learn; Routledge:
New York, NY, USA, 2013.
4. Chaudron, S. Young Children (0–8) and Digital Technology: A Qualitative Exploratory Study across Seven Countries; EUR 27052 EN;
ISPRA, Joint Research Centre: Ispra, Italy, 2015.
5. Marsh, J.; Kontovourki, S.; Tafa, E.; Salomaa, S. Developing Digital Literacy in Early Years Settings: Professional Development
Needs for Practitioners. White Paper Cost Action IS1410. 2017. Available online: http://digilitey.eu/wp-content/uploads/2017
/01/WG2-LR-jan-2017-1.pdf (accessed on 10 April 2021).
6. Cordes, C.; Miller, E. Fool’s Gold: A Critical Look at Computers at Childhood; ED 445 803; Alliance for Childhood: College Park, MD,
USA, 2000.
7. Cuban, L. Cyberteaching in Preschools and Kidergartens. In Oversold and Underused Computer in the Classroom; Harvard University
Press: Cambridge, MA, USA, 2001.
8. Mayer, R. Learning with Technology. In The Nature of Learning Using Research to Inspire Practice; Dumont, H., Instance, D.,
Benavides, F., Eds.; OECD Centre for Educational Research and Innovation: Paris, France, 2010; pp. 179–196.
9. NAEYC. Fred Rogers Center for Early Learning and Children’s Media. Technology and Interactive Media as Tools in Early
Childhood Programs Serving Children from Birth through Age 8. In Joint Position Statement; NAEYC: Washington, DC, USA, 2012.
10. Chassiakos, Y.; Radesky, J.; Christakis, D.; Moreno, M.; Cross, C.; Council on Communications and Media. Children and
Adolescents and Digital Media. Pediatrics 2016, 138, e20162593.
11. Burnett, C. Technology and Literacy in Early Childhood Educational Settings: A Review of Research. J. Early Child. Lit. 2010, 10,
247–270. [CrossRef]
12. Hsin, C.; Li, M.; Tsai, C. The Influence of Young Children’s Use of Technology on Their Learning: A Review. Educ. Technol. Soc.
2014, 17, 85–99.
13. Lieberman, D.; Bates, C.; So, J. Young Children’s Learning with Digital Media. Comput. Sch. 2009, 26, 271–283. [CrossRef]
14. Amante, L. As TIC Na Escola e No Jardim de Infância: Motivos e Factores Para a Sua Integração. Sísifo Rev. Ciênc. Educ. 2007,
3, 51–64.

67
Educ. Sci. 2021, 11, 229

15. Amante, L.; Faria, Á. Sentido(s) Emergente(s) Das Tecnologias Digitais No Jardim de Infância. In Ensinar e Aprender Online Com
Tecnologias Digitais: Abordagens Teóricas e Metodológicas; Moreira, J.A., Monteiro, A., Eds.; Porto Editora: Porto, Portugal, 2012;
pp. 45–64.
16. Ramos, A. Crianças, Tecnologias e Aprendizagem: Contributo Para Uma Teoria Substantiva. Ph.D. Thesis, Universidade do
Minho, Braga, Portugal, 2005.
17. Clements, D. Young Children and Technology; ERIC: Washington, DC, USA, 1998.
18. Ramos, A. Crianças, Tecnologias e Aprendizagem: Contributos Para Uma Teoria Substantiva. A Minha Tese de Doutoramento
. . . Mais de Dez Anos Depois. In Pesquisar; Chiado Editora: Lisboa, Portugal, 2018; pp. 23–52.
19. Common Sense Media. The Common-Sense Census: Media Use by Kids Age Zero to Eight; Common Sense Media: San Francisco, CA,
USA, 2017.
20. Ponte, C.; Simões, J.; Batista, S.; Jorge, A.; Castro, T. Crescendo Entre Ecrãs: Usos de Meios Eletrónicos Por Crianças (3–8 Anos);
ERC—Entidade Reguladora da Comunicação Social: Lisboa, Portugal, 2017.
21. Gardner, H.; Davis, K. The App. Generation: How Today’s Youth Navigate Identity, Intimacy, and Imagination in a Digital World; Yale
University Press: New HAvem, CT, USA, 2013.
22. Clarke, B.; Svanaes, S. An Updated Literature Review on the Use of Tablets in Education; Tablets for Schools; Family Kids & Youth:
London, UK, 2014.
23. Goodwin, K.; Highfield, K. ITouch and ILearn: An Examination of “Educational” Apps. In Proceedings of the Early Education
and Technology for Children Conference, Salt Lake City, UT, USA, 14–16 March 2012; pp. 14–16.
24. Clements, D.; Sarama, J. Teaching with Computers in Early Childhood Education: Strategies and Professional Development. J.
Early Child. Teach. Educ. 2002, 23, 215–226. [CrossRef]
25. Hillman, M.; Marshall, J. Evaluation of Digital Media for Emergent Literacy. Comput. Sch. 2009, 26, 256–270. [CrossRef]
26. Papert, S. The Connected Family: Bridging the Digital Generation Gap; Longstreet Press: Atlanta, GA, USA, 1996.
27. Prensky, M. Digital Game-Based Learning. Comput. Entertain. 2003, 1, 21–25. [CrossRef]
28. Gros, B. Digital Games in Education: The Design of Games-Based Learning Environments. J. Res. Technol. Educ. 2007, 40, 23–38.
[CrossRef]
29. Silva, I.; Marques, L.; Mata, L.; Rosa, M. Orientações Curriculares Para a Educação Pré-Escolar; Ministério da Educação: Lisboa,
Portugal, 2016.
30. van den Akker, J.; Gravemeijer, K.; Mckenney, S.; Nieveen, N. Introducing Educational Design Research. In Educational Design
Research; Routledge: New York, NY, USA, 2006; pp. 3–7.
31. Brown, A. Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom
Settings. J. Learn. Sci. 1992, 2, 141–178. [CrossRef]
32. van den Akker, J. Principles and Methods of Development Research. In Design Approaches and Tools in Education and Training; van
den Akker, J., Branch, R., Gustafson, K., Nieveen, N., Plomp, T., Eds.; Springer Science & Business Media: Berlin/Heidelberg,
Germany, 1999; pp. 1–22.
33. Plomp, T. Educational Design Research: An Introduction. In Educational Design Research; Plomp, T., Nieveen, N., Eds.; Institute
for Curriculum Development: Enschede, The Netherlands, 2013; pp. 10–51.
34. The Design-Based Research Collective. Design-Based Research: An Emerging Paradigm for Educational Inquiry. Educ. Res. 2003,
32, 5–8. [CrossRef]
35. Nieveen, N.; Folmer, E. Formative Evaluation in Educational Design Research. Des. Res. 2013, 153, 152–169.
36. Reeves, T. Enhancing the Worth of Instructional Technology Research through “design Experiments” and Other Development
Research Strategies. In International Perspectives on Instructional Technology Research for the 21st Century; Annual Meeting of the
American Educational Research Association: New Orleans, LA, USA, 2000.
37. Garrett, J. The Elements of User Experience: User-Centered Design for the Web and Beyond, 2nd ed.; Pearson Education: London,
UK, 2003.
38. Sova, D.; Nielsen, J. 234 Tips and Tricks for Recruiting Users as Participants in Usability Studies; Nielsen Norman Group: Fremont,
CA, USA, 2003; p. 190.
39. Creswell, J. Collecting Qualitative Data. In Educational Research: Planning, Conducting and Evaluating Quantitative and Qualitative
Research; Pearson Education: London, UK, 2012.
40. Cohen, L.; Manion, L.; Morrison, K. Research Methods in Education; Routledge: London, UK, 2007.
41. Fraenkel, J.; Wallen, N. Observation and Interviewing. In Design and Evaluate Research in Education; McGraw-Hill Higher
Education: New York, NY, USA, 2009; pp. 439–470.
42. Bogdan, R.; Biklen, S. Trabalho de Campo: O Contínuo Participante/Observador. In Investigação Qualitativa em Educação: Uma
Introdução à Teoria e aos Métodos; Porto Editora: Porto, Portugal, 1994; pp. 125–133.
43. Laranjeiro, D.; Laranjeiro, J.; Mouta, L.; Zagalo, N. Relatório de Estudos Preliminares do Projeto Aprender XXI; Criamagin: Aveiro,
Portugal, 2017. [CrossRef]
44. Couse, L.; Chen, D. A Tablet Computer for Young Children? Exploring Its Viability for Early Childhood Education. J. Res. Technol.
Educ. 2010, 43, 75–96. [CrossRef]
45. Sandvik, M.; SmØrdal, O.; Østerud, S. Exploring IPads in Practitioners’ Repertoires for Language Learning and Literacy Practices
in Kindergarten. Nord. J. Digit. Lit. 2012, 7, 204–221. [CrossRef]

68
Educ. Sci. 2021, 11, 229

46. Zaranis, N.; Kalogiannakis, M.; Kontovourki, S. Using Mobile Devices for Teaching Realistic Mathematics in Kindergarten
Education. Creat. Educ. 2013, 4, 1–10. [CrossRef]
47. Dominguez, A. IPads Na #sala5; Universidade de Lisboa, Instituto de Educação: Lisboa, Portugal, 2013.
48. Pereira, A.; Moraes, A. Possibilidades De Uso Do Tablet Na Educação Infantil. Rev. Semin. Midias E Educ. 2015, 1, 423–426.
49. Zagalo, N.; Laranjeiro, D. Brinquedos e Jogos Digitais Para o Jardim de Infância. In Atas do 4.o Encontro sobre Jogos e Mobile
Learning; Centro de Estudos Interdisciplinares do Século XX: Coimbra, Portugal, 2018; pp. 479–489.
50. European Commission/EACEA/Eurydice/Eurostat. Key Data on Early Childhood Education and Care in Europe; Eurydice and
Eurostat Report 2014 Edition; Publications Office of the European Union: Luxembourg, 2014.
51. Laranjeiro, D. Relatório Do Estudo Piloto—Projeto Aprender XXI; Criamagin: Aveiro, Portugal, 2019. [CrossRef]
52. Kelly, A.; Baek, J.; Lesh, R.; Bannan-Ritland, B. Enabling Innovations in Education and Systematizing Their Impact. In Handbook of
Design Research Methods in Education—Innovations in Science, Technology, Engineering and Mathematics Learning and Teaching; Kelly,
A., Baek, J., Eds.; Routledge: New York, NY, USA, 2008; pp. 3–19.
53. Bannan-Ritland, B. The Role of Design in Research: The Integrative Learning Design Framework. Educ. Res. 2003, 32, 21–24.
[CrossRef]

69
 education
sciences
Article
Mobile App for Science Education: Designing the
Learning Approach
Rita Tavares 1, * , Rui Marques Vieira 1 and Luís Pedro 2

1 CIDTFF, Department of Education and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal;
[email protected]
2 DigiMedia, Department of Communication and Arts, University of Aveiro, 3810-193 Aveiro, Portugal;
[email protected]
* Correspondence: [email protected]

Abstract: This paper reports research work related to a wider study, aimed at developing a mobile
app for Science Education in primary-school. Several studies reveal that Science Education can be
improved by using technology, namely educational software. However, to promote a structured
use of technology, innovative learning approaches must be designed for educational software. This
paper aims to answer how the interaction between students and a mobile app for Science Education
can promote students’ scientific competences development and self-regulated learning. To achieve
this, a learning approach was designed, combining the Universal Design for Learning principles,
Inquiry-Based Science Education and the BSCS 5E – teaching model for Science Education designed
by the Biological Sciences Curriculum Study, which results in the acronym of the model. The 5E is
related to each phase of the model: Engagement; Exploration; Explanation; Elaboration; Evalua-
tion. The proposed was based on a grounded, participatory, and user-centred approach, crossing
literature contributions with data collected among primary-school teachers through the application



of a questionnaire (n = 118). Data collected allowed deductions about the expected adequacy of


the learning approach, according to Nieveen’s criteria for high quality educational interventions.
Citation: Tavares, R.; Marques Vieira, This adequacy was revealed through the teachers’ conceptions about the potential impact of the
R.; Pedro, L. Mobile App for Science conceptualized mobile app (i) to provide a comprehensive and practical Science Education learning;
Education: Designing the Learning
and (ii) to enhance students’ scientific competences development and self-regulated learning. The
Approach. Educ. Sci. 2021, 11, 79.
paper aims to contribute to the design of an innovative learning approach in Science Education and
https://doi.org/10.3390/
to share it with other researchers since it can be expanded to other educational software.
educsci11020079

Keywords: mobile application; Science Education; learning approach; scientific competences; Uni-
Academic Editor: Diego Vergara
Received: 3 December 2020
versal Design for Learning; Inquiry-Based Science Education; BSCS 5E; Educational Data Mining
Accepted: 1 February 2021
Published: 18 February 2021

Publisher’s Note: MDPI stays neutral 1. Introduction

with regard to jurisdictional claims in The importance of Science Education is increasing within a more democratic, informed
published maps and institutional affil- and enabled society, faced with the great challenges that the technological world brings
iations. to new generations [1–7]. For this reason, Science Education is advocated from the early
years, especially in primary schools. Some of the main reasons pointed out for this are its
potential (i) to help students to develop (new) ideas and to “make sense of the world”, from
what they hear and see in their daily lives; (ii) to promote opportunities for the clarification
Copyright: © 2021 by the authors. of students’ (pre-)concepts and to confront these with scientific evidence; (iii) to avoid
Licensee MDPI, Basel, Switzerland. students’ belated conceptual change, by testing their ideas through scientific experiences;
This article is an open access article and (iv) to promote and enhance positive attitudes towards Science, extending and/or
distributed under the terms and amplifying the number of students who will pursue scientific careers, and, among others,
conditions of the Creative Commons decreasing the problem of the growing lack of girls’ interest in Science [2,3,6,8,9]. In this
Attribution (CC BY) license (https:// regard, almost all European countries propose as main objectives (a) to improve students’
creativecommons.org/licenses/by/ understanding regarding the application of Science in real life; and (b) to strengthen Science
4.0/).

71
Educ. Sci. 2021, 11, 79

Education in schools in order to increase recruitment in areas such as Mathematics and

Technology [4].
For the last 30 years, several authors and organizations have highlighted the impor-
tance of a deeper understanding of the potential and/or effectiveness of the different
Science Education learning approaches, for instance, in students’ scientific competences
development [10,11]. Furthermore, in the last decade, many European countries have been
promoting several actions and efforts to integrate Science Education in a more contextual-
ized way, e.g., by supporting the development and maintenance of international networks
and databases to share and provide free (digital) educational resources and practices [10].
Despite these initiatives, Science Education is far from assuming the same importance as
such disciplines as mother tongue and Mathematics in primary school [4].
In European countries Science Education is taught as one general integrated subject
in primary-school, based on the broad acceptance that in real life knowledge and practice
are not split. This approach highlights the integration and iteration between knowledge
and practice, between theory and action, purporting a meaningful and contextualized
knowledge construction practice. This comprehensive approach relates scientific concepts
to other disciplinary contents/subjects and can help the students to develop logical rea-
soning, critical thinking, and an integrated and extended knowledge of reality [11]. This
can also potentially enhance students’ interest, stimulating new ideas, questions, and the
understanding of (new) complementary concepts based on personal experience and/or
real situations.
Several studies reveal that Science Education can be improved by using technol-
ogy, namely personal computers, smartphones, tablets and different types of educational
software [12–19]. The increasing usage of technological devices is an international trend,
underlined in several reports of the development and usage of digital educational resources
such as mobile applications (mobile apps) [17,20–22]. In the last OECD report related to
innovation in education, the importance of these devices in Science Education is underlined,
namely to enhance the development of students’ content and procedural knowledge [23].
In line with the above mentioned, this paper reports research work related to a wider
study aimed at developing a mobile app for Science Education in primary-school. One
comprehensive research question has been designed: Which type of mobile app can promote
primary-school students’ scientific competences development and self-regulated learning? To an-
swer this, eight additional questions have oriented the wider study. The present paper is
focused on one of those eight additional questions, related to the interaction between the
students and the mobile app and how that can promote students’ scientific competences
development and self-regulated learning: How can students interact with the mobile app and
how can the mobile app respond in real time to students’ interactions, simulating the teaching and
learning process and promoting the students’ scientific competences development and self-regulated
learning? To answer this question, a learning approach was designed, combining the
Universal Design for Learning (UDL) principles, Inquiry-Based Science Education (IBSE)
and the BSCS 5E (5E) – teaching model for Science Education designed by the Biological
Sciences Curriculum Study, which results in the acronym of the model. The 5E is related
to each phase of the model: Engagement; Exploration; Explanation; Elaboration; Evalua-
tion. This approach was designed and validated by crossing literature contributions with
data collected among primary-school teachers through the application of a questionnaire
(n = 118). Among other aspects not reported in this paper, the questionnaire application
allowed deductions about the expected adequacy of the proposed learning approach.
Our study aimed (i) to contribute to the design of innovative learning approaches [3,5,23]
by combining the UDL principles, the IBSE and the 5E; and (ii) to contribute to (research in)
Science Education by proposing a learning approach that can be expanded and/or applied
to other digital educational solutions besides mobile apps, aiming at (a) to facilitate ap-
proaches to scientific concepts/topics/phenomena; and (b) to promote students’ scientific
competences development: scientific knowledge, skills and attitudes, with particular focus
on self-regulated learning.

72
Educ. Sci. 2021, 11, 79

Because our research was developed in Portugal, based on the most recent published
data, our study also aimed to contribute to enhance and promote innovative practices in
Portuguese primary schools, regarding the following reported aspects [23]:
• Technology can promote students’ scientific knowledge development, its application
and deepening, promoting understanding of scientific concepts and procedures.
• Portugal is one of the three countries with a great percentage of students having access
to computers and tablets and using them for educational purposes.
• Despite this, the use of technology for practicing skills and procedures in Portuguese
4th grade Science lessons has been declining.
• There is great potential in the use of computers and tablets in Science Education,
including learning through games, simulations, and real time assessment.
• The use of simulations by 4th grade science students (9 to 10 years old) remains uncommon.
In the following sections, the authors present how the study was implemented
(Section 2) and how it can contribute to the above aspects, namely by proposing an in-
novative literature-based learning approach (Section 3.1); and presenting the expected
adequacy of the integration of the proposed digital educational resources in the mobile app
(Section 3.2); the potential impact of the conceptualized mobile app (Section 3.3); and the
expected adequacyof the proposed learning approach in promoting scientific competences
development (Section 3.4).

2. Materials and Methods

Since the wider study aimed at the development of a mobile app according to the
future end-users’ needs and expectations, a participatory and user-centred design approach
was adopted. At the same time, the mobile app development was research-based.
For this mixed approach, a research plan was conceptualized involving (i) scientific
knowledge deepening; (ii) the collaboration between researchers, experts and future end-
users; (iii) mixed methods; and (iv) interactive, cyclic and flexible phases of analysis, design,
development, implementation, evaluation and revision.
This research plan resulted in a participatory framework proposal nested within the
larger framework of Educational Design Research [24]. Educational Design Research revealed
to be the most adequate methodological approach to the study rationale since it [25,26]:
• encompasses interactive and iterative phases: Preliminary research, Prototyping phase,
and Assessment phase;
• comprises systematic and flexible processes of analysis, design, development, imple-
mentation, evaluation and revision;
• uses mixed methods;
• involves different participants in the study;
• aims to solve educational problems through scientific knowledge deepening and the
development of educational solutions;
In line with the above, the study fits in with this methodological approach by:
• being developed according to the three previously mentioned phases;
• foreseeing different moments of analysis, design, development, implementation, eval-
uation and revision;
• using different techniques and instruments for data collection and analysis (e.g.,
questionnaire, focus group, document analysis);
• involving the collaboration of researchers, experts and future end-users (primary-
school students and teachers) in different study phases/moments;
• resulting in different scientific products: frameworks, guidelines and the mobile
app prototypes.
The present paper reports on the Preliminary research focused on the analysis and
design of the proposed mobile app (see Figure 1), presenting and discussing a literature-
based learning approach proposal and data collected among primary-school teachers
through the application of a questionnaire (n = 118).
73
Educ. Sci. 2021, 11, 79

Figure 1. Conceptual scheme of the wider study.

Data collected with the questionnaire application was used in other moments of the
wider study (e.g., to define the mobile app’s target audience, i.e., 4th grade primary-school;
to define the topic to be approached in the mobile app, i.e., the Human Body). In the study,
data collected via the questionnaire allowed deductions about the expected adequacy of
the proposed learning approach and the potential impact of the conceptualized mobile app.
Since the mobile app’s learning approach is quite comprehensive, for its design four
correlated moments within the Preliminary research were considered:
• Analysis, definition and combination of the theoretical frameworks that sustain the
mobile app’s learning approach: UDL, IBSE and 5E;
• Definition of the types of digital educational resources and learning management
components integrated in the mobile app, and their relationship with the mobile app’s
learning approach rationale;
• Data collection, comprising the design, validation and application of the following
questionnaire: Primary-school teachers’ conceptions about their knowledge and their educa-
tional practices in Science Education using digital educational resources [27];
• Data analysis and discussion, with the production of the guideline Preliminary draft,
that allowed the definition of the proposed mobile app: (i) target audience – 4th grade
primary-school students; (ii) Science Education topic approached in the mobile app –
the Human Body; (iii) digital educational resources to integrate in the mobile app –
animations, games, simulations, quizzes and information areas; (iv) learning approach;
and (v) learning management components – formative feedback, recommendations
and real-time help triggered by the mobile app according to an Educational Data
Mining (EDM) framework developed for the mobile app that derives from the authors’
74
Educ. Sci. 2021, 11, 79

preliminary proposal of the conceptual EDM framework for Science Education –

Relational structure: questions and events that result in the conceptual EDM framework for
Science Education [28].
All the study phases were implemented in Portugal, the country where the authors
developed their research. For this reason, the questionnaire and the original data are only
available in Portuguese [27,29].

2.1. Participants
Since teachers represent one of the most legitimate and reliable sources of information
regarding the teaching and learning process, to ground and validate the proposed learning
approach the authors opted to involve teachers in the Preliminary research reported in
this paper.
Since the aim was to survey primary-school teachers’ conceptions about their knowl-
edge and their educational practices in Science Education using digital educational re-
sources, the study sample was a convenience sample [29,30]. This option was to assure
that most of the teachers answering the questionnaire used technological resources to teach
Science Education. The study’s population profile was the following: Primary-school teachers
that use and/or frequently interact with digital educational resources to teach Science Education.
To select our sample, the authors first surveyed teachers with the mentioned profile.
For this, Portuguese open access repositories dedicated to Science Education on the web
that included registered primary school teachers were searched. From the search, the
authors were able to find an open access repository in Portugal with these requirements:
House of Sciences – originally Casa das Ciências [10].
This repository delivers digital educational resources in subjects such as Introduction
to Science, Biology, Physics, Geology, Mathematics and Chemistry, allowing teachers
to upload, share, access and/or download the following digital educational resources:
animations, simulations, videos, interactive presentations, games, interactive whiteboard
resources, documents, and activities exploration guides. According to the House of Sciences
stakeholders, at the time of the survey the repository had 1046 primary-school teachers
registered. From those, 118 primary-school teachers answered the questionnaire.

2.2. Data Collection

The paper reports on the Preliminary research, focused on analysis of the mobile
app’s requirements and design, presenting and discussing a proposal for a literature-
based learning approach and data collected among primary-school teachers through the
application of a questionnaire (n = 118).
The questionnaire was a multidimensional and a priori structured instrument, crossing
four instruments validated and implemented in national and international settings. The
authors adapted and adopted ten items and conceptualized another six. For this, the
following stages were considered: (1) design of the pilot version of the questionnaire;
(2) validation of the pilot version (3) implementation of the pilot version; (4) analysis of the
data gathered; (5) design of the final version of the instrument; and (6) implementation of
the final version.
Regarding the adapted item, the following instruments were used: Avaliação do
Impacte Programa de Formação de Professores do 1.º Ciclo do Ensino Básico em Ensino
Experimental das Ciências nas práticas docentes de ensino experimental (Evaluation of the
Impact of the Primary-School Teachers Training Programme in Experimental Science Teach-
ing on the Teachers’ Practices) [31]; and TIMSS 2015 Grade 4 Teacher Questionnaire [32].
For the adopted items, the following instruments were considered: Self-Efficacy Teaching
and Knowledge Instrument for Science Teachers [33]; and Survey of Preservice Teach-
ers’ Knowledge of Teaching and Technology [34]. A formal consent was requested of the
instruments’ authors, both to translate and to apply the adopted items.
According to the proposed and adopted methodology presented in Section 2 and
Nieveen’s criteria [35], in the Preliminary research teachers’ conceptions provided as-

75
Educ. Sci. 2021, 11, 79

sumptions about the expected adequacy of the proposed learning approach regarding its
(a) consistency; (b) expected practicality; and (c) expected effectiveness (see Table 1).

Table 1. Nieveen’s criteria for high quality educational interventions [35] (p. 94).

Criterion
Relevance (also referred to as There is a need for the intervention and its design is based on
content validity) state-of-the-art (scientific) knowledge.
Consistency (also referred to as
The intervention is “logically” designed.
construct validity)
Expected: The intervention is expected to be usable in the
settings for which it has been designed and developed.
Practicality
Actual: The intervention is usable in the settings for which it
has been designed and developed.
Expected: Using the intervention is expected to result in
Effectiveness desired outcomes.
Actual: Using the intervention results in desired outcomes.

The questionnaire’s application was authorized and supported by House of Sciences

stakeholders, by sending an e-mail containing the invitation to participate in the study and
the hyperlink to the questionnaire to all the possible participants (N = 1046). In this way,
it was assured that only primary-school teachers registered in the repository participated
in the study, and allowed to control the number of e-mails read to reinforce the invitation
if needed.
The instrument was implemented using the University of Aveiro Questionnaires plat-
form [27] and it was available for 38 days. Besides the first e-mail containing the invitation
sent on 27 February 2017, the House of Sciences stakeholders sent two more e-mails to
reinforce the participation on March 13th 2017 and on April 3rd 2017.
The instrument was validated by five external experts and by the Portuguese School
Surveys Monitoring team – originally Monitorização de Inquéritos em Meio Escolar (MIME). The
preliminary pilot version of the instrument was sent to the external experts, who analysed
it with a qualitative approach. Based on their appreciation, a convergence analysis was
adopted [36]. Considering the consensus of comments, suggestions, points of view and
ideas pointed out by the experts, the final pilot version of the instrument was generated
and submitted for the MIME’s validation and approval to be implemented in a school
setting. Once approved, the questionnaire was implemented among a random sample of
primary school teachers, according to the defined profile (n = 17).
The final version of the questionnaire comprised nine questions (Q). On the first
page, the study and some clarifications about the instrument were presented, such as the
normal duration of time to fill it in (around eight minutes). The second page was related
to participants’ informed consent and agreement to participate in the study. The third
page comprised two questions, one Likert scale with eight items related to the teachers’
knowledge and educational practices in Science Education (Q1); and one dichotomous
“Yes/No” question asking the participants if they used digital educational resources to
teach Sciences (Q2). By answering “Yes” the participants proceeded with the remaining
questions (pages four and five), and by answering “No” the participants move to Q7
(page five).
The fourth page had four questions, all closed-ended one-choice or multiple-choice
questions: Q3 related to the frequency of usage of digital educational resources; Q4 related
to which school grades were most privileged to use them; Q5 related to the most used
digital educational resources; and Q6 related to the usage of digital educational resources
in Science Education (e.g., explore concepts/topics using games).
Finally, the fifth page comprised three questions. The first two were closed-ended
multiple-choice questions: Q7 related to the Experimental Science Education topics most
76
Educ. Sci. 2021, 11, 79

commonly explored by teachers, and Q8 related to the two topics most easily explored
using digital educational resources. The last question (Q9) was an open-answer question
related to the potential of the conceptualized mobile app.
Because the questionnaire was implemented in Portugal, the adopted items of the Q1
scale [36,37] were translated into Portuguese. To assure the correct version of the translation,
the items were submitted to a process of translation and back translation, ensured by two
external experts. To assure the internal consistency and reliability of the questionnaire, as
mentioned, a pilot version of the instrument was applied to 17 primary-school teachers,
allowing its validation. For both the pilot and the final version of the instrument, the
adequacy of the sample, the internal consistency, and the reliability was verified [37–45]
(see Table 2).

Table 2. Measures of the internal consistency and reliability of the questionnaire.

Cronbach’s α Pearson’s Coefficient

Knowledge 0.76
Pilot version 0.86 0.63 *
Educational practices 0.89
Knowledge 0.79
Final version 0.87 0.71 *
Educational practices 0.82
* p < 0.01.

In the present paper, to deduce the expected adequacy of the proposed learning
approach, revealed through the teachers’ conceptions about the potential impact of the con-
ceptualized mobile app (i) to provide comprehensive and practical Science Education learn-
ing; and (ii) to enhance students’ scientific competences development and self-regulated
learning, from the nine questions available in the questionnaire, data from the teachers’
answers to Q5, Q6, and Q9 were considered:
• Q5–From the following options, please select the digital educational resources you frequently
use in Science lessons using computers (including tablets).
• Q6–From the following options, please tell us how you use digital educational resources in
Science lessons?
• Q9–If you had a set of digital educational resources related to each other in a single mobile
app (e.g., an animation, a game, and a simulation related to liquid float), would you use it to
explore the areas mentioned above? Why?
By asking the participants if they use digital educational resources to teach Science
(Q2) the sample was split into two independent groups: group one–Primary-school teachers
that use digital educational resources to teach Science, and group two–Primary-school teachers
that do not use digital educational resources to teach Science. Q2 data analysis, verified that only
20.3% of the teachers did not use digital educational resources. This result confirmed the
adequacy of the convenience sample, and collected significant data from which to infer the
expected adequacy of the proposed learning approach.
From 230 answers to the questionnaire, 78 were incomplete and 34 were not properly
saved in the University of Aveiro Questionnaires platform due a system failure. For these
reasons, 118 complete and valid answers were considered, which will be analysed in the
following sections for the three questions.

2.3. Data Analysis

For Q5 and Q6–closed-ended multiple-choice questions–descriptive statistical analysis
was applied, using the software SPSS Statistics 24® (IBM Corporation, Armonk, NY, USA).
Descriptive statistical analysis was conducted to deduce (i) the expected adequacy of the
integration of the proposed digital educational resources in the mobile app; (ii) the potential
impact of the conceptualized mobile app; and (iii) the expected adequacy of the proposed
learning approach to promote scientific competences development.

77
Educ. Sci. 2021, 11, 79

For Q9–open-answer question–content analysis was conducted using the software

webQDA® (Universidade de Aveiro, CIDTFF–Research Centre on Didactics and Technology
in the Education of Trainers, Micro I/O, and Ludomedia). Content analysis was conducted
to deduce the potential impact of the conceptualized mobile app and its potential future
usage, namely regarding the expected adequacy of the proposed learning approach. For
this, a deductive system of categories was designed to analyse the teachers’ answers. The
system of categories was designed according to the “User Experience Honeycomb” [46],
reflecting the theoretical frameworks adopted (UDL, IBSE and 5E) and the proposed
learning approach (see Table A1–Appendix A).
The system of categories was framed in the content analysis software by creating
a project with the defined categories organized in a coding scheme, and with the data
sources: teachers’ answers to Q9 organized by identifiers (ID) [47]. The IDs were analysed
according to the categories. To assure the adequacy of the system of categories and the
coded IDs, clone versions of the project were submitted for experts’ validation. For this,
two external experts coded 10% of the coded ID selected randomly. Then, the coded ID
were crossed and the reliability (r) of the system of categories was calculated according
to the following [48]: r = (TaTa
+ Td)
, where TA represents the total of agreements and Td the
total of disagreements between the researchers’ coded ID and the experts. By applying this
validation approach, the system of categories’ reliability was verified (r = 0.73). Although
the value is low [44,45,49–51], according to DeVellis [51] in some Social Sciences studies,
namely those with a small sample such as the present one, reliability values from 0.6 can
be considered as acceptable. For this reason, r = 0.73 was considered acceptable to proceed
with the system of categories application.
To deduce the expected adequacy of the proposed learning approach, aspects related
to the User Experience –Valuable subcategory and the aspects related to the Scientific Com-
petences category were analysed, finding for both a total of 387 references. The original
references were translated into English, bearing in mind textual coherence. Although some
of the translated sentences suffered small adjustments, semantic, idiomatic, cultural, and
conceptual equivalence were preserved.

3. Results
3.1. Literature-Based Learning Approach Proposal
The proposed learning approach designed for the mobile app combines the Universal
Design for Learning (UDL) principles, the Inquiry-Based Science Education (IBSE) and
the BSCS 5E (5E). UDL is an educational framework focused on the development of
learning environments, designed to meet individual learning differences and to promote
and facilitate the learning process [52,53]. Therefore, UDL proposes that the curriculum
must be designed to promote equal learning opportunities, since information access is not
enough for students’ knowledge development [54]. Knowledge construction will depend
on several aspects, such as learning goals, teaching and learning approaches, educational
resources adopted and learning assessment methodology [53]. In this regard, UDL sets
three main principles presented as follow according to the study scope [53–55]:
• Promote multiple means of (information) representation–the “what” of learning: to
allow students to explore the same educational content in several ways;
• Promote multiple means of expression (and interaction with the information)–the
“how” of learning: to allow students to explore flexible alternatives for performance
and knowledge assessment;
• Promote multiple means of engagement–the “why” of learning: to allow students to
explore challenging ways to contact with difficult concepts/topics, helping them to
maintain interest and persistence in learning.
Regarding the theoretical frameworks adopted towards the teaching and learning
process simulated by the mobile app, in the last decade several authors have implemented
the IBSE approach according to the 5E inquiry curriculum model [1,56,57]. Both approaches

78
Educ. Sci. 2021, 11, 79

propose five highly related phases, adopted in the study according to two complementary
points of view:
(i) IBSE–Teachers’ point of view: teachers’ role in the learning process simulated by the
mobile app [28,55]
• Orientation phase: to stimulate students’ curiosity about a certain scientific
concept/topic;
• Conceptualization phase: to confront students’ (pre-)concepts and/or inquire
about them, and to promote the generation of new ideas/assumptions;
• Investigation phase: to lead the students to plan and apply investigation pro-
cesses (e.g., collect, analyse and interpret data to test the assumptions);
• Conclusion phase: to lead the students to draw conclusions by comparing/confronting
their (pre-)concepts with new evidence;
• Discussion phase: to confront students’ ideas and/or results, promoting a reflec-
tion and learning process (self-)evaluation - this phase is adopted as transversal
to the previous phases).
(ii) 5E–Students’ point of view: students’ learning process by interacting with the mobile
app [28,56]
• Engage phase: to stimulate students’ interest and promote their personal and
active learning involvement;
• Explore phase: to lead the students to build their own understanding about
concepts/topics, by confronting and experimenting with scientific phenomena;
• Explain phase: to promote the students’ opportunity to communicate their
knowledge/findings and to establish a theoretical framework;
• Elaborate phase: to lead the students to apply their (new) knowledge, deepening
scientific concepts/topics and/or proceeding towards new learning paths;
• Evaluate phase: to help students to develop self-awareness about their learning
path and about their knowledge construction (this phase is adopted as transversal
to the previous phases).
Besides the intrinsic relationship between both approaches found in the literature and
the proposed complementary points of view endorsed for each one, their simultaneous
adoption aims to address the importance of comprehensive and practical activities in the
development of scientific competences [2,58–60]. Thus, the proposed mobile app first allows
the students to contact (new) scientific concepts/topics and/or confront their previous
knowledge, hoping that such confrontation will allow them to become actively engaged in
reflective, exploratory and (self-)evaluative activities. For this, the mobile app integrates
five types of digital educational resources, organized according to learning sequences (the
mobile app games levels) and three learning management components.
Both the learning sequences and the learning management components are supported
by an Educational Data Mining (EDM) framework integrated in the mobile app, according
to the adopted methods and techniques. So that the mobile app can provide formative
feedback, recommendations and real-time help and tailored to the students’ needs, the
app will need to provide (in terms of computer programming) a structure that allows the
system (the mobile app) to read every single interaction between students and the mobile
app (and vice versa). For this, as presented in Figure 1, in the Preliminary research the
authors have also defined which questions should be “asked” of the system and which
events should be read and analysed through the EDM methods and techniques. The set of
questions and events resulted in the so-called Relational structure: questions and events that
result in the conceptual EDM framework for Science Education [28]. The Relational structure is
based on the learning approach proposed, relating it to the possibility of the mobile app
(i) to assess in real-time students’ performance levels; (ii) to identify in real-time difficulties
experienced by the students; and (iii) to guide students in real-time students along the
most adequate learning path [61].

79
Educ. Sci. 2021, 11, 79

The EDM framework and its influence on the availability of digital educational
resources and learning management components aims to simultaneously promote the
students’ development of scientific competences–scientific knowledge, skills and attitudes–
and self-regulation with regard to attitudes [1,62–67] (see Figure 2). Thus, the mobile app
includes several game levels related to different scientific concepts/topics. Each level
supports a learning sequence organized according to the following digital educational
resources: animation, game, simulation, quiz, and information areas. The integration of
these resources reflects the relationship between the UDL principles and the theoretical
frameworks adopted [68–77] (see Table 3).

Figure 2. Proposed learning approach scheme.

Table 3. Relationship between the five types of digital educational resources and the three learning management components
integrated in the mobile app with the UDL principles, the IBSE and the 5E approaches, and the student’s scientific competences.

UDL IBSE 5E Scientific Competences

Watch and explore (interactive) animations
To help the students to develop

As a means of scientific Orientation Engage - factual scientific knowledge

information (e.g., concept/topic-specific
- to stimulate students’ - to draw students’ attention/interest
representation, details)
curiosity about a par- - to involve students in a personal - scientific skills (e.g., iden-
enhancing the ticular concept/topic way
introduction and tify or formulate criteria to
- to promote students’ - to stimulate students to predict, re-
presentation of scientific draw possible answers)
self-evaluation about late and evaluate their previous
concepts/topics - attitudes (e.g., access avail-
previous knowledge knowledge
able help to solve a prob-
lem)

80
Educ. Sci. 2021, 11, 79

Table 3. Cont.

UDL IBSE 5E Scientific Competences

Explore games
Explore To help the students to develop
- to promote students’ active learn- - conceptual scientific knowl-
As a means of Conceptualization- to lead ing edge (e.g., classes, cate-
engagement (expression, the students to form - to stimulate students to analyse in- gories, principles, systems
and interaction with the assumptions- to test formation, observe and compare and scientific phenomena)
information), enhancing students’ assumptions phenomena, variables and concepts - scientific skills (e.g., decide
the exploration of according to the - to help students to identify require- (by attempting) the best ac-
scientific established dynamics ments and variables that influence tion/procedure)
concepts/topics through inquiring outcomes - attitudes (e.g., follow rec-
- to help students to interpret results ommendations for learning
- to stimulate students to draw and reinforcement and/or deep-
confront conclusions ening)
Explore simulations
Explain
- to stimulate students’ reflection To help the students to develop
about how they structure their con-
ceptual framework and the de- - procedural scientific knowl-
Investigation
As a means of signed research path edge (e.g., define and/or in-
expression (and - to lead the students terpret experimental proce-
- to help students to draw conclu-
interaction with the to form assumptions, dures)
sions and structure their knowledge - scientific skills (e.g., ob-
information), enhancing to plan processes,
to test assumptions, - to lead the students to confront
the application of serve scientific systems
and to collect, anal- their initial ideas with the results of and/or phenomenon varia-
scientific knowledge and the experimental activity
skills yse and interpret tions)
data - to help students to establish a theo- - attitudes (e.g., find alterna-
retical framework tives to validate the set cri-
- to help students to establish rela- teria)
tionships between their choices and
the initial research question
Answer quizzes
To help the students to develop
- conceptual scientific knowl-
Elaborate edge in order to deepen
- to promote students’ knowledge their knowledge (e.g.,
mobilization deepen scientific concepts
- to help students to discover and and/or specific details
understand the implications of the related to the concept/topic
phenomena explored addressed)
- to help students to establish - scientific skills (e.g., identify
relationships with other con- or formulate criteria for pos-
Conclusion sible answers)
cepts/topics
As a means of - to lead the students - attitudes (e.g., analyse state-
expression, enhancing to draw conclusions ments and (ir)relevant infor-
the deepening of - to help students to mation)
scientific knowledge and reflect about how To help the students to develop
skills they construct their
knowledge - conceptual scientific knowl-
Evaluate edge in order to assess
- to lead the students to evaluate knowledge (e.g., verify the
their understanding of a scientific domain of scientific con-
concept/topic cepts)
- to lead the students to apply their - scientific skills (e.g., inter-
(new) knowledge pret statements and answer
- to lead the students to deepen their questions)
- attitudes (e.g., use their
conceptual framework or advance
knowledge to analyse state-
towards new research paths
ments, seek relevant infor-
mation, and answer cor-
rectly)

81
Educ. Sci. 2021, 11, 79

Table 3. Cont.

UDL IBSE 5E Scientific Competences

Explore games
Access information areas/available help
To help the students to develop
As a means of scientific - conceptual scientific knowl-
information edge (e.g., deepen scientific
representation,
Conclusion & Elaborate phenomena)
enhancing the
- scientific skills (e.g., iden-
deepening of scientific - to lead the students to deepen/expand their knowledge tify necessary assumptions
knowledge and skills, - to help students to clarify doubts to understand scientific con-
and helping the students cepts/topics)
to proceed in their - attitudes (e.g., find ways to
learning path
be well informed about sci-
entific concepts/topics)
Read formative feedback and accept recommendations
Evaluate To promote students’
- to lead the students to constantly - reflection on knowledge
Discussion
and continuously be aware about construction (e.g., decide to
As a means of - to reinforce/deepen how much they have learned and access an information area
engagement, enhancing students’ knowledge how their conceptual framework in order to learn more about
the students’ interest - to help students to evolved a particular concept/topic
and persistence in self-regulate their - to help students to a greater un- and, thus, improve their
learning process learning (e.g., what derstanding of the scientific compe- performance)
content to explore) tences developed - self-awareness of their
- to help students to find ways of self- learning (e.g., performance
correction and readjustment level)

The integration of animations aims at the representation of scientific information,

enhancing the introduction and presentation of scientific concepts/topics. By integrating
games, the authors intend to provide means of engagement, expression, and interaction
with the information, enhancing the exploration of scientific concepts/topics. The simula-
tions intend to be a means of expression and interaction with the information, enhancing
the application of scientific knowledge and skills. By integrating the quizzes, the authors
aim to provide a means of knowledge expression, enhancing the deepening of scientific
knowledge and skills. Finally, by integrating information areas the authors intend to pro-
vide means of scientific information representation, enhancing the deepening of scientific
knowledge and skills, and helping the students to proceed in their learning path.
As referenced, besides the integration of digital educational resources, the mobile app
includes three components related to the learning management process: available help,
formative feedback, and recommendations. As mentioned, these components are related
to the authors’ proposal of an EDM framework for Science Education integrated into the
mobile app, aiming simultaneously at students’ development of scientific competences,
and at students’ self-regulation: reflection, (self-)evaluation and self-awareness [28]. The
focus on self-regulation aims that the students are able to (i) to identify personal interests
and learning needs; (ii) set learning objectives and pathways according to personal inter-
ests and needs; and (iii) search for personal skills consolidation and deepening learning
opportunities.
Regarding the IBSE and the 5E approaches, the mobile app game levels (the learning
sequences) and the learning management components allow the students to go through
the five phases of the adopted approaches by exploring scientific concepts/topics of in-
troduction, exploration, application and deepening activities. Each level is related to a
specific scientific concept/topic which means that the level related to “A” comprises one
animation, one game, one simulation, one quiz, and information areas related to “A”; the
level related to “B” comprises one animation, one game, one simulation, one quiz, and
information areas related to “B”; and so on.
According to the proposed learning approach scheme (see Figure 1) and the previous
examples, for instance, the students can begin to explore game “A”, continue to animation

82
Educ. Sci. 2021, 11, 79

“A”, browse an information area “A”, answer quiz “A” and explore the simulation “A2; or
the students can run each learning sequence: animation “A” → game “A” → simulation “A”
→ quiz “A” → information areas “A”. The mobile app is set to propose this last sequence
of exploration, so the student can go through the theoretical learning structure proposed.
However, to allow the students to set learning objectives and pathways according to
personal interests and needs, and to promote the students’ development of self-awareness
about their learning path and their knowledge construction (5E Evaluate phase), the mobile
app allows students to explore each learning sequence both linearly and non-linearly.
To ensure that all the phases are completed, students can only advance to the next level
(learning sequence) when they complete the previous one.
Thus, either exploring the mobile app linearly and/or non-linearly, students will
have the opportunity to develop scientific knowledge (factual, conceptual and procedural),
scientific skills and attitudes by exploring the five types of digital educational resource.
In addition and simultaneously, the three learning management components will ideally
help to promote the development of students’ scientific competences and self-regulated
learning, since the mobile app (i) gives the students the opportunity to choose what digital
educational resources are more suitable for their learning path, and so to personalize it;
and (ii) supports students’ digital educational resources exploration by giving them real
time formative feedback and recommendations, and identifying when they need help (e.g.,
to propose that the students access available help to solve a problem).
In the following section, data analysis from primary-school teachers’ answers to the
questionnaire (n = 118) detail the expected adequacy of the proposed learning approach.

3.2. Expected Adequacy of the Integration of the Proposed Digital Educational Resources in the
Mobile App
For Q5 and Q6 descriptive statistical analysis was applied, calculating percentages
according to the total sample.
As presented above, the proposed learning approach provides learning sequences–the
mobile app game levels. Each level has a set of five correlated digital educational resources:
an animation, a game, a simulation, a quiz and information areas. To deduce the expected
adequacy of the integration of these types of digital educational resources in the mobile
app, the teachers’ answers to Q5 were analysed: From the following options, please select the
digital educational resources you frequently use in Science lessons using computers (including
tablets). From the listed resources, the most frequently used by teachers in their Science
lessons were animations (68.6%) and games (55.9%) (see Figure 3).

Figure 3. Primary-school teachers’ conceptions about the digital educational resources most frequently used in their Science
lessons using computers (including tablets) (n = 118).

To deduce the expected adequacy of the usage of the listed digital educational re-
sources to introduce, explore, apply, and/or deepen scientific concepts/topics, data gath-
ered from Q6 was analysed: From the following options, please tell us how you use digital
educational resources in Science lessons? The analysis demonstrated that teachers used mainly

83
Educ. Sci. 2021, 11, 79

animations to introduce scientific concepts/topics (57.6%); simulations to explore scientific

concepts/topics (45.8%); games to apply scientific concepts/topics (52.5%); and simulations
to deepen scientific concepts/topics (39%) (see Figure 4.)

Figure 4. Primary-school teachers’ conceptions about how they use digital educational resources in Science lessons (n = 118).

Finally, the proposed integration of digital educational resources in the mobile app,
aiming at scientific concepts/topics’ introduction, exploration, application and deepening
according to UDL principles (see Table 3), was matched with the collected data (see Table 4).

Table 4. Proposed integration of the digital educational resources vs. Teachers’ conceptions about the expected adequacy of
the usage of the listed digital educational resources (n = 118).

Introduce Scientific Explore Scientific Apply Scientific Deepen Scientific

Concepts/Topics Concepts/Topics Concepts/Topics Concepts/Topics
Proposed digital educational Animations Games Simulations Quizzes Information
resources areas
Teachers’ conceptions about the
expected adequacy of the usage of
Animations (57.6%) Simulations (45.8%) Games (52.5%) Simulations (39%)
the listed digital educational
resources (%)

3.3. Potential Impact of the Conceptualized Mobile App

Regarding the User Experience –Valuable subcategory, data analysis demonstrated
that most of the teachers privileged a mobile app providing digital educational resources
correlated with the possibility of implementing Orientation and Engage phases, and to pro-
mote scientific information Representation (66 references registered). The teachers’ answers
revealed the following aspects related to the potential of the mobile app to:
• represent a means to obtain and/or to appeal to students’ interest, stimulating them
to learn (e.g., “to stimulate learning process”–ID 116);
• enhance students’ interaction with scientific contents/topics, promoting a more dy-
namic, meaningful and comprehensive learning process (e.g., “more appealing and
interactive, getting more easily the students’ attention and promoting a more mean-
ingful learning”–ID 188);
• facilitate information presentation and exploration (e.g., “allows to present/explore
concepts easily”–ID 219);
• enhance scientific concepts/topics observation and/or exploration, promoting oppor-
tunities for students to apply and/or evaluate their knowledge (e.g., “observe, inform,
apply and evaluate knowledge”–ID 57);
• facilitate a comprehensive, systematic, interdisciplinary and “hands-on” scientific
concepts/topics/phenomena approach (e.g., “global, interdisciplinary, and applied
vision of the phenomena”–ID 66);
• promote students’ motivation to learn more (e.g., “to enhance student’s motivation to
discover and learn more”–ID 170).
84
Educ. Sci. 2021, 11, 79

For the aspects related to the potential of the mobile app to implement Conceptual-
ization and Explore phases, and to promote Engagement, Expression, and interaction with
scientific concepts/topics, 29 references were registered in teachers’ answers. The references
emphasized aspects related to the potential of the mobile app to promote:
• students’ active and playful learning (e.g., “learn by doing”–ID 167);
• the development, consolidation and deepening of scientific knowledge (e.g., “to
materialize concepts, to consolidate knowledge and to promote a better perception of
reality”–ID 50);
• different approaches to apply scientific competences (e.g., “exploring and deepening
(scientific knowledge) using games and simulations–ID 39).
Analysing data for aspects related to the potential of the mobile app to implement
Investigation and Explain phases, and to promote multiple ways of Expression of scientific
competences and interaction with scientific concepts/topics, 39 references were registered
in teachers’ answers. The references emphasized aspects related to the potential of the
mobile app to help the students and/or the teachers:
• to manipulate scientific concepts, variables and/or phenomena with accuracy (e.g.,
“the mobile app provides accuracy and control of the variables”–ID 143);
• to view, demonstrate, materialize, simulate and/or experiment with scientific phe-
nomena (e.g., “phenomena visualization”–ID 105; “to simulate schemes”–ID 33);
• to compare scientific data and/or phenomena (e.g., “to compare and experiment
several phenomena”–ID 23);
• to mobilize scientific knowledge and skills (e.g., “allows to articulate concepts and
procedures easily, facilitating knowledge systematization and application”–ID 212);
• to learn in an active way (e.g., “it is via experimenting that one learns”–ID 213).
For the aspects related to the potential of the mobile app to implement Conclusion and
Elaborate phases, and to promote multiple ways of Expression of scientific competences,
20 references were registered in teachers’ answers. The references emphasized aspects
related to the potential of the mobile app to help the students:
• to apply scientific knowledge (e.g., “to apply knowledge”–ID 29);
• to consolidate and deepen scientific knowledge (e.g., “deepen their knowledge”–
ID 113);
• to proceed towards new learning paths (e.g., “prepares the student for future learning”–
ID 147).
Analysing data for aspects related to the potential of the mobile app to implement
Discussion and Evaluate phases, and to promote multiple ways of Engagement with scien-
tific concepts/topics, six references were registered in teachers’ answers. The references
emphasized aspects related to the potential of the mobile app to help the students:
• to deepen scientific concepts/topics understanding (e.g., “deepen complex scientific
topics understanding–ID 108);
• to discuss/compare ideas/evidence/results (e.g., “compare ( . . . ) several situations”–
ID 23);
• to reflect about topics/problems/challenges (e.g., “promotes a better understanding,
and prepares the students for ( . . . ) curiosities that may emerge during the research”–
ID 147);
• to evaluate their scientific knowledge development (e.g., “(the mobile app) would
allow to ( . . . ) evaluate the knowledge”–ID 57).

3.4. Expected Adequacy of the Proposed Learning Approach to Promote Scientific

Competences Development
Regarding the Scientific Competences category, data analysis demonstrated that most
of the teachers consider that the use of a mobile app integrating correlated digital educa-
tional resources could promote students’ Scientific Knowledge development (47 references

85
Educ. Sci. 2021, 11, 79

registered). The references emphasized aspects related to the potential of the mobile app
to promote:
• an organized, comprehensive, interdisciplinary and “hands-on” learning (e.g., “easy
articulation of the concepts and the procedures, so that knowledge can easily be
systematized and applied in practice”–ID 212);
• concepts/topics understanding (e.g., “to understand everyday life situations and
phenomena”–ID 192);
• scientific knowledge deepening (e.g., “students have the opportunity to deepen their
knowledge”–ID 113).
For aspects related to the potential of the mobile app to promote students’ Scientific
Skills development, 16 references were registered in teachers’ answers. The references
emphasized aspects related to the potential of the mobile app to promote:
• idea confrontation, observation analysis and/or discussion and knowledge application
(e.g., “encourage new knowledge application”–ID 102);
• the definition and/or operationalization of scientific strategies/experiments (e.g., “to
change the experiment variables”–ID 15).
Analysing data for the aspects related to the potential of the mobile app to promote
students’ Scientific Attitudes development, 40 references were registered in teachers’ an-
swers. The references emphasized aspects related to the potential of the mobile app to
enhance students’:
• curiosity, questioning and digging for more information (e.g., “a way to promote
students’ interest, developing an attitude of appreciation for science”–ID 170);
• involvement and maintaining motivation in learning process (e.g., “appealing to
students and promotes their involvement in learning”–ID 11);
• critical reflection, sympathy, and respect for others, for the environment and for objects
(e.g., “students will have a better perception of reality”–ID 35).

4. Discussion
Data analysis allowed inference about the expected adequacy of the proposed learning
approach and its potential to promote a comprehensive and practical approach in Science
Education, by allowing exploration of correlated digital educational resources, namely
animations, games, simulations, quizzes and information areas. Furthermore, the teachers’
answers also allowed inferences about the expected adequacy of the proposed learning
approach to promote the student’s scientific knowledge, skills and attitudes development.
First, the study demonstrated that most of the teachers adopting technology in their
Science lessons used mainly animations (68.6%) and games (55.9%) (see Figure 2). The
fact that only 38.1% of the teachers selected simulations from the options available in Q5
could be an indicator of the low availability of this typology in the Portuguese language
(develop and/or adapted), as well as the low availability of simulations for primary school
grades at the time of data gathering [78,79]. Since quizzes are a game typology, in Q5
and Q6 this typology was considered as a game option [72,80–83]. Regarding information
areas, since they are not very common, they were not presented in the questionnaire.
In this regard, it is important to refer to the fact that, within the Preliminary research,
between 1 September 2015 and 18 September 2017 a survey was performed a survey
related to the state of the art of mobile apps for Science Education, among others, (i) for
primary-school students; (ii) developed by Portuguese stakeholders; (iii) available in
Portuguese language; (iv) including (correlated) digital educational resources such as
animations, games, simulations, quizzes, and information areas. The survey found nine
mobile apps with those features, four from those from the same collection/stakeholder
providing information areas. Another piece of useful evidence is the fact that, despite
all the digital educational resources that House of Sciences provides, information areas
are not available. This means that the study sample was not familiar with this type of
digital educational resource. Instead, it was familiar with animations, simulations, videos,
86
Educ. Sci. 2021, 11, 79

interactive presentations, games, interactive whiteboard resources, documents and activity

exploration guides.
Regarding the use of digital educational resources in Science lessons, the study also
demonstrated that most of the teachers used animations and games to introduce and apply
scientific concepts/topics, respectively (see Figure 3). When matching teacher conceptions
with the leaning approach proposal, although most of the teachers used simulations to
explore and to deepen concepts/topics in their Science lessons, the verified frequency
of usage was below 50% of the total sample (see Table 4). Thus, games typology was
considered as a means to explore scientific concepts/topics, as proposed in the designed
learning approach, and quizzes and information areas to deepen scientific concepts/topics.
It is also important to mention that, at the time of data gathering, most of the available
simulations in the Portuguese language and for primary school grades were similar to
games [78,79]. In this regard, by choosing to maintain the proposed learning approach, the
mobile app, simultaneously, allows diversification of the typology of digital educational
resources available, and allows the students to use the most appropriate resources for
each one of the possibilities proposed: to introduce, explore, apply, and deepen scientific
concepts/topics.
Whether using technology in their Science lessons or not, data analysis demonstrated
that most of the teachers expressed that they privileged a mobile app integrating correlated
digital educational resources (i) to promote the students’ orientation and engagement in the
learning process, and (ii) to (re)present scientific information (see Section 3.3). Data analysis
also demonstrated that most of the teachers privileged the use of a mobile app like the
proposed one to promote students’ scientific knowledge and attitudes development (see
Section 3.4). Since references to all the defined categories and subcategories were found,
data analysis predicted the expected adequacy of the proposed learning approach regarding
the following: (i) integration of correlated digital educational resources in the mobile app;
(ii) adoption of the UDL principles; and (iii) the simultaneous adoption of the IBSE and 5E
theoretical frameworks from two related points of view: the teachers’ role in the learning
process simulated by the mobile app (IBSE) [28,84] and the students’ learning process when
interacting with the mobile app (5E) [30,62]; providing a comprehensive and practical
Science Education approach and facilitating students’ scientific competences development
and self-regulated learning. In this regard, the proposed learning approach revealed itself
to be adequate in providing a comprehensive and practical Science Education learning
tool, and enhancing students’ scientific competences development and self-regulated
learning [2,58–60].
Finally, by crossing the theorized aspects with data collected, the expected adequacy
of the proposed learning approach according to teachers’ conceptions could be deduced.
Therefore, the participatory and user-centred design approach allowed to confront and
ground the literature-based rationale of the proposed learning approach. Data analysis
and its impact detailed in the study aimed to share the foundations of the proposal with
other researchers and enhance the importance of the design and (preliminary) validation
of technological educational solutions among future end-users. Furthermore, the proposed
learning approach and the adopted method to design and demonstrate its potential could
represent a contribution to the development of innovative learning approaches in (Science)
Education, aiming at students’ engagement and helping them to deepen, understand and
develop (new) competences.
The study presented minor limitations. Besides the limitation mentioned in Section 2.2
related to the fact that from the 230 answers to the questionnaire, 78 were incomplete and
34 were not properly saved in the University of Aveiro Questionnaires platform due a system
failure, the authors had to wait 23 days for formal consent to adapt to Portuguese language
items of one of the instruments. Three e-mails had to be sent to reinforce the request. This
constraint resulted in a one-month delay in the following processes: (i) translation of the
adopted items; (ii) backtranslation of the adopted items; (iii) questionnaire pilot version

87
Educ. Sci. 2021, 11, 79

design; (iv) questionnaire pilot version validation; and (v) questionnaire pilot version
implementation in the University of Aveiro Questionnaires platform.
The authors support the idea that the learning approach proposal can contribute to
(research in) Science Education by proposing the combination of the UDL principles, the
IBSE and the 5E. From data analysis, the authors also support the idea that the proposed
approach can (a) facilitate scientific concepts/topics/phenomena approaches; (b) represent
an opportunity for students to explore scientific contents/topics in an organised, compre-
hensive, and practical approach; (c) promote students’ scientific competences development
(scientific knowledge, skills and attitudes); and (d) facilitate students’ self-regulated learn-
ing. Since the proposed learning approach is quite comprehensive, it could be expanded to
other educational software. With this paper, the authors also aim to contribute to Education
Science by providing a validated questionnaire aimed at surveying primary-school teachers’
conceptions about their knowledge and educational practices in Science Education using
digital educational resources.
With wider study, among other aspects, the authors aim at to contribute to Education
Science by (a) providing a participatory framework proposal for guiding researchers
through an educational mobile app development [24], representing an opportunity for
researchers in Education and Multimedia (in Education) to develop educational software
based not only on the state-of-the-art, literature in the area and their own rationales, but also
on the users’ perceptions, ideas and needs, and on experts’ validation; and (b) deepening
knowledge in the area of EDM and proposing a relational structure [28] and a conceptual
EDM framework developed for the mobile app, both extendable to other Science Education
software, representing a new perspective on technology enhanced learning and on how to
extract valuable educational data to guide students’ scientific competences development
and self-regulation of learning, as well as to help teachers and researchers to understand
and support students in those processes.
Future work is related to the mobile app development, testing and implementation in
a school setting, to investigate the adequacy of the learning approach, namely the actual
practicality and the actual effectiveness. By validating practicality, the authors intend to
understand if the mobile app is usable in the settings for which it has been designed and
developed. Finally, by validating its actual effectiveness, the authors intend to investigate
mobile app usage and relevance according to the desired outcomes: students’ scientific
development and self-regulated learning.

Author Contributions: Conceptualization, R.T.; Methodology, R.T.; Formal analysis, R.T.; Investiga-
tion, R.T.; Data curation, R.T.; Writing—original draft preparation, R.T.; Writing—review and editing,
R.T., R.M.V. and L.P. All authors have read and agreed to the published version of the manuscript.
Funding: This research was funded by Fundação para a Ciência e a Tecnologia – FCT I.P. – Portugal,
grant number PD/00173/2014 within the PhD Program Technology Enhanced Learning and Societal
Challenges; and SFRH/BD/107808/2015 under the Human Capital Operational Program, supported by
the European Social Fund and national funds of the Ministry of Science, Technology and Higher Education.
Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.
All the participants have been informed about the aim of this study and of the use of data collected.
Data Availability Statement: The anonymized data is available open access on figshare® [29]. This
study meets the necessary ethical requirements and does not include activities or results that pose
safety problems for the participants. All data were treated confidentially, and the participants were
treated anonymously in the study.
Acknowledgments: The authors thank the reviewers for their insightful comments.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or
in the decision to publish the results.

88
Educ. Sci. 2021, 11, 79

Appendix A

Table A1. Deductive System of Categories: potential and future usage of the mobile app regarding the expected adequacy
of the proposed learning approach.

Categories Subcategories Description

The visual aesthetics and the presentation of scientific concepts/topics/phenomena
Desirable
became more attractive using the mobile app
Scientific concepts/topics/phenomena are easily explored/manipulated using the
Usable
mobile app
Scientific concepts/topics/phenomena are easily organized/approached using the
Useful
mobile app
Orientation (IBSE): the mobile app aims to stimulate students’ curiosity about a
particular concept/topic; and to promote students’ self-evaluation about previous
knowledge.
Valuable: Orientation, Engage (5E): the mobile app aims to draw students’ attention/interest; to involve
Engage, and the students in a personal way; and to stimulate the students to predict, relate and
Representation evaluate their previous knowledge.
Representation (UDL): the mobile app promotes multiple ways of scientific
information representation, enhancing the introduction and presentation of scientific
concepts/topics (e.g., animations).
Conceptualization (IBSE): the mobile app aims to lead the students to form
assumptions; and to test students’ assumptions according to the established
dynamics through inquiring.
Explore (5E): the mobile app aims to promote students’ actively learning; to
Valuable: stimulate the students to analyse information, observe and compare phenomena,
Conceptualization, variables and concepts; to help the students to identify requirements and variables
Explore, and Engagement that influence outcomes; to help the students to interpret results; and to stimulate
the students to draw and confront conclusions.
Engagement (UDL): the mobile app promotes multiple ways of engagement,
expression and interaction, enhancing the exploration of scientific concepts/topics
(e.g., games).
Investigation (IBSE): the mobile app aims to lead the students to form assumptions,
to plan processes, to test assumptions, and to collect, analyse and interpret data.
Explain (5E): the mobile app aims to stimulate students’ reflection about how they
structure their conceptual framework and the designed research path; to help the
User Experience students to draw conclusions and structure their knowledge; to lead the students to
Potential/added value to Valuable: Investigation, confront their initial ideas with the results of the experimental activity; to help the
approach scientific
Explain, and Expression students to establish a theoretical framework about their meaning; and to help the
concepts/topics/phenomena
students to establish relationships between their choices and the initial research
[49,58,61,62] question.
Expression (UDL): the mobile app promotes multiple ways of expression and
interaction with the information, enhancing the application of scientific knowledge
and skills (e.g., simulations).
Conclusion (IBSE): the mobile app aims to lead the students to draw conclusions;
and to help the students to reflect about how they construct their knowledge.
Elaborate (5E): the mobile app aims to promote students’ knowledge mobilization;
to help the students to discover and understand the implications of the phenomena
explored; to help students to establish relationships with other concepts/topics; to
Valuable: Conclusion, lead the students to deepen/expand their knowledge; and to help the students to
Elaborate, Expression and clarify doubts.
Representation Expression (UDL): the mobile app promotes multiple ways of expression, enhancing
the deepening of scientific knowledge and skills (e.g., quizzes).
Representation (UDL): the mobile app promotes multiple ways of scientific
information representation, enhancing the deepening of scientific knowledge and
skills, and helping the students to proceed in their learning path (e.g., information
areas).
Discussion (IBSE): the mobile app aims to reinforce/deepen students’ knowledge;
and to help the students to self-regulate their learning (e.g., what content to explore).
Evaluate (5E): the mobile app aims to lead the students to evaluate their
understanding of a scientific concept/topic; to lead the students to apply their (new)
knowledge; to lead the students to deepen their conceptual framework or advance
Valuable: Discussion, towards new research paths; to lead the students to constantly and continuously be
Evaluate, and aware about how much they have learned and how their conceptual framework
Engagement evolved; to help the students to a greater understanding of the scientific
competences developed; to help the students to find ways of self-correction and
readjustment.
Engagement (UDL): the mobile app promotes multiple ways of engagement,
enhancing the students’ interest and persistence in learning process (e.g., formative
feedback, recommendations).

89
Educ. Sci. 2021, 11, 79

Table A1. Cont.

Categories Subcategories Description

Potential to promote scientific knowledge development: the app mobile aims to lead
the students to explore/contact with concepts; terminologies; concept/topic-specific
details; classes, categories, principles, systems and scientific phenomena; define
Scientific knowledge
and/or interpret experimental procedures; deepen scientific concepts and/or
specific details related to the concept/topic addressed; verify the domain of
scientific concepts; deepen scientific phenomena.
Potential to promote scientific skills development: the mobile app aims to lead the
Scientific Competences students to identify or formulate criteria to draw possible answers; decide (by
Potential/added value to attempting) about the best action/procedure; observe, analyse and/or interpret
promote scientific Scientific skills scientific systems and/or phenomenon variations; interpret statements and answer
competences development questions; analyse statements and (ir)relevant information; analyse and resume
[1,67–72]. ideas, statements, arguments; perform strategies and research plans; identify
necessary assumptions to understand scientific concepts/topics.
Potential to promote scientific attitudes development: the mobile app aims to lead
the students to access to more information to solve a problem; find alternatives to
validate the set criteria; mobilize knowledge to analyse statements, relevant
Scientific attitudes
information, and answer correctly; find ways to be well informed about scientific
concepts/topics; and to promote students’ reflection on knowledge construction;
self-awareness of their learning; and self-evaluation about previous knowledge.

References
1. Harlen, W. Assessment & Inquiry-Based Science Education: Issues in Policy and Practice; Global Network of Science Academies -
Science Education Programme: Trieste, Italy, 2013.
2. Harlen, W. Teaching Science for Understanding in Elementary and Middle Schools; Heinemann: Portsmouth, UK, 2015.
3. European Commission. Horizon 2020 > Science Education. Available online: https://ec.europa.eu/programmes/horizon2020/
en/h2020-section/science-education (accessed on 10 September 2019).
4. Forsthuber, B.; Horvath, A.; Almeida Coutinho, A.; Motiejūnaitė, A.; Baïdak, N. Science Education in Europe: National Policies,
Practices and Research; Eurydice—Education, Audiovisual and Culture Executive Agency: Brussels, Belgium, 2012. [CrossRef]
5. Nistor, A.; Angelopoulos, P.; Gras-Velazquez, A.; Grenon, M.; Mc Guinness, S.; Mitropoulou, D.; Ahmadi, M.; Coelho, M.J.; Greca,
I.M.; Kalambokis, E.; et al. STEM in Primary Education; Scientix Observatory: Brussels, Belgium, 2019.
6. Rocard, M.; Csermely, P.; Jorde, D.; Lenzen, D.; Walberg-Henriksson, H.; Valerie, H. Science Education Now: A Renewed Pedagogy for
the Future of Europe; European Commission—High Level Group on Science Education: Brussels, Belgium, 2007.
7. Tavares, R.; Almeida, P. Metodologia Inquiry Based Science Education No 1.o e 2.o CEB Com Recurso a Dispositivos Móveis –
Uma Revisão Crítica de Casos Práticos (Inquiry Based Science Education Methodology in 1st and 2nd Cycles of Basic Education
Using Mobile Devices - a Critical. Educ. Formação Tecnol. 2015, 8, 28–41.
8. Brotman, J.; Moore, F. Girls and Science: A Review of Four Themes in the Science Education Literature. J. Res. Sci. Teach. 2008,
45, 971–1002. [CrossRef]
9. Martins, I. Educação e Educação Em Ciências (Education and Science Education), 1st ed.; Universidade de Aveiro - Departamento
de Didáctica e Tecnologia Educativa, publicação financiada por Ciência Viva - Agência Nacional para a Cultura Científica e
Tecnológica: Aveiro, Portugal, 2002.
10. Tavares, R.; Moreira, A. Implications of Open Access Repositories Quality Criteria and Features for Teachers’ TPACK Development, 1st ed.;
Springer International Publishing: New York, NY, USA, 2017. [CrossRef]
11. Czerniak, C. Interdisciplinary Science Teaching. In Handbook of Research on Science Education; Lawrence Erlbaum and Associates:
Mahwah, NJ, USA, 2007; pp. 537–559. [CrossRef]
12. Sykes, B.E.R.; College, S. New Methods of Mobile Computing. TechTrends 2014, 58, 26–37. [CrossRef]
13. Teri, S.; Acai, A.; Griffith, D.; Mahmoud, Q.; Ma, D.W.L.; Newton, G. Student Use and Pedagogical Impact of a Mobile Learning
Application. Biochem. Mol. Biol. Educ. 2014, 42, 121–135. [CrossRef]
14. Kalogiannakis, M.; Papadakis, S. Gamification in Science Education: A Systematic Review of the Literature. Educ. Sci. 2021, 11, 22.
[CrossRef]
15. Stoyanova, D.; Kafadarova, N.; Stoyanova-Petrova, S. Enhancing Elementary Student Learning in Natural Sciences through
Mobile Augmented Reality Technology. Bulg. Chem. Commun. 2015, 47, 533–537.
16. Ekici, M.; Erdem, M. Developing Science Process Skills through Mobile Scientific Inquiry. Think. Skills Creat. 2020, 36, 1–12.
[CrossRef]
17. Sinclair, A. Mobile Education Landscape Report; GSMA Head Office: London, UK, 2011; p. 61.
18. Song, Y. “Bring Your Own Device (BYOD)” for Seamless Science Inquiry in a Primary School. Comput. Educ. 2014, 74, 50–60.
[CrossRef]
19. Felicia, P. Using Educational Games in the Classroom: Guidelines for Successful Learning Outcomes; Hertz, B., Pinzi, V., Sefen, M., Eds.;
European Schoolnet & EUN Partnership AISBL: Brussels, Belgium, 2020.

90
Educ. Sci. 2021, 11, 79

20. Adkins, S. The 2012-2017 Worldwide Mobile Learning Market - The Global Mobile Learning Market Is in a Boom Phase: Consumers and
Academic Buyers Dominate the Market; Ambient Insight Research: Monroe, MI, USA, 2013; p. 65.
21. MarketsandMarkets. Mobile Learning Market worth $37.60 Billion by 2020. Available online: https://www.marketsandmarkets.
com/PressReleases/mobile-learning.asp (accessed on 21 November 2020).
22. MarketsandMarkets. Mobile Learning Market by Solution (Mobile Content Authoring, E-books, Portable LMS, Mobile and
Video-based Courseware, Interactive Assessments, Content Development, M-Enablement), by Applications, by User Type, & by
Region - Global Forecast to 2020. Available online: https://www.marketsandmarkets.com/Market-Reports/mobile-learning-
market-73008174.html (accessed on 21 November 2020).
23. Vincent-Lancrin, S.; Urgel, J.; Kar, S.; Jacotin, G. Measuring Innovation in Education 2019: What Has Changed in the Classroom?
Educational Research and Innovation; OECD Publishing: Paris, France, 2019. [CrossRef]
24. Tavares, R.; Vieira, R.; Pedro, L. A Participatory Framework Proposal for Guiding Researchers through an Educational Mobile
App Development. Res. Learn. Technol. 2020, 28. [CrossRef]
25. McKenney, S.; Reeves, T. Conducting Educational Design Research; Routledge: New York, NY, USA, 2012.
26. Plomp, T. Educational Design Research: An Introduction. In Educational Design Research; Plomp, T., Nieveen, N., Eds.; SLO -
Netherlands Institute for Curriculum Development: Enschede, The Netherlands, 2013; pp. 10–51.
27. Tavares, R.; Vieira, R.; Pedro, L. Questionnaire: Primary-school teachers’ conceptions about their knowledge and their educational
practices in Science Education using digital educational resources. figshare 2019. [CrossRef]
28. Tavares, R.; Vieira, R.; Pedro, L. Preliminary Proposal of a Conceptual Educational Data Mining Framework for Science Education:
Scientific Competences Development and Self-Regulated Learning. In Proceedings of the 2017 International Symposium on
Computers in Education (SIIE), Lisbon, Portugal, 9–11 November 2017; Institute of Electrical and Electronics Engineers (IEEE):
Lisbon, Portugal, 2017; pp. 1–6. [CrossRef]
29. Saumure, K.; Given, L. Convenience Sample. In The SAGE Encyclopedia of Qualitative Research Methods; Given, L., Ed.; SAGE
Publications Ltd: Thousand Oaks, CA, USA, 2008; p. 125.
30. Lavrakas, P. Encyclopedia of Survey Research Methods; SAGE Publications Ltd: Thousand Oaks, CA, USA, 2008.
31. Martins, I.; Tenreiro-Vieira, C.; Vieira, R.; Sá, P.; Rodrigues, A.; Teixeira, F.; Couceiro, F.; Veiga, M.; Neves, C. Avaliação Do Impacte
Do Programa de Formação Em Ensino Experimental Das Ciências: Um Estudo de Âmbito Nacional - Relatório Final (Evaluation of the
Impact of the Training Programme in Experimental Science Teaching: A National Scope Study—Final Report); Ministério da Educação e
Ciência - Direção-Geral da Educação: Lisbon, Portugal, 2011; p. 334.
32. IEA. TIMSS 2015 Grade 4 Teacher Questionnaire; International Association for the Evaluation of Educational Achievement:
Washington, DC, USA, 2015; p. 24.
33. Roberts, K.; Henson, R. Self-Efficacy Teaching and Knowledge Instrument for Science Teachers: A Proposal for a New Efficacy
Instrument. In Annual Meeting of the Mid-South Educational Research Association; Mid-South Educational Research Association: OH,
USA, 2000; pp. 2–28.
34. Schmidt, D.; Baran, E.; Thompson, A.; Koehler, M.; Mishra, P.; Shin, T. Survey of Preservice Teachers’ Knowledge of Teaching and
Technology; Iowa State University: Ames, IA, USA; Michigan State University: East Lansing, MI, USA, 2009.
35. Nieveen, N. Formative Evaluation in Educational Design Research. In Educational Design Research; Plomp, T., Nieveen, N., Eds.;
SLO - Netherlands Institute for Curriculum Development: Enschede, The Netherlands, 2010.
36. Leite, S.; Áfio, A.; Carvalho, L.; Silva, J.; Almeida, P.; Pagliuca, L. Construção e Validação de Instrumento de Validação de
Conteúdo Educativo Em Saúde (Design and Validation of a Educational Health Content Validation Tool). Rev. Bras. Enferm. 2018,
71, 1732–1738. [CrossRef] [PubMed]
37. Anastasi, A. Psychological Testing, 4th ed.; MacMillan: New York, NY, USA, 1976.
38. Guilford, J.; Fruchter, B. Fundamental Statistics in Psychology and Education, 5th ed.; McGraw-Hill: New York, NY, USA, 1973.
39. Gall, M.; Gall, J.; Borg, W. Educational Research: An Introdution, 7th ed.; Pearson Education: Boston, MA, USA, 2003.
40. Abell, S.; Lederman, N. Handbook of Research on Science Education; Routledge: New York, NY, USA, 2010.
41. Bland, J.; Altman, D. Statistics Notes: Cronbach’s Alpha. Br. Med. J. 1997, 314, 314–572. [CrossRef]
42. Burgess, T. Guide to the Design of Questionnaires: A General Introduction to the Design of Questionnaires for Survey Research; University
of Leeds: Leeds, UK, 2001.
43. Cohen, L.; Manion, L.; Morrison, K. Research Methods in Education, 5th ed.; RoutledgeFalmer: London, UK; New York, NY,
USA, 2000.
44. Cronbach, L. Coefficient Alpha and the Internal Structure of Tests. Psychometrik 1951, 16, 297–334. [CrossRef]
45. Nunnally, J.; Bernstein, I. Psychometric Theory, 3rd ed.; McGraw-Hill, Inc.: New York, NY, USA, 1994.
46. Morville, P. Intertwingled: Information Changes Everything, 1st ed.; Morville, P., Ed.; Semantic Studios: Ann Arbor, MI, USA, 2014.
47. Tavares, R.; Vieira, R.; Pedro, L. Teachers’ responses to "Primary-school teachers’ conceptions about their knowledge and their
educational practices in Science Education using digital educational resources" questionnaire. figshare 2019. [CrossRef]
48. Amado, J. Manual de Investigação Qualitativa Em Educação (Handbook of Qualitative Research in Education), 2nd ed.; Imprensa da
Universidade de Coimbra: Coimbra, Portugal, 2014. [CrossRef]
49. Murphy, K.; Davidshofer, C. Psychological Testing: Principles and Applications, 6th ed.; Pearson Prentice Hall: Upper Saddle River,
NJ, USA, 2005.

91
Educ. Sci. 2021, 11, 79

50. Marôco, J.; Garcia-Marques, T. Qual a Fiabilidade Do Alfa de Cronbach? Questões Antigas e Soluções Modernas? (What Is the
Reliability of Cronbach’s Alpha? Old Questions and Modern Solutions?). Laboratório Psicol. 2006, 4, 65–90.
51. DeVellis, R. Scale Development: Theory and Applications, 2nd ed.; SAGE Publications Inc.: Thousand Oaks, CA, USA, 2003.
52. Hall, T.; Meyer, A.; Rose, D. Universal Design for Learning in the Classroom: Practical Applications; Guilford Press: New York, NY,
USA, 2012.
53. Rose, D.; Gravel, W. Universal Design for Learning. In International Encyclopedia of Education; Peterson, P., Baker, E., McGraw, B.,
Eds.; Elsevier: Oxford, UK, 2010.
54. Tavares, R.; Oliveira, D.; Laranjeiro, D.; Almeida, M. Universal Design for Learning: Potencial de Aplicação No Ensino Superior
Com Alunos Com NEE e Por Recurso a Tecnologias Mobile (Universal Design for Learning: Potential Application in Higher
Education to Students with Special Educational Needs Using Mobil. Educ. Formação Tecnol. 2015, 8, 84–94.
55. Meyer, A.; Rose, H.; Gordon, D. Universal Design for Learning: Theory and Practice; CAST: Wakefield, MA, USA, 2014.
56. Chitman-Booker, L.; Kopp, K. The 5Es of Inquiry-Based Science; Shell Education: Huntington Beach, CA, USA, 2013.
57. Wilson, C.; Taylor, J.; Kowalski, S.; Carlson, J. The Relative Effects and Equity of Inquiry-Based and Commonplace Science
Teaching on Students’ Knowledge, Reasoning, and Argumentation. J. Res. Sci. Teach. 2010, 47, 276–301. [CrossRef]
58. Bybee, R. The BSCS 5E Instructional Model: Creating Teachable Moments; National Science Teachers Association: Arlington, TX,
USA, 2015.
59. Marôco, J.; Lourenço, V.; Mendes, R.; Gonçalves, C. TIMSS 2015 – Portugal. Volume I: Desempenhos Em Matemática e Em Ciências
(TIMSS 2015 – Portugal. Volume I: Mathematics and Science Peformances); Instituto de Avaliação Educativa: Lisbon, Portugal, 2016.
60. Martins, I.; Veiga, M.; Teixeira, F.; Tenreiro-Vieira, C.; Vieira, R.; Rodrigues, A.; Couceiro, F. Colecção Ensino Experimental Das
Ciências: Educação Em Ciências e Ensino Experimental – Formação de Professores (Experimental Science Education Collection: Science
Education and Experimental Teaching - Teacher Training); Ministério da Educação e Ciência - Direcção-Geral de Inovação e de
Desenvolvimento Curricular: Lisbon, Portugal, 2007.
61. Prabha, L.; Shanavas, M. Educational Data Mining Applications. Oper. Res. Appl. An Int. J. 2014, 1, 23–29.
62. Martins, G.; Gomes, C.; Brocardo, J.; Pedroso, J.; Carrillo, J.; Ucha, L.; Encarnação, M.; Horta, M.; Calçada, M.; Nery, R.; et al. Perfil
Dos Alunos à Saída Da Escolaridade Obrigatória (Students Profile at the End of the Compulsory Education); Ministério da Educação:
Lisbon, Portugal, 2017.
63. Pujol, R. Didáctica de Las Ciencias En La Educación Primaria (Sciences Education in Primary Education); SINTESIS: Madrid, Spain, 2003.
64. Tenreiro-Vieira, C.; Vieira, R. Construindo Práticas Didático-Pedagógicas Promotoras Da Literacia Científica e Do Pensamento Crítico
(Building Didactic and Pedagogical Practices to Promote Scientific Literacy and Critical Thinking); OEI & Iberciencia: Madrid, Spain, 2014.
65. Tenreiro-Vieira, C.; Vieira, R. Literacia e Pensamento Crítico: Um Referencial Para a Educação Em Ciências e Em Matemática
(Literacy and Critical Thinking: A Framework for Science and Mathematics Education). Rev. Bras. Educ. 2013, 18, 163–242.
[CrossRef]
66. Vieira, R.; Tenreiro-Vieira, C. Estratégias de Ensino/Aprendizagem: O Questionamento Promotor Do Pensamento Crítico (Teaching and
Learning Strategies: The Questioning That Promotes Critical Thinking); Instituto Piaget: Lisbon, Portugal, 2005.
67. Zabala, A.; Arnau, L. Como Aprender e Ensinar Competências (How to Learn and Teach Competences); Artmed Editora: Porto Alegre,
Brazil, 2010.
68. Berney, S.; Bétrancourt, M. Does Animation Enhance Learning? A Meta-Analysis. Comput. Educ. 2016, 101, 150–167. [CrossRef]
69. Boyle, E.; MacArthur, E.; Connolly, T.; Hainey, T.; Manea, M.; Kärki, A.; van Rosmalen, P. A Narrative Literature Review of Games,
Animations and Simulations to Teach Research Methods and Statistics. Comput. Educ. 2014, 74, 1–14. [CrossRef]
70. Cox, R.; Ainsworth, S. Use Technology to Understand Better. In System Upgrade Realising the vision for UK education – A report from
the ESRC/EPSRC Technology Enhanced Learning; Northen, S., Ed.; tel.ac.uk: London, UK, 2012.
71. Guralnick, D.; Levy, C. Educational Simulations: Learning from the Past and Ensuring Success in the Future. In Design and
Implementation of Educational Games: Theoretical and Practical Perspectives; Zemliansky, P., Wilcox, D., Eds.; IGI Global: Hershey, PA,
USA, 2010. [CrossRef]
72. Kebritchi, M.; Hirumi, A.; Bai, H. The Effects of Modern Mathematics Computer Games on Mathematics Achievement and Class
Motivation. Comput. Educ. 2010, 55, 427–443. [CrossRef]
73. Laurillard, D. Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology; Taylor & Francis: New York,
NY, USA, 2013.
74. Schwartz, R.; Milne, C.; Homer, B.; Plass, J. Designing and Implementing Effective Animations and Simulations for Chemistry
Learning. In Pedagogic Roles of Animations and Simulations in Chemistry Courses; Suits, J., Sanger, M., Eds.; ACS Symposium Series;
American Chemical Society: Washington, DC, USA, 2013; pp. 43–76. [CrossRef]
75. Tavares, R.; Vieira, R. Formação Contínua de Professores Do 1.o CEB Em TIC: O Desenvolvimento de RED Com Recurso a
Ferramentas Da Web 2.0 (Primary-School Teachers Continuing Professional Development in ICT: Digital Educational Resources
Development Using Web 2.0 Tools). In V Conferência Internacional Investigação, Práticas e Contextos em Educação (2016); Alves, D.,
Pinto, H., Dias, I., Abreu, M., Muñoz, R., Eds.; Escola Superior de Educação e Ciências Sociais - Instituto Politécnico de Leiria:
Leiria, Portugal, 2016.
76. Ulicsak, M.; Williamson, B. A FUTURELAB Handbook: Computer Games and Learning; FUTURELAB: Bristol, UK, 2010.
77. Wang, P.-Y.; Vaughn, B.; Liu, M. The Impact of Animation Interactivity on Novices’ Learning of Introductory Statistics.
Comput. Educ. 2011, 56, 300–311. [CrossRef]

92
Educ. Sci. 2021, 11, 79

78. Centro de Física Computacional. Mocho > Simulações. Available online: http://www.mocho.pt/Ciencias/Fisica/simulacoes/
(accessed on 26 November 2015).
79. University of Colorado Boulder. PhET Interactive Simulations > Ensino Primário. Available online: https://phet.colorado.edu/
pt/simulations/filter?levels=elementary-school&sort=alpha&view=grid (accessed on 7 January 2017).
80. Underwood, J.S.; Kruse, S.; Jakl, P. Moving to the next Level: Designing Embedded Assessments into Educational Games. In
Design and Implementation of Educational Games: Theoretical and Practical Perspectives; Zemliansky, P., Wilcox, D., Eds.; IGI Global:
Hershey, PA, USA, 2010. [CrossRef]
81. Tsai, F.; Tsai, C.; Lin, K. The Evaluation of Different Gaming Modes and Feedback Types on Game-Based Formative Assessment
in an Online Learning Environment. Comput. Educ. 2015, 81, 259e269. [CrossRef]
82. Huang, W.; Soman, D. A Practitioner’s Guide To Gamification Of Education [Research Report Series Behavioural Economics in Action];
Rotman School of Management, University of Toronto: Toronto, ON, Canada, 2013.
83. Connolly, T.; Boyle, E.; MacArthur, E.; Hainey, T.; Boyle, J. A Systematic Literature Review of Empirical Evidence on Computer
Games and Serious Games. Comput. Educ. 2012, 59, 661–686. [CrossRef]
84. Pedaste, M.; Mäeots, M.; Siiman, L.; de Jong, T.; van Riesen, S.; Kamp, E.; Manoli, C.; Zacharia, Z.; Tsourlidaki, E. Phases of
Inquiry-Based Learning: Definitions and the Inquiry Cycle. Educ. Res. Rev. 2015, 14, 47–61. [CrossRef]

93
 education
sciences
Article
Application of Virtual Reality in Computer Science Education:
A Systemic Review Based on Bibliometric and Content
Analysis Methods
Friday Joseph Agbo 1, * , Ismaila Temitayo Sanusi 1 , Solomon Sunday Oyelere 2, * and Jarkko Suhonen 1

1 School of Computing, University of Eastern Finland, FIN-80101 Joensuu, Finland;

[email protected] (I.T.S.); [email protected] (J.S.)
2 Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology,
971 87 Luleå, Sweden
* Correspondence: [email protected] (F.J.A.); [email protected] (S.S.O.)

Abstract: This study investigated the role of virtual reality (VR) in computer science (CS) education
over the last 10 years by conducting a bibliometric and content analysis of articles related to the use of
VR in CS education. A total of 971 articles published in peer-reviewed journals and conferences were
collected from Web of Science and Scopus databases to conduct the bibliometric analysis. Furthermore,
content analysis was conducted on 39 articles that met the inclusion criteria. This study demonstrates
that VR research for CS education was faring well around 2011 but witnessed low production output
between the years 2013 and 2016. However, scholars have increased their contribution in this field
recently, starting from the year 2017. This study also revealed prolific scholars contributing to the
field. It provides insightful information regarding research hotspots in VR that have emerged recently,





 which can be further explored to enhance CS education. In addition, the quantitative method remains
the most preferred research method, while the questionnaire was the most used data collection
Citation: Agbo, F.J.; Sanusi, I.T.;
Oyelere, S.S.; Suhonen, J. Application
technique. Moreover, descriptive analysis was primarily used in studies on VR in CS education. The
of Virtual Reality in Computer study concludes that even though scholars are leveraging VR to advance CS education, more effort
Science Education: A Systemic needs to be made by stakeholders across countries and institutions. In addition, a more rigorous
Review Based on Bibliometric and methodological approach needs to be employed in future studies to provide more evidence-based
Content Analysis Methods. Educ. Sci. research output. Our future study would investigate the pedagogy, content, and context of studies on
2021, 11, 142. https://doi.org/ VR in CS education.
10.3390/educsci11030142

Keywords: computer science education; virtual reality; VR; content analysis; bibliometric analysis;
Academic Editor: João Piedade immersion; 3D simulation; presence; game-based learning

Received: 22 February 2021

Accepted: 16 March 2021
Published: 23 March 2021
1. Background of the Study
Publisher’s Note: MDPI stays neutral
Virtual reality (VR) has recently become a popular technology in different contexts
with regard to jurisdictional claims in
such as entertainment, military, and education [1]. VR combines technologies to provide an
published maps and institutional affil- immersive presence through highly interactive objects in a virtual environment but stimu-
iations. lates users’ sensory awareness to perceive being in an almost natural environment. The use
of VR in education to support training, teaching, and learning through 3D simulation and
visualization of learning content in a virtual presence has grown recently [2]. This increasing
VR application growth in the educational field is evident, as revealed by the literature,
Copyright: © 2021 by the authors.
including a recent VR study in computer science education [3]. VR technology provides an
Licensee MDPI, Basel, Switzerland.
opportunity to develop a state-of-the-art smart learning environment with a high level of
This article is an open access article
interaction, engagement, and motivation for an enhanced learning experience [1–8]. This
distributed under the terms and study refers to computer science (CS) education as the art and science involved in learning
conditions of the Creative Commons and teaching computer science, including computing, algorithmic and computational think-
Attribution (CC BY) license (https:// ing [9]. For example, the science behind curriculum design, pedagogical approach, and
creativecommons.org/licenses/by/ instructional tools and techniques educators adopt to support computer science teaching
4.0/). and learning.

95
Educ. Sci. 2021, 11, 142

This study investigated the role of VR in CS education by conducting a comprehen-

sive content and bibliometric analysis of relevant articles published between 2011 and
2020 in journals and conferences. Bibliometric and content analysis of articles focused on
VR in CS education would provide a deeper understanding of the evolution of research
conducted in this field and how VR applications have advanced CS education over the
years [4,10,11]. From the standpoint of bibliometric mapping analysis, this study inves-
tigates the publication growth of studies on VR in CS education within the last 10 years,
reveals the most active authors and affiliations contributing to the development of VR in
CS education, and anticipates the future direction on the basis of the co-occurrence pattern
analysis of current studies. In addition, this study explicates the role of VR in CS education
from the perspective of methodological approaches used in studies related to VR in CS
education [7], the kind of data collected for such studies, the sample size, and the types of
data analysis conducted.
Research on VR in education has claimed several benefits, such as positively af-
fecting users’ attitude [12,13], presenting an effective and efficient learning and training
environment [14,15], and increasing students’ motivation to learn within a virtual environ-
ment [14–17]. Furthermore, many systematic review studies related to VR in education
have been published in recent years. However, there have been only a limited number
of such studies focused on computer science education. For example, Pirker et al. [3]
conducted a systematic literature review of VR in CS education, focusing on the technology
used to deploy VR applications for CS education, the learning objectives, and challenges
recorded in studies related to VR in CS education. Pirker and colleagues revealed that VR
desktop applications using Oculus Rift and HTC Vive dominate the technology currently
used to deploy VR in CS education. On the other hand, the majority of studies on VR in CS
education focused on cognitive learning with topics such as fundamental components of
algorithms and object-oriented programming [3].
Similarly, Oyelere et al. [1] studied VR games in CS education, focusing on devel-
opmental features such as the technology, pedagogy, and gaming elements used in such
studies. In terms of technology, Oyelere et al. [1] finding was in congruence with that of
Pirker et al. [3], where Oculus Rift, HTC Vive, and PC-based applications dominate the
technology aspect. Both studies show that mobile-based VR applications for CS education
are still growing, with less than 15% of deployment of VR applications on mobile devices.
We could find only a few studies regarding recent studies that focused on content and
bibliometric analysis of articles related to VR in education. For example, Arici et al. [11]
conducted content and bibliometric mapping analysis of augmented reality (AR) in science
education. Lorenzo et al. [17] investigated VR articles’ scientific production for inclusive
learning of people with autism spectrum disorder (ASD). Sobral and Pestana [18] studied a
bibliometric analysis of articles related to VR application to learn about dementia from 1998
until 2018 by focusing on articles’ intellectual structure and emerging trends. Lai et al. [19]
conducted a bibliometric analysis of VR research in engineering education published and
indexed in the Scopus database that spans over 26 years. Thus, Lai et al. [19] provided
valuable insights in terms of article production, trends, and co-occurrence network of VR
studies within the field of engineering. Another bibliometric study related to VR in CS
field-specific was recently conducted by Enebechi and Duffy [20]. This study [21] focused
on bibliometric analysis of VR and artificial intelligence (AI) articles in mobile computing
and applied ergonomics.
While all these related studies highlighted above are relevant and provided essential
knowledge about the field, our current research would expand on the existing research
rather than re-inventing the wheel. For example, while the work of Pirker et al. [3] mainly
focused on the technology used to deploy VR application for CS education, the learning
objectives, and challenges recorded in studies related to VR in CS education, our research
would address the aspect of methodological approach used in studies on VR in CS education,
kind of data collected for such studies, the sample size, and types of data analysis conducted.
The majority of these related studies analyzed a small sample size, limiting the study, and

96
Educ. Sci. 2021, 11, 142

cannot justify the generalization of their findings. For example, Pirker et al. [3] analyzed
13 pieces of data, Lorenzo et al. [17] revealed 18 articles, Lai et al. [19] conducted bibliometric
analysis on 274 articles, and Enebechi and Duffy [20] presented a content analysis of 8 papers.
Our study took a different approach by analyzing more extensive data to discover more
profound knowledge in the field. It is worth mentioning that our study drew motivation
from [11] by focusing content analysis of variables such as materials and method trends,
sample sizes, and method of an investigation conducted by articles on VR in CS education
in the last 10 years. The authors hope that the approach used in this study would contribute
to the existing knowledge in terms of unveiling how VR has supported CS education and
what scientific achievement have been made in this field.
As a result of this comprehensive content and bibliometric analysis of studies on VR
in CS education, we hoped that our findings would contribute to the existing knowledge
by providing a deeper understanding of VR applications’ role in honing CS education over
the last decade. In addition, the authors believe that this study will unveil information
regarding what scholars have made a scientific achievement in this field in terms of
advancing teaching and learning of CS topics in the different contexts, which will serve
as a boost for active researchers. In contrast, new scholars would derive motivation and
valuable resources for future studies. To achieve objectives, this study set out to answer the
following research questions:
RQ1 How is the growth of research publication and citation of articles on VR in computer
science education?
RQ2 Who are the most active authors, institutions, and countries publishing articles on the
use of VR in computer science education?
RQ3 What co-occurrence patterns exist in studies on the use of VR in computer science education?
RQ4 What is the trend of the research methodology employed in articles on VR in computer
science education?
RQ5 What were the most preferred data collection tools and sampling methods in articles
on the use of VR in computer science education?
RQ6 What were the sample sizes in articles on the use of VR in computer science education?
RQ7 What were the most preferred data analysis methods in articles on the use of VR in
computer science education?

2. Methods
The method explored in this study was centered on content and bibliometric mapping
analysis. This study followed the recommended workflow for science mapping provided
by Aria and Coccurullo [21] to conduct our bibliometric mapping analysis. In contrast, the
approach shown by [11] was followed to present the content analysis, respectively.
Article selection process
The article selection process for this study includes 3 phases similar to the one pre-
sented by [4], namely, (i) literature search and data collection; (ii) data extraction, loading,
and conversion; and (iii) data synthesis. A graphical representation of the data collection
process is presented in Figure 1, showing detailed actions in each phase.
(i) Literature search and data collection
This study obtained data from 2 databases, i.e., the Web of Science (WoS) and the
Scopus databases. These 2 databases have been acclaimed to contain comprehensive data
of scientific outputs relevant to this study [14]. To conduct an extensive data collection
needed for this study, we define the search keywords to include “virtual reality” “VR”,
“computer science”, and “computing education”. A number of common protocols for data
collection were applied to both databases. They include the same search keywords used in
combination with the binary operators such as “OR” and “AND” across the 2 databases,
limited time span to the period from 2011 to 2020, and language selected as “English”.
Table 1 presents details of the search protocol, how they were applied in each database,
and the result obtained.

97
 format, we conducted data extraction and conversion into a comma-separated values CSV
file to merge the 2 datasets from WoS and Scopus. The process of merging the data is
presented in Table 2, followed by executing command line instructions (CLI) shown in
Figure 2. R-studio is an integrated development environment for R programming lan-
guage (https://rstudio.com)) software was used to combine the data into a single CSV file
Educ. Sci. 2021, 11, 142
before uploading it to biblioshiny (Biblioshiny is a web interface for bibliometrix r-pack-
age (https://www.bibliometrix.org/Biblioshiny.html)) for bibliometrix R-package [17].

Figure 1.
Figure The procedure
1. The procedure followed
followed to
to obtain
obtain data
data used
used for
for bibliometric
bibliometric and
and content
content analysis.
analysis.

Table 1. Data search procedures and obtained amount of data.

Combination of Search String Based on

Database Description of the Protocol Search Outcome
Database Algorithm
Applying the search keywords in
TOPIC: (“virtual reality” OR “VR”) AND TOPIC:
quotation to the WoS TOPIC field 80
(“computer science” OR “computing education”).
with binary operators.
TOPIC: (“virtual reality” OR “VR”) AND TOPIC:
WoS Additional conditions were (“computer science”).Refined by: DOCUMENT
applied by limiting the results to TYPES: (ARTICLE OR PROCEEDINGS PAPER)
only articles and proceedings AND PUBLICATION YEARS: (2020 OR 2014 OR 58
papers, with time span set to 2019 OR 2013 OR 2018 OR 2012 OR 2017 OR 2011
2011–2020. OR 2016 OR 2015)Timespan: All years. Indexes:
SCI-EXPANDED, SSCI, A&HCI, ESCI.
Applying the search keywords in
(TITLE-ABS-KEY (“virtual reality” OR “VR”)
quotation to Scopus title, abstract,
AND TITLE-ABS-KEY (“computer science” OR
and keywords field with binary 1058
“computing education”)) AND PUBYEAR > 2010
operators and limiting the time
AND PUBYEAR < 2021.
span to 2011–2020.
Scopus
(TITLE-ABS-KEY (“virtual reality” OR “VR”)
AND TITLE-ABS-KEY (“computer science” OR
Applying additional conditions
“computing education”)) AND PUBYEAR > 2010
by limiting to only articles and 962
AND PUBYEAR < 2021 AND (LIMIT-TO
conference papers.
(DOCTYPE, “cp”) OR LIMIT-TO (DOCTYPE,
“ar”)) AND (LIMIT-TO (LANGUAGE, “English”)).
After merging both files, we removed 971
49 duplicated documents. Total

(ii) Data extraction, loading, and conversion

After data from the independent databases were collected and downloaded in BibTex
format, we conducted data extraction and conversion into a comma-separated values CSV
file to merge the 2 datasets from WoS and Scopus. The process of merging the data is
presented in Table 2, followed by executing command line instructions (CLI) shown in
Figure 2. R-studio is an integrated development environment for R programming language
(https://rstudio.com, accessed on 18 January 2018) software was used to combine the data
into a single CSV file before uploading it to biblioshiny (Biblioshiny is a web interface for
bibliometrix r-package (https://www.bibliometrix.org/Biblioshiny.html, accessed on 18
January 2018) for bibliometrix R-package [17].

98
 Educ. Sci. 2021, 11, 142
Educ. Sci. 2021, 11, x FOR PEER REVIEW 6 of 25

Table 2. Data conversion and merging steps.

Steps Authors
Instructions on How to Merge Two Points of Data from WoS and Scopus Databases2738
1
Author appearances
Download in BibTex format independently from databases (in this case, WoS and Scopus).
3308
Authors of single-authored documents 98
2 Save data in a directory with a name that says “rawData”.
Authors of multi-authored documents 2640
Open RStudio and import the bibliometrix library by running the script < library(“bibliometrix”) > in
3 Author collaboration
the command-line interface (CLI).
Single-authored
In Rstudio CLI, rundocuments 102 where
the script < setwd (“C:/../ . . . / . . . /rawData”) > to open the directory
4 data would be
Documents imported
per authorfrom and saved. Not that the ellipsis ( . . . ) indicates the paths0.355
to the
directory and should be correctly inserted.
Authors per document 2.82
5 Download in BibTex format
Co-authors per documents independently from databases (in this case, WoS and Scopus).
3.41
6 Save data in a directory
Collaboration index with a name that says “rawData”. 3.04

Figure 2. (A) shows the set of commands to be executed in R-Studio command line instructions (CLI) to implement the
Figure 2. (A) shows the set of commands to be executed in R-Studio command line instructions (CLI) to implement the
conversion and to merge of data downloaded from Web of Science (WoS) and Scopus databases; (B) shows the output of the
conversion and to merge of data downloaded from Web of Science (WoS) and Scopus databases; (B) shows the output of
executed commands; (C) depicts the console for the CLI where the line execution returns a value including line errors.
the executed commands; (C) depicts the console for the CLI where the line execution returns a value including line errors.
After completing the steps in Table 2, we executed the line of commands (lines 6 to
3. Results
15) in Figure 2 to complete the remaining process of data conversion and merging. This
3.1.merging of from
Findings the two converted points
Bibliometric Mappingof data by running the command in line 11 of Figure 2A
Analysis
triggered the R- Function that identified 49 similar articles from WoS and Scopus databases.
TheThis sectionsimilar
identified presents our were
articles findings fromtothe
removed bibliometric
avoid analysis
having duplicate on the
data. basis of the
Removing
data generated
duplicate fromleft
articles WoSthe and Scopusdata
remaining databases. This bibliometric
at 971, which was uploadedanalysis intendsfor
to biblioshiny to pro-
vide insight into how studies on the use of VR for CS education have
bibliometric mapping analysis. The search was conducted on 2 January 2021. grown in the last 10
years. In addition, the result reveals authors, institutions, and countries who have been
contributing to the field by actively
99 publishing research related to VR in CS education.
Furthermore, the result presents how studies on VR in CS education have had an impact
in terms of their citations and authors co-occurrence pattern analysis. The section deline-
ates the analysis of common keywords used in articles on VR for CS education, thereby
Educ. Sci. 2021, 11, 142

(iii) Data Synthesis

In Table 3, we present the synthesized data used for the bibliometric analysis. However,
for the content analysis, 3 researchers screened the entire data by reading each paper’s
abstract to decide whether it was relevant or not. Further criteria for selecting relevant
papers suitable for the content analysis included:

Table 3. Data synthesis indicating the primary information about the data and document type.

Description Results
Main information about data
Timespan 2011–2020
Sources (journals, books, etc.) 378
Documents 971
Average years from publication 4.53
Average citations per documents 3.754
Average citations per year per doc 0.7841
References 21,021
Document types
Article 157
Conference paper 814
Document contents
Keywords plus (ID) 6281
Author’s keywords (DE) 2848
Authors
Authors 2738
Author appearances 3308
Authors of single-authored documents 98
Authors of multi-authored documents 2640
Author collaboration
Single-authored documents 102
Documents per author 0.355
Authors per document 2.82
Co-authors per documents 3.41
Collaboration index 3.04

(i) the paper must focus on virtual reality for education in computer science education;
(ii) the paper designed a study or developed a solution to facilitate CS education in a
VR environment;
(iii) the study reported any outcome by evaluating with users (students, educators, or experts);
(iv) the paper is open access and could be downloaded for detailed review.
After applying the criteria, we arrived at 39 papers that met the content analysis
requirements presented in Section 3.2.

3. Results
3.1. Findings from Bibliometric Mapping Analysis
This section presents our findings from the bibliometric analysis on the basis of the
data generated from WoS and Scopus databases. This bibliometric analysis intends to
provide insight into how studies on the use of VR for CS education have grown in the
last 10 years. In addition, the result reveals authors, institutions, and countries who
have been contributing to the field by actively publishing research related to VR in CS
education. Furthermore, the result presents how studies on VR in CS education have had an
impact in terms of their citations and authors co-occurrence pattern analysis. The section
delineates the analysis of common keywords used in articles on VR for CS education,
thereby presenting the thematic area of the current research landscape and topic hotspots.

100
 Educ.
Educ. Sci.Sci. 11,11,
2021,
2021, 142 PEER REVIEW
x FOR 7 of 25

3.1.1.
3.1.1.Research
ResearchPublication
Publication Growth of Articles
Growth of Articleson
onthe
theUse
UseofofVR
VRinin Computer
Computer
Science Education
Science Education
Figure33shows
Figure shows the
the articles’
articles’ distribution
distributionininterms
termsofofthe
thepublication
publication year regarding
year regarding
the article production and development across 10 years. The overall
the article production and development across 10 years. The overall publicationpublication trend of of
trend
articlesrelated
articles relatedto
toVR
VR in
in CS
CS education
educationshows
showsthat
that2011
2011witnessed
witnessed the highest
the production
highest production
year, reaching 148 articles, followed closely by 135 articles in
year, reaching 148 articles, followed closely by 135 articles in 2018.2018.

160
140
120
100
80
60
40
20
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Articles

Figure 3. 3.
Figure Annual scientific
Annual scientificproduction
productionof
ofarticles
articles on virtual reality
on virtual reality(VR)
(VR)inincomputer
computer science
science (CS)
(CS) education.
education.

Thepublication
The publication volume
volume decreased
decreased from
from2012
2012toto2015
2015and
andfrom
from 2019
2019to to
2020. There
2020. There
wasan
was anincrease
increasein
inarticle
article production
production from
from2016
2016toto2018
2018before
beforethe
theslight decline
slight declineuntil 2020.
until 2020.
This trend occurred probably because the selected articles were limited to only
This trend occurred probably because the selected articles were limited to only education,education,
leavingout
leaving outother
otherdomains,
domains, such
such as
as health,
health,business,
business,entertainment,
entertainment, and
andmedia.
media.
3.1.2. Most Active Authors, Institutions, and Countries Publishing Articles on the Use of
3.1.2.
VR in Most ActiveScience
Computer Authors, Institutions, and Countries Publishing Articles on the Use of
Education
VR in Computer Science Education
Regarding authors’ production over time, we investigated the top 20 authors. Our
findings showed
Regarding that most
authors’ of those top
production authors
over time,were already publishing
we investigated the top articles on VR in
20 authors. Our
CS education
findings showed by that
2011.most
However, about
of those tophalf of those
authors authors
were were
already not active articles
publishing from 2019. As in
on VR
CSshown in Figure
education 4, many
by 2011. articles related
However, to VR
about half ofinthose
CS education
authors werewere notpublished
active between
from 2019.
As2011 and 2020.
shown in Figure 4, many articles related to VR in CS education were published be-
As we
tween 2011 and can2020.
see in Figure 4, the author Li Y. had the highest publication over time,
having
As we can see articles
had several in Figure published yearly for
4, the author 7 years
Li Y. fromhighest
had the 2011 topublication
2020, except over
in 2013,
time,
2014, and 2016. With the least productivity over time was the author
having had several articles published yearly for 7 years from 2011 to 2020, except in is Dengel A., with
2013,
publications
2014, and 2016. only in 2019
With the and
least2020.
productivity over time was the author is Dengel A., with
We analyzed the top 20 authors’ number citations across the production years (m-
publications only in 2019 and 2020.
index) regarding their impact. M-index is calculated by dividing the total number of
We analyzed the top 20 authors’ number citations across the production years (m-
citations by the total number of years of production. In order words, this study measures
index) regarding
the authors’ their
impact byimpact.
dividing M-index is calculated
the H-index by dividing
by the total number of the totalofnumber
years of cita-
production.
tions
Note that the total years of production varied for different authors. Although the totalthe
by the total number of years of production. In order words, this study measures
authors’
numberimpact
of years byinvestigated
dividing theinH-index by the
this study total number
remained of years
at 10, some of production.
authors did not startNote
that the totalfrom
publishing years2011;
of production
therefore, varied
such anfor different
author’s authors.
total number Although
of years the total number
of production
ofwould
years count from theinyear
investigated thisthe author
study published
remained his/her
at 10, somefirst paper.did
authors Fornotexample, Dengel
start publishing
A. started
from publishingsuch
2011; therefore, articles
an on VR in total
author’s CS education
number in of 2019;
yearshence, the total number
of production of
would count
from the year the author published his/her first paper. For example, Dengel A. started
101
publishing articles on VR in CS education in 2019; hence, the total number of years
Educ.
Educ. Sci. 2021,
Educ. Sci.
Sci. 2021, 11, xx142
2021, 11,
11, FOR
FOR PEER
PEER REVIEW
REVIEW 88 of
of 25
25

years remained
remained
remained at two.atTherefore,
at two. two. Therefore,
Therefore, the the m-index
the m-index
m-index wouldwould
would be thebe
be the thenumber
total
total total number
number of of citations
of citations
citations in in
in 2019
2019
20192020,
and
and and 2020,
2020, divided
divided
divided by 2. by 2.
by 2.

Figure 4.
Figure
Figure 4. Top 20
4. Top
Top 20 authors
20 authors publishing
authors publishing articles
publishing articles on
articles on VR in
on VR
VR in CS
in CS education
CS education between
education between 2011
between 2011 and
2011 and 2020:
and 2020: the size
2020: the
the size of
size of each
of each circle
each circle indicates
circle indicates
indicates
the number
the
the number of
number of articles.
of articles. The amount
articles. The
The amount of
amount of boldness
of boldness of
boldness of the
of the circles
the circles shows
circles shows the
shows the number
the number of
number of citations
of citations in
citations in that
in that year.
that year.
year.

As shown
As
As shown in
shown in Figure
in Figure 5,
Figure 5, the
5, the authors’
the authors’ m-index
authors’ m-indexwas
m-index washighest
was highestat
highest at1.0
at 1.0(to
1.0 (to aaa single
(to single decimal).
single decimal).
decimal).
Therefore, the
Therefore,
Therefore, the result
the result indicates
result indicates that
indicates that Dengel
that Dengel A.,
Dengel A., with
A., with the
with the highest
the highest m-index,
highest m-index, remained the
m-index, remained
remained the
the
most
most impactful
impactful author
author at
at the
the end
end of
of2020.
2020. This
This finding
findingsuggests
suggests that
thatDengel
Dengel
most impactful author at the end of 2020. This finding suggests that Dengel A. had had A. had
A. had
had an
had
unbroken
an
an unbrokenresearch
unbroken research activity
research activity in the
activity in area
in the of
the area VR
area of in
of VR CS
VR in education
in CS since
CS education the
education since first
since the publication
the first and
first publication
publication
had had
and
and received
had a significant
received
received number
aa significant
significant numberof citations.
number of
of citations.
citations.

1.0
1.0
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
m-index
m-index

0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
GOURANT…
GOURANT…
ZHANG YY
DENGEL AA

RIVA GG

BARREAU JJ

W
GAUGNE RR

SERAFIN SS

LIN CC
SHARMA SS

LI YY
CIPRESSO PP

WANG YY

CHEN YY

ZHU SS

LI JJ
WANG XX

LIU XX

WANG JJ

ZHAO W
KRAUS M

BARREAU

LI
WANG
SHARMA

SERAFIN

ZHU
ZHANG
LI
WANG

CHEN

LIN
CIPRESSO

WANG

LIU

GAUGNE
DENGEL

RIVA

KRAUS

ZHAO

Authors
Figure
Figure 5.
Figure 5. Top-20
5. Top-20 authors’
Top-20 authors’ impact
authors’ impact analysis
impact analysis within
analysis within 10
within 10 years.
10 years.
years.

102
Educ. Sci. 2021, 11, x FOR PEER REVIEW 9 of 25
Educ. Sci. 2021, 11, 142

Our analysis revealed some top universities regarding institutions (authors’ affilia-
Our analysis revealed some top universities regarding institutions (authors’ affilia-
tions) and countries
tions) and countriesfronting
fronting VR in CS
VR in CSeducation.
education.AsAs shown
shown in Figure
in Figure 6, some
6, some of these
of these
universities, to toname
universities, nameaafew,
few, were theUniversity
were the Universityof of Southern
Southern California,
California, USA;USA;
AalborgAalborg
University,
University, Denmark;and
Denmark; and University
University ofofRennes,
Rennes,France.
France.

FigureFigure 6. Three-field
6. Three-field plotplot of activeinstitutions
of active institutions and
andcountries
countriespublishing articles
publishing related
articles to VR in
related toCS
VReducation between 2011
in CS education between
and 2020.
2011 and 2020.
Figure 6 shows the USA as the most productive country in terms of publishing articles
Figure
related to 6VR
shows the USA in
in CS education ascountries.
the mostFrom
productive country
the European in terms
continent, of publishing
France, Denmark, arti-
clesItaly, the UK,
related Germany,
to VR in CSand Spain made
education significant contributions.
in countries. Only China
From the European and JapanFrance,
continent,
made contributions
Denmark, regarding
Italy, the UK, VR inand
Germany, CS education
Spain made fromsignificant
the Asian continent.
contributions. Only China
and Japan made contributions regarding VR in CS education from the Asian continent.
3.1.3. Keywords Co-Occurrence Patterns of Studies on the Use of VR in Computer
Science Education
3.1.3. Keywords Co-Occurrence
A keywords co-occurrence Patterns of Studies
pattern (KCP) focuseson onthe Use of VR in
understanding theComputer
knowledge
Science Education
components and knowledge structure of a scientific field by examining the links between
keywords
A keywordsin the co-occurrence
published articles within(KCP)
pattern the same area [4].
focuses on understanding the knowledge
components and knowledge structure of a scientific field of
Figure 7 focuses on keyword co-occurrence patterns bystudies on the
examining theuse of VR
links between
in computer science education. As observed in Figure 7, the root keyword in the field
keywords in the published articles within the same area [4].
remains “virtual reality”. Other keywords that are frequently used by articles on VR in CS
Figure 7 focuses on keyword co-occurrence patterns of studies on the use of VR in
education are shown in red color. For instance, we notice keywords such as gamification,
computer science
simulation, highereducation.
education,As observed
mixed reality,inserious
Figuregames,
7, theand
rootmore.
keyword in the as
In addition, field re-
mains “virtual
expected, reality”.
keywords thatOther
definekeywords that are
the characteristics of frequently
virtual realityused by articles
technology on VR
were seen to in CS
be strongly
education areconnected
shown intored thecolor.
root keyword. For example,
For instance, we observe
we notice a thicksuch
keywords line connecting
as gamification,
keywordshigher
simulation, such aseducation,
immersion,mixed
interaction,
reality,and presence,
serious to the
games, androot keyword
more. “virtual as ex-
In addition,
reality”.
pected, Moreover,
keywords thatvirtualization, cloud computing,
define the characteristics and virtual
of virtual machine
reality are keywords
technology were seen to
that show a strong connection. Other keywords that show a close relationship to virtual
be strongly connected to the root keyword. For example, we observe a thick line connect-
reality include augmented reality and computer science.
ing keywords such as immersion, interaction, and presence, to the root keyword “virtual
reality”. Moreover, virtualization, cloud computing, and virtual machine are keywords
that show a strong connection. Other keywords that show a close relationship to virtual
reality include augmented reality and computer science.

103
 Educ.
Educ. Sci.
Sci. 2021, 11,11,
2021, 142 PEER REVIEW
x FOR 10 of 25

Figure
Figure 7. 7. Co-occurrence
Co-occurrence patterns
patterns ofof authors’
authors’ keywords
keywords inin articles
articles onon
VRVR
inin
CSCS education
education between
between 2011and
2011 and2020.
2020.

Furthermore,
Furthermore, Figure
Figure 8 presents
8 presents a avisualization
visualizationofoffrequently
frequentlyused usedkeywords
keywordsininVR VR
for CS education. It is clear from the size of the nodes that other related
for CS education. It is clear from the size of the nodes that other related terms used for terms used for
virtual
virtual reality,for
reality, forexample,
example,“virtualization”
“virtualization”and and“virtual
“virtualenvironment”
environment” werefound
were foundtotobebe
highly connected to “computer science” and “education”. In addition,
highly connected to “computer science” and “education”. In addition, some pedagogical some pedagogical
concepts for teaching and learning, such as games, gamification, collaborative learning, and
concepts for teaching and learning, such as games, gamification, collaborative learning,
immersive learning, are visible in the network. Figure 8 also shows clustering of concepts
and immersive learning, are visible in the network. Figure 8 also shows clustering of con-
where terms such as virtualization, virtual environment, computer science, and education
cepts where terms such as virtualization, virtual environment, computer science, and ed-
form clusters depicted with different colors.
ucation form clusters depicted with different colors.
One way to examine how VR application has influenced CS education is to analyze
One way to examine how VR application has influenced CS education is to analyze
trending topics over the period considered in this study. Figure 9 presents the trending
trending topics over the period considered in this study. Figure 9 presents the trending
topics or approaches scholars have explored to provide VR intervention for CS education.
topics or approaches scholars have explored to provide VR intervention for CS education.
This study analyzed the authors’ keywords to determine what research hotspot in
This study analyzed the authors’ keywords to determine what research hotspot in
terms of topics and approaches have been explored by VR applications in CS education
terms of topics and approaches have been explored by VR applications in CS education in
in the last decade. This analysis was conducted through the word cloud of authors’
the last decade.
keywords, which This analysis
gives wastoconducted
a pointer through
what has been the wordinterest.
the scholars’ cloud of authors’
This analysiskey-
also
words, which gives a pointer to what has been the scholars’ interest. This
provides insight regarding the future outlook of VR interventions in CS education. Figure analysis also 9
provides insight
delineates thatregarding the future
virtualization, cloudoutlook of VRthe
computing, interventions
virtual world, in CSand
education. Figure
virtual machine
9 delineates
dominate VR thatstudies
virtualization, cloud computing,
in CS education between thethe virtual
years 2011 world,
and 2015.andInvirtual machine
addition, slightly
dominate VR studies in CS education between the years 2011 and
different changes were observed where keywords such as computer science education, 2015. In addition,
slightly
seriousdifferent
games, changes
and higher were observedemerged
education where keywords
among the such as computer
trending topics science
betweenedu-2015
cation, serious
and 2017. games, and higher education emerged among the trending topics between
2015 and 2017.

104
Educ. Sci. 2021, 11, 142
Educ. Sci. 2021, 11, x FOR PEER REVIEW 11 of 25

Figure Visualized
8. 2021,
Figure
Educ. Sci. 11, x FORauthors’
8. Visualized keywords
authors’
PEER REVIEW
keywordsco-occurrence analysisofofarticles
co-occurrence analysis articles
on on
VRVR in CS
in CS education:
education: these these are among
are among
12 of the
the
25
highest number of repetitive keywords within the field.
highest number of repetitive keywords within the field.
Furthermore, it was observed that between the years 2018 and 2020, new keywords
such as augmented reality, immersion, presence, gamification, game-based learning, and
human–computer interaction were added to the trending topics. Therefore, topics such as
immersion, presence, human–computer interaction, gamification, and game-based learn-
ing dominate the list of research hotspots in recent times. This finding suggests that one
of the most appreciated learning and teaching approaches used by studies on VR applica-
tion in CS education is game-based learning.

FigureFigure 9. (a–c)
9. (a–c) WordWord cloud
cloud showing
showing thethe trendingtopics
trending topicsof
of VR
VR in
in CS
CS education
educationinin
terms of authors
terms keywords.
of authors WhileWhile
keywords. (a) (a)
shows the trending topics between 2011 and 2015, (b) presents the topics between 2015 and 2017, and (c) depicts the
shows trending
the trending topics between 2011 and 2015, (b) presents
topics on VR in CS education between 2018 and 2020.
the topics between 2015 and 2017, and (c) depicts the trending
topics on VR in CS education between 2018 and 2020.
3.2. Findings from Content Analysis
105
This section presents the content analysis findings to address some of the research
questions (RQ4 to RQ7). Moreover, an overview of the data analyzed in this section is
presented as an Appendix A. In the Appendix A, information regarding the study focus
Educ. Sci. 2021, 11, 142

Furthermore, it was observed that between the years 2018 and 2020, new keywords
such as augmented reality, immersion, presence, gamification, game-based learning, and
human–computer interaction were added to the trending topics. Therefore, topics such as
immersion, presence, human–computer interaction, gamification, and game-based learning
dominate the list of research hotspots in recent times. This finding suggests that one of the
most appreciated learning and teaching approaches used by studies on VR application in
CS education is game-based learning.

3.2. Findings from Content Analysis

This section presents the content analysis findings to address some of the research
questions (RQ4 to RQ7). Moreover, an overview of the data analyzed in this section is
presented as an Appendix A. In the Appendix A, information regarding the study focus
and outcome are highlighted to showcase how the selected articles have employed VR in
CS education.

3.2.1. Trends of the Research Methodology Employed in Articles on the Use of VR in

Computer Science Education
According to Figure 10, 47% of the articles used a quantitative design approach,
16% used a qualitative design, 3% used mixed design, and 12% utilized a design and
development research approach. In comparison, others may include review/meta-analysis
Educ. Sci. 2021, 11, x FOR PEER REVIEW 13 of 25
research accounts for 5%.

Design and development Qualitative Mixed method Quantitative Others

Figure 10. Frequency of research methods in articles on VR in CS education between 2011 and 2020.
Figure 10. Frequency of research methods in articles on VR in CS education between 2011 and 2020.

Figure 11 revealed the research method trends related to VR in CS education in the

past 10 years. The use of quantitative
106 methods increased in 2018 and declined from 2019
to 2020. The next prominent method utilized is the design research method used in 2011
and in 2014, and witnessed an increase in 2020. While mixed methods are almost inexist-
ent, qualitative and other methods showed no significant distribution variations over
Educ. Sci. 2021, 11, 142

Figure 11 revealed the research method trends related to VR in CS education in the

past 10 years. The use of quantitative methods increased in 2018 and declined from 2019 to
2020. The next prominent method utilized is the design research method used in 2011 and
in 2014, and witnessed an increase in 2020. While mixed methods are almost inexistent,
qualitative and other methods showed no significant distribution variations over time.
Educ. Sci. 2021, 11, x FOR PEER REVIEW 14 of 25
Review and meta-analysis began to be used in 2019 as the quantitative design was found
to be the most used research method over the years.

Figure11.
Figure Trends of
11.Trends of research
research methods
methods in
in articles
articles on
on VR
VR in
in CS
CS in
in the
the past
past 10
10 years.
years.

3.2.2. The Most Preferred Data Collection Tools and Sampling Methods in Articles on the
3.2.2. The Most Preferred Data Collection Tools and Sampling Methods in Articles on the
Use of VR in Computer Science Education
Use of VR in Computer Science Education
Data collection tools and sampling methods in research conducted on VR in CS edu-
Data collection tools and sampling methods in research conducted on VR in CS
cation show
education thatthat
show the the
questionnaire (46%)
questionnaire remains
(46%) thethe
remains most
mostused
usedtool.
tool.However,
However,quite
quite aa
number of studies (23%) either did not conduct evaluation or did not specify what
number of studies (23%) either did not conduct evaluation or did not specify what method method
of
ofdata
data collection
collection was
was used.
used.
As
As shown in Figure 12,
shown in Figure 12, the
the use
use of
of interviews
interviews (13%)
(13%) is
is still
still growing
growing asas fewer
fewer studies
studies
have been seen to use the method.
have been seen to use the method.

3.2.3. Sample Populations and Sample Sizes in Articles on the Use of VR in Computer
Science Education
According to Figure 13, the most commonly used sample size in articles published
between 2011 and 2020 fell between 11–20 participants. Closely followed were 1–10 per-
sons and 51–100 people. Although other studies utilized samples between 21–50 and
101–200 respondents, a few studies did not specify the sample size they used.

107
Educ. Sci. 2021, 11, 142
Educ. Sci. 2021, 11, x FOR PEER REVIEW 15 of 25

Survey Observation Interview Questionnaire

Questionnaire and interview System generated data Primary data Not evaluated/unspecified

Educ. Sci. 2021, 11, x FOR PEER REVIEW 16 of 25

Figure 12. Data collection tools and sampling methods of articles on the use of VR in CS education.
Figure 12. Data collection tools and sampling methods of articles on the use of VR in CS education.

3.2.3. Sample Populations and Sample Sizes in Articles on the Use of VR in Computer
Science Education
According to Figure 13, the most commonly used sample size in articles published
between 2011 and 2020 fell between 11–20 participants. Closely followed were 1–10 per-
sons and 51–100 people. Although other studies utilized samples between 21–50 and 101–
200 respondents, a few studies did not specify the sample size they used.

Figure
Figure 13. Frequencyof
13. Frequency of use
use of
of sample
sample sizes
sizes in
in articles.
articles.

3.2.4. Most Preferred Data Analysis Methods in Articles on the Use of VR in Computer
Science Education 108

The findings show that most studies were performed using descriptive analysis re-
garding the most preferred data analysis conducted in studies focused on VR in CS edu-
cation.
 Figure 13. Frequency of use of sample sizes in articles.
Educ. Sci. 2021, 11, 142

3.2.4. Most Preferred Data Analysis Methods in Articles on the Use of VR in Computer
Science Education
3.2.4.
TheMost Preferred
findings Datathat
show Analysis
most Methods in Articles
studies were on the Use
performed of VR
using in Computer
descriptive analysis re-
Science Education
garding the most preferred data analysis conducted in studies focused on VR in CS edu-
The findings show that most studies were performed using descriptive analysis regard-
cation.
ing the most preferred data analysis conducted in studies focused on VR in CS education.
Other preferred analysis methods, as shown in Figure 14, are meta-analysis and con-
Other preferred analysis methods, as shown in Figure 14, are meta-analysis and
tent analysis.
content Moreover,
analysis. some
Moreover, somestudies adopteda atheoretical
studies adopted theoretical approach
approach whilewhile
some some
other other
studies did not conduct any form of research, and therefore we categorized these types of
studies did not conduct any form of research, and therefore we categorized these types of
studiesas
studies as“others/not
“others/not specified”.
specified”.

Figure14.
Figure Mostpreferred
14.Most preferred data
dataanalysis
analysismethod
methodbetween 20112011
between and and
2020.2020.

4. Discussion
The bibliometric method’s potential is seen by earlier research [4]. It was opined that
bibliometric study advances complement meta-analysis and qualitative research for the
scientific evaluation of literature. This study delved into VR’s role in CS education to
provide a deeper understanding of the evolution of research conducted in this field and
anticipate the future direction on the basis of the analysis of the co-occurrence pattern
of keywords used in studies conducted in the last 10 years. The study contributes to
knowledge by presenting valuable findings that can boost the morale of prolific scholars
who have been contributing to this field and researchers and practicing managers who
may be starting to research into VR for CS education. This current study obtained its
bibliometric and content analysis data from the Web of Science and Scopus databases.
The bibliometric analysis of articles related to the use of VR in CS education, together
with the methodological research trends over the last 10 years, was revealed. Bibliometric
analysis results showed that the year 2011 was the highest in article production (148 articles).
This result was closely followed by the year 2018 with 135 articles. This finding implies
that between 2012 and 2017, articles related to VR in CS education dwindled. Regarding
the authors’ production over time, Li Y. had the highest number of articles produced in the
field, which is not surprising as the author consistently published in 2011–2012, 2015, and
2017–2020.
Moreover, we analyzed studies’ impact by investigating the number of citations
obtained by authors within 10 years. The analysis was focused on the m-index of each
author. Considering the 10 years duration in this study, we calculated the m-index by
dividing the total number of citations by the total number of years authors have been
109
Educ. Sci. 2021, 11, 142

publishing. For example, Dengel A. emerged as the most impactful author because this
author had produced one paper per year for only two years. This means that Dengel’s
impact analysis was computed on the basis of the output of these two years. However, it
was surprising to discover that Li Y., who had the highest number of articles produced
over the years, was not as impactful as Dengel A., who had a limited number of articles
published within just two years. Earlier studies have examined intrinsic factors affecting
the number of citations of articles [22,23]; however, some indicators are not directly related
to the quality or content of articles’ extrinsic factors [24]. The previous finding reveals that
price index, number of references, keywords, and length of studies are essential explanatory
factors [24]. It can be concluded that it is likely that Li’s articles are easily accessible to
researchers via open access medium. The relevancy of their topic or even the quality of their
paper in terms of content and presentation may account for the citations and rapid impact.
Regarding the institutions and countries contributing to VR in CS education, the
results further showed that the University of Southern California, USA; Aalborg University,
Denmark; and the University of Rennes, France, remain the top universities in terms
of publishing VR in CS education articles. On the other hand, the USA emerged as the
most productive country. However, other countries from Europe (France, Denmark, Italy,
the UK, and Germany) and Asia (China) are making a significant contribution towards
advancing CS education using VR technology. The co-occurrence pattern of authors’
keywords revealed that VR characteristics are leveraged for CS education. For example,
immersion, presence, interaction, and gamification are being explored in advancing CS
education [1,16,18]. Moreover, these keywords also form the research hotspots in VR,
primarily to support learning. Therefore, this study anticipates that VR in CS education
would continue to be researched within the scope of these keywords [14].
The content analysis results showed that quantitative studies (47%) dominate the
studies in terms of research methodology. The reason for quantitative method preference
may be due to the simplified way of presenting quantitative research, as well as less time
and effort required to conduct and analyze quantitative data [25]. It might also be the case
that the generalization and replicability that the quantitative approach provides accounts
for its dominance in the studies. The percentage for the use of mixed methods studies
was meager, reflecting that the use of mixed approach studies presents methodological
difficulties and challenges [12]. It is safe to conclude that only a few studies consider the
potential of mixed-method research, which adds rigor and validity to research through
triangulation and convergence of multiple and different sources of information [26,27].
Moreover, few qualitative studies have been conducted in the last 10 years. This may have
been due to the rigor and non-use of numbers, making it difficult to simplify findings
and observations [25]. On the contrary, Johnson and Christensen [28] assert that reliance
on collecting non-numerical primary data such as words and pictures makes qualitative
research well-suited for providing factual and descriptive information.
Regarding the frequency of the sampling size utilized over the years, the most used
sample sizes were 11–20. We were surprised to find out that most published articles on
VR in CS education were evaluated with about 11 to 20 participants. Since the research
method’s preference was quantitative research, we expected that many studies would have
used more participants to arrive at a generalized outcome. Although studies that used
51–100 sample sizes were also seen in the result, one could have thought that 20 participants
may be too small for a quantitative study. According to Faber and Fonseca [29], very small
samples undermine the internal and external validity, while huge samples tend to transform
minor differences into statistically significant differences.
Our findings revealed that the questionnaire is the most used data collection tool, while
descriptive analysis remains the preferred data analysis method. One way to reflect on this
result is that the questionnaire seems more straightforward, quicker, and cost-effective to
collect data from participants. Moreover, the preference for descriptive analysis may be
used to simplify data efficiently [30]. The researcher may have adopted this data analysis
method to reduce the time and effort required to format and present beneficial, easily

110
Educ. Sci. 2021, 11, 142

interpretable results to practitioners, policymakers, and other researchers to understand a

phenomenon better.

5. Conclusions
This study provides a comprehensive view of scientific papers on VR in CS education
published in peer-reviewed journals and conferences between 2011 and 2020. Two main
approaches were explored to answer the research questions presented in this study. First,
the bibliometric analysis answered the questions regarding the article production growth
in the field within a decade, prolific scholars and their affiliations publishing to advance
VR in CS education, and research hotspots in the field may guide scholar’s future research
focus. Second, content analysis of articles that met the inclusion criteria for this study
was analyzed to provide a methodological overview of studies conducted on VR in CS
education. Several findings were presented in this study. These findings show that VR
research for CS education has fared well; however, some of the years (between 2013 and
2016) witnessed low article production. The study also revealed the prolific scholars
and authors’ impact analysis in this field and provided insightful information regarding
research hotspots by analyzing the authors’ keywords co-occurrence.
Regarding the scientific methodology and data sampling technique used by studies on
VR in CS education, the most preferred is the quantitative method. At the same time, the
questionnaire was the most used data collection technique. Moreover, descriptive analysis
was mainly used to analyze data in studies on VR in CS education.
This study witnessed a limitation regarding the content analysis. It would be inter-
esting to see the educational context where VR technology is being used and the learning
contents deployed in the VR application for CS education. Nonetheless, this study con-
tributes to knowledge in significant ways. The study revealed that pedagogical approaches
such as game-based learning and gamification were explored for VR education in CS edu-
cation. The findings from this study can provide insight into how VR technology research
has progressed in a decade. Moreover, the result can be generalized since this study could
obtain relevant data from two databases (WoS and Scopus) to conduct its analysis. The
process for merging these data is another contribution as scholars interested in running a
similar study would find this helpful study. Our future study would address the limitations
by providing answers regarding the pedagogy, content, and context of studies on VR in
CS education.
By implication, we conclude that findings from this study suggest that even though
scholars are leveraging VR to advance teaching and learning in the field of CS, more effort
needs to be made, especially from continents, countries, and institutions that were not
reported among the top-20 list revealed in this study. In addition, a more rigorous method-
ological approach needs to be employed in a future study to provide more evident-based
research output. For example, our study revealed only a few studies that used a mixed-
methods approach, which has been more rigorous in terms of quality of scientific research.

Author Contributions: Conceptualization, F.J.A., I.T.S.; Data curation, F.J.A., I.T.S.; Formal analysis,
F.J.A., I.T.S., S.S.O. and J.S.; Investigation, F.J.A., I.T.S. and S.S.O.; Methodology, F.J.A., I.T.S., S.S.O.
and J.S.; Project administration, F.J.A.; Resources, F.J.A., I.T.S.; Software, F.J.A., I.T.S., S.S.O. and
J.S.; Supervision, S.S.O. and J.S.; Validation, S.S.O. and J.S.; Writing—original draft, F.J.A. and I.T.S.;
Writing—review & editing, F.J.A., I.T.S., S.S.O. and J.S. All authors have read and agreed to the
published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Data sharing is not applicable to this article.
Conflicts of Interest: The authors declare no conflict of interest.

111
Educ. Sci. 2021, 11, 142

Appendix A

Table A1. Published articles contained in the content analysis of VR for CS education (2011–2020).

Authors Aim of the Study Results of the Study

Virtual reality (VR) programming environment
called VRASP was developed allow students to Findings from the study show that students
Nguyen et al. [31] produce an avatar (agent) in a virtual world were able to communicate with the environment
that is able to answer questions in spoken intuitively with an accuracy of 78%.
natural language.
The study focused on conducting an
The authors anticipated that findings can be
experiment with the virtual reality mobile app
Srimadhaven et al. [32] useful to higher education students and enhance
in order to assess the cognitive level of the
the performance of all levels of learners.
students in a Python course.
This study presented a VR-based learning The authors envisaged that the designed game
game to support the teaching and learning of would spark interest for learning CS
Bouali et al. [33]
object-oriented programming (OOP) concepts programming concepts such as IF condition,
in computing education. Arrays, and Loops.
This study demonstratred how metaphorical
The study anticipated measuring students’
representations in VR can enhance the
Dengel [34] cognition, presence, usability, and satisfaction
understanding of theoretical computer science
in their future study.
concepts by using the Treasure Hunt game.
This study presented an immersion 3D
environment in the form of a video game. The The authors anticipated a positive impact of
Bolivar et al. [35] environment offers the player the opportunity the framework when their future research
to explore basic CS concepts without removing is completed.
any of the entertaining aspects of games.
This authors developed a virtual reality The study presented several cases and sample
Parmar et al. [36] tool—VEnvI—to support CS students in projects developed to assist teachers in
learning about the fundamental of CS. their classes.
This study proposed a virtual environment The author envisaged that this approach could
Kerdvibulvech [37]
framework for human–computer interaction. provide significant ducational values.
The authors have developed curriculum
materials for several disciplines both for
Demonstration and evaluation of the tool was
Rodger et al. [38] student and teacher use. The curriculum
expected to produce useful outcome.
materials include tutorials, sample projects,
and challenges for teaching CS topics.
This study aimed to set a medium of This study was still a work in progress, and
Vallance [39]
collaboration within a 3D virtual world. hence a concrete result was not presented.
This study designed and implemented Dr.
The study measured students’ cognition,
Chestr, a virtual human in a virtual
presence, usability, and satisfaction and found
Arrington et al. [40] environment game aimed at supporting the
that students enjoyed the experience and were
understanding and retention of introductory
successfully engaged the virtual world.
programming cources.
The study found that when receiving lectures
This study present a VR application where the in a virtual environment by a teacher, the child
Vanderdonckt and Vatavu [41] user, a psychologist, controls a virtual puppet was calm, focused, and capable of working on
(a cartoon-like character in VR). his assignments without showing any
disruptive behaviors.
The authors developed a VR tool—VEnvI—to
Participants who tested the VR tool agreed
support CS students in learning about the
Parmar et al. [42] that the visual aspect improved the overall
fundamental CS concepts such as sequences,
learning experience.
loops, variables, conditionals, and functions.

112
Educ. Sci. 2021, 11, 142

Table A1. Cont.

Authors Aim of the Study Results of the Study

This study investigated the feasibility of using The study provided guidelines to educators
VR to reduce disruptive classroom behavior of and instructional designers who wish to offer
Adjorlu and Serafin [43]
a child diagnosed with autism spectrum interactive and engaging learning activities to
disorder (ASD). their students.
A VR educational platform MYR was built to
spark student interest in computer science by
Evaluation with CS students shows that MYR
allowing them to write code that generates
is hard for CS students to provide clear 3D
three-dimensional, animated scenes in virtual
Berns et al. [44] representations for programming concepts;
reality environment. The goal of the project
however, the study was able to derive some
was to gain insight into computing students’
common figures.
success, motivation, and confidence in
learning computing.
Authors investigared what effect instructional
Evaluation of this tool suggests that users’
design decisions have on motivation and
Christopoulos et al. [45] experience is enhanced through the
engagement of students learning in virtual and
3D animation.
physical world.
The study developed a 3D virtual
programming language to provide an The study reported that the method creates fun
Ortega et al. [46]
interactive tool for beginners and intermediate and effective means of interdisciplinary study.
students to learn programming concepts.
This study proposed a virtual 3D tool
(touchless interface) to support people
Feedback from the workshop participants
Sanna et al. [47] without any prior knowledge in code
generally shows that they had a good experience.
writing to promote user friendliness and
usability experience.
This study explored a style of teaching youths The study tested educational virtual
Cleary et al. [48] how to write computer program using reactive environments (EVEs) with pre- and post-test
programming in a 3D virtual environment. and found to be significantly effective.
The authors created “Move the World”
Overall comments from participants of the
workshop in a summer camp to increase high
Domik et al. [49] workshop revealed that learning in the virtual
school juniors’ interest in computer science by
world is appealing and inspiring.
leveraging math and virtual worlds.
The study discusses students’ preconceptions
The study modeled three computer scienc
towards the inclusion of 3D virtual learning
topics- asymmetric encryption/decryption,
Dengel [50] environments in the context of their studies
and finite state machines in a 3 D immersive
and further elicit their thoughts related to the
VR to teach these topics.
impact of the “hybrid” interactions
The study reported positive impact on
This study used a VR game (Mazes) to teache computer science education by increasing
Koltai et al. [51]
CS concepts. engagement, knowledge acquisition, and
self-directed learning.
The study generally indicated that participants
perceived a high joy of use while playing
This authors developed a tool—FunPlogs FunPlogs, which indicated that despite the
Christopoulos et al. [52]
application—to deply microlearning. simple game concept, complex matters as the
while-loop could be transported to
programming laymen.
The study explored a summer course that uses
visual design problem-based learning The study concluded that interactions in VR
Banic and Gamboa [53]
pedagogy with virtual environments as a plays a crucial role in learner engagement.
strategy to teach computer science.
This study introduced a VR puzzle The study outcome shows that the proposed
Horst et al. [54] mini-game for learning fundamental module helps students learn stacks and queues
programming principles. while being satisfactorily usable.

113
Educ. Sci. 2021, 11, 142

Table A1. Cont.

Authors Aim of the Study Results of the Study

Authors examined the impact that the virtual
Results show that the self-overlapping maze is
reality learning process has on university
Christopoulos and Conrad [55] experienced as freely walkable while the map
students who study CS and have almost no
is mostly understandable.
experience in the use of virtual worlds.
This study designed a game theme-based
instructional (GTI) module to teach The analysis of SEQ usability test shows
Stigall and Sharma [56]
undergraduate CS majors about stacks good acceptance.
and queues.
This study investigated how working with VR
Analysis of the usability and likeability of the
setups can be walkable in small physical
Serubugo et al. [57] survey shows that students felt motivated and
spaces or included in non-HMD participants
engaged in learning programming concepts.
using self-overlapping maze
The study reported that students gained
This study designed a VR tool to assist in cognitive thinking process and had a greater
Pilatásig et al. [58]
training and rehabilitation of hands and wrist range of expressing sufficiently alternative to
self-explanatory solutions.
This study designed a VR application
(VR-OCKS) to teach basic programming The initial evaluation of this tool shows that it
Segura et al. [59]
concepts such as flow statements and enhanced creative thinking of young children.
conditional selections.
Evaluation results demonstrated positive
The authors explored a 3D simulation game to student perceptions about the use of gaming
Pellas and Vosinakis [60]
teach computational problem-solving. instructional modules to advance student
learning and understanding of the concepts.
This study designed and developed two
Result analysis suggests that participants
gaming modules for teaching CS students
Stigall and Sharma [61] showed similar connectedness in affiliative
object-oriented programming (OOP) and
tour and competitive design.
binary search.
The study measured participants’ intentions
The authors developed virtual reality
toward majoring in a computing discipline,
instructional (VRI) modules for teaching loops
Sharma and Ossuetta [62] attitudes toward computing, and overall
and arrays that can provide a better
satisfaction with the camp, and showed
understanding of the concept.
positive indication.
This study proposed a VR exergaming
This study argued that comparative studies are
platform that combines a recumbent tricycle
a useful method for analyzing benefits of
Ijaz et al. [63] and real-world panoramic images where the
different approaches to controlling
player can navigate real locations in a safe
virtual agents.
virtual environment
This study demonstrated that familiarity may
This study reported the experience of a
reduce working memory load and increase
summer camp that introduced computing
Hulsey et al. [64] children’s spatial memory capacity for
concepts to middle school girls in the context
acquiring sequential temporal–spatial
of an online, multiplayer, virtual world.
information from virtual displays.
This study presents results of comparing the
Outcome of the experiment with CodeSpells
usability of an academic technique designed
shows that students were able to understand
Gemrot et al. [65] for programming intelligent agents’ behavior
and write basic Java code after only 8 h of
with the usability of an unaltered classical
playing the game.
programming language.
This study examined the potential use of Outcome of the study provide overview of the
Korallo et al. [66] virtual environment in general computer two reviewed approaches for implementing
knowledge in virtual environment. VR gestures, which may guide experts.

114
Educ. Sci. 2021, 11, 142

References
1. Oyelere, S.S.; Bouali, N.; Kaliisa, R.; Obaido, G.; Yunusa, A.A.; Jimoh, E.R. Exploring the trends of educational virtual reality
games: A systematic review of empirical studies. Smart Learn. Environ. 2020, 7, 1–22. [CrossRef]
2. Carruth, D.W. Virtual reality for education and workforce training. In Proceedings of the 2017 15th International Conference on
Emerging eLearning Technologies and Applications (ICETA), Stary Smokovec, Slovakia, 26–27 October 2017; pp. 1–6.
3. Pirker, J.; Dengel, A.; Holly, M.; Safikhani, S. Virtual Reality in Computer Science Education: A Systematic Review. In Proceedings
of the 26th ACM Symposium on Virtual Reality Software and Technology; Association for Computing Machinery (ACM), Tokyo,
Japan, 1–4 November 2020; pp. 1–8.
4. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Tukiainen, M. Scientific production and thematic breakthroughs in smart learning environ-
ments: A bibliometric analysis. Smart Learn. Environ. 2021, 8, 1–25. [CrossRef]
5. Agbo, F.J.; Oyelere, S.S.; Bouali, N. A UML approach for designing a VR-based smart learning environment for programming
education. In Proceedings of the 2020 IEEE Frontiers in Education Conference (FIE), Uppsala, Sweden, 21–24 October 2020;
pp. 1–5.
6. Agbo, F.J.; Oyelere, S.S. Smart mobile learning environment for programming education in Nigeria: Adaptivity and context-
aware features. In Intelligent Computing-Proceedings of the Computing Conference; Springer: Berlin/Heidelberg, Germany, 2019;
pp. 1061–1077.
7. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Tukiainen, M. Identifying potential design features of a smart learning environment for
programming education in Nigeria. Int. J. Learn. Technol. 2019, 14, 331–354. [CrossRef]
8. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Tukiainen, M. Smart learning environment for computing education: Readiness for
implementation in Nigeria. In Proceedings of the EdMedia+ Innovate Learning, Association for the Advancement of Computing
in Education (AACE), Amsterdam, The Netherlands, 24–28 June 2019; pp. 1382–1391.
9. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Adewumi, S. A systematic review of computational thinking approach for programming
education in higher education institutions. In Proceedings of the 19th Koli Calling International Conference on Computing
Education Research, Koli, Finland, 21–24 November 2019; pp. 1–10.
10. Zupic, I.; Čater, T. Bibliometric methods in management and organisation. Organ. Res. Methods 2015, 18, 429–472. [CrossRef]
11. Arici, F.; Yildirim, P.; Caliklar, Ş.; Yilmaz, R.M. Research trends in the use of augmented reality in science education: Content and
bibliometric mapping analysis. Comput. Educ. 2019, 142, 103647. [CrossRef]
12. Buchholtz, N. Planning and conducting mixed methods studies in mathematics educational research. In Compendium for Early
Career Researchers in Mathematics Education ICME-13 Monographs; Kaiser, G., Presmeg, N., Eds.; Springer: Cham, Switzerland, 2019;
pp. 131–152.
13. Tüysüz, C. The Effect of the Virtual Laboratory on Students’Achievement and Attitude in Chemistry. Int. Online J. Educ. Sci. 2010,
2, 37–53.
14. Cheng, K.; Tsai, C. Students’ motivational beliefs and strategies, perceived immersion and attitudes towards science learning
with immersive virtual reality: A partial least squares analysis. Br. J. Educ. Technol. 2020, 51, 2140–2159. [CrossRef]
15. Bogusevschi, D.; Muntean, C.; Muntean, G.M. Teaching and Learning Physics using 3D Virtual Learning Environment: A Case
Study of Combined Virtual Reality and Virtual Laboratory in Secondary School. J. Comput. Math. Sci. Teach. 2020, 39, 5–18.
16. Huang, H.-M.; Liaw, S.-S. An Analysis of Learners’ Intentions Toward Virtual Reality Learning Based on Constructivist and
Technology Acceptance Approaches. Int. Rev. Res. Open Distrib. Learn. 2018, 19. [CrossRef]
17. Lorenzo, G.; Lorenzo-Lledó, A.; Lledó Carreres, A.; Pérez-Vázquez, E. Approach from a Bibliometric Perspective of the Educational
Application of Virtual Reality in People with Autism Spectrum Disorder. Educ. Knowl. Soc. 2020, 21, 4–14.
18. Sobral, M.; Pestana, M.H. Virtual reality and dementia: A bibliometric analysis. Eur. J. Psychiatry 2020, 34, 120–131. [CrossRef]
19. Lai, N.Y.G.; Wong, K.H.; Yu, L.J.; Kang, H.S. Virtual Reality (VR) in Engineering Education and Training: A Bibliometric
Analysis. In Proceedings of the 2020 2nd World Symposium on Software Engineering, San Francisco, CA, USA, 18–20 May 2020;
pp. 161–165.
20. Enebechi, C.N.; Duffy, V.G. Virtual Reality and Artificial Intelligence in Mobile Computing and Applied Ergonomics: A Biblio-
metric and Content Analysis. In Proceedings of the Constructive Side-Channel Analysis and Secure Design, Lugano, Switzerland,
1–3 April 2020; pp. 334–345.
21. Aria, M.; Cuccurullo, C. Bibliometrix: An R-tool for comprehensive science mapping analysis. J. Inf. 2017, 11, 959–975. [CrossRef]
22. Chen, C. Predictive effects of structural variation on citation counts. J. Am. Soc. Inf. Sci. Technol. 2011, 63, 431–449. [CrossRef]
23. Didegah, F.; Thelwall, M. Determinants of research citation impact in nanoscience and nanotechnology. J. Am. Soc. Inf. Sci.
Technol. 2013, 64, 1055–1064. [CrossRef]
24. Onodera, N.; Yoshikane, F. Factors affecting citation rates of research articles. J. Assoc. Inf. Sci. Technol. 2015, 66, 739–764.
[CrossRef]
25. Eyisi, D. The Usefulness of Qualitative and Quantitative Approaches and Methods in Researching Problem-Solving Ability in
Science Education Curriculum. J. Educ. Pract. 2016, 7, 91–100.
26. Jokonya, O. The significance of mixed methods research in information systems research. In Proceedings of the Midwest
Association for Information Systems Conference (MWAIS), Springfield, IL, USA, 19–20 May 2016.
27. Regnault, A.; Willgoss, T.; Barbic, S. Towards the use of mixed methods inquiry as best practice in health outcomes research.
J. Patient-Rep. Outcomes 2018, 2, 19. [CrossRef] [PubMed]

115
Educ. Sci. 2021, 11, 142

28. Johnson, R.B.; Christensen, L.B. Educational Research: Quantitative, Qualitative, and Mixed Approaches, 6th ed.; SAGE Publications:
Thousand Oaks, CA, USA, 2017.
29. Faber, J.; Fonseca, L.M. How sample size influences research outcomes. Dent. Press J. Orthod. 2014, 19, 27–29. [CrossRef]
30. Loeb, S.; Dynarski, S.; McFarland, D.; Morris, P.; Reardon, S.; Reber, S. Descriptive Analysis in Education: A Guide for Researchers.
NCEE 2017-4023 Natl. Cent. Educ. Eval. Reg. Assist. 2017, 1–40.
31. Nguyen, V.T.; Zhang, Y.; Jung, K.; Xing, W.; Dang, T. VRASP: A Virtual Reality Environment for Learning Answer Set Pro-
gramming. In International Symposium on Practical Aspects of Declarative Languages; Springer: Berlin/Heidelberg, Germany, 2020;
pp. 82–91.
32. Srimadhaven, T.; AV, C.J.; Harshith, N.; Priyaadharshini, M. Learning analytics: Virtual reality for programming course in higher
education. Procedia Comput. Sci. 2020, 172, 433–437.
33. Bouali, N.; Nygren, E.; Oyelere, S.S.; Suhonen, J.; Cavalli-Sforza, V. Imikode: A VR Game to Introduce OOP Concepts.
In Proceedings of the 19th Koli Calling International Conference on Computing Education Research 2019, Koli, Finland,
21–24 November 2019; pp. 1–2.
34. Dengel, A. Seeking the treasures of theoretical computer science education: Towards educational virtual reality for the visualiza-
tion of finite state machines. In Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning
for Engineering (TALE), Wollong, Australia, 4–7 December 2018; pp. 1107–1112.
35. Bolivar, S.; Perez, D.; Carrasquillo, A.; Williams, A.S.; Rishe, N.D.; Ortega, F.R. 3D Interaction for Computer Science Educational
VR Game. In International Conference on Human-Computer Interaction; Springer: Berlin/Heidelberg, Germany, 2019; pp. 408–419.
36. Parmar, D.; Babu, S.V.; Lin, L.; Jörg, S.; D’Souza, N.; Leonard, A.E.; Daily, S.B. Can embodied interaction and virtual peer
customization in a virtual programming environment enhance computational thinking? In Proceedings of the 2016 Research on
Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT), Atlanta, GA, USA, 11–13 August
2016; pp. 1–2.
37. Kerdvibulvech, C. Vision and virtual-based human computer interaction applications for a new digital media visualization.
In Proceedings of the WSCG 2015 Conference on Computer Graphics, Visualization and Computer Vision, Pilsen near Prague,
Czech Republic, 8–12 June 2015; pp. 247–254.
38. Rodger, S.H.; Brown, D.; Hoyle, M.; MacDonald, D.; Marion, M.; Onstwedder, E.; Ward, E. Weaving computing into all middle
school disciplines. In Proceedings of the 2014 Conference on Innovation Technology in Computer Science Education 2014,
Uppsala, Sweden, 21–25 June 2014; pp. 207–212.
39. Vallance, M. The affect of collaboratively programming robots in a 3D virtual simulation. In Proceedings of the 2013 8th
ACM/IEEE International Conference on Human-Robot Interaction (HRI), Tokyo, Japan, 3–6 March 2013; pp. 245–246.
40. Arrington, C., Jr.; Wilson, D.M.; Lehmann, L. Improving performance and retention in computer science courses using a virtual
game show. In Proceedings of the 49th Annual Southeast Regional Conference 2011, Kennesaw, GA, USA, 24–26 March 2011;
pp. 320–321.
41. Vanderdonckt, J.; Vatavu, R.D. A pen user interface for controlling a virtual puppet. In Proceedings of the 12th ACM SIGCHI
Symposium on Engineering Interactive Computing Systems 2020, Sophia Antipolis, France, 23–26 June 2020; pp. 1–6.
42. Parmar, D.; Isaac, J.; Babu, S.V.; D’Souza, N.; Leonard, A.E.; Jörg, S.; Daily, S.B. Programming moves: Design and evaluation
of applying embodied interaction in virtual environments to enhance computational thinking in middle school students.
In Proceedings of the 2016 IEEE Virtual Reality (VR), Greenville, SC, USA, 19–23 March 2016; pp. 131–140.
43. Adjorlu, A.; Serafin, S. Head-mounted display-based virtual reality as a tool to reduce disruptive behavior in a student diagnosed
with autism spectrum disorder. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer: Berlin/Heidelberg,
Germany, 2019; pp. 739–748.
44. Berns, C.; Chin, G.; Savitz, J.; Kiesling, J.; Martin, F. Myr: A web-based platform for teaching coding using VR. In Proceedings of
the 50th ACM Technical Symposium on Computer Science Education 2019, Minneapolis, MN, USA, 27 February–2 March 2019;
pp. 77–83.
45. Christopoulos, A.; Conrad, M.; Shukla, M. Learner experience in hybrid virtual worlds: Interacting with pedagogical agents.
In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019), Heraklion, Crete, Greece,
2–4 May 2019; pp. 488–495.
46. Ortega, F.R.; Bolivar, S.; Bernal, J.; Galvan, A.; Tarre, K.; Rishe, N.; Barreto, A. Towards a 3D virtual programming language to
increase the number of women in computer science education. In Proceedings of the 2017 IEEE Virtual Reality Workshop on K-12
Embodied Learning through Virtual Augmented Reality (KELVAR), Los Angeles, CA, USA, 19 March 2017; pp. 1–6.
47. Sanna, A.; Lamberti, F.; Bazzano, F.; Maggio, L. Developing touch-less interfaces to interact with 3D contents in public exhibitions.
In International Conference on Augmented Reality, Virtual Reality and Computer Graphics; Springer: Berlin/Heidelberg, Germany,
2016; pp. 293–303.
48. Cleary, A.; Vandenbergh, L.; Peterson, J. Reactive game engine programming for stem outreach. In Proceedings of the 46th ACM
Technical Symposium on Computer Science Education 2015, Kansas City, MO, USA, 4–7 March 2015; pp. 628–632.
49. Domik, G.; Arens, S.; Stilow, P.; Friedrich, H. Helping High Schoolers Move the (Virtual) World. IEEE Comput. Graph. Appl. 2013,
33, 70–74. [CrossRef]
50. Dengel, A. How Important is Immersion for Learning in Computer Science Replugged Games? In Proceedings of the 51st ACM
Technical Symposium on Computer Science Education 2020, New York, NY, USA, 11–14 March 2014; pp. 1165–1171.

116
Educ. Sci. 2021, 11, 142

51. Koltai, B.G.; Husted, J.E.; Vangsted, R.; Mikkelsen, T.N.; Kraus, M. Procedurally Generated Self Overlapping Mazes in Virtual
Reality. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer International Publishing: Berlin/Heidelberg,
Germany, 2020; pp. 229–243.
52. Christopoulos, A.; Conrad, M.; Shukla, M. What Does the Pedagogical Agent Say? In Proceedings of the 2019 10th International
Conference on Information, Intelligence, Systems and Applications (IISA), Piraeus, Greece, 15–17 July 2019; pp. 1–7.
53. Banic, A.; Gamboa, R. Visual Design Problem-based Learning in a Virtual Environment Improves Computational Thinking and
Programming Knowledge. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka,
Japan, 23–27 March 2019; pp. 1588–1593.
54. Horst, R.; Naraghi-Taghi-Off, R.; Diez, S.; Uhmann, T.; Müller, A.; Dörner, R. FunPlogs–A Serious Puzzle Mini-game for
Learning Fundamental Programming Principles Using Visual Scripting. In International Symposium on Visual Computing; Springer
International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 494–504.
55. Christopoulos, A.; Conrad, M.; Shukla, M. Increasing student engagement through virtual interactions: How? Virtual Real. 2018,
22, 353–369. [CrossRef]
56. Stigall, J.; Sharma, S. Usability and Learning Effectiveness of Game-Themed Instructional (GTI) Module for Teaching Stacks and
Queues. SoutheastCon 2018, 1–6. [CrossRef]
57. Serubugo, S.; Škantárová, D.; Evers, N.; Kraus, M. Self-overlapping Maze and Map Design for Asymmetric Collaboration in
Room-Scale Virtual Reality for Public Spaces. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer International
Publishing: Berlin/Heidelberg, Germany, 2018; pp. 194–203.
58. Pilatasig, M.; Tigse, J.; Chuquitarco, A.; Pilatasig, P.; Pruna, E.; Acurio, A.; Buele, J.; Escobar, I. Interactive System for Hands and
Wrist Rehabilitation. In Advances in Intelligent Systems and Computing; Springer International Publishing: Berlin/Heidelberg,
Germany, 2018; pp. 593–601.
59. Segura, R.J.; Del Pino, F.J.; Ogáyar, C.J.; Rueda, A.J. VR-OCKS: A virtual reality game for learning the basic concepts of
programming. Comput. Appl. Eng. Educ. 2020, 28, 31–41. [CrossRef]
60. Pellas, N.; Vosinakis, S. Learning to Think and Practice Computationally via a 3D Simulation Game. In Interactive Mobile
Communication, Technologies and Learning; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 550–562.
61. Stigall, J.; Sharma, S. Virtual reality instructional modules for introductory programming courses. In Proceedings of the 2017
IEEE Integrated STEM Education Conference (ISEC), Princeton, NJ, USA, 11 March 2017; pp. 34–42.
62. Sharma, S.; Ossuetta, E. Virtual Reality Instructional Modules in Education Based on Gaming Metaphor. Electron. Imaging 2017,
2017, 11–18. [CrossRef]
63. Ijaz, K.; Wang, Y.; Milne, D.; Calvo, R.A. VR-Rides: Interactive VR Games for Health. In Joint International Conference on Serious
Games; Springer International Publishing: Berlin/Heidelberg, Germany, 2016; Volume 9894, pp. 289–292.
64. Hulsey, C.; Pence, T.B.; Hodges, L.F. Camp CyberGirls: Using a virtual world to introduce computing concepts to middle school
girls. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education, Atlanta, GA, USA, 5 March 2014;
pp. 331–336.
65. Gemrot, J.; Hlavka, Z.; Brom, C. Does High-Level Behavior Specification Tool Make Production of Virtual Agent Behaviors
Better? In International AAMAS Workshop on Cognitive Agents for Virtual Environments; Springer: Berlin/Heidelberg, Germany,
2013; Volume 7764, pp. 161–183.
66. Korallo, L.; Foreman, N.; Boyd-Davis, S.; Moar, M.; Coulson, M. Do challenge, task experience or computer familiarity influence
the learning of historical chronology from virtual environments in 8–9 year old children? Comput. Educ. 2012, 58, 1106–1116.
[CrossRef]

117
 MDPI
St. Alban-Anlage 66
4052 Basel
Switzerland
Tel. +41 61 683 77 34
Fax +41 61 302 89 18
www.mdpi.com

Education Sciences Editorial Office

E-mail: [email protected]
www.mdpi.com/journal/education
MDPI
St. Alban-Anlage 66
4052 Basel
Switzerland
Tel: +41 61 683 77 34
Fax: +41 61 302 89 18
www.mdpi.com ISBN 978-3-0365-2315-6