TECHNICAL REPORT 2005-004

THE EFFECTIVENESS OF
INSTRUCTIONAL GAMES:
A LITERATURE REVIEW
AND DISCUSSION

NOVEMBER 2005

Robert T. Hays
Naval Air Warfare Center Training Systems Division

NAVAL AIR WARFARE CENTER

TRAINING SYSTEMS DIVISION
ORLANDO, FL 32826-3275
DISTRIBUTION STAT0,17ThTA
Approved for Public Release
APPROVED FOR PUBLIC RELEASE Distribution Unlimited
DISTRIBUTION IS UNLIMITED

1717 ... •/" • iretr•

"'ecnc BS, ietr

D. J. DWYER, Head, D. Sj YO G.J. L

Human Systems Research Technical irector Science & Technology Office
& Engineering Division
GOVERNMENT RIGHTS DATA STATEMENT
Reproduction of this publication in whole or in
part is permitted for any purpose of the United
States Government.

20051219 042
 Technical Report 2005-004
REPOT DCUMNTATON AGEForm Approved
REPOT DCUMETATON PGEOMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average I hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the
data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing
this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management
and Budget. Paperwork Reduction Project (0704-0188), Washington, DC 20503.

1. AGENCY USE ONLY (Leave Blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED
Technical Report
November 2005 July - October 2005
4. TITLE AND SUBTITLE 5. FUNDING NUMBERS
THE EFFECTIVENESS OF INSTRUCTIONAL GAMES: A LITERATURE
REVIEW AND DISCUSSION (U)

6. AUTHOR(S)
Robert T. Hays

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION

Naval Air Warfare Center Training Systems Division REPORT NUMBER

(AIR 4652) Technical Report

12350 Research Parkway 2005-004
Orlando, FL 32826-3275
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING
Naval Air Warfare Center Training Systems Division AGENCY REPORT NUMBER

Special Emphasis Programs (Code 1195)

12350 Research Parkway
Orlando, FL 32826-3275
11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

Approved for public release; distribution is unlimited. "A"

13. ABSTRACT (Maximum 200 words)

This report documents a review of 48 empirical research articles on the effectiveness of instructional games. It also includes
summaries of 26 other review articles and 31 theoretical articles on instructional gaming. Based on this review the following
5 conclusions and 4 recommendations are provided. Conclusions: (1) The empirical research on the instructional
effectiveness of games is fragmented, filled with ill defined terms, and plagued with methodological flaws. (2) Some games
provide effective instruction for some tasks some of the time, but these results may not be generalizable to other games or
instructional programs. (3) No evidence indicates that games are the preferred instructional method in all situations. (4)
Instructional games are more effective if they are embedded in instructional programs that include debriefing and feedback.
(5) Instructional support during play increases the effectiveness of instructional games. Recommendations: (1) The decision
to use a game for instruction should be based on a detailed analysis of learning requirements and tradeoffs among alternate
instructional approaches. (2) Program managers and procurement officials should insist that instructional game developers
demonstrate how their game will support instructional objectives. (3) Games should be used as adjuncts and aids, not as
stand-alone instruction. (4) Instructor-less approaches (e.g., web-based instruction) must include all "instructor functions."
14. SUBJECT TERMS 15. NUMBER OF PAGES
63
16. PRICE CODE

17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF ABSTRACT
OF REPORT OF THIS PAGE OF ABSTRACT
UNCLASSIFIED UNCLASSIFIED UNCLASSIFIED

NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89)

Prescribed by ANSI Std 239-18

298-102

3
 Technical Report 2005-004

THIS PAGE INTENTIONALLY LEFT BLANK

 Technical Report 2005-004

EXECUTIVE SUMMARY

PROBLEM

The Navy, DoD, and private industry need new, innovative and effective instructional
methods and media to ensure that personnel have the necessary skills to perform in today's high
technology, rapidly changing environment. There is a growing advocacy for the use of
instructional games as a primary delivery medium for this required instruction. However, the
decision to use instructional games is often made on the basis of "leaps of faith" rather than
empirical data on the effectiveness of instructional games.

OBJECTIVE
The objective of this effort was twofold:
* To conduct a review of the literature on instructional games with a focus on the empirical
research on the instructional effectiveness of games.
* To document conclusions and recommendations from this literature review.

APPROACH

This effort proceeded through the following steps:

* Researchers in the field of instructional games were contacted to obtain literature reviews
either published or in progress.
"*The Internet was searched to obtain additional documents.
"*Review articles were read to identify empirical research articles.
"*Articles that documented empirical data on the instructional effectiveness of games were
obtained.
"*The empirical articles were read and summarized.
"*The empirical data were used to draw conclusions and recommendations about the design
and use of games as instructional media.

RESULTS

Two hundred and seventy-four documents on the design, use, and evaluation of games were
reviewed. Of these, 169 could not be used:
e 77 of these articles were not used because they only provided the author's opinion on the
potential of instructional games
* 11 articles were on game design and analysis
* 29 articles were simulation studies (not games)
* 14 articles were computer-based instruction studies (not games)
* 36 articles documented studies with major methodological problems, and
* 2 were proposals for future research.

One hundred and five articles are documented in this report. These included:
* 26 review articles

5
 Technical Report 2005-004

* 31 theoretical articles, and

* 48 articles that provided empirical data on the effectiveness of instructional games.

CONCLUSIONS AND RECOMMENDATIONS

This review of empirical research on the effectiveness of instructional games leads to the
following five conclusions and four recommendations. The conclusions are:
* The empirical research on the effectiveness of instructional games is fragmented. The
literature includes research on different tasks, age groups, and types of games. The research
literature is also filled with ill defined terms, and plagued with methodological flaws.
9 Although research has shown that some games can provide effective learning for a variety of
learners for several different tasks (e.g., math, attitudes, electronics, and economics), this does
not tell us whether to use a game for our specific instructional task. We should not generalize
from research on the effectiveness of one game in one learning area for one group of learners
to all games in all learning areas for all learners.
o There is no evidence to indicate that games are the preferred instructional method in all
situations.
o Instructional games should be embedded in instructional programs that include debriefing
and feedback so the learners understand what happened in the game and how these events
support the instructional objectives.
o Instructional support to help learners understand how to use the game increases the
instructional effectiveness of the gaming experience by allowing learners to focus on the
instructional information rather than the requirements of the game.

The following four recommendations may help the instructional gaming industry produce
more instructionally effective games.
o The decision to use a game should be based on a detailed analysis of the learning
requirements and an analysis of the tradeoffs among alternate instructional approaches.
o Program managers and procurement personnel should insist that game developers clearly
demonstrate how the design of a game will provide interactive experiences that support
properly designed instructional objectives (see for example, Gagn6 & Briggs, 1979; Merrill,
1983; 1997 for guidance on the proper design of instructional objectives).
* Instructors should view instructional games as adjuncts and aids to help support instructional
objectives. Learners should be provided with debriefing and feedback that clearly explains
how their experiences with the game help them meet these instructional objectives.
o Instructor-less approaches (e.g., web-based instruction) must include all "instructor
functions." These include performance evaluation, debriefing, and feedback.

6
 Technical Report 2005-004

TABLE OF CONTENTS

Page
Introduction ............ ............ ................................. 9

Problem ................................................................. 9
Objective............................................................... 9
Organization of the Report ................................................... 9
Confusion in Term s ......................................................... 9

Descriptions and Classifications of Games .............................. 11

H ybrid G ames ............................................................. 13

Elem ents of Enjoym ent ...................................................... 14
W orking Definition of a Game ................................................ 15

Instructional Uses of Games ...................................... 17

Research on Instructional Games ....................................... 19

Review Articles on the Effectiveness of Instructional Games ......................... 19

Summaries of Research on Specific Instructional Games ........................... 24
Games Used in Schools (Pre-K-12)......................................... 24
Games Used in College -........ .......................................... 29
Games Used in the Workplace ... ........................................... 33
Other Issues in Instructional Gaming... ........................................ 37
Types of Competition in a Game ........................................... 37
Adding Instructional Support to a Game .. .................................... 38
What is Learned in an Adventure Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
The Danger of Evaluator Interest......................................... 39
The Instructional Value of a Game Developed for Another Purpose ................. 40
The Importance of Debriefing .............................................. 40

Summary of Empirical Evidence on Instructional Games .............. 43

A Reexamination of the "Claims" ... ........................................ 46

To Game or Not to Game? .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Conclusions and Recommendations ................................ 53

References .............. ................................................ 55

7
 Technical Report 2005-004

Table of Contents
(Continued)

LIST OF FIGURES
Page
1. Relationship among Games, Simulations and Case Studies ....................... 14
2. Instructional Effectiveness as Degree of Overlap among Instructional Objectives
and Gam e Characteristics .................................................. 49
3. A Systematic Approach to Instructional Game Design and Use .................... 50

LIST OF TABLES
Page
1. Design Heuristics for Motivating Instructional Environments ..................... 18
2. Percentage of Articles by Game Function (Dempsey, et al., 1993-1994) ............. 22
3. Percentage of Articles by Learning Outcome (Dempsey, et al., 1993-1994) ........... 22
4. Summary of Empirical Effectiveness Studies Reviewed .......................... 44
5. Evidence in Support of the "Claims": Then and Now ............................ 47
6. Design Recommendations for Instructional Games .............................. 52
 Technical Report 2005-004

INTRODUCTION

PROBLEM

Recently there has been increased interest in the use of instructional games. It has been
argued that young people, raised playing video games, have changed in ways that turn them off
to conventional instruction (Prensky, 2001, p. 01-6). It has also been suggested that instructional
games, because of their applied and dynamic nature, can heighten the learner's motivation and
interest more effectively than classroom lectures (Greenblat, 1981, p. 147). Another suggested
benefit of instructional games is that they may improve the retention of learned skills and
knowledge (Pierfy, 1977).

Unfortunately, many decisions about whether to use games for instruction are based on
unfounded assumptions about the ability of games to provide effective and efficient instruction.
This problem was stated by a senior naval officer as follows. "Lack of quantifiable metrics or
data to validate use of gaming technology as an effective delivery media [sic] for various training
solutions has relegated decisions of the same to being 'leaps of faith' vice based on solid
business case solutions" (M. K. Gritton, personal communication, September 26, 2005).

OBJECTIVE

The objective of this report is to provide a review and discussion of empirical research on the
effectiveness of instructional games. Based on this review, a series of conclusions and
recommendations are provided.

ORGANIZATION OF THE REPORT

This report contains six sections. Following the introduction, the second section summarizes
a variety of descriptions and classifications of games. The third section discusses how games are
not always used for instructional purposes, but how the characteristics of games (e.g., challenge,
rules, and context) can motivate learners. The fourth section provides summaries of review
articles on the effectiveness of various games and empirical articles on the effectiveness of
specific games. The fifth section summarizes the empirical research and compares what we
know about instructional games today with several "claims" made about instructional games
twenty-five years ago. Finally, a series of conclusions and recommendations are provided in the
final section, followed by a complete reference list.

CONFUSION IN TERMS

Certainly, it can be argued that games can attract and hold an individual's attention. It is less
certain that they always provide instructional value. Before discussing the research on
instructional games, it is important to distinguish games from other instructional activities. In
the literature on instructional games, we often find the terms simulations, games, simulation-
games, and computer games used interchangeably (e.g., Greenblat & Duke, 1981; Reiber, 1996;
Thomas, Cahill & Santilli, 1997). Greenblat (1981) observed that "in many studies, 'games' and

9
 Technical Report 2005-004

'simulations' are at least implicitly treated as homogeneous" (p. 181). Although aware of the
problem, even Greenblat used the term simulation in one sentence and the term game in the next
sentence to describe the same thing (p. 144).

All simulations are based on models of reality. A model can be defined as "a physical,
mathematical, or otherwise logical representation of a system, entity, phenomenon, or process
(Department of Defense, 1997, p. 138). A simulation is "a method for implementing a model
over time" (p. 160). Thus, a model provides the rules and data that are used to represent some
portion of reality. Then, a simulation uses these rules and data to dynamically represent these
aspects of reality for some purpose. A specific type of simulation is sometimes called a
microworld(e.g., Miller, Lehman, & Koedinger, 1999). Microworlds are simulations that
attempt to capture the relevant aspects of some topic or phenomena so learners can interact
within it to observe the effects of their interactions. For our purposes, microworlds will be
referred to as simulations unless there is a reason to make a distinction between the two terms.
Simulations can incorporate aspects of games (e.g., competition, rules, rewards, etc.) or be used
as games, but only if the simulation includes these game aspects, should it be called a simulation
game.

One of the purposes of simulations can be to provide instruction, but only if they are designed
to do so. For example, Simons (1993) believes that "the educational value of a simulation comes
with repeated simulation of a model" (p. 137). In this type of application, the learner is able to
try various policies or parameters and observe the results. Through this closed-loop process, the
learner gradually builds an understanding of the simulated system. This is a powerful
instructional benefit of simulation games. Unfortunately, most of these games do not meet their
potential. Simons explains that simulation games are ineffective because they do not "directly
communicate their underlying models and the reasons for those models' behaviors" (p. 148).
One might rephrase this in instructional terms-the simulation game is ineffective because it
does not directly address instructional objectives and does not ensure that the learner has met
those objectives.

Not every game is intended to be used for instruction. Most games are intended for
enjoyment. Just putting a game in a course does not ensure that it will facilitate the learning of
the intended knowledge or skill. Games can be designed for instruction, but only if they are
designed to support specific instructional objectives and are incorporated logically into an
instructional program. Before elaborating on this point, it is useful to discuss the various types
of games and the characteristics that make them games.

10
 Technical Report 2005-004

DESCRIPTIONS AND CLASSIFICATIONS OF GAMES

Some authors (e.g., de Felix & Johnson, 1993) described games by listing their structural
components, such as dynamic visuals, interaction, rules, and goals. Others (e.g., Gredler, 1996)
stated that the essential elements of a game are the task, the player's role, the multiple paths to
the goal, and the degree of player control. Baranauskas, Neto, and Borges (1999) stated that the
essence of gaming is challenge and risk.

Csikszentmihalyi (1990) discussed how Roger Caillois (1958/2001), the French

psychological anthropologist, classified games into four broad classes based on the kind of
experiences they provide.
1. Agnostic games are those that have competition as their main feature (e.g., sports and
athletic events). These types of games stretch the player's skills to meet the challenge
provided by the skills of their opponents.
2. Aleatory games are games that involve the element of chance (e.g., dice or bingo). These
games give the illusion that the player somehow controls the inscrutable future.
3. Vertigo or Ilinix games are activities that alter one's consciousness by scrambling
ordinary perception. Examples are riding a merry-go-round, skydiving, or young
children turning around in circles until they are dizzy.
4. Mimicry games allow the player to create alternative realities (e.g., dance, theater, and
the arts in general). These games make one feel as though they are more than they
actually are through fantasy, pretence, and disguise. Caillois (1958/2001) uses the term
simulation as a synonym for mimicry (p. 36).

Any given game may include some or all of these experiences, but it is usually designed to
only provide a subset of these experiences. Furthermore, games are played with a variety of
media, from game boards to networked computers. It is not the medium on which the game is
played, but its characteristics that make it a game. One of the most important of these
characteristics is the specificity of the game's rules. Caillois (1958/2001) discussed the issue of
rules by defining a continuum anchored by the terms paidiaand ludus. He definedpaidia as
"the spontaneous manifestations of the play instinct" (pp. 27-28). Games at the paidiaend of the
continuum have few or no rules and are played for shear joy. At the other end of the continuum
we find ludus, which "is complementary to and a refinement ofpaidia, which it disciplines and
enriches" (p. 29). Ludus refers to games with rules and requirements for play. The more a game
is bounded by specific rules, the closer it falls toward the ludus end of the continuum. For
example the paidia feelings found in illinix (vertigo) games are challenged and constrained by
the rules and skills required in mountain climbing or tightrope walking.

The "Folk Model" divides games into 4 (non-exclusive) categories: games of skill, games of
chance, games of strategy, and simulation games (Wikipedia, 2005).
1. Games of skill include: board games, card games, letter games, mathematical games,
puzzle games, guessing games, word games, games of physical skill, and instructional
games.
2. Games of chance include: dice games, card games, casino games, lottery-type games,
Bingo, and Pifiata.

11
 Technical Report 2005-004

3. Games of strategy include: Checkers, Chess, Go, and Mastermind.

4. Simulation games include: role-playing games, board games like Monopoly, and
computer and video games.

Bj~rk and Holopainen (2003) describe games in terms of four overlapping conceptual groups.
1. The overall activity of the game. This is how the players understand the meaning of the
activity and how the activity unfolds. Sub components of activities include:
9 The game instance, which is the specific players, their experience, and the location
and requirements of a single completion of a game.
9 The game session, which is the activity defined by the time spent on playing a
game instance.
2. The boundary components of the game. These include the rules, modes of play, and
goals of the game.
3. The temporal components of the game. These are used to record the activity of playing
the game. They include discrete actions (e.g., moving a chess piece) or continuous
actions (e.g., driving a racing car). Temporal components include:
* Events, which are discrete points in the game where the game state changes.
Events usually occur as the result of player actions, but can also be the result of
trigger events such as elapsed time.
"*Closures are a change of game state resulting from completion of a goal or subgoal.
"*End conditions, which specify when a closure occurs.
"*Evaluation functions, which determine the outcome of an event.
4. The objective components of the game, such as the types of players, the game interface,
and the physical and logical components that inform players about the current game state
(e.g., tokens that represent which player is currently in control or physical elements that
describe the game space like chess squares).

Leemkvil, de Jong, and Ootes (2000) and Bright and Harvey (1984) discussed several
characteristics of games. According to these authors, all games include:
9Voluntary Play. A game is freely engaged in. Persons normally play games because they
want to, not because it is required.
* Some goal state that must be reached. The goal state may be the same each time a game
is played or it can change over time. The goal may also consist of subgoals that may
change during play (e.g., when a player reaches a certain proficiency and moves to a
higher difficulty level). Players can sometimes set their own goals.
* Constraints and rules. These define which actions are allowed and which are not. They
also define the setting and goals of the game. Certain actions may introduce additional
constraints, often defined by if-then statements (e.g., If a player misses a target three
times, then he or she must complete a practice session and achieve a criterion score before
the game can continue). Constraints are sometimes induced by setting time limits on
certain actions.
• Competition. Players can compete in several ways:
"o With other players or teams by beating them to a goal or outperforming them by
achieving a higher score.
"o With the system (e.g., by moving to higher difficulty levels).

12
 Technical Report 2005-004

o With themselves (e.g., by improving their performance).

9 A specific context. A game is always situated in a specific context that makes it more or
less realistic, appealing, or motivating to the players. This context is separatefrom real
life in terms of time and space. Leemkvil et al. quote Graham and Grey concerning how
game players must place themselves into the situated game context. "In one sense all
gaming involves role playing since the individual participants are asked to assume the
situation assigned" (Leemkvil, et al., 2000, p. 15 citing Graham and Grey).

Dorn (1989) defines a game as "any contest or play among adversaries or players operating
under constraints or rules for an objective goal" (p. 2). He defines a simulation as "an operating
representation of central features of reality" (p. 2). He then defines a simulation game as
"activities undertaken by players whose actions are constrained by a set of explicit rules
particular to that game and by a predetermined end point. The elements of the game constitute a
more or less accurate representation or model of some external reality with which players
interact by playing roles in much the same way as they would interact with reality itself' (p. 3).

A Game Designer's Definition of a Game. Greg Costikyan is a game designer and

commentator on gaming. He defines a game as "an interactive structure of endogenous meaning
that requires players to struggle toward a goal" (Costikyan, 2002, p. 22). Let us look at each of
these components in greater detail. Costikyan maintains that a game's structure is the rules,
components, and other characteristics (e.g., delivery medium) that define the way people play
the game. He states that this structure is interactive because the game changes based upon the
actions of the players. This structure also provides the meaning of the game. Thus, a game's
meaning is endogenous-it is created by the internal structure of the game and has no meaning
outside of the game. Furthermore, the game's endogenous meaning defines the goals which
game players attempt to achieve. Finally, players must struggle to achieve the goals of a game.
In this sense, struggle means that the players must work to achieve the goals. A good game
challenges the players, often allowing them to move to greater levels of difficulty as they
become more skilled at the game.

HYBRID GAMES

As discussed above, the terms game and simulation are often confused. Leemkvil, et al.
(2000) discuss the relationships among games, simulations, and case studies. Figure 1 illustrates
these relationships. As defined above, a simulation is "a method for implementing a model over
time" (Department of Defense, 1997, p. 138). A case study is an actual or hypothetical problem
situation taken from the real world. As illustrated in Figure 1, games, simulations, or case
studies can be found in a pure form or as hybrids with the others. In the literature on games, the
simulation game is the most often encountered hybrid.

13
 Technical Report 2005-004

Figure 1:
Relationship among Games, Simulations and Case Studies

ELEMENTS OF ENJOYMENT

One of the aspects that attract people to games is that they are enjoyable. Csikszetmihalyi
(1990) has studied many enjoyable activities, including games. He argued that there are several
elements (components) that make an activity enjoyable.
A challenging activity that requires skills. Individuals report that they enjoy activities
that challenge their skills. "...the overwhelming proportion of optimal experiences are reported
to occur within sequences of activities that are goal-directed and bounded by rules--activities
that require the investment of psychic energy, and that could not be done without the
appropriate skills" (p. 49). Csikszetmihalyi goes on to explain that "an 'activity' need not be
active in the physical sense and the 'skill' necessary to engage in it need not be a physical
skill" (p. 49). What is necessary is that the individual directs his or her attention (psychic
energy) to the task and applies skills to accomplish it. One of the aspects of games that offer
challenge is competition. However, "competition is enjoyable when it is a means to perfect
one's skills; when it becomes an end in itself, it ceases to be fun" (p. 50).
* The merging of action and awareness. Another element of enjoyment is when one
"loses" oneself in the activity. "People become so involved in what they are doing that the
activity becomes spontaneous, almost automatic; they stop being aware of themselves as
separate from the actions they are performing" (p. 53). This merging of an individual's action
and activity is why Csikszetmihalyi calls an optimal experience "flow." "The short and simple
word describes well the sense of seemingly effortless movement" (p. 54).

14
 Technical Report 2005-004

* Clear goals and feedback. "The reason it is possible to achieve such complete
involvement in a flow experience is that goals are usually clear, and feedback immediate" (p.
54). This is another important feature of games. It is usually clear if a goal has been made or a
target hit. One does not have to wonder if they performed correctly, the game provides
immediate feedback.
0 Paradox of control. Enjoyable experiences involve a sense of control---or, "more
precisely, as lacking the sense of worry about losing control that is typical in many situations of
normal life" (p. 59). "What people enjoy is not the sense of being in control, but the sense of
exercising control in difficult situations.. .Only when a doubtful outcome is at stake, and one is
able to influence that outcome, can a person really know whether she is in control" (p. 61,
author's emphasis).
0 The loss of self-consciousness. Because attention is so focused, because the activity is so
engrossing, "there is not enough attention left over to allow a person to consider either the past
or the future, or any other temporarily irrelevant stimuli" (p. 62). When enjoying an activity,
like a game, one of the items that disappears from awareness is our thinking about our own self.
Loss of self-consciousness "does not involve a loss of self, and certainly not a loss of
consciousness, but rather, only a loss of consciousness of the self' (p. 64, author's emphasis).
* The transformation of time. In an enjoyable activity, "time no longer seems to pass the
way it ordinarily does" (p. 66). One second can seem to stretch out for minutes or conversely
an hour can seem like only a few minutes.
* The Autelic (intrinsically rewarding) experience. An enjoyable activity is intrinsically
rewarding. The word autelic derives from two Greek words, auto meaning self and telos
meaning goal. "It refers to a self-contained activity, one that is done not with the expectation
of some future benefit, but simply because the doing itself is the reward" (p. 67).

WORKING DEFINITION OF A GAME

Based on the above definitions and for the purposes of this paper, the following working
definition of game will be used.

A game is an artificiallyconstructed,competitive activity with a specific goal, a

set of rules and constraintsthat is located in a specific context.

A game is not reality. It is a constructed activity that resembles portions of reality. It provides a
competitive environment for a player by challenging him or her to reach a goal. The purpose of
the game (e.g., enjoyment, information, instruction, etc.) helps define the goals, rules, and
context of the game.

15
 Technical Report 2005-004

THIS PAGE INTENTIONALLY LEFT BLANK

16
 Technical Report 2005-004

INSTRUCTIONAL USES OF GAMES

Most definitions of games, including the working definition above do not include any
reference to instruction. Most games are played because they provide enjoyment, not because
the player wants to learn something. However, instruction is a specific type of interaction. It is
an interactive dialogue between the learner and the instructional material (Jacobs & Dempsey,
1993). The control of the learning experience is an essential feature of instruction. Without this
control, we cannot be sure that the student learned what is required from a given instructional
product. Instruction, as a minimum, must include the following four elements. First, instruction
must be designed to support specific instructional objectives, which are determined by job
requirements. Second, instruction must include the opportunity for a learner to interact with the
instructional content in a meaningful way. Third, the student's performance must be assessed to
determine if he or she has learned what was intended. Finally, the results of the assessment must
be presented to the student in a relevant and timely manner to either reinforce correct actions or
to provide remediation for incorrect actions. If these four elements are not present, we are not
dealing with instruction.

Games can be used as instructional activities or some of the aspects of games can be
incorporated into other instructional activities to make them more enjoyable for the learner
(Garris, Ahlers, & Driskell, 2002). For example, Parker and Lepper (1992) found that learners
preferred educational programs that included fantasy elements. They also found that learners
showed greater learning and retention in the fantasy conditions when compared to activities
without fantasy elements.

Hays and Singer (1989) discussed several ways that games could be used for instruction.
They observed that potentially, games can:
"*Be used to assess entry level performance
"*Measure criterion performance
"*Aid in formative and summative evaluations of instructional approaches and programs
"*Provide instructional information on specific knowledge and skills
"*Help change attitudes
"*Serve as advance organizers prior to other forms of instruction
"*Replace alternate forms of instruction to transmit facts, teach skills, and provide insights
"*Serve as a means for drill and practice
"*Help integrate and maintain skills
"• Illustrate the dynamics or abstract principles of a task

Based on their review of several studies of motivating computer games, Malone (1981) and
Malone and Lepper (1987) provide a framework for designing intrinsically motivating
instructional environments. Many of these design heuristics can be used to help design
motivational instructional activities other than games. However, they are most relevant for the
design of instructional games. Table 1 provides brief descriptions of these motivational
heuristics. The reader should consult the original articles for additional details.

17
 Technical Report 2005-004

Table 1:
Design Heuristics for Motivating Instructional Environments
Goals: (a) clear, fixed goals; or (b) ability for learners to
INDIVIDUAL Challenge generate goals for themselves
MOTIVATIONS Uncertain Outcomes: (a) variable difficulty; (b) multiple
levels of goals; (c) hidden information, selectively
revealed; and (d) randomness
Performance Feedback: frequent, clear, constructive, and
encouraging
Self-Esteem: (a) gradually increasing difficulty levels to
promote feelings of competence; (b) goals that are
meaningful to the learner
Curiosity Sensory Curiosity: may be promoted using variable audio
& visual effects
Cognitive Curiosity: may be promoted by: (a) using
surprise, paradoxes, incompleteness; and (b) using
activities that contain topics in which the learner is already
interested
Control Contingency: learning environment should be responsive
to learner actions
Choice: activities should provide learner with choice over
various aspects of the learning environment (e.g., narration
or full text)
Power: activity should allow learner to produce powerful
effects
Fantasy Emotional aspects: appeal to the emotional needs of
learners
Cognitive Aspects: use appropriate metaphors or analogies
for the material to be learned
Endogeneity: fantasies should have an integral
_(endogenous) relationship to the material to be learned

INTERPERSONAL Cooperation: design some activities to promote cooperation among learners

MOTIVATIONS Competition: design some activities to require learners to compete with one
another (e.g., actions affect each other)
Recognition: learners' efforts should receive social recognition so they are
appreciated by others

As should be apparent from the above discussions, it is difficult to distinguish games from
other instructional activities that incorporate game elements. Nevertheless, there have been
many efforts that have tried to evaluate the effectiveness of instructional games. The next
sections summarize some of the research on the instructional effectiveness of games.

18
 Technical Report 2005-004

RESEARCH ON INSTRUCTIONAL GAMES

There have been many articles written and published on the use of instructional games. Most
of this literature is based on the writers' opinions about the potential of instructional games (e.g.,
Driskell & Dwyer, 1984; Rieber, Smith, & Noah, 1998) and the questions that must be answered
about how games will be developed to make them instructionally sound (e.g., Oblinger, 2004).
Far fewer articles have documented the empirical data on the effectiveness of instructional
games. The following sections provide summaries of the results of some of these empirical
research efforts. The first section summarizes the results of review articles, which attempted to
draw general conclusions about the effectiveness of instructional games from collections of
research efforts. Other review articles are not summarized, but were used to locate original
articles on research efforts (e.g., Dempsey, Rasmussen, & Lucassen, 1996; Hogle, 1996;
Leemkuil, do Jong, & Ootes, 2000). Subsequent sections provide summaries of individual
research efforts that have evaluated specific instructional games for different types of learners
(e.g., school children, college students, or military trainees) and for different types of learning
tasks (e.g., factual information, technical skills, etc.). These summaries focus on the effects of
each game on learner performance and also, where possible, describe the characteristics of the
games used in the studies. Where the information is available, the summaries include how the
game was used in a given instructional program.

REVIEW ARTICLES ON THE EFFECTIVENESS OF INSTRUCTIONAL GAMES

Simulation games have been used extensively in sociology and social science courses. Pierfy
(1977) concluded from a review of 22 research studies that "in terms of fostering student
learning, simulation games are no more effective than conventional classroom instruction" (p.
266). However, he did find that some research indicated that the use of games might improve
the retention of learned information. Dorn (1989) also reviewed the use of simulation games in
this area. He maintained that simulation games are based on the model of experiential learning.
In this model, learners first act in a particular instance of application. In the second stage, they
attempt to understand the effects of their behaviors and decisions in the particular instance.
Thirdly, they seek to understand the general principles under which that instance falls. Finally,
they apply the principles to new circumstances so that the learning is useful to their future
behavior.

Dorn (1989) found mixed results from his review. He observed that there are consistent
results that show that games generate interest and motivation in learners. This does not
automatically produce greater interest in the subject matter or in learning in general. He cited
some research that showed that simulation games did not increase learners' interest in
comparison to other active learning techniques like case studies. However, in general, he found
that games appear to be as effective as other techniques for teaching factual information,
principles, and concepts.

Dorn observed that the results of research that evaluated the ability of simulation games to
affect attitudes are confusing and contradictory. Although games appear to change attitudes, this
does not occur every time a game is played. Furthermore, some games have been shown to
increase positive attitudes and others resulted in an increase in negative attitudes.

19
 Technical Report 2005-004

To meet instructional goals, Dorn maintained that simulation games must be designed and
used properly. Instructional games should:
* Be selected to accomplish particular goals and should be supplemented with discussions,
lectures, and other instructional methods.
* Not be inserted into a course as random events. Rather they should be designed to meet
specific instructional purposes.
* Include debriefing after the game. Debriefing is crucial and should be more than a simple
recounting of the game. It should be a structured, guided, activity that brings meaning to the
experience and fosters learning from that meaning.

In 1981, after reviewing books, articles, newsletters, and advertisements about simulation
games, Cathy Stein Greenblat listed the claims made in support of their use. These claims were
divided into 6 categories: (1) motivation and interest, (2) cognitive learning, (3) changes in later
course work, (4) affective learning concerning'the subject matter, (5) general affective learning,
and (6) changes in the classroom structure and relations. She then summarized the available data
in support of these claims.

The strongest empirical support was for the claims that games improve learner motivation and
interest (category 1). However, Greenblat observed that most of the discussions concerning the
heightened motivation provided by games were supported by anecdotal evidence. There were
some data to support the claims in category 2 (cognitive learning), but these data were weak.
She found no evidence to support the claims in category 3 (changes in later course work) and
little data on the claims in category 6 (changes in the classroom). Some evidence indicated that
learners' attitudes about the subject matter (category 4) could be changed through the use of
games. However, sometimes these attitudes changed in the opposite direction to that which was
desired by the game designers. There was even less evidence in support of general affective
changes (category 5). Her conclusion was that "there is, at the moment, little hard data to show
that such participation leads to greater interest in the subject matter, the course, or learning in
general" (Greenblat, 1981, p. 149).

Bredemeir and Greenblat (1981) attempted to synthesize the findings on the educational
effectiveness of simulation games available at that time. They found little hard evidence on the
effectiveness of these games except for some support that games result in increased retention of
material. However, like many other reviewers, they concluded that simulation games "are at
least as effective as other methods in facilitating subject matter learning" (p. 165). However,
their overall conclusions were that "we do not yet have (1) a theoretically based taxonomy of
games with (2) clear theories about (a) what aspects of them are expected to have (b) what sorts
of distinct effects (c) on what sorts of students (d) for what reasons" (p. 169).

VanSickle (1986) conducted a quantitative analysis of simulation gaming compared to other

instructional procedures. He concluded that there was weak support for simulation gaming over
other approaches (primarily classroom lecture). This conclusion is suspect for several reasons.
First, he did not describe the characteristics of the simulation games used in his analysis.
Furthermore, six of the twenty-two studies did not compare learners who participated in
simulation games with other forms of instruction. Finally, he applied several mathematical

20
 Technical Report 2005-004

transformations to the data prior to drawing his conclusions (e.g., converted means and standard
deviations to effect sizes, then averaged these). These transformations may have introduced
errors into his results. Even without these cautions, his results only showed a moderate
advantage of simulation games. When looking at the specific studies in terms of learner
performance:
9 only 5 findings favored simulation gaming and 13 findings showed no differential impact on
immediate recall of knowledge; and
* only 2 findings favored simulation gaming and 15 findings showed no significant differences
on retention of knowledge.
Thus, from this review, it is not possible to conclude that simulation gaming is the preferred
instructional approach.

Randel, Morris, Wetzel, and Whitehill (1992) reviewed 68 studies that compared the
instructional effectiveness of games to conventional classroom instruction. These studies
covered a 28-year span of time. Of the 68 studies, 38 showed no differences between games and
conventional instruction; 22 favored games (5 additional studies favored games, but their
controls were questionable); and 3 favored classroom instruction. The studies were used to
provide instruction in social sciences, math, language arts, logic, physics, and biology. Math
was the subject area with the greatest percentage of results favoring games. The review also
indicated that games resulted in greater retention of the learned information. The authors
reached several conclusions based upon their review:
• "That only 68 studies were reported in 28 years reflects a trend to use descriptive reports
rather than empirical studies comparing games with classroom instruction" (Randel, et al.,
1992, p. 269).
* A consistent finding is that games are rated as more interesting than conventional
instruction.
0 Careful consideration should be given to the measures used to demonstrate the effects of
games. "If the test for effectiveness does not match what the game is teaching, negative results
will occur" (p. 271).
* The experimental designs used to evaluate games need to be more rigorous. Reliability
and validity are often not reported. Random sampling is often not used. Experimental designs
need to reduce confounding variables such as, Hawthorne effect, teacher bias, selection effects,
and time differences for treatments.

Dempsey, Lucassen, Gilley, and Rasmussen (1993-1994) conducted a review of the gaming
literature. They collected 51 journal articles based on the results of electronic searches (e.g.,
ERIC, PSYCHLIT). Most of the articles were discussions (n = 28). A smaller number
summarized research efforts (n = 16). The remainder were theoretical discussions and reviews
(n=4) or descriptions of how a given game was designed or developed (n=3). In terms of the
types of games described, simulation games accounted for the largest number of articles (n = 30).
Puzzle and adventure games were described in only 3 articles. They also tallied the types of
learning outcomes and the functions of the games described in the articles. Table 2 shows the
frequency of articles for each type of game function reported. Table 3 shows the frequency of
articles by type of the learning outcome sought through using the game.

21
 Technical Report 2005-004

Table 2:
Percentage of Articles by Game Function from
Dempsey, et al. (1993-1994)

Function of Game Percentage of Articles

Learn New Skills 23%
Practice Existing Skills 21%
Not Able to Determine 14%
Change Attitudes 11%
Other 11%
Drill Existing Skills 9%
Tutor 6%
Promote Self-Esteem 3%
Amuse 2%

Table 3:
Percentage of Articles by Learning Outcomes from
Dempsey, et al. (1993-1994)

Learning Outcomes Percentage of Articles

Problem Solving 23%
Not Able to Determine 21%
Attitude 13%
Verbal Information 10%
Cognitive Strategy 9%
Concrete Concept 9%
Rules 7%
Other 4%
Defined Concept 3%

A few interesting observations can be made from looking at these frequency data. Table 2
shows that most games are used to teach new skills or to practice existing skills. The next
highest percentage of game functions is "not able to determine." This is a problem with the way
gaming research is presented. In the opinion of this author, if one wishes to publish the results of
research, he or she should ensure that enough information is provided so the reader can fully
understand the intent and results of the research. The function of the instructional game is
certainly an important item of information if we wish to base future decisions on the results of
the research.

In terms of the learning outcome of the games (Table 3) they found that the largest number
sought to help learners in problem solving (23%), followed by attitudes (13%) and verbal
information (10%). Again an unfortunate result is that like game functions, a similar

22
 Technical Report 2005-004

informational gap was found in the literature on learning outcomes. The second most frequent
percentage that Dempsey, et al. (1993-1994) found when looking for learning outcomes is "not
able to determine" (21%). If future instructional game designers and developers are to benefit
from previous research, the research must be documented in a manner that will provide all the
necessary information.

Lee (1999) conducted a meta-analysis on instructional simulations (his term). Nineteen

studies met the criteria for inclusion in the analysis. Three modes of simulations were identified:
"*presentations:simulations used to supplement expository instruction
"*practice: used to provide learners with a large number of examples and guidance for
interaction with the material to be learned
* hybrids: combinations of the other two modes
Lee found many methodological problems and confounding variables that make it difficult to
draw firm conclusions about the effectiveness of the instructional simulations. Nevertheless,
several initial conclusions can be regarded as first steps toward a better understanding of
instructional games in the future. First, when used for presentations, hybrid simulations
appeared to be more effective than pure simulations. Second, hybrid simulations seem to be
equally effective for both presentation and practice. Third, learners seem to perform better when
provided with specific guidance on how to use the simulation.

Egenfeldt-Nielsen (2003) conducted a review of the educational usage of games by surveying

both single studies and other review articles. The review focused on non-electronic games, but
did not distinguish between simulations and games. He concluded that there is "currently no
evidence for a better or worse learning outcome, when games are used" (p. 1).

Maria Klawe, a senior game developer stated in a public lecture on "The Effective Design
and Use of Educational Computer Games" that the "most common problem with educational
software is that students don't pay attention to or learn the way that designers intended" (quoted
in Jenson & de Castell, 2002, p. 6 of 15). Caftori (1994) supported this view by providing
several examples of how educational software games do not always play an educational role, at
least not the role they were intended to play. One history simulation game (The Oregon Trail
Game) was intended to introduce children to the life of covered-wagon travelers on their way to
Oregon in 1848. It contained a number of problem-solving situations like crossing a river,
managing food, or dealing with disease outbreaks. It also provided opportunities for the children
to shoot at dangerous animals and other targets. Unfortunately, the children concentrated on
reaching the end of the trail as fast as possible and shooting animals for the sake of shooting.
The critical information about the type of terrain and the animals associated with it were not
noticed and/or learned by the children. Similar problems were encountered with other games.
Even though the children interacted with the games, "they were able to do it in such a way that at
least some (if not all) of the specified educational objectives have been missed" (Caftori &
Paprzycki, 1997, p. 1 of 8). This is a conclusion that we will encounter throughout our review of
the literature: an instructionalgame will only be effective if it is designed to meet specific
instructionalobjectives and used as it was intended.

23
 Technical Report 2005-004

SUMMARIES OF RESEARCH ON SPECIFIC INSTRUCTIONAL GAMES

The following sections contain summaries of research on the instructional effects of specific
games. The organization of the summaries is based on the type of learner and/or the type of task
that was targeted by the game. The first section summarizes the effectiveness of games used to
teach school children in pre-kindergarten through the 12 th grade. This is followed by summaries
of instructional games used with college students. Games used to provide instruction in the
workplace are summarized in the final section.

Games Used in Schools (Pre-K-12)

Many different games have been used to teach a variety of topics to school children in pre-
kindergarten through 12 th grade. The following sections summarize the results of some of this
research. The summaries are organized by topic area.

Games Used to Teach Math. Koran and McLaughlin (1990) compared the effectiveness of a
drill and practice math game for teaching basic multiplication facts. Twenty-eight fifth grade
students participated in ten days of instruction. Their performance was measured using daily
tests and a test at the end of the program. Both the game and the drill and practice conditions
were found to be equally effective, but the students preferred the game.

Two different card games were evaluated in an elementary school classroom environment to
help teach fractions, decimals, and percentages (Rowe, 2001). The students played both games
and were then tested on standard questions about the math topics. Only one of the games,
PercentRummy, was found to improve the learners' performance. The other game, Find the
Missing Number, was difficult to use, did not stimulate the learners and did not result in
improved performance. These results indicate that every game is not useful instructionally. The
instructional effectiveness of a game depends on its characteristics and how it is used.

Van Eck and Dempsey (2002) evaluated the effects of different ways of using an instructional
game. They measured the math problem-solving skills of 7 th and 8 th grade students after they
had been given instruction using a computer simulation game using competition and contextual
advisement. Some of the students experienced the game in a "competitive" context and others
experienced it in a "non-competitive" context. The competition group was told that they were
competing with a computer character. A face icon, representing the character, was continuously
shown on the computer screen. The non-competition group was only encouraged to work
quickly and accurately. Some students were able to access contextual advisement by watching
videos of computer characters that provided hints and encouragement. Other students did not
have access to contextual advisement. These conditions resulted in a 2x2 design. The
performance of these groups was compared to an outside control group who solved identical
word problems outside of a game context.

The math problems in both game and control conditions involved determining how much
paint, wallpaper, and other materials were necessary to fix up a room. The performance of the
students was assessed on a transfer task requiring them to apply the same skills to determine how
much paint and other materials were required to fix up a different room. The researchers found

24
 Technical Report 2005-004

an interesting interaction between competition and contextual advisement. The non-competition

group had higher transfer when they had access to contextual advisement, but the competition
group had higher transfer when they did not access contextual advisement. Unfortunately, these
results may be moot because the control group performed as well as the game groups.

An Internet game was used in conjunction with other classroom activities to teach
probabilities and statistics to 4- and 5-year old children (Pange, 2003). After a number of
activities such as throwing dice and coins, the children played the game to improve their
understanding of probabilities. The game required the children to pick one of three doors to find
a hidden car, demonstrating a one-in-three chance. After the game, the teacher discussed all of
the activities. The children showed improvement in their understanding of probability. The
teachers liked using the game and reported that it excited the students' curiosity and made the
teaching more interesting. No determination can be made about the separate instructional
effectiveness of the game.

Laffey, Espinosa, Moore, and Lodree (2003) attempted to evaluate the potential of interactive
computer technology (ICT) for teaching math skills to young, low income, urban children. The
ICT environment included several commercially available math games, such as Mighty Math and
Millie 's Math House. The 61 study participants were all Pre-K, Kindergarten, or first-graders.
The children were randomly assigned to treatment (ICT) or control groups and took a math
pretest. Some of the children were also identified as "at-risk," because of previous behavior
problems. Both groups received the same math instruction in their classroom. The ICT group
participated in two 20-25-minute ICT sessions per week over an eight week period. Both groups
later answered posttest questions. Results were based on the difference between pre- and
posttest scores. The ICT group had significantly higher gain scores than the comparison group.
Within the treatment group, the not-at-risk children gained more than the at-risk children. From
this study, it is not possible to draw conclusions about the effectiveness of the games because no
details are provided. The results are also confounded because the ICT group received more math
instruction than the control group. The authors conclude that their study demonstrates the
"potential" of ICT environments. Like many other.studies that demonstrate "potential," this
research does not provide guidance on how to design and implement instructional games.

A video-game approach was evaluated for teaching basic math and reading comprehension
skills to first- and second-graders in economically disadvantaged schools in Chile (Rosas,
Nussbaum, Cumsille, Marianov, Correa, Flores, Grau, Lagos, L6pez, L6pez, Rodriguez, &
Salinas, 2003). The games were designed to play on a device that looked almost identical to the
popular Gameboy system. The performance of an experimental group, who used the
videogames, was compared to a group from the same school, who did not use the games, and a
group from another school. Although both groups from the target school outperformed the
external group, there was no difference in their performance. Thus, there is no support for the
effectiveness of the game approach over traditional instruction on these basic learning tasks.

Teaching Vocabulary. Malouf (1987-1988) investigated the motivational effect of computer

games for special education students. He compared the performance of sixth-, seventh-, and
eighth-grade learning-disabled students who were learning vocabulary skills. His results showed
no difference between a computer program with game features and the same program without the

25
 Technical Report 2005-004

game features. However, the game condition resulted in significantly higher levels of continuing
motivation than the non-game condition.

Teaching Pedestrian Safety. Renaud and Stolovitch (1988) investigated if a game could be
used to help five-year-old children improve their pedestrian safety. One hundred and thirty-six
5-year olds were assigned to 4 groups: 3 experimental groups and 1 control group (who received
no instruction). The experimental groups included 3 variations of a simulation game. The first
version of the game included role playing and interactions among players. The second version
of the game included behavior modeling and specific instruction from the experimenter. The
third game included all the elements of the other games. The childrens' learning was measured
using pictures of a road, to which the children added stickers to answer a series of questions. All
of the experimental groups outperformed the control group. The group that engaged in role
playing and player interactions performed slightly better than the groups that experienced
behavior modeling.

Using Games to Study Motivation. Motivation can be assessed by the amount of involvement
players demonstrate in a game. Wishart (1990) examined three game characteristics (control,
challenge, and complexity) to determine their effects on learner involvement. She developed
three versions of a role-playing game to teach children fire safety knowledge. Three hundred
children (ranging in age from 5 years 9 months to 12 years 3 months with a mean age of 8 years
11 months) participated in the study. Six versions of the game were developed, each with a
different combination of player control, game challenge, and game complexity. All the subjects
took a pretest on fire safety knowledge before playing one of the versions of the game. In the
game, a player was presented with a fire safety situation and given a choice of actions. After
making the choice, the player advanced to another situation. At the end of the game, the players
answered posttest questions about fire safety. She found that learner involvement was increased
the most when the players had more control of the game. The separate addition of complexity
and challenge to control did not increase learner involvement, but together they did.
Furthermore, increased learner involvement was shown to increase the improvement of posttest
scores. This research demonstrates that specific game designs can positively affect learning
outcomes. Thus, it is important to design instructional games to increase learner involvement.

Costable, De Angeli, Roselli, Lanzilotti, and Plantamura (2003) also investigated motivation.
However, they did not look at how a game motivated learners. Rather, they investigated the
relationship between learner motivation and the effectiveness of the game. They conducted two
experiments with an educational software product called Logiocando. It was developed to teach
9-10 year-olds the basic concepts of logic. The product contained Explanation Sections, Logic
Games Sections, and Test Sections. The games increased in complexity as the learners
successfully completed the simpler exercises. Costable, et al. compared the performance of
learners in a lecture group to that of a group of learners who used Logiocando. In their first
experiment, they found that the lecture group outperformed the game group. However, in their
second experiment, they specifically motivated the learners prior to their interactions with the
games. They explained that the learners' performance would be carefully monitored by their
teachers and would be considered as part of their class work. This time, the learners who used
Logiocando performed as well as the learners who received lectures. The researchers concluded

26
 Technical Report 2005-004

that with proper motivation, these types of gaming exercises were as effective as traditional
lecture methods in helping children improve their knowledge of logic.

The Costable, et al. study introduces an interesting issue about how to use instructional
games. Many people believe that the game itself will motivate learners. However, this research
indicates that other sources of motivation may also be necessary if a game is to be instructionally
effective. Thus, for games, as with other types of instructional activities, how the activity is used
is as important as the design of the activity itself.

As discussed earlier, Malouf (1987-1988) investigated whether playing a vocabulary skill

game would increase children's motivation to continue studying the topic. He compared two
groups of sixth-, seventh-, and eighth-grade learning disabled students who played either the
game or used a computer program without game features. He found that the game and nongame
programs produced equal gains in vocabulary skill. In addition, the learners who played the
game demonstrated significantly higher levels of continuing motivation than the learners who
used the nongame program. This indicates that instructional games can have motivational
advantages beyond immediate learning gains.

Games Used to Teach Geography. Wiebe and Martin (1994) investigated the effectiveness of
a commercial game for teaching geography facts. The game, Where in the World is Carmen
Sandiego, was compared to map drawing and non-computer games (e.g., "Concentration") with
two groups of fifth- and sixth-grade students. Posttests revealed no significant differences
between the two groups in their recall of geography facts or their attitudes towards studying
geography. The authors concluded that non-computer games and activities can be just as
beneficial as computer-based games. Unfortunately, it is not possible to determine if the non-
computer games or the map drawing exercises had separate effects on learning or if it was a
combination of these activities that was as effective as the computer game.

A tutoring software system that used an adventure game interface and adventure-type
scenarios was compared to a tutoring system that did not use these game aspects for teaching
geography to fourth graders (Virvou, Katsionis, & Manos, 2005). Based on the number of errors
on a posttest, it was found that the learners who used the adventure game interface improved
their performance over the learners who used the other system. It was also found that learners
who had the poorest performance in geography before using the game received the most benefit.
The success of the game can be attributed to the instructional support (e.g., advice, suggestions)
that was incorporated into the game. Similar instructional support was not as effective in the
non-game condition. Thus, the combination of instructional support with game characteristics
appears to be an effective method for teaching geography principles to children.

Teaching Electronic Circuits. Although not specifically called a game, the following study
provides useful information that can apply to instructional games. A simulation was used as a
supplement to laboratory instruction and exercises to teach two-person teams of 15 year-olds
tasks involving electronic circuits (Ronen & Eliahu, 2000). The learners in the experimental
condition were given the option to use the simulation to help complete two exercises. The
learners in the control group only used existing materials (e.g., workbooks). Although some
indications of higher achievement were shown in the group that used the simulation, the authors'

27
 Technical Report 2005-004

observations about learners who did not benefit from the simulation may be more interesting.
They observed that the simulation was not effective for three groups:
* Learners who did not need additional assistance (about 10% of the total) and did not use the
simulation. Some did use it in later more advanced tasks.
* Learners with insufficient understanding of the domain (about 10-15%), who only performed
random trials on the simulation rather that actions that would help them on their specific tasks.
* A few students would not use the computer (about 5%) because they "hated computers."

Teaching Physics. White (1984) used a series of simple force and motion video games to
teach high-school students the principles of Newtonian physics. The games all used a
representation of a space ship to illustrate the effects of force on motion in a frictionless
environment. The games required the students to move the space ship to various targets by
applying different amounts and directions of force. The performance of the students on a series
of posttest force and motion problems was compared to a group that received no instruction
(both groups scored similarly on a pretest). The students who played the games improved their
answering of the force and motion problems more than those who did not play the games. This
study does show that these games can provide effective instruction on Newtonian force and
motion principles. However, there was no comparison to other instructional approaches, so it is
not possible to conclude that games are the most effective way to teach these principles.

Use of Games for Health Care Education. Games have been used to help teach people about
health care issues and the health care system. Sleet and Stadskley (1977) provided an annotated
bibliography of 66 simulation games used in health education. These games were used in a
variety of health care areas including: disease management; drug use and abuse; ecology; family
planning and human sexuality; nursing; nutrition; physical fitness; and safety. The games
allowed individuals to experience various roles and requirements of both health care personnel
and their patients. These authors provided no information on the instructional effectiveness of
the games.

Thomas, et al. (1997) used an interactive computer game, Life Challenge, as a tool to enhance
adolescents' sense of self-efficacy in HIV/AIDS prevention. The game allowed learners to
travel to different times and role play with imaginary partners, then to hear their recorded
statements and decide to "try again" or stay with their original statements. There was also a
game show segment where the learners picked the contestant they felt gave the best line in each
situation. A pretest, posttest comparison of true-false questions was used to determine if there
were learning gains on 7 knowledge items. Learning gains were found on 3 of the 7 items.
Learners also showed improved self-efficacy when their pre-test and posttest scores were
compared on several self-efficacy items. All learners showed increased self-efficacy with the
strongest gains shown by those who began with the lowest self-efficacy scores. Unfortunately, it
is not possible to attribute these effects to the game alone since there were no comparisons to
other instructional methods.

Brown, Lieberman, Gemeny, Fan, Wilson, and Pasta (1997) assessed the effectiveness of a
role-playing video game to teach young diabetic patients about diabetes. They compared two
groups of young diabetics (ages 8 to 16). One group played an interactive adventure video game
(Packy & Marlon), in which they played the role of a character that had diabetes. The other

28
 Technical Report 2005-004

group played a video game containing no diabetes-related content. The two groups played their
games at home as often as they wished. They were interviewed and their parents filled out a
questionnaire at the beginning of the study, after three months, and after six months. After six
months, the group that played the diabetes game showed higher perceived self-efficacy for
diabetes self-management, increased their communication with their parents about diabetes, and
improved their daily diabetes self-management behaviors (e.g., monitoring the blood glucose
levels and taking insulin as needed). The control group did not change their behaviors.
Although the game shows promise, the study did not compare game use with other instructional
approaches.

Similar conclusions can be drawn about the evaluation of a game to teach bilingual children
about the food pyramid and better eating habits. Serrano and Anderson (2004) found that fifth
graders had better knowledge about nutrition and the food pyramid after playing a game when
compared to a group that had no instruction. This shows that a game can be better than nothing,
but does not help us decide whether using a game is the most effective instructional approach.

Games Used in College

Teaching Social Science. Szafran and Mandolini (1980) examined the effectiveness of a
simulation game called SIMSOC: Simulated Society for teaching undergraduates in an
introductory sociology course. The non-computer game was intended to create a situation where
the student examined the processes of social conflict and social control in a simulated society.
The results of the game were compared to the performance of students who received
conventional classroom instruction. Performance was evaluated in terms of improvements in the
students' test scores and their ability to recognize sociological concepts in nonsociological
writings (e.g., excerpts from a novel, a magazine story, and a newspaper article). No statistically
significant differences were found in the performance of the two groups. They concluded that
"there continues to be no evidence that simulation games substantially increase cognitive
knowledge" (p. 334).

Teaching Abnormal Psychology. Brewster (1996) used games in both an interactive

multimedia computer environment and in a more traditional classroom environment to teach
undergraduates the principles of abnormal psychology. The games were modifications of
Concentrationand Jeopardy. She found no differences among the groups on any measure of
performance. It appears that multimedia is not necessary for games to provide an effective
instructional approach. However, this study does not tell us when to use the games rather than
other instructional methods.

Teaching Economics. Fraas (1982) compared a simulation gaming approach to the lecture-
discussion method for teaching college-level economics principles. The game condition
consisted of seven commercial simulation games used in succession. Unfortunately, detailed
descriptions of the games were not provided in this publication. He found that neither method
was superior to the other in overall effectiveness. However, he also found that learners with less
previous knowledge of economics showed better improvement with the simulation game
approach. This was also true of students with lower SAT scores. The reverse was also found.
Students with higher SAT scores and/or more previous economic knowledge appeared to learn

29
 Technical Report 2005-004

better from the lecture-discussion. This study indicates that games may be differentially
effective for learners with different abilities or experiences.

Gremmen and Potters (1995) evaluated whether lectures supplemented with a game were
more effective for teaching economics principles than lectures alone. They compared the
performance of two groups on a multiple-choice economics test. The group who received a
lectures supplemented with the game outperformed the group that only received the lectures.

Reasoning Skills. Wood and Stewart (1987) tried to determine if use of a computerized
version of the game Mastermind would help college students improve their skills in practical
reasoning. Thirty students were divided into two groups. Both groups took a pre and posttest on
the Watson-Galser Thinking Appraisal. The experimental group played the game between test
administrations, but the control group received no treatment. The experimental group reduced
their errors on the posttest relative to the control group and indicated that they enjoyed the
experience. However, it is not clear whether the game directly affected these results or if they
were changed due to repetition of the test and/or more time on the task. If the game did
influence the results, it is still not clear what game characteristics might have affected the
learners' performance.

Teaching About Learning. Klein and Freitag (1991) compared the instructional effectiveness
of using a board game to the use of a worksheet for teaching the principles of the information-
processing model of learning to undergraduate education majors. The game and worksheet were
designed to supplement and provide practice on material from assigned textbook reading. The
board game included a set of 25 game cards with practice questions about the topic. The rules of
the game "were developed to encourage cooperation, competition, and active participation" (p.
304). The worksheet included the same 25 questions that appeared on the game cards. No
performance differences were found as a result of the two methods of instruction. However, the
learners who reported that they had read the textbook assignment performed better than those
who did not complete the assignment.

Games Used to Change Attitudes. Williams (1980) evaluated the effectiveness of a

simulation game to change the attitudes of college students toward a negatively-presented
historical character. The students first read a fictional history of a struggle among four
noblemen. One nobleman was presented as a "villain." After the reading, they all filled out an
attitude questionnaire about the characters. Then they were separated into two experimental
groups and one control group. The control group engaged in a non-related activity while the two
experimental groups engaged in a simulation game in which they had to "play" the role of the
negative character. One experimental group (dissonance) was not given praise or reward for
playing the character. The other experimental group (incentive) was given praise and a
monetary reward for their role play. It was assumed that cognitive dissonance would increase in
the group that had to play the negative character with no outside incentives. Both of the
experimental groups changed their attitudes toward the negative character to a significantly
greater degree than the students in the control group. No differences were found in the attitude
change between the two experimental groups.

30
 Technical Report 2005-004

Another study (Bredemeier, Bernstein, & Oxman, 1982) found that a game designed to
change college students' attitudes on dogmatism and ethnocentrism was effective when used in
an anthropology class compared to another anthropology class that did not use the game.
However, the same game, used in a different course (philosophy) did not result in the same
degree of attitude change. These results indicate the importance of placing instructional games
in the appropriate instructional context.

Teaching Empathic Understanding. A role-playing game was compared to classroom lectures

for improving the empathic understanding of first-year university students (Barak, Engle, Katzir,
& Fisher, 1987). The game was designed to improve the communication skills involved in
empathy (understanding another person and communicating this understanding to him or her). It
involved analysis, diagnosis, and understanding a target "client." Learners in the game group
took turns playing the role of the "client" while other players listened to the "client" describing
his or her situation. The control group participated in a lecture and group discussion about
listening and understanding others feelings. The performance of the experimental group
significantly improved, but the performance of the control group did not change. The authors
concluded that the game was an effective method for improving empathic skills.

Teaching Physics, Electronics, and Engineering Principles. Rieber and Noah (1997) used a
game to teach university students the relationships between acceleration and velocity. When
compared to participants who were taught these topics without a gaming context, it was found
that the game actually interferedwith the participants' learning of these principles. The authors
examined several patterns of interaction with the game that may have led to this outcome. The
participants who were given assistance by an outside agent were better able to consider the
relevancy of the game for learning the science principles. "When left on their own, participants
had much difficulty in focusing their attention on how to manipulate the task in order to learn
more about the content" (p. 8). Without assistance, "participants focused far too much on the
competitive nature of the game and, as a result their ability to monitor their own comprehension
was inhibited" (p. 8). This study, although exploratory in nature, illustrates the important issue
of the instructional support that surrounds a game and how the game is used to support
instruction. Without instructional support, a game may be not only ineffective, but it may be
detrimental to learning.

Miller, et al. (1999) examined three versions of an instructional simulation game (Electronic
Field Hockey) to determine which was most effective in helping undergraduate students develop
a better understanding of the physics of electrical interactions. The simulation was intended to
allow learners to observe how the trajectory of a "puck" with an electrical charge was influenced
by obstacles with various electrical charges. The three versions of the simulation were:
"*A no-goal condition, which included no obstacles, no net, and no specific task.
"*A standard-goalcondition, in which the learners' task was to try to position the obstacles so
that their charges would cause the "puck" to avoid them and go into the hockey net.
9 A specific path condition, which showed the learners an ideal trajectory which they
attempted to duplicate. It also showed the difference between the ideal trajectory and their
actual trajectory that was obtained by their placement of the obstacles.

31
 Technical Report 2005-004

Both the no-goal and specific-path conditions produced higher learner scores than the
standard-goal condition. The authors concluded that these two conditions required learners to
engage in broader exploration and to be more selective in interpreting the evidence than the
standard-goal condition. The study highlights the importance of carefully selecting and
analyzing the intended goals of an instructional game before it is implemented.

Crown (2001) conducted a study to determine the effect of web-based games on the
visualization skills of engineering graphics students. However, it is difficult to determine the
effect of his instructional games because the game pages were only part of an instructional CD.
Although learners who used the instructional CD showed positive performance on exams, it is
not possible to separate the effect of the game pages from the effect of other instructional
material presented on the CD.

Teaching Principles of Underwater Sound Propagation. Shrestha (1990) compared a simple

game to more traditional lecture-type instruction on the principles of underwater sound
propagation. She found no learning differences among two groups of college students.
However, she did find that the learners in the game condition spent more time interacting with
the game. She concluded that games may have motivational benefits over the traditional
instruction.

Games to Teach Health Care Decisions. Westbrook and Braithwaite (2001) evaluated a web-
based educational game to help college undergraduates make better health care decisions. The
game presented four families in a series of health care events. It then generated problems that
required the learners to seek out information to assist the families in making decisions like which
type of care to seek and how much it would cost. The participants were 55 students from
Australia and South-East Asia. They worked in competing teams of six persons. Each team
member contributed to a common discussion log over a four week period. The evaluation
consisted of a pre- and post-questionnaire that assessed: (1) the participants' interest in and
knowledge of the health care system, (2) their experience with computers, (3) their views on
team work, and (4) their demographics. Only two questions assessed the learners' factual
knowledge.

Although a higher percentage of the participants correctly answered the factual questions on
the posttest, these results were highly confounded. Prior to the game, the participants had
already received 9 weeks of lectures which included the topics covered in the game. The authors
concluded that the improvement on the factual questions indicated the effectiveness of the game.
However, other events during the 13 week period (9 weeks of lectures and 4 weeks of game
participation) could have influenced their answers.

The Westbrook and Braithwaite (2001) study illustrates many of the problems that make it
difficult to reach conclusions about either the effectiveness of a specific instructional game or the
effectiveness of games in general. Some of these problems include:
e The evaluators' interest in "proving" the effectiveness of their game (i.e., they developed
the game and wanted it to work).
* No control group who experienced an alternate form of instruction to compare to the
game.

32
 Technical Report 2005-004

* No clear and detailed description of the game itself (e.g., what characteristics made it a
"game").
* Lack of control of possible confounding variables (e.g., events prior to and during the
evaluation).

Teaching Marketing, Business, and Management Principles. Fritzsche (1981) compared the
use of a marketing simulation game, Marketing in Action, as either the central delivery vehicle
for instructional information or as a supplement to the standard lecture on the same material. No
differences were found between the two conditions on college students' midterm and final
examination grades. Thus, Fritzsche concluded that the game alone was as effective as the
lecture with the game as a supplement. The equivalence in effectiveness may be because the
game (Day, 1962) included detailed instructions on how and why to fill out the simulated
management decision forms. These may have had as much effect on later performance as
playing the game itself.

Wellington and Faria (1996) studied the effect of team cohesion, player attitudes, and
performance expectations on the instructional effectiveness of a simulation game. Although they
do not specifically describe the simulation games, it appears to be a marketing simulation
specifically developed for an introductory marketing course. The game was played by 389
students from two sections of a Principlesof Marketing course. They were divided into 108
teams of three or four players. The teams competed with one another and each team had to make
six decisions during the competition. The teams that were more cohesive at the start of the
competitions outperformed the teams that were less cohesive. The game did not appear to
change the cohesiveness of the teams over the course of the competition. Furthermore, learner
attitudes were not found to be statistically related to final game performance although learner
expectations were positively related to performance. Unfortunately, this research tells us very
little about the specific characteristics of the game. It also does not compare the use of the game
with other instructional approaches.

Rowland and Gardner (1973) did not find positive support for the instructional value of a
business and management game. In their study of 200 college students who played the Marksim
business game as seven-person teams, there was no positive relationship between playing the
game and grades in a marketing course. They concluded that business games were not the
educational panacea envisaged by many advocates.

Games Used in the Workplace

Teaching Attention Allocation Skills. Gopher, Weil, and Barebet (1994) found positive
learning effects from a video game called Space Fortress. They found that attention allocation
skills could be trained using the game. These attention allocation skills were found to transfer to
piloting skills in complex fighter aircraft. However, these results are ambiguous because other
training (e.g., verbal tips and other part-task training sessions) may also have affected the
outcomes.

Teaching Periscope Skills. Garris, et al. (2002) described an initial evaluation of a game
called BOTTOM GUN. It was developed to help Navy personnel learn periscope skills such as

33
 Technical Report 2005-004

distance estimation and angle-on-the-bow (i.e., the angle at which an observed ship is visually
presented to the periscope observer). The game was designed to incorporate game
characteristics such as curiosity, competition and control, as well as, visual and sound effects.
Although an initial evaluation, their results indicated that learners using the game showed more
improved performance than those using a trainer without game elements.

Teaching Principles of Chemical, Biological, and Radiological Defense. Ricci, Salas, and
Cannon-Bowers (1996) investigated the learning effects of a very simple game. They compared
three groups of Navy trainees (mean age = 20 yrs.) who were taught the basic principles of
chemical, biological, and radiological defense. One group (text group) studied a printed text (a
63-page pocket handbook). The second group (test group) was given a printed copy of 88
multiple-choice questions with answers. The third group (game group) interacted with a
computer program that presented the 88 questions in random fashion in boxes that resembled a
slot machine. This group was given 3 minutes to answer each question and received points for
correct answers. Both the test and game groups performed better than the text group on a
posttest. However, no performance differences were shown between the test and game groups
although the game group scored higher on a retention test given four weeks later. The results of
this study are questionable because the game condition was basically a way to randomize the
presentation of the questions and may not have been a true game.

Teaching Technical Skills. Oxford, Harman, and Holland (1987) reported the initial
evaluations of two hand-held computerized game-based training aids for Army trainees. The
two training aids, called Tutor and CHIP,used simple games (e.g., matching pictures to verbal or
visual stimuli to send a projectile against an "enemy" target) to teach technical terminology and
basic technical skills. Learners who used the aids completed more instructional units than a
group who used a workbook containing the same content. The training aid group also
outperformed the workbook group. Although these data were very tentative and the games were
simplistic, they did show the instructional potential of gaming activities.

Whitehill and McDonald (1993) compared the effects of the presentation of circuit repair
problems in a drill format to presentation of the same problems in a game context. The drill
context simply presented the circuit problems one at a time. However, the game context required
learners to simulate the role of an electrician repairing the circuits. They used a computer to
move a cursor around a Navy ship's floor plan to locate and solve the problems. Video game
sound effects accompanied the movement of the cursor through the ship. The payoff for solving
the problems was either fixed or variable based on problem difficulty. Although they found no
performance differences between learners in either the drill or game conditions, they did find
that combining the game with variable payoff resulted in increased learner persistence.

Parchman, Ellis, Christinaz, and Vogel (2000) evaluated four different instructional
conditions, including a game, to train basic electricity and electronics fundamentals to beginning
Navy electronic technicians. They compared conventional classroom instruction (CI) with
computer-based drill and practice (CBDR), enhanced computer-based instruction (ECBI) that
added compelling graphics, animations, and simulations to the CBDR, and a role-playing
adventure game. The trainees in the CBDR and ECBI conditions outperformed those trained in
either the conventional instruction or the game. The CBDR and ECBI conditions were equally

34
 Technical Report 2005-004

effective in teaching symbols and definitions. However, cause-and-effect relationships were

learned better by trainees in the ECBI condition. This is probably because of the enhanced
visualizations provided by the ECBI program. The ECBI-trained learners were also more
confident than those in the other groups that they had learned the required material.

These researchers concluded that instructional developers need to be cautious and not rush
into exclusively using games until additional data are available on their effectiveness. Other
computer-based instructional approaches may be more effective than games for certain types of
instruction. They also recommended that instructional developers not overdevelop computer-
based instruction. Simple computer-based instruction may be sufficient when the objective is to
teach definitions and symbols. When more complex material needs to be learned (e.g., cause-
and-effect relationships), enhanced computer-based instruction may be required.

Business and Management Games. Gaming has been used to teach business and management
skills for many years. Faria (1989) observed that business games are direct descendents of war
games, dating back at least to the German Kriegspiel of the mid-19th century. He also observed
that the RAND Corporation developed an instructional simulation game in 1955. Called
Monopologs, it was intended to allow U. S. Air Force logistics personnel to make logistics
decisions in a "risk-free" environment. No data could be located on the effectiveness of the
game.

The first widely known business game was developed by the American management
Association in 1956. Called AMA 's Top ManagementDecision Simulation, it provided an
environment in which two teams of players could represent officers of firms and make business
decisions (Cohen & Rhenman, 1961). Five teams with three to five persons each produced a
single product which they sold in competition with other teams. Many varieties of this game
were developed in subsequent years, although no data on its effectiveness were found.

In the early 1960s, a game was used to teach diplomats international relations skills (Benson,
1961). The Simple Diplomatic Game reproduced, in simplified form, features of the
international political arena and allowed learners to see the effects of their actions in a larger
action-counteraction cycle. No data were found on its effectiveness.

Faria (1989) conducted surveys of training managers, businesses, and business schools on
their use of games. Of the 223 training managers who responded, 54.7% indicated that their
companies used simulation games in their training programs. Results also indicated that at least
5000 firms were using business games and 1,700 four-year business schools used games in their
programs. Wolfe (1997) observed that by 1974 hundreds of articles on business gaming had
appeared in the business and professional press.

Cohen and Rhenman (1961) and Morris (1976) maintain that all management games share
several common features.
"*They allow the presentation of feedback based on the results of players' actions.
"*They represent the environment in logical or mathematical relations. Some of these relations
are known to the players as "rules" while others are only vaguely qualified and become known
during play.

35
 Technical Report 2005-004

* They allow interaction between players and the environment. Sometimes players can assume
the roles of managers in different functional areas within a company to learn how these areas
differ from their own.
9 They provide a simplified view of reality so players can focus on single issues or areas as
they learn.

Citing Cohen and Rhenman (1961) and Morris (1976), Hays and Singer (1989, pp. 198-200)
mentioned several educational properties of business and management games:
* They can provide training on the interactions of functional specialties within companies.
The players can learn how their specialty is dependent on others, and how to interact more
effectively to reach common goals.
o Playing the game may sensitize learners to the fact that in the real world they must take
particular actions solely for the sake of information gathering.
"• Games can offer the trainee the opportunity to learn and refine a variety of analytic tools.
"*Games may allow learners to become aware of the psychological and organizational
interactions and interdependencies in business.
* Players may learn that most decisions are made by teams of several persons and that
these decisions are constrained by a variety of variables, such as time, complexity of the
decision, and personality factors of the players.
* Games can teach institutional facts about the learners' company (e.g., standard operating
procedures).
These properties and many of those mentioned by other authors also apply to games outside of
business and management.

Thornton and Cleveland (1990) described management gaming simulations as a hierarchy of

increasing complexity. Their hierarchy included:
* One-on-one interview simulationsthat are "short, yet powerful, simulations of specific
interpersonal interactions" (p. 192). The authors list several studies that indicated that these
were effective in enhancing supervisory skills. However they also cited contradictory
evidence that showed no behavior change as a result of participating in this type of gaming
simulations.
* Leaderless group discussions simulate "numerous ad hoc committee situations in
organizations in which problem analysis and decision making take place" (p. 192). The
authors cited no specific data on the instructional effectiveness of these discussion simulations.
* In-basket technique. A type of simulation named after the in-tray on a manager's desk has
been used to practice administrative skills. Although the technique has been shown to have
predictive validity and is used in "virtually all assessment centers" (p. 193) for managerial
selection, Thornton and Cleveland cited no data on their instructional effectiveness.
9 Complex decision-makingsimulations have been used to study the components of managerial
decision making (Streufert, 1986). However, Thornton and Cleveland found no evaluations of
their instructional effectiveness.
* Large-scalebehavioralsimulations involve "multiple business problems and opportunities,
intensive and extensive interaction among managers, and observation by trained staff' (p. 193-
194). Little published empirical information was available on these simulations.

36
 Technical Report 2005-004

Morris and Holman (1976) report the successful application of business gaming at a large
pharmaceutical company. A business management competition was held with 638 people
divided into 192 teams. These teams, from various divisions, learned an appreciation of
business, as a whole, rather than just their work functions. Some of the lessons learned included:
methods of increasing profits by either increasing income or decreasing expenditures; team work
skills; methods to get along with the competition; and marketing techniques.

Carlson and Hill (1982) examined the effectiveness of a gaming approach to reducing
absenteeism and tardiness at a small manufacturing firm. They found that the impact of gaming
was minimal on these two areas. However, they did find that when gaming was used as a
vehicle for employee communication, it resulted in statistically significant improvement in
employee attitudes and cooperation.

Wolfe (1997) conducted a review of business games used to teach strategic management
skills. He reviewed seven studies conducted between 1966 and 1988 that: 1) compared game use
with at least one other instructional approach, 2) had predefined, objectively measured
instructional objectives, and 3) objectively measured learning outcomes. He divided the studies
into those that concentrated on results (substantive evaluations) and those that examined the
effects of how the game was used (procedural studies).

From his review, Wolfe concluded that there is evidence that computer-based general
management games are effective in producing knowledge gains. He found that the only
alternative instructional approach that was compared to games was case studies. Both methods
produced learning, but games appeared to be superior. It should be remembered, as discussed
above, games can be designed as hybrids, incorporating the characteristics of case studies.
Although he reported no data on these game-case study hybrids, in future studies, this approach
may be found to be effective.

Wolfe found less clear data on the way games are used. Some studies found less complex
games as effective as more complex games while others found that more complex games
provided better learning outcomes. However, this result is confounded in at least one study
because the more complex game included more relevant topics than the less complex games.
Wolfe found little data on the role of the instructor in influencing the learning outcomes of
games other than that some type of instructor involvement (instructional support) appeared to be
required. Instructional support is one of the topics discussed in the next section.

OTHER ISSUES IN INSTRUCTIONAL GAMING

Types of Competition in a Game

As noted above, competition (challenge) is one of the most important characteristics that
make games enjoyable (Csikszentmihalyi, 1990) and motivating (Malone, 1981; Malone &
Lepper, 1987). Fisher (1976) examined three different types of motivation in a vocabulary skills
game (The DictionaryGame). Three groups of college students played the game under different
types of competition conditions and their performance was compared with each other and with a
control group that did not play the game. The first treatment group played the game with 1-to-I

37
 Technical Report 2005-004

person interpersonal competition. The second treatment group played with 1-to-2 person
interpersonal competition. The third treatment group played as three-person groups with 1-to-I
intergroup competition. Post-game performance was measured on a ten-item multiple-choice
test. Results showed that learners in all three game conditions performed better than the non-
game group. The 1-to-2 interpersonal competition group had the highest score, followed by the
1-to-I interpersonal competition group and the 1-to-I intergroup competition group. These
results show that specific types of competition can affect the instructional effectiveness of a
game. However, since the game was not compared to any alternate types of instruction, it is not
possible to say that gaming is the preferred approach to teach vocabulary skills.

Adding Instructional Support to a Game

Leutner (1993) investigated the effectiveness of two varieties of instructional support in a

computer-based simulation game to teach economic concepts to seventh-graders and later to
university students. The game, Hunger in the Sahel, is a role-playing simulation where learners
run a small farm and must deal with issues such as rainfall and temperature extremes. The two
varieties of instructional support were system-initiated advice and learner-requested background
information. He found that learners without any instructional support learned to play the game,
but only learned a minimal amount about domain-specific concepts. The opposite occurred with
the learners given advice. They learned more domain-specific concepts, but only learned to play
the game to a limited degree. These results indicate that instructional support is essential if the
goal of the game is to provide instruction. Leutner (1993) stated that it is "very useful to make
pieces of information, which are implicitly available in the system, explicit through appropriate
instructional support during system exploration" (p. 219).

The importance of instructional support is also discussed by de Jong and van Joolingen
(1998). These authors review the effectiveness of discovery learning in simulated environments.
Discovery learning is often cited as one of the strengths of instructional simulations and games.
The assumption is that learners gain a deeper understanding of a topic if they are allowed to
freely explore a domain and "discover" important information and interactions among
phenomena. However, after reviewing many studies on discovery learning in simulations, these
authors stated: "The general conclusion that emerges from these studies is that there is no clear
and univocal outcome in favor of simulations" (de Jong and van Joolingen, 1998, p. 181). They
explain these results by highlighting several intrinsic problems with discovery learning. These
problems center on difficulties that learners have with forming and testing hypotheses in these
discovery environments. The reader is encouraged to consult the original source for details on
these problems. The authors conclude that the crucial aspects of successful discovery learning
are well designed instructional goals and instructional support for the learner.

Mayer, Mautone, and Prothero (2002) investigated the effect of adding prior pictorial
representations of possible features that would be encountered in a game to teach geology
principles. These researchers reported the results of 4 experiments with college students. A
game called The Profile Game was designed to help learners determine which geological
features were present on a certain portion of the planet's surface. Prior to playing the game in
each experiment, the students either received prior pictorial support or did not. The specifics of
how the support was used differed across the experiments. In the first two experiments, no

38
 Technical Report 2005-004

differences were found due to the pictorial support. However, in the second two experiments,
the pictorial support groups outperformed the no-support groups. The support given in the
second two experiments was accompanied with more detailed instructions on how to use the
support. Thus, it appears that instructional support can enhance the instructional effectiveness of
games if it supports "guided discovery" rather than just providing information.

What is Learned in an Adventure Game?

Adventure games are very popular and many have advocated their use for instructional
purposes. To determine what is learned in this type of game, Ju and Wagner (1997) had 12
college students play IndianaJones and the Fate ofAtlantis in which they had to find their way
through a South American jungle. The students played the game individually. After an hour of
play, they stopped the game and asked the players to answer a set of questions, referring to the
material encountered during previous play. No player answered all questions correctly. When
the responses were analyzed by question type, they found that more correct answers were given
to the questions that asked about physical objects in the game or attributes about the game (mean
correct responses of .83 and .84 respectively). Questions about concepts and cause-effect
thinking only produced mean correct responses of .55 and .50. The least successful answers (.38
correct) were on questions that required the user to respond with a "plan" or procedure. These
results show that different types of learning can result from an adventure game. This is a strong
indication of the importance of designing instructional games to contain events that meet specific
instructional objectives.

The Danger of Evaluator Interest

An area of game research that indicates how the evaluator's "interests" may influence the
results is the investigation of chess playing as a way to develop children's cognitive skills and
scholastic achievement. Dauvergn (2000) summarized several studies that he claimed show that
"chess is one of the most powerful educational tools available to strengthen a child's mind" (p. 2
of 6). However, as Thompson (2003) pointed out, most of these studies were quasi-experimental
in nature and were conducted by chess enthusiasts (some from the U. S. Chess Federation).
Thompson (2003) conducted a controlled study that compared the scholastic performance of
chess players and non-chess players. When he controlled for grade level and IQ, he found no
significant effect of chess playing on scholastic performance. He concluded that the chess
enthusiasts want to see a positive effect from playing chess and therefore do not control for other
factors like general intelligence. His belief is that children who are interested in chess also tend
to be more capable in general scholastic skills. Whichever view is correct, it is important that
the interests and goals of researchers be accounted for when interpreting the results of their
investigations.

The Instructional Value of a Game Developed for Another Purpose

Recently, a game developed for Army recruiting was evaluated to determine its instructional
effectiveness. The game, America's Army, was developed by the Office of Economic and
Manpower Analysis at the United States Military Academy as a recruiting tool. It was intended

39
 Technical Report 2005-004

to inform the "recruiting age" public about the U. S. Army. Because it was so popular (over 2
million players registered on the web site), some believed it could be assessed to identify
features that motivated players and provided instruction.

Belanich, Sibley, and Orvis (2004) assessed the knowledge that 21 players gained from the
game using a 35-question posttest. The questions were designed to assess procedural, episodic,
and factual knowledge that was either relevant to game play or that did not impact game play.
They found that players correctly answered a higher percentage of procedural questions,
followed by episodic and factual questions in descending order. They also found that the
realism, challenge, exploration, and control afforded by the game were the factors that
influenced player motivation.

On the surface, these results seem to indicate that the game provided effective instruction.
However, when one examines the study more closely, it becomes apparent that the players only
learned procedures, experiences, and facts about the game. There is no indication that any of the
learned information is relevant to real-world Army requirements. For example, one of the
procedural questions asks how to select a "Flash/Bang" grenade. The possible answers involve
pressing a number key or clicking the mouse. These procedures are irrelevant in the real-world.
From reading the report, it does not seem that the game provided real-world relevant instruction,
such as when to choose this type of grenade.

This study illustrates one of the greatest dangers of using games for instruction. Yes, they
may be motivational and they may help learners retain information. However, if the game is not
designed to directly support specific instructional objectives related to actual job requirements,
much of the learning may be irrelevant. This is also what was found in the evaluation of The
Oregon Trail Game (Caftori, 1994). As discussed earlier, children only learned "shoot-em-up"
skills rather than the historical information that game designers hoped the game would provide.

This study (Belanich, et al., 2004), like many others, indicates the potential of games for
instruction. However, it also highlights the need to carefully design and develop games that will
support specific instructional objectives and that will fit into larger programs of instruction.

The Importance of Debriefing

After participation in an instructional game, it is very important to conduct a debriefing

session with the participants (Crookall, 1992; Garris, et al., 2002). Debriefing gives the learners
the opportunity to reflect on their experience with the game and understand how this experience
supported the instructional objectives of the course or program of instruction. Lederman (1992)
reviewed the literature on debriefing and presented a model of the debriefing process.
According to her model, debriefing should consist of three phases:
* Phase 1: Systematic reflection and analysis. This phase introduces the learners to a
systematic self-reflective process about their game experience. It should include a summary of
the game experience as it supports the instructional objectives.
e Phase 2: Intensification and personalization. The second phase is a deeper examination of
the learners' game experiences. Each learner should be guided through reflections on their
own individual experiences and the meanings these experiences have for them. It is the job of

40
 Technical Report 2005-004

the debriefing facilitator (e.g., instructor) to help ensure that the meaning of the experiences tie
into the instructional objectives.
o Phase 3: Generalization and application. Here, the learners are led beyond their own
individual experiences to the broader applications of the experiences. For example, they might
engage in discussions of how their experiences can contribute to their success on the job or in
additional learning situations.

The key point is that no instructional game should be conceived as a stand-alone activity. It
should be designed and implemented as part of a larger instructional program that meets the
specific instructional requirements of the task and the learners. As Lederman puts it, "The
process of debriefing is not ancillary to the educational experience to which it is tied. Debriefing
is an integral part of any learning experience that is designed to be experience based"
(Lederman, 1992, p. 158).

41
 Technical Report 2005-004

THIS PAGE INTENTIONALLY LEFT BLANK

42
 Technical Report 2005-004

SUMMARY OF EMPIRICAL EVIDENCE ON INSTRUCTIONAL GAMES

Instructional games have been used for a wide variety of tasks with learners of different ages.
Past reviews have found that the majority of articles on instructional games are opinions about
whether to use games and about the potential of games to provide effective instruction (e.g.,
Dempsey, et al., 1993-1994). Far fewer articles provided empirical data on research that has
investigated the instructional effectiveness of games. This was also found in the present review.
Over 270 documents were obtained from the literature on games. Of these, only 48 provided
empirical data on the instructional effectiveness of games. This review discussed the results of
other review articles and also articles on the effectiveness of specific games. Table 4 provides a
summary of the 48 empirical articles discussed in earlier sections.

Several observations can be derived from an examination of Table 4. First, the empirical
research on instructional games examined their effects for a wide range of age groups. Twelve
of the studies examined the learning effects of games with elementary school children and two
with high school students. Twenty-one studies used instructional games with college students.
Six studies examined game effects with military trainees and the remainder of the studies
evaluated games for adults working in various industries. The results of any given study must be
evaluated in a restricted manner. We cannot necessarily generalize the results of a study
conducted with one age group to another age group.

The empirical research also includes studies that examined the instructional effectiveness of
games for many different tasks. These tasks range from basic mathematics principles to complex
business and marketing decision making. They include general reasoning skills and specific
electronics repair tasks. Care should also be taken when generalizing from one instructional task
to tasks in another domain.

The empirical research does not make a compelling casefor games as the preferred
instructionalmethod. In most cases, the research shows no instructional advantage of games
over other instructional approaches (such as lectures). In several cases, games were shown to
provide effective learning, but were not compared to other instructional methods. The research
does not allow us to conclude that games are more effective than other well-designed
instructional activities.

Too much of the empirical research on instructional games contains methodological problems
(e.g., experimental confounds) that make it difficult to draw valid conclusions about the
effectiveness of the games. Researchers need to ensure that they understand experimental design
and apply sound decisions when designing and reporting their research. In addition, editors of
educationally oriented journals need to filter out studies that do not follow sound experimental
design procedures.

43
 Technical Report 2005-004

Table 4:
Summary of Empirical Effectiveness Studies Reviewed

Study Learning Task Age/Grade Results

Koran & McLaughlin (1990) Math Facts Gr.
5 th No Differences
Rowe (2001) Math Elementary School Only one of two games
provided learning. No
comparison to other approaches.
VanEck & Dempsey (2002) Math Problem 7-8t Gr. No Differences
Solving
Pange (2003) Probability 4-5 yrs. Can't Determine
Laffey, et al. (2003) Math Skills Pre-K-1sl Gr. Can't Determine
Rosas, et al. (2003) Basic Math & 1st & 2 nd Gr. No Differences
Reading
Malouf (1987-88) Vocabulary 6 - 8th Gr. No Differences
Renaud & Stolovitch (1988) Pedestrian Safety 5 yrs. Learning, but no comparison
Wishart (1990) Motivation 5 - 12 yrs. Learning, but no comparison
Costable, et al. (2003) Basic Logic 9-10 yrs. No Differences
Wiebe & Martin (1994) Geography Facts 5 th- 6 th Gr. No Differences
Virvou, et al. (2005) Geography 4 th Gr. Game favored over non-game
software
Ronen & Eliahu (2000) Electronics 15 yrs. Slight, but not effective for
certain groups
White (1984) Physics H.School Learning, but no comparison
Thomas, et al. (1997) Self-efficacy in Adolescents Can't Determine
HIV/AIDS prevention
Brown, et al. (1997) Diabetes Facts 8-16 yrs. Can't Determine
Serrano & Anderson (2004) Nutrition Facts 5 th Gr. Learning, but no comparison
Szafron & Mandolini (1980) Sociology College Learning, but no comparison
Brewster (1996) Psychology College No Differences
Fraas (1982) Economics College No Differences
(game better for learners with
less previous knowledge)
Gremmen & Potters (1995) Economics College Game supplementing lecture
favored over lecture alone
Wood & Stewart (1987) Practical Reasoning College Learning, but only compared to
no instruction.
Klein & Freitag (1991) Principles of Learning College No Differences
Williams (1980) Attitudes toward College Favored Game over non-
subject matter related activity
Bredemeier, et al. (1982) Attitudes toward College Game successful in appropriate
subject matter context
Barak, et al. (1987) Empathy College Favored Game over lecture &
discussion.
Rieber & Noah (1997) Physics Principles College Game interfered with learning

44
 Technical Report 2005-004

Table 4: (Continued)

Study Learning Task Age/Grade Results

Miller, et al. (1999) Electrical Interactions College No comparison to alternate instruction.
Different versions of game were compared.
Crown (2001) Engineering College Can't Determine
Visualization
Shrestha (1990) Underwater Sound College No Differences
Propagation
Westbrook & Health Care Decisions College Can't Determine
Braithwaite (2001)
Frizsche (1981) Marketing College Game alone as good as game and lecture
Wellington & Faria Team Cohesion & College No Comparison
(1996) Attitudes
Rowland & Gardner Marketing College No support for game
(1973)
Gopher, et al. (1994) Attention Allocation Pilot Trainees Learning, but no comparison
Skills
Garris, et al. (2002) Periscope Skills Navy Favored Game (over trainer without game
Trainees elements)
Ricci, et al. (1996) CBRD Defense Navy No Differences ("game" slightly better for
Principles Trainees retention)
(Mean age =
20yrs.)
Oxford, et al. (1987) Technical Skills & Army Favored Game (over workbook)
Terminology Trainees
Whitehill & Electrical Repairs Navy No Differences
McDonald (1993) Trainees
Parchman, et al. Electronics Navy Drill & Practice favored over game
(2000) Fundamentals Trainees
Morris & Holman Business Issues Adults No Comparison
(1976)
Carlson & Hill (1982) Attendance Attitudes Adults No effect
Fisher (1976) Different types of College No Comparison
Competition
Leutner (1993) Economics Concepts 7th Gr. & Without instructional support, Only
College Learned Game
Mayer, et al. (2002) Geology College Instructional support enhanced value of
game
Ju & Wagner (1997) Playing an adventure College Learned more about physical objects in
game game than concepts
Thompson (2003) Effect of playing chess Elementary No effect
on scholastic school
performance children
Belanich, et al. What is learned in a Learned only game-relevant skills not real-
(2004) game? world skills

45
 Technical Report 2005-004

A REEXAMINATION OF THE "CLAIMS"

As discussed earlier, Greenblat (1981) summarized evidence in support of six categories of

"claims" about the efficacy of games used for instructional purposes. We can use the same
categories of claims to help us summarize the research on the instructional uses of games since
that time. By comparing the evidence presented by Greenblat in 1981 to the results of later
research we can draw some conclusions about the progress that has been made over the last 25
years in our understanding of the effectiveness of games for instructional purposes. Table 5
summarizes the evidence, then and now.

Comparing the evidence on the instructional effectiveness of games that was available in 1981
to the results of more recent research shows that there are still many questions that need to be
answered about when and how to use games. The first "claim," that games improve the
motivation and interest of learners, is still weakly supported. There has been little additional
evidence beyond a few studies that use self-reports or time-on-task to show that games are
motivational. Unfortunately, there is little evidence that these measures of motivation are related
to improved task performance. Games do motivate. They motivate players to play the game.
This can be beneficial if the game is designed to target and meet instructional objectives.
Otherwise, learners may spend their time learning to be successful at the game without receiving
instructional benefits from these experiences.

The second "claim," that games enhance cognitive learning, continues to be supported. The
research shows that people can learn from games. However, the research does not indicate that
games are superior to other instructional methods in all cases. Like any instructional activity,
games should be chosen because they provide learners with interactive experiences that help
them meet instructional objectives.

The third and sixth "claims" were not supported. No additional evidence was found that
games change later course work. This effect may occur, but no research was found to support
this "claim." This may be because it is difficult to track learners from one class to another and
document this effect. Likewise, no evidence was found for the sixth "claim," that the use of
games changes the classroom structure and relations. This does not mean that instructors who
use games in their courses do not change their classroom structure or that game players do not
change how they relate to other learners. Some anecdotal evidence indicates that these changes
do occur, but no empirical evidence was found.

Finally, the fourth and fifth "claims," that games change learners' feelings (affect) about the
learning domain and learning in general has mixed support. Two studies (Williams, 1980; and
Bredemeier, et al., 1982) provide some support for the utility of games to change learners'
attitudes toward subject matter (claim 4). No empirical research was found that supports the
claim that games can change learners' attitudes about learning in general (claim 5).

46
 Technical Report 2005-004

Table 5:
Evidence in Support of the "Claims":
Then and Now

Categories of Original Evidence Later Evidence

Claims (prior to 1981) (1981 - 2005)
1. Motivation & Strongest support. A great deal of Little additional evidence. A few studies
Interest anecdotal reports. Only one study found that learners indicated that they
(Robinson et al., 1966) used several enjoyed games and spent more time
indicators of motivation to show playing. However, only weak
simulation-games generated greater connection between this and improved
interest than other modes of teaching. performance.
2. Cognitive Learning Some weak empirical evidence Similar pattern. Some studies show that
favoring games. Some showing no games are effective for some learning
differences, tasks, but do not show them superior to
other instructional approaches. Some
evidence that games can be detrimental
to learning if they do not include
instructional support. Games have been
shown to be more effective if they are
followed by a debriefing session that
highlights the importance of the game
experiences in terms of instructional
objectives.
3. Changes in Later None None Found
Course Work
4. Affective Learning Mixed results. Some anecdotal Two studies provide some additional
(re: Subject Matter) evidence. Empirical evidence shows support. There are indications that a
increases in both positive and in game is more effective if used in the
negative attitudes. appropriate context.
5. General Affective Almost none None Found
Learning
6. Changes in None None Found
Classroom Structure
& Relations

In summary, this review identified several general problems with instructional game research
that make it difficult to draw firm conclusions. These problems include:
* There are many published articles, but most are opinions about the "potential" of games.
There are few empirical studies.
* There is considerable confusion in the use of terms. When is a simulation a game? Are
all competitive activities games? It is often difficult to determine if the instructional
activity was actually a game.
* There are many methodological flaws in the empirical game research, such as poor
experimental designs that lack the control of possible confounding variables (e.g., events
prior to and during the evaluation).

47
 Technical Report 2005-004

* Some of the research appears to be biased by the evaluators' interest in "proving" the
effectiveness of their game (i.e., they developed the game and wanted it to work).
9 Too much research fails to use control groups who experienced an alternate form of
instruction to compare to the game.
e Most published articles provide no clear and detailed description of the game itself (e.g.,
what characteristics made it a "game").

TO GAME OR NOT TO GAME?

As can be seen from the above discussions, the research on the use of instructional games is
mixed. It appears that games can be of instructional value if they are well designed and targeted
to meet specific instructional objectives. Unfortunately, many program managers and game
developers do not appreciate the importance of instructional design. They often assume that the
game is sufficient, in itself, to provide the necessary instruction. Squire (2005) conducted case
studies of three companies that develop game-based learning products. "It is worth noting that
none of the featured companies started in instructionaldesign...they come from business
strategy, marketing, and the games industry" (p. 13, my emphasis). Although each company
used interdisciplinary design teams to create their instructional games, none of the teams
included instructional developers. The teams usually consisted of: 1) graphic artists, 2) program
managers, and 3) programmers. Commenting on the avoidance of instructional designers, Squire
stated, "Most game-based learning approaches do not employ that particular category of expert
whatsoever" (p. 35). In most cases, the game designers fulfilled the role of instructional
developer. It appears that the "instructional gaming" industry does not value the skills of
instructional developers.

This is not a new phenomenon. In the late 1980s, Greenblat (1987) observed that the
teaching enterprise was undervalued in our society. Technological issues involved in game
development seem better able to catch people's interest, while the development of instructional
objectives and logical programs of instruction seems to be boring. The data on instructional
effectiveness of games (e.g., Randel, et al., 1992; Parchman, et al., 2000; Wolfe, 1997) indicate
that the role of the instructor and the way a game is incorporated into an instructional program
are major factors in whether a game will contribute to successful learning. Nevertheless, it is not
clear that current game-development teams understand the principles of instructional
development. It may be up to the program managers and other individuals who procure
instructional games to demand that they support instructional objectives and that this support be
demonstrated. This will then probably require that game companies begin to include individuals
who understand learning and instruction on their development teams. As illustrated in Figure 2,
it is more likely that games will be instructionally effective if the specific characteristics of the
game (e.g., setting, player roles and activities, rules, etc.) overlap with specific instructional
objectives. This overlap must be consciously structured on the basis of a thorough analysis of
the reasons for the instruction and the instructional objectives to be met.

48
 Technical Report 2005-004

Instruction

Figure 2:
Instructional Effectiveness as Degree of Overlap
among Instructional Objectives and Game Characteristics

Rieber (1996) advocated a mixing of simulations and games in a hybrid learning

environment. This approach would use the strengths of simulations (e.g., dynamic and
interactive representations of real-world systems or phenomena) and the strengths of games (e.g.,
challenge and fantasy) to create instructionally sound, realistic learning activities. Additional
research needs to be conducted that will provide guidance about how to create effective hybrid
learning environments (e.g., how to determine if the specific features of specific hybrid games
will support instructional objectives).

Much of the empirical research indicates that instructional games are only effective if they are
designed to support instructional objectives. This is more likely if the design of the instructional
game is the result of a systematic analysis process. Atkinson (1977) advocated that the design
and development of instructional games should follow the same basic systems approach that is
required for the design of any instructional activity. The basic steps in this approach are shown
in Figure 3 as applied to the design and use of instructional games. The first section of the
approach includes steps that are exactly the same, no matter what the outcome (e.g., choosing a
game or another instructional activity). The first two steps are to identify the instructional
problem and the instructional objectives. A problem statement is a general statement of the
overall instructional problem (e.g., improve students overall understanding of and competence in
a topic area). Next, specific instructional objectives must be determined. These objectives
should be stated in terms of observable learner behaviors, under specific conditions that help the

49
 Technical Report 2005-004

SECTION 1: Understanding the Instructional Environment

ieStep 1: Develop Problem Statement

IStep 2: Develop Instructional Objectives

(eng.,aplayStep
3: Select Game Strategyr
SIdentify
Alternative A
Strategies SECTION 2: Deva louina the Game

Game Model
Identify /IStep 4: Develop

Rsources• t
Step 5: Developludents'
Students' Role in Game

Students' ESteps6:&evlo Rulesi

Objectives Eet,&WnigCiei

SECTION 3: Implementing & Evaluating the Game

AStep 7: Develop Supporting Game Resources

(e.g., player's manuals, teacher's manuats) e

r St~ep 8-: Evaluate Game Compared

to Alternate Instruction

bStep 9: Modify Game Based

on Results of Evaluation

Figure 3:
A Systematic Approach to Instructional Game Design and Use

learners reach an acceptable level of performance. These foundational requirements are next
used to help determine the possible alternative instructional strategies and/or approaches that can
be selected. The choice of a specific instructional strategy is then determined on the basis of the
constraints of each instructional situation (e.g., numbers of students, available facilities, budget,
etc.). Only when a gaming approach has been selected from the alternative instructional
strategies does one move to the next section: the design or modification of a specific game.

The second section of Figure 3 shows the basic steps in developing an instructional game.
The fourth step is to develop the game model. Like any model, it is a simplified representation
of only certain elements of reality. The specific elements chosen for the game should be selected
on the basis of how they will support the instructional objectives already identified. Step 5 is the
identification of how the students will participate in the game. Several questions, like the
following, must be answered to support game design. Will each student play the same role or
different roles? Will they play as individuals or as a team? Will learner actions affect the
outcome of the game or do they just observe? These and other questions will lead to the
development of specific rules and events in the game. Finally, the criteria for winning the game

50
 Technical Report 2005-004

must be developed. Winning the game should involve improved learner performance on the
instructional objectives. Sometimes an existing game can be modified for a new instructional
situation. Even if an existing game is chosen, it must be modified to meet the specific
requirements of the new instructional situation.

The third section of Figure 3 shows the three main steps involved in the implementation and
evaluation of an instructional game. In addition to the game, supporting materials, like players'
instructions and teachers' manuals must be developed. As indicated by the data from studies
such as Leutner (1993) and Mayer, et al. (2002), instructional support that explains the purpose
of the game and how to play it can enhance the instructional value of the game. In step 8, the
instructional game is evaluated. Ideally this evaluation should compare the effectiveness of the
game (as measured by learner performance) to the effectiveness of alternate instructional
approaches. The final step in this process uses the results of the evaluation to modify the game
to improve its instructional effectiveness.

In any systematic process, like the one depicted in Figure 3, it is necessary to make specific
design decisions about the game. Malone (1981, 1982) reported the results of a series of studies
that detailed the characteristics that made several simple video games enjoyable to students. As
discussed earlier, he used his results to develop a general framework for analyzing the appeal of
instructional environments, computer games, and computer interfaces (see Table 1). The
framework was based on three categories: challenge, fantasy, and curiosity. Because this
framework is consistent with the important characteristics of games and motivational
instructional environments summarized earlier (e.g., Malone & Lepper, 1987; Lepper & Malone,
1987; Csikszentmihalyi, 1990; Leemkuil, et al., 2000), it has been adapted to help us make
recommendations about the design of effective and motivational instructional games. Table 6
shows these design recommendations applied specifically to instructional games.

One thing that is fairly clear is that instructional games are not a panacea. Like any
instructional medium or approach, games must provide a means for learners to engage in
cognitive and/or motor interactions that directly support instructional objectives. Gratuitous use
of games or the use of games with no clear instructional goals will probably increase the cost of
the instruction without providing the instructional benefit that learners require (see the earlier
discussion of de Jong & van Joolingen, 1998). If a game is chosen as an instructional approach,
it is important that it includes instructional support, such as emphasizing important information
in the game that support the instructional objectives (e.g., Leutner, 1993).

If designed with instructional goals in mind, games can provide effective instruction. Jacobs
and Dempsey (1993) expressed the opinion that "By facilitating learner involvement via
simulation and gaming and incorporating sound instructional features, learning outcomes should
show improvements relative to other training methods that are less engaging or that provide less
effective means of instructional interaction" (p. 198). They suggest that instructional gaming
design should take several factors into account. Instructional games should include relevant
activities that make up the job or domain of interest. The relative "criticality" of the various
activities should be determined and the most critical should be included in the game. Finally, the
performance of learners in the game activities should be evaluated so learners are rewarded for
performing correctly and given remediation to improve incorrect performance.

51
 Technical Report 2005-004

Table 6:
Design Recommendations for Instructional Games
Design Category Recommendations
A. Goal.
Challenge e There should be a clear goal to the game.
"*The goal should be consistent with the instructional objective.
"*Performance feedback should be provided about how close the learner is to
achieving the goal.
B. Uncertain Outcome.
* The outcome of reaching the goal should be uncertain. The learner should
have to exert effort to achieve the goal.
e If the subject matter is complex, the game should include multiple levels of
difficulty or complexity. Sub-goals should help the learner successfully
complete enabling instructional objectives.
C. Competition.
"*A game can be made more challenging by introducing competition.
"*Competition can be against a live opponent, against a computer-controlled
opponent, or against a criterion score. Achieving a criterion score can be one
of the factors that allows a learner to advance to higher levels of difficulty.
* Game scores should be clearly related to the achievement of instructional
objectives (e.g., not just how many shots are fired accurately or how many
opponents are defeated).
A. Emotional Appeal.
Fantasy 9 The fantasy should embody emotional appeal for the learners (e.g., not
everyone reacts positively to shooting weapons or conversely, to complex
interpersonal situations).
B. Fantasy Metaphor.
* The metaphor used in the fantasy should embody physical or other
characteristics that the learner already understands.
A. Optimal Level of complexity.
Curiosity * The game should provide an optimal level of informational complexity to meet
the needs of the learners.
o The level of complexity should increase as learners gain higher levels of
expertise.
B. Incorporation of Interesting Media.
9 The game should use interesting audio and visual effects to enhance the
fantasy and emphasize instructional content.
• The game can incorporate elements of randomness to avoid boredom for the
learner.
A. Logical Instructional Structure.
Instructional a The game should capitalize on the learner's desire to have "well-formed"
Quality knowledge structures.
o Elements of the game should build on and reinforce each other to help build
the learner's knowledge structures.
* Learners should be able to easily see when their knowledge structures are
incomplete. Feedback during the game should be used to help learners
complete their knowledge structures.
B. Incorporation into the Larger Instructional Program.
* The game should not "stand alone." Rather, it should be part of a larger
instructional program.
e Debriefing and feedback should be provided after the game to help the learner
understand how the experiences in the game supports instructional objectives.

52
 Technical Report 2005-004

CONCLUSIONS AND RECOMMENDATIONS

This review of empirical research on the effectiveness of instructional games leads to the
following five conclusions and four recommendations. The conclusions are:
* The empirical research on the effectiveness of instructional games is fragmented. The
literature includes research on different tasks, age groups, and types of games. The research
literature is also filled with ill defined terms, and plagued with methodological flaws.
* Although research has shown that some games can provide effective learning for a variety of
learners for several different tasks (e.g., math, attitudes, electronics, and economics), this does
not tell us whether to use a game for our specific instructional task. We should not generalize
from research on the effectiveness of one game in one learning area for one group of learners
to all games in all learning areas for all learners.
e There is no evidence to indicate that games are the preferred instructional method in all
situations.
* Instructional games should be embedded in instructional programs that include debriefing
and feedback so the learners understand what happened in the game and how these events
support the instructional objectives.
* Instructional support to help learners understand how to use the game increases the
instructional effectiveness of the gaming experience by allowing learners to focus on the
instructional information rather than the requirements of the game.

The following four recommendations may help the instructional gaming industry produce
more instructionally effective games.
o The decision to use a game should be based on a detailed analysis of the learning
requirements and an analysis of the tradeoffs among alternate instructional approaches.
o Program managers and procurement personnel should insist that game developers clearly
demonstrate how the design of a game will provide interactive experiences that support
properly designed instructional objectives (see for example, Gagn6 & Briggs, 1979; Merrill,
1983; 1997 for guidance on the proper design of instructional objectives).
o Instructors should view instructional games as adjuncts and aids to help support instructional
objectives. Learners should be provided with debriefing and feedback that clearly explains
how their experiences with the game help them meet these instructional objectives.
o Instructor-less approaches (e.g., web-based instruction) must include all "instructor
functions." These include performance evaluation, debriefing, and feedback.

Herbert Simon (2000) provides a succinct prescription for designing improved instruction.
Although he was not speaking specifically about instructional games, his prescription applies
equally well to instructional games as to other forms of instructional activities. As Simon puts it,
"A first step toward improved instruction is to examine the understanding we hope students will
acquire, the things they should be able to do with their knowledge... .Next, we must design a
series of experiences that will enable students to learn the relevant cues in the situations they
encounter and to evoke from memory the actions that are effective and appropriate in specific
situations" (pp. 117-118).

53
 Technical Report 2005-004

THIS PAGE INTENTIONALLY LEFT BLANK

54
 Technical Report 2005-004

REFERENCES

Atkinson, F. D. (1977). Designing simulation/gaming activities: A systems approach.

Educational Technology, 17(2), 38-43.

Barak, A., Engle, C, Katzir, L, & Fisher, W. A. (1988). Increasing the level of empathic
understanding by means of a game. Simulation & Games, 18(4), 45 8-470.

Baranauskas, M. C. C., Neto, N. G. G., & Borges, M. A. F. (1999). Learning at work through a
multi-user synchronous simulation game. In Proceedings of the PEG'99 Conference,
Exeter, UK (pp. 137-144). Exeter, UK: University of Exeter. Retrieved on 8/15/2005
from http://www.ic.unicamp.br/-maborges/PEG99.htm.

Belanich, J., Sibley, D. E., & Orvis, K. L. (2004). Instructionalcharacteristicsand motivational

features of a PC-based Game (Research Report 1822). Alexandria, VA: U. S. Army
Research Institute for the Behavioral and Social Sciences.

Benson, 0. (1961). A simple diplomatic game. In J. N. Rosenau (Ed.), Internationalpolitics and

foreignpolicy: A reader in researchand theory (pp. 504-511). New York: The Free Press
of Glencoe, Inc.

Bj6rk, S., & Holopainen, J. (2003, November). Describing games: An interaction-centric

structural framework. Proceedings of the Level Up Conference. Utrecht, The
Netherlands. Retrieved on 7/25/2005 from http://www.digra.org/dl/db/05150.10348.

Bredemeier, M. E., Bernstein, G., & Oxman, W. (1982). BA FA BA FA and

dogmatism/ethnocentrism: A study of attitude change through simulation-gaming.
Simulation & Gaming, 13(4), 413-436.

Bredemeier, M. E., & Greenblat, C. S. (1981). The educational effectiveness of simulation

games: A synthesis of findings. Simulation & Games, 12(3), 307-332. Also in C. S.
Greenblat & R. D. Duke (Eds.). (1981). Principlesand practicesof gaming-simulation.
(pp. 155-169). Beverly Hills, CA: Sage Publications.

Brewster, J. (1996). Computers in teaching: Teaching abnormal psychology in a multimedia

classroom. Teaching of Psychology, 23(4), 249-252.

Bright, G. W., & Harvey, J. G. (1984). Computer games as instructional tools. Computers in the
Schools, 13, 73-79.

Brown, S. J., Lieberman, D. A., Gemeny, B. A., Fan, Y. C., Wilson, D. M., & Pasta, D. J.
(1997). Educational video game for juvenile diabetes: results of a controlled trial.
Medical Informatics,22(1), 77-89.

Caillois, R. (2001). Man, play and games (M. Barash, Trans.). Urbana and Chicago: University
of Illinois Press. (Original work published 1958)

55
 Technical Report 2005-004

Caftori, N. (1994). Educational effectiveness of computer software. Technical Horizons in

Education (THE.) Journal,22(1), 62-65.

Caftori, N., & Paprzycki, M. (1997). The design, evaluation and usage of educational software
(web version). Originally in J. D. Price, K. Rosa, S. McNeil, & J. Willis (Eds.).
Technology and teacher education annual,1997 (CD-ROM edition). Charlottesville, VA:
Association for the Advancement of Computing in Education. Retrieved from
http://www.webcom.com/Joumal/Caftori.html on March 12, 2003.

Carlson, J. G., & Hill, K. D. (1982). The effect of gaming on attendance and attitude. Personnel
Psychology, 35, 63-73.

Cohen, K. J., & Rhenman, E. (1961). The role of management games in education and research.
ManagementScience, 7(2), 131-166.

Costabile, M. F., De Angeli, A., Roselli, T., Lanzilotti, R., & Plantamura, P. (2003). Evaluating
the educational impact of a tutoring hypermedia for children. Information Technology in
ChildhoodEducationAnnual, 289-308.

Costikyan, G. (2002). I have no words & I must design: Toward a critical vocabulary for games.
In, F. Mdiyrd (Ed.), Proceedings of the Computer Games and Digital Cultures Conference
(pp. 9-33). Tampere, Finland: Tampere University Press. Retrieved from on August 1,
2005 from http://www.digra.org/dl/db/05164.51146.

Crookall, D. (1992). Editorial: Debriefing. Simulation & Gaming, 23(2), 141-142.

Crown, S. W. (2001). Improving visualization skills of engineering graphics students using

simple JavaScript web based games. Journalof EngineeringEducation. Retrieved on
8/4/2005 from
http://www.findarticles.com/p/articles/mi qa3886/is 200107/ai n8988842/print.

Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper &
Row, Publishers.

Dauvergn, P. (2000). The case for chess as a tool to develop our children's minds. Retrieved on
8/3/2005 from http://www.auchess.org.au/articles/chessmind.htm.

Day, R. L. (1962). Marketing in action: A dynamic business decision game. Homewood, IL:
Richard D. Irwin, Inc.

de Felix, J. W., & Johnson, R. T. (1993). Learning from video games. Computers in the Schools,
9(2/3), 119-134.

de Jong, T., & van Joolingen (1998). Scientific discovery learning with computer simulations of
conceptual domains. Review ofEducationalResearch, 68(2), 179-201.

56
 Technical Report 2005-004

Dempsey, J., Lucassen, B., Gilley, W., & Rasmussen, K. (1993-1994). Since Malone's theory of
intrinsically motivating instruction: What's the score in the gaming literature? Journalof
Educational Technology Systems, 22(2), 173-183.

Dempsey, J. V., Rasmussen, K., & Lucassen, B. (1996). The instructionalgaming literature:
Implications and 99 sources (Technical Report 96-1). College of Education, University
of South Alabama. Retrieved on 8/5/2005 from
https://www.southalabama.edu/coe/coe/programs/TechReports/tr96 1.pdf.

Department of Defense. (1997, Dec.). DoD Modeling and Simulation (M&S) Glossary (DoD
5000.59-M). Alexandria, VA: Defense Modeling and Simulation Office.

Dorn, D. S. (1989). Simulation games: One more tool on the pedagogical shelf. Teaching
Sociology, 17(1), 1-18.

Driskell, J. E., & Dwyer, D. J. (1984, February). Microcomputer videogame-based training.

EducationalTechnology, 11-16.

Egenfeldt-Nielsen, S. (2003, November). Review of the research on educational usage of games.

Retrieved on July 25, 2005 from http://www.it-
c.uk/people/sen/papers/Reviewing%20the%201iterature%20onsimulations%20games%
20for%201earning vO.5.doc.

Faria, A. J. (1989). Business gaming: Current usage levels. Journalof Management

Development, 8(2), 58-66.

Fisher, J. E. (1976). Competition and gaming: An experimental study. Simulation & Games,
7(3), 321-328.

Fraas, J. W. (1982). The influence of student characteristics on the effectiveness of simulations

in the Principles Course. The JournalofEconomic Education, 13(1), 56-61.

Fritzsche, D. J. (1981). The role of simulation games: Supplement or central delivery vehicle?
Journalof ExperientialLearning and Simulation, 2, 205-211.

Gagn6, R. M., & Briggs, L. J. (1979). Principlesof instructionaldesign ( 2 nd ed.). New York:
Holt, Rinehart and Winston.

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and
practice model. Simulation & Gaming, 33(3), 441-467.

Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from a computer game trainer to
flight. Human Factors,36(3), 387-405.

57
 Technical Report 2005-004

Gredler, M. E. (1996). Educational games and simulations: A technology in search of a

(research) paradigm. In D. H. Jonassen (Ed.), Handbook of researchfor educational
communications and technology: A project of the associationfor educational
communications and technology. New York: Simon & Schuster Macmillan.

Greenblat, C. S. (1981). Teaching with simulation games: A review of claims and evidence. In
C. S. Greenblat & R. D. Duke (Eds.), Principlesand practicesof gaming-simulation
(pp. 139-153). Beverly Hills, CA: Sage Publications.

Greenblat, C. S. (1987). Communicating about simulation design: It's not only (sic) pedagogy.
In D. Crookall, C. S. Greenblat, A. Coote, J. H. G. Klabbers, & D. R. Watson (Eds.),
Simulation - gaming in the late 1980s: Proceedings of the InternationalSimulation and
GamingAssociation's 1 7 th InternationalConference, Universit6 de Toulon et dur Var,
France,1-4 July 1987 (pp. 23-33). Oxford, UK: Pergamon Press.

Greenblat, C. S., & Duke, R. D. (Eds.). (1981). Principlesandpractices of gaming-simulation.

Beverly Hills, CA: Sage Publications.

Gremmen, H., & Potters, H. (1995). Assessing the efficacy of gaming in economics education.
Retrieved on 9/12/2005 from http://amo.uvt.nl/show.cgi?fid=3278.

Hays, R. T., & Singer, M. J. (1989). Simulationfidelity in trainingsystem design: Bridging the
gap between reality and training.New York: Springer-Verlag.

Hogle, J. G. (1996). Considering games as cognitive tools: In search of effective "edutainment."

Department of Instructional Technology, University of Georgia. Retrieved on 7/25/2005
from http://twinpinefarm.com/pdfs/games.pdf.

Jacobs, J. W., & Dempsey, J. V. (1993). Simulation and gaming: Fidelity, feedback, and
motivation. In J. V. Dempsey & G. C. Sales (Eds.), Interactive instruction andfeedback
(pp. 197-227). Englewood Cliffs, NJ: Educational Technology Publications.

Jenson, J., & de Castell, S. (2002). Seriousplay: Challenges of educationalgame design. Paper
presented at AERA Annual Meeting, New Orleans, LA, April 1-5, 2002. Retrieved on
May 6, 2003 from http://www.yorku.ca/jjenson/papers/aera2002.htm.

Ju, E., & Wagner, C. (1997). Personal computer adventure games: Their structure, principles,
and applicability for training. The DATA BASE for Advances in Information Systems,
28(2), 78-91.

Klein, J. D., & Freitag, E. (1991). Effects of using an instructional game on motivation and
performance. JournalofEducationalResearch, 84(5), 303-308.

Koran, L. J., & McLaughlin, T. F. (1990). Games or drill: Increasing the multiplication skills of
students. JournalofInstructionalPsychology, 17(4), 222-230.

58
 Technical Report 2005-004

Laffey, J. M., Espinosa, L., Moore, J., & Lodree, A. (2003). Supporting learning and behavior of
at-risk young children: Computers in urban education. JournalofResearch on
Technology in Education, 35(4), 423-440.

Lederman, L. C. (1992). Debriefing: Toward a systematic assessment of theory and practice.

Simulation & Gaming, 23(2), 145-160.

Lee, J. (1999). Effectiveness of computer-based instructional simulation: A meta-analysis.

InternationalJournalof InstructionalMedia, 26(1), 71-85.

Leemkuil, H., de Jong, T., & Ootes, S. (2000). Review of educational use of games and
simulations (Knowledge management Interactive Training System Project No. IST-1999-
13078). The Netherlands: University of Twente. Retrieved on 8/1/2005 from
http://kits.edte.utwente.nl/documents/D l.pdf.

Lepper, M. R., & Malone, T. W. (1987). Intrinsic motivation and instructional effectiveness in
computer-based education. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning,and
instruction volume 3: Conative and affective process analyses (pp. 255-286). Hillsdale,
NJ: Lawrence Erlbaum Associates, Publishers.

Leutner, D. (1993). Guided discovery learning with computer-based simulation games: Effects
of adaptive and non-adaptive instructional support. Learning and Instruction, 3, 113-132.

Malone, T. W. (1981). Toward a theory of intrinsically motivating instruction. Cognitive

Science, 4, 333-369.

Malone, T. W. (1982). Heuristics for designing enjoyable user interfaces: Lessons from
computer games. In Proceedingsof the 1982 conference on Human Factorsin
Computing Systems, Gaithersburg,MD (pp. 63-68). Association for Computing
Machinery.

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic
motivation for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learningand
Instructionvolume 3: Conative and affective process analyses (pp. 223-253). Hillsdale,
NJ: Lawrence Erlbaum Associates, Publishers.

Malouf, D. B. (1987-1988). The effect of instructional computer games on continuing student

motivation. The Journalof Special Education,21(4), 27-3 8.

Mayer, R. E., Mautone, P., & Prothero, W. (2002). Pictorial aids for learning by doing in a
multimedia geology simulation game. Journalof EducationalPsychology, 94(1), 171-
185.

Merrill, M. D. (1983). Component display theory. In C. M. Reigeluth (Ed.), Instructionaldesign

theories and models: An overview of their currentstatus (pp. 279-333). Hillsdale, NJ:
Erlbaum.

59
 Technical Report 2005-004

Merrill, M. D. (1997). Instructional strategies that teach. CBT Solutions, Nov./Dec., 1-11.
Retrieved from http://www.id2.usu.edu/Papers/Consistency.PDF on June 16, 2004.

Miller, C. S., Lehman, J. F., & Koedinger, K. R. (1999). Goals and learning in microworlds.
Cognitive Science, 23(3), 305-336.

Morris, R. (1976). Simulation in training - Part 4: Business games are not funny. Industrial
TrainingInternational,11(7-8), 241-243.

Morris, R., & Holman, J. (1976). Simulation in training - Part 5. IndustrialTraining

International,11(9), 267-271.

Oblinger, D. G. (2004). The next generation of educational engagement. Journalof Interactive

Media in Education, 8, 1-18.

Oxford, R. L., Harman, J., & Holland, V. M. (1987). Advances in the development of hand-held,
computerized game-based training devices. In D. Crookall, C. S. Greenblat, A. Coote, J.
H. G. Klabbers, & D. R. Watson (Eds.), Simulation - gaming in the late 1980s:
Proceedingsof the InternationalSimulation and Gaming Association's 1 7th International
Conference, UniversitW de Toulon et dur Var, France, 1-4 July 1987 (pp. 65-72). Oxford,
UK: Pergamon Press.

Pange, J. (2003). Teaching probabilities and statistics to preschool children. Information

Technology in ChildhoodEducationAnnual, 163-172.

Parchman, S. W., Ellis, J. A., Christinaz, D., & Vogel, M. (2000). An evaluation of three
computer-based instructional strategies in basic electricity and electronics training.
Military Psychology, 12(1), 73-87.

Parker, L. E., & Lepper, M. R. (1992). Effects of fantasy contexts on children's learning and
motivation: Making learning more fun. Journalof Personalityand SocialPsychology,
62(4), 625-633.

Pierfy, D. A. (1977). Comparative simulation game research: Stumbling blocks and stepping
stones. Simulation and Games, 8(2), 255-268.

Prensky, M. (2001). Digitalgame-basedlearning (on-line version). New York: McGraw-Hill.

Randel, J. M., Morris, B. A., Wetzel, C. D., & Whitehill, B. V. (1992). The effectiveness of
games for educational purposes: A review of recent research. Simulation & Gaming,
23(3), 261-276.

Ricci, K., Salas, E., & Cannon-Bowers, J. A. (1996). Do computer games facilitate knowledge
acquisition and retention? Military Psychology, 8(4), 295-307.

60
 Technical Report 2005-004

Rieber, L. P. (1996). Seriously considering play: Designing interactive learning environments

based on the blending of microworlds, simulations, and games. EducationalTechnology
Research and Development, 44(2), 43-58.

Rieber, L. P., & Noah, D. (1997). Effect of gaming and visual metaphors on reflective cognition
within computer-based simulations. Paper presented at the 1997 AERA conference in
Chicago, IL. Retrieved on 9/14/05 from http://it.coe.uga.edu/-lrieber/gaming-
simulation/Rieber-gaming-simulation.pdf.

Rieber, L. P., Smith, L., & Noah, D. (1998). The value of serious play. EducationalTechnology,
38(6), 29-37.

Renaud, L., & Stolovitch, H. (1988). Simulation gaming: An effective strategy for creating
appropriate traffic safety behaviors in five-year-old children. Simulation & Games, 19(3),
328-345.

Robinson, J. A., Anderson, L. F., Hermann, M. G., & Snyder, R. C. (1966). Teaching with Inter-
Nation simulation and case studies. American PoliticalScience Review, 60(March), 53-
64.

Ronen, M., & Eliahu, M. (2000). Simulation-a bridge between theory and reality: The case of
electric circuits. Journalof ComputerAssisted Learning, 16, 14-26.

Rosas, R., Nussbaum, M. Cumsille, P., Marianov, V., Correa, M., Flores, P., Grau, V., Lagos, F.,
L6pez, X., L6pez, V., Rodriguez, P., & Salinas, M. (2003). Beyond Nintendo: Design
and assessment of educational video games for first and second grade students.
Computers & Education, 40, 71-94.

Rowe, J. C. (2001). An experiment in the use of games in the teaching of mental arithmetic.
Philosophyof Mathematics Education Journal,14. Retrieved on 8/1/2005 from
http://www.ex.ac.uk/-PErnest/pome 14/rowe.htm.

Rowland, K. M., & Gardner, D. M. (1973). The uses of business gaming in education and
laboratory research. Decision Sciences, 4, 268-283.

Serrano, E. L., & Anderson, J. E. (2004). The evaluation of food pyramid games, a bilingual
computer nutrition education program for Latino youth. Journaloffamily and Consumer
Sciences Education,22(1). Retrieved on 8/3/2005 from
http://www.natefacs.org/JFCSE/v22no1/v22no1 Serrano.pdf.

Shrestha, L. B. (1990). Computer based training: The effects of game characteristicson

motivation and learning.Unpublished master's thesis, University of Central Florida,
Orlando.

Simon, H. A. (2000). Observations on the sciences of science learning. Journal ofApplied

Developmental Psychology, 21(1), 115-121.

61
 Technical Report 2005-004

Simons, K. L. (1993). New technologies in simulation games. System Dynamics Review, 9(2),
135-152.

Sleet, D. A., & Stadsklev, R. (1977). Annotated bibliography of simulations and games in health
education. Health EducationMonographs,5 (supplement 1), 74-90.

Squire, K. (2005). Game-basedleaning:Presentandfuture state of the field. Masie Center e-

Learning Consortium, February, 2005. Retrieved on 8/26/2005 from
http://www.masie.com/xlearn!Game-Based Learning.pdf.

Streufert, S. (1986). Assessment of taskperformance via a quasi-experimentalsimulation

technology. Unpublished report, Pennsylvania State University, College of Medicine,
University Park, PA.

Szafran, R. F., & Mandolini, A. F. (1980). Test performance and concept recognition: The effect
of a simulation game on two types of cognitive knowledge. Simulation & Games, 11(3),
326-335.

Thomas, R., Cahill, J., & Santilli, L. (1997). Using an interactive computer game to increase skill
and self-efficacy regarding safer sex. Health Educationand Behavior, 24(1), 71-86.

Thompson, M. (2003). Does the playing of chess lead to improved scholastic achievement?
Issues in EducationalResearch, 13. Retrieved on 8/3/2005 from
http://education.curtin.edu.au/iier/lierl3/Thompson.html.

Thornton, G. C., III, & Cleveland, J. N. (1990). Developing managerial talent through
simulation. American Psychologist,45(2), 190-199.

Van Eck, R., & Dempsey, J. (2002). The effect of competition and contextualized advisement on
the transfer of mathematics skills in a computer-based instructional simulation game.
EducationalTechnology Research and Development, 50(3). Retrieved on 8/3/2005 from
http://www.gamespace.nl/content/MAThesis DBNieborg.pdf.

VanSickle, R. L. (1986). A quantitative review of research on instructional simulation gaming: A

twenty-year perspective. Theory and Research in Social Education,X/V(3), 245-264.

Virvou, M., Katsionis, G., & Manos, K. (2005). Combining software games with education:
Evaluation of its educational effectiveness. EducationalTechnology & Society, 8(2), 54-
65.

Wellington, W. J., & Faria, A. J. (1996). Team cohesion, player attitude, and performance
expectations in simulation. Simulation & Gaming, 27(1), 23-40.

Westbrook, J. I., & Braithwaite, J. (2001). The Health Care Game: An evaluation of a heuristic,
web-based simulation. JournalofInteractive LearningResearch, 12(1), 89-104.

62
 Technical Report 2005-004

White, B. Y. (1984). Designing computer games to help physics students understand Newton's
laws of motion. Cognition and Instruction, 1(1), 69-108.

Whitehill, B. V., & McDonald, B. A. (1993). Improving learning persistence of military

personnel by enhancing motivation in a technical training program. Simulation &
Gaming, 24(3), 294-313.

Wiebe, J. H., & Martin, N. J. (1994). The impact of a computer-based adventure game on
achievement and attitudes in geography. Journalof Computingin ChildhoodEducation,
5(1), 61-71.

Wikipedia. (2005). Game classification. In Wikipedia, the free encyclopedia. Retrieved on July
29, 2005 from http://en.wikipedia.org/wiki/Game classification.

Williams, R. H. (1980). Attitude change and simulation games: The ability of a simulation game
to change attitudes when structured in accordance with either the cognitive dissonance or
incentive models of attitude change. Simulation & Games, 11(2), 177-196.

Wishart, J. (1990). Cognitive factors related to user involvement with computers and their effects
upon learning from an educational computer game. Computers and Education, 15(1-3),
145-150.

Wolfe, J. (1997). The effectiveness of business games in strategic management course work.
(Special Issue: Teaching Strategic Management). Simulation & Gaming, 28(4), 360-376.

Wood, L. E., & Stewart, P. W. (1987). Improvement of practical reasoning skills with a
computer game. Journalof Computer-BasedInstruction, 14(2), 49-53.

63