See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/234793015

Augmented Reality in the Classroom

Article in Computer · July 2012

DOI: 10.1109/MC.2012.111

CITATIONS READS
816 19,285

2 authors:

Mark Billinghurst Andreas Duenser

University of South Australia The Commonwealth Scientific and Industrial Research Organisation
768 PUBLICATIONS 32,630 CITATIONS 101 PUBLICATIONS 4,523 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Andreas Duenser on 22 May 2014.

The user has requested enhancement of the downloaded file.

 C ov er F e at u re

Augmented
Reality in the
Classroom
Mark Billinghurst and Andreas Dünser, The HIT Lab NZ, University of Canterbury

Evaluations of AR experiences in an edu- derstand the earth’s position relative to the sun by reading
cational setting provide insights into how text and seeing a 2D picture, but they can better grasp the
this technology can enhance traditional nuances of that position by visualizing a 3D solar system.
Dynamic processes become animated models, and direct
learning models and what obstacles stand
tangible interaction provides an intuitive way to interact
in the way of its broader use. with the digital content.
Finally, AR can enable new forms of face-to-face and

T
remote collaboration and shared learning experiences.
hrough the years, new technologies have often Multiple users can experience 3D objects from differ-
enabled new opportunities for education. For ent viewpoints, interacting with the object and sharing
example, decades of research have shown that insights, making learning essentially a collaborative expe-
computer technology in the classroom can enrich rience. Such capabilities can become the basis for a range
teaching and learning and boost student achievement, of novel educational experiences.
compared to teaching without such aids.1 Augmented Because of its possibilities for advanced spatial visual-
reality (AR), which enables the seamless connection of ization and interaction, AR provides a superior learning
the digital and physical domains, is one of the newest environment. However, before AR can enjoy wider use in
technologies applied to education. It has already begun to the classroom, researchers must answer important ques-
show promise in helping students learn more effectively tions about the technology’s application in an educational
and increase knowledge retention, relative to traditional setting, such as does AR enhance elementary and high
2D desktop interfaces. school education, and if so, how is AR superior to other
AR supports the understanding of complex phenomena technologies that promote learning?
by providing unique visual and interactive experiences To answer these questions, we informally surveyed user
that combine real and virtual information and help studies investigating AR’s value in both elementary and
communicate abstract problems to learners. With AR, high school classrooms. We found that both research re-
designers can superimpose virtual graphics over real sults and classroom studies of educational AR applications
objects, allowing users to interact with digital content were largely positive, supporting the idea that AR can be a
through physical manipulation. The result is a more ef- valuable teaching tool at these levels. However, our results
fective demonstration of spatial and temporal concepts, also show that evaluation of this technology’s unique ben-
as well as of the contextual relationships among real and efits must be deeper before AR can reach its full potential
virtual objects. For example, students can abstractly un- in an educational setting.

42 computer Published by the IEEE Computer Society 0018-9162/12/$31.00 © 2012 IEEE

 (a) (b) (c)

Figure 1. Reading a story in the MagicBook. (a) The reader views the page as he would in a print book. (b) By using the hand-
held AR interface, the reader can view the virtual story characters in a 3D scene. (c) The reader has “flown” into an immersive
view of the battle scene. (Source: M. Billinghurst, H. Kato, and I. Poupyrev, “The MagicBook—Moving Seamlessly Between
Reality and Virtuality,” IEEE Computer Graphics and Applications, May/June 2001, pp. 6-8.)

THE AR LEARNING EXPERIENCE still viewing it, Mary sees John as a small character in the
AR typically requires a CPU, display, and tracking immersive scene. By providing these interactive options,
system, as well as computer graphics hardware and ap- the augmented book transforms reading into a multiscale
plication software. These components can take many immersive experience.
forms, but two leading formats for AR learning experiences Some early MagicBook interfaces showed educational
are augmented books and AR applications for handheld content, such as a guide for assembling a chair, visualiza-
devices. tion data on oil fields, or plant structure. MagicBook users
found the interface very natural to use and reported that
Augmented books being able to fly in and out of the virtual scene helped them
Augmented books resemble print books except that understand the content in ways that were not possible by
their pages have virtual graphics superimposed on them. viewing static printed pictures. They also enjoyed having
They offer a broad perspective on the educational AR ex- a shared educational experience, in which a learning part-
perience because the pages provide ideal images for AR ner guided them through the content from an external
visual tracking, and even young children know how to top-down view.
open and read books. In a sense, augmented books are A strong advantage of augmented books over print-only
digital versions of the familiar pop-ups, in which a 3D card- books is the additional interactivity. Users can manipu-
board construction rises off the page as the reader opens late the book by rotating or tilting the pages to experience
the book. The virtual content can provide an animated the virtual content from different positions, or by flipping
scene that complements print content, and in some cases pages to show different AR scenes. These basic interaction
supports simple interactivity. techniques are natural extensions of the physical book,
The MagicBook is an example of an educational but augmented books can also support more sophisticated
augmented book.2 The book’s initial version featured a interactions. Gaze input, for example, lets users interact
handheld display and transitional interface. Other ver- simply by looking at the page. Using gesture input, the user
sions with other interface types have since appeared. As points at different parts of the page and triggers actions
Figure 1 shows, MagicBook users hold an AR display to such as highlighting or moving virtual objects or starting
view virtual content overlaid on the real book pages. In an animation.3 Tangible user interfaces provide novel and
addition to seeing AR scenes, users can push a button on intuitive ways for students to learn and can support playful
the display and fly into a fully immersive virtual environ- and collaborative learning.
ment. To explore that environment, a user simply turns in Researchers have built on these MagicBook experiences
the desired direction and touches a second button to fly to create other augmented books, and in 2008, the first
in that direction. commercial augmented books became available. Since
The MagicBook also supports collaborative viewing so then, interest in AR has grown considerably, and more
that two or more people can explore the same content. If augmented books have entered the market, many provid-
John flies into the page’s immersive scene while Mary is ing educational content.

JULY 2012 43
 C ov er F e at u re

with a traditional interface used a digital map to access the

same information.
The study found that the AR interface facilitated col-
laboration. Those using an AR interface often pointed to
the map and environment. The pointing helped establish
common ground and served as a starting point for discus-
sion and problem solving among the users.
With indoor mobile AR applications, users tend to inter-
act much like readers interacting with augmented books.
Computer vision software running on the mobile phone
tracks printed images and overlays virtual content onto
them.
Whether they are outdoors or indoors, mobile AR ap-
Figure 2. The CityViewAR application showing the history plication users enjoy engaging in a topic through game-like
of earthquake-damaged Christchurch. The application experiences, and developers have already created some in-
uses the mobile phone’s GPS and compass sensors to teresting education applications, such as having a 3D virtual
superimpose a virtual undamaged building on the site of
character teach users about famous works of art.5 Game-
the real building remains.
engine technology animates the character and creates an
entertaining encounter, which users find more enjoyable
Mobile AR applications than learning about art from traditional print sources.
As smartphone use has dramatically increased over
the past few years, interest in mobile AR applications has EFFECTIVE AR USE
grown. Current smartphones and tablets combine a fast Providing AR experiences does not necessarily mean
processor with graphics hardware, a large touchscreen, that people are learning more effectively from them. As
and onboard sensors (camera, GPS, compass, accelerom- with any technology, important lessons must be learned
eters), making them ideal for both indoor and outdoor AR about how best to use AR in an educational setting.
experiences. Researchers and commercial developers have
used these platforms to create educational applications, Blending AR with traditional learning
providing novel learning experiences. AR might be most effective as a complement to tradi-
With mobile devices, users can have an AR experience tional learning methods. Schools could use augmented
anywhere, which means that students can remain actively books along with normal print material as a way to teach
engaged in the learning process outside as well as inside storytelling. In one such storytelling workshop,6 sponsored
the classroom. CityViewAR (www.hitlabnz.org/cityviewar) by the HIT Lab NZ, teachers showed 10- to 14-year-olds an
is an example of unique experiential learning. As Figure 2 augmented book version of Giant Jimmy Jones by Gavin
shows, students can use this mobile phone application to Bishop. Animated virtual scenes appeared over the physi-
walk through the city of Christchurch and “see” buildings cal pages, enhanced by audio effects, including a sound
as they were before the 2011 earthquake made it necessary track of the author reading the story (www.youtube.com/
to demolish them. The phone GPS and compass sensors set watch?v=M7bZxZXy2o8). Teachers then taught the stu-
the user’s location and viewpoint so that the application dents about narrative, storytelling, graphic design, 3D
can position the virtual reconstructed building within the modeling, and animation using a mixture of whiteboard
remains of the real building. The user can touch the build- lessons, print material, and digital technology. Finally,
ings on the screen to see a history of the building on the they asked the students to work together to create their
site along with geolocated panoramic photographs taken own story version.
directly after the earthquake. Many mobile AR applications AR technology was a minor part of storytelling cur-
provide a similarly realistic experience at the physical site riculum, but it was the part that enticed students to delve
that directly relates to a lesson topic. more deeply into related topics. Although few of those
Studies show that providing AR experiences on mobile participating in the workshop had any experience with
devices can have unique benefits over offering non-AR modeling software or computer graphics, they all suc-
content on the same topic. One study compared the expe- ceeded in producing storyboards and creating their own
riences of people using different mobile phone interfaces 3D scenes, which they displayed in augmented books.
to play a location-based treasure hunt that integrated Figure 3 shows some of the projects.
learning and environmental exploration.4 Users with the The children overwhelmingly enjoyed the workshop
AR-based interface could augment a physical map of the and were excited about the technology. Teachers observed
environment and get additional information, while users each day and were impressed by how the technology sup-

44 computer
ported the learning of a wide range of skills. As one teacher
observer commented, “[the augmented book technology]
provides a multisensory approach to learning that links
text, image, sound, and movement and is a highly motiva-
tional communication format.”
The teachers’ report concludes that this activity exer-
cised the children’s problem-solving skills and that there
is “no question that AR will prove to be a highly effective
medium both for entertainment and education.” Since
the workshop, the HIT Lab NZ has run a series of similar
classes with comparable outcomes.

Authoring tools
Building AR scenes can be an educational experience
in itself, since students must think about how to use the
technology to represent complex concepts. Suppose, for
example, that the assignment is to make an AR scene of
the solar system. Once students learn about the planets
and their motion, they will need AR content-creation tools
to build the scene. To meet that need, we are developing
simple authoring tools that students without programming
expertise can use to create AR scenes.
Figure 4 shows the use of one such tool, BuildAR (www.
buildar.org), which lets users design simple AR scenes
through a graphical user interface without writing code.
Users can load virtual text, images, video, or 3D content
and attach them onto tracking markers or printed images,
thus quickly creating the AR scene. They can choose their
own tracking images and use an intuitive mouse-driven 3D
user interface to position content on the images. Teachers
and students can even add AR content to existing printed Figure 3. Students and their 3D models. As part of a story-
educational material. telling workshop students created 3D scenes after reading
As of April 2012, users have downloaded more than an augmented book version of Giant Jimmy Jones. The stu-
25,0000 copies, making BuildAR one of the most popular dents had little or no experience with modeling software
AR authoring tools for schools. or computer graphics. (Image reprinted with permission
from the University of Canterbury).
We are using BuildAR with free modeling software such
as The Art of Illusion (www.artofillusion.org) to create
teaching modules for grade-school children. Students as
young as age seven have successfully used the software
to build AR scenes. In doing so, they use a variety of skills,
such as mathematical problem solving, 2D and 3D digital
content creation, drawing, and storytelling.
Teachers can use BuildAR to support a range of activities
by integrating AR into their traditional curriculum. The
tool is simple enough that users can focus on the educa-
tional content without worrying about interface or coding
issues. Figure 5 shows two sample scenes that users have
developed.
Our experience with AR content-creation tools in the
classroom provides evidence that even young children
can create AR scenes and that the creation process can Figure 4. Student using the BuildAR authoring interface.
be a powerful educational experience. To create an AR To create an AR scene, the student simply loads content
scene, children must understand the topic, which pro- onto tracking markers or printed images. Virtual content
can be text, images, video, or a 3D object.
motes learning. Creating content is an engaging creative

JULY 2012 45
 C ov er F e at u re

(a) (b)

Figure 5. Content that users have created using BuildAR. (a) The Stirling engine is fully animated with virtual text tags
explaining each machine part, while (b) the physics book provides 3D models and gives the mathematical formulas for each
model shown.

experience that can impart practical 3D design skills while and using handheld paddles to move virtual characters and
deeply involving the children with educational content and solve simple puzzles. The children work in pairs, which fa-
problem solving. cilitates collaborative problem solving and increases their
chances of finding new solutions.8
EVALUATING EDUCATIONAL BENEFIT The augmented book combines text-based sections and
Despite these positive AR content-creation experiences, interactive AR sections. We compared how much readers
an important question remains: Does AR technology in- with high and low ability (teacher-provided classifications)
crease learning? Building and using AR scenes combines remembered from the two media formats. From text-based
active complex problem solving and teamwork to create story sections, those in the high-ability group could retell
engaging educational experiences to teach science, math or significantly more key story points than the low-ability
language skills, and studies have found that this activity en- group.9 However, the two groups showed no significant
hances student motivation, involvement, and engagement.7 difference in recalling key story points from the inter-
However, what is the real education benefit from using this active AR sections. Arguably then, because they provide
technology in the classroom? Answering this question can an opportunity for interactive content engagement, aug-
be challenging because education quality stems from many mented books could benefit students who are less able to
factors, some of which are hard to quantify. comprehend text-based learning materials—currently the
predominant education medium.
Reading comprehension Interactivity seems to be central to content engagement.
Story recall and reading comprehension are two areas Children referred to the augmented books as games, and
in which we have seen positive educational results. In when asked what they liked best, they consistently named
Figure 6, children are reading an augmented storybook the interactive events. These comments indicate that AR
educational media could be a valuable and engaging ad-
dition to classroom education and overcome some of the
limitations of textbook-based methods, allowing students
to absorb the material according to their preferred learn-
ing style.

Teaching topics with a spatial component

AR can also be more effective in teaching topics with a
strong spatial component. In a study after the reading com-
prehension evaluation, we compared learning outcomes
using a print-only text about electromagnetism and an aug-
mented book with the same content. As Figure 7 shows, the
augmented book provides animated 3D models of electro-
Figure 6. Using an augmented book in a collaborative set- magnetic concepts and permits simple interaction, such as
ting. Working in pairs, children use paddles to move virtual
characters and solve puzzles.
letting the user change a magnet’s polarity by pointing or
using a paddle.

46 computer
 In a test administered directly after exposure to the
two books, the AR group answered more questions cor-
rectly—both factual and inferential—with a mean score
of 72 percent, relative to 60 percent for the non-AR group.
In a follow-up retention test administered four weeks later,
the AR group again scored significantly higher, with a 55
percent score, relative to 45 percent for the non-AR group.
From these results, it appears that the AR interface’s ad-
ditional interactivity and visual representation enhance
subject learning.

Overall value
Most educational AR systems are single-use prototypes
for specific projects, so it is difficult to generalize evalu-
ation results. Overall, augmented books are promising
educational tools, both in the laboratory and real class- (a)
room settings. They can enhance learning with print-only
books or traditional interfaces by offering novel ways to
experience educational content. The intuitive interac-
tion possibilities and spatial visualization of 3D content
that augmented books afford seem to be beneficial for
learning. AR can also add more meaning to topics that
students cannot possibly experience in the real world.
For example, undergraduates used AR to learn about the
relationship of the earth to the sun in terms of axial tilt
and solstices, which the augmented book depicted as real
physical props.10
The results of our reading comprehension study show
that AR seems to particularly benefit students who strug-
gle in traditional text-based learning environments. Our
results support earlier research that shows the role of (b)
multimodality and interactivity in increasing engagement
and immersion and in general learning support. Some re- Figure 7. Using an interactive augmented book to teach
searchers argue that interactivity can promote learning by electromagnetism. (a) The user views a 3D model of the
activating certain cognitive processes.11 Interactivity can earth’s magnetic field. (b) The student uses a paddle to
change a magnet’s polarity.
activate knowledge stored in long-term memory and cause
the brain to integrate it with incoming information. Stu-
dents who interact with the content can remember more only on visually enhancing the real world, but they have
than students who receive information only passively and established several key ideas about AR use in the classroom:
better transfer what they have learned to new problems.
Interacting students work harder to make sense of the •• AR technology is robust enough to deliver learning ex-
material, and they rate their interest in the material higher. periences, especially in augmented books and mobile
So far, however, relatively few user studies have in- AR applications;
vestigated AR’s educational value in classroom settings. •• AR experiences should complement rather than re-
Conclusive results require more thorough study of educa- place traditional curriculum material;
tional AR experiences and how they can enhance learning. •• valuable learning occurs during the development of
Most studies test learning only right after training. Ideally, AR content as well as in using the AR application itself;
future studies will investigate longitudinal effects as well and
and evaluate the retention of educational benefits. •• AR provides real benefit for reading comprehension
and in understanding spatial data, especially for those
WHAT’S NEXT? with low reading ability.
AR use in the classroom is still relatively new, but the
technology’s impact is increasing as AR platforms become As AR use broadens, however, the education model is
more widely available. Current AR experiences have focused likely to be quite different for students who have AR-ready

JULY 2012 47
 C ov er F e at u re

mobile phones in their pockets and easy-to-use content text documents or presentation files that students now use
tools at hand. in classrooms. This is very different from most standard
educational practices in which students are meant to either
More media channels passively consume lecture-style classes or learn mainly
Future educational AR experiences will build on the through the visual channel by reading textbooks.
results of studies with a visual-only channel, adding audio
and kinesthetic channels and incorporating interactivity in Purposeful design
myriad ways. For example, while walking around with a Current AR educational experiences are largely the ad
mobile AR application, the student should be able to hear hoc creation of either educators with little understanding
audio linked to a specific location, as well as see visual of the technology or developers with little understanding of
cues. This multimodality can be part of materials that education. The design of next-generation AR experiences
enrich the learning experience and convey ideas in various will be purposeful. Education and sound learning theory
forms. Such materials can cater to students with different will be at the forefront of design decisions from the bottom
learning preferences and styles and actively engage them up. Such thoughtful design will aid in maximizing learning
in new ways. outcomes and ensure that educators apply AR technology
where it is likely to have the most benefit.

R
Mobile AR systems are especially esearch results and classroom studies of educa-
useful for learning activities that get tional AR applications strongly suggest that the
students actively engaged both inside technology can be a valuable teaching tool. Aug-
and outside the classroom. mented books are an extension of traditional books and
can be a good means to enrich traditional classroom
teaching and learning at home. Although most applica-
tions have focused on young children, these books are
The current focus is on augmented books and mobile suitable for students of all ages.
AR applications with a standard form factor, but the next Results show that AR’s high level of interactivity en-
generation of AR technology will support a wider variety of hances learning, particularly for students who learn
interfaces and devices. Hardware such as flexible and fold- through kinesthetic, visual, and other non-text-based
able displays, 3D capture cameras, or pico projectors could methods. Engaging children in creating their own AR ex-
provide new ways to view and interact with AR content. For periences provides a further learning avenue. Mobile AR
example, by using a 3D capture camera in an augmented systems are especially useful for learning activities that get
physics textbook, students could use their hands to directly students actively engaged both inside and outside the class-
manipulate a virtual magnet to change its polarity, rather room. Such applications motivate students to explore their
than relying on separate tangible input objects, such as the surroundings and collaboratively develop their problem-
paddle in the augmented book about electromagnetism. solving skills. These systems seem particularly suitable for
Providing AR experiences in this hands-free manner would high school settings.
extend the interaction space into the real world and provide Some obstacles remain in making AR experiences
intuitive visualization as users move around 3D models. part of the average classroom. One is the lack of content-
creation tools. Many educational content developers, such
Broader experiences as teachers, do not have the highly developed program-
Future AR technology will provide a wider range of ming and 3D modeling skills currently required to design
educational experiences. New interactive learning applica- AR experiences. Unless tools become usable without such
tions can initiate actions impossible in the real world, with skills, AR interfaces most likely will not catch on in the
unique consequences. For example, students can use AR to mainstream curriculum.
see and manipulate magnetic field lines with their hands or In addition, as more AR systems become mobile, new
see history superimposed over the location in which they opportunities for creating highly engaging and interactive
are standing. This exploration into novel areas opens pos- educational experiences will emerge. Future research must
sibilities in teaching practical as well as abstract concepts. address the unique opportunities and challenges of mobile
Technology will continue to define content creation. systems and how they can enrich classroom teaching, as
Students will be able to create their own 3D AR content well as offer novel authoring solutions for students and
and experiences in the real world, thus expanding the educators.
classroom to outside locations or print media to physical In many ways, AR technology is where 2D interactive
objects. Building AR scenes will be as easy as creating the multimedia was in the 1980s. In the decades since, mul-

48 computer
timedia technology developed to the degree that young 8. A. Dünser and E. Hornecker, “Lessons from an AR Book
children can now create their own games and interactive Study,” Proc. 1st Int’l Conf. Tangible and Embedded Interac-
experiences. Likewise, we expect AR technology to mature tion (TEI 07), ACM, 2007, pp. 179-182.
enough that students in 2030 will be routinely building AR 9. A. Dünser, “Supporting Low-Ability Readers with Inter-
active Augmented Reality,” Ann. Rev. CyberTherapy and
educational content, thereby tightly connecting the class- Telemedicine: Changing the Face of Healthcare, Interactive
room experience to the world around them. Media Institute, 2008, pp. 41-48.
10. B. Shelton and N. Hedley, “Using Augmented Reality for
Teaching Earth-Sun Relationships to Undergraduate Geog-
References raphy Students,” Proc. IEEE Int’l Augmented Reality Toolkit
1. R. Tamim et al., “What Forty Years of Research Says about Workshop, IEEE CS, 2002, p. 8.
the Impact of Technology on Learning: A Second-Order 11. R. Moreno and R.E. Mayer, “Role of Guidance, Reflection,
Meta-Analysis and Validation Study,” Rev. Educational Re- and Interactivity in an Agent-Based Multimedia Game,”
search, Mar. 2011, vol. 81, no. 1, pp. 4-28. J. Educational Psychology Rev., vol. 97, no. 1, 2005, pp.
2. M. Billinghurst, H. Kato, and I. Poupyrev, “The Magic- 117-128.
Book—Moving Seamlessly Between Reality and Virtuality,”
IEEE Computer Graphics and Applications, May/June 2001, Mark Billinghurst is the director of the HIT Lab NZ and a
pp. 6-8. professor of human interface technology at the University
3. R. Grasset et al., “The Mixed Reality Book: A New Multime- of Canterbury. His research interests include developing
dia Reading Experience,” Proc. 25th Conf. Human Factors innovative computer interfaces that explore how to merge
in Computing Systems (CHI 07), ACM, 2008, pp. 1953-1958. virtual and real worlds. Billinghurst received a PhD in
4. A. Morrison et al., “Like Bees Around the Hive: A Compara- electrical engineering from the University of Washington.
tive Study of a Mobile Augmented Reality Map,” Proc. 27th He is a member of IEEE and ACM. Contact him at mark.
Int’l Conf. Human Factors in Computing Systems (CHI 09), [email protected].
ACM, 2009, pp. 1889-1898.
Andreas Dünser is a research scientist in the HIT Lab NZ at
5. D. Wagner, D. Schmalstieg, and M. Billinghurst, “Handheld
the University of Canterbury. His research interests include
AR for Collaborative Edutainment,” Proc. 16th Int’l Conf.
interaction design, human-computer interaction, the evalu-
Artificial Reality and Telexistence (ICAT 06), Springer, 2006,
pp. 85-96.
ation of user experiences with novel computer interfaces,
and the application of AR- and VR-based technologies in
6. J. McKenzie and D. Darnell, The eyeMagic Book. A Report
into Augmented Reality Storytelling in the Context of a
education, training, assessment, and rehabilitation. Dünser
Children’s Workshop, 2003, New Zealand Centre for Chil- received a PhD in psychology from the University of Vienna.
dren’s Literature and Christchurch College of Education, He is a member of IEEE and ACM. Contact him at andreas.
2004. [email protected].
7. P. O’Shea et al., “Lessons Learned about Designing Aug-
mented Realities,” Int’l J. Gaming and Computer-Mediated Selected CS articles and columns are available
Simulations, no. 1, vol. 1, 2008, pp. 1-15. for free at http://ComputingNow.computer.org.

JULY 2012 49
View publication stats