Stackable Music: A Marker-Based Augmented Reality Music

Synthesis Game
Max Chen Shano Liang Gillian Smith
[email protected] [email protected] [email protected]
Worcester Polytechnic Institute Worcester Polytechnic Institute Worcester Polytechnic Institute
Worcester, Massachusetts, USA Worcester, Massachusetts, USA Worcester, Massachusetts, USA

Figure 1: An illustration of stacking transparent AR markers and photos of the Stackable Music application. When multiple
transparent AR markers are stacked and recognized by the camera, the application takes them as input and generates a variation
of jazz.

ABSTRACT ACM Reference Format:

Augmented reality (AR) allows the rendering of digital content on Max Chen, Shano Liang, and Gillian Smith. 2023. Stackable Music: A Marker-
Based Augmented Reality Music Synthesis Game. In Companion Proceedings
top of the physical space, which is a promising medium for tangible
of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY
interaction. Marker-based AR is widely used thanks to its low cost Companion ’23), October 10–13, 2023, Stratford, ON, Canada. ACM, New York,
and ease of integration, but the gameful aspect of manipulating NY, USA, 7 pages. https://doi.org/10.1145/3573382.3616071
the physical AR markers remains understudied. In this paper, we
explored the stacking mechanics of transparent AR markers and 1 INTRODUCTION
described the creation of an AR music game called Stackable Music.
Augmented Reality (AR) technology merges digital content with
Stackable Music can be developed, assembled, and set up at the
the physical environment [1], which is an emerging field of study
home or office with a printer using several sheets of transparent
in human-computer interaction (HCI). Based on the method of map-
film and a PC or mobile device with a camera.
ping digital objects to physical surroundings, AR can be divided
into two categories: markerless and marker-based [30]. AR markers
CCS CONCEPTS function as reference points in the physical environment, helping
• Human-centered computing → Mixed / augmented reality; to place virtual objects and content in the user’s field of view. Mark-
Human computer interaction (HCI); Interface design prototyping. erless AR identifies anchor objects, like flat surfaces (such as walls,
ceilings, and tabletops), or human body parts such as limbs and
KEYWORDS hands, as reference points in the physical environment. Despite
Marker-based AR, AR Game, Tangible Interaction, Transparent AR significant progress in head-mounted AR devices and marker-less
Markers AR interactions, marker-based AR maintains extensive usage due
to its affordability and simplicity of integration. This approach
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed enables tangible interaction and serves as a liaison between physi-
for profit or commercial advantage and that copies bear this notice and the full citation cal and digital spaces [4], thereby asserting its importance in the
on the first page. Copyrights for components of this work owned by others than the post-screen era [16]. The tangible characteristic of printed mark-
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission ers has gained justifiable popularity in fostering creative activities
and/or a fee. Request permissions from [email protected]. [39], storytelling [7, 13, 24, 38], games[21], learning[6, 11], physical
CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada prototyping [20], and enhancing social interaction [35].
© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 979-8-4007-0029-3/23/10. . . $15.00 Using marker-based AR in card games is an intuitive design
https://doi.org/10.1145/3573382.3616071 decision, as cards are physical media for printing images or QR
CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada Max Chen, Shano Liang, and Gillian Smith

the AR marker, for example, by putting the markers on the surfaces

of a cube[17] or on everyday objects[8]. Other works looked into
materials that allow dynamic changes. For example, HueCode uses
a physical bandpass filter so that the robot camera can capture
both ArUco and QR codes printed in different colors on plain paper
[28, 34]. Peiris et al. used thermochromic inks to create dynamic
AR markers so that the pattern can morph into a new one for each
temperature range[29].
More recently, scholars started to investigate intra- and inter-
marker interactions. Intra-marker interaction uses physical defor-
mation and paper crafting to influence the presence and absence of
a marker. Zheng et al. explored tangible experiences by combining
paper mechanics with printed fiducial markers [37]. An interesting
aspect is that the markers are not directly presented in the front
Figure 2: Renowned existing fiducial marker systems includ- scene for interaction but rather positioned in the background for
ing ARToolkit, ArUco, ARTag, InterSense, and ReacTIVision. the camera to capture. Building upon this research, Tinycade, an
codes. AR can add dynamics and graphical elements by overlaying introductory platform for alternate controller design, employs rota-
animated characters, visual effects, and 3D models onto the physical tion and the presence or absence of AR markers in the background
card, which is helpful in reminding players of the rules and spells in to detect and represent "joystick" behaviors [14, 15]. Inspired by
the game [3]. AR can be used to recognize card and player actions to [37], Scheirer and Harrison presented DynaTags, a comprehensive
help calculate the damage and result of a battle[22]. AR card games library of 23 mechanisms that enhance the content payload of indi-
are also widely used in educational settings, such as allowing users vidual markers [31]. One significant distinction is that DynaTags
to mimic physical actions such as shaking and pouring in chemistry works with a standard and unmodified smartphone, providing a
experiments using card-based or box-based marker design[23]. versatile solution for marker-based interactions. DynaTags intro-
Fiducial markers, artificial patterns which balance fast and pre- duced stacking mechanics by cutting windows into the markers
cise pose estimation with a straightforward and cost-effective de- and letting information through from a lower layer. When the struc-
ployment, play a critical role [10]. As illustrated in Fig.2, renowned ture is purposefully designed, the physical ordering of the stacked
fiducial marker systems include square-shaped marker systems elements can be conveyed and deciphered [2].
such as ARToolkit[19], ArUco[12], and ARTag[9], and irregular- Inter-marker interaction uses a combination of relationships (e.g.,
shaped marker systems such as InterSense[27] and ReacTIVision[18]. distance, presence, and absence) to serve as input variables for the
Using fiducial markers rather than image markers allows standard- interactive system [26, 32]. This technique has been widely applied
ized production because it does not require individual training over towards using a camera to detect the physical distance between
each image tracker. locations.
However, the use of printed fiducial markers in HCI introduces All of the aforementioned works are designed and intended
several limitations. For instance, they are permanent; once printed, for fiducial markers to be printed on plain paper, while our work
few modifications can be made. In addition, most AR applications explores the possibilities of transparent film.
only use AR markers as reference points. The physical affordances
of materials and the interaction of AR markers are less studied and 2.2 Transparent Film Paper and AR Markers
utilized in playable media. One unique inter-marker mechanic that Transparent film papers are thin sheets with coatings that allow
can bring creative potential to AR interaction is stacking. When AR light to pass through without significant distortion. They are com-
markers are printed on transparent film and stacked, the formation monly used in crafting projects for various creative purposes. There
of new markers can be captured and recognized by the camera, have been works studying transparent markers from a technical
thus becoming a new input modality to the interactive system. perspective. Teixeira et al. presented a study titled "Towards Trans-
In this paper, we describe using transparent film as a physical parent Marker," which showcased the marker’s unique capabilities
material for AR markers and experiment with stacking transparent of being visible from both sides and the marker interference in-
AR markers. As a proof-of-concept, we developed Stackable Music, volved in stacking[33]. Zhang et al. proposed a method of stack-
a card game that leverages stacking transparent AR markers to ing multiple transparent colored AR markers in space to display
enhance tangible interaction in music synthesis. Further, we discuss and manipulate multiple objects’ order and rotation, proposing
the affordances and limitations of the transparent AR markers and the mathematical constraints involved in marker design and the
propose future research directions. implementation of detecting the stacking order [36].
Our work delves into the HCI aspect and outlines the design of
2 BACKGROUND the stacking mechanism for AR interactions.
2.1 Dynamic Fiducial Markers
Previous studies in dynamic characteristics of fiducial markers 2.3 AR in Musical Expression
proved the potential of multimodal physical-digital experiences. Researchers and practitioners are enthusiastic about applying AR in
Some works investigated the physical shape of the media to hold artistic and musical expressions. Zünd et al. proposed the concept
Stackable Music CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada

of Augmented Creativity and illustrated 6 prototyped applications

to enhance real-world creative activities, one of which is a music ar-
rangement application that uses image-based markers for a mobile
application to detect and render 3D models of the instruments and
audio[39]. Lucas reported the design of AuSynthAR, an affordable
AR musical instrument that requires only a mobile device, a set of
tokens, a sound output device, and, optionally, a MIDI controller
[25]. Kelly et al. presented ARcadic with an implementation of
a "disc jockey (DJ)" table interface for music performance using Figure 3: An example of stacking two markers from the stan-
fiducial markers and cardboard [20]. DynaTags introduced a music dard ArUco dictionary generates a pattern that is not in-
synthesizer with modular blocks that can be connected to create cluded in the dictionary.
looping sounds, where a sequence of three blocks displays two
markers detected by an overhead phone, allowing the creation of
songs by assembling multiple sequences [31].
Our work uncovers the creative potential of the stacking me-
chanic and transparent AR markers by experimenting with proto-
typing and analyzing them in a music arrangement activity.

3 DESIGN
3.1 Music Arrangement
Figure 4: An example of marker interference. In this example,
Music arrangement is an activity in music synthesis that involves two markers have overlapping patterns, causing the stacked
creating and arranging music through computer software or a syn- deck indistinguishable.
thesizer. We explore tangible AR interaction in a trading card game
in which individuals with varying degrees of musical knowledge
can work together to create music. We classify musical expressions each marker causes on the others. Figure 4 shows an example in
into three categories: drums (like claps, kicks, hats, cymbals, and which two markers have overlapping patterns, causing the stacked
rims), instruments (such as bass, guitar, keyboard, and orchestral), marker cannot be distinguished. To bypass the complexity of over-
and vocals. When creating a mix, the guideline is to use no more lapping patterns, a simple solution is to create a set of markers
than one selection from each category. In this work, music samples in which each two of the markers have no interference. Inspired
are collected from free sources online. The selection is purposeful by "orthogonal" markers [33], here we introduce the concepts of
to encompass a variety of styles. "group" and "ring". A group is a set of markers; only markers from
different groups can be stacked together. When groups are allocated
3.2 The Design of Transparent AR Markers distinct territories that do not overlap, stacking markers from differ-
The design of the transparent AR marker uses Custom Marker ent groups will not result in any interference among markers. One
Creator1 , a design tool to create a customized ArUco dictionary. An way to allocate distinct territories is to divide the marker content
ArUco marker is a square marker composed of a black border and an into concentric squares, where each square locates at a different
inner content that determines its identifier through a binary matrix. "radius". As a matter of convenience, we refer to these segments as
Predefined ArUco dictionaries are useful for ease of integration "rings". When the marker’s width is an odd number, the central bit
and robust camera pose estimation. Designing a customized ArUco stands alone. While a single bit might convey minimal information,
dictionary is necessary because combining multiple markers was it can serve as a toggle button. This functionality proves valuable
not considered when the standard ArUco dictionaries were created. in the context of stackable marker mechanics.
For example, as illustrated in Figure 3, stacking ID = 1 and ID = 2 In Stackable Music, we utilized a 7x7 marker template, consisting
from the standard ArUco dictionary creates a pattern that is not of 3 rings and a central square. As illustrated in Figure 6, Each
included in the same dictionary. category (Drum, Instrument, and Vocal) is assigned a unique layer
Customizing an ArUco dictionary starts with indicating the num- and corresponding ring. The central square functions as a Stop-and-
ber of bits in the marker content by determining the dimensions of Play control; audio can only be played when this square is present
the square matrix. Typical matrix dimensions include 5x5, 6x6, and as shown in Figure 7. The area outside the marker is left open for
7x7. The size of the matrix influences the number of identifiable graphic design. In this instance, as seen in Figure 8, we opted for a
markers in the dictionary and the expressive space for individual simple hand-drawn frame with the letter "E" positioned in the top
marker content. left corner to indicate the marker’s front/back side and orientation.
Another restriction in designing stackable transparent AR mark-
ers is marker interference. To ensure the stacked pile of markers is 3.3 Unity Implementation
recognizable, the maximum number of markers and their orienta- We developed Stackable Music AR using Unity. OpenCV for Unity 2
tion are significant factors, as they are related to the interference is a paid Unity plug-in that allows developers to use OpenCV[5],
1 https://jingruchenmax.github.io/StackAR/ 2 https://enoxsoftware.com/opencvforunity/
CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada Max Chen, Shano Liang, and Gillian Smith

Figure 5: An illustration of dividing the marker content into Figure 8: A decorative frame surrounds the marker on the
rings. outside.

Figure 9: An example of a combined marker that consists of

four groups, including drum, instrument, vocal, and play.
Figure 6: Each music category occupies a layer and is assigned
a ring area and a central square as an on/off button.
in Fig. 9. All combinations of markers from different groups are
calculated and added to the marker dictionary for detection. The
combined markers consist of multiple basic markers, meaning that
the audio, model, and particle effects are also derived from the basic
markers.

3.4 System Setup

Stackable Music requires transparent AR markers, a computer, and
a camera, shown in Fig. 10. It is recommended to use a top-down
camera with a desktop for an optimal experience. The choice of
transparent film product depends on the type of printer. In our case,
we used a low-cost inkjet printer and compatible transparent paper
for creating the prototype. After that, the transparent AR markers
are ready to be cut.

3.5 Gameplay
The gameplay of Stackable Music involves stacking a subset of
provided transparent markers to create a musical composition, em-
ploying pre-recorded music clips. The primary objective of the game
Figure 7: The audio player and animation will only be trig- is to acquaint players with the distinct features of each music clip
gered when the central square is occupied. and facilitate the generation of harmonious compositions. When
the Stop-and-Play marker is included in the stack which is captured
by the camera, the corresponding music clip will be played and
an open-source computer vision and machine learning software looped. The order in which the markers are stacked does not affect
library, in Unity. OpenCV for Unity was used to detect custom ArUco the outcome. Additionally, if multiple stacked decks of AR markers
code. We implemented audio, 3D model, and particle effects. Once a are observed by the camera, they will be played simultaneously.
marker is detected, the corresponding audio, model, and animation Stackable Music is a solvable game, because the number of mark-
are presented. The concept of decoding a combined marker is shown ers is countable, resulting in finite potential marker combinations.
Stackable Music CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada

Figure 10: An example system setup with printed transparent markers and a desktop computer with a top-down camera.

While there isn’t a traditional win-or-lose condition, players will OpenCV demonstrates reasonable performance in this regard. How-
eventually realize they have solved the game by iterating through ever, the accuracy of detection also relies on the design of the mark-
all possible combinations. ers themselves. Traditionally, researchers have computed and tested
To add replayability and uncertainty, Stackable Music is recom- various patterns to determine the optimal set for fiducial marker
mended to be played by multiple players (2-4) in a group. Players systems. However, in the context of supporting stacking mechanics,
can share the music they create with others and seek feedback. custom AR markers are created that may not be fully optimized
for accuracy. To address this, future efforts can be directed toward
4 DISCUSSION, LIMITATIONS, AND FUTURE simulating and predicting the quality of markers within the custom
WORK marker design system. Such simulations would allow designers to
determine the expected accuracy of their marker designs.
Through the design of the technology and prototyping the experi-
Another significant challenge is lighting. While ample lighting
ence of the game Stackable Music, we explore what new expressive
is preferred for camera detection, the transparent film used often
potentials are enabled by this technology, and also identify some
has reflective properties. This becomes problematic for detection
potential future directions for research. The discussion is fueled by
when the light source is either point-like or directional, as it can
reflecting on the transparent AR marker system through the design
create unwanted reflection circles. One potential avenue for future
process, rather than a conducted user study.
exploration involves experimenting with alternative transparent
materials that minimize or eliminate these reflection issues. Addi-
4.1 Low-cost "Do it yourself" (DIY) and tangible tionally, integrating the system with controlled and even lighting
AR sources, such as light boxes, could offer a promising direction for
This work explored the novel stacking mechanics of AR markers mitigating the challenges posed by lighting conditions.
using affordable materials - transparent film sheets. Being tangible
and touchable are important features for an AR starter DIY project. 4.3 Customization and creative coding
The unique interaction of stacking markers is fun and intuitive, An interesting direction is to integrate creative coding functions
which also leads the creators to think about the design affordances with the AR marker so that the music piece or art can be customized.
and limitations of objects. In particular, the marker design aspect To support the creative coding aspect, an implementation of a code-
explicitly requires the designers to think about the size of the marker sharing server could be highly beneficial for the community. This
and the restrictions of marker interference. server would provide a platform for artists to share their code snip-
The prototype requires a certain amount of effort in crafting and pets, techniques, and innovative ideas. This exchange of knowledge
hands-on practice. Aligning the markers when stacking requires and expertise would not only enhance the quality of individual
the edge of each marker to be precisely cut. Currently, the stacked creations but also contribute to the evolution and growth of the
markers need to be precisely aligned, with an acceptable error entire Stackable Music ecosystem.
tolerance of 1-2 mm.
Additionally, the transparent film itself is a great medium for 4.4 Social and community building
artistic expression. The blank space outside of the marker content The gameplay in Stackable Music is similar to a trading card game.
can be designed with decoration or with meaningful information Trading card games encompass a card-based gameplay format that
like themes and marker annotations. Besides graphics, one can combines strategic deck-building elements with card trading me-
think of all kinds of physical shapes, for example, triangles, circles, chanics among players, both during and outside of gameplay. This
or other metaphors that work well with AR content. type of game emphasizes social interaction as a significant aspect
of the overall gaming experience, alongside the card battles them-
4.2 Occlusion, lighting, and reflection selves. AR implementation typically involves showcasing 3D mod-
Occlusion poses a significant challenge for all fiducial marker sys- els, animations, and audio effects to enhance the visual appeal and
tems, as it raises questions about their resilience in detecting partially- immersive nature of the user experience. In addition, our research
covered markers. Currently, the customized ArUco dictionary with takes a distinct approach by placing an emphasis on facilitating
CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada Max Chen, Shano Liang, and Gillian Smith

tangible interactions and social engagement between players, ad- REFERENCES

dressing different aspects of the AR game experience. In future [1] Ronald T. Azuma. 1997. A Survey of Augmented Reality. Presence: Teleoperators
studies, a user study and research in the wild can help us gain and Virtual Environments 6, 4 (Aug. 1997), 355–385. https://doi.org/10.1162/pres.
1997.6.4.355
valuable insights into the design elements and social aspects that [2] Tom Bartindale and Chris Harrison. 2009. Stacks on the surface: resolving physical
hold the highest significance for players. order using fiducial markers with structured transparency. In Proceedings of the
ACM International Conference on Interactive Tabletops and Surfaces. ACM, Banff
Alberta Canada, 57–60. https://doi.org/10.1145/1731903.1731916
4.5 Sharing physical product [3] Santiago Bedoya-Rodriguez, Cristian Gomez-Urbano, Alvaro Uribe-Quevedoy,
and Christian Quintero. 2014. Augmented reality RPG card-based game. In
The introduction of stacked markers in the gameplay of Stackable 2014 IEEE Games Media Entertainment. IEEE, Toronto, ON, Canada, 1–4. https:
Music has brought about a unique advantage in terms of selection //doi.org/10.1109/GEM.2014.7118433
and arrangement. Unlike plain paper markers, where it can be [4] Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. 2008. Tangible augmented
reality. In ACM SIGGRAPH ASIA 2008 courses on - SIGGRAPH Asia ’08. ACM Press,
challenging to convey the distance between markers and the overall Singapore, 1–10. https://doi.org/10.1145/1508044.1508051
layout, stacked markers provide a clear visual representation of the [5] G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools
music composition. This makes it easier for players to share their (2000).
[6] Sandra Camara, Valentina Nisi, and Teresa Romão. 2021. Enhancing Children
creations and communicate their intended arrangements to others. Spatial Skills with Augmented Reality Serious Games. In Extended Abstracts
To enhance the accessibility and portability of the stacked mark- of the 2021 Annual Symposium on Computer-Human Interaction in Play (CHI
PLAY ’21). Association for Computing Machinery, New York, NY, USA, 94–100.
ers, an intriguing option for storage and transport would be the https://doi.org/10.1145/3450337.3483470
use of an acrylic photo frame keychain. This compact and versatile [7] Gabriel De Ioannes Becker and Eva Hornecker. 2021. Sally&Molly: A Children’s
accessory could serve as an ideal container for the markers. The Book with Real-Time Multiplayer Mobile Augmented Reality. In Extended Ab-
stracts of the 2021 Annual Symposium on Computer-Human Interaction in Play
acrylic frame would provide protection from wear and tear, ensur- (CHI PLAY ’21). Association for Computing Machinery, New York, NY, USA,
ing that the markers remain intact and legible. Additionally, the 80–86. https://doi.org/10.1145/3450337.3483498
transparency of the frame would allow users to easily view and [8] Adam Drogemuller, James Walsh, Ross T. Smith, Matt Adcock, and Bruce H
Thomas. 2021. Turning everyday objects into passive tangible controllers. In
identify the stacked markers without needing to remove them from Proceedings of the Fifteenth International Conference on Tangible, Embedded, and
the keychain. Embodied Interaction (TEI ’21). Association for Computing Machinery, New York,
NY, USA, 1–4. https://doi.org/10.1145/3430524.3442460
The keychain aspect of the storage solution adds a practical [9] Mark Fiala. 2004. Artag, an improved marker system based on artoolkit. National
element to the design. By attaching the acrylic photo frame to a Research Council Canada, Publication Number: NRC 47419 (2004), 2004.
keychain, players can conveniently carry their stacked markers [10] Mark Fiala. 2010. Designing Highly Reliable Fiducial Markers. IEEE Transactions
on Pattern Analysis and Machine Intelligence 32, 7 (July 2010), 1317–1324. https://
with them wherever they go. This promotes spontaneous music doi.org/10.1109/TPAMI.2009.146 Conference Name: IEEE Transactions on Pattern
creation and sharing opportunities, as players can readily showcase Analysis and Machine Intelligence.
their compositions and engage in collaborative experiences with [11] Anna Fuste and Chris Schmandt. 2019. HyperCubes: A Playful Introduction
to Computational Thinking in Augmented Reality. In Extended Abstracts of the
others. Annual Symposium on Computer-Human Interaction in Play Companion Extended
Furthermore, the acrylic photo frame keychain offers an aes- Abstracts (CHI PLAY ’19 Extended Abstracts). Association for Computing Machin-
ery, New York, NY, USA, 379–387. https://doi.org/10.1145/3341215.3356264
thetic appeal. It allows players to showcase their creativity and [12] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez.
personalized markers, turning them into portable works of art. The 2014. Automatic generation and detection of highly reliable fiducial markers
transparent nature of the frame also provides a visually pleasing under occlusion. Pattern Recognition 47, 6 (June 2014), 2280–2292. https://doi.
org/10.1016/j.patcog.2014.01.005
way to display the stacked markers when not in use. [13] Terrell Glenn, Ananya Ipsita, Caleb Carithers, Kylie Peppler, and Karthik Ramani.
2020. StoryMakAR: Bringing Stories to Life With An Augmented Reality &
5 CONCLUSION Physical Prototyping Toolkit for Youth. In Proceedings of the 2020 CHI Conference
on Human Factors in Computing Systems (CHI ’20). Association for Computing
Marker-based AR offers a unique and engaging experience by in- Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376790
corporating the tangible manipulation of trackable objects as an [14] Peter Gyory. 2022. Creating Platforms to Support Craft and Creativity in Game
Controller Design. In Creativity and Cognition (C&C ’22). Association for
input to interactive systems. The interaction between and within Computing Machinery, New York, NY, USA, 708–710. https://doi.org/10.1145/
markers adds an extra layer of fun to tangible interactions. In this 3527927.3533733
[15] Peter Gyory, Perry Owens, Matthew Bethancourt, Amy Banic, Clement Zheng,
paper, we presented the design of transparent AR markers and and Ellen Yi-Luen Do. 2022. Build Your Own Arcade Machine with Tinycade. In
demonstrated their integration into a captivating card game called Creativity and Cognition (C&C ’22). Association for Computing Machinery,
Stackable Music. Additionally, we examined the potential that stack- New York, NY, USA, 312–322. https://doi.org/10.1145/3527927.3533023
[16] Robert J.K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn,
ing transparent markers brings and highlighted certain limitations. Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based
Based on our findings, we put forth future directions in the in- interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI
teractive and creative domains that can benefit from the use of Conference on Human Factors in Computing Systems (CHI ’08). Association for
Computing Machinery, New York, NY, USA, 201–210. https://doi.org/10.1145/
transparent AR markers. By further exploring these possibilities, 1357054.1357089
we can unlock new realms of interactive experiences and foster [17] Carmen M. Juan, Giacomo Toffetti, Francisco Abad, and Juan Cano. 2010. Tangible
Cubes Used as the User Interface in an Augmented Reality Game for Edutainment.
innovation in various fields. In 2010 10th IEEE International Conference on Advanced Learning Technologies.
IEEE, Sousse, Tunisia, 599–603. https://doi.org/10.1109/ICALT.2010.170 ISSN:
ACKNOWLEDGMENTS 2161-377X.
[18] Martin Kaltenbrunner and Ross Bencina. 2007. reacTIVision: A Computer-Vision
We thank Karen Royer and Albert Ruixiang Han for contributing Framework for Table-Based Tangible Interaction. In Proceedings of the 1st Interna-
to the brainstorming phase of this research, as well as Shawn Gilles tional Conference on Tangible and Embedded Interaction (Baton Rouge, Louisiana)
(TEI ’07). Association for Computing Machinery, New York, NY, USA, 69–74.
for proofreading the manuscript. We also thank the reviewers for https://doi.org/10.1145/1226969.1226983
their insightful comments.
Stackable Music CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada

[19] Hirokazu Kato and Mark Billinghurst. 1999. Marker tracking and hmd calibration Gwenn Englebienne, Hamid Aghajan, and Ben J. A. Kröse (Eds.). Springer, Berlin,
for a video-based augmented reality conferencing system. In Proceedings 2nd Heidelberg, 195–199. https://doi.org/10.1007/978-3-642-25167-2_24
IEEE and ACM International Workshop on Augmented Reality (IWAR’99). IEEE, [30] Ihsan Rabbi and Sehat Ullah. 2013. A Survey on Augmented Reality Challenges
San Francisco, CA, USA, 85–94. https://doi.org/10.1109/IWAR.1999.803809 and Tracking. Acta graphica : znanstveni časopis za tiskarstvo i grafičke komu-
[20] Annie Kelly, R. Benjamin Shapiro, Jonathan de Halleux, and Thomas Ball. 2018. nikacije 24, 1-2 (Feb. 2013), 29–46. https://hrcak.srce.hr/102430 Publisher: Acta
ARcadia: A Rapid Prototyping Platform for Real-time Tangible Interfaces. In Graphica d.o.o..
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems [31] Cassandra Scheirer and Chris Harrison. 2022. DynaTags: Low-Cost Fiducial
(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–8. https: Marker Mechanisms. In INTERNATIONAL CONFERENCE ON MULTIMODAL
//doi.org/10.1145/3173574.3173983 INTERACTION. ACM, Bengaluru India, 432–443. https://doi.org/10.1145/3536221.
[21] Maren Klimm, Dominik Walczak, and Daniel Ayen. 2019. JumpAR – Augmented 3556591
Reality Platformer. In Extended Abstracts of the Annual Symposium on Computer- [32] Anurag Kumar Singh and Jayesh S. Pillai. 2022. Exploration of inter-marker
Human Interaction in Play Companion Extended Abstracts (CHI PLAY ’19 Extended interactions in Tangible AR. In Proceedings of the 28th ACM Symposium on Vir-
Abstracts). Association for Computing Machinery, New York, NY, USA, 261–266. tual Reality Software and Technology (VRST ’22). Association for Computing
https://doi.org/10.1145/3341215.3358249 Machinery, New York, NY, USA, 1–2. https://doi.org/10.1145/3562939.3565642
[22] Albert H. T. Lam, Kevin C. H. Chow, Edward H. H. Yau, and Michael R. Lyu. 2006. [33] Joao Marcelo Xavier Natario Teixeira, Alexandre de Queiroz Burle, Arthur de
ART: augmented reality table for interactive trading card game. In Proceedings Andrade Almeida, Thiago Buarque de Gusmao Lafayette, Vinicius Lima Ventura,
of the 2006 ACM international conference on Virtual reality continuum and its Vitor Mendes Carvalho, and Veronica Teichrieb. 2022. Towards Transparent
applications (VRCIA ’06). Association for Computing Machinery, New York, NY, Markers. In Symposium on Virtual and Augmented Reality (SVR’21). Association
USA, 357–360. https://doi.org/10.1145/1128923.1128987 for Computing Machinery, New York, NY, USA, 74–83. https://doi.org/10.1145/
[23] Meng Chun Lam, Hwei Kei Tee, Siti Soleha Muhammad Nizam, Nurhazarifah Che 3488162.3488212
Hashim, Nur Asylah Suwadi, Siok Yee Tan, Nazatul Aini Abd Majid, Haslina [34] Yoshiki Yokota, Daiki Fujikura, Yoshito Okada, Kazunori Ohno, Kenjiro Tadakuma,
Arshad, and Sook Yee Liew. 2020. Interactive Augmented Reality with Natural and Satoshi Tadokoro. 2022. HueCode2: An Illumination-Robust Meta-Marker
Action for Chemistry Experiment Learning. TEM Journal 9, 1 (Feb. 2020), 351–360. Overlaying Multiple Fiducial Markers using Optimal Color Scheme. In 2022 IEEE
https://doi.org/10.18421/TEM91-48 Num Pages: 351-360 Place: Novi Pazar, Serbia 18th International Conference on Automation Science and Engineering (CASE). IEEE,
Publisher: UIKTEN - Association for Information Communication Technology Mexico City, Mexico, 583–588. https://doi.org/10.1109/CASE49997.2022.9926583
Education and Science. ISSN: 2161-8089.
[24] Jingya Li, Erik D. van der Spek, Jun Hu, and Loe Feijs. 2019. Turning Your [35] Fabio Zambetta, William Raffe, Marco Tamassia, Florian ’Floyd‚ Mueller, Xi-
Book into a Game: Improving Motivation through Tangible Interaction and aodong Li, Niels Quinten, Rakesh Patibanda, Daniel Dang, and Jon Satterley. 2020.
Diegetic Feedback in an AR Mathematics Game for Children. In Proceedings Reducing Perceived Waiting Time in Theme Park Queues via an Augmented
of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY Reality Game. ACM Transactions on Computer-Human Interaction 27, 1 (Jan 2020),
’19). Association for Computing Machinery, New York, NY, USA, 73–85. https: 3:1–3:30. https://doi.org/10.1145/3361524
//doi.org/10.1145/3311350.3347174 [36] Xuan Zhang, Jonathan Lundgren, Yoya Mesaki, Yuichi Hiroi, and Yuta Itoh.
[25] Pedro Pablo Lucas. 2019. AuSynthAR: A simple low-cost modular synthesizer 2020. Stencil Marker: Designing Partially Transparent Markers for Stacking
based on Augmented Reality. Zenodo, Porto Alegre, Brazil. https://doi.org/10. Augmented Reality Objects. In 2020 IEEE International Symposium on Mixed
5281/ZENODO.3673011 and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, Recife, Brazil, 255–257.
[26] Kyzyl Monteiro, Ritik Vatsal, Neil Chulpongsatorn, Aman Parnami, and Ryo https://doi.org/10.1109/ISMAR-Adjunct51615.2020.00073
Suzuki. 2023. Teachable Reality: Prototyping Tangible Augmented Reality with [37] Clement Zheng, Peter Gyory, and Ellen Yi-Luen Do. 2020. Tangible Interfaces
Everyday Objects by Leveraging Interactive Machine Teaching. In Proceedings with Printed Paper Markers. In Proceedings of the 2020 ACM Designing Interactive
of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Systems Conference (DIS ’20). Association for Computing Machinery, New York,
Association for Computing Machinery, New York, NY, USA, 1–15. https://doi. NY, USA, 909–923. https://doi.org/10.1145/3357236.3395578
org/10.1145/3544548.3581449 [38] Zhiying Zhou, Adrian David Cheok, JiunHorng Pan, and Yu Li. 2004. Magic Story
[27] Leonid Naimark and Eric Foxlin. 2002. Circular data matrix fiducial system and Cube: an interactive tangible interface for storytelling. In Proceedings of the 2004
robust image processing for a wearable vision-inertial self-tracker. In Proceedings. ACM SIGCHI International Conference on Advances in computer entertainment
International Symposium on Mixed and Augmented Reality. IEEE, Darmstadt, technology (ACE ’04). Association for Computing Machinery, New York, NY, USA,
Germany, 27–36. https://doi.org/10.1109/ISMAR.2002.1115065 364–365. https://doi.org/10.1145/1067343.1067404
[28] Yoshito Okada, Daiki Fujikura, Yu Ozawa, Kenjiro Tadakuma, Kazunori Ohno, [39] Fabio Zünd, Mattia Ryffel, Stéphane Magnenat, Alessia Marra, Maurizio Nitti,
and Satoshi Tadokoro. 2021. HueCode: A Meta-marker Exposing Relative Pose Mubbasir Kapadia, Gioacchino Noris, Kenny Mitchell, Markus Gross, and
and Additional Information in Different Colored Layers. In 2021 IEEE International Robert W. Sumner. 2015. Augmented creativity: bridging the real and virtual
Conference on Robotics and Automation (ICRA). IEEE, Xi’an, China, 5928–5934. worlds to enhance creative play. In SIGGRAPH Asia 2015 Mobile Graphics and
https://doi.org/10.1109/ICRA48506.2021.9561783 ISSN: 2577-087X. Interactive Applications (SA ’15). Association for Computing Machinery, New
[29] Roshan Lalintha Peiris, Owen Noel Newton Fernando, and Adrian David Cheok. York, NY, USA, 1–7. https://doi.org/10.1145/2818427.2818460
2011. A Dynamic AR Marker for a Paper Based Temperature Sensor. In Ambient
Intelligence (Lecture Notes in Computer Science), David V. Keyson, Mary Lou Received 2023-06-22; accepted 2023-08-03
Maher, Norbert Streitz, Adrian Cheok, Juan Carlos Augusto, Reiner Wichert,