Mixed Reality Installation Research Articles

Design and User Experience of a Hybrid Mixed Reality Installation that Promotes Tinian Marble Crafts Heritage

Hybrid physical-digital installations in museums are interactive systems or exhibits that seamlessly combine physical (tangible) artifacts with virtual environments. In the museum, hybrid installations offer direct, hands-on experiences to visitors and thus may enhance their interest and engagement. Moreover, the entanglement of tangible and virtual interfaces may reinforce learning about the respective heritage. This article reports on the design, implementation, and evaluation of an interactive installation that promotes the heritage of Tinian marble crafts that is included in the representative list of intangible heritage by UNESCO and resides in the Museum of Marble Crafts in Tinos, Greece. The installation puts the museum visitor in the role of a crane operator in a virtual reconstruction of a historic quarry and requires them to operate the crane and move marble volumes with safety in cooperation with non-playable characters (NPCs). Our design approach aimed to engage visitors in a playful learning experience, incorporating aspects of the two museum's exhibits (crane, quarry). The empirical evaluation took place at the lab and the museum with promising results on UX and learning. In addition, we identify and discuss several prospects and drawbacks of designing playful tangible interfaces for other practitioners to consider in similar applications.

Entering hyper-reality: “Resonance-In-Sight,” a mixed-reality art installation

Enhanced virtual communicative technologies extend human perception in extraordinary ways that junction in detaching or augmenting its operator from physical reality. Prospects of the metaverse are progressing towards a possibly fully immersive cyberspace in which users will be entirely disconnected from their analogue physical surroundings. As we advance towards such an alternative reality, the practice-based research project discussed in this paper, titled “Resonance-In-Sight,” foresees and demonstrates the near-future of the metaverse to be a blended hyper-reality in which fiction and reality seamlessly blend into a “Mixed Reality” (MR)-based, co-urban design environment. As digital content in such an environment creatively coexists with our physical cityscape, the research question posed by this project is how such media environment can be used as a proactive cultural educational tool to engage with and inform user audiences of otherwise challenging art-historical content. Through this demonstrator project, the article discusses and reflects on applied methods for user experience (UX) design and the development of MR mobile applications for effective mixed-reality installations. It further reports on employed “Simultaneous Localisation And Mapping” (SLAM) techniques, which are essential for merging co-virtual design content in a hybrid metaverse, and concludes with an assessment of the project’s community impact from an analysis of available user data.

EShadow+: Mixed Reality Storytelling Inspired by Traditional Shadow Theatre

eShadow is a digital storytelling platform inspired by traditional Shadow Theatre. It enables the creation of digital stories within a project-based approach that may start from scenario development and include the creation of digital puppets and sceneries, the set-up and recording of story scenes and the final assembly of a digital story. This paper presents details about eShadow architecture, features and how its use has been enhanced, employing the PerFECt framework, to combine it with immersive technologies such as projection mapping, to create mixed reality installations that offer rich learning experiences to participants in informal learning settings. Two installations are described and compared. The evaluation results demonstrate the effectiveness of the approach towards engaging learning experiences that combine mixed reality approaches with arts such as black light theatre.

Political Engagement in Carne y Arena by Alejandro González Iñárritu

Created by Mexican awarded director Alejandro González Iñárritu, Carne y Arena is an immersive mixed-reality installation that allows visitors to experience traumatic and violent incidents with illegal immigrants crossing the Mexican–US border. Carne y Arena’s mixed reality combines VR experience with physical components, turning it into a multisensory, bodily immersive experience. As part of the art installation, the whole VR arena is surrounded by the remains of a wall’s border; while inside, actual immigrants’ clothes and objects are also exhibited. Another component is the documentary aspect, where real-life characters recount their stories through video testimonies. Iñárritu immerses and makes the visitors experience refugees’ stories first-hand while exploring their emotional reactions to traumatic realities through a spiral of corporeal sensations and entertainment spectacle. According to Iñárritu, the intent is to subordinate technology to the human condition. Technology does mean nothing unless it can reveal or denounce people’s situations. Therefore, technology must be subordinated to humans, humanity, and art. “I despise technology,” says the filmmaker. But, has film lost the power to engage the viewers emotionally? Can virtual reality simulate refugees’ dispossession (the sense of the self) and alleviate society’s consciousness? In this paper, I examine the role of a museum installation featuring refugees’ discourses; the VR technology in bringing forward the visitor’s social engagement; and the issues the filmmaker address, such as the refugee’s experience in contemporary global society.

Mixed-Reality Demonstration and Training of Glassblowing

Traditional crafts exhibit tangible and intangible dimensions. Intangible dimensions include the practitioner’s gestural know-how in craft practice and have received smaller attention than tangible dimensions in digitization projects. This work presents the process of representation and presentation of the glasswork and is exemplified in the re-creation of a historical object. Following an articulated pipeline approach for data collection, annotation, the crafting process is represented visually and semantically in a way that can be meaningfully presented and utilized in craft training and preservation. The outcomes of the proposed approach were used to implement a Mixed Reality installation. The installation targets craft presentation through an exploration of the workspace, as well as craft training through an interactive experience where users re-enact gestures of a glass master holding a tool and receiving audiovisual feedback on the accuracy of their performance. Preliminary evaluation results show high acceptance of the installation and increased user interest.

Kurika - uma instalação interactiva de realidade mista e arte em realidade virtual.

“Kurika” presents itself as a research project / tribute to the literature work “Cão como Nós” by Manuel Alegre. It was publicly presented as an artwork at “Escritaria2019” both as an interactive mixed reality and Virtual Reality installation. In it, a three-dimensional dog made up of words (all of the Alphabet) responds to voice commands (barking if its name “Kurika” is said) and to movements of the audience, triggering sounds and moving images of places of experiences and affections. The main elements of a continuous nonlinear narrative triggered by the interaction. Therefore, the present paper speaks about the development of this project and argues the role of a luminous plane where virtual and real words relate to memory.

Co-Design of a Playful Mixed Reality Installation: An Interactive Crane in the Museum of Marble Crafts

Interactive installations in museums usually adopt hybrid technologies that combine physical elements with digital content, and studies so far show that this approach enhances the interest and engagement of visitors compared to non-interactive media or purely digital environments. However, the design of such systems is complicated, as it involves a large number of stakeholders and specialists. Additionally, the functional components need to be carefully orchestrated to deliver a rich user experience. Thus, there is a need for further research on tools and methods that facilitate the process. In this paper we present the design and development of a mixed reality installation for the Museum of Marble Crafts in Tinos island in Greece, which places visitors in the role of the crane operator and they have to complete challenges in a gamified version of the old quarry. The system lets users operate a tangible controller and their actions are executed by digital workers in a rich 3D environment. Our design approach involved iterative prototyping, research and co-design activities. The creative process has been supported by a series of organized workshops. The evaluation results indicate that mixed reality can be a promising medium for rich interactive experiences in museums that combine tangible and intangible heritage.

Body RemiXer: Extending Bodies to Stimulate Social Connection in an Immersive Installation

Body RemiXer is a mixed reality installation that connects immersants across the virtual/actual divide through emergent tactility and abstract embodiment. Using a virtual reality headset, Kinect and projections, the installation explores the potential of immersive technology to create copresent experiences that foster intercorporeal relationships between immersants wearing a headset and those using the projections. Immersants’ bodies are at the center of the installation, activated as a site for social exchange. Body RemiXer has been exhibited at an art festival and at several smaller events. The authors’ observations during these exhibits reveal Body RemiXer's capacity to disrupt social norms and stimulate new connections.

Virtually Present, Physically Invisible: Alejandro G. Iñárritu’s Mixed Reality InstallationCarne y Arena

This article proposes a media and philosophical analysis of Carne y Arena (2017), an immersive mixed reality installation by Mexican director Alejandro G. Iñárritu. Drawing from Chouliaraki’s notion of the “distant sufferer,” the article compares the medium specificity of television with the characteristics of mixed reality to question whether the gap between spectator and migrant can be bridged. The article analyzes how Carne y Arena positions its users not as spectators but as visitors or participants, thereby turning the “distant” sufferer into a “close and proximate” sufferer. To better understand how this immediacy effect is realized, the article introduces the concept of “intuition,” as theorized by the French philosopher Henri Bergson (1859–1941). In its closing section, the article discusses the ways in which Iñárritu’s work is part of a political intervention that proposes to challenge the limits of mediation and to promote social justice through feeling, acting, and knowing otherwise.

Within an Ocean of Light: Creating Volumetric Lightscapes

This paper documents explorations into an alternative platform for immersive and affective expression within spatial mixed reality installation experiences. It discusses and analyzes experiments that use an advanced LED cube to create immersive, interactive installations and environments where visitors and visuals share a common physical space. As a visual medium, the LED cube has very specific properties and affordances, and optimizing the potential for such systems to create meaningful experiences presents many interlinked challenges. Two artworks exploring these possibilities are discussed. Both have been exhibited internationally in a variety of settings. Together with this paper, the works shed some light on the design considerations and experiential possibilities afforded by LED cubes and arrays. They also suggest that LED grids have potential as an emerging medium for immersive volumetric visualizations that occupy physical space.

Adrift in the virtuality continuum

This article explores the induction of presence upon individuals through illusion in mixed-reality installation environments, using the author's work “Dislocation” [2005] as a case study. The article reports on the design and production of “Dislocation,” concentrating on the development of techniques for creating illusion. These approaches can be used to produce compelling works that are not only technically and aesthetically sophisticated, but also provide a sustained engaging experience for the user. It reflects on the effectiveness of these techniques from observations of audience interaction in gallery settings and examines future possibilities and developments.

Designing performativity for mixed reality installations

This article takes up the concept of performativity prevalent in the humanities and applies it to the design of installation arts in mixed reality mode. Based on the design, development and public access to two specific works, the concept is related to a form of research by design. We argue that the concept of performativity may be further usefully employed in investigations (design and research, artistic and public) into digital arts where complex intersections between concepts, technologies, dramaturgy, media and participant actions are in flux and together constitute the emergence and experience of a work. Theories of performativity are related to these two works in an argument that further suggests there is room in research by design to also include ‘performative design’. The article is the result of a wide-ranging interdisciplinary collaboration and aims to convey some sense of that in its reporting style, content and analysis.

The Effects of Explicit and Implicit Interaction on User Experiences in a Mixed Reality Installation: The Synthetic Oracle

Virtual and mixed reality environments (VMRE) often imply full-body human-computer interaction scenarios. We used a public multimodal mixed reality installation, the Synthetic Oracle, and a between-groups design to study the effects of implicit (e.g., passively walking) or explicit (e.g., pointing) interaction modes on the users' emotional and engagement experiences, and we assessed it using questionnaires. Additionally, real-time arm motion data was used to categorize the user behavior and to provide interaction possibilities for the explicit interaction group. The results show that the online behavior classification corresponded well to the users' interaction mode. In addition, contrary to the explicit interaction, the engagement ratings from implicit users were positively correlated with a valence but were uncorrelated with arousal ratings. Interestingly, arousal levels were correlated with different behaviors displayed by the visitors depending on the interaction mode. Hence, this study confirms that the activity level and behavior of users modulates their experience, and that in turn, the interaction mode modulates their behavior. Thus, these results show the importance of the selected interaction mode when designing users' experiences in VMRE.

Disquiet in the plasma

The metaphor of plasma is taken up to present and discuss movement and engagement by participants in mixed reality installation arts. Two works involving full body video portraits exhibited through large plasma screens in a variety of public settings are covered. Machinic mediations of video realism are considered in terms of embodied interaction in which viewer-participants contribute to the ‘disquiet’ of gendered figuring. Processural, proximal and personal aspects of responsive engagement are discussed. This is extended to performativity that may lead us to critical reflection of our own actions and responses in mixed reality arts.

Media Beat: uncharted pages from a voyage of the beagle

Uncharted pages from a voyage of the beagle is a mixed-reality installation created for a new media symposium with the theme of ecologies of self-replicating systems. The work uses both the virtual domain of Second Life and a computer-mediated interactive installation to create a hybrid space. At the center of this work is an investigation into self-replicating and self-reproducing objects and systems. The process of creating this work was directed by my practice as an artist, where metaphor and poetic license enable me to investigate ways of looking at associations between philosophical and poetic concerns relating to memory, narrative, self, and the biological and physical sciences, using - as is so often the case in my own practice - the contrivance of narrative to do so.