Multisensory Augmented Reality Research Articles

Comparison of visual and multisensory augmented reality for precise manual manipulation tasks

Precise manual manipulation is an important skill in daily life, and Augmented Reality (AR) is increasingly being used to support such operations. This article reports on a study investigating the usability of visual and multisensory AR for precise manual manipulation tasks, in particular the representation of detailed deviations from the target pose. Two AR instruction interfaces were developed: the visual deviation instruction and the multisensory deviation instruction. Both interfaces used visual cues to indicate the required directions for manipulation. The difference was that the visual deviation instruction used text and color mapping to represent deviations, whereas the multisensory deviation instruction used sonification and vibration to represent deviations. A user study was conducted with 16 participants to compare the two interfaces. The results found a significant difference only in speed, without significant differences in accuracy, perceived ease-of-use, workload, or custom user experience elements. Multisensory deviation cues can speed up precise manual manipulation compared to visual deviation cues, but inappropriate sonification and vibration strategies can negatively affect users’ subjective experience, offsetting the benefits of multisensory AR. Based on the results, several recommendations were provided for designing AR instruction interfaces to support precise manual manipulation.

Boosting heritage recommendations through awe and augmented reality

PurposeDrawing on appraisal theories, this paper aims to attempt to demonstrate how augmented reality (AR) strengthens relationships in the cognitive-affective-conative mechanism to shape visitors' recommendation of awe-inspiring heritage experiences.Design/methodology/approachA mixed-method approach was undertaken. First, an exploratory pre-study (n = 98) analyzed the multidimensionality of awe appraisals, eliciting the most salient dimensions in tourism experiences for informing the main study. Subsequently, an on-site survey was used to examine the relationships in the cognitive-affective-conative sequence, based on 316 questionnaires collected at a highly acclaimed religious heritage equipped with a multisensory AR activity.FindingsTwo distinct appraisals underlined in the awe concept were elicited during the exploratory phase: altered emotions (aggregating positive self-transformative dimensions) and the need for accommodation. The survey results demonstrated that the influence of authenticity on recommendation is shaped by emotions. Altered emotions hold a central mediating role and are differentiated from other discrete emotions. AR and the need for accommodation are revealed as contingencies that regulate the mediated mechanism.Practical implicationsThis paper provides the hospitality and tourism stakeholders with a better understanding of how immersive technologies can be used for boosting recommendations, by enhancing the emotional responses toward authenticity perceptions.Originality/valueThis study introduces a novel conceptual framework to broaden our understanding of how cognitive and affective mental processes are linked to visitors’ recommendations, and how AR amplifies the outcomes. To the best of the authors’ knowledge, it is the first research to distinguish and document the specific effects of two different types of awe appraisal.

Head-mounted augmented reality to support reassurance and social interaction for autistic children with severe learning disabilities

Augmented Reality (AR) is promising to complement autism approaches, but so far has mainly focused on training socio-emotional abilities for autistic children with mild learning disabilities. To better consider autistic children with severe learning disabilities and complex needs (SLN), stakeholders advise using collaborative AR sensory-based mediation approaches. Magic Bubbles is a multisensory AR environment created based on stakeholders’ interviews, then adapted for a day hospital setting in collaboration with practitioners, and finally validated in terms of acceptability and usability for autistic children with SLN. In this paper, we report on our latest study that explores three main research questions: 1) To what extent can Magic Bubbles secure autistic children with SLN? 2) To what extent can Magic Bubbles prompt the dyadic relationship between an autistic child with SLN and a practitioner? 3) What is the overall quality of experience for autistic children with SLN when using Magic Bubbles? To answer these questions, seven autistic children with SLN participated in at least six weekly sessions over three months in a day hospital setting. Data collection and analysis used qualitative and quantitative methods, mainly drawing upon grounded theory to evaluate their experiences. Findings validate the three research questions, offer a detailed account of children’s experiences with AR, and outline future directions.

Cognitive Computing and Workplace Monitoring Systems, Generative Artificial Intelligence and Contextual Awareness Tools, and Multi-Sensory Augmented Reality and Spatial Data Mining Algorithms in Multi-Organizational Collaborative Environments

Cognitive Computing and Workplace Monitoring Systems, Generative Artificial Intelligence and Contextual Awareness Tools, and Multi-Sensory Augmented Reality and Spatial Data Mining Algorithms in Multi-Organizational Collaborative Environments

Spatial Data Mining and Image Processing Computational Algorithms, Generative Artificial Intelligence and Multi-Sensory Augmented Reality Systems, and Productivity Software and Emotional State Prediction Tools in Virtual Work Settings

Spatial Data Mining and Image Processing Computational Algorithms, Generative Artificial Intelligence and Multi-Sensory Augmented Reality Systems, and Productivity Software and Emotional State Prediction Tools in Virtual Work Settings

Tactile Echoes: Multisensory Augmented Reality for the Hand.

Touch interactions are central to many human activities, but there are few technologies for computationally augmenting free-hand interactions with real environments. Here, we describe Tactile Echoes, a finger-wearable system for augmenting touch interactions with physical objects. This system captures and processes touch-elicited vibrations in real-time in order to enliven tactile experiences. In this article, we process these signals via a parametric signal processing network in order to generate responsive tactile and auditory feedback. Just as acoustic echoes are produced through the delayed replication and modification of sounds, so are Tactile Echoes produced through transformations of vibrotactile inputs in the skin. The echoes also reflect the contact interactions and touched objects involved. A transient tap produces discrete echoes, while a continuous slide yields sustained feedback. We also demonstrate computational and spatial tracking methods that allow these effects to be selectively assigned to different objects or actions. A large variety of distinct multisensory effects can be designed via ten processing parameters. We investigated how Tactile Echoes are perceived in several perceptual experiments using multidimensional scaling methods. This allowed us to deduce low-dimensional, semantically grounded perceptual descriptions. We present several virtual and augmented reality applications of Tactile Echoes. In a user study, we found that these effects made interactions more responsive and engaging. Our findings show how to endow a large variety of touch interactions with expressive multisensory effects.

Development and Evaluation of an Outdoor Multisensory AR System for Cultural Heritage

Enhancing tourist visits to cultural heritage sites by making use of mobile augmented reality has been a tendency in the last few years, presenting mainly audiovisual experiences. However, these explorations using only visuals and sounds, or narratives, do not allow users to be presented with, for example, a particular smell that can be important to feel engaged or to better understand the history of the site. This article pursues the goal of creating an experience that puts the user in a scene planned to evoke several stimuli with SensiMAR prototype - a Multisensory Augmented Reality system that aims to be used in cultural heritage outdoors. When using SensiMAR, the user will be involved with visual reconstructions, surrounded by the soundscape of ancient times, and is exposed to a particular smell very common that time. Given the novelty of this proposal, ascertaining the usability of such a system was raised as a foremost demand. Thus, in addition to its development and implementation specifications, an experimental study was conducted to evaluate the usability of the system in end-users' perspective. The results obtained from random visitors of an archaeological site were analysed according to their sex, age, previous experience with augmented reality technology, and provided condition - audiovisual condition, and multisensory condition, with visual, audio, and smell stimuli. Results were collected from a total of 67 participants and show that this multisensory prototype achieved good usability results across all groups. No statistically differences were found, demonstrating good usability of the SensiMAR system regardless of their sex, age, previous experience with the technology or provided condition.

Raising Awareness about the Consequences of Human Activities on Natural Environments through Multisensory Augmented Reality: Amazon Rainforest and Coral Reef Interactive Experiences

Raising Awareness about the Consequences of Human Activities on Natural Environments through Multisensory Augmented Reality: Amazon Rainforest and Coral Reef Interactive Experiences

The Kandinsky Experience: A Multisensory Augmented Reality Application for Cultural Heritage

The Kandinsky Experience: A Multisensory Augmented Reality Application for Cultural Heritage

Multisensory Augmented Reality in Cultural Heritage: Impact of Different Stimuli on Presence, Enjoyment, Knowledge and Value of the Experience

Little is known about the impact of the addition of each stimulus in multisensory augmented reality experiences in cultural heritage contexts. This paper investigates the impact of different sensory conditions on a user's sense of presence, enjoyment, knowledge about the cultural site, and value of the experience. Five different multisensory conditions, namely, Visual, Visual + Audio, Visual + Smell, and Visual + Audio + Smell conditions, and regular visit referred to as None condition, were evaluated by a total of 60 random visitors distributed across the specified conditions. According to the results, the addition of particular types of stimuli created a different impact on the sense of presence subscale scores, namely, on spatial presence, involvement, and experienced realism, but did not influence the overall presence score. Overall, the results revealed that the addition of stimuli improved enjoyment and knowledge scores and did not affect the value of the experience scores. We concluded that each stimulus has a differential impact on the studied variables, demonstrating that its usage should depend on the goal of the experience: smell should be used to privilege realism and spatial presence, while audio should be adopted when the goal is to elicit involvement.

Touching the Untouchable: Exploring Multi-Sensory Augmented Reality in the Context of Online Retailing

Mental intangibility during product evaluation remains one of the greatest drawbacks for online purchasing. However, emerging multi-sensory Augmented Reality (m-AR) applications offer a potential solution for this online retailing problem. Drawing on active inference theory, this article proposes a conceptual framework to assess how sensory control and feedback modalities affect consumer value judgements by reducing mental intangibility. We show how touch control, compared to voice control, positively affects consumers’ willingness-to-pay. The underlying mechanism is a sequential process of reduced mental intangibility and increased feeling of decision comfort. In addition, we highlight a positive moderating effect of congruent auditory feedback on decision comfort. We also demonstrate a novel consumer boundary condition. Consumers high in assessment orientation experience a stronger reduction in mental intangibility. The results are consistently replicated across three experiments implying theoretical and managerial contributions for m-AR in the context of online retailing.

Augmenting Museum Communication Services to Create Young Audiences

The aim of this paper is to demonstrate how museums use Augmented Reality (AR) to enhance communication services with their audiences and attract new ones. Still, there is no definite answer to how young audiences perceive the educational effects of experiencing this augmented space of communication as an immersive medium. This study is based on a survey of 400 students after they visited an AR technology-enhanced exhibition held by a local history museum. Two stimulus–response marketing scale metrics, widely used to assess TV commercials, were adapted for AR experiences and validated. The mediation analysis revealed an intervening emotional mechanism, in which the multisensory AR experience has educational effects through entertainment and empathy. An improved stimulus–response empirical model is proposed, in which AR technologies, as environmental multisensory stimuli, produce cognitive responses through emotional immersion. The findings have significance in improving how museums encode their message using AR technologies as a secondary communication medium with young audiences. This study could help museum professionals and application developers to find AR implementation solutions as service tools to enhancing user experience by using a widely tested scale for evaluating TV commercials applied to measure AR experiences.

Digilog book for temple bell tolling experience based on interactive augmented reality

We first present the concept of the Digilog Book, an augmented paper book that provides additional multimedia content stimulating readers’ five senses using augmented reality (AR) technologies. We also develop a prototype to show the usefulness and effectiveness of the book. The Digilog book has the following characteristics: AR content descriptions for updatable multisensory AR contents; enhanced experience with multisensory feedback; and interactive experience with computerized vision-based manual input methods. As an example of an entertaining and interactive Digilog Book, this paper presents a “temple bell experience” book and its implementation details. Informal user observation and interviews were conducted to verify the feasibility of the prototype book. As a result, this case study of the Digilog book can be useful in guiding the design and implementation of other Digilog applications, including posters, pictures, newspapers, and sign boards.

Augmented Multisensory Interface Design (AMID): A Human-Centric Approach to Unisensory and Multisensory Augmented Reality Design

This study demonstrates that the use of augmented reality interfaces, designed based on a theoretically driven human-centered framework, can significantly improve task performance. The domain of unmanned ground vehicle landmine/unexploded ordnance detection and disposal was selected because it is a real-world example of a complex human-robot interaction task. Participants in this study were assigned to one of seven experimental conditions. A new method (the augmented multisensory interface design, or AMID, method) was used to identify two unisensory and one multisensory performance-enhancing augmented reality (AR) interface solutions, along with three less appropriate interface solutions. All participants were trained in the operation of the AR devices they were assigned and also in telerobotic landmine detection operations. After completing the training, participants were given the task of clearing a minefield. Results show that participants in the AMID-compliant solution groups significantly outperformed the control group and their non-AMID-compliant group counterparts in terms of the numbers of landmines correctly identified. The AMID-compliant multisensory condition was found to result in significantly increased landmine detection, increased sensitivity, and decreased task time.

Improving Tele-robotic Landmine Detection through Augmented Reality Devices

Tele-robotic landmine/UXO (unexploded ordinance) detection and disposal is a real world example of a complex human robot interaction task. This study demonstrates that the use of augmented reality interfaces, designed based on a theoretically driven framework, can significantly improve performance, and that these same devices can be used as effective training technologies. Participants in this study were assigned to one of four experimental conditions (two uni-sensory and one multisensory augmented reality interface, and a control group). All participants were trained in tele-robotic landmine detection and the groups performed these tasks first with augmentation (session one) and then without (session two/transfer task). The results show that participants in the augmented groups significantly outperformed the control group in terms of the numbers of landmines correctly identified (in both sessions). The multisensory condition was found to result in both significantly increased landmine detection and significantly decreased task time (in both sessions).