Interactive Augmented Reality Research Articles

Effects of User Interface Orientation on Sense of Immersion in Augmented Reality

The sense of immersion provided by overlying computer-generated graphics over actual items is one of augmented reality’s distinguishing features. It is affected by multiple factors such as the user interface orientation in the virtual physical hybrid space. There have been many studies on realistic graphic rendering and multimodal interaction in augmented reality. Few studies, however, have systematically investigated the influence of the user interface orientations, more specifically the user-based orientation (UO) and object-based orientation (OO), on various aspects of immersion. We developed the UO and OO user interface prototypes and evaluated the sense of immersion in terms of engagement, engrossment, and overall immersion. The results showed that the user interface orientations in augmented reality had a significant influence on users’ perceived engagement and engrossment, whereas there was no significant difference in overall immersion. Particularly, compared to the OO user interfaces, the UO user interfaces exhibited higher perceived usability and slower focus of attention switching. Generalising implications for alternative augmented reality are provided.

Beyond Books: Exploring AR in Education with Unity3D

Augmented Reality (AR) is a technology that enables the blending of digital information, such as 3D models, images, or videos, into our real-world environment in real time. Unlike virtual reality, which creates an entirely artificial environment, AR enhances reality by overlaying digital elements. Augmented Reality (AR) is a technological innovation that combines the physical and digital worlds. It overlays contextual and interactive digital information onto the environment we see, creating an immersive experience. AR can revolutionize various fields such as computer science, human-computer interaction, psychology, medicine, and education. Although the concept of AR dates back to the 1960s, recent advancements in sensor technologies, computational power, and mobile devices have made it more accessible. As AR gains popularity, researchers explore its potential in different fields and study the cognitive processes involved in AR interactions. Understanding the fundamental principles of AR systems is key to unlocking its transformative capabilities. As we go further, we will explore the dimensions of augmented reality, including its evolution, current scientific applications, and prospects for scientific inquiry and innovation. Key Words: Augmented Reality (AR), Digital Integration, Immersive Experience, Interdisciplinary Research, Technological Innovation

Tangible and Mid-Air Interactions in Hand-Held Augmented Reality for Upper Limb Rehabilitation: An Evaluation of User Experience and Motor Performance

Hand-held augmented reality (AR) offers accessible, interactive rehabilitation options for patients with upper limb motor deficits. Incorporating hand-involved interactions (e.g., tangible and mid-air interactions) into hand-held AR provides patients with intuitive manners to perform rehabilitation exercises mimicking real-world activities. Previous work has shown the importance of user experience and motor performance in rehabilitation systems, but little was known in the literature regarding the impact of hand-involved interactions in hand-held AR on user experience and motor performance in rehabilitation exercises. Hence, this study aims to evaluate user experience and motor performance when using three types of hand-involved interactions in hand-held AR rehabilitation: (1) tangible cube (i.e., a space-multiplexed tangible interaction with a physical cube acting as a real proxy to manipulate a virtual object in the same form); (2) tangible controller (i.e., a time-multiplexed tangible interaction with a physical controller applied to manipulate a virtual object); and (3) hand motion (i.e., a form of mid-air interaction to move a virtual object with hands). Based on the findings from self-report, electroencephalography (EEG), and performance measures, this study reveals the advantages of the tangible cube over the tangible controller, both superior to the hand motion in hand-held AR rehabilitation regarding user experience and motor performance. This study offers new understanding of the advantages and disadvantages of various interaction techniques in hand-held AR rehabilitation, emphasizing crucial design considerations for these systems, with a focus on user experience and motor performance in upper limb rehabilitation.

A Fast and Interactive Augmented Reality System for PET/CT-guided Intervention of Neuroblastoma.

Neuroblastoma is the most common type of extracranial solid tumor in children and can often result in death if not treated. High-intensity focused ultrasound (HIFU) is a non-invasive technique for treating tissue that is deep within the body. It avoids the use of ionizing radiation, avoiding long-term side-effects of these treatments. The goal of this project was to develop the rendering component of an augmented reality (AR) system with potential applications for image-guided HIFU treatment of neuroblastoma. Our project focuses on taking 3D models of neuroblastoma lesions obtained from PET/CT and displaying them in our AR system in near real-time for use by physicians. We used volume ray casting with raster graphics as our preferred rendering method, as it allows for the real-time editing of our 3D radiologic data. Some unique features of our AR system include intuitive hand gestures and virtual user interfaces that allow the user to interact with the rendered data and process PET/CT images for optimal visualization. We implemented the feature to set a custom transfer function, set custom intensity cutoff points, and region-of-interest extraction via cutting planes. In the future, we hope to incorporate this work as part of a complete system for focused ultrasound treatment by adding ultrasound simulation, visualization, and deformable registration.

Augmented Reality Visualization and Quantification of COVID-19 Infections in the Lungs

The ongoing COVID-19 pandemic has had a significant impact globally, and the understanding of the disease’s clinical features and impacts remains insufficient. An important metric to evaluate the severity of pneumonia in COVID-19 is the CT Involvement Score (CTIS), which is determined by assessing the proportion of infections in the lung field region using computed tomography (CT) images. Interactive augmented reality visualization and quantification of COVID-19 infection from CT allow us to augment the traditional diagnostic techniques and current COVID-19 treatment strategies. Thus, in this paper, we present a system that combines augmented reality (AR) hardware, specifically the Microsoft HoloLens, with deep learning algorithms in a user-oriented pipeline to provide medical staff with an intuitive 3D augmented reality visualization of COVID-19 infections in the lungs. The proposed system includes a graph-based pyramid global context reasoning module to segment COVID-19-infected lung regions, which can then be visualized using the HoloLens AR headset. Through segmentation, we can quantitatively evaluate and intuitively visualize which part of the lung is infected. In addition, by evaluating the infection status in each lobe quantitatively, it is possible to assess the infection severity. We also implemented Spectator View and Sharing a Scene functions into the proposed system, which enable medical staff to present the AR content to a wider audience, e.g., radiologists. By providing a 3D perception of the complexity of COVID-19, the augmented reality visualization generated by the proposed system offers an immersive experience in an interactive and cooperative 3D approach. We expect that this will facilitate a better understanding of CT-guided COVID-19 diagnosis and treatment, as well as improved patient outcomes.

An interactive augmented reality framework to enhance the user experience and operational skills in electronics laboratories

Augmented reality (AR) stands as a widely embraced technology that significantly enhances learning experiences for students. AR offers an instructional approach supported by technological design, thereby fostering enriched learning interactions. This research proposes an interactive AR framework, intended to create an augmented reality learning environment (ARLE) for the specific needs of electronics engineering laboratory hardware operations. The ARLE functions as an active learning system designed using a user-centered methodology. It offers interactive 3D models of laboratory equipment, providing learners with preliminary training in equipment operation. The real-time connection between the laboratory apparatus and the AR environment is established using the Arduino board. This interface empowers users to control the AR simulation through the laboratory equipment seamlessly. An experimental study involving 80 engineering students was conducted to evaluate the impact of AR intervention on user experience, usability, and operational skills. The participants were divided into two groups: the experimental group (N = 40) and the control group (N = 40). The experimental group underwent electronics equipment training using ARLE, while the control group followed instructions from a standard instrument handbook. To assess the usability and user experience of ARLE, the system usability scale (SUS) and the user experience questionnaire (UEQ) were employed (N = 40). The findings revealed an SUS score of 80.9 for ARLE, categorizing it as “good” according to SUS ratings. Additionally, the UEQ results demonstrated significantly favorable scores across the six scales when compared to the benchmark dataset. The study's outcomes demonstrate that AR intervention offers learners significant pedagogical value, resulting in a substantial positive impact on operational skills in electronics laboratories.

Creating Interactive Books with Augmented Reality

This workshop helped educators to create interactive Augmented Reality (AR) books at the Association for Educational Communications and Technology (AECT) conference. The goal of the workshop was to train participants to gain skills in integrating AR into their professional lives. All participants created an AR book or curriculum vitae during the workshop. This manuscript describes the specific activities, assigned times, and educational tools used during the workshop. This workshop was designed for participants with no previous knowledge or skills in developing AR materials. Educators interested in teaching educational technologies, especially AR, could be instructors for this lesson. Preservice teachers could also take this lesson.

Symphonic Ornaments in Parametric Architecture through Music

The research explores a pioneering approach to intertwining music and architecture through the development of reliefs with musical identities, inspired by Goethe's perspective that music and architecture are intrinsically linked. By employing augmented reality (AR) and QR code technology, the initiative enables a synchronized visual and auditory narrative. The methodology encompasses three pivotal sections: Firstly, Generative Design, which leverages artificial intelligence-optimized algorithms to morph numerical musical outputs into 3D geometrical patterns. Secondly, technology application, employing Pure Javascript for QR code scanning and utilizing Asp.Net 6 and Kestrel for streaming audio information. Thirdly, AR Interaction, which utilizes AR technology to offer an immersive visual and auditory experience via mobile devices. This innovative integration of music, architecture, and technology not only pioneers a new experiential journey through form and melody but also delivers an interactive, dynamic user experience.

Visuo-Haptic Rendering of the Hand during 3D Manipulation in Augmented Reality.

Manipulating virtual objects with bare hands is a key interaction in Augmented Reality (AR) applications. However, there are still several limitations that affect the manipulation, including the lack of mutual visual occlusion between virtual and real content as well as the lack of haptic sensations. To address the two abovementioned matters, the role of the visuo-haptic rendering of the hand as sensory feedback is investigated. The first experiment explores the effect of showing the hand of the user as seen by the AR system through an avatar, comparing six visual hand rendering. The second experiment explores the effect of the visuo-haptic hand rendering by comparing two vibrotactile contact techniques provided at four delocalized positions on the hand and combined with the two most representative visual hand renderings from the first experiment. Results show that delocalized vibrotactile haptic hand rendering improved perceived effectiveness, realism, and usefulness when provided close to the contact point. However, the farthest rendering position, i.e., on the contralateral hand, gave the best performance even though it was largely disliked. The visual hand rendering was perceived as less necessary for manipulation when the haptic hand rendering was available, but still provided useful feedback on the hand tracking.

Investigating the Effects of Avatarization and Interaction Techniques on Near-field Mixed Reality Interactions with Physical Components.

Mixed reality (MR) interactions feature users interacting with a combination of virtual and physical components. Inspired by research investigating aspects associated with near-field interactions in augmented and virtual reality (AR & VR), we investigated how avatarization, the physicality of the interacting components, and the interaction technique used to manipulate a virtual object affected performance and perceptions of user experience in a mixed reality fundamentals of laparoscopic peg-transfer task wherein users had to transfer a virtual ring from one peg to another for a number of trials. We employed a 3 (Physicality of pegs) X 3 (Augmented Avatar Representation) X 2 (Interaction Technique) multi-factorial design, manipulating the physicality of the pegs as a between-subjects factor, the type of augmented self-avatar representation, and the type of interaction technique used for object-manipulation as within-subjects factors. Results indicated that users were significantly more accurate when the pegs were virtual rather than physical because of the increased salience of the task-relevant visual information. From an avatar perspective, providing users with a reach envelope-extending representation, though useful, was found to worsen performance, while co-located avatarization significantly improved performance. Choosing an interaction technique to manipulate objects depends on whether accuracy or efficiency is a priority. Finally, the relationship between the avatar representation and interaction technique dictates just how usable mixed reality interactions are deemed to be.

An interactive augmented reality software for facial reconstructive surgeries

Background and ObjectiveSurgical trainees need a lot of training and practice before being able to operate independently. The current approach of surgical education mainly involves didactic teaching and psychomotor training through physical models or real tissue. Due to the unavailability of physical resources and lack of objective ways of evaluation, there is a demand for developing alternative training methods for surgeons. In this paper, we present an application that provides additional training opportunities to surgical trainees in the field of facial reconstructive surgeries. MethodsWe built a mobile augmented reality application that helps the user to visualize important concepts and experiment with different surgical plans for facial reconstructive surgeries. The application can overlay relaxed skin tension lines on a live video input or a patient's photo, which serve as bases for aligning a skin flap. A surgical trainee can interactively compare different skin flap design choices with estimated final scars on a photo of a patient. Data collection capability is also added to the application, and we performed a Monte Carlo experiment with simulated users (five classes of 100 users each) as an example of objectively measuring user performance. ResultsThe application can overlay relaxed skin tension lines on a patient's face in real time on a modern mobile device. Accurate overlays were achieved in over 91% as well as 84% and 88% out of 263 generated face images, depending on the method. Visual comparisons of the three overlay methods are presented on sample faces from different population groups. From the Monte Carlo experiment, we see that user actions in each class follow a normal distribution with a distinct set of parameters. ConclusionsThis application can serve as a basis for teaching surgical trainees the fundamentals of different facial reconstructive procedures, especially concepts related to relaxed skin tension lines and skin flaps. It can objectively evaluate the performance of surgical trainees in a course. This setup focuses on illustrating the relationship between the orientation of skin flaps and relaxed skin tension lines, which is a prerequisite of minimizing scarring in patients in addition to good surgical techniques.

Characterizing information access needs in gaze-adaptive augmented reality interfaces: implications for fast-paced and dynamic usage contexts

ABSTRACT Gaze-adaptive interfaces can enable intuitive hands-free augmented reality (AR) interaction but unintentional selection (i.e. “Midas Touch”) can have serious consequences during high-stakes real-world AR use. In the present study, we assessed how simulated gaze-adaptive AR interfaces, implementing single and dual gaze inputs, influence Soldiers’ human performance and user experience (UX) in a fast-paced virtual reality marksmanship task. In Experiment 1, we investigated 1- and 2-stage dwell-based interfaces, finding confirmatory dual gaze dwell input effectively reduced Midas Touch but also reduced task performance and UX compared to an always-on (AO) interface. In Experiment 2, we investigated gaze depth-based interfaces, finding similar negative impacts of confirmatory dwell on Midas Touch, task performance, and UX. Overall, compared to the AO interface, single gaze input interfaces (e.g. single dwell or gaze depth threshold) reduced viewing of task-irrelevant information and yielded similar task performance and UX despite being prone to Midas Touch. Broadly, our findings demonstrate that AR users performing fast-paced dynamic tasks can tolerate some unintentional activation of AR displays if reliable and rapid information access is maintained and point to the need to develop and refine gaze depth estimation algorithms and novel gaze depth-based interfaces that provide rapid access to AR display content.

Surveying the Social Comfort of Body, Device, and Environment-Based Augmented Reality Interactions in Confined Passenger Spaces Using Mixed Reality Composite Videos

Augmented Reality (AR) headsets could significantly improve the passenger experience, freeing users from the restrictions of physical smartphones, tablets and seatback displays. However, the confined space of public transport and the varying proximity to other passengers may restrict what interaction techniques are deemed socially acceptable for AR users - particularly considering current reliance on mid-air interactions in consumer headsets. We contribute and utilize a novel approach to social acceptability video surveys, employing mixed reality composited videos to present a real user performing interactions across different virtual transport environments. This approach allows for controlled evaluation of perceived social acceptability whilst freeing researchers to present interactions in any simulated context. Our resulting survey (N=131) explores the social comfort of body, device, and environment-based interactions across seven transit seating arrangements. We reflect on the advantages of discreet inputs over mid-air and the unique challenges of face-to-face seating for passenger AR.

Augmented digital human vs. human agents in storytelling marketing: Exploratory electroencephalography and experimental studies

AbstractAs the fourth industrial revolution unfolds and the use of digital humans becomes more commonplace, understanding digital humans' potential to replace real human interaction or enhance it, particularly in storytelling marketing contexts, is becoming evermore important. To promote interaction and increase the entertainment value of technology‐enhanced storytelling marketing, brands have begun to explore the use of augmented digital humans as storytelling agents. In this article, we examine the effectiveness of leveraging advanced technologies and delivering messages via digital humans in storytelling advertisements. In Study 1, we investigate the effectiveness of narrative transportation on behavioral responses after exposure to an interactive augmented reality mobile advertisement with a digital human storyteller. In Study 2, we compare how consumers respond to augmented digital human versus real human storytelling advertisements after conducting an exploratory neurophysiological electroencephalography study. The findings show that both types of agents promote narrative transportation when the story fits the product well. Moreover, a digital human perceived as more human‐like elicits stronger positive consumer responses, suggesting an effective new approach to storytelling marketing.

Effects of a Shoulder-support Exoskeleton on Shoulder Strain during Augmented Reality Interactions

This study determined whether a passive shoulder-support exoskeleton would reduce biomechanical load in the shoulders during augmented reality (AR) interactions. Twenty participants were recruited to a repeated-measures laboratory study to perform AR tasks with and without a shoulder-support exoskeleton. The AR tasks consisted of two standardized tasks (omni-directional pointing and cube placing task). Shoulder muscle activity, postures, discomfort, and task performance (i.e., completion time) were measured during the experimental sessions. The result showed that the shoulder muscle activity (trapezius and deltoid muscles) and subjective shoulder discomfort were significantly lower with the exoskeleton compared to the no-exoskeleton condition (p < 0.05). The shoulder posture data showed that the exoskeleton use increased the 50th percentile of the shoulder flexion (p = 0.02). In contrast, the exoskeleton had limited effects on task performance (p > 0.83). These results suggested that a shoulder-support exoskeleton may have potential to reduce shoulder strain during AR interactions.

Errors in Augmented Reality Interactions Affected Muscular Loads in the Neck and Shoulders

The use of augmented reality (AR) head-mounted displays is becoming increasingly prevalent due to their ability to provide interactive experiences with high degrees of freedom. However, AR interactions have been shown to be associated with increased biomechanical loads in the neck and shoulders. This repeated-measures study evaluated the effects of interaction errors on neck and shoulder biomechanical loads during AR tasks. Twenty participants performed two standardized AR tasks (omni-directional pointing and cube placing) with and without AR interaction errors. During the tasks, neck and shoulder angles and muscle activity were collected. The results showed that the presence of errors led to increased neck extension and shoulder flexion angles. Peak muscle activity in the shoulders (anterior and medial deltoids) also increased with errors. These findings highlight the importance of reducing interaction errors in AR interfaces to minimize risks of musculoskeletal discomfort and injuries in the neck and shoulders

Learning across Diverse Training Modalities is Different across Gender: Insights from a Neuroergonomics Approach

Recent challenges such as COVID-19 call for innovative methods of delivering training. In this study, we compare the effectiveness of augmented reality (AR) based psychomotor learning compared to videobased training. Participants completed either AR or video training to learn four basic AR interactions, before completing an AR evaluation. The AR group reported greater workload during training but demonstrated better performance during evaluation. We used functional near infrared spectroscopy (fNIRS) to compare brain activation and functional connectivity across groups during evaluation. We found that participants in the AR group displayed smaller brain networks during the evaluation, suggesting more efficient utilization of neural resources. Sex differences in activation and connectivity suggest that men and women potentially employ different neural strategies while learning motor skills in AR. Future studies should examine how these differences could impact performance and user experience in future AR-based training programs.

Adaptive Training on Basic AR Interactions: Bi-Variate Metrics and Neuroergonomic Evaluation Paradigms

Augmented Reality (AR) training is a cost-effective and safe alternative to traditional instructional methods. However, training novices in basic mid-air AR interactions remains challenging. To address this, we aimed to: (a) develop a robust metric to evaluate user performance across different AR interaction techniques and develop adaptation models to predict additional training requirements; (b) evaluate the adaptation models using a neuroergonomics approach. We conduct a two-phase study during which, novice participants perform simple AR interactions: poking and raycasting. In Phase-I, twenty-seven participants’ data is used to identify a bi-variate performance metric based on median completion ime and consistency. Unsupervised models are trained using this metric to classify participants as low/high performers. In Phase-II, we evaluate the models on twenty-one new participants and analyze the differences in performance, neural activity and heart-rate variability between low/high performers. Our study showcases the effectiveness of our models and further discusses the potential of integrating neuroergonomics for advanced AR-based training applications.

Augmented reality in the metaverse market: the role of multimodal sensory interaction

PurposeIn the growing information systems (IS) literature on metaverse, augmented reality (AR) technology is regarded as a cornerstone of the metaverse which enables interaction services. Interaction has been identified as a core technology characteristic of metaverse shopping environments. Based on previous human–technology interaction research, the authors further explicate interaction to be multimodal sensory. The purpose of this study is thus to better understand the unique nature of interaction in AR technology and highlight the technology's benefits for shopping in metaverse spaces.Design/methodology/approachAn experiment has been conducted to empirically examine the authors' research model. The authors use the structural equation modeling (SEM) approach to analyze the collected data.FindingsThis study conceptualizes image, motion and touchscreen interactions as the three dimensions of multimodal sensory interaction, which can reflect visual-, kinesthetic- and haptic-based sensation stimulation. The authors' findings show that multimodal sensory interaction of AR activates consumers' intention to purchase via a psychological process. To delineate this psychological process, the authors use feelings-as-information theory to posit that experiential factors can influence cognitive factors. More specifically, multimodal sensory interaction is shown to increase multisensory experience and spatial presence, which can effectively reduce product uncertainty and information overload. The two outcomes have been considered to be key issues in online shopping environments.Originality/valueThis study is one of the first ones that shed light on the multimodal sensory peculiarity of AR interactions in the extant IS literature. The authors further highlight the benefits of AR in addressing major online shopping concerns about product uncertainty and information overload, which are largely overlooked by prior research. This study uses feelings-as-information theory to explain the impacts of AR interactions, which reveal the essential role of the experiential process in sensory-enabling technologies. This study enriches the existing theoretical frameworks that mostly focus on the cognitive process. The authors' findings about AR interactions provide noteworthy guidelines for the design of metaverse environments and extend the authors' understanding of how the metaverse may bring benefits beyond traditional online shopping settings.

Large-field Gesture Tracking and Recognition for Augmented Reality Interaction

In recent years, with the continuous development of computer vision and artificial intelligence technology, gesture recognition is widely used in many fields, such as virtual reality, augmented reality and so on. However, the traditional binocular camera architecture is limited by its limited field of view Angle and depth perception range. Fisheye camera is gradually applied in gesture recognition field because of its advantage of larger field of view Angle. Fisheye cameras offer a wider field of vision than previous binocular cameras, allowing for a greater range of gesture recognition. This gives fisheye cameras a distinct advantage in situations that require a wide field of view. However, because the imaging mode of fisheye camera is different from traditional camera, the image of fisheye camera has a certain degree of distortion, which makes the calculation of gesture recognition more complicated. Our goal is to design a distortion correction processing strategy suitable for fisheye cameras in order to extend the range of gesture recognition and achieve large field of view gesture recognition. Combined with binocular technology, we can use the acquired hand depth information to enrich the means of interaction. By taking advantage of the large viewing Angle of the fisheye camera to expand the range of gesture recognition, make it more extensive and accurate. This will help improve the real-time and precision of gesture recognition, which has important implications for artificial intelligence, virtual reality and augmented reality.