Augmented Reality Interface Research Articles

The influence of augmented reality interaction design on Pilot's perceived workload and situation awareness

This work explored the potential for Augmented Reality (AR) rendering information superimposed over the flight deck to increase a pilot's situation awareness (SA). This emerging technology introduced novel human-computer interaction paradigms that would have impact on pilot's cognitive demands. The objective of this research was to evaluate both the pilot's perceived workload and SA while interacting with an AR device using different interactive modes. Participants performed traditional landing checklists as a baseline to compare with the AR gesture-command and voice-command checklists. The research results showed that gesture-commands created additional cognitive and physical demands. Conversely, voice-command checklists could constitute a significant improvement in terms of reducing participants' perceived workload and maximising SA performance. The findings provided evidence that the interactive modes of AR user interface design could influence participant's cognitive information processing and perceived workload in flight operations. However, there were some limitations with AR applications that included latency on response time, narrow field of view, accuracy of voice recognition, calibration within dynamic environment and inexplicable movements of the head position that required further investigation. An AR device can be a great tool for training at the initial stage to increase cost-efficiency in flight operations. Furthermore, the implementation of an AR design may provide part of the potential solution for single pilot operations in the future.

Evaluating an augmented reality interface for sailing navigation: a comparative study with a immersive virtual reality simulator

Sailing navigation is an activity that requires acquiring and processing information from the surrounding environment. The advancement of technology has enabled sailboats to have an increasing number of onboard sensors that make sailing more user-friendly. However, data provided by these sensors are still visualized on 2D digital displays that imitate traditional analog interfaces. Although these displays are strategically placed on the sailboat, the user needs to divert attention from the primary navigation task to look at them, thus spending a significant amount of cognitive resources. AR-based technologies have the potential to overcome these limitations by displaying information registered in the real environment, but there are no studies in the literature for validating the effectiveness of this technology in the field of sailing. Thus, we designed a head-mounted display AR-based interface to assist users in monitoring wind data to avoid user diversion from the primary task of sailing. We conducted a user study involving 45 participants in an Immersive Virtual Reality simulated environment. We collected objective and subjective measures to compare the AR-based interface with a traditional data visualization system. The AR-based interface outperformed the traditional data visualization system regarding reaction time, cognitive load, system usability, and user experience.

BARI: An Affordable Brain-Augmented Reality Interface to Support Human–Robot Collaboration in Assembly Tasks

Human–robot collaboration (HRC) is a new and challenging discipline that plays a key role in Industry 4.0. Digital transformation of industrial plants aims to introduce flexible production lines able to adapt to different products quickly. In this scenario, HRC can be a booster to support flexible manufacturing, thus introducing new interaction paradigms between humans and machines. Augmented reality (AR) can convey much important information to users: for instance, information related to the status and the intention of the robot/machine the user is collaborating with. On the other hand, traditional input interfaces based on physical devices, gestures, and voice might be precluded in industrial environments. Brain–computer interfaces (BCIs) can be profitably used with AR devices to provide technicians solutions to effectively collaborate with robots. This paper introduces a novel BCI–AR user interface based on the NextMind and the Microsoft Hololens 2. Compared to traditional BCI interfaces, the NextMind provides an intuitive selection mechanism based on visual cortex signals. This interaction paradigm is exploited to guide a collaborative robotic arm for a pick and place selection task. Since the ergonomic design of the NextMind allows its use in combination with the Hololens 2, users can visualize through AR the different parts composing the artifact to be assembled, the visual elements used by the NextMind to enable the selections, and the robot status. In this way, users’ hands are always free, and the focus can be always on the objects to be assembled. Finally, user tests are performed to evaluate the proposed system, assessing both its usability and the task’s workload; preliminary results are very encouraging, and the proposed solution can be considered a starting point to design and develop affordable hybrid-augmented interfaces to foster real-time human–robot collaboration.

Robotic on-site adaptive thin-layer printing: Challenges and workflow for design and fabrication of bespoke cementitious plasterwork at full architectural scale

This paper describes the 1:1 scale application of Robotic Plaster Spraying (RPS), a novel, adaptive thin-layer printing technique, using cementitious base coat plaster, realized in a construction setting. In this technique, the print layers are vertical unlike most 3DCP processes. The goal is to explore the applicability and scalability of this spray-based printing technique. In this study, RPS is combined with an augmented interactive design setup, the Interactive Robotic Plastering (IRoP), which allows users to design directly on the construction site, taking the building structure, as-built state of the on-going fabrication and the material behavior into consideration. The experimental setup is an on-site robotic system that consists of a robotic arm mounted on a semi-mobile vertical axis with an integrated, automated pumping and adaptive spraying setup that is equipped with a depth camera. The user interaction is enabled by a controller-based interaction system, interactive design tools, and an augmented reality interface. The paper presents the challenges and the workflow that is needed to work with a complex material system on-site to produce bespoke plasterwork. The workflow includes an interactive design procedure, localization on-site, process control and a data collection method that enables predicting the behavior of complex-to-simulate cementitious material. The results demonstrate the applicability and scalability of the adaptive thin-layer printing technique and address the challenges, such as maintaining material continuity and working with unpredictable material behavior during the fabrication process.

Augmented reality (AR) posture training for manual material handling: iterative design, evaluation, and recommendation for AR interface

Manual material handling (MMH) tasks can pose risks of musculoskeletal disorders for workers, which makes posture training programs essential. The ultimate goal of this research was to design and evaluate an augmented reality (AR) based training platform, namely the virtual instructor application (VIA). Specifically, the usability problems revealed from the preliminary version (VIA-1) were addressed and enhanced features were implemented in the current version (VIA-2). The usability of VIA-2 was assessed by ten individuals with experience of MMH. Participants explored VIA-2 while performing a series of predefined tasks. A usability questionnaire was administered, and a brief interview was conducted. Results showed that VIA-2 had increased usability compared to VIA-1. While some visual content display issues occurred, a future minor refurnish can address these problems. Finally, the iterative design and evaluation scheme can be applied to the future design of AR-based training programs and evaluation protocols.

Design and Evaluation of an Augmented Reality Head-mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

We provide an experimental evaluation of a wearable augmented reality (AR) system we have developed for human-robot teams working on tasks requiring collaboration in shared physical workspace. Recent advances in AR technology have facilitated the development of more intuitive user interfaces for many human-robot interaction applications. While it has been anticipated that AR can provide a more intuitive interface to robot assistants helping human workers in various manufacturing scenarios, existing studies in robotics have been largely limited to teleoperation and programming. Industry 5.0 envisions cooperation between human and robot working in teams. Indeed, there exist many industrial tasks that can benefit from human-robot collaboration. A prime example is high-value composite manufacturing. Working with our industry partner towards this example application, we evaluated our AR interface design for shared physical workspace collaboration in human-robot teams. We conducted a multi-dimensional analysis of our interface using established metrics. Results from our user study (n = 26) show that, subjectively, the AR interface feels more novel and a standard joystick interface feels more dependable to users. However, the AR interface was found to reduce physical demand and task completion time, while increasing robot utilization. Furthermore, user’s freedom of choice to collaborate with the robot may also affect the perceived usability of the system.

Multimodal Data Integration and User Interaction for Avatar Simulation in Augmented Reality

Augmented Reality (AR) allows users to interact with the virtual world in the real-world environment. This paper proposes a child avatar simulation framework using multimodal data integration and user interaction in the AR environment. This framework generates the child avatar that can interact with the user and respond to his/her behaviors and consists of three subsystems: (1) the avatar interaction system scrutinizes user behaviors based on the user data, (2) the avatar action control system generates naturalistic avatar activities (actions and voices) according to the avatar internal status, and (3) the avatar display system renders the avatar through the AR interface. In addition, a child tantrum management training application has been built based on the proposed framework. And a light machine learning model has been integrated to enable efficient and effective speech emotion recognition. A sufficiently realistic child tantrum management training based on the evaluation of clinical child psychologists is enabled, which helps users get familiar with child tantrum management.

Pervasive Augmented Reality—Technology and Ethics

In the foreseeable future, mobile and wearable computing technology with an augmented reality (AR) interface can provide an omnipresent, environmentally adaptive, and everyday reality augmentation. This new pervasive AR technology will lead to a continuous moderation of experienced reality with the potential to support better and faster decision-making, the exploration of new information, and novel ways of communication, interaction, and collaboration. However, pervasive AR technology will also have undesired consequences, e.g., in the areas of privacy, commercial exploitation, distractions, digital inequality, and our perception of what is true and real. Little is known about how severe these effects will be when AR has become pervasive and how they can be prevented or mitigated. We draw on current developments in research and the market, sketch a near-time future of pervasive AR technology, identify ethical considerations, and discuss the development of pervasive AR systems.

Duplicated Reality for Co-located Augmented Reality Collaboration.

When two or more users attempt to collaborate in the same space with Augmented Reality, they often encounter conflicting intentions regarding the occupation of the same working area and self-positioning around such without mutual interference. Augmented Reality is a powerful tool for communicating ideas and intentions during a co-assisting task that requires multi-disciplinary expertise. To relax the constraint of physical co-location, we propose the concept of Duplicated Reality, where a digital copy of a 3D region of interest of the users' environment is reconstructed in real-time and visualized in-situ through an Augmented Reality user interface. This enables users to remotely annotate the region of interest while being co-located with others in Augmented Reality. We perform a user study to gain an in-depth understanding of the proposed method compared to an in-situ augmentation, including collaboration, effort, awareness, usability, and the quality of the task. The result indicates almost identical objective and subjective results, except a decrease in the consulting user's awareness of co-located users when using our method. The added benefit from duplicating the working area into a designated consulting area opens up new interaction paradigms to be further investigated for future co-located Augmented Reality collaboration systems.

The Effect of Context Switching, Focal Switching Distance, Binocular and Monocular Viewing, and Transient Focal Blur on Human Performance in Optical See-Through Augmented Reality.

In optical see-through augmented reality (AR), information is often distributed between real and virtual contexts, and often appears at different distances from the user. To integrate information, users must repeatedly switch context and change focal distance. If the user's task is conducted under time pressure, they may attempt to integrate information while their eye is still changing focal distance, a phenomenon we term transient focal blur. Previously, Gabbard, Mehra, and Swan (2018) examined these issues, using a text-based visual search task on a one-eye optical see-through AR display. This paper reports an experiment that partially replicates and extends this task on a custom-built AR Haploscope. The experiment examined the effects of context switching, focal switching distance, binocular and monocular viewing, and transient focal blur on task performance and eye fatigue. Context switching increased eye fatigue but did not decrease performance. Increasing focal switching distance increased eye fatigue and decreased performance. Monocular viewing also increased eye fatigue and decreased performance. The transient focal blur effect resulted in additional performance decrements, and is an addition to knowledge about AR user interface design issues.

Comparing Desktop vs. Mobile Interaction for the Creation of Pervasive Augmented Reality Experiences.

This paper presents an evaluation and comparison of interaction methods for the configuration and visualization of pervasive Augmented Reality (AR) experiences using two different platforms: desktop and mobile. AR experiences consist of the enhancement of real-world environments by superimposing additional layers of information, real-time interaction, and accurate 3D registration of virtual and real objects. Pervasive AR extends this concept through experiences that are continuous in space, being aware of and responsive to the user’s context and pose. Currently, the time and technical expertise required to create such applications are the main reasons preventing its widespread use. As such, authoring tools which facilitate the development and configuration of pervasive AR experiences have become progressively more relevant. Their operation often involves the navigation of the real-world scene and the use of the AR equipment itself to add the augmented information within the environment. The proposed experimental tool makes use of 3D scans from physical environments to provide a reconstructed digital replica of such spaces for a desktop-based method, and to enable positional tracking for a mobile-based one. While the desktop platform represents a non-immersive setting, the mobile one provides continuous AR in the physical environment. Both versions can be used to place virtual content and ultimately configure an AR experience. The authoring capabilities of the different platforms were compared by conducting a user study focused on evaluating their usability. Although the AR interface was generally considered more intuitive, the desktop platform shows promise in several aspects, such as remote configuration, lower required effort, and overall better scalability.

MINA: A Robotic Assistant for Hospital Fetching Tasks

In this paper, a robotic Multitasking Intelligent Nurse Aid (MINA) is proposed to assist nurses with everyday object fetching tasks. MINA consists of a manipulator arm on an omni-directional mobile base. Before the operation, an augmented reality interface was used to place waypoints. Waypoints can indicate the location of a patient, supply shelf, and other locations of interest. When commanded to retrieve an object, MINA uses simultaneous localization and mapping to map its environment and navigate to the supply shelf waypoint. At the shelf, MINA builds a 3D point cloud representation of the shelf and searches for barcodes to identify and localize the object it was sent to retrieve. Upon grasping the object, it returns to the user. Collision avoidance is incorporated during the mobile navigation and grasping tasks. We performed experiments to evaluate MINA’s efficacy including with obstacles along the path. The experimental results showed that MINA can repeatedly navigate to the specified waypoints and successfully perform the grasping and retrieval task.

COVI3D: Automatic COVID-19 CT Image-Based Classification and Visualization Platform Utilizing Virtual and Augmented Reality Technologies.

Recently many studies have shown the effectiveness of using augmented reality (AR) and virtual reality (VR) in biomedical image analysis. However, they are not automating the COVID level classification process. Additionally, even with the high potential of CT scan imagery to contribute to research and clinical use of COVID-19 (including two common tasks in lung image analysis: segmentation and classification of infection regions), publicly available data-sets are still a missing part in the system care for Algerian patients. This article proposes designing an automatic VR and AR platform for the severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) pandemic data analysis, classification, and visualization to address the above-mentioned challenges including (1) utilizing a novel automatic CT image segmentation and localization system to deliver critical information about the shapes and volumes of infected lungs, (2) elaborating volume measurements and lung voxel-based classification procedure, and (3) developing an AR and VR user-friendly three-dimensional interface. It also centered on developing patient questionings and medical staff qualitative feedback, which led to advances in scalability and higher levels of engagement/evaluations. The extensive computer simulations on CT image classification show a better efficiency against the state-of-the-art methods using a COVID-19 dataset of 500 Algerian patients. The developed system has been used by medical professionals for better and faster diagnosis of the disease and providing an effective treatment plan more accurately by using real-time data and patient information.

Interactive IIoT-Based 5DOF Robotic Arm for Upper Limb Telerehabilitation

Significant advancements in contemporary telemedicine applications enforce the demand for effective and intuitive telerehabilitation tools. Telerehabilitation can minimize the distance, travel burden, and costs between rehabilitative patients and therapists. This research introduces an interactive novel telerehabilitation system that integrates the Industrial Internet of Things (IIoT) platform with a robotic manipulator named xARm-5, aiming to deliver rehabilitation therapies to individuals with upper limb dysfunctions. With the proposed system, a therapist can provide upper limb rehab exercises remotely using an augmented reality (AR) user interface (UI) developed using Vuforia Studio, which transmits bidirectional data through the IIoT platform. The proposed system has a stable communication architecture and low teleoperation latency. Experimental results revealed that with the developed telerehabilitation framework, the xArm-5 could be teleoperated from the developed AR platform and/or use a joystick to provide standard upper limb rehab exercises. Besides, with the designed AR-based UI, a therapist can monitor rehab/robot trajectories along with the AR digital twin of the robot, ensuring that the robot is providing passive therapy for shoulder and elbow movements.

A Systematic Review of User Acceptance in Industrial Augmented Reality

In the industrial work context, Augmented Reality (AR) can support work processes and employees’ cognitive relief through the location-specific and context-related superimposition of real objects with virtual information. The AR-based support of industrial work processes ranges over product development, manufacturing, assembly, maintenance, and training. In all these areas, numerous location-based AR support functions are being prototypically implemented, aiming to improve work efficiency, communication in mobile work situations, or employee qualification in the work process. In contrast to the increasing number of developed AR solutions in recent years, there is no widespread use of these solutions in industrial practice. AR systems’ successful introduction is closely related to user acceptance, which has not been comprehensively considered over the system development process. In addition to improving AR hardware ergonomic features, usability or user interface design play an essential role in user acceptance. Particularly in the context of employee qualification, increasing employee engagement can be named as a success factor. Previous user studies of industrial AR systems only include individual user acceptance aspects. The use of game elements has not been widely addressed in connection with manual tasks in production environments, including AR-based assistance systems. This paper aims to examine user acceptance of industrial AR systems and the relevant factors for investigating user acceptance, e.g., ease of use or enjoyment, based on a systematic literature review. An analysis of existing review articles on industrial AR systems elaborates the current state of the art and identifies the research gap. This review of 109 scientific articles from 2011 to 2020 provides an overview of the current state of research on the inclusion of user acceptance in industrial AR systems. The identified papers from the scientific databases, Scopus, Web of Science, IEEE Xplore Digital Library, ACM Digital Library, and Science Direct, are evaluated for their relevance and selected for further analysis based on inclusion and exclusion criteria, e.g., year of publication. This review presents the current challenges regarding user acceptance of industrial AR systems and future possibilities for the comprehensive integration of user acceptance factors into the development, evaluation, and implementation process.

The Potential Applications of Augmented Reality in Fetoscopic Surgery for Antenatal Treatment of Myelomeningocele

To develop a preoperative planning method using augmented reality (AR) of a specific surgical procedure: fetoscopy for myelomeningocele repair. Imaging data were acquired of a pregnant woman at 27 weeks of gestation whose fetus was diagnosed with myelomeningocele. The patient was identified as a candidate for fetoscopic repair of the spine defect, and an AR application for mobile device simulation was developed. The virtual customized model was created by analysis of the presurgical magnetic resonance imaging. A real-time AR interface was developed by using an application that enhanced the anatomical aspects of both mother and fetus. A virtual model for planning fetoscopy repair for myelomeningocele was developed. Preoperative and postoperative procedures were successfully carried out, emphasizing the beneficial role of the AR application. The use of the AR model allowed the multidisciplinary team to engage in discussion to determine the appropriate surgical approach. It also allowed a clearer explanation of the procedure to the parents enabling a better understanding of the parents regarding specifics characteristics of their baby's spine defect. This new preoperative platform using a virtual model represents an important tool to improve patient's comprehension, multidisciplinary discussion, and surgical planning. In addition, it can be used worldwide as a teaching tool in the fetal surgery field.

Applying the ARCS Motivation Theory for the Assessment of AR Digital Media Design Learning Effectiveness

This research adopts Keller’s ARCS motivation theory as a method to create a teaching experiment by integrating augmented reality (AR) into teaching in order to enhance learning interest and learning effectiveness in a digital media design course. The purpose of this research is to examine the application of AR in quarantine during the COVID-19 pandemic, whereby students can enhance their learning interest, learning satisfaction, and learning performance. Augmented reality acts as a tool for this research, wherein it is applied with the course of a 3D model-based interface and built-in learning contexts for the “digital media design” of the learning topics. The learning performance of the test group students was examined through the aspects of attention, relevance, confidence, and satisfaction, according to the ARCS motivation theory. According to the results of quantitative validation, experimental teaching with AR is more effective than traditional teaching methods. The learning feedback of the test group students obtained a positive result by using the AR. This research concludes with three results: firstly, integrating AR into teaching can improve students’ concentration with respect to digital media design practice; secondly, video teaching with an AR interface can help increase students’ confidence in digital media design learning; lastly, applying AR during the learning process can enhance the digital media visual effects that effectively enhance students’ self-learning abilities.

Pre-define Rotation Amplitudes Object Rotation in Handheld Augmented Reality

Interaction is one of the important topics to be discussed since it includes the interface where the end-user communicates with the augmented reality (AR) system. In handheld AR interface, the traditional interaction techniques are not suitable for some AR applications due to the different attributes of handheld devices that always refer to smartphones and tablets. Currently interaction techniques in handheld AR are known as touch-based technique, mid-air gesture-based technique and device-based technique that can led to a wide discussion in related research areas. However, this paper will focus to discover the device-based interaction technique because it has proven in the previous studies to be more suitable and robust in several aspects. A novel device-based 3D object rotation technique is proposed to solve the current problem in performing 3DOF rotation of 3D object. The goal is to produce a precise and faster 3D object rotation. Therefore, the determination of the rotation amplitudes per second is required before the fully implementation. This paper discusses the implementation in depth and provides a guideline for those who works in related to device-based interaction.

Patching interpretable And‐Or‐Graph knowledge representation using augmented reality

AbstractWe present a novel augmented reality (AR) interface to provide effective means to diagnose a robot's erroneous behaviors, endow it with new skills, and patch its knowledge structure represented by an And‐Or‐Graph (AOG). Specifically, an AOG representation of opening medicine bottles is learned from human demonstration and yields a hierarchical structure that captures the spatiotemporal compositional nature of the given task, which is highly interpretable for the users. Through a series of psychological experiments, we demonstrate that the explanations of a robotic system, inherited from and produced by the AOG, can better foster human trust compared to other forms of explanations. Moreover, by visualizing the knowledge structure and robot states, the AR interface allows human users to intuitively understand what the robot knows, supervise the robot's task planner, and interactively teach the robot with new actions. Together, users can quickly identify the reasons for failures and conveniently patch the current knowledge structure to prevent future errors. This capability demonstrates the interpretability of our knowledge representation and the new forms of interactions afforded by the proposed AR interface.

Calculating and Analyzing Angular Head Jerk in Augmented and Virtual Reality: Effect of AR Cue Design on Angular Jerk

In this work, we propose a convenient method for evaluating levels of angular jerk in augmented reality (AR) and virtual reality (VR). Jerk is a rarely analyzed metric in usability studies, although it can be measured and calculated easily with most head-worn displays and can yield highly relevant information to designers. Here, we developed and implemented a system capable of calculating and analyzing jerk in real-time based on orientation data from an off-the-shelf head-worn display. An experiment was then carried out to determine whether the presence of AR user interface annotations results in changes to users’ angular head jerk when conducting a time-pressured visual search task. Analysis of the data indicates that a decrease in jerk is significantly associated with the use of AR augmentations. As noted in the limitations section, however, the conclusions drawn from this work should be limited, as this analysis method is novel in the VR/AR space and because of methodological limitations that limited the reliability of the jerk data. The work presented herein considerably facilitates the use of jerk as a quick component measure of usability and serves as an initial point off which future research involving jerk in VR and AR can be performed.