Augmented Reality Interface Research Articles

Toward AI-enabled augmented reality to enhance the safety of highway work zones: Feasibility, requirements, and challenges

Highway work zones are considered among the most hazardous working environments. In 2018 alone, 124 workers lost their lives to fatal accidents. The lack of predictive safety systems that notify workers of upcoming dangers in advance is a major reason to blame in the highway maintenance and operation community. This article presents an integrative design framework for bringing recent advances in Augmented Reality (AR) and Artificial Intelligence (AI) to enhance the safety of highway workers through real-time multimodal notifications on-spot. To this end, this article conceptualizes and co-designs three major pillars: (1) AR user interface design for multimodal notification, (2) real-time AI at the edge for vehicle detection/classification from distance, and (3) real-time wireless communication in work zone setting to enable latency-aware operation between AI and AR components. Our early results demonstrate that we can achieve 24.83 FPS end-to-end execution latency on the Xavier AGX Jetson board with 48.7% mAP on BDD100K dataset, and a real-time communication covering 120 meters with an average latency of 5.1 ms at the farthest distance. Our mixed-method user research also reveals an acceptable level of excitement and engagement from the body of highway workers toward both the proposed technology and the designed user interface. Overall, this article provides a proof-of-concept toward AI-enabled AR safety systems in highway work zones.

A comparative evaluation of the wearable augmented reality-based data presentation interface and traditional methods for data entry tasks

Data entry is a ubiquitous task performed in today's offices. Persistent data entry is linked with high workload and fatigue due to poor ergonomic workplace design and poor posture. This study aims to alleviate data entry operators' workload and improve data entry performance by applying wearable augmented reality (AR) technology to data entry tasks. An AR-based interface was developed and used to present data to the participants, who entered the data in web-based data entry forms. A total of eighteen participants performed data entry tasks to evaluate the AR interface with traditional methods – extra desktop monitor and paper-based data presentation methods. Each method's performance was judged on the task completion time, typing error rate, workload, and usability. The usability and overall perceived workload while using an AR interface for data entry were similar to the traditional way of using paper, despite the additional burden due to the weight of the AR headset. AR interface did not perform better than the extra desktop monitor interface for usability and overall perceived workload. The results from this study can be utilized to design AR devices that are suited for data entry tasks.

Development of an Augmented Reality Concept for Icebreaker Assistance and Convoy Operations

A vessel convoy is a complex and high-risk operation completed during icebreaking operations in the Arctic. Icebreaker navigators need to continuously communicate with their crew while monitoring information such as speed, heading, and distance between vessels in the convoy. This paper presents an augmented reality user interface concept, which aims to support navigators by improving oversight and safety during convoy operations. The concept demonstrates how augmented reality can help to realize a situated user interface that adapts to user’s physical and operational contexts. The concept was developed through a human-centered design process and tested through a virtual reality simulator in a usability study involving seven mariners. The results suggest that augmented reality has the potential to improve the safety of convoy operations by integrating distributed information with heads-up access to operation-critical information. However, the user interface concept is still novel, and further work is needed to develop the concept and safely integrate augmented reality into maritime operations.

Investigating a Virtual Reality Based Subterranean Scenario Examining Augmented Reality Implications for Military Operators

Virtual and augmented reality have made it possible to test new technologies for military missions - such as using augmented reality to support complicated subterranean (sub-t) missions. The military will soon offer operators the ability to wear augmented reality (AR) head-mounted displays (HMDs) while conducting missions. Therefore, it is crucial to understand how this emerging technology will positively and negatively impact Soldiers’ ability to execute missions. This paper describes the development of a virtual reality (VR) scenario designed to support user experience research for evaluating the efficacy of Soldier AR systems during sub-t operations. We describe usability test results that captured feedback and reactions to the VR scenario and the AR interface that presented critical information through a VR HMD. We conclude with lessons learned from our development process and future research plans.

Different System Error Rates in Augmented Reality Interface Affected Cognitive Stress

Different System Error Rates in Augmented Reality Interface Affected Cognitive Stress

THE EVALUATION OF CASCADING AND RADIAL MENUS FOR AUGMENTED REALITY IN A SMARTPHONE

The article describes an evaluation of cascading menu and radial menu in augmented reality (AR) on a smartphone. To evaluate the menus, we measured time to finish the tasks (in seconds) and usability (learnability, efficiency, memorability, errors, and satisfaction) using the System Usability Scale Questionnaire. Data obtained is then analyzed using paired-samples t-test with repeated measures and Wilcoxon signed-rank test to find the statistically significant difference. Overall results showed a preference for the cascading menu type over the radial menu in terms of usability, different from other studies. Statistically, a significant difference was found performance-wise, which indicates that participants are more familiar with cascading menus which contributes to better task time. However, the performance improved the more prolonged the participants got used to the menu when it comes to the radial menu. For errors rates, it is found that the radial menu is less error-prone than cascading menu. This evaluation can help interface designers decide whether they should use the cascading menu or radial menu when designing an interface for AR on a smartphone. The research looks into filling the evaluation gap on the radial and cascading menu, specifically in augmented reality on a smartphone. While other studies have studied and evaluated the cascading menu and radial menu in other environments, this research, on the other hand, tested and evaluated the menu in an augmented reality environment instead of the more traditional desktop environment.

Towards Indistinguishable Augmented Reality

Adding virtual information that is indistinguishable from reality has been a long-awaited goal in Augmented Reality (AR). While already demonstrated in the 1960s, only recently have Optical See-Through Head-Mounted Displays (OST-HMDs) seen a reemergence, partially thanks to large investments from industry, and are now considered to be the ultimate hardware for augmenting our visual perception. In this article, we provide a thorough review of state-of-the-art OST-HMD-related techniques that are relevant to realize the aim of an AR interface almost indistinguishable from reality. In this work, we have an initial look at human perception to define requirements and goals for implementing such an interface. We follow up by identifying three key challenges for building an OST-HMD-based AR interface that is indistinguishable from reality: spatial realism, temporal realism, and visual realism. We discuss existing works that aim to overcome these challenges while also reflecting against the goal set by human perception. Finally, we give an outlook into promising research directions and expectations for the years to come.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

AUGMENTED REALITY ASSET TRACKING USING HOLOLENS

Abstract. Asset Tracking is an essential component of building management process. It involves creating and maintaining a database of detailed information of assets such as location, condition, brand, and type. This information can help building owners make informed decisions for cost-effective maintenance of building assets. Existing approaches to perform asset tracking require a manual process of measuring and recording the asset condition and location, which is labour-intensive and costly. The typical approach usually includes a human operator with pen and paper inspecting the site and manually recording the information about the asset. In this paper, we propose an augmented reality asset tracking system using HoloLens to reduce the manual labour involved in this process. The system can automatically detect the asset, record and update its related information by visual inspection. Assets are detected by feeding images captured by the HoloLens built-in camera to a pre-trained object detection network. Using a combination of various sensor readings from the HoloLens, the system can estimate the location of the asset using visual simultaneous localization and mapping (vSLAM). This information is then viewed and verified by the user using the augmented reality user interface. Upon the user confirmation, this information will be uploaded to a database. As a case study, we demonstrate a vending machine tracking system which is able to detect and localise the vending machines in an indoor environment and create a database of vending machine information. The system can detect vending machines with a mean average precision of 94.8% and a localization accuracy of 2.3 meters without pre-screening or user input.

"Disruptive Technology" in Spine Surgery and Education: Virtual and Augmented Reality.

Technological advancements are the drivers of modern-day spine care. With the growing pressure to deliver faster and better care, surgical-assist technology is needed to harness computing power and enable the surgeon to improve outcomes. Virtual reality (VR) and augmented reality (AR) represent the pinnacle of emerging technology, not only to deliver higher quality education through simulated care, but also to provide valuable intraoperative information to assist in more efficient and more precise surgeries. To describe how the disruptive technologies of VR and AR interface in spine surgery and education. We review the relevance of VR and AR technologies in spine care, and describe the feasibility and limitations of the technologies. We discuss potential future applications, and provide a case study demonstrating the feasibility of a VR program for neurosurgical spine education. Initial experiences with VR and AR technologies demonstrate their applicability and ease of implementation. However, further prospective studies through multi-institutional and industry-academic partnerships are necessary to solidify the future of VR and AR in spine surgery education and clinical practice.

Eye See What You See: Exploring How Bi-Directional Augmented Reality Gaze Visualisation Influences Co-Located Symmetric Collaboration

Gaze is one of the predominant communication cues and can provide valuable implicit information such as intention or focus when performing collaborative tasks. However, little research has been done on how virtual gaze cues combining spatial and temporal characteristics impact real-life physical tasks during face to face collaboration. In this study, we explore the effect of showing joint gaze interaction in an Augmented Reality (AR) interface by evaluating three bi-directional collaborative (BDC) gaze visualisations with three levels of gaze behaviours. Using three independent tasks, we found that all bi-directional collaborative BDC visualisations are rated significantly better at representing joint attention and user intention compared to a non-collaborative (NC) condition, and hence are considered more engaging. The Laser Eye condition, spatially embodied with gaze direction, is perceived significantly more effective as it encourages mutual gaze awareness with a relatively low mental effort in a less constrained workspace. In addition, by offering additional virtual representation that compensates for verbal descriptions and hand pointing, BDC gaze visualisations can encourage more conscious use of gaze cues coupled with deictic references during co-located symmetric collaboration. We provide a summary of the lessons learned, limitations of the study, and directions for future research.

A Holographic Augmented Reality Interface for Visualizing of MRI Data and Planning of Neurosurgical Procedures.

The recent introduction of wireless head-mounted displays (HMD) promises to enhance 3D image visualization by immersing the user into 3D morphology. This work introduces a prototype holographic augmented reality (HAR) interface for the 3D visualization of magnetic resonance imaging (MRI) data for the purpose of planning neurosurgical procedures. The computational platform generates a HAR scene that fuses pre-operative MRI sets, segmented anatomical structures, and a tubular tool for planning an access path to the targeted pathology. The operator can manipulate the presented images and segmented structures and perform path-planning using voice and gestures. On-the-fly, the software uses defined forbidden-regions to prevent the operator from harming vital structures. In silico studies using the platform with a HoloLens HMD assessed its functionality and the computational load and memory for different tasks. A preliminary qualitative evaluation revealed that holographic visualization of high-resolution 3D MRI data offers an intuitive and interactive perspective of the complex brain vasculature and anatomical structures. This initial work suggests that immersive experiences may be an unparalleled tool for planning neurosurgical procedures.

The Design of a Gamified Responsible Use of Social Media

Social media is part of almost everyone's daily life. Its networking facilities redefines the way people connect and interact with each other. However, social media is reported being misused in different ways, especially the millennials. There is a need to elevate the teens' level of empowerment on the responsible use of social media. Other technological innovations like augmented reality (AR) and digital gamification provides pedagogical benefits. Digital gamification in the classroom is a teaching strategy that translates content and delivery into a game using digital technology. On the other hand, AR is an emerging technology to enlarge real-life situations in multimedia. Research shows both technologies increase interactivity well as attention span among the learners. Additionally, these technologies, including social media, are among the many useful tools in teaching twenty first century learners once correctly used. With this, a game-based mobile application was developed to advocate responsible use of social media among teens. The learning content was gamified in augmented reality to provide an innovative teaching and learning way at Silliman University. This paper describes the gamification design of the learning trail on the topic of responsible use of social media. Specifically, it presents the publishing process of the augmented reality mobile application about responsible use of social media. Kuhlmann's 3C Model (Challenge, Choices, Consequences) was utilized to formulate the learning content. It also demonstrates the design phases, game mechanics, and the general evaluation of the learning application. Four challenges were developed. These are (a) be familiar with security and privacy policies, (b) do not express concerns about others, even if you think you are anonymous, (c) respond to digital offenders, (d) do not tell the world about an upcoming vacation. These challenges were translated into graphics and animations. The animated material was compiled, programmed, and published to a server of the mobile application. The gamified learning trail on responsible use of social media is accessible through QR codes leading to the augmented reality interface. The design was validated and found to be relevant and engaging.

Gesture Controlled Tactile Augmented Reality Interface for the Visually Impaired

This paper proposes to create an augmented reality interface for the visually impaired, enabling a way of haptically interacting with the computer system by creating a virtual workstation, facilitating a natural and intuitive way to accomplish a multitude of computer-based tasks (such as emailing, word processing, storing and retrieving files from the computer, making a phone call, searching the web, etc.). The proposed system utilizes a combination of a haptic glove device, a gesture-based control system, and an augmented reality computer interface which creates an immersive interaction between the blind user and the computer. The gestures are recognized, and the user is provided with audio and vibratory haptic feedbacks. This user interface allows the user to actually “touch, feel, and physically interact” with digital controls and virtual real estate of a computer system. A test of applicability was conducted which showcased promising positive results.

Augmented Reality Interfaces Using Virtual Customization of Microstructured Electronic Skin Sensor Sensitivity Performances

AbstractElectronic skins equip robots and biomedical devices with intuitive skin‐like sensitivity. Performance‐driven design of electronic skins is a critical need for electronic or biomedical applications. Prior research primarily focuses on investigating effects of microstructures on sensor performance at low pressure ranges. However, having predictive and tunable electro–mechanical responses across an extensive pressure range (>100 kPa) is paramount. Here, the authors propose a system that virtually customizes micropyramids for e‐skin sensors. The associations between geometry parameters, material properties, and single‐pyramid performance are systematically explored via numerical simulations, empirical characterizations, and analytical solutions. These experimentally validated models allow for the determination of the sensor parameters for the desired performance. An augmented reality interface system for surgery skills training by optimizing sensitivities that match varying tissue stiffnesses is further demonstrated. The platform enables greater effectiveness in rapidly iterating and designing micropyramidal e‐skin for applications in augmented reality interfaces, robotics, and telehealthcare.

The utility of augmented reality in lateral skull base surgery: A preliminary report

ObjectiveTo discuss the utility of augmented reality in lateral skull base surgery. PatientsThose undergoing lateral skull base surgery at our institution. Intervention(s)Cerebellopontine angle tumor resection using an augmented reality interface. Main outcome measure(s)Ease of use, utility of, and future directions of augmented reality in lateral skull base surgery. ResultsAnecdotally we have found an augmented reality interface helpful in simulating cerebellopontine angle tumor resection as well as assisting in planning the incision and craniotomy. ConclusionsAugmented reality has the potential to be a useful adjunct in lateral skull base surgery, but more study is needed with large series.

Comparing State-of-the-Art and Emerging Augmented Reality Interfaces for Autonomous Vehicle-to-Pedestrian Communication

Providing pedestrians and other vulnerable road users with a clear indication about a fully autonomous vehicle status and intentions is crucial to make them coexist. In the last few years, a variety of external interfaces have been proposed, leveraging different paradigms and technologies including vehicle-mounted devices (like LED panels), short-range on-road projections, and road infrastructure interfaces (e.g., special asphalts with embedded displays). These designs were experimented in different settings, using mockups, specially prepared vehicles, or virtual environments, with heterogeneous evaluation metrics. Promising interfaces based on Augmented Reality (AR) have been proposed too, but their usability and effectiveness have not been tested yet. This paper aims to complement such body of literature by presenting a comparison of state-of-the-art interfaces and new designs under common conditions. To this aim, an immersive Virtual Reality-based simulation was developed, recreating a well-known scenario represented by pedestrians crossing in urban environments under non-regulated conditions. A user study was then performed to investigate the various dimensions of vehicle-to-pedestrian interaction leveraging objective and subjective metrics. Even though no interface clearly stood out over all the considered dimensions, one of the AR designs achieved state-of-the-art results in terms of safety and trust, at the cost of higher cognitive effort and lower intuitiveness compared to LED panels showing anthropomorphic features. Together with rankings on the various dimensions, indications about advantages and drawbacks of the various alternatives that emerged from this study could provide important information for next developments in the field.

Social Media Data in an Augmented Reality System for Situation Awareness Support in Emergency Control Rooms

During crisis situations, emergency operators require fast information access to achieve situation awareness and make the best possible decisions. Augmented reality could be used to visualize the wealth of user-generated content available on social media and enable context-adaptive functions for emergency operators. Although emergency operators agree that social media analytics will be important for their future work, it poses a challenge to filter and visualize large amounts of social media data. We conducted a goal-directed task analysis to identify the situation awareness requirements of emergency operators. By collecting tweets during two storms in Germany we evaluated the usefulness of Twitter data for achieving situation awareness and conducted interviews with emergency operators to derive filter strategies for social media data. We synthesized the results by discussing how the unique interface of augmented reality can be used to integrate social media data into emergency control rooms for situation awareness support.

Evaluating the Potential of Augmented Reality Interfaces for Exploring Underwater Historical Sites

Underwater cultural heritage sites represent an attractive and exciting experience for diving tourists, even if often it is complicated for them to understand the significance and value of the remains that are usually strongly damaged and covered by the marine organisms. Thanks to the recent advancements in technologies that overcome these problems, augmented reality is nowadays possible even in such harsh conditions, opening new possibilities for enhancing the diver’s experience. However, no user study has formally evaluated the usefulness and usability of augmented reality in open sea underwater environments. This paper presents two novel solutions for underwater augmented reality: a compact marker-based system for small areas, and a complex acoustic system for large areas. Both of them were deployed at an underwater cultural heritage site and evaluated by ten divers in experiments analyzing their perception and remembrance, interests, and user experience. For comparison, the same study was also performed with non-divers assessing the marker-based system on land. Results show that both systems allow divers to encounter new and exciting moments and provide valuable insights for underwater augmented reality applications.

A Perceptual Color-Matching Method for Examining Color Blending in Augmented Reality Head-Up Display Graphics.

Augmented reality (AR) offers new ways to visualize information on-the-go. As noted in related work, AR graphics presented via optical see-through AR displays are particularly prone to color blending, whereby intended graphic colors may be perceptually altered by real-world backgrounds, ultimately degrading usability. This work adds to this body of knowledge by presenting a methodology for assessing AR interface color robustness, as quantitatively measured via shifts in the CIE color space, and qualitatively assessed in terms of users' perceived color name. We conducted a human factors study where twelve participants examined eight AR colors atop three real-world backgrounds as viewed through an in-vehicle AR head-up display (HUD); a type of optical see-through display used to project driving-related information atop the forward-looking road scene. Participants completed visual search tasks, matched the perceived AR HUD color against the WCS color palette, and verbally named the perceived color. We present analysis that suggests blue, green, and yellow AR colors are relatively robust, while red and brown are not, and discuss the impact of chromaticity shift and dispersion on outdoor AR interface design. While this work presents a case study in transportation, the methodology is applicable to a wide range of AR displays in many application domains and settings.