Wearable Augmented Reality System Research Articles

A real-time wearable AR system for egocentric vision on the edge

Real-time performance is critical for Augmented Reality (AR) systems as it directly affects responsiveness and enables the timely rendering of virtual content superimposed on real scenes. In this context, we present the DARLENE wearable AR system, analysing its specifications, overall architecture and core algorithmic components. DARLENE comprises AR glasses and a wearable computing node responsible for several time-critical computation tasks. These include computer vision modules developed for the real-time analysis of dynamic scenes supporting functionalities for instance segmentation, tracking and pose estimation. To meet real-time requirements in limited resources, concrete algorithmic adaptations and design choices are introduced. The proposed system further supports real-time video streaming and interconnection with external IoT nodes. To improve user experience, a novel approach is proposed for the adaptive rendering of AR content by considering the user’s stress level, the context of use and the environmental conditions for adjusting the level of presented information towards enhancing their situational awareness. Through extensive experiments, we evaluate the performance of individual components and end-to-end pipelines. As the proposed system targets time-critical security applications where it can be used to enhance police officers’ situational awareness, further experimental results involving end users are reported with respect to overall user experience, workload and evaluation of situational awareness.

Visualization modality for augmented reality guidance of in-depth tumour enucleation procedures.

Recent research studies reported that the employment of wearable augmented reality (AR) systems such as head-mounted displays for the in situ visualisation of ultrasound (US) images can improve the outcomes of US-guided biopsies through reduced procedure completion times and improved accuracy. Here, the authors continue in the direction of recent developments and present the first AR system for guiding an in-depth tumour enucleation procedure under US guidance. The system features an innovative visualisation modality with cutting trajectories that 'sink' into the tissue according to the depth reached by the electric scalpel, tracked in real-time, and a virtual-to-virtual alignment between the scalpel's tip and the trajectory. The system has high accuracy in estimating the scalpel's tip position (mean depth error of 0.4 mm and mean radial error of 1.34 mm). Furthermore, we demonstrated with a preliminary user study that our system allowed us to successfully guide an in-depth tumour enucleation procedure (i.e. preserving the safety margin around the lesion).

Design and Evaluation of an Augmented Reality Head-mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

We provide an experimental evaluation of a wearable augmented reality (AR) system we have developed for human-robot teams working on tasks requiring collaboration in shared physical workspace. Recent advances in AR technology have facilitated the development of more intuitive user interfaces for many human-robot interaction applications. While it has been anticipated that AR can provide a more intuitive interface to robot assistants helping human workers in various manufacturing scenarios, existing studies in robotics have been largely limited to teleoperation and programming. Industry 5.0 envisions cooperation between human and robot working in teams. Indeed, there exist many industrial tasks that can benefit from human-robot collaboration. A prime example is high-value composite manufacturing. Working with our industry partner towards this example application, we evaluated our AR interface design for shared physical workspace collaboration in human-robot teams. We conducted a multi-dimensional analysis of our interface using established metrics. Results from our user study (n = 26) show that, subjectively, the AR interface feels more novel and a standard joystick interface feels more dependable to users. However, the AR interface was found to reduce physical demand and task completion time, while increasing robot utilization. Furthermore, user’s freedom of choice to collaborate with the robot may also affect the perceived usability of the system.

Augmented reality flavor: cross-modal mapping across gustation, olfaction, and vision

Gustatory display research is still in its infancy despite being one of the essential everyday senses that human practice while eating and drinking. Indeed, the most important and frequent tasks that our brain deals with every day are foraging and feeding. The recent studies by psychologists and cognitive neuroscientist revealed how complex multisensory rely on the integration of cues from all the human senses in any flavor experiences. The perception of flavor is multisensory and involves combinations of gustatory and olfactory stimuli. The cross-modal mapping between these modalities needs to be more explored in the virtual environment and simulation, especially in liquid food. In this paper, we present a customized wearable Augmented Reality (AR) system and olfaction display to study the effect of vision and olfaction on the gustatory sense. A user experiment and extensive analysis conducted to study the influence of each stimulus on the overall flavor, including other factors like age, previous experience in Virtual Reality (VR)/AR, and beverage consumption. The result showed that smell contributes strongly to the flavor with less contribution to the vision. However, the combination of these stimuli can deliver richer experience and a higher belief rate. Beverage consumption had a significant effect on the flavor belief rate. Experience is correlated with stimulus and age is correlated with belief rate, and both indirectly affected the belief rate.

Metrology-Based Design of a Wearable Augmented Reality System for Monitoring Patient’s Vitals in Real Time

In this work, an augmented-reality (AR) system for monitoring patient's vitals in real time during surgical procedures is proposed and metrologically characterized in terms of transmission error rates and latency. These specifications, in fact, are crucial to ensure real-time response which is a critical requirement for Health applications. The proposed system automatically acquires the vitals from the operating room instrumentation, and shows them in real time directly on a set of wearable AR glasses. In this way, the surgical team has a real-time visualization of a comprehensive set of patient's information, without constantly looking at the instrumentation. The system was designed to ensure modularity, flexibility, ease of use and, most importantly, a reliable communication. The system was designed, implemented and validated experimentally through on-field tests carried out at the Academic Hospital of Federico II University, using instrumentation typically available in the operating room: namely, a respiratory ventilator and a patient monitor. Results of the experimental validation highlight the need for the metrological approach in conceiving advanced monitoring surgical procedures in operating room.

Wearable AR and 3D Ultrasound: Towards a Novel Way to Guide Surgical Dissections

Nowadays, ultrasound (US) is increasingly being chosen as imaging modality for both diagnostic and interventional applications, owing to its positive characteristics in terms of safety, low footprint, and low cost. The combination of this imaging modality with wearable augmented reality (AR) systems, such as the head-mounted displays (HMD), comes forward as a breakthrough technological solution, as it allows for hands-free interaction with the augmented scene, which is an essential requirement for the execution of high-precision manual tasks, such as in surgery. What we propose in this study is the integration of an AR navigation system (HMD plus dedicated platform) with a 3D US imaging system to guide a dissection task that requires maintaining safety margins with respect to unexposed anatomical or pathological structures. For this purpose, a standard scalpel was sensorized to provide real-time feedback on the position of the instrument during the execution of the task. The accuracy of the system was quantitatively assessed with two different experimental studies: a targeting experiment, which revealed a median error of 2.53 mm in estimating the scalpel to target distance, and a preliminary user study simulating a dissection task that requires reaching a predefined distance to an occult lesion. The second experiment results showed that the system can be used to guide a dissection task with a mean accuracy of 0.65 mm, with a mean angular error between the ideal and actual cutting plane of 2.07°. The results encourage further studies to fully exploit the potential of wearable AR and intraoperative US imaging to accurately guide deep surgical tasks, such as to guide the excision of non-palpable breast tumors ensuring optimal margin clearance.

A scalable wearable AR system for manual order picking based on warehouse floor-related navigation

In manual order picking systems, a paper list specifying the name, location, and amount of each item is often given, and this would exert mental pressure on pickers while finding target shelf bins in high-density picking environments. Augmented reality (AR) can provide a friendly alternative to improve the manual order picking performance by conveying picking information into visual instruction. Nevertheless, there is still no systematic consensus to deploy the pick-by-AR associated with actual warehouse workplaces. To establish a spatial correspondence between the visual guidance with the actual workplace, the multi-marker-based global map about the warehouse floor is established in advance. Instead of traditional single marker-based pick-by-AR methods, the warehouse floor-related map can provide an accurate and continuous navigation performance for intuitive AR guidance, allowing the picker to move freely on the warehouse floor without limiting to some certain locations. Besides, a systematic pick-by-AR solution is available by integrating the proposed method to a lightweight wearable AR device, and this easy-to-deploy pick-by-AR solution can alleviate the worker’s mental effort while doing picking action. Finally, the pick-by-AR system described in the paper is deployed in the automobile assembly line, and experimental results illustrate that the proposed method can improve the picking efficiency while reducing the picking errors compared with the current paper-based order picking.

Enhancing User Experiences of Mobile-Based Augmented Reality via Spatial Augmented Reality: Designs and Architectures of Projector-Camera Devices

As smartphones, tablet computers, and other mobile devices have continued to dominate our digital world ecosystem, there are many industries using mobile or wearable devices to perform Augmented Reality (AR) functions in their workplaces in order to increase productivity and decrease unnecessary workloads. Mobile-based AR can basically be divided into three main types: phone-based AR, wearable AR, and projector-based AR. Among these, projector-based AR or Spatial Augmented Reality (SAR) is the most immature and least recognized type of AR for end users. This is because there are a small number of commercial products providing projector-based AR functionalities in a mobile manner. Also, prices of mobile projectors are still relatively high. Moreover, there are still many technical problems regarding projector-based AR that have been left unsolved. Nevertheless, it is projector-based AR that has potential to solve a fundamental problem shared by most mobile-based AR systems. Also the always-visible nature of projector-based AR is one good answer for solving current user experience issues of phone-based AR and wearable AR systems. Hence, in this paper, we analyze what are the user experience issues and technical issues regarding common mobile-based AR systems, recently widespread phone-based AR systems, and rising wearable AR systems. Then for each issue, we propose and explain a new solution of how using projector-based AR can solve the problems and/or help enhance its user experiences. Our proposed framework includes hardware designs and architectures as well as a software computing paradigm towards mobile projector-based AR systems. The proposed design is evaluated by three experts using qualitative and semiquantitative research approaches.

The Role of Camera Convergence in Stereoscopic Video See-through Augmented Reality Displays

In the realm of wearable augmented reality (AR) systems, stereoscopic video see-through displays raise issues related to the user’s perception of the three-dimensional space. This paper seeks to put forward few considerations regarding the perceptual artefacts common to standard stereoscopic video see-through displays with fixed camera convergence. Among the possible perceptual artefacts, the most significant one relates to diplopia arising from reduced stereo overlaps and too large screen disparities. Two state-of-the-art solutions are reviewed. The first one suggests a dynamic change, via software, of the virtual camera convergence, whereas the second one suggests a matched hardware/software solution based on a series of predefined focus/vergence configurations. Potentialities and limits of both the solutions are outlined so as to provide the AR community, a yardstick for developing new stereoscopic video see-through systems suitable for different working distances.

Gathering, evaluating and managing customer feedback during aircraft production.

This paper proposes a systematic approach for gathering requirements during production through customers' remote access to the partially and fully assembled aircraft and its modules. The paper also proposes an evaluation and management of these recorded requirements and their utilization in the development of an aircraft. Modular product architecture is used for the modular organization of the product, product-service, and production system for the gathering, evaluation and management of feedback for product development perspectives. A mobile and wearable augmented reality system is used to virtually walk through the partially or fully manufactured product and to compare the status of the production with the product model to be produced. Change requests are captured as customer feedback. The knowledge thus acquired can be overlaid (augmented) on the real product, i.e. the aircraft. This approach is able to record the dynamic requirements of targeted customers. These changes can be carried out in the current version of the aircraft, and also incorporated into future versions. The implementation using case studies is presented for gathering feedback during assembly as well as for evaluating and managing the recorded feedback for exemplary modules (cabin and galley) of an aircraft. The use of the evaluation results in the development of an aircraft is also presented in the paper.

ウェアラブル拡張現実感のためのナビゲーション情報呈示に関する検討 : 天体観察支援システムを用いて( ウエアラブルVR)

ウェアラブル拡張現実感のためのナビゲーション情報呈示に関する検討 : 天体観察支援システムを用いて( ウエアラブルVR)

Experiencing touchless interaction with augmented content on wearable head-mounted displays in cultural heritage applications

The cultural heritage could benefit significantly from the integration of wearable augmented reality (AR). This technology has the potential to guide the user and provide her with both in-depth information, without distracting her from the context, and a natural interaction, which can further allow her to explore and navigate her way through a huge amount of cultural information. The integration of touchless interaction and augmented reality is particularly challenging. On the technical side, the human---machine interface has to be reliable so as to guide users across the real world, which is composed of cluttered backgrounds and severe changes in illumination conditions. On the user experience side, the interface has to provide precise interaction tools while minimizing the perceived task difficulty. In this study, an interactive wearable AR system to augment the environment with cultural information is described. To confer robustness to the interface, a strategy that takes advantage of both depth and color data to find the most reliable information on each single frame is introduced. Moreover, the results of an ISO 9241-9 user study performed in both indoor and outdoor conditions are presented and discussed. The experimental results show that, by using both depth and color data, the interface can behave consistently in different indoor and outdoor scenarios. Furthermore, the results show that the presence of a virtual pointer in the augmented visualization significantly reduces the users error rate in selection tasks.

A Prototyping Method to Simulate Wearable Augmented Reality Interaction in a Virtual Environment - A Pilot Study

Recently, we have seen an intensified development of head mounted displays (HMD). Some observers believe that the HMD form factor facilitates Augmented Reality (AR) technology, a technology that mixes virtual content with the users' view of the world around them. One of many interesting use cases that illustrate this is a smart home in which a user can interact with consumer electronic devices through a wearable AR system. Building prototypes of such wearable AR systems can be difficult and costly, since it involves a number of different devices and systems with varying technological readiness level. The ideal prototyping method for this should offer high fidelity at a relatively low cost and the ability to simulate a wide range of wearable AR use cases. This paper presents a proposed method, called IVAR (Immersive Virtual AR), for prototyping wearable AR interaction in a virtual environment (VE). IVAR was developed in an iterative design process that resulted in a testable setup in terms of hardware and software. Additionally, a basic pilot experiment was conducted to explore what it means to collect quantitative and qualitative data with the proposed prototyping method. The main contribution is that IVAR shows potential to become a useful wearable AR prototyping method, but that several challenges remain before meaningful data can be produced in controlled experiments. In particular, tracking technology needs to improve, both with regards to intrusiveness and precision.

Multi-cue hand detection and tracking for a head-mounted augmented reality system

With the recent developments in wearable augmented reality (AR), the role of natural human---computer interaction is becoming more important. Utilization of auxiliary hardware for interaction introduces extra complexity, weight and cost to wearable AR systems and natural means of interaction such as gestures are therefore more desirable. In this paper, we present a novel multi-cue hand detection and tracking method for head-mounted AR systems which combines depth, color, intensity and curvilinearity. The combination of different cues increases the detection rate, eliminates the background regions and therefore increases the tracking performance under challenging conditions. Detected hand positions and the trajectories are used to perform actions such as click, select, etc. Moreover, the 6 DOF poses of the hands are calculated by approximating the segmented regions with planes in order to render a planar menu (interface) around the hand and use the hand as a planar selection tool. The proposed system is tested on different scenarios (including markers for reference) and the results show that our system can detect and track the hands successfully in challenging conditions such as cluttered and dynamic environments and illumination variance. The proposed hand tracker outperforms other well-known hand trackers under these conditions.

Multimodal interaction with a wearable augmented reality system

An augmented reality system enhances a mobile user's situational awareness and provides new visualization functionality. The custom-built multimodal interface provides access to information encountered in urban environments. In this article, we detail our experiences with various input devices and modalities and discuss their advantages and drawbacks in the context of interaction tasks in mobile computing. We show how we integrated the input channels to use the modalities beneficially and how this enhances the interface's overall usability

Through-Walls Communication for Medical Emergency Services

The authors present a model for bringing the coordination power of workflow management systems to outdoor wearable augmented reality (AR) systems. They portray how mobile equipment may be integrated with adaptive, context-aware work environments. A scenario of a medical emergency task is described to illustrate the functionality of this form of collaboration system. Appropriate information stickers are introduced to support data collection in medical emergency scenarios in a sophisticated form through a hands-free user interface for medical personnel. They propose the use of new user interface technology, including multimedia, AR information stickers, and the allocation of patient medical records to identified locations of the human body. A key feature is the access to relevant information for users in the mobile environment as well as for those in the advanced control room. An additional advantage is the automatic recording of on-site data, which helps to build the medical record of a patient without interfering with the work of the emergency team.

ウェアラブル拡張現実システムのための利用者のビューペースト位置・方位取得手法( 複合現実感2)

ウェアラブル拡張現実システムのための利用者のビューペースト位置・方位取得手法( 複合現実感2)

HMI, The Window to the Manufacturing and Process Industry

The human-machine interface, or HMI for short, has become an important factor in the automation world over the past few years, and its importance never ceases to grow. The advance of IT technologies - especially the Internet - will continue to pick up speed and will determine what HMI systems look like in the future.HMI systems with open standards also permit automation functions that involve more than just operator control and monitoring, such as diagnostics or control tasks. Human-machine systems are increasingly becoming an information hub between the automation and corporate IT worlds, right up to MES (Manufacturing Execution System) or ERP (Enterprise Resource Planning) solutions.Moreover, speech recognition, virtual reality, and even wearable augmented reality systems will open up new possibilities for interaction between humans and technology.