Wearable Augmented Reality Research Articles

MobileSP: An FPGA-Based Real-Time Keypoint Extraction Hardware Accelerator for Mobile VSLAM

Keypoint extraction is a key technique for Visual Simultaneous Localization and Mapping (VSLAM). Recently, Convolutional Neural Network (CNN) has been used in the keypoint extraction for improving the accuracy. As one of the state-of-the-art CNN based keypoint extraction techniques, the SuperPoint ranked top in the CVPR2020 image matching challenge. However, the use of complex CNN makes it difficult to meet the real-time performance on a mobile platform with limited resource such as mobile robots and wearable Augmented Reality (AR) devices. In this work, based on the SuperPoint, we proposed an FPGA-based real-time keypoint extraction hardware accelerator through algorithm-hardware co-design for mobile VSLAM applications, which is named as MobileSP. Several algorithm and hardware level design techniques have been proposed to reduce the computation and improve the processing speed while maintaining high accuracy, including a partially shared detection & description encoding architecture, a pre-sorting based Non-Maximum Suppression (NMS) engine and a software-hardware hybrid pipeline computing technique. The design has been implemented and evaluated on a ZCU104 FPGA board. It achieves real-time performance of 42 fps with low Absolute Trajectory Error (ATE) of 1.82 cm simultaneously, outperforming several state-of-the-art designs.

Object Tracking in Augmented Reality: Enhancement Using Convolutional Neural Networks

Augmented reality (AR) has been used in maintenance, simulation, and remote assistance, among other applications. In AR systems, one of the significant issues is the placement of objects in augmented physical environments. Given the importance of object placement in AR systems, we proposed deep learning-based object placement, covering both object detection and object segmentation, to address relevant issues. Deep learning can help users complete tasks by providing the right information effectively, with the method taking into account dynamically changing environments and users’ situations in real time. The problem is that it is rarely used in AR, thereby prompting the combination of deep learning-based object detection and instance segmentation with wearable AR technology to improve the performance of complex tasks. This challenge was addressed in this work through the use of convolutional neural networks in the detection and segmentation of objects in actual environments. We measured the performance of AR technology on the basis of detection accuracy under environmental conditions of different intensities. Experimental results showed satisfactory segmentation and accurate detection

Actuating compact wearable augmented reality devices by multifunctional artificial muscle

An artificial muscle actuator resolves practical engineering problems in compact wearable devices, which are limited to conventional actuators such as electromagnetic actuators. ing the fundamental advantages of an artificial muscle actuator provides a small-scale, high-power actuating system with a sensing capability for developing varifocal augmented reality glasses and naturally fit haptic gloves. Here, we design a shape memory alloy-based lightweight and high-power artificial muscle actuator, the so-called compliant amplified shape memory alloy actuator. Despite its light weight (0.22 g), the actuator has a high power density of 1.7 kW/kg, an actuation strain of 300% under 80 g of external payload. We show how the actuator enables image depth control and an immersive tactile response in the form of augmented reality glasses and two-way communication haptic gloves whose thin form factor and high power density can hardly be achieved by conventional actuators.

Design and Evaluation of an Augmented Reality Head-mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

We provide an experimental evaluation of a wearable augmented reality (AR) system we have developed for human-robot teams working on tasks requiring collaboration in shared physical workspace. Recent advances in AR technology have facilitated the development of more intuitive user interfaces for many human-robot interaction applications. While it has been anticipated that AR can provide a more intuitive interface to robot assistants helping human workers in various manufacturing scenarios, existing studies in robotics have been largely limited to teleoperation and programming. Industry 5.0 envisions cooperation between human and robot working in teams. Indeed, there exist many industrial tasks that can benefit from human-robot collaboration. A prime example is high-value composite manufacturing. Working with our industry partner towards this example application, we evaluated our AR interface design for shared physical workspace collaboration in human-robot teams. We conducted a multi-dimensional analysis of our interface using established metrics. Results from our user study (n = 26) show that, subjectively, the AR interface feels more novel and a standard joystick interface feels more dependable to users. However, the AR interface was found to reduce physical demand and task completion time, while increasing robot utilization. Furthermore, user’s freedom of choice to collaborate with the robot may also affect the perceived usability of the system.

On-chip meta-optics for semi-transparent screen display in sync with AR projection

On-chip integrated meta-optics could enable high-performance, lightweight, and compact integrated photonic devices for augmented reality (AR). Despite previous endeavors in controlling guided waves for holographic phase control, such devices lack versatile performance with the full optical controllability in both amplitude and phase needed to generate multi-functional displays. Here, we propose and experimentally demonstrate an on-chip metasurface integrated on a waveguide to enable a multiplexing semi-transparent screen display in sync with an AR holographic display for human eyes. Through judicious engineering of on-chip meta-diatom displacement and interference at the nanoscale, we can locally modulate the on-chip optical scattering intensities to create a semi-transparent screen display. More intriguingly, we can project a simultaneous dual-channel AR holographic display by incorporating independent encoding freedom of its detour phase. Beyond the all-dielectric properties and high transparency, the projected AR images are free of zeroth-order diffraction interference due to the on-chip optical propagation scheme. We envision that the proposed on-chip meta-optic display could be readily applied to next-generation wearable AR displays, multiplexing optical displays, colored signage, information storage/encryption, and cosmetic visual applications.

Dynamic Augmented Reality Display by Layer‐Folded Metasurface via Electrical‐Driven Liquid Crystal

AbstractHeading toward the next‐generation versatile meta‐devices and meta‐systems with increasing complexity, it inevitably demands the cascading and coupling between multi‐layer metasurfaces to create new design space for meta‐optics. However, the accurate requirement for multilayer alignment at the nanoscale is significantly challenging due to the actual fabrication limitation, thus hindering the state‐of‐the‐art meta‐optics device integration and its practical application. Herein, a dynamic meta‐system for augmented reality (AR) holographic display functionality based on a layer‐folded metasurface integrated with an electrical‐driven liquid crystal (LC) platform is originally proposed and realized. By ingeniously folding the optical path between meta‐device vertical space, a beam‐steering metasurface in parallel with the other two holographic metasurfaces placed on a single surface is successfully cascaded. Such a layer‐folded scheme enjoys single‐time lithography processing for multi‐patterning with nanoscale alignment accuracy and avoids any vertical alignment issue. Furthermore, by combining with an electrical‐driven LC tuning scheme, the integrated meta‐system could actively switch independent‐encoded holographic images in real‐time floating in the actual‐world scene. Overall, it is envisioned that such a compact dynamic meta‐system suggests a feasible route for dynamic tunable wearable AR displays, intelligent dynamic displays, multi‐functional devices, etc.

Augmented Reality Enabled by On‐Chip Meta‐Holography Multiplexing

AbstractThe exploit of on‐chip metasurfaces with full optical controllability and multiplexing capability holds great promise for photonic integration circuits (PIC). Despite previous endeavors in metasurfaces controlling guided waves, it still faces critical challenges to realize practical display applications, for instance, augmented reality (AR). Here, a new type of AR based on multiplexed on‐chip meta‐holography is proposed and experimentally demonstrated by integrating judiciously engineered meta‐atoms above waveguide. Through hybridizing detour and Pancharatnam–Berry phases, on‐chip metasurfaces can independently manipulate both guided waves and free‐space light to enable a triple‐channel holography‐multiplexing with independent‐encoding freedom. Eventually, an RGB‐coloring AR hologram free from zero‐order diffraction that can project virtual images into a real‐world environment is realized, showcasing its significant potential for wearable devices (glasses or contact lens) integration. This study finds a feasible route toward multifunctional PIC devices and applications in wearable AR displays, imaging multiplexing, information storage, etc.

Designing Wearable Augmented Reality Concepts to Support Scalability in Autonomous Vehicle-Pedestrian Interaction

Wearable augmented reality (AR) offers new ways for supporting the interaction between autonomous vehicles (AVs) and pedestrians due to its ability to integrate timely and contextually relevant data into the user's field of view. This article presents novel wearable AR concepts that assist crossing pedestrians in multi-vehicle scenarios where several AVs frequent the road from both directions. Three concepts with different communication approaches for signaling responses from multiple AVs to a crossing request, as well as a conventional pedestrian push button, were simulated and tested within a virtual reality environment. The results showed that wearable AR is a promising way to reduce crossing pedestrians' cognitive load when the design offers both individual AV responses and a clear signal to cross. The willingness of pedestrians to adopt a wearable AR solution, however, is subject to different factors, including costs, data privacy, technical defects, liability risks, maintenance duties, and form factors. We further found that all participants favored sending a crossing request to AVs rather than waiting for the vehicles to detect their intentions—pointing to an important gap and opportunity in the current AV-pedestrian interaction literature.

Real-World Effectiveness of Wearable Augmented Reality Device for Patients With Hearing Loss: Prospective Study

BackgroundHearing loss limits communication and social activity, and hearing aids (HAs) are an efficient rehabilitative option for improving oral communication and speech comprehension, as well as the psychosocial comfort of people with hearing loss. To overcome this problem, over-the-counter amplification devices including personal sound amplification products and wearable augmented reality devices (WARDs) have been introduced.ObjectiveThis study aimed to evaluate the clinical effectiveness of WARDs for patients with mild to moderate hearing loss.MethodsA total of 40 patients (18 men and 22 women) with mild to moderate hearing loss were enrolled prospectively in this study. All participants were instructed to wear a WARD, Galaxy Buds Pro (Samsung Electronics), at least 4 hours a day for 2 weeks, for amplifying ambient sounds. Questionnaires including the Korean version of the abbreviated profile of hearing aid benefit (K-APHAB) and the Korean adaptation of the international outcome inventory for hearing aids (K-IOI-HA) were used to assess personal satisfaction in all participants. Audiologic tests, including sound field audiometry, sound field word recognition score (WRS), and the Korean version of hearing in noise test (K-HINT), were administered to 14 of 40 patients. The tests were performed under two conditions: unaided and aided with WARDs.ResultsThe mean age of the participants was 55.4 (SD 10.7) years. After 2 weeks of the field trial, participants demonstrated a benefit of WARDs on the K-APHAB. Scores of 3 subscales of ease of communication, reverberation, and background noise were improved significantly (P<.001). However, scores regarding aversiveness were worse under the aided condition (P<.001). K-IOI-HA findings indicated high user satisfaction after the 2-week field trial. On audiologic evaluation, the K-HINT did not show significant differences between unaided and aided conditions (P=.97). However, the hearing threshold on sound field audiometry (P=.001) and the WRS (P=.002) showed significant improvements under the aided condition.ConclusionsWARDs can be beneficial for patients with mild to moderate hearing loss as a cost-effective alternative to conventional hearing aids.

Intraoperative Navigation in Plastic Surgery with Augmented Reality: A Preclinical Validation Study.

Augmented reality allows users to visualize and interact with digital images including three-dimensional holograms in the real world. This technology may have value intraoperatively by improving surgical decision-making and precision but relies on the ability to accurately align a hologram to a patient. This study aims to quantify the accuracy with which a hologram of soft tissue can be aligned to a patient and used to guide intervention. A mannequin's face was marked in a standardized fashion with 14 incision patterns in red and nine reference points in blue. A three-dimensional photograph was then taken, converted into a hologram, and uploaded to HoloLens (Verto Studio LLC, San Diego, Calif.), a wearable augmented reality device. The red markings were then erased, leaving only the blue points. The hologram was then viewed through the HoloLens in augmented reality and aligned onto the mannequin. The user then traced the overlaid red markings present on the hologram. Three-dimensional photographs of the newly marked mannequin were then taken and compared with the baseline three-dimensional photographs of the mannequin for accuracy of the red markings. This process was repeated for 15 trials (n = 15). The accuracy of the augmented reality-guided intervention, when considering all trials, was 1.35 ± 0.24 mm. Markings that were positioned laterally on the face were significantly more difficult to reproduce than those centered around the facial midline. Holographic markings can be accurately translated onto a mannequin with an average error of less than 1.4 mm. These data support the notion that augmented reality navigation may be practical and reliable for clinical integration in plastic surgery.

Towards a More Socially Sustainable Advanced Pilot Training by Integrating Wearable Augmented Reality Devices

For flying all types of turbine-engine airplanes, a pilot must undergo an intense type rating (TR) course. This study investigated the learning conditions and TR course content, and specifies the most difficult course elements that could be tackled by augmented reality (AR) applications. Because women are underrepresented in the worldwide pilot population, it is important to address gender-specific preferences and needs in the development of AR-based wearable technologies for advanced pilot training. A gender-sensitive survey of the learning conditions and course contents was conducted with 31 pilots and 22 instructors. Despite many similarities, the results confirm that there are gender-specific needs and preferences for the development of future AR-based applications for TR training. In addition, the views of both pilots’ and instructors’ are required to obtain a comprehensive assessment of the learning contents and conditions related to TR. The results also show that time pressure increased the perceived difficulty of the course for some trainees. These results are important because they indicate the directions to be taken in developing future AR-based training applications for a more learner-centered and inclusive TR training. Future directions to foster a socially sustainable development of AR-based training means for TR with special focus on gender diversity are presented.

DARLENE - Improving situational awareness of European law enforcement agents through a combination of augmented reality and artificial intelligence solutions.

Background: Augmented reality (AR) and artificial intelligence (AI) are highly disruptive technologies that have revolutionised practices in a wide range of domains, including the security sector. Several law enforcement agencies (LEAs) employ AI in their daily operations for forensics and surveillance. AR is also gaining traction in security, particularly with the advent of affordable wearable devices. Equipping police officers with the tools to facilitate an elevated situational awareness (SA) in patrolling and tactical scenarios is expected to improve LEAs' safety and capacity to deliver crucial blows against terrorist and/or criminal threats. Methods: In this paper we present DARLENE, an ecosystem incorporating novel AI techniques for activity recognition and pose estimation tasks, combined with a wearable AR framework for visualization of the inferenced results via dynamic content adaptation according to the wearer's stress level and operational context. The concept has been validated with end-users through co-creation workshops, while the decision-making mechanism for enhancing LEAs' SA has been assessed with experts. Regarding computer vision components, preliminary tests of the instance segmentation method for humans' and objects' detection have been conducted on a subset of videos from the RWF-2000 dataset for violence detection, which have also been used to test a human pose estimation method that has so far exhibited impressive results, constituting the basis of further developments in DARLENE. Results: Evaluation results highlight that target users are positive towards the adoption of the proposed solution in field operations, and that the SA decision-making mechanism produces highly acceptable outcomes. Evaluation of the computer vision components yielded promising results and identified opportunities for improvement. Conclusions: This work provides the context of the DARLENE ecosystem and presents the DARLENE architecture, analyses its individual technologies, and demonstrates preliminary results, which are positive both in terms of technological achievements and user acceptance of the proposed solution.

On the suitability of current augmented reality head-mounted devices

Simulation is a recognized and much-appreciated tool in healthcare and education. Advances in simulation have led to the burgeoning of various technologies. In recent years, one such technological advancement has been Augmented Reality (AR). Augmented Reality simulations have been implemented in healthcare on various fronts with the help of a plethora of devices including cellphones, tablets, and wearable AR headsets. AR headsets offer the most immersive experience of the AR simulation as they are head-mounted and offer a stereoscopic view of the superimposed 3D models through the attached goggles overlaid on real-world surfaces. To this effect, it is important to understand the performance capabilities of the AR headsets based on workload. In this paper, our objective is to compare the performances of two prominent AR headsets of today, the Microsoft Hololens and the Magic Leap One. We use surgical AR software that allows the surgeons to show internal structures, such as the rib cage, to assist in the surgery as a reference application to obtain performance numbers for those AR devices. Based on our research, there are no performance measurements and recommendations available for these types of devices in general yet.

Metaverse Employee Socialization and Operations Management, Mobile Biometric and Sentiment Data, and Wearable Augmented Reality Devices in Immersive Workspaces

Metaverse Employee Socialization and Operations Management, Mobile Biometric and Sentiment Data, and Wearable Augmented Reality Devices in Immersive Workspaces

Virtual Employee Engagement and Immersive Workspaces, Wearable Augmented Reality Technology, and Spatial Data Mining and Simulation Modeling Algorithms in the Metaverse Interactive Environment

Virtual Employee Engagement and Immersive Workspaces, Wearable Augmented Reality Technology, and Spatial Data Mining and Simulation Modeling Algorithms in the Metaverse Interactive Environment

A comparative evaluation of the wearable augmented reality-based data presentation interface and traditional methods for data entry tasks

Data entry is a ubiquitous task performed in today's offices. Persistent data entry is linked with high workload and fatigue due to poor ergonomic workplace design and poor posture. This study aims to alleviate data entry operators' workload and improve data entry performance by applying wearable augmented reality (AR) technology to data entry tasks. An AR-based interface was developed and used to present data to the participants, who entered the data in web-based data entry forms. A total of eighteen participants performed data entry tasks to evaluate the AR interface with traditional methods – extra desktop monitor and paper-based data presentation methods. Each method's performance was judged on the task completion time, typing error rate, workload, and usability. The usability and overall perceived workload while using an AR interface for data entry were similar to the traditional way of using paper, despite the additional burden due to the weight of the AR headset. AR interface did not perform better than the extra desktop monitor interface for usability and overall perceived workload. The results from this study can be utilized to design AR devices that are suited for data entry tasks.

Augmented Reality-Assisted Craniotomy for Parasagittal and Convexity En Plaque Meningiomas and Custom-Made Cranio-Plasty: A Preliminary Laboratory Report.

Background: This report discusses the utility of a wearable augmented reality platform in neurosurgery for parasagittal and convexity en plaque meningiomas with bone flap removal and custom-made cranioplasty. Methods: A real patient with en plaque cranial vault meningioma with diffuse and extensive dural involvement, extracranial extension into the calvarium, and homogeneous contrast enhancement on gadolinium-enhanced T1-weighted MRI, was selected for this case study. A patient-specific manikin was designed starting with the segmentation of the patient’s preoperative MRI images to simulate a craniotomy procedure. Surgical planning was performed according to the segmented anatomy, and customized bone flaps were designed accordingly. During the surgical simulation stage, the VOSTARS head-mounted display was used to accurately display the planned craniotomy trajectory over the manikin skull. The precision of the craniotomy was assessed based on the evaluation of previously prepared custom-made bone flaps. Results: A bone flap with a radius 0.5 mm smaller than the radius of an ideal craniotomy fitted perfectly over the performed craniotomy, demonstrating an error of less than ±1 mm in the task execution. The results of this laboratory-based experiment suggest that the proposed augmented reality platform helps in simulating convexity en plaque meningioma resection and custom-made cranioplasty, as carefully planned in the preoperative phase. Conclusions: Augmented reality head-mounted displays have the potential to be a useful adjunct in tumor surgical resection, cranial vault lesion craniotomy and also skull base surgery, but more study with large series is needed.

Augmented reality flavor: cross-modal mapping across gustation, olfaction, and vision

Gustatory display research is still in its infancy despite being one of the essential everyday senses that human practice while eating and drinking. Indeed, the most important and frequent tasks that our brain deals with every day are foraging and feeding. The recent studies by psychologists and cognitive neuroscientist revealed how complex multisensory rely on the integration of cues from all the human senses in any flavor experiences. The perception of flavor is multisensory and involves combinations of gustatory and olfactory stimuli. The cross-modal mapping between these modalities needs to be more explored in the virtual environment and simulation, especially in liquid food. In this paper, we present a customized wearable Augmented Reality (AR) system and olfaction display to study the effect of vision and olfaction on the gustatory sense. A user experiment and extensive analysis conducted to study the influence of each stimulus on the overall flavor, including other factors like age, previous experience in Virtual Reality (VR)/AR, and beverage consumption. The result showed that smell contributes strongly to the flavor with less contribution to the vision. However, the combination of these stimuli can deliver richer experience and a higher belief rate. Beverage consumption had a significant effect on the flavor belief rate. Experience is correlated with stimulus and age is correlated with belief rate, and both indirectly affected the belief rate.

Dielectric Metalens: Properties and Three-Dimensional Imaging Applications

Recently, optical dielectric metasurfaces, ultrathin optical skins with densely arranged dielectric nanoantennas, have arisen as next-generation technologies with merits for miniaturization and functional improvement of conventional optical components. In particular, dielectric metalenses capable of optical focusing and imaging have attracted enormous attention from academic and industrial communities of optics. They can offer cutting-edge lensing functions owing to arbitrary wavefront encoding, polarization tunability, high efficiency, large diffraction angle, strong dispersion, and novel ultracompact integration methods. Based on the properties, dielectric metalenses have been applied to numerous three-dimensional imaging applications including wearable augmented or virtual reality displays with depth information, and optical sensing of three-dimensional position of object and various light properties. In this paper, we introduce the properties of optical dielectric metalenses, and review the working principles and recent advances in three-dimensional imaging applications based on them. The authors envision that the dielectric metalens and metasurface technologies could make breakthroughs for a wide range of compact optical systems for three-dimensional display and sensing.

32‐1: RGBW OLED Microdisplays for Wearable Augmented Reality

In order to meet the demand of high luminance and low power consumption for wearable AR (Augmented Reality) applications, two types of 0.71‐inch RGBW OLED micro‐displays were developed, reaching luminance of more than 5000 cd/m2 with resolution of FHD (1920 × 1080). RGBW mapping and sub‐pixel rendering algorithms were proposed to achieve good performance.