Augmented Reality Registration Research Articles

Study of Various Computer Vision Algorithms for Registration in Augmented Reality: A Survey

Abstract: Augmented reality is defined as the fusion of virtual object in real world. Registration is basic step for creating an immersive AR experience to the user. The registration algorithms helps in placing the virtually created object in the most accurate position in the real world. This paper shows an in-depth comparative analysis of various registration algorithm under computer vision that are used for AR applications. The comparative analysis helps in identifying the strengths, weaknesses and the applicability of each algorithm in various AR fields by studying different case studies and real-world applications which helps in showing the practical implication of these algorithms. Also, the challenges associated with AR are discussed and future scope of this aims at advancement of the registration algorithms so as to make the AR experience more functionable and smooth. This survey paper can help researchers, developers who have interest in knowing or exploring the field of computer vision algorithms for solving the registration problem in Augmented Reality.

Feasibility and accuracy of a real-time depth-based markerless navigation method for hologram-guided surgery

BackgroundTwo-dimensional (2D) medical visualization techniques are often insufficient for displaying complex, three-dimensional (3D) anatomical structures. Moreover, the visualization of medical data on a 2D screen during surgery is undesirable, because it requires a surgeon to continuously switch focus. This switching focus problem also results in extensive use of intraoperative radiation to gain additional insights for a 3D configuration. The use of augmented reality (AR) has the potential to overcome these problems, for instance by using markers on target points that are aligned with the AR solution. However, placing markers for a precise virtual overlay are time-costly, always have to be visible within the field of view and disrupt the surgical workflow. In this study, we developed ARCUS, a depth-based, markerless AR navigation system, which overlays 3D virtual elements onto target body parts to overcome the limitations of 2D medical visualization techniques.Methods and resultsIn a phantom study, our markerless ARCUS system was evaluated for accuracy and precision by comparing it to a Quick Response (QR) code-based AR registration method. The evaluation involved measuring the Euclidean distance between target points on a 3D-printed face and their corresponding points on the virtual overlay using a robotic arm for precise measurements. Correlations between the measuring points provided by our markerless system and the actual measuring points on the 3D-print were high, with promising consistent Euclidean distances between the 3D points and the virtual points generated by both our markerless system and the Vuforia QR Code system. We also show two clinical examples of ex vivo case studies on cadaveric human specimens where our markerless ARCUS system could be applicable to.ConclusionThe markerless AR navigation system holds strong potential as a 3D visualization method in clinical settings. While both ARCUS and the Vuforia QR code-based method fell short of meeting the surgical threshold of a 2 mm offset, our markerless system demonstrated promising features such as instant registration, markerless operation, and potential compatibility with non-rigid structures. Its automated virtual overlay onto target body parts offers significant advantages, paving the way for investigations into future clinical use.

Deep learning-based augmented reality work instruction assistance system for complex manual assembly

The manual assembly process of complex products is lengthy, the assembly requirements are difficult to recall, and the assembly quality requirements are high. The separation of the operation guidance information from the real physical object in the traditional paper manual easily distracts the operator, increasing their cognitive burden. To address this issue, we integrate augmented reality (AR) and artificial intelligence (AI) technologies for manual assembly assistance. A novel encoding and decoding convolutional neural network (CNN) is built to realize accurate AR registration under mark-less assembly environment. An assembly quality inspection method integrating a neural network and virtual model matching was proposed for AR systems. The specific process of the proposed method includes two stages: offline and online. In the first stage, a monocular RGB image dataset was built for assembling object keypoints and assembly object detection. CNN models were designed and trained for deployment in an AR-assisted assembly system. Second, the deployed CNN models were used to perform AR registration and assembly quality inspection. Experimental results demonstrate that the proposed method can accurately present AR work instruction guidance and assembly quality inspection for manual assembly. The proposed AR registration method can achieve a frame rate of 25FPS, which satisfies the timeliness requirements of the AR system. The accuracy rate of the proposed assembly quality inspection method for the MONA component assembly reached 92.5%.

Local geometric edge features based registration for textureless object in augmented reality assisted assembly

Image-based methods have been widely used in augmented reality (AR) assistant assembly systems. However, due to the lack of sufficient texture information on the surface of assembly part, traditional image feature matching methods still face challenges. This paper proposes a coarse-to-fine AR registration method for textureless assembly part. In the first stage, a new feature matching method which is called line neighborhood edge descriptor (LNED) is presented to find the coarse camera pose from textureless image. The LNED take the contour line of assembly part as the description object, and use local geometric edge of assembly part to describe the contour line. During the image matching, the binary encoding is used to reduce the computational consumption for LNED. In the second stage, spatial points in the CAD model of assembly part are reverse projected to the textureless image based on the coarse camera pose. And the bundle adjustment method based on the edge distance of the textureless image is adopted to iteratively calculate the precise camera pose. In the experimental evaluation, the proposed registration method shows high accuracy and fast speed in comparison with conventional registration methods, which demonstrates that our method can effectively solve the problem of AR registration for textureless assembly part.

Application of augmented reality technology in visual design of cultural product packaging

Abstract Integrating intelligent terminal devices into product packaging to amplify brand influence represents a significant trend in contemporary packaging design research. This study introduces an augmented reality (AR) cultural product packaging design model predicated on the principles of AR packaging visual information interactivity and content interaction. This model facilitates a precise comprehension of shifts in user preferences for cultural product packaging. It primarily employs computer vision technology for AR tracking and registration, ensuring accurate overlay of virtual objects onto real-world spaces. Within the scope of this model, user experience metrics—ease of use, interactivity, and satisfaction—were quantitatively assessed, yielding scores of 14.523, 13.49, and 13.918, respectively, with variance values spanning from 0.031 to 0.1418. Notably, users rated the packaging design creativity of cultural product A, which utilizes this model, highest at 6.66. This demonstrates that the AR cultural product packaging design model delivers a stable and impactful user experience. Consequently, this research provides a robust and productive framework for advancing intelligent and interactive product packaging designs.

Application of augmented reality using automatic markerless registration for facial plastic and reconstructive surgery

This study aimed to present a novel markerless augmented reality (AR) system using automatic registration based on machine-learning algorithms that visualize the facial region and provide an intraoperative guide for facial plastic and reconstructive surgeries.This study prospectively enrolled 20 patients scheduled for facial plastic and reconstructive surgeries. The AR system visualizes computed tomographic data in three-dimensional (3D) space by aligning with the point clouds captured by a 3D camera. Point cloud registration consists of two stages: the preliminary registration gives an initial estimate of the transformation using landmark detection, followed by the precise registration using Iterative Closest Point algorithms. Computed Tomography (CT) data can be visualized as two-dimensional slice images or 3D images by the AR system. The AR registration error was defined as the cloud-to-cloud distance between the surface data obtained from the CT and 3D camera. The error was calculated in each facial territory, including the upper, middle, and lower face, while patients were awake and orally intubated, respectively.The mean registration errors were 1.490 ± 0.384 mm and 1.948 ± 0.638 mm while patients were awake and orally intubated, respectively. There was a significant difference in the errors in the lower face between patients while they were awake (1.502 ± 0.480 mm) and orally intubated (2.325 ± 0.971 mm) when stratified by facial territories (p = 0.006).The markerless AR can accurately visualize the facial region with a mean overall registration error of 1–2 mm, with a slight increase in the lower face due to errors arising from tube intubation.

Augmented Reality Registration System for Visualization of Skull Landmarks

BackgroundAugmented reality (AR) is an emerging technology in neurosurgery with the potential to become a strategic tool in the delivery of care and education for trainees. Advances in the technology have demonstrated promising use for improving visualization and spatial awareness of critical neuroanatomical structures. In this report, we employ a novel AR registration system for the visualization and targeting of skull landmarks. MethodsA markerless AR system was used to register 3D reconstructions of suture lines onto the head via a head mounted display. Participants were required to identify craniometric points with and without AR assistance. Targeting error was measured as the Euclidian distance between the user-defined location and the true craniometric point on the subjects’ head. ResultsAll participants successfully registered 3D reconstructions onto the subjects’ head. Targeting accuracy was significantly improved with AR (3.59 ± 1.29 mm) . Across all target points, AR increased accuracy by an average of 19.96 ± 3.80 mm. Post-test surveys revealed that participants felt the technology increased their confidence in identifying landmarks (4.6/5) and that the technology will be useful for clinical care (4.2/5). ConclusionsWhile several areas of improvement and innovation can further enhance the use of AR in neurosurgery, this report demonstrates the feasibility of a markerless headset-based AR system for visualizing craniometric points on the skull. As the technology continues to advance, AR is expected to play an increasingly significant role in neurosurgery, transforming how surgeries are performed and improving patient care.

SURG-37. NEURO-ONCOLOGICAL AUGMENTED REALITY: A MORE INTUITIVE APPROACH TO RESECTION PLANNING

Abstract BACKGROUND When preparing for the resection of an intracranial lesion, its borders and optimal approach are often determined using neuronavigation with a tracked pointer. This can sometimes prove challenging, especially for deep-seated lesions. Augmented reality (AR) can simplify and improve this step by directly displaying the lesion on the patient's skin. METHODS A proprietary inside-out infrared tracking solution was developed, allowing for heads-up displaying AR scenes on the Microsoft HoloLens II without the need for an external tracking camera or computer. We included twenty patients with an intracerebral lesion planned for resection. After semi-automatic hologram-to-patient registration, different participants marked the lesion outlines on the patient’s skin, consecutively aided by the Brainlab neuronavigation system and the HoloLens. Each registration on both systems provided a registration transform that was compared for accuracy and consistency. Participant performance was quantified in terms of duration and accuracy for both patient registration and lesion delineation, and compared to expert performance. RESULTS When using AR both registration and delineation were significantly faster than with conventional neuronavigation (p = 0.02 and p < 0.001, respectively, and p < 0.001 for the total duration), taking 79.23 ± 17.48 and 39.58 ± 39.10 seconds while neuronavigation required 96.61 ± 24.54 and 90.80 ± 44.09 seconds. AR had a registration offset of 3.3mm and 3.4°, and was more consistent compared to neuronavigation. AR facilitated more accurate and detailed lesion delineation, while neuronavigation often overestimated lesion size. CONCLUSION Augmented reality provides a faster and more accurate alternative for resection planning. Lesion delineation was more intuitive while retaining high accuracy. Future research should focus on further intraoperative implementations.

VSLAM method based on object detection in dynamic environments.

Augmented Reality Registration field now requires improved SLAM systems to adapt to more complex and highly dynamic environments. The commonly used VSLAM algorithm has problems such as excessive pose estimation errors and easy loss of camera tracking in dynamic scenes. To solve these problems, we propose a real-time tracking and mapping method based on GMM combined with YOLOv3. The method utilizes the ORB-SLAM2 system framework and improves its tracking thread. It combines the affine transformation matrix to correct the front and back frames, and employs GMM to model the background image and segment the foreground dynamic region. Then, the obtained dynamic region is sent to the YOLO detector to find the possible dynamic target. It uses the improved Kalman filter algorithm to predict and track the detected dynamic objects in the tracking stage. Before building a map, the method filters the feature points detected in the current frame and eliminates dynamic feature points. Finally, we validate the proposed method using the TUM dataset and conduct real-time Augmented Reality Registration experiments in a dynamic environment. The results show that the method proposed in this paper is more robust under dynamic datasets and can register virtual objects stably and in real time.

A Quick Development Toolkit for Augmented Reality Visualization (QDARV) of a Factory

Augmented reality (AR) technology has been applied to the industrial field; however, at present, most industrial AR applications are developed for specific application scenarios, which are difficult to develop and have a long development cycle, and lack universality. To this end, this paper proposed a quick development toolkit for augmented reality visualization (QDARV) of a factory by using the script configuration and parsing approach. With QDARV, designers can quickly develop industrial AR applications, achieve AR registration based on quick response (QR) code and simultaneous localization and mapping (SLAM), and superpose information such as disassembly animations, industrial instruments, pictures, and texts on real scenes. In QDARV, an AR registration method based on SLAM map reloading is proposed. Through saving and reloading the map and the configuration and analysis of the SLAM-AR display content script, the AR scene configured by the designer is displayed. Meanwhile, the objects detected using YOLOv5 are used as a landmark to assist the SLAM system in positioning. The QDARV can be applied to realize AR visualization of factories with a large area.

Multicenter assessment of augmented reality registration methods for image-guided interventions.

To evaluate manual and automatic registration times and registration accuracies on HoloLens 2 for aligning a 3D CT phantom model onto a CT grid, a crucial step for intuitive 3D navigation during CT-guided interventions; to compare registration times between HoloLens 1 and 2. Eighteen participants in various stages of clinical training across two academic centers performed registration of a 3D CT phantom model onto a CT grid using HoloLens 2. Registration times and accuracies were compared among different registration methods, clinical experience levels, and consecutive attempts. Registration times were also compared retrospectively to prior HoloLens 1 results. Mean aggregate manual registration times were 27.7s, 24.3s, and 72.8s for one-handed gesture, two-handed gesture, and Xbox controller, respectively; mean automatic registration time was 5.3s (ANOVA p < 0.0001). No significant difference in registration times was found among attendings, residents and fellows, and medical students (p > 0.05). Significant improvements in registration times were detected across consecutive attempts using hand gestures (p < 0.01). Compared to prior HoloLens 1 data, hand gesture registration was 81.7% faster with HoloLens 2 (p < 0.05). Registration accuracies were not significantly different across manual registration methods, measuring at 5.9mm, 9.5mm, and 8.6mm with one-handed gesture, two-handed gesture, and Xbox controller, respectively (p > 0.05). Manual registration times decreased significantly on HoloLens 2, approaching those of automatic registration and outperforming Xbox controller registration. Fast, adaptive, and accurate registration of holographic models of cross-sectional imaging is paramount for the implementation of augmented reality-assisted 3D navigation during CT-guided interventions.

Microscope-Based Augmented Reality with Intraoperative Computed Tomography-Based Navigation for Resection of Skull Base Meningiomas in Consecutive Series of 39 Patients.

Simple SummaryThe aim of surgery for skull base meningiomas is maximal resection with minimal damage to the involved cranial nerves and cerebral vessels. Compared to non-skull base meningiomas, these lesions show a reduced rate of gross total resection (GTR). Therefore, the use of technologies for improved orientation in the surgical field, such as neuronavigation and augmented reality (AR), is of interest. We confirmed in a consecutive series of 39 patients who underwent surgery for skull base meningiomas that automatic registration with intraoperative computed tomography (iCT) showed high registration accuracy and that microscope-based AR largely facilitated the resection by increasing surgical precision and providing improved intraoperative orientation by visualizing the tumor and the critical neurovascular structures in the operative microscope. No injuries to critical neurovascular structures occurred. There were 26 patients (66.6%) who underwent GTR. Additionally, 33 out of 35 patients who lived to follow-up could ambulate.Background: The aim of surgery for skull base meningiomas is maximal resection with minimal damage to the involved cranial nerves and cerebral vessels; thus, implementation of technologies for improved orientation in the surgical field, such as neuronavigation and augmented reality (AR), is of interest. Methods: Included in the study were 39 consecutive patients (13 male, 26 female, mean age 64.08 ± 13.5 years) who underwent surgery for skull base meningiomas using microscope-based AR and automatic patient registration using intraoperative computed tomography (iCT). Results: Most common were olfactory meningiomas (6), cavernous sinus (6) and clinoidal (6) meningiomas, meningiomas of the medial (5) and lateral (5) sphenoid wing and meningiomas of the sphenoidal plane (5), followed by suprasellar (4), falcine (1) and middle fossa (1) meningiomas. There were 26 patients (66.6%) who underwent gross total resection (GTR) of the meningioma. Automatic registration applying iCT resulted in high accuracy (target registration error, 0.82 ± 0.37 mm). The effective radiation dose of the registration iCT scans was 0.58 ± 1.05 mSv. AR facilitated orientation in the resection of skull base meningiomas with encasement of cerebral vessels and compression of the optic chiasm, as well as in reoperations, increasing surgeon comfort. No injuries to critical neurovascular structures occurred. Out of 35 patients who lived to follow-up, 33 could ambulate at their last presentation. Conclusion: A microscope-based AR facilitates surgical orientation for resection of skull base meningiomas. Registration accuracy is very high using automatic registration with intraoperative imaging.

Pose Estimation and Non-Rigid Registration for Augmented Reality During Neurosurgery.

A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a non-rigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.

Augmented reality registration algorithm based on T-AKAZE features

Three-dimensional (3D) registration plays a pivotal step in augmented reality (AR) systems. Traditional 3D registration methods have the disadvantages of poor accuracy and robustness. This paper proposes a novel registration method, we believe, for AR systems based on the AKAZE and Tanimoto similarity measurement method. In this paper, the image feature points are extracted and matched by combining the AKAZE algorithm and the Tanimoto similarity measurement method. Then, the camera pose is estimated by calculating the constraint relationship of the feature points. Finally, the 3D registration and real-time tracking of the virtual objects are realized by the Lucas-Kanade (LK) optical flow tracking algorithm. We use Tanimoto to determine the similarity of feature points to improve the matching accuracy of the AKAZE algorithm, and this method not only retains the advantages of strong scale adaptation but also has the advantage of high-precision matching. Experiments show that the method proposed in this paper has the benefits of high registration accuracy, low time consumption, and strong robustness. Under the premise of ensuring accuracy, when the marker is rotated or blocked, it can be accurately registered. In addition, when the external environment changes, for example, the light intensity or the size of the parallax, the registration can still be stable.

Contextualizing benefits and limitations reported for augmented reality in construction research

Augmented Reality (AR) has been in use for years, and is increasingly being adopted in the construction industry. Research has explored various applications of AR in construction to identify its benefits and drawbacks. To synthesize these findings, prior works have conducted literature reviews about AR in construction, but these papers generally focus on identifying attributes for which there is consensus about AR benefits or drawbacks. While this is important, it highlights the need to identify trends in AR literature for which there is divergence in reporting to better understand the underlying contextual factors that may impact the success of AR implementation. This paper investigates trends in AR literature by studying both benefits and limitations reported when using current generation AR devices for construction applications. This is done by reviewing 49 articles from 2013 to 2020 found through a key word search for “augmented reality”, “mixed reality” and “construction”. The papers are categorized according to the most common attributes reported. The benefits and limitations identified are analysed based on how the technology was used. For several AR attributes, including ease of implementation, learning/training time, field of view, hardware and software performance, occlusion and immersiveness, there was consensus in reporting among researchers. For other AR attributes, including cost, cognitive performance, development time, tracking and registration, attitude towards the technology, and efficiency, there was divergence in reports among researchers. For each of these divergent attributes, the authors provide a discussion related to the contextual factors that were present. Beyond illustrating that some attributes do not lead to agreement between researchers, the results also indicate contextual factors that may contribute to the difference in reports. These findings contribute to the literature by enabling researchers to include or exclude contextual factors to gain or mitigate previously reported benefits or challenges when using AR in construction.

Abstract No. 486 Augmented reality registration of three-dimensional models: a multicenter assessment using the next-generation HoloLens 2

Comparison of registration methods on HoloLens 2 for aligning 3D holographic models onto a CT grid amongst two academic centers to assess efficacy and reproducibility. A 3D holographic model of an abdominal CT phantom (CIRS 071B) with an overlying CT grid (Beekley Medical 117) was segmented using ITK-SNAP and projected using Microsoft HoloLens 2. Custom code was developed in Unity and C# for model registration using four methods: 1) one-handed gestures, 2) optional two-handed gestures, 3) Xbox controller, and 4) computer vision with automated image detection using Vuforia SDK. Code was shared and deployed across institutions. Registration times for aligning the model to the CT grid were recorded until visual satisfaction and alignment errors were < 1 cm in the x-, y-, and z-planes. Eighteen users from two academic centers (3 attendings, 3 trainees, and 3 medical students at each center) attempted each registration method using HoloLens 2 three times. Comparisons were made with paired t-tests within each center, F-tests, and unpaired t-tests between centers. Center 1 had mean manual registration times of 22.6s, 25.2s, and 77.2s, and Center 2 had times of 32.7s, 23.4s, 68.5s for one-handed gestures, two-handed gestures, and Xbox controller, respectively. Both centers showed faster registration times using one- and two-handed hand gestures compared to controller (P < 0.001 at both centers). Between centers, there was no significant difference in registration times for two-handed gestures and controller, but one-handed gestures were faster at Center 1 compared to Center 2 by 31% (P < 0.05). Both centers had fastest registration times with automated computer vision. Center 2’s mean automatic registration time was 3.4s versus Center 1’s time of 7.2s (52% reduction, P < 0.001). Registration times demonstrated similar trends within each center but showed variability in certain registration methods between centers. Prior exposure with the HoloLens may impact hologram interaction and ambient lighting conditions may affect computer vision detection. These findings encourage cross-institutional development and reinforce multicenter evaluation for new technologies, like augmented reality, in the IR suite.

Augmented reality-based robot teleoperation system using RGB-D imaging and attitude teaching device

Augmented reality (AR)-based programming using the demonstration method has been widely studied. However, studies on AR-based programming for remote robots are lacking because of the limitation of human–computer interaction. This paper proposes an AR-based robot teleoperation system and method using RGB-D imaging and an attitude teaching device. By sending the color and depth images of the remote robot environment to the local side, the operators can complete the teleoperation of the robot at the local side. First, the operators select key positions on the motion path of the robot endpoint from color images via a mouse, and the computer calculates the 3D coordinates of these key points in the robot base coordinate system to complete the position teaching process. In the robot attitude teaching process, the AR technology is used to superimpose the virtual robot model onto the color images of the robot teleoperation environment, so as to make the virtual robot endpoint to move along the teaching path. An operator can use the portable attitude teaching device designed in this study to control the robot movement parameters, such as the attitude and motion speed, during the movement of the virtual robot. After the position and attitude teaching processes, the robot movement trajectory can be generated. To make the base coordinate system of the virtual model consistent with that of the physical robot, we propose an online AR registration method, which does not require manually placing the AR registration marker. The proposed AR-based robot teleoperation system can quickly and easily complete robot teleoperation at the local side.

Laparoscopic augmented reality registration for oncological resection site repair

PurposeResection site repair during laparoscopic oncological surgery (e.g. laparoscopic partial nephrectomy) poses some unique challenges and opportunities for augmented reality (AR) navigation support. This work introduces an AR registration workflow that addresses the time pressure that is present during resection site repair.MethodsWe propose a two-step registration process: the AR content is registered as accurately as possible prior to the tumour resection (the primary registration). This accurate registration is used to apply artificial fiducials to the physical organ and the virtual model. After the resection, these fiducials can be used for rapid re-registration (the secondary registration). We tested this pipeline in a simulated-use study with N=18 participants. We compared the registration accuracy and speed for our method and for landmark-based registration as a reference.ResultsAcquisition of and, thereby, registration with the artificial fiducials were significantly faster than the initial use of anatomical landmarks. Our method also had a trend to be more accurate in cases in which the primary registration was successful. The accuracy loss between the elaborate primary registration and the rapid secondary registration could be quantified with a mean target registration error increase of 2.35 mm.ConclusionThis work introduces a registration pipeline for AR navigation support during laparoscopic resection site repair and provides a successful proof-of-concept evaluation thereof. Our results indicate that the concept is better suited than landmark-based registration during this phase, but further work is required to demonstrate clinical suitability and applicability.

Early Feasibility Studies of Augmented Reality Navigation for Lateral Skull Base Surgery.

An augmented reality (AR) platform using a head mounted display (HMD) can successfully display and register virtual computerized tomography (CT) data to corresponding real world anatomy using fiducial markers. AR is a growing field in surgical planning, and has been used by this group to aide in orthopedic surgery applications. Intraoperative AR registration of preoperative CT images could improve identification of critical anatomical structures during lateral skull base surgery. An AR based HMD platform was created to allow for virtual display and real time manipulation of a temporal bone phantom model. Four fiducial markers were incorporated into this model and used to allow manual alignment of surface markers in real-time. To test the accuracy of this platform virtual lines were created in the AR environment running through the visualized real-world fiducial marker points. Target registration error (TRE) was calculated by measuring the orthogonal distance between these virtual lines and the corresponding virtual fiducial marker for each of the four markers from varying angles. Ten consecutive experiments were performed. The AR based platform successfully displayed CT renderings in the AR environment and allowed real time translation and rotation for manual alignment. A TRE of 10.62 ± 5.90 mm was found. Our results suggest that AR visualization of CT imaging can be registered to patient surface landmarks, but current limitations of AR hardware resulted in TREs too large to be used in the lateral skull base. Future advancements in AR technology will hopefully allow for reduction of registration error.

Deep learning-based smart task assistance in wearable augmented reality

• Combining deep learning with wearable AR for innovative and user-centered task assistance. • Reconstructing 2.5D and 3D replicas without pre-defined models for effective 3D annotation and sharing. • Applying instance segmentation and 3D spatial mapping in wearable AR for more intuitive task guidance. • Conducting experimental analyses through realistic manufacturing tasks for proving usability and usefulness. Wearable augmented reality (AR) smart glasses have been utilized in various applications such as training, maintenance, and collaboration. However, most previous research on wearable AR technology did not effectively supported situation-aware task assistance because of AR marker-based static visualization and registration. In this study, a smart and user-centric task assistance method is proposed, which combines deep learning-based object detection and instance segmentation with wearable AR technology to provide more effective visual guidance with less cognitive load. In particular, instance segmentation using the Mask R-CNN and markerless AR are combined to overlay the 3D spatial mapping of an actual object onto its surrounding real environment. In addition, 3D spatial information with instance segmentation is used to provide 3D task guidance and navigation, which helps the user to more easily identify and understand physical objects while moving around in the physical environment. Furthermore, 2.5D or 3D replicas support the 3D annotation and collaboration between different workers without predefined 3D models. Therefore, the user can perform more realistic manufacturing tasks in dynamic environments. To verify the usability and usefulness of the proposed method, we performed quantitative and qualitative analyses by conducting two user studies: 1) matching a virtual object to a real object in a real environment, and 2) performing a realistic task, that is, the maintenance and inspection of a 3D printer. We also implemented several viable applications supporting task assistance using the proposed deep learning-based task assistance in wearable AR.