HoloLens Device Research Articles

Delivering clinical tutorials to medical students using the Microsoft HoloLens 2: A mixed-methods evaluation

BackgroundMixed reality offers potential educational advantages in the delivery of clinical teaching. Holographic artefacts can be rendered within a shared learning environment using devices such as the Microsoft HoloLens 2. In addition to facilitating remote access to clinical events, mixed reality may provide a means of sharing mental models, including the vertical and horizontal integration of curricular elements at the bedside. This study aimed to evaluate the feasibility of delivering clinical tutorials using the Microsoft HoloLens 2 and the learning efficacy achieved.MethodsFollowing receipt of institutional ethical approval, tutorials on preoperative anaesthetic history taking and upper airway examination were facilitated by a tutor who wore the HoloLens device. The tutor interacted face to face with a patient and two-way audio-visual interaction was facilitated using the HoloLens 2 and Microsoft Teams with groups of students who were located in a separate tutorial room. Holographic functions were employed by the tutor. The tutor completed the System Usability Scale, the tutor, technical facilitator, patients, and students provided quantitative and qualitative feedback, and three students participated in semi-structured feedback interviews. Students completed pre- and post-tutorial, and end-of-year examinations on the tutorial topics.ResultsTwelve patients and 78 students participated across 12 separate tutorials. Five students did not complete the examinations and were excluded from efficacy calculations. Student feedback contained 90 positive comments, including the technology’s ability to broadcast the tutor’s point-of-vision, and 62 negative comments, where students noted issues with the audio-visual quality, and concerns that the tutorial was not as beneficial as traditional in-person clinical tutorials. The technology and tutorial structure were viewed favourably by the tutor, facilitator and patients. Significant improvement was observed between students’ pre- and post-tutorial MCQ scores (mean 59.2% Vs 84.7%, p < 0.001).ConclusionsThis study demonstrates the feasibility of using the HoloLens 2 to facilitate remote bedside tutorials which incorporate holographic learning artefacts. Students’ examination performance supports substantial learning of the tutorial topics. The tutorial structure was agreeable to students, patients and tutor. Our results support the feasibility of offering effective clinical teaching and learning opportunities using the HoloLens 2. However, the technical limitations and costs of the device are significant, and further research is required to assess the effectiveness of this tutorial format against in-person tutorials before wider roll out of this technology can be recommended as a result of this study

Three-Dimensional Coordinate Calibration Models for Augmented Reality Applications in Indoor Industrial Environments

The calibration of three-dimensional (3D) coordinates in augmented reality systems is a complex activity. It involves the recognition of environmental characteristics and technology that models the 3D space of the device, determining its position and orientation. Single markers suffer from numerical instability, particularly when they are small within the camera image. In the industrial environment, it is common for augmented reality applications to cover large spaces, making it difficult to maintain attributes such as precision and accuracy. To address this issue, our study proposes a two-step calibration model that leverages multiple markers for accurate localization in a larger indoor environment. We developed the calibration model using Unity3D, Mixed Reality ToolKit, and Vuforia and evaluated it in terms of precision and accuracy in a proof of concept using the MS Hololens device. Our findings reveal that employing two markers significantly reduces angular discrepancies between points in the real and augmented environments. Moreover, our results underscore that registration accuracy improves as the number of calibration points increases. The results show improvements in determining the axes that define the 3D space, with a direct influence on the position of the points observed in the experiment.

A novel mixed reality-guided dental implant placement navigation system based on virtual-actual registration

BackgroundsThe key to successful dental implant surgery is to place the implants accurately along the pre-operative planned paths. The application of surgical navigation systems can significantly improve the safety and accuracy of implantation. However, the frequent shift of the views of the surgeon between the surgical site and the computer screen causes troubles, which is expected to be solved by the introduction of mixed-reality technology through the wearing of HoloLens devices by enabling the alignment of the virtual three-dimensional (3D) image with the actual surgical site in the same field of view. MethodsThis study utilized mixed reality technology to enhance dental implant surgery navigation. Our first step was reconstructing a virtual 3D model from pre-operative cone-beam CT (CBCT) images. We then obtained the relative position between objects using the navigation device and HoloLens camera. Via the algorithms of virtual-actual registration, the transformation matrixes between the HoloLens devices and the navigation tracker were acquired through the HoloLens-tracker registration, and the transformation matrixes between the virtual model and the patient phantom through the image-phantom registration. In addition, the algorithm of surgical drill calibration assisted in acquiring transformation matrixes between the surgical drill and the patient phantom. These algorithms allow real-time tracking of the surgical drill's location and orientation relative to the patient phantom under the navigation device. With the aid of the HoloLens 2, virtual 3D images and actual patient phantoms can be aligned accurately, providing surgeons with a clear visualization of the implant path. ResultsPhantom experiments were conducted using 30 patient phantoms, with a total of 102 dental implants inserted. Comparisons between the actual implant paths and the pre-operatively planned implant paths showed that our system achieved a coronal deviation of 1.507 ± 0.155 mm, an apical deviation of 1.542 ± 0.143 mm, and an angular deviation of 3.468 ± 0.339°. The deviation was not significantly different from that of the navigation-guided dental implant placement but better than the freehand dental implant placement. ConclusionOur proposed system realizes the integration of the pre-operative planned dental implant paths and the patient phantom, which helps surgeons achieve adequate accuracy in traditional dental implant surgery. Furthermore, this system is expected to be applicable to animal and cadaveric experiments in further studies.

A novel deep learning-driven approach for predicting the pelvis soft-tissue deformations toward a real-time interactive childbirth simulation

Background and objectiveSoft-tissue dynamics plays an essential role in the mechanical functions of the human body. Numerical approaches using finite element modeling and mass-spring framework have been currently used to estimate the biological soft tissue dynamics. However, these approaches still have important computational cost due to the use of a mesh configuration in the formulation of the dynamic equilibrium equation and unstable convergence issue. MethodsWe present in this study a novel approach based on the deep learning framework to predict the deformation of soft-tissues. In particular, the Long Short-term Memory (LSTM) neural network and deep neural network (DNN) were used to deal with high-frequency oscillation signals. Different learning strategies (with and without data dimension reduction) were also applied. A simulation-based database was generated using our HyperMSM model for training and testing purposes. The application of the proposed approach on the childbirth simulation was addressed. ResultsUsing the root mean square error, the LSTM- and DNN-derived deformation deviation range from 0.139 mm to 1.062 mm (0.266%–2.028%) for both training and testing processes. The Pearson correlation coefficient of 0.994–0.999 demonstrates the strong similarity between predicted outputs and ground truth data. ConclusionsThis present work showed the capacity of the deep learning neural networks to predict complex physiological signals of the human body functions. As a perspective, this approach will be coupled with the Hololens device toward a novel interactive childbirth training tool with real-time feedback on the soft tissue deformations.

Fast soft-tissue deformations coupled with mixed reality toward the next-generation childbirth training simulator.

High-quality gynecologist and midwife training is particularly relevant to limit medical complications and reduce maternal and fetal morbimortalities. Physical and virtual training simulators have been developed. However, physical simulators offer a simplified model and limited visualization of the childbirth process, while virtual simulators still lack a realistic interactive system and are generally limited to imposed predefined gestures. Objective performance assessment based on the simulation numerical outcomes is still not at hand. In the present work, we developeda virtual childbirth simulator based on the Mixed-Reality (MR) technology coupled with HyperMSM (Hyperelastic Mass-Spring Model) formulation for real-time soft-tissue deformations, providing intuitive user interaction with the virtual physicalmodel and a quantitative assessment to enhance the trainee's gestures. Microsoft HoloLens 2 was used and the MR simulator was developed including a complete holographic obstetric model. A maternalpelvis system modelof a pregnant woman (including the pelvis bone, the pelvic floor muscles, the birth canal, the uterus, and the fetus) was generated, and HyperMSM formulation was applied to simulate the soft tissue deformations. To induce realistic reactions to free gestures, the virtual replicas of the user's detected hands were introduced into the physical simulation and were associated with a contact model between the hands and the HyperMSM models. The gesture of pulling any part of the virtual models with two hands was also implemented. Two labor scenarios were implemented within the MR childbirth simulator: physiological labor and forceps-assisted labor. A scoring system for the performance assessment was included based on real-time biofeedback. As results,our developedMR simulation application was developed in real-time with a refresh rate of 30-50 FPS on the HoloLens device. HyperMSM model was validated using FE outcomes: high correlation coefficients of [0.97-0.99] and weighted root mean square relative errors of 9.8% and 8.3% were obtained for the soft tissue displacement and energy density respectively. Experimental tests showed that the implemented free-user interaction system allows to apply the correct maneuvers (in particular the "Viennese" maneuvers) during the labor process, and is capable to induce a truthful reaction of the model. Obtainedresults confirm alsothe possibility of using our simulation's outcomes to objectively evaluate the trainee's performance with a reduction of 39% for the perineal strain energy density and 5.6mm for the vertical vaginal diameter when the "Viennese" technique is applied. This present study provides, for the first time, an interactive childbirth simulator with an MR immersive experience with direct free-hand interaction, real-time soft-tissue deformation feedback, and an objective performance assessment based on numerical outcomes. This offers a new perspective for enhancing next-generation training-based obstetric teaching. The used models of the maternal pelvic system and the fetus will be enhanced, and more delivery scenarios (e.g. instrumental delivery, breech delivery, shoulder dystocia) will be designed and integrated. The third stage of labor will be also investigated to include the delivery of the placenta, and the clamping and cutting of the umbilical cord.

Digital Imperfection

Digital Imperfection was a temporary installation at the German University in Cairo (GUC) that combined the use of mixed-reality tools and earth as a sustainable and multifaceted material. The project involved two separate processes that came together during the final montage procedure: on the one hand, the design of handmade earth bricks, and on the other, the design of a parametric wall and the coding of the montage procedure on the mixed-reality platform.&#x0D; The project aimed to reconnect both students and a wider audience with a traditional craft through the use of modern digital tools. Hand-crafted bricks were stacked to create a wall with the help of a HoloLens device, which overlaid a digital four-dimensional model over the physical world. Despite the mediation of digital apparatus, the aim was to engage participants in a comprehensive workflow that involved aspects of both handmade production and interactive assembly, rather than promoting a mere robotic process. During the research phase, we investigated the relationship between high-tech and low-tech tools through the following questions:&#x0D; • How can we incorporate digital technology without losing human interaction?&#x0D; • How can we measure and account for manufacturing imperfections?&#x0D; • How can we minimise those imperfections within the design and its montage?&#x0D; • What benefits and opportunities are offered by the combination of low- and high-tech techniques? &#x0D; The process accounted for various imperfections and height irregularities (resulting, for example, from differences in mortar thickness or manufacturing), sustaining a constant loop with real-time feedback: the physical model was updated with new bricks while the digital model was updated with height corrections.&#x0D; The research offers multiple benefits. Firstly, it introduces students (and a broader academic public) to the use of sustainable materials in combination with parametric design. Secondly, it produces a digitally-designed installation (of relative complexity) without the need for printed documentation. Finally, it demonstrates a resource-saving method in which both building procedure and instructions are entirely virtual, eliminating the need for framework or printed plans.&#x0D; Digital Imperfection puts humans at the centre of the digital assembly process; humans are not replaced by robots or algorithms but instead collaborate with them in ways that maximise the advantages they offer.&#x0D;

Verify the Validity of Guidance Sign in Buildings: A New Method Based on Mixed Reality with Eye Tracking Device

Evacuation signs are one of the key factors which influence the building evacuation efficiency and evacuees’ route choice behaviors. It is important to test the effectiveness of evacuation signs, and an eye movement test was verified to be a good way. In this paper, both reality and MR (Mixed-Reality) evacuation experiments were carried out to collect the eye movement data of participants, such as gaze position, fixation duration, and number of gazes. In the reality experiment, participants took part in an emergency evacuation at a T-junction corridor with the glass-type eye-tracking device. In order to collect eye movement data in the MR experiment, the eye tracking device was integrated into the Microsoft HoloLens device (MR-ET for short). In either reality or MR experiments, each participant took part in five sub-experiments with five different types of evacuation signs. It was found that (1) the evacuation signs with a black background, green foreground and flashing lights have a better effect in guidance; (2) the flashing lights can better attract the attention of the participants, and enhanced the participants’ understanding of the signs. (3) An X-mark will hinder participants’ understanding of the signs and increase their reaction time. According to the results of the experiments, it is also found that the eye-tracking test in an MR environment is also a good method to test the effectiveness of the evacuation signs, and such a method can also be used in building safety design without damaging a wall or changing the reality environment of a building.

The use of mixed reality technology for the objective assessment of clinical skills: a validation study

BackgroundMixed Reality technology may provide many advantages over traditional teaching methods. Despite its potential, the technology has yet to be used for the formal assessment of clinical competency. This study sought to collect validity evidence and assess the feasibility of using the HoloLens 2 mixed reality headset for the conduct and augmentation of Objective Structured Clinical Examinations (OSCEs).MethodsA prospective cohort study was conducted to compare the assessment of undergraduate medical students undertaking OSCEs via HoloLens 2 live (HLL) and recorded (HLR), and gold-standard in-person (IP) methods. An augmented mixed reality scenario was also assessed.ResultsThirteen undergraduate participants completed a total of 65 OSCE stations. Overall inter-modality correlation was 0.81 (p = 0.01), 0.98 (p = 0.01) and 0.82 (p = 0.01) for IP vs. HLL, HLL vs. HLR and IP vs. HLR respectively. Skill based correlations for IP vs. HLR were assessed for history taking (0.82, p = 0.01), clinical examination (0.81, p = 0.01), procedural (0.88, p = 0.01) and clinical skills (0.92, p = 0.01), and assessment of a virtual mixed reality patient (0.74, p = 0.01). The HoloLens device was deemed to be usable and practical (Standard Usability Scale (SUS) score = 51.5), and the technology was thought to deliver greater flexibility and convenience, and have the potential to expand and enhance assessment opportunities.ConclusionsHoloLens 2 is comparable to traditional in-person examination of undergraduate medical students for both live and recorded assessments, and therefore is a valid and robust method for objectively assessing performance. The technology is in its infancy, and users need to develop confidence in its usability and reliability as an assessment tool. However, the potential to integrate additional functionality including holographic content, automated tracking and data analysis, and to facilitate remote assessment may allow the technology to enhance, expand and standardise examinations across a range of educational contexts.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery.

We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic () and maxillofacial surgeries (). For preoperative input data, computed tomography (CT) (), and surface rendered models () were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (); this was achieved by surface tracking of fiducials (), external tracking (), or manual placement (). Microsoft HoloLens devices ( in 2021, in 2022) were the most frequently used OST-HMDs; gestures and/or voice () served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.

Augmented reality calibration using feature triangulation iteration-based registration for surgical navigation

BackgroundMarker-based augmented reality (AR) calibration methods for surgical navigation often require a second computed tomography scan of the patient, and their clinical application is limited due to high manufacturing costs and low accuracy. MethodsThis work introduces a novel type of AR calibration framework that combines a Microsoft HoloLens device with a single camera registration module for surgical navigation. A camera is used to gather multi-view images of a patient for reconstruction in this framework. A shape feature matching-based search method is proposed to adjust the size of the reconstructed model. The double clustering-based 3D point cloud segmentation method and 3D line segment detection method are also proposed to extract the corner points of the image marker. The corner points are the registration data of the image marker. A feature triangulation iteration-based registration method is proposed to quickly and accurately calibrate the pose relationship between the image marker and the patient in the virtual and real space. The patient model after registration is wirelessly transmitted to the HoloLens device to display the AR scene. ResultsThe proposed approach was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with six advanced AR calibration methods. The proposed method obtained average fusion errors of 0.70 ± 0.16 and 0.91 ± 0.13 mm in phantom and volunteer experiments, respectively. The fusion accuracy of the proposed method is the highest among all comparison methods. A volunteer liver puncture clinical simulation experiment was also conducted to show the clinical feasibility. ConclusionsOur experiments proved the effectiveness of the proposed AR calibration method, and revealed a considerable potential for improving surgical performance.

HoloLens 1 vs. HoloLens 2: Improvements in the New Model for Orthopedic Oncological Interventions.

This work analyzed the use of Microsoft HoloLens 2 in orthopedic oncological surgeries and compares it to its predecessor (Microsoft HoloLens 1). Specifically, we developed two equivalent applications, one for each device, and evaluated the augmented reality (AR) projection accuracy in an experimental scenario using phantoms based on two patients. We achieved automatic registration between virtual and real worlds using patient-specific surgical guides on each phantom. They contained a small adaptor for a 3D-printed AR marker, the characteristic patterns of which were easily recognized using both Microsoft HoloLens devices. The newest model improved the AR projection accuracy by almost 25%, and both of them yielded an RMSE below 3 mm. After ascertaining the enhancement of the second model in this aspect, we went a step further with Microsoft HoloLens 2 and tested it during the surgical intervention of one of the patients. During this experience, we collected the surgeons’ feedback in terms of comfortability, usability, and ergonomics. Our goal was to estimate whether the improved technical features of the newest model facilitate its implementation in actual surgical scenarios. All of the results point to Microsoft HoloLens 2 being better in all the aspects affecting surgical interventions and support its use in future experiences.

Framework for Armature-Based 3D Shape Reconstruction of Sensorized Soft Robots in eXtended Reality.

Soft robots are typically intended to operate in highly unpredictable and unstructured environments. Although their soft bodies help them to passively conform to their environment, the execution of specific tasks within such environments often requires the help of an operator that supervises the interaction between the robot and its environment and adjusts the actuation inputs in order to successfully execute the task. However, direct observation of the soft robot is often impeded by the environment in which it operates. Therefore, the operator has to depend on a real-time simulation of the soft robot based on the signals from proprioceptive sensors. However, the complicated three-dimensional (3D) configurations of the soft robot can be difficult to interpret using traditional visualization techniques. In this work, we present an open-source framework for real-time 3D reconstruction of soft robots in eXtended Reality (Augmented and Virtual Reality), based on signals from their proprioceptive sensors. This framework has a Robot Operating System (ROS) backbone, allowing for easy integration with existing soft robot control algorithms for intuitive and real-time teleoperation. This approach is demonstrated in Augmented Reality using a Microsoft Hololens device and runs at up to 60 FPS. We explore the influence that system parameters such as mesh density and armature complexity have on the reconstruction's key performance metrics (i.e., speed, scalability). The open-source framework is expected to function as a platform for future research and developments on real-time remote control of soft robots operating in environments that impede direct observation of the robot.

Application of Mixed Reality in Medical Training and Surgical Planning Focused on Minimally Invasive Surgery

Introduction:Medical training is a long and demanding process, in which the first stages are usually based on two-dimensional, static, and unrealistic content. Conversely, advances in preoperative imaging have made it an essential part of any successful surgical procedure. However, access to this information often requires the support of an assistant and may compromise sterility in the surgical process. Herein, we present two solutions based on mixed reality that aim to improve both training and planning in minimally invasive surgery.Materials and Methods:Applications were developed for the use of the Microsoft HoloLens device. The urology training application provided access to a variety of anatomical and surgical training contents. Expert urological surgeons completed a questionnaire to evaluate its use. The surgical planning solution was used during laparoscopic renal tumorectomy in an experimental model and video-assisted right upper lobectomy in an adult patient. Surgeons reported their experience using this preoperative planning tool for surgery.Results:The solution developed for medical training was considered a useful tool for training in urological anatomy, facilitating the translation of this knowledge to clinical practice. Regarding the solution developed for surgical planning, it allowed surgeons to access the patient’s clinical information in real-time, such as preoperative imaging studies, three-dimensional surgical planning models, or medical history, facilitating the surgical approach. The surgeon’s view through the mixed reality device was shared with the rest of the surgical team.Conclusions:The mixed reality-based solution for medical training facilitates the transfer of knowledge into clinical practice. The preoperative planning tool for surgery provides real-time access to essential patient information without losing the sterility of the surgical field. However, further studies are needed to comprehensively validate its clinical application.

İnsan - Makine Etkileşiminde Hologram Kullanılarak İnsan Niyetinin Yönlendirilmesi

Decision making could be critically important for people in some situations. People have intentions to choose a side if it is time to make decision. These intentions are strongly related to the knowledge and experience. But sometimes outer effects could reshape their intentions easily. In this paper, an experimental work is studied for Human-Machine Interaction in which if holograms could change or affect the human intention. And also the question which asks whether people trust on a hologram agent while making decision or not is researched. To study this research, a memory game application is developed and this application is run on Microsoft Hololens device. Hololens is used to maintain the Augmented Reality (AR) environment with holograms. An algorithm with a Finite State Machines (FSM) is developed to manage the response of hologram agent while giving hint to the confused users. The accuracy of the hints changes nonlinearly. 3 different game stages are trained on users to see how they are affected by both virtual and real world noises. According to the results, intention of majority of users was affected by the hologram while making the decisions. Also it is observed that some users who were concentrated too much to memorize the order of objects did not realize the hologram, and some few could not understand the actions of hologram.

Лапароскопическая резекция почки с использованием технологии дополненной реальности: новый вектор развития оперативной урологии?

Aim to evaluate the efficiency and usefulness of augmented reality (AR) technology using HoloLens glasses for laparoscopic partial nephrectomy (LPN). Materials and methods From July to December 2019, a total of 5 patients with localized kidney cancer (cT1aN0M0) underwent AR-assisted LRP. The mean RENAL score was 6 points (5-8). Preoperatively, all patients underwent contrast-enhanced multispiral computed tomography (MSCT). The three-dimensional reconstructions of the kidney, tumor, part of the abdominal aorta with the renal artery and its branches, part of the inferior vena cava with the renal vein were segmented with color coding and connected into a single virtual 3D model, which was loaded into the program in order to display image in AR glasses. The duration of surgery and thermal ischemia, type and frequency of intraoperative complications, as well as the time spent on preparing the 3D model and the Microsoft HoloLens device were evaluated. To assess the feasibility of using AR technology intraoperatively, a Likert scale was filled out by the surgeon. Results It took 10 (9-11) hours to prepare the model, including time to optimize the model and to set up its display and interactions. The setup of HoloLens required an average of 7.8 (5- 12) min. The total duration of the operation and the period of warm ischemia was 108 (90-120) and 20 (15-25) min, respectively, while intraoperative blood loss was 160 (110-250) ml. In all cases, a negative surgical margin was found. The surgeon who performed all the operations assessed the use of AR technology with the HoloLens device as highly beneficial in all clinical cases. Conclusion The use of AR technology with a HoloLens holographic device during LPN can lead to improved treatment outcomes.

Experimental Setup Employed in the Operating Room Based on Virtual and Mixed Reality: Analysis of Pros and Cons in Open Abdomen Surgery.

Currently, surgeons in operating rooms are forced to focus their attention both on the patient's body and on flat low-quality surgical monitors, in order to get all the information needed to successfully complete surgeries. The way the data are displayed leads to disturbances of the surgeon's visuals, which may affect his performances, besides the fact that other members of the surgical team do not have proper visual tools able to aid him. The idea underlying this paper is to exploit mixed reality to support surgeons during surgical procedures. In particular, the proposed experimental setup, employed in the operating room, is based on an architecture that put together the Microsoft HoloLens, a Digital Imaging and Communications in Medicine (DICOM) player and a mixed reality visualization tool (i.e., Spectator View) developed by using the Mixed Reality Toolkit in Unity with Windows 10 SDK. The suggested approach enables visual information on the patient's body as well as information on the results of medical screenings to be visualized on the surgeon's headsets. Additionally, the architecture enables any data and details to be shared by the team members or by external users during surgical operations. The paper analyses in detail advantages and drawbacks that the surgeons have found when they wore the Microsoft HoloLens headset during all the ten open abdomen surgeries conducted at the IRCCS Hospital “Giovanni Paolo II” in the city of Bari (Italy). A survey based on Likert scale demonstrates how the use of the suggested tools can increase the execution speed by allowing multitasking procedures, i.e., by checking medical images at high resolution without leaving the operating table and the patient. On the other hand, the survey also reveals an increase in the physical stress and reduced comfort due to the weight of the Microsoft HoloLens device, along with drawbacks due to the battery autonomy. Additionally, the survey seems to encourage the use of DICOM Viewer and Spectator View both for surgical education and for improving surgery outcomes. Note that the real use of the conceived platform in the operating room represents a remarkable feature of this paper, since most if not all the studies conducted so far in literature exploit mixed reality only in simulated environments and not in real operating rooms. In conclusion, the study clearly highlights that, despite the challenges required in the forthcoming years to improve the current technology, mixed reality represents a promising technique that will soon enter the operating rooms to support surgeons during surgical procedures in many hospitals across the world.

A Wearable Augmented Reality Navigation System for Surgical Telementoring Based on Microsoft HoloLens.

This paper reports a new type of augmented reality (AR) system that integrates a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery. In this system, a stereo camera is used to track the 3D position of a scalpel and transfer its coordinates wirelessly to a HoloLens device. In the scenario of surgical training, a virtual surgical scene with pre-recorded surgical annotations is superimposed with the actual surgical scene so that the surgical trainee is able to operate following virtual instructions. In the scenario of telementored surgery, the virtual surgical scene is co-registered with the actual surgical scene so that the virtual scalpel remotely mentored by an experienced surgeon provides the AR guidance for the inexperienced on-site operator. The performance characteristics of the proposed AR telementoring system are verified by benchtop experiments. The clinical applicability of the proposed system in telementored skin grafting surgery and fasciotomy is validated in a New Zealand rabbit model. Our benchtop and in vivo experiments demonstrate the potential to improve surgical performance and reduce healthcare disparities in remote areas with limited resources.

A Comparative Evaluation of a Virtual Reality Table and a HoloLens-Based Augmented Reality System for Anatomy Training

Anatomy training with real cadavers poses many practical problems for which new training and educational solutions have been developed making use of technologies based on real-time 3-D graphics. Although virtual reality (VR) and augmented reality (AR) have been previously used in the medical field, it is not easy to select the right 3-D technology or setup for each particular problem. For this reason, this article presents a comprehensive comparative study with 82 participants between two different 3-D interactive setups: an optical-based AR setup, implemented with a Microsoft HoloLens device, and a semi-immersive setup based on a VR Table. Both setups are tested using an anatomy training software application. Our primary hypothesis is that there would be statistically significant differences between the use of the AR application and the use of the VR Table. Our secondary hypothesis is that user preference and recommendation for the VR setup would be higher than for the HoloLens-based system. After completing two different tasks with both setups, the participants filled two questionnaires about the use of the anatomy training application. Three objective measures are also recorded (time, number of movements, and a score). The results of the experiments show that more than two-thirds of the users prefer, recommend, and find more useful the VR setup. The results also show that there are statistically significant differences in the use of both systems in favor of the VR Table.

Measuring the Performance Impact of Using the Microsoft HoloLens 1 to Provide Guided Assembly Work Instructions

Abstract The human performance benefits of four types of guided assembly instructions, including the Microsoft HoloLens 1, were analyzed in the context of a realistic assembly task. Several studies have confirmed the benefits of using augmented reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware, including head-mounted displays, for complex assembly tasks. Participants completed a mock wing assembly task using the Microsoft HoloLens 1, and completion time, error count, and net promoter score (NPS) were recorded. These data were compared to data from previous research which employed desktop model-based instructions (MBIs), tablet MBI, and tablet AR instructions for the same task. The use of HoloLens AR instructions led to time saving of 16% over the tablet AR instructions. HoloLens users also had lower error rates than non-AR users. Despite the performance benefits of the HoloLens AR instructions, this condition had a lower NPS than the tablet AR group. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the users favored the tablet AR condition, the HoloLens condition had significantly faster assembly times. The authors recommend using the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders understand the benefits of different AR technologies for manual assembly tasks.

Real-Time Volume-to-Plane Comparison for Mixed Reality–Based Progress Monitoring

On-site construction inspection for progress monitoring is important because accurate and timely assessment of progress allows managers to make adjustments to minimize time and cost overrun when deviations from the schedule occur. However, current practice still remains manual, time consuming, and labor intensive, which usually requires exhaustive manual data extraction from drawings and other databases. Inspection of the interior environment can be even more challenging due to the complexity and interconnectivity of tasks. Systems based on mobile devices could potentially enhance the inspection process and reduce the required time by allowing the inspector to acquire progress data by simply walking around the site. This paper presents an automated method that allows a real-time comparison between the registered as-built and as-planned data while the inspector navigates inside a building under construction to perform inspection wearing a Microsoft HoloLens device. The proposed method automatically detects if volumetric objects such as beams, columns, walls, and so on exist in reality according to plan. Different parameters of the method were tested, and 95.2% accuracy, 96.5% precision, and 97.3% recall was achieved by using the optimum set of parameters.