3D Mixed Reality Research Articles

Validation of a dynamic 4D microsurgical bypass simulator for training and teaching microvascular anastomosis techniques with blood flow and fluorescence imaging

ObjectiveMicrovascular anastomosis is challenging, and training surgeons to develop and maintain skills is imperative. Current training models either miss the simulation of the surgical workflow, lack 3D key-hole space, need ethical approval, require special preparation, or lack realism. To circumvent these issues, this study describes the use of a mixed reality 3D printed model with integrated blood flow for training cerebral anastomosis and assesses its validity. MethodsDifferent-sized 3D-printed artificial micro artery models in a 3D brain space with a keyhole opening were used. The model was connected to a 4D simulator to induce pulsatile blood flow. Neurosurgical microscopes and exoscopes were used for visualization. Nine participants (n = 6 board-certified cerebrovascular neurosurgeons; n = 3 in-training) participated in the study and practiced anastomosis techniques with the simulator. Two senior, experienced vascular neurosurgeons mentored live teaching activity on the simulator. Participants completed a survey to score the face and content validity of the simulation on a 5-point Likert scale. Simulation time and anastomosis score differences between in-training and board-certified participants were compared for construct validity. ResultsParticipants scored the simulation difficulty similar to actual surgery, proving face validity. All participants agreed that practice on the provided simulator models would improve bypass techniques (μ = 4.67/5 ± 0.47) and instrument handling (μ = 4.56/5 ± 0.68). Board-certified participants had better anastomosis scores than in-training participants (non-significant difference). ConclusionsThe 4D simulator and the high-fidelity artificial 3D printed model effectively simulated actual bypass surgery in a key-hole fashion with blood flow abilities. Limited resources and preparation time are needed for the training setup. The model provides benefits in learning and maintaining anastomosis skills and allows for easy adaptation to different microanatomical scenarios.

Targeted augmented reality-guided transperineal prostate biopsies study: initial experience.

Transperineal biopsy of magnetic resonance imaging (MRI)-detected prostate lesions is now the established technique used in prostate cancer (CaP) diagnostics. Virtual Surgery Intelligence (VSI) Holomedicine by Apoqlar (Hamburg, Germany) is a mixed reality (MR)/augmented reality (AR) software platform that runs on the HoloLens II system (Microsoft, Redford, USA). Multiparametric prostate MRI images were converted into 3D holograms and added into a MR space, enabling visualization of a 3D hologram and image-assisted prostate biopsy. The Targeted Augmented Reality-GuidEd Transperineal (TARGET) study investigated the feasibility of performing AR-guided prostate biopsies in a MR framework, using the VSI platform in patients with MRI-detected prostate lesions. Ten patients with a clinical suspicion of CaP on MRI (Prostate Imaging-Reporting and Data System, PI-RADS 4/5) were uploaded to the VSI HoloLens system. Two MR/AR-guided prostate biopsies were then acquired using the PrecisionPoint Freehand transperineal biopsy system. Cognitive fusion biopsies were performed as standard of care following the MR/AR-guided prostate biopsies. All 10 patients successfully underwent MR/AR-guided prostate biopsy after 3D MR images were overlaid on the patient's body. Prostatic tissue was obtained in all MR/AR-guided specimens. Seven patients (70%) had matching histology in both the standard and MR/AR-guided biopsies. The remaining three had ISUP (International Society of Urological Pathology) Grade 2 CaP. There were no immediate complications. We believe this is a world first. The initial feasibility data from the TARGET study demonstrated that an MR/AR-guided prostate biopsy utilizing the VSI Holomedicine system is a viable option in CaP diagnostics. The next stage in development is to combine AR images with real-time needle insertion and to provide further data to formally appraise the sensitivity and specificity of the technique.

Educational value of mixed reality combined with a three-dimensional printed model of aortic disease for vascular surgery in the standardized residency training of surgical residents in China: a case control study

BackgroundThe simulated three-dimensional (3D) printed anatomical model of the aorta, which has become the norm in medical education, has poor authenticity, tactility, feasibility, and interactivity. Therefore, this study explored the educational value and effect of mixed reality (MR) combined with a 3D printed model of aortic disease in training surgical residents.MethodFifty-one resident physicians who rotated in vascular surgery were selected and divided into traditional (27) and experimental (24) teaching groups using the random number table method. After undergoing the experimental and traditional training routines on aortic disease, both the groups took a theoretical test on aortic disease and an assessment of the simulation based on the Michigan Standard Simulation Experience Scale (MiSSES) template. Their scores and assessment results were compared. The study was conducted at the Department of Vascular Surgery of Peking University People’s Hospital, Beijing, China.ResultsIn the theoretical test on aortic disease, the experimental teaching group obtained higher mean total scores (79.0 ± 9.1 vs. 72.6 ± 7.5, P = 0.013) and higher scores in anatomy/ pathophysiology (30.8 ± 5.4 vs. 24.8 ± 5.8; P < 0.001) than the traditional teaching group. The differences in their scores in the differential diagnosis (25.8 ± 3.0 vs. 23.3 ± 4.9; P = 0.078) and treatment (22.5 ± 11.8 vs. 24.5 ± 8.2; P = 0.603) sessions were insignificant. The MR-assisted teaching stratified the vascular residents through the MiSSES survey. Overall, 95.8% residents (23/24) strongly or somewhat agreed that the MR was adequately realistic and the curriculum helped improve the ability to understanding aortic diseases. Further, 91.7% residents (22/24) strongly or somewhat agreed that the MR-assisted teaching was a good training tool for knowledge on aortic diseases. All residents responded with “Good” or “Outstanding” on the overall rating of the MR experience.ConclusionsMR combined with the 3D printed model helped residents understand and master aortic disease, particularly regarding anatomy and pathophysiology. Additionally, the realistic 3D printing and MR models improved the self-efficacy of residents in studying aortic diseases, thus greatly stimulating their enthusiasm and initiative to study.

X-Space: Interaction design of extending mixed reality space from Web2D visualization

Mixed reality offers a larger visualization space and more intuitive means of interaction for data exploration, and many works have been dedicated to combining 2D visualizations on screen with mixe reality. However, for each combination, we need to customize the implementation of the corresponding mixed reality 3D visualization. It is a challenge to simplify this development process and enable agile building of mixed reality 3D visualizations for 2D visualizations. In addition, many existing 2D visualizations do not provide interfaces oriented to immersive analytics, so how to extend the mixed reality 3D space from existing 2D visualizations is another challenge. This work presents an agile and flexible approach to interactively transfer visualizations from 2D screens to mixed reality 3D spaces. We designed an interactive process for spatial generation of mixed-reality 3D visualizations, defined a unified data transfer framework, integrated data deconstruction techniques for 2D visualizations, implemented interfaces to immersive visualization building tool-kits, and encapsulated these techniques into a tool named X-Space. We validated that the approach is feasible and effective through 2D visualization cases including scatter plots, stacked bar charts, and adjacency matrix. Finally, we conducted expert interviews to discuss the usability and value of the method.

Application of virtual and mixed reality for 3D visualization in intracranial aneurysm surgery planning: a systematic review.

Precise preoperative anatomical visualization and understanding of an intracranial aneurysm (IA) are fundamental for surgical planning and increased intraoperative confidence. Application of virtual reality (VR) and mixed reality (MR), thus three-dimensional (3D) visualization of IAs could be significant in surgical planning. Authors provide an up-to-date overview of VR and MR applied to IA surgery, with specific focus on tailoring of the surgical treatment. A systematic analysis of the literature was performed in accordance with the PRISMA guidelines. Pubmed, and Embase were searched to identify studies reporting use of MR and VR 3D visualization in IA surgery during the last 25 years. Type and number of IAs, category of input scan, visualization techniques (screen, glasses or head set), inclusion of haptic feedback, tested population (residents, fellows, attending neurosurgeons), and aim of the study (surgical planning/rehearsal, neurosurgical training, methodological validation) were noted. Twenty-eight studies were included. Eighteen studies (64.3%) applied VR, and 10 (35.7%) used MR. A positive impact on surgical planning was documented by 19 studies (67.9%): 17 studies (60.7%) chose the tailoring of the surgical approach as primary outcome of the analysis. A more precise anatomical visualization and understanding with VR and MR was endorsed by all included studies (100%). Application of VR and MR to perioperative 3D visualization of IAs allowed an improved understanding of the patient-specific anatomy and surgical preparation. This review describes a tendency to utilize mostly VR-platforms, with the primary goals of a more accurate anatomical understanding, surgical planning and rehearsal.

A case study towards assessing the impact of mixed reality-based inspection and resolution of MEP issues during construction

Despite advances in 3D clash detection during preconstruction, mechanical, electrical, and plumbing (MEP) installations are still prone to the detection of unforeseen clashes during construction. These issues must be resolved as quickly as possible to prevent significant schedule delays. Through interviews and field observations, this case study investigates the impact of mixed reality (MR) on the inspection and resolution of field-detected MEP issues from product, organization, and process (POP) perspectives. For the product impact, preliminary findings from the field interviews show that MR-based inspection would increase the quality of MEP installation by identifying errors easily and resolving them faster. For the organizational impact, we modeled and compared the current (as-is) and MR-integrated (to-be) MEP field issue resolution workflows using Business Process Model and Notation (BPMN) and determined that MR-based inspections can decrease the coordination overhead between MEP engineers and superintendents by up to 75%. This translates into at least a 50% faster resolution of an MEP issue for the process impact. The paper contributes to the practice of MR-based field inspection by providing a method to quantify potential time savings by integrating MR into the MEP field issue resolution workflow and field interview questions for MEP engineers and superintendents to further examine the use of MR during inspection activities in construction projects. Our observations of MEP superintendents and engineers during field inspection showed that not all building information visualized in MR is useful for their inspection tasks. We developed a classification for building information usefulness to help construction project managers who are deploying MR determine useful information for the task at hand that needs to be integrated into the 3D MR model for MR-based inspections.

Mixed Reality Improves 3D Visualization and Spatial Awareness of Bone Tumors for Surgical Planning in Orthopaedic Oncology: A Proof of Concept Study.

In orthopedic oncology, computer navigation and 3D-printed guides facilitate precise osteotomies only after surgical exposure. Before surgeries start, it is challenging to mentally process and superimpose the virtual medical images onto patients' anatomy for preoperative surgical planning. Mixed Reality (MR) is an immersive technology merging real and virtual worlds, and users can interact with digital objects in real time. Through Head-Mounted Displays, surgeons directly visualize holographic models that overlaid on tumor patients. The technology may facilitate surgical planning before skin incisions. Nine bone tumor patients were included (July 2021 - Dec 2022). There were six primary bone sarcomas, two benign bone tumors, and one revision pelvic prosthesis. MR applications were created using patients' preoperative medical images. The surgeon examined each patient clinically using the conventional method of viewing 2D images and MR via HMD, Hololens 2. A Likert-Scale (LS) questionnaire and The National Aeronautics and Space Administration-Task Load Index (NASA-TLX) score were used to evaluate and compare the effectiveness of surgical planning and the surgeon's clinical cognitive workload for the two methods. The qualitative survey of the LS questionnaire suggested that the MR group had superior spatial awareness of tumors and was considered more effective as a preoperative planning tool than the conventional group. For NASA-TLX scores, the overall cognitive workload was lower in MR 3D hologram group than in the 2D Group for preoperative clinical assessment. When using MR technology with HMDs, the surgeon reported no discomfort. MR technology may improve 3D visualization and spatial awareness of bone tumors in patients' anatomies and may facilitate surgical planning before skin incisions in orthopedic oncology surgery. With less cognitive load and better ergonomics, surgeons can focus on patients and surgical tasks with MR technology. Further studies must investigate whether MR technology improves clinical outcomes.

A 3D mixed reality visualization of network topology and activity results in better dyadic cyber team communication and cyber situational awareness.

Cyber defense decision-making during cyber threat situations is based on human-to-human communication aiming to establish a shared cyber situational awareness. Previous studies suggested that communication inefficiencies were among the biggest problems facing security operation center teams. There is a need for tools that allow for more efficient communication of cyber threat information between individuals both in education and during cyber threat situations. In the present study, we compared how the visual representation of network topology and traffic in 3D mixed reality vs. 2D affected team performance in a sample of cyber cadets (N = 22) cooperating in dyads. Performance outcomes included network topology recognition, cyber situational awareness, confidence in judgements, experienced communication demands, observed verbal communication, and forced choice decision-making. The study utilized network data from the NATO CCDCOE 2022 Locked Shields cyber defense exercise. We found that participants using the 3D mixed reality visualization had better cyber situational awareness than participants in the 2D group. The 3D mixed reality group was generally more confident in their judgments except when performing worse than the 2D group on the topology recognition task (which favored the 2D condition). Participants in the 3D mixed reality group experienced less communication demands, and performed more verbal communication aimed at establishing a shared mental model and less communications discussing task resolution. Better communication was associated with better cyber situational awareness. There were no differences in decision-making between the groups. This could be due to cohort effects such as formal training or the modest sample size. This is the first study comparing the effect of 3D mixed reality and 2D visualizations of network topology on dyadic cyber team communication and cyber situational awareness. Using 3D mixed reality visualizations resulted in better cyber situational awareness and team communication. The experiment should be repeated in a larger and more diverse sample to determine its potential effect on decision-making.

Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

Remote robotic interventions and maintenance tasks are frequently required in hazardous environments. Particularly, missions with a redundant mobile manipulator in the world’s most complex machine, the CERN Large Hadron Collider, are performed in a sensitive underground environment with radioactive or electromagnetic hazards, bringing further challenges in safety and reliability. The mission’s success depends on the robot’s hardware and software, and when the tasks become too unpredictable to execute autonomously, the operators need to make critical decisions. However, in most current human-machine systems, the state of the human is neglected. In this context, a novel 3D Mixed Reality (MR) human-robot interface with the Operator Monitoring System (OMS) was developed to advance safety and task efficiency with improved spatial awareness, advanced manipulator control, and collision avoidance. However, new techniques could increase the system’s sophistication and add to the operator’s workload and stress. Therefore, for operational validation, the 3D MR interface had to be compared with an operational 2D interface, which has been used in hundreds of interventions. With the 3D MR interface, the execution of precise approach tasks was faster, with no increased workload or physiological response. The new 3D MR techniques improved the teleoperation quality and safety while maintaining similar effects on the operator. The OMS worked jointly with the interface and performed well with operators with varied teleoperation backgrounds facing a stressful real telerobotic scenario in the LHC. The paper contributes to the methodology for human-centred interface evaluation incorporating the user’s physiological state: heart rate, respiration rate and skin electrodermal activity, and combines it with the NASA TLX assessment method, questionnaires, and task execution time. It provides novel approaches to operator state identification, the GUI-OMS software architecture, and the evaluation of the 3D MR techniques. The solutions can be practically applied in mission-critical applications, such as telesurgery, space robotics, uncrewed transport vehicles and semi-autonomous machinery.

A Mixed Reality-enhanced Rapid Prototyping Approach for Industrial Articulated Products

The design of industrial articulated products is challenging due to the consideration of multiple poses, movements, and especially in-field constraints, such as the design of a personalized robot arm or the layout of articulated hoists. Traditionally, depicting 2D sketches from multiple perspectives and poses of the product is tedious. Meanwhile, on-site evaluation after fabricating the physical prototype is costly and time-consuming. In order to assist in context-aware decision-making at the early design stage, this research proposes a mixed reality-enhanced rapid prototyping approach for industrial articulated products. In a 3D mixed reality environment that incorporates the surroundings, designers can depict key joints and rough shapes through intuitive gesture-based interaction. After a quick prototype creation, the kinematics preview is provided to designers for preliminary functional verification. Finally, a user study shows that the proposed approach can help designers rapidly complete the MR prototype of articulated products at the early design stage, provide a better understanding of the product movement, assist in-field solution generation and facilitate communication. It is hoped this explorative study can offer insightful ideas with effective toolkits for assisting the new product design process more intuitively and interactively.

Impact of Mixed Reality Presentation on STEM Engagement and Comprehension: A Pilot Study on Adult Scientists

Current educational presentation software used in STEM education fail to maximize student engagement and comprehension. Mixed reality presentation is one specific type of digital presentation software that has shown to significantly improve student engagement and comprehension. In this paper, we describe a pilot study on adult scientists which evaluates the usage of an integrated mixed reality presentation software in the Zyndo platform as an enhanced alternative to Adobe PDFs. A group of adult scientists (N = 20), with higher education of at least a bachelor’s degree, from an academic research center at Harvard Medical School were randomized and asked to read two articles (one on Immunology and the other on Bioengineering) presented through either the mixed reality presentation or PDFs. Our results indicate that participants improved in nearly all metrics for engagement (ranging from + 4 to 51% improvement depending on engagement metric and subject matter) when viewing the mixed reality presentation over the traditional PDFs for both articles. Specifically, the participants demonstrated improved comprehension of the scientific content and time spent viewing the presentation in a content-dependent manner. Therefore, 3D mixed reality environments can potentially be applied to enhance student learning in STEM fields, particularly Biomedical Engineering in both on-line and in person classroom settings.

Mixed-reality hologram for diagnosis and surgical planning of double outlet of the right ventricle: a pilot study

To evaluate the mixed-reality (MR) hologram, a novel technology based on two-dimensional images, which simulates three-dimensional (3D) images and provides a dynamic and interactive alternative, for its usefulness in the diagnosis and surgical planning of double outlet of the right ventricle (DORV). Thirty-four patients who were suspected of DORV based on ultrasound findings underwent cardiac computed tomography angiography (CTA). The patients were assigned randomly to the MR holographic guidance (MRHG) group or the control group. For the patients in the MRHG group, the CTA images were converted into Standard Template Library (STL) files after segmentation, 3D reconstruction, colourisation, and transparentisation, and then exported for MR holographic visualisation. The CTA images of the patients in the control group were analysed using routine 3D reconstruction only. Diagnostic accuracy and surgical planning were compared between the two groups based on visualisation at surgery. In the MRHG group, the 3D hologram observation was in concordance with the actual anatomical findings, and the DORV type was classified accurately in all patients. The diagnostic accuracy for the malformation was 95.5% in the MRHG group and 89.7% in the control group, but the difference was not significant (p=0.3). All the procedures were exactly the same as planned based on the 3D MR holographic model. The surgical planning time was shorter for the MRHG group (51.65±11.11 min) than that for the control group (65.71±18.07 min, p<0.05). MR 3D holograms may provide a clear and deeper anatomical perception of DORV and improve surgical planning.

V010 - 3D mixed reality guidance for percutaneous puncture during kidney stones surgical treatment

V010 - 3D mixed reality guidance for percutaneous puncture during kidney stones surgical treatment

3D mixed reality holograms for preoperative surgical planning of nephron-sparing surgery: evaluation of surgeons' perception.

3D reconstructions are gaining a wide diffusion in nephron-sparing surgery (NSS) planning. They have usually been studied on common 2D flat supports, with limitations regarding real depth comprehension and interaction. Nowadays, it is possible to visualize kidney 3D reconstructions as holograms in a "mixed reality" (MR) setting. The aim of this study was to test the face and content validity of this technology, and to assess the role of 3D holograms in aiding preoperative planning for highly complex renal tumors amenable by NSS. We evaluated surgeons' perception of mixed reality for partial nephrectomy during a urological international meeting organized at our Institution in January 2019. Thanks to the images of preoperative CT, hyper-accuracy 3D (HA3DTM) reconstructions were performed. Then, a virtual environment was created, and it interacted with the models in mixed reality setting by using HoloLens. We submitted to all the attendees a questionnaire, expressed by the Likert scale (1-10), about their opinion over the use and application of the MR. Moreover, the attendees had the chance to perform a first-hand MR experience; then, they were asked to choose their clamping and resection approach. Overall 172 questionnaires were collected. The scores obtained regarding both surgical planning (scored 8/10) and anatomical accuracy (9/10) were very positive. High satisfaction toward the potential role of this technology in surgical planning and understanding of surgical complexity (both scored 9/10) were expressed. After a first-hand experience with HoloLens and MR, 64.4% and 44.4% of the surgeons changed their clamping and resection approach, respectively - compared to CT image visualization only - choosing a more selective one. Our study suggests that surgeons perceive holograms and MR as a useful and interesting tool for the preoperative setting before partial nephrectomy, in the direction of an ever more precise surgery.

3D mixed reality holograms for preoperative surgical planning of nephron-sparing surgery: evaluation of surgeons' perception.

BACKGROUND 3D reconstructions are gaining a wide diffusion in nephron-sparing surgery (NSS) planning. They have usually been studied on common 2D flat supports, with limitations regarding real depth comprehension and interaction. Nowadays, it is possible to visualize kidney 3D reconstructions as holograms in a (MR) setting. The aim of this study was to test the face and content validity of this technology, and to assess the role of 3D holograms in aiding preoperative planning for highly complex renal tumors amenable by NSS. METHODS We evaluated surgeons' perception of mixed reality for partial nephrectomy during a urological international meeting organized at our Institution in January 2019. Thanks to the images of preoperative CT, hyper-accuracy 3D (HA3DTM) reconstructions were performed. Then, a virtual environment was created, and it interacted with the models in mixed reality setting by using HoloLens. We submitted to all the attendees a questionnaire, expressed by the Likert scale (1-10), about their opinion over the use and application of the MR. Moreover, the attendees had the chance to perform a first-hand MR experience; then, they were asked to choose their clamping and resection approach. RESULTS Overall 172 questionnaires were collected. The scores obtained regarding both surgical planning (scored 8/10) and anatomical accuracy (9/10) were very positive. High satisfaction toward the potential role of this technology in surgical planning and understanding of surgical complexity (both scored 9/10) were expressed. After a first-hand experience with HoloLens and MR, 64.4% and 44.4% of the surgeons changed their clamping and resection approach, respectively - compared to CT image visualization only - choosing a more selective one. CONCLUSIONS Our study suggests that surgeons perceive holograms and MR as a useful and interesting tool for the preoperative setting before partial nephrectomy, in the direction of an ever more precise surgery.

From photo to 3D to mixed reality: A complete workflow for cultural heritage visualisation and experience

The domain of cultural heritage is on the verge of adopting immersive technologies; not only to enhance user experience and interpretation but also to satisfy the more enthusiastic and tech-savvy visitors and audiences. However, contemporary academic discourse seldom provides any clearly defined and versatile workflows for digitising 3D assets from photographs and deploying them to a scalable 3D mixed reality (MxR) environment; especially considering non-experts with limited budgets. In this paper, a collection of open access and proprietary software and services are identified and combined via a practical workflow which can be used for 3D reconstruction to MxR visualisation of cultural heritage assets. Practical implementations of the methodology has been substantiated through workshops and participants’ feedback. This paper aims to be helpful to non-expert but enthusiastic users (and the GLAM sector) to produce image-based 3D models, share them online, and allow audiences to experience 3D content in a MxR environment.

Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery.

Orthopaedic surgeons are still following the decades old workflow of using dozens of two-dimensional fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This Letter presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. A red–green–blue–depth camera is rigidly attached to a mobile C-arm and is calibrated to the cone-beam computed tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model-based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localising the entry point and orienting the surgical tool during screw placement. The authors validate the augmentation by measuring target registration error and also evaluate the tracking accuracy in the presence of partial occlusion.

A mixed reality 3D system for the integration of X3DoM graphics with real-time IoT data

A mixed reality system is a set of interlinked real-time sensors’ and actuators’ data presented within a virtual world with graphics and texts. This paper describes a methodology for the implementation of mixed reality systems that interconnect real-world IoT systems with 3D virtual worlds. In particular, this paper presents the steps for integrating X3DoM graphics with real-time data and discusses the associated technologies from a developer’s point of view. Such a mixed reality 3D system, interconnecting real-time sensors and actuators from a real-world wastewater management system, with all the corresponding components and controls was developed. This mixed reality 3D world was created using open source web technologies that provide users, in this case the building tenants, with the ability to monitor and control the water consumption of their homes at anytime and from anywhere. It provides not only a comprehensive monitoring 3D model of the real building management system but also an easy to use controlling platform, where the commands are executed in the real building infrastructure.

Teleoperating Assistive Robots: A Novel User Interface Relying on Semi-Autonomy and 3D Environment Mapping

[abstFig src='/00290002/11.jpg' width='300' text='User-assisted pick and place task' ] Despite remarkable progress of service robotics in recent years, it seems that a fully autonomous robot which would be able to solve everyday household tasks in a safe and reliable manner is still unachievable. Under certain circumstances, a robot’s abilities might be supported by a remote operator. In order to allow such support, we present a user interface for a semi-autonomous assistive robot allowing a non-expert user to quickly asses the situation on a remote site and carry out subtasks which cannot be finished automatically. The user interface is based on a mixed reality 3D environment and fused sensor data, which provides a high level of situational and spatial awareness for teleoperation as well as for telemanipulation. Robot control is based on low-cost commodity hardware, optionally including a 3D mouse and stereoscopic display. The user interface was developed in a human-centered design process and continuously improved based on the results of five evaluations with a total of 81 novice users.