Stereoscopic Augmented Reality Research Articles

Spatial perception in stereoscopic augmented reality based on multifocus sensing

In many areas ranging from medical imaging to visual entertainment, 3D information acquisition and display is a key task. In this regard, in multifocus computational imaging, stacks of images of a certain 3D scene are acquired under different focus configurations and are later combined by means of post-capture algorithms based on image formation model in order to synthesize images with novel viewpoints of the scene. Stereoscopic augmented reality devices, through which is possible to simultaneously visualize the three dimensional real world along with overlaid digital stereoscopic image pair, could benefit from the binocular content allowed by multifocus computational imaging. Spatial perception of the displayed stereo pairs can be controlled by synthesizing the desired point of view of each image of the stereo-pair along with their parallax setting. The proposed method has the potential to alleviate the accommodation-convergence conflict and make augmented reality stereoscopic devices less vulnerable to visual fatigue.

STEREOSCOPIC AUGMENTED REALITY FOR INTRAOPERATIVE GUIDANCE IN ROBOTIC SURGERY

Augmented reality (AR) technology is increasingly adopted in the surgical field and recently it has been also introduced in robotic-assisted urologic surgery. This work describes the design and development of an AR intraoperative guide system with stereoscopic visualization (SAR, stereoscopic augmented reality) for the Da Vinci surgical robot. As a major novelty, the developed SAR system allows the surgeon to have the virtual 3D model of patient anatomy superimposed on the real field, without losing the stereoscopic view of the operative field. The workflow starts with the 3D model generation of the anatomical district of interest for surgery, from patient diagnostic imaging. Then, the 3D model is uploaded in the developed SAR application, navigated using a 3D space mouse, and superimposed to the operative field using computer vision algorithms. The SAR system was tested during 30 robot-assisted surgeries, including 20 partial nephrectomies, 1 kidney explant, and 9 radical prostatectomies. The SAR guidance system received overall great appreciation from surgeons and helped in localizing hidden structures, such as arteries or tumoral masses, increasing the understanding of surgical anatomy with depth perception, and facilitating intraoperative navigation. Future efforts will be addressed to improve the automatic superimposition of digital 3D models on the intraoperative view.

Quantitative evaluation of stereoscopic augmented reality visualization using an optical see-through head-mounted display

Purpose The objectives of the current study were to propose a calibration method for accurate augmented reality visualisation using an optical see-through head-mounted display, and to qualitatively evaluate visualisation accuracy for the application in computer assisted surgery. Methods An adaptation of stereo single-point active alignment method was proposed as the calibration procedure together with verification Three tests were performed: display of points in 3D space, on the plane, and on the surface of a skull phantom on inexperience users ( 5 participants) and experienced users ( 17 participants). Results The highest accuracy of visualization was obtained for skull phantom visualisation for an inexperienced user (3.00 mm, std 0.75 mm), while the lowest accuracy was obtained in a 3D visualisation test for an inexperienced user (22.95 mm, std 18.04 mm). The largest error was related to the depth component and amounted to 18.49 mm, std 18.10 mm for an inexperienced user. Conclusions It is possible to achieve relatively high visualisation accuracy (less than 5 mm for visualisation in space) for selected users but providing it for the group of inexperienced users seems to remain a major challenge. The accuracy of point indication can be substantially and statistically significantly increased by visualising objects on surfaces. The proposed methods and obtained results can serve as a basis for further implementation of augmented reality visualisation on an optical see-through head-mounted display in applications requiring high-quality augmented reality guidance of manual tasks.

Perceptual Guidelines for Optimizing Field of View in Stereoscopic Augmented Reality Displays

Near-eye display systems for augmented reality (AR) aim to seamlessly merge virtual content with the user’s view of the real-world. A substantial limitation of current systems is that they only present virtual content over a limited portion of the user’s natural field of view (FOV). This limitation reduces the immersion and utility of these systems. Thus, it is essential to quantify FOV coverage in AR systems and understand how to maximize it. It is straightforward to determine the FOV coverage for monocular AR systems based on the system architecture. However, stereoscopic AR systems that present 3D virtual content create a more complicated scenario because the two eyes’ views do not always completely overlap. The introduction of partial binocular overlap in stereoscopic systems can potentially expand the perceived horizontal FOV coverage, but it can also introduce perceptual nonuniformity artifacts. In this arrticle, we first review the principles of binocular FOV overlap for natural vision and for stereoscopic display systems. We report the results of a set of perceptual studies that examine how different amounts and types of horizontal binocular overlap in stereoscopic AR systems influence the perception of nonuniformity across the FOV. We then describe how to quantify the horizontal FOV in stereoscopic AR when taking 3D content into account. We show that all stereoscopic AR systems result in a variable horizontal FOV coverage and variable amounts of binocular overlap depending on fixation distance. Taken together, these results provide a framework for optimizing perceived FOV coverage and minimizing perceptual artifacts in stereoscopic AR systems for different use cases.

VE26 - RARP nerve sparing with 3D reconstruction and stereoscopic augmented reality

VE26 - RARP nerve sparing with 3D reconstruction and stereoscopic augmented reality

Understanding differences between combinations of 2D and 3D input and output devices for 3D data visualization

Focusing on interaction needs for scientific data exploration, we evaluated people’s performance using a 2D mouse, 3D SpaceMouse, or 3D-tangible tablet as input devices to interact with visualizations on 2D screens or stereoscopic augmented reality (AR) head-mounted displays. The increasing availability and power of immersive displays drives us to try to understand how to choose input devices, interaction techniques and output displays for the visualization of scientific data, thus to finally help us guide the interaction design for hybrid AR and PC visualization systems. With a docking task and a clipping plane placement/orientation task, we measure our participants’ performance (completion time and accuracy) with each of the different combinations of input and output. We also report on their perceived workload, their preference, and on other qualitative feedback. Results show that the mouse remains good with any display, especially for tasks that require a high accuracy. Our results highlight the potential to retain the mouse as a primary input device, and to complement it with other 3D interaction devices for specific uses.

When virtual and real worlds coexist: Visualization and visual system affect spatial performance in augmented reality.

New visualization approaches are being actively developed aiming to mitigate the effect of vergence-accommodation conflict in stereoscopic augmented reality; however, high interindividual variability in spatial performance makes it difficult to predict user gain. To address this issue, we investigated the effects of consistent and inconsistent binocular and focus cues on perceptual matching in the stereoscopic environment of augmented reality using a head-mounted display that was driven in multifocal and single focal plane modes. Participants matched the distance of a real object with images projected at three viewing distances, concordant with the display focal planes when driven in the multifocal mode. As a result, consistency of depth cues facilitated faster perceptual judgments on spatial relations. Moreover, the individuals with mild binocular and accommodative disorders benefited from the visualization of information on the focal planes corresponding to image planes more than individuals with normal vision, which was reflected in performance accuracy. Because symptoms and complaints may be absent when the functionality of the sensorimotor system is reduced, the results indicate the need for a detailed assessment of visual functions in research on spatial performance. This study highlights that the development of a visualization system that reduces visual stress and improves user performance should be a priority for the successful implementation of augmented reality displays.

The Effect of Stereoscopic Augmented Reality Visualization on Learning Anatomy and the Modifying Effect of Visual-Spatial Abilities: A Double-Center Randomized Controlled Trial.

Monoscopically projected three-dimensional (3D) visualization technology may have significant disadvantages for students with lower visual-spatial abilities despite its overall effectiveness in teaching anatomy. Previous research suggests that stereopsis may facilitate a better comprehension of anatomical knowledge. This study evaluated the educational effectiveness of stereoscopic augmented reality (AR) visualization and the modifying effect of visual-spatial abilities on learning. In a double-center randomized controlled trial, first- and second-year (bio)medical undergraduates studied lower limb anatomy with stereoscopic 3D AR model (n=20), monoscopic 3D desktop model (n=20), or two-dimensional (2D) anatomical atlas (n=18). Visual-spatial abilities were tested with Mental Rotation Test (MRT), Paper Folding Test (PFT), and Mechanical Reasoning (MR) Test. Anatomical knowledge was assessed by the validated 30-item paper posttest. The overall posttest scores in the stereoscopic 3D AR group (47.8%) were similar to those in the monoscopic 3D desktop group (38.5%; P=0.240) and the 2D anatomical atlas group (50.9%; P=1.00). When stratified by visual-spatial abilities test scores, students with lower MRT scores achieved higher posttest scores in the stereoscopic 3D AR group (49.2%) as compared to the monoscopic 3D desktop group (33.4%; P=0.015) and similar to the scores in the 2D group (46.4%; P=0.99). Participants with higher MRT scores performed equally well in all conditions. It is instrumental to consider an aptitude-treatment interaction caused by visual-spatial abilities when designing research into 3D learning. Further research is needed to identify contributing features and the most effective way of introducing this technology into current educational programs.

46.4: Compact Stereoscopic Augmented Reality Goggle with Single Display Panel

A light path for stereoscopic near eye display utilizing single display panel has been proposed and embedded in an existing snow goggle for AR display function with its original form factor and ID design intact. The prototype demonstrates sufficient quality to display numbers or symbols for data information or guiding.

Dual focal plane augmented reality interactive display with gaze-tracker

Stereoscopic augmented reality (AR) displays have a fixed focus plane and they suffer from visual discomfort due to vergence-accommodation conflict (VAC). In this study, we demonstrated a biocular (i.e. common optics for two eyes and same images are shown to both eyes) two focal-plane based AR system with real-time gaze tracker, which provides a novel interactive experience. To mitigate VAC, we propose a see-through near-eye display mechanism that generates two separate virtual image planes at arm’s length depth levels (i.e. 25 cm and 50 cm). Our optical system generates virtual images by relaying two liquid crystal displays (LCDs) through a beam splitter and a Fresnel lens. While the system is limited to two depths and discontinuity occurs in the virtual scene, it provides correct focus cues and natural blur effect at the corresponding depths. This allows the user to distinguish virtual information through the accommodative response of the eye, even when the virtual objects overlap and partially occlude in the axial direction. The system also provides correct motion parallax cues within the movement range of the user without any need for sophisticated head trackers. A road scene simulation is realized as a convenient use-case of the proposed display so that a large monitor is used to create a background scene and the rendered content in the LCDs is augmented into the background. Field-of-view (FOV) is 60 × 36 degrees and the eye-box is larger than 100 mm, which is comfortable enough for two-eye viewing. The system includes a single camera-based pupil and gaze tracker, which is able to select the correct depth plane based on the shift in the interpupillary distance with user’s convergence angle. The rendered content can be distributed to both depth planes and the background scene simultaneously. Thus, the user can select and interact with the content at the correct depth in a natural and comfortable way. The prototype system can be used in tasks that demand wide FOV and multiple focal planes and as an AR and vision research tool.

The Role of Camera Convergence in Stereoscopic Video See-through Augmented Reality Displays

In the realm of wearable augmented reality (AR) systems, stereoscopic video see-through displays raise issues related to the user’s perception of the three-dimensional space. This paper seeks to put forward few considerations regarding the perceptual artefacts common to standard stereoscopic video see-through displays with fixed camera convergence. Among the possible perceptual artefacts, the most significant one relates to diplopia arising from reduced stereo overlaps and too large screen disparities. Two state-of-the-art solutions are reviewed. The first one suggests a dynamic change, via software, of the virtual camera convergence, whereas the second one suggests a matched hardware/software solution based on a series of predefined focus/vergence configurations. Potentialities and limits of both the solutions are outlined so as to provide the AR community, a yardstick for developing new stereoscopic video see-through systems suitable for different working distances.

Sense of Immersion in Computer Games Using Single and Stereoscopic Augmented Reality

ABSTRACTAs a novel interaction and presentation technology, Augmented Reality (AR) has been widely used in education, tourism, entertainment, psychology, and many other applications. In the game industry, this revolutionary technology has introduced a new form of interaction between players and the gaming environment. In this article, we aim to find if exploiting AR in computer games can enhance the sense of immersion, and consequently, satisfying the players. To this end, a mobile game called Ladybug is implemented using three different techniques including the regular virtual reality (VR) camera, single AR camera, and stereoscopic AR camera. The results of a comprehensive user study on these techniques showed that the single AR camera and stereoscopic AR camera can result in more sense of immersion in players in comparison to the VR camera. However, the hypothesis that the stereoscopic AR results in more sense of presence in comparison to the single AR camera is rejected. On the other hand, our findings show that the AR stereoscopic camera can lead to a better emotional involvement and enjoyment in comparison to the single AR camera and VR camera.

Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution.

The purpose of this work was to develop a clinically viable laparoscopic augmented reality (AR) system employing stereoscopic (3-D) vision, laparoscopic ultrasound (LUS), and electromagnetic (EM) tracking to achieve image registration. We investigated clinically feasible solutions to mount the EM sensors on the 3-D laparoscope and the LUS probe. This led to a solution of integrating an externally attached EM sensor near the imaging tip of the LUS probe, only slightly increasing the overall diameter of the probe. Likewise, a solution for mounting an EM sensor on the handle of the 3-D laparoscope was proposed. The spatial image-to-video registration accuracy of the AR system was measured to be [Formula: see text] and [Formula: see text] for the left- and right-eye channels, respectively. The AR system contributed 58-ms latency to stereoscopic visualization. We further performed an animal experiment to demonstrate the use of the system as a visualization approach for laparoscopic procedures. In conclusion, we have developed an integrated, compact, and EM tracking-based stereoscopic AR visualization system, which has the potential for clinical use. The system has been demonstrated to achieve clinically acceptable accuracy and latency. This work is a critical step toward clinical translation of AR visualization for laparoscopic procedures.

Robust and Accurate Algorithm for Wearable Stereoscopic Augmented Reality with Three Indistinguishable Markers

In the context of surgical navigation systems based on augmented reality (AR), the key challenge is to ensure the highest degree of realism in merging computer-generated elements with live views of the surgical scene. This paper presents an algorithm suited for wearable stereoscopic augmented reality video see-through systems for use in a clinical scenario. A video-based tracking solution is proposed that relies on stereo localization of three monochromatic markers rigidly constrained to the scene. A PnP-based optimization step is introduced to refine separately the pose of the two cameras. Video-based tracking methods using monochromatic markers are robust to non-controllable and/or inconsistent lighting conditions. The two-stage camera pose estimation algorithm provides sub-pixel registration accuracy. From a technological and an ergonomic standpoint, the proposed approach represents an effective solution to the implementation of wearable AR-based surgical navigation systems wherever rigid anatomies are involved.

Stereoscopic augmented reality for single camera endoscopy: a virtual study

Endoscopic surgery causes less tissue injury compared to open surgical techniques, thus promoting more rapid recuperation and reduced post-operative pain. Endoscopy, however, allows the surgeon to visualise only the anatomical surface of the surgical site, with a relatively narrow field of view. Moreover, the 2D video captured by the conventional endoscope does not provide depth perception of the surgical scene. In this study, these limitations have been addressed with the development of an augmented reality (AR) system with stereoscopic visualisation. A phantom and its 3D CT model were used, respectively, to form the real and virtual parts of the AR. The virtual environment camera pose was tracked using algorithms for image feature detection, feature matching and Perspective-n-Point applied on the endoscopic image and the 3D virtual model-rendered image. The endoscope video frame- and the virtual model-rendered images were superimposed to form the AR composite view. The depth buffer (z-buffer) of the rendering window was further used to make a stereo pair of the AR image. The AR system produced a stereo composite view having well-aligned real and virtual components. The RMS error of the real and virtual image contours registration was 9.6 ± 6.7 mm. Correlation coefficients between the depth map from z-buffer and a depth camera was between 0.60 and 0.96 (p < 0.05). The AR system requires further improvement to be applicable at a higher frame rate of endoscope image acquisition. It also needs to include motion and deformation models when applied to animals or patients.

3D Stereoscopic Augmented Reality with a Monocular Camera

3D Stereoscopic Augmented Reality with a Monocular Camera

Stereoscopic augmented reality in visual interface for flight control

Head-up displays (HUD) become common equipment in aircraft cockpits. One of the possible and adequate to the HUD specifics visual pilot's interfaces is the “tunnel-in-the-sky” (3D structure on a flat screen). But according to the recent studies the tunnel-in-the-sky does not provide crucial advantages in comparison with traditional methods of presenting navigation information to pilots. This paper considers a stereoscopic version of the “tunnel-in-the-sky” realized as an augmented reality (AR) pocket-size system with see-through light-weight glasses. It consists of low-cost elements and is free of drawbacks that are inherent in the existing synthetic/enhanced vision systems. The design of experiments with desktop simulators of different AR pilot's interfaces (2D, 3D and stereo 3D conditions) and their results are described. The experiment results prove the effectiveness of the proposed stereo solution.

Stereoscopic augmented reality for laparoscopic surgery

Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field. The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine. The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy. We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

Calibrated depth and color cameras for accurate 3D interaction in a stereoscopic augmented reality environment

A Human–machine interaction system requires precise information about the user’s body position, in order to allow a natural 3D interaction in stereoscopic augmented reality environments, where real and virtual objects should coherently coexist. The diffusion of RGB-D sensors seems to provide an effective solution to such a problem. Nevertheless, the interaction with stereoscopic 3D environments, in particular in peripersonal space, requires a higher degree of precision. To this end, a reliable calibration of such sensors and an accurate estimation of the relative pose of different RGB-D and visualization devices are crucial. Here, robust and straightforward procedures to calibrate a RGB-D camera, to improve the accuracy of its 3D measurements, and to co-register different calibrated devices are proposed. Quantitative measures validate the proposed approach. Moreover, calibrated devices have been used in an augmented reality system, based on a dynamic stereoscopic rendering technique that needs accurate information about the observer’s eyes position.

Stereoscopic augmented reality for da Vincii™ robotic biliary surgery

INTRODUCTIONNew laparoscopic techniques put distance between the surgeon and his patient. PRESENTATION OF CASE3D volume rendered images directly displayed in the da Vinci surgeon's console fill this gap by allowing the surgeon to fully immerse in its intervention. DISCUSSIONDuring the robotic operation the surgeon has a greater control on the procedure because he can stay more focused not being obliged to turn is sight out of his operative field. Moreover, thanks to depth perception of the rendered images he had a precise view of important anatomical structures. CONCLUSIONWe describe our preliminary experience in the quest of computer-assisted robotic surgery.