Holographic Augmented Reality Research Articles

Full-colour 3D holographic augmented-reality displays with metasurface waveguides

Emerging spatial computing systems seamlessly superimpose digital information on the physical environment observed by a user, enabling transformative experiences across various domains, such as entertainment, education, communication and training1–3. However, the widespread adoption of augmented-reality (AR) displays has been limited due to the bulky projection optics of their light engines and their inability to accurately portray three-dimensional (3D) depth cues for virtual content, among other factors4,5. Here we introduce a holographic AR system that overcomes these challenges using a unique combination of inverse-designed full-colour metasurface gratings, a compact dispersion-compensating waveguide geometry and artificial-intelligence-driven holography algorithms. These elements are co-designed to eliminate the need for bulky collimation optics between the spatial light modulator and the waveguide and to present vibrant, full-colour, 3D AR content in a compact device form factor. To deliver unprecedented visual quality with our prototype, we develop an innovative image formation model that combines a physically accurate waveguide model with learned components that are automatically calibrated using camera feedback. Our unique co-design of a nanophotonic metasurface waveguide and artificial-intelligence-driven holographic algorithms represents a significant advancement in creating visually compelling 3D AR experiences in a compact wearable device.

Waveguide holography for 3D augmented reality glasses

Near-eye displays are fundamental technology in the next generation computing platforms for augmented reality and virtual reality. However, there are remaining challenges to deliver immersive and comfortable visual experiences to users, such as compact form factor, solving vergence-accommodation conflict, and achieving a high resolution with a large eyebox. Here we show a compact holographic near-eye display concept that combines the advantages of waveguide displays and holographic displays to overcome the challenges towards true 3D holographic augmented reality glasses. By modeling the coherent light interactions and propagation via the waveguide combiner, we demonstrate controlling the output wavefront using a spatial light modulator located at the input coupler side. The proposed method enables 3D holographic displays via exit-pupil expanding waveguide combiners, providing a large software-steerable eyebox. It also offers additional advantages such as resolution enhancement capability by suppressing phase discontinuities caused by pupil replication process. We build prototypes to verify the concept with experimental results and conclude the paper with discussion.

Design and Development of an Augmented Reality Interface for Space Exploration

The Holographic Augmented Reality Visualization Interface for Exploration (HARVIE) was developed for the 2022 NASA SUITS (Spacesuit User Interface for Students) challenge. HARVIE assists astronauts with elevated demands of the lunar surface through navigation, terrain sensing, and an optimal display of suit status elements (e.g., oxygen, battery, and heart rate). Considering environmental constraints, the system architecture promotes efficient cross modal communication between the mission control center, other astronauts, and the user interface. Currently, the system utilizes a hands-free modality such as speech recognition. Throughout the design process, we conducted heuristic evaluations on a low-fidelity prototype. Then, we implemented HARVIE into a high-fidelity prototype on the HoloLens 2 and utilized the Rapid Iterative Testing & Evaluation (RITE) method for human-in-the-loop testing. Lastly, we evaluated our final design at NASA Johnson Space Center. Our interface serves as a novel approach to enhance how astronauts navigate on missions using augmented reality.

Swave Photonics Secures €10M Seed Funding

Funding to support Holographic Augmented Reality development

Eyebox enlargement in holographic AR glasses.

Despite the high output characteristics of holographic optical elements (HOEs), there are still no affordable holographic AR glasses that combine qualities such as a wide field of view (FOV) and a large eyebox (EB). In this study, we propose an architecture for holographic augmented reality glasses that covers both needs. Our solution is based on the combination of an axial HOE with a directional holographic diffuser (DHD) illuminated by a projector. A transparent-type DHD redirects the light from the projector, increasing the angular aperture of the image beams and providing a large EB. A reflection-type axial HOE redirects the light, transforming the spherical beams into parallel ones and providing a wide FOV for the system. The main feature of our system is the coincidence of the DHD position with the planar intermediate image of the axial HOE. This unique condition makes the system free of off-axial aberrations and ensures high output characteristics. The proposed system has a horizontal FOV of 60deg and an EB width of 10mm. We used modeling and a prototype to prove our investigations.

High Resolution Multiview Holographic Display Based on the Holographic Optical Element.

Limited by the low space-bandwidth product of the spatial light modulator (SLM), it is difficult to realize multiview holographic three-dimensional (3D) display. To conquer the problem, a method based on the holographic optical element (HOE), which is regarded as a controlled light element, is proposed in the study. The SLM is employed to upload the synthetic phase-only hologram generated by the angular spectrum diffraction theory. Digital grating is introduced in the generation process of the hologram to achieve the splicing of the reconstructions and adjust the position of the reconstructions. The HOE fabricated by the computer-generated hologram printing can redirect the reconstructed images of multiview into multiple viewing zones. Thus, the modulation function of the HOE should be well-designed to avoid crosstalk between perspectives. The experimental results show that the proposed system can achieve multiview holographic augmented reality (AR) 3D display without crosstalk. The resolution of each perspective is 4K, which is higher than that of the existing multiview 3D display system.

Dual‐view holographic AR display based on Bragg mismatched reconstruction of the holographic optical element

AbstractA method for dual‐view holographic display based on Bragg mismatched reconstruction of holographic optical element (HOE) is proposed. Under the Bragg mismatched condition, the reconstructed images are guided into two separated viewing zones to realize dual‐view holographic display. Meanwhile, the viewing angle of each perspective is increased to 11.2°, which is almost 2.5 times as large as the traditional holographic display system. The design process of HOE is simple only by interference of plane reference wave and converging spherical signal wave, which has high practicability. Furthermore, the HOE can mix the virtual 3D image with real‐world scenes, which could implement augmented reality (AR) display. Experiments validate that the proposed system can achieve dual‐view holographic AR three‐dimensional (3D) display with accommodation effect.

Synergistic Digital Twin and Holographic Augmented-Reality-Guided Percutaneous Puncture of Respiratory Liver Tumor

Thermal ablation is an exciting new minimally invasive treatment that destroys liver tumors without removing them. It uses image guidance to place a needle through the skin into a liver tumor, which is highly dependent on surgeons’ experience. With the development of digital medicine, augmented reality (AR) has become a more intuitive and safer way to achieve real-time navigation. However, the technology is still in its infancy due to its limited accuracy and real-time performance. To address these problems, we syncretized the holographic AR with the digital twin technique to track the dynamic surgical scene and provide the 3-D navigation of heterogeneous target regions via internal motion prediction. To tackle the dilemma of real-time performance and precise internal motion estimation, a dynamic adaptation scheme is proposed to compensate for the time cost induced by the external/internal correlation model and data transmission. We carried out a series of experiments to validate our methods. With the proposed external/internal correlation model, the average estimation errors of the tumor and vessels are 2.18 and 2.79 mm, respectively. Besides, we performed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in vivo</i> experiments on two beagle dogs with an artificial lesion in their liver, respectively, and the puncture accuracy of our method are 2.5 and 2.17 mm. The results show that on one hand, our method can fulfill the real-time requirement of AR via using the intraoperative data, which is also more precise than that with preoperative data. On the other hand, our method can provide more 3-D information for surgeons, such as vessels, which can well ensure the safety of operation.

Binocular holographic display based on the holographic optical element.

Due to the limited pixel pitch of the spatial light modulator (SLM), the field of view (FOV) is insufficient to meet binocular observation needs. Here, an optimized controlling light method of a binocular holographic three-dimensional (3D) display system based on the holographic optical element (HOE) is proposed. The synthetic phase-only hologram uploaded onto the SLM is generated with the layer-based angular spectrum diffraction theory, and two different reference waves are introduced to separate the left view and the right view of the 3D scene. The HOE with directional controlling light parameters is employed to guide binocular information into the left-eye and the right-eye viewing zones simultaneously. Optical experiments verify that the proposed system can achieve binocular holographic augmented reality 3D effect successfully with real physical depth, which can eliminate the accommodation-vergence conflict and visual fatigue problem. For each perspective, the FOV is 8.7° when the focal length of the HOE is 10 cm. The width of the viewing zone is 2.3 cm when the viewing distance is 25 cm.

Multiplane holographic augmented reality head-up display with a real-virtual dual mode and large eyebox.

We propose a multiplane augmented reality (AR) head-up display (HUD) with a real-virtual dual mode based on holographic optical elements (HOEs). The picture generation unit (PGU) is only a single free-focus projector, and the optical combiner includes a HOE lens (HOEL) for long-distance virtual image display and a HOE diffuser (HOED) for in-plane real image display. A HOED with directional scattering characteristics in the real image mode can significantly increase the size of the eyebox (EB) without increasing the size of the HOE, and a HOEL with a flexible design for the optical focal length in the virtual image mode can be used to achieve a different depth of the AR display. The proposed AR HUD system, which has a compact structure and offers high light transmittance, high energy usage, a multiplane display, and a large EB, is expected to be widely used in the future.

Holographic near-eye display system with large viewing area based on liquid crystal axicon.

In this paper, a liquid crystal axicon based holographic near-eye display system with large viewing area is proposed. The viewing area of the system is extended by implementing the liquid crystal axicon. The error diffusion algorithm is used to calculate the computer-generated hologram (CGH). When incident on the liquid crystal axicon placed at the back focal plane of Fourier lens, the reconstruction light modulated by the CGH is deflected into two directions resulting in a viewing area extension. Meanwhile, to illustrate the potential of the proposed system, two-dimensional viewing area extension is demonstrated. It combines the frequency spectrum shift with the proposed system and achieves a double expansion of the horizontal viewing area and three-times expansion of the vertical viewing area. Feasibility of the proposed system is verified by optical experiments. The proposed system has potential applications in holographic augmented reality (AR) display.

Advances in perforator imaging through holographic CTA and augmented reality: a systematic review

**Introduction** Free tissue transfer has become a mainstay in reconstructive plastic surgery, and techniques to plan such surgery continue to evolve. Novel technologies and increases in computational power have enabled computed tomographic angiography (CTA) data augmentation onto patients to assist in pedicle identification and dissection. Given the rapidly evolving field and research in this domain, a systematic review was un-undertaken to establish the evidence for its usefulness in pedicle identification and dissection.**Methods** An extensive search using keywords in EMBASE and PubMed® with bibliographic linkage following PRISMA guidelines was performed. We identified 107 articles. Duplicate articles were removed prior to review. Two reviewers independently screened the titles for appropriate topic relevance. Full articles were then screened for review. **Results** Eleven articles were appropriate for review. Two articles analysed the time taken to identify perforators using augmented reality (AR) compared to Doppler ultrasound. The remainder of the articles analysed time to perforator identification, differences between projected location and dissected perforator location, qualitative feedback from surgeons on the use of AR systems for perforator identification and proof of concept and the usefulness of AR in perforator flap surgery. **Conclusion** This review demonstrates that while established methods of data rendering and projection can achieve holographic projection and AR, there is a lack of objective outcome data to demonstrate its usefulness. This, combined with a cost analysis, are the main obstructions to this technology being more widely adopted.

Holographic augmented reality display with conical holographic optical element for wide viewing zone

In this study, we propose a holographic augmented reality (AR) display with a wide viewing zone realized by using a special-designed reflective optical element. A conical holographic optical element (HOE) is used as such a reflective optical element. This conical HOE was implemented to reconstruct a diverging spherical wave with a wide spread angle. It has a sharp wavelength selectivity by recording it as a volume hologram, enabling augmented reality (AR) representation of real and virtual 3D objects. The quality of the generated spherical wave and the spectral reflectivity of the fabricated conical HOE were investigated. An optical superimposition between real and virtual 3D objects was demonstrated, thereby enhancing the validity of our proposed method. A horizontal viewing zone of 140° and a vertical viewing zone of 30° were experimentally confirmed. The fabrication procedure for the conical HOE is presented, and the calculation method of the computer-generated hologram (CGH) based on Fermat’s principle is explained in detail.

Feasibility of the Application of Holographic Augmented Reality in Endovascular Surgery Using Microsoft HoloLens Head-Mounted Display

Advances in virtual, augmented (AR) and mixed reality have led to the development of wearable technologies including head mounted displays (HMD). The aim of this study was to investigate the feasibility to use HMD during endovascular surgery. We propose an adaptation of AR-HMD using Microsoft HoloLens. Software was developed to enable visualization of the vascular system during endovascular procedures. A video was performed to present an overview of the device and show its use in real conditions. The device allowed a successful visualization of perioperative angiography during peripheral angioplasty, carotid angioplasty and aortic aneurysm endovascular repair. The device was operated on voice command, preserving the environment sterility. This video illustrated the feasibility of the application of holographic AR during endovascular intervention and brings perspectives to use artificial-intelligence derived tools for image-guided surgery.

Evaluation of how users interface with holographic augmented reality surgical scenes: Interactive planning MR-Guided prostate biopsies.

User interfaces play a vital role in the planning and execution of an interventional procedure. The objective of this study is to investigate the effect of using different user interfaces for planning transrectal robot-assisted MR-guided prostate biopsy (MRgPBx) in an augmented reality (AR) environment. End-user studies were conducted by simulating an MRgPBx system with end- and side-firing modes. The information from the system to the operator was rendered on HoloLens as an output interface. Joystick, mouse/keyboard, and holographic menus were used as input interfaces to the system. The studies indicated that using a joystick improved the interactive capacity and enabled operator to plan MRgPBx in less time. It efficiently captures the operator's commands to manipulate the augmented environment representing the state of MRgPBx system. The study demonstrates an alternative to conventional input interfaces to interact and manipulate an AR environment within the context of MRgPBx planning.

A Holographic Augmented Reality Interface for Visualizing of MRI Data and Planning of Neurosurgical Procedures.

The recent introduction of wireless head-mounted displays (HMD) promises to enhance 3D image visualization by immersing the user into 3D morphology. This work introduces a prototype holographic augmented reality (HAR) interface for the 3D visualization of magnetic resonance imaging (MRI) data for the purpose of planning neurosurgical procedures. The computational platform generates a HAR scene that fuses pre-operative MRI sets, segmented anatomical structures, and a tubular tool for planning an access path to the targeted pathology. The operator can manipulate the presented images and segmented structures and perform path-planning using voice and gestures. On-the-fly, the software uses defined forbidden-regions to prevent the operator from harming vital structures. In silico studies using the platform with a HoloLens HMD assessed its functionality and the computational load and memory for different tasks. A preliminary qualitative evaluation revealed that holographic visualization of high-resolution 3D MRI data offers an intuitive and interactive perspective of the complex brain vasculature and anatomical structures. This initial work suggests that immersive experiences may be an unparalleled tool for planning neurosurgical procedures.

A Holographic Augmented Reality Guidance System for Patient Alignment: A Feasibility Study.

PurposeTo evaluate the accuracy of an augmented reality holographic guidance system for potential use in patient alignment in radiotherapy applications.MethodsA cubic phantom was scanned on a CT simulator and a 3D mesh was extracted using the Eclipse Scripting API. An application was created for the Microsoft HoloLens to allow users to see the scanned mesh as a hologram overlaid in the treatment vault. Six therapists were equipped with the HoloLens glasses and instructed to move the real phantom to align with the perceived spatial hologram using only couch controls. The initial couch coordinates were recorded and then recorded at each step as the therapist moved the phantom to each new location. The application varied the position of the virtual phantom to 10 preprogrammed locations within a 40-cm cubic volume in a combination of vertical, longitudinal, and lateral axis shifts. The absolute position difference between the holographic world and real-world phantom was recorded at each step. Also, the relative position from one position to the next was recorded.ResultsFifty shifts were collected across the six therapists. The mean difference between the physical position and instructed holographic position was 0.58 ± 0.31 cm for relative shifts and 0.51 ± 0.33 cm for absolute position. The maximum difference between the holographic position and the actual post shift position was 1.53 cm for relative and 1.58 cm for absolute.ConclusionHolographic augmented reality guidance using the Microsoft HoloLens provides adequate accuracy for initial treatment alignment but lacks the fine alignment accuracy of X-ray imaging systems.

42.3: Compact Design of Holographic Augmented Reality Display Using Bragg Grating Noise Filter and Holographic Lens

42.3: Compact Design of Holographic Augmented Reality Display Using Bragg Grating Noise Filter and Holographic Lens

Trans-catheter treatments of superior sinus venosus atrial septal defects

Superior sinus venosus atrial septal defect is a communication between the two atria located above the upper margin of the oval fossa, immediately caudal to the junction of the superior vena cava and the right atrium. Commonly, it is associated with partial anomalous pulmonary venous drainage, particularly of the right upper pulmonary vein.Surgical repair has been the gold-standard approach to close this congenital anomaly, with excellent results. However, some recent studies have shown promising results from a trans-catheter approach by placing a covered stent in the superior vena cava to close the defect and redirect the anomalous pulmonary venous return posterior to the stent and into the left atrium. This procedure has proven effective and feasible in selected patients. Additionally, the development of a percutaneous approach has opened the possibility of treating patients with contraindications to surgical correction.Procedural planning using 3D printed models, simulation, holographic augmented reality, and interrogation of the superior vena cava with a balloon has made it possible to increase the effectiveness and safety of the transcatheter correction of this type of atrial septal defects.Based on our experience and literature data, transcatheter correction of sinus venosus atrial septal defect with partial anomalous pulmonary venous drainage may be an alternative to surgery in adult patients and would reduce the need for cardiopulmonary bypass. The short and medium-term results are promising, but longer follow-up studies on the use of this technique are needed to validate the procedure.

Holographic Augmented Reality for DIEP Flap Harvest.

During a deep inferior epigastric perforator (DIEP) flap harvest, the identification and localization of the epigastric arteries and its perforators are crucial. Holographic augmented reality is an innovative technique that can be used to visualize this patient-specific anatomy extracted from a computed tomographic scan directly on the patient. This study describes an innovative workflow to achieve this. A software application for the Microsoft HoloLens was developed to visualize the anatomy as a hologram. By using abdominal nevi as natural landmarks, the anatomy hologram is registered to the patient. To ensure that the anatomy hologram remains correctly positioned when the patient or the user moves, real-time patient tracking is obtained with a quick response marker attached to the patient. Holographic augmented reality can be used to visualize the epigastric arteries and its perforators in preparation for a deep inferior epigastric perforator flap harvest. Potentially, this workflow can be used visualize the vessels intraoperatively. Furthermore, this workflow is intuitive to use and could be applied for other flaps or other types of surgery.