Near-eye Display Research Articles

Holographic near-eye display system with large viewing area based on liquid crystal axicon.

In this paper, a liquid crystal axicon based holographic near-eye display system with large viewing area is proposed. The viewing area of the system is extended by implementing the liquid crystal axicon. The error diffusion algorithm is used to calculate the computer-generated hologram (CGH). When incident on the liquid crystal axicon placed at the back focal plane of Fourier lens, the reconstruction light modulated by the CGH is deflected into two directions resulting in a viewing area extension. Meanwhile, to illustrate the potential of the proposed system, two-dimensional viewing area extension is demonstrated. It combines the frequency spectrum shift with the proposed system and achieves a double expansion of the horizontal viewing area and three-times expansion of the vertical viewing area. Feasibility of the proposed system is verified by optical experiments. The proposed system has potential applications in holographic augmented reality (AR) display.

Design, analysis and optimization of a waveguide-type near-eye display using a pin-mirror array and a concaved reflector

Waveguides have become one of the most promising optical combiners for see-through near-eye displays due to the thickness, weight, and transmittance. In this study, we propose a waveguide-type near-eye display using a pin-mirror array and a concaved reflector with a compact outlook, optimized image uniformity and stray light. Issues have been discussed in detail, which include field of view (FOV), eye-box, resolution, depth of field (DOF), display uniformity and stray light artifacts. It can be shown that the DOF can be extended (when compared with traditional waveguide-type near-eye displays) to alleviate the vergence-accommodation conflict (VAC) problem, and the uniformity & stray light can be improved with an optimal structure. Moreover, reflective surfaces have been introduced as the input and output coupling with a compact outlook, an easy-processing structure and the achromatic performance. A prototype based on the proposed method have been successfully developed, and virtual images with an extended DOF can be shown along with the real-world.

Real scene acquisition and holographic near-eye display system based on a zoom industrial endoscope.

In this paper, we propose a real scene acquisition and holographic near-eye display system based on a zoom industrial endoscope. By controlling the driving current of the liquid lens, the working distance and focal length of the zoom industrial endoscope can be tuned accordingly. Thus, the object at different depths can be captured. Then, the sub-sampling algorithm is used to generate the hologram. By adjusting the hologram sampling rate of the objects with different depths, the holographic near-eye 3D display can be realized. Experimental results demonstrate that the working distance of the zoom industrial endoscope can be tuned from 20 mm to 200 mm with the driving current changing from 80 mA to 190 mA. With the proposed system, the human eye can intuitively see the depth relationships among the real objects. The proposed system is expected to be applied to 3D display and industrial inspection fields.

Enlarging the FOV of lensless holographic retinal projection display with two-step Fresnel diffraction

The field of view (FOV) of holographic retinal projection display (RPD) is always restricted by the diffraction angle of a spatial light modulator (SLM). In this paper, we demonstrate a large FOV holographic RPD by using two-step Fresnel diffraction calculation. By adding the pupil plane as the intermediate plane and decreasing the sampling interval, the FOV can be enlarged to nearly two times of the diffraction angle limit without any physical change. Due to the added spherical wave phase, the influence of aliasing is eliminated because the adjacent orders are projected to different viewpoints. The nonuniform image intensity caused by the sinc function modulation is compensated by image pre-processing. Optical experiments verify that the proposed method can present a uniform holographic near-eye display with a large FOV and adjustable viewpoint position.

216. The arrival of augmented reality in MIS: initial results of use for percutaneous pedicle screw instrumentation

<h3>BACKGROUND CONTEXT</h3> Augmented reality (AR) employs computer-generated sensory inputs to enhance the user's perception of his or her environment. Pedicle screw instrumentation has become a mainstay of spinal stabilization. In general, pedicle screw accuracy rates have improved with increasing use of technology; navigation-based instrumentation have been described as accurate in 89-100% of cases. Some inherent difficulties persist with regards to ergonomics and efficiency of spinal navigation. Traditional navigation requires removing focus from the surgical field to interpret and coordinate input of external visual data. Line-of-sight disturbance and attention-shift has known impact on cognitive load and optimization of workflow in surgery. Emerging AR technology in spine surgery aims to reduce the impact of these pitfalls. <h3>PURPOSE</h3> To describe the first known series of in vivo pedicle screws placed percutaneously using AR technology for minimally invasive surgery (MIS) applications. <h3>STUDY DESIGN/SETTING</h3> After IRB approval, three senior surgeons at two institutions collected cases from June, 2020 - October, 2021, in which AR was used for placement of percutaneous pedicle screw instrumentation. <h3>PATIENT SAMPLE</h3> A total of 148 total MIS cases were studied; the average patient age was 59.7 ±13.84 and average BMI was 30.8 ±7.2. <h3>OUTCOME MEASURES</h3> Patient demographics and surgical metrics including total posterior construct time (defined as time elapsed from pre-incision instrument registration, percutaneous posterior fusion and final screw placement) were recorded. Screw revision intraoperatively after fluoroscopic evaluation and revision rates postoperatively were recorded. <h3>METHODS</h3> The xvision^TM (Augmedics Inc., Arlington Heights, IL, USA) system features a wireless headset with transparent near-eye display which projects intraoperative 3D imaging directly onto the surgeon's retina. After patient positioning, one percuntaneous and one superficial reference marker are placed. Intra-operative CT data is processed to the headset and integrates into the surgeon's visual field creating a "see-through" 3D effect in addition to 2D standard navigation images. MIS pedicle screw placement is then carried out percutaneously through single line of sight using navigated instruments. <h3>RESULTS</h3> Of the 148 cases, 139 (93.9%) were performed for degenerative pathology, 6 (4.1%) for tumor and 3 (2.0%) for spinal deformity. A total of 535 pedicle screws were placed; 519 (97.0%) lumbar and 16 (3.0%) thoracic. Eighty-four (58.8%) cases were 1-level, 42 (28.4%) were 2-level, 16 (10.8%) were 3-level and 6 (4.1%) were 4-level, equating to an average of 3.51 screws per case. A total of one screw (0.20%) was replaced intraoperatively after examining position with fluoroscopy. No instrumentation was revised postoperatively at final available follow-up ranging from 3-18 months. The mean total posterior construct time was 5 minutes and 56 seconds per screw overall; cases later in the learning curve achieved total posterior construct mean time of 2 minutes and 51 seconds per screw. <h3>CONCLUSIONS</h3> This study represents the first report of the use of AR to place MIS pedicle screws. These data support an exceedingly low rate of complications with screw placement and confirms a rapid learning curve with a high level of surgical efficiency demonstrated. These advances are achieved with an image- and implant-agnostic system which requires no physical footprint in the operating room. <h3>FDA DEVICE/DRUG STATUS</h3> xvision^TM (Augmedics Inc., Arlington Heights, IL, USA) (Approved for this indication)

Two‐Dimensional Angular Bandwidth Broadening of Metasurface Grating

Waveguide‐type near‐eye displays (NED) have attracted increasing attention in the market of augmented reality (AR) and virtual reality (VR) devices, thanks to their outstanding performance within ultra‐compact size. The performance of waveguide‐type NED mainly depends on the waveguide coupler that transmits the incident image to the user's eye through the total internal reflection (TIR) inside the waveguide. However, the waveguide couplers, that have been widely used, exhibit nonuniform diffraction efficiency within the transferable angular range, i.e., field of view (FoV) of waveguide‐type NED, so that the image intensity distribution distortion is induced. Thus, it is needed that many researchers focus on broadening angular bandwidth to overcome the non‐uniformity of the diffraction efficiency. Herein, the 2D angular bandwidth broadening metasurface grating that exhibits uniform and high diffraction efficiency for the 2D FoV range is proposed. Specifically, the proposed device is designed specifically, and the diffraction efficiency of this device shows uniform efficiency of about 75% with a variance of 0.25% for the FoV range, and it is demonstrated both numerically and experimentally. Consequently, it is expected that the proposed device can be a game‐changer in the market for waveguide‐type NED for the enhancement of image uniformity.

Compact full-color augmented reality near-eye display using freeform optics and a holographic optical combiner.

We develop a compact full-color augmented reality near-eye display system with a multicolor holographic optical combiner and a freeform relay system. The digital image is produced by a full-color micro organic light-emitting diode (Micro-OLED) display module. The freeform relay system includes four freeform optics and a holographic optical mirror, which are employed to correct both the monochromatic and chromatic aberrations caused by the holographic optical combiner. The two multicolor holographic mirrors have a three-layer laminated structure and are delicately fabricated to yield an improved diffractive efficiency and a reduced efficiency difference for red, green, and blue colors. The high degrees of freedom of freeform optics, and the thin and light nature of the holographic optical combiner yield a compact form factor near-eye display system with a diagonal field of view (FOV) of 20° and the eye-box of 5 mm × 5 mm. Two prototypes are built to demonstrate the feasibility of the proposed display system.

Near-eye display with a triple-channel waveguide for metaverse.

We present a near-eye display featuring a triple-channel waveguide with chiral liquid crystal gratings. Our triple-channel waveguide is capable of dividing one field of view into three through both the polarization orthogonality and angular separation. To illustrate its principle, a k-space diagram, which takes into account the aspect ratio of field of view, is depicted. Our results demonstrate that its diagonal field of view reaches 90°, eye relief is 10 mm, exit pupil is 4.9 × 4.9 mm2, transmittance is 4.9%, and uniformity is 89%.

Convolutional Neural Network for Phase-Only Hologram Optimization Based on the Point Source Method With the Holographic Viewing-Window

An optimization algorithm for phase-only hologram (POH) based on the point source method (PSM) with the holographic viewing-window (HVW) by using a convolutional neural network (CNN) is proposed. The network training adopts an unsupervised strategy that maps the target image to optimized constant phases (OCPs) corresponding to the wavefront of each point source instead of random constant phases (RCPs), and then the optimized POH can be obtained by wavefront superposition. The simulation and experimental results show that the speckle noise of the reconstruction is significantly suppressed compared with the initial random phase method (RPM), which demonstrates the feasibility of the proposed method and shows a strong generalization ability of the trained CNN. The HVW of a holographic near-eye display (NED) has a continuous observation range of about 6 mm in the horizontal direction without using additional optical expansion elements, and a complete high-quality image can be observed within this range.

Simultaneous multi-channel near-eye display: a holographic retinal projection display with large information content

Augmented reality (AR) near-eye displays (NEDs) are emerging as the next-generation display platform. The existing AR NED only present one single video channel at a time, same as traditional media such as TVs and smartphones. In this Letter, to the best of our knowledge, we propose for the first time a multi-channel holographic retinal projection display (RPD), which can provide multi-channel image sources simultaneously, thus greatly increasing the information content. Due to the superposition capacity of a hologram, multiple images are projected to different viewpoints simultaneously through multiple spherical wave encoding, so that the viewer can switch among playing channels very fast through eye rotation. A full-color dynamic multi-channel holographic near-eye display is demonstrated in the optical experiment. The proposed method provides a good prospect that the future AR glasses can play dozens of video channels in parallel, and the user can switch among channels freely and efficiently just through a simple eye rotation.

Design of a dual focal-plane near-eye display using diffractive waveguides and multiple lenses.

We propose a method to construct a compact dual focal-plane optical see-through near-eye display using diffractive waveguides and multiple lenses. A virtual image from a display device is projected into a three-grating waveguide using an objective lens, and a virtual image can be shown at a far distance with an extended eye box. One negative lens is employed to reduce the focus distance of the virtual image, and a corresponding positive lens is used to compensate for the distortion and accommodation errors. Thus, not only can a virtual image with a near distance be achieved, but also a virtual plane with a further distance can be generated by introducing another projection module and waveguide. Only two waveguides and two pieces of lenses are used in front of one eye to obtain a lightweight outlook. To verify the proposed method, a proof-of-concept prototype was developed to provide vivid virtual images at different depths in front of the human eye.

Design method of a wide-angle AR display with a single-layer two-dimensional pupil expansion geometrical waveguide.

Waveguide near-eye displays (NEDs) consist of a planar waveguide combiner and a coupling-in projection system. A two-dimensional geometrical waveguide (TDGW) can achieve an ultra-thin, large exit pupil diameter (XPD), wide-angle NED. The design method of a single-layer TDGW is presented and discussed in detail in this paper. A high-precision processing technology that can effectively guarantee the parallelism accuracy is also presented. A miniature coupling-in projection optics is designed with a catadioptric structure and integrated with the waveguide accordingly. Finally, a TDGW with a thickness of 1.75 mm is designed and analyzed. The results show that the stray light over the normal light is less than 0.5%, and the illuminance uniformity is well optimized. The field of view is up to 55°, and the XPD exceeds 12mm×10mm at an eye relief (ERF) of 18 mm. A proof-of-concept prototype was fabricated and demonstrated.

Low-stray-light gratings fabricated with scanning exposure method based on Lloyd's mirror for a high-contrast near-eye display in augmented reality.

The stray light of gratings lowers the image contrast of augmented reality display devices based on lightguide gratings. We propose to reduce the stray light of gratings fabricated with the scanning exposure method in a Lloyd's mirror interferometer setup. The photoresist-coated substrate is moved in the longitudinal direction parallel to the exposure interference fringes during exposure to average out the laser speckle-induced noise. A phase locking module is designed to compensate for the unwanted lateral displacement caused by the straightness error and vibration of the translation stage. The stability and accuracy of phase locking in the Lloyd's mirror interferometer are analyzed with control system theory. Low-stray-light lightguide gratings were fabricated successfully. The stray light level was decreased by more than 50%, and the imaging contrast was increased from 65% to 85%.

The XVS System During Open Spinal Fixation Procedures in Patients Requiring Pedicle Screw Placement in the Lumbosacral Spine

This pilot study was undertaken to evaluate the safety, performance, and usability of the Xvision-Spine (XVS) System (Augmedics, Arlington Heights, IL) during open spinal fixation procedures in patients requiring pedicle screw placement in the lumbosacral spine. The XVS System is an augmented reality head-mounted display (HMD) based on a computer navigation system designed to assist surgeons in accurately placing pedicle screws. It uses an HMD-mounted tracking camera to provide optical tracking technology, and provides the surgeon a translucent direct near-eye display of the navigated surgical instrument's location relative to the computed tomographic image. We report the preliminary results of a prospective series of all consecutive patients who underwent augmented reality-assisted pedicle screw placement in the lumbosacral vertebrae at 3 institutions. Clinical accuracy for each pedicle screw was graded with Gertzbein-Robbins scores by 2 independent and blinded neuroradiologists. The 19 study participants included 8 men and 11 women with a mean age of 59.13 ± 12.09 and 59.91 ± 12.89 years, respectively. Seventeen procedures were successfully completed via the XVS System. Two procedures were not completed due to technical issues with the system's intraoperative scanner. A total of 86 screws were inserted. The accuracy of the XVS System was 97.7%. The XVS System's performance in accurate placement of pedicle screws in the lumbosacral vertebrae had an overall accuracy of 97.7%. These preliminary results were comparable to the accuracy of other manual computer-assisted navigation systems reported in the literature.

Color near-eye display with high exit-pupil uniformity based on optimized meta-grating

A diffractive waveguide based on surface relief grating is one of the most significant components for near-eye displays (NEDs). Here, we propose a colorful NED system using meta-grating as the coupler to achieve high exit-pupil brightness uniformity and color uniformity. The whole system has a lightweight and compact configuration and allows for flexible integration with ordinary smart glass. Our work provides a feasible way to design a colorful and miniaturized NED.

37‐2: Invited Paper: Enabling Augmented‐Reality Near‐Eye and Head‐Up Displays with Neural Holography

Holography promises unprecedented capabilities for augmented reality (AR) display systems, but the current achievable image quality is quite limited. We leverage Neural Holography, a family of artificial intelligence‐driven computer‐generated holography (CGH) techniques, to enable two holographic see‐through AR displays: a lightguide coupled near‐eye display (NED) and a partial reflective head‐up display (HUD). Experimental results demonstrate the promise of enabling next‐generation AR 3D holographic displays.

Multifocal Near-Eye Display: Timing of Optical Diffuser Elements and Synchronization to DLP-Based Projection Microunit

Abstract The paper present the key technical details of a multifocal near-eye display concept. Along with an overview of the basic architecture, a particular implementation that utilises a digital light processing (DLP®) based spatial light modulator as the image source is provided in the study. The investigated approach involves the utilisation of a small-scale volumetric screen formed by a stack of fast-switching optical diffuser elements based on liquid crystal technology. The volumetric screen is illuminated by a rear image projector. To make the whole system functional and small, the challenge lies within the development of integrated control board for the projection modules as well as the synchronization of the DLP® projector image output to the optical diffuser element switching-cycle. The main difficulty of the development process is accounting for the peculiarities of in-house developed diffuser elements and the off-the-shelf DLP®, which is the main focus of this paper. There is no direct control over the full set of DLP® operational parameters, an indirect method for adjusting frame dead time is proposed, showing that an increase in dead time close to 0.3 ms (from 0.3 ms to 0.6 ms in the particular setup) can be achieved without significantly sacrificing image colour depth or quality. Tuneable dead time mitigates the limitations set by the non-instantaneous switching of liquid crystal diffuser elements as longer dead times allow for the removal of image bleeding between frames.

Uniformity improvement of a reconstructed-holographic image in a near-eye display system using off-axis HOE.

When a near-eye display (NED) device reproduces an image at a location close to the eye, the virtual image is implemented at a large angle. The uniformity of the image is unbalanced due to the change in diffraction efficiency by the hologram recording angle and angular selectivity. This study proposes a method for implementing an optimal uniform image by analyzing the diffraction efficiency and the reconstructed image was analyzed using angular selectivity generated while reproducing the source point of the diffused image as an intermediate element by holographic optical element (HOE). This research provides practical results for displaying high diffraction efficiency and immersive holographic images in the NED system with HOE as uniformed intermediate elements.

Foveated integral imaging system for near-eye 3D displays

Integral imaging displays have been presented as the most effective solution to reduce the visual discomfort in three-dimensional (3D) images caused by vergence-accommodation conflict (VAC). However, due to the resolution degradation, it is still challenging to adapt the integral imaging system to near-eye display (NED) devices. In this paper, we propose a resolution-enhanced integral imaging NED using foveated imaging system with two display panels and an optical combiner. We use a microdisplay combined with a lens array to provide the integral imaging 3D images with relatively high pixel density to the central foveal area. And the peripheral area is covered by an additional display panel which offers background images with wide field-of-view (FOV). Combining these two images with an optical combiner, the foveated integral imaging system results in highly resolution- enhanced integral imaging 3D images concentrated on the foveal area. The proposed NED system effectively provides integral imaging 3D images with approximately 4.5 times improved resolution in the foveal area through an optimally designed foveated imaging system.

Chromatic aberration correction in bi-focal augmented reality display by the multi-layer Pancharatnam-Berry phase lens.

Chromatic aberration is a main obstacle for the commercial application of augmented reality displays. The current digital and optical compensation methods of reducing the chromatic aberration suffer from processing time, power consumption or complex design. Here, a simple strategy of chromatic aberration correction in bi-focal augmented reality near-eye display based on multi-layer Pancharatnam-Berry phase lens has been demonstrated and verified by experimental results. The multi-layer Pancharatnam-Berry phase lens, as a part of optical combiner, is fabricated by three liquid crystal polymer phase lenses with central wavelength in red, green, and blue, respectively. The multi-layer Pancharatnam-Berry phase lens can effectively reduce the chromatic aberration in both convex and concave mode of bi-focal augmented reality system, where the color breakup of virtual images captured in bi-focal augmented reality display is significantly alleviated. Comparing to the value of ΔK = 1.3 m-1 in single green Pancharatnam-Berry phase lens, the multi-layer Pancharatnam-Berry phase lens system significantly reduce the ΔK to 0.45 m-1 with reduction of 65.4%, which finally decreases the longitudinal chromatic aberration and improve the quality of images. The proposed broadband multi-layer Pancharatnam-Berry phase lens can benefit augmented reality displays and find widespread application in the near-eye displays.