3D Eye Research Articles

S3: Speech, Script and Scene driven Head and Eye Animation

We present S 3 , a novel approach to generating expressive, animator-centric 3D head and eye animation of characters in conversation. Given speech audio, a Directorial script and a cinematographic 3D scene as input, we automatically output the animated 3D rotation of each character's head and eyes. S 3 distills animation and psycho-linguistic insights into a novel modular framework for conversational gaze capturing: audio-driven rhythmic head motion; narrative script-driven emblematic head and eye gestures; and gaze trajectories computed from audio-driven gaze focus/aversion and 3D visual scene salience. Our evaluation is four-fold: we quantitatively validate our algorithm against ground truth data and baseline alternatives; we conduct a perceptual study showing our results to compare favourably to prior art; we present examples of animator control and critique of S 3 output; and present a large number of compelling and varied animations of conversational gaze.

Head motion-corrected eye gaze tracking with the da Vinci surgical system.

To facilitate the integration of point of gaze (POG) as an input modality for robot-assisted surgery, we introduce a robust head movement compensation gaze tracking system for the da Vinci Surgical System. Previous surgical eye gaze trackers require multiple recalibrations and suffer from accuracy loss when users move from the calibrated position. We investigate whether eye corner detection can reduce gaze estimation error in a robotic surgery context. A polynomial regressor is first used to estimate POG after an 8-point calibration, and then, using another regressor, the POG error from head movement is estimated from the shift in 2D eye corner location. Eye corners are computed by first detecting regions of interest using the You Only Look Once (YOLO) object detector trained on 1600 annotated eye images (open dataset included). Contours are then extracted from the bounding boxes and a derivative-based curvature detector refines the eye corner. Through a user study (n = 24), our corner-contingent head compensation algorithm showed an error reduction in degrees of visual angle of 1.20 (p = 0.037) for the left eye and 1.26 (p = 0.079) for the right compared to the previous gold-standard POG error correction method. In addition, the eye corner pipeline showed a root-mean-squared error of 3.57 (SD = 1.92) pixels in detecting eye corners over 201 annotated frames. We introduce an effective method of using eye corners to correct for eye gaze estimation, enabling the practical acquisition of POG in robotic surgery.

Realistic 3D human saccades generated by a 6-DOF biomimetic robotic eye under optimal control.

We recently developed a biomimetic robotic eye with six independent tendons, each controlled by their own rotatory motor, and with insertions on the eye ball that faithfully mimic the biomechanics of the human eye. We constructed an accurate physical computational model of this system, and learned to control its nonlinear dynamics by optimising a cost that penalised saccade inaccuracy, movement duration, and total energy expenditure of the motors. To speed up the calculations, the physical simulator was approximated by a recurrent neural network (NARX). We showed that the system can produce realistic eye movements that closely resemble human saccades in all directions: their nonlinear main-sequence dynamics (amplitude-peak eye velocity and duration relationships), cross-coupling of the horizontal and vertical movement components leading to approximately straight saccade trajectories, and the 3D kinematics that restrict 3D eye orientations to a plane (Listing's law). Interestingly, the control algorithm had organised the motors into appropriate agonist-antagonist muscle pairs, and the motor signals for the eye resembled the well-known pulse-step characteristics that have been reported for monkey motoneuronal activity. We here fully analyse the eye-movement properties produced by the computational model across the entire oculomotor range and the underlying control signals. We argue that our system may shed new light on the neural control signals and their couplings within the final neural pathways of the primate oculomotor system, and that an optimal control principle may account for a wide variety of oculomotor behaviours. The generated data are publicly available at https://data.ru.nl/collections/di/dcn/DSC_626870_0003_600.

Pistol: Pupil Invisible Supportive Tool in the Wild

This paper is an in the wild evaluation of the eye tracking tool Pistol. Pistol supports Pupil Invisible projects and other eye trackers (Dikablis, Emke GmbH, Look, Pupil, and many more) in offline mode. For all eye tracking recordings, Pistol is able to compute pupil and iris ellipse, eye aperture, eye lids, 3D eye ball, optical vector estimation, and eye movement types. For Pupil Invisible projects, it also supports the detection of a special marker and allows the 2D and 3D gaze estimation mapped on the scene camera. In this extended journal paper for Pistol we focus on challenging in the wild situations, such as reflections on glasses, eye tracking during sport, changing lightning conditions, and this for two eye trackers. The tool Pistol can be downloaded here Link

For higher accuracy: 3D eye movement gaze depth estimation method based on Mix-TCN

Currently, the biggest obstacle in applying eye tracking technology in Virtual Reality (VR) and Augmented Reality (AR) scenes is the particular difficulty in choosing distance and object distance in 3D scenes. In previous research, geometric calculation methods using vestibular ocular reflex (VOR) and research on binocular visual angle have been studied to some extent, but unfortunately, their effects have not reached a practical level. The paper proposes a new research idea to estimate the depth of binocular gaze using a method of time series eye movement data analysis based on depth learning, and proposes a Mix-Temporal Convolutional Network (TCN) optical time series network. By combining VOR and deep learning theory, the paper has realized the current state-of-the-art technology for estimating the depth of gaze from the movement of the gaze.

Unexpected corneal reflection phenomenon alters smartphone 3D image-based models of the eye

Abstract Reconstruction of a 3D eye model by photogrammetry from a smartphone video could be prospectively used in self-diagnosis, screening and telemedicine monitoring of diseases of the front part of the eye and its surroundings. The main use could be found in the treatment of diseases of the curvature and surface of the cornea and in follow-up after some refractive procedures. In our work, we create 3D image-based models of the eye after scanning the face with a smartphone. An unexpected phenomenon appeared during the reconstruction of the transparent cornea – a crater-like depression was formed at the place where nearby objects reflected on the cornea, which corresponds to the first Purkinje image, the so-called glint. We thus encountered complications that may arise when modelling transparent living structures from a video taken in a normal environment, which will need to be solved if we want to create such 3D models of the eye using this method for medical purposes. Another 3D reconstruction approach or additional algorithms must be considered as a future work.

Longer duration entry mitigates nystagmus and vertigo in 7-Tesla MRI.

Patients and technologists commonly describe vertigo, dizziness, and imbalance near high-field magnets, e.g., 7-Tesla (T) magnetic resonance imaging (MRI) scanners. We sought a simple way to alleviate vertigo and dizziness in high-field MRI scanners by applying the understanding of the mechanisms behind magnetic vestibular stimulation and the innate characteristics of vestibular adaptation. We first created a three-dimensional (3D) control systems model of the direct and indirect vestibulo-ocular reflex (VOR) pathways, including adaptation mechanisms. The goal was to develop a paradigm for human participants undergoing a 7T MRI scan to optimize the speed and acceleration of entry into and exit from the MRI bore to minimize unwanted vertigo. We then applied this paradigm from the model by recording 3D binocular eye movements (horizontal, vertical, and torsion) and the subjective experience of eight normal individuals within a 7T MRI. The independent variables were the duration of entry into and exit from the MRI bore, the time inside the MRI bore, and the magnetic field strength; the dependent variables were nystagmus slow-phase eye velocity (SPV) and the sensation of vertigo. In the model, when the participant was exposed to a linearly increasing magnetic field strength, the per-peak (after entry into the MRI bore) and post-peak (after exiting the MRI bore) responses of nystagmus SPV were reduced with increasing duration of entry and exit, respectively. There was a greater effect on the per-peak response. The entry/exit duration and peak response were inversely related, and the nystagmus was decreased the most with the 5-min duration paradigm (the longest duration modeled). The experimental nystagmus pattern of the eight normal participants matched the model, with increasing entry duration having the strongest effect on the per-peak response of nystagmus SPV. Similarly, all participants described less vertigo with the longer duration entries. Increasing the duration of entry into and exit out of a 7T MRI scanner reduced or eliminated vertigo symptoms and reduced nystagmus peak SPV. Model simulations suggest that central processes of vestibular adaptation account for these effects. Therefore, 2-min entry and 20-s exit durations are a practical solution to mitigate vertigo and other discomforting symptoms associated with undergoing 7T MRI scans. In principle, these findings also apply to different magnet strengths.

Ray-tracing-guided myopic LASIK: real-world clinical outcomes.

To assess effectiveness of individualized ray-trace based laser in situ keratomileusis (LASIK) for correction of myopia in everyday clinical practice. Single-site private practice. Retrospective nonrandomised unmasked chart review. Consecutive, myopic eyes (range ≤-8.25 diopters [D] sphere; astigmatism 0 to -4.25 D) treated with ray-trace based LASIK were included. Patients underwent wavefront, tomography, and biometry assessment using the InnovEyes Sightmap diagnostic device. The ray-trace based algorithm (InnovEyes algorithm) then generated an individualized 3D eye model and calculated a customized LASIK ablation profile. Postoperative visual acuity, refractive error and whole eye higher-order aberrations (HOAs) were evaluated over 3 months. The procedure was performed on 400 eyes (200 patients). Mean preoperative manifest refraction spherical equivalent was -3.39 ± 1.58 D (right eye -3.84 ± 1.63 D, left eye -3.98 ± 1.75 D). At month 3, uncorrected distance visual acuity (UDVA) was ≥20/20 in all eyes, ≥20/16 in 89% (right eye 90%, left eye 89%), ≥20/12 in 51% (54% right eye; 47% left eye), and 20/10 in 8% (right eye 8%; left eye 9%) of eyes respectively. UDVA was within 1 line of preoperative corrected distance visual acuity in 98% of eyes (right eye 98.5%; left eye 98%) and 39% of eyes (right eye 38%; left eye 39%) gained 1 line improvement. There was a statistically but not clinically significant increase in total HOAs (right eye 0.06 ± 0.133 μm; left eye 0.057 ± 0.125 μm; P < .001). The spherical aberration decreased (right eye -0.047 ± 0.095 μm, P < .001; left eye -0.051 ± 0.091 μm, P < .001). Ray-trace based LASIK was safe and effective for correction of myopia with and without astigmatism. Approximately, half the eyes achieved ≥20/12.5 UDVA and 8% achieved 20/10. There was no clinically significant increase in total whole eye HOAs.

Eye detection and coarse localization of pupil for video-based eye tracking systems

A video-based eye tracking system generally captures NIR images, each of which contains one or two eyes of a subject. The subject’s point of gaze is then determined using 3D eye model and pupil centre corneal reflection technique. Eye detection and pupil localization play significant roles in video-based eye tracking systems. However, face rotation, wearing glasses, eye-shape variation and illumination variation make it difficult to detect an eye and localize a pupil accurately in the images captured by video-based eye tracking systems. In this paper, we proposed an eye detector that adopts a coarse-to-fine strategy. The eye detector consists of three classifiers: an ATLBP-THACs feature-based cascade classifier, a branch CNN and a multi-task CNN. We also proposed a method for coarse pupil localization. Coarse localization is an important step for pupil localization since it can provide initial pupil coordinates for a fine pupil localization method. Given a downscaled eye image, a shallow CNN is used to estimate the location of seven landmarks. On this basis, pupil center and radius are estimated. A method for small dim target enhancement is used to increase the contrast between pupil and background. The main goal of pupil enhancement is to make it easier to binarize an eye image. At last, component filtering is made by utilizing the estimated pupil center and radius. We collected a dataset named neepuEYE dataset that consists of 5500 NIR eye images from 109 people. The images can be used to generate augmented samples for training an eye detector since they contain eyes with different shape, orientation and pupil localization. Experimental results show that our eye detector is a fast and robust detector. Furthermore, our method for coarse pupil localization can obtain not only high detection rate but also high localization speed.

Adaptive Resolution Enhancement for Visual Attention Regions Based on Spatial Interpolation.

Resolution enhancement is crucial for human vision. However, it can be resource-consuming in the display pipeline. Therefore, there is a need to develop a lightweight resolution improvement algorithm specifically targeting visual attention regions. This paper presents a spatial-interpolation-based algorithm to improve the resolution of the visual attention area. The eye-tracking system consists of a near-infrared camera and an event camera is proposed to obtain the 3D gaze vector and eye moving trajectory. Secondly, the observation coordinates are obtained by gaze vectors, and the visual attention region is defined by the sensitive field-of-view angle. Then, interpolation-based adaptive spatial resolution enhancement and contrast enhancement adjustment are performed in the visual attention area. Finally, the feasibility of the proposed method is tested on both qualitative and quantitative dimensions. The experimental results demonstrate that the proposed method can significantly improve the visual effects.

Been There, Seen That: Visualization of Movement and 3D Eye Tracking Data from Real‐World Environments

AbstractThe distribution of visual attention can be evaluated using eye tracking, providing valuable insights into usability issues and interaction patterns. However, when used in real, augmented, and collaborative environments, new challenges arise that go beyond desktop scenarios and purely virtual environments. Toward addressing these challenges, we present a visualization technique that provides complementary views on the movement and eye tracking data recorded from multiple people in real‐world environments. Our method is based on a space‐time cube visualization and a linked 3D replay of recorded data. We showcase our approach with an experiment that examines how people investigate an artwork collection. The visualization provides insights into how people moved and inspected individual pictures in their spatial context over time. In contrast to existing methods, this analysis is possible for multiple participants without extensive annotation of areas of interest. Our technique was evaluated with a think‐aloud experiment to investigate analysis strategies and an interview with domain experts to examine the applicability in other research fields.

Design and fabrication method of holographic waveguide near-eye display with 2D eye box expansion.

Augmented reality near-eye display (AR-NED) technology has attracted enormous interests for its widespread potential applications. In this paper, two-dimensional (2D) holographic waveguide integrated simulation design and analysis, holographic optical elements (HOEs) exposure fabrication, prototype performance evaluation and imaging analysis are completed. In the system design, a 2D holographic waveguide AR-NED integrated with a miniature projection optical system is presented to achieve a larger 2D eye box expansion (EBE). A design method for controlling the luminance uniformity of 2D-EPE holographic waveguide by dividing the two thicknesses of HOEs is proposed, which is easy to fabricate. The optical principle and design method of the HOE-based 2D-EBE holographic waveguide are described in detail. In the system fabrication, laser exposure fabrication method of eliminating stray light for HOEs is proposed, and a prototype system is fabricated and demonstrated. The properties of the fabricated HOEs and the prototype are analyzed in detail. The experimental results verified that the 2D-EBE holographic waveguide has a diagonal field of view (FOV) of 45°, an ultra-thin thickness of 1 mm, and an eye box of 16 mm × 13 mm at an eye relief (ERF) of 18 mm, the MTF values of different FOVs at different 2D-EPE positions can be better than 0.2 at 20 lp/mm, and the whole luminance uniformity is 58%.

In vitro–in vivo correlation of drug release profiles from medicated contact lenses using an in vitro eye blink model

There is still a paucity of information on how in vitro release profiles from drug-loaded contact lenses (CLs) recorded in 3D printed eye models correlate with in vivo profiles. This work aims to evaluate the release profiles of two drug-loaded CLs in a 3D in vitro eye blink model and compare the obtained results with the release in a vial and the drug levels in tear fluid previously obtained from an animal in vivo study. In vitro release in the eye model was tested at two different flow rates (5 and 10 µL/min) and a blink speed of 1 blink/10 s. Model CLs were loaded with two different drugs, hydrophilic pravastatin and hydrophobic resveratrol. The release of both drugs was more sustained and lower in the 3D eye model compared to the in vitro release in vials. Interestingly, both drugs presented similar release patterns in the eye model and in vivo, although the total amount of drugs released in the eye model was significantly lower, especially for resveratrol. Strong correlations between percentages of pravastatin released in the eye model and in vivo were found. These findings suggest that the current 3D printed eye blink model could be a useful tool to measure the release of ophthalmic drugs from medicated CLs. Nevertheless, physiological parameters such as the composition of the tear fluid and eyeball surface, tear flow rates, and temperature should be optimized in further studies.Graphical abstract

A detailed methodology to model the Non Contact Tonometry: a Fluid Structure Interaction study.

Understanding the corneal mechanical properties has great importance in the study of corneal pathologies and the prediction of refractive surgery outcomes. Non-Contact Tonometry (NCT) is a non-invasive diagnostic tool intended to characterize the corneal tissue response in vivo by applying a defined air-pulse. The biomarkers inferred from this test can only be considered as indicators of the global biomechanical behaviour rather than the intrinsic biomechanical properties of the corneal tissue. A possibility to isolate the mechanical response of the corneal tissue is the use of an inverse finite element method, which is based on accurate and reliable modelling. Since a detailed methodology is still missing in the literature, this paper aims to construct a high-fidelity finite-element model of an idealized 3D eye for in silico NCT. A fluid-structure interaction (FSI) simulation is developed to virtually apply a defined air-pulse to a 3D idealized eye model comprising cornea, limbus, sclera, lens and humors. Then, a sensitivity analysis is performed to examine the influence of the intraocular pressure (IOP) and the structural material parameters on three biomarkers associated with corneal deformation. The analysis reveals the requirements for the in silico study linked to the correct reproduction of three main aspects: the air pressure over the cornea, the biomechanical properties of the tissues, and the IOP. The adoption of an FSI simulation is crucial to capture the correct air pressure profile over the cornea as a consequence of the air-jet. Regarding the parts of the eye, an anisotropic material should be used for the cornea. An important component is the sclera: the stiffer the sclera, the lower the corneal deformation due to the air-puff. Finally, the fluid-like behavior of the humors should be considered in order to account for the correct variation of the IOP during the test which will, otherwise, remain constant. The development of a strong FSI tool amenable to model coupled structures and fluids provides the basis to find the biomechanical properties of the corneal tissue in vivo.

Effect of anatomical differences and intraocular lens design on negative dysphotopsia.

To assess the effect of ocular anatomy and intraocular lens (IOL) design on negative dysphotopsia (ND). Department of Ophthalmology, Leiden University Medical Center, Leiden, the Netherlands. Ray-tracing study based on clinical data. Ray-tracing simulations were performed to assess the effect of anatomical differences and differences in IOL design on the peripheral retinal illumination. To that end, eye models that incorporate clinically measured anatomical differences between eyes of patients with ND and eyes of pseudophakic controls were created. The anatomical differences included pupil size, pupil centration, and iris tilt. The simulations were performed with different IOL designs, including a simple biconvex IOL design and a more complex clinical IOL design with a convex-concave anterior surface. Both IOL designs were analyzed using a clear edge and a frosted edge. As ND is generally considered to be caused by a discontinuity in peripheral retinal illumination, this illumination profile was determined for each eye model and the severity of the discontinuity was compared between eye models. The peripheral retinal illumination consistently showed a more severe discontinuity in illumination with ND-specific anatomy. This difference was the least pronounced, 8%, with the frosted edge clinical IOL and the most pronounced, 18%, with the clear edge biconvex IOL. These results show that small differences in the ocular anatomy or IOL design affect the peripheral retinal illumination. Therewith, they can increase the severity of ND by up to 18%.

MRI and FUNDUS image fusion for improved ocular biometry in Ocular Proton Therapy

IntroductionOcular biometry in Ocular Proton Therapy (OPT) currently relies on a generic geometrical eye model built by referencing surgically implanted markers. An alternative approach based on image fusion of volumetric Magnetic Resonance Imaging (MRI) and panoramic fundus photography was investigated. Materials and methodsEighteen non-consecutive uveal melanoma (UM) patients, who consented for an MRI and had their tumour base visible on panoramic fundus photography, were included in this comparative analysis. Through generating digitally-reconstructed projections from MRI images using the Lambert azimuthal equal-area projection, 2D-3D image fusion between fundus photography and an eye model delineated on MRI scans was achieved and allowed for a novel definition of the target base (MRI + FCTV). MRI + FCTV was compared with MRI-only delineation (MRIGTV) and the conventional (EyePlan) target definition (EPCTV). ResultsThe combined use of fundus photography and MRI to define tumour volumes reduced the average discrepancies by almost 65% with respect to the MRI only tumour definitions when comparing with the conventionally planned EPCTV. With the proposed method, shallow sub-retinal tumour infiltration, otherwise invisible on MRI, can be included in the target volume definition. Moreover, a novel definition of the fovea location improves the accuracy and personalisation of the 3D eye model. ConclusionMRI and fundus image fusion overcomes some of the limitations of ophthalmological MRI for tumour volume definition in OPT. This novel eye tumour modelling method might improve treatment planning personalisation, allowing to better anticipate which patients could benefit from prophylactic treatment protocols for radiation induced maculopathy.

The use of three-dimensional models for the teaching anatomical structures in high school biology lessons

The effects on the success of students in examination by using a three-dimensional plastic model and a three-dimensional digital computer application for the teaching of the subject of "Eye anatomy" under the title of "Sensory organs" were aimed in the study. The study was conducted on the three groups of high school students (N=43). The groups were split into different laboratories for freelance work. The first group was given standard lecture notes, the second group was given 3D plastic eye models and the third group was given a 3D digital eye model application in the computer environment and they were left to work for equal periods. Pre-test and post-test achievement exams were used as data collection tools to measure the achievement levels of the students on the subject of "Eye anatomy". At the end of this study, no significant difference was found between the groups according to the results of the statistical analysis. It is assumed that the results may be related to the study habits of the students for the university entrance exam in Turkey; the characteristics of the three-dimensional models used; or the fact that only short-term memory has been tested. So, extensive research is needed considering these issues.

Human–System Interaction Based on Eye Tracking for a Virtual Workshop

With the constant exploration and development of intelligent manufacturing, the concept of digital twins has been proposed and applied. In view of the complexity and intellectualization of virtual workshop systems, real workshops can link with virtual workshosp based on AR under the structure of digital twins, which allows users to interact with virtual information and perceive the virtual information superimposed on the real world with great immersion. However, the three-dimensionality of virtual workshops and interaction with complex workshop information can be challenging for users. Due to a shortage of input bandwidth and the nontraditional mode of interaction, a more natural interaction technique for virtual workshops is required. To solve such problems, this paper presents a technical framework for 3D eye movement interaction applied to a virtual workshop. An eye movement interaction technique, oriented to implicit interaction and explicit interaction, is developed by establishing behavior recognition and interaction intention understanding. An eye-movement experiment verifies the former’s accuracy is above 90% and had better recognition performance. A better feature vector group of the latter is selected to establish a model and verify its feasibility and effectiveness. Finally, the feasibility of the framework is verified through the development of an application example.

Autostereoscopic 3D Display System for 3D Medical Images

Recent advances in autostereoscopic three-dimensional (3D) display systems have led to innovations in consumer electronics and vehicle systems (e.g., head-up displays). However, medical images with stereoscopic depth provided by 3D displays have yet to be developed sufficiently for widespread adoption in diagnostics. Indeed, many stereoscopic 3D displays necessitate special 3D glasses that are unsuitable for clinical environments. This paper proposes a novel glasses-free 3D autostereoscopic display system based on an eye tracking algorithm and explores its viability as a 3D navigator for cardiac computed tomography (CT) images. The proposed method uses a slit-barrier with a backlight unit, which is combined with an eye tracking method that exploits multiple machine learning techniques to display 3D images. To obtain high-quality 3D images with minimal crosstalk, the light field 3D directional subpixel rendering method combined with the eye tracking module is applied using a user’s 3D eye positions. Three-dimensional coronary CT angiography images were volume rendered to investigate the performance of the autostereoscopic 3D display systems. The proposed system was trialed by expert readers, who identified key artery structures faster than with a conventional two-dimensional display without reporting any discomfort or 3D fatigue. With the proposed autostereoscopic 3D display systems, the 3D medical image navigator system has the potential to facilitate faster diagnoses with improved accuracy.

Eye MG 3D Application - A comprehensive ocular anatomy and pathophysiology 3D atlas with real-time true color confocal images to enhance ophthalmology education and e-Counseling.

Concepts pertaining to ophthalmology have lots of theoretical frameworks. Neophyte residents and novice surgeons may have to mentally visualize these concepts during the initial days of training. Only a powerful cognitive tool such as a three-dimensional (3D) eyeball model, with real-time TrueColor confocal images (and not animated images or models), can fill in these intellective mental gaps. Giving the users (i.e., residents and students) the power to choose and visualize various parts of the eye, with multiple magnitudes of zoom, is mandatory for optimal e-learning. To make ophthalmic concept learning better, we have developed a 3D app Eye MG 3D (patent pending) comprising ocular anatomy and pathophysiological 3D models, built on an advanced interactive 3D touch interface, by using patient’s real-time confocal images to serve as a new-age pedagogical tool and e-counseling. According to our knowledge, there are no applications to date that incorporate real-time high-resolution multimodal confocal fundus images and photoreal visuals for interactive and immersive 3D learning.