3D Eye Research Articles

Autostereoscopic 3D display using directional subpixel rendering.

In this paper we present an autostereoscopic 3D display using a directional subpixel rendering algorithm in which clear left-right images are expressed in real time based on a viewer's 3D eye positions. In order to maintain the 3D image quality over a wide viewing range, we designed an optical layer that generates a uniformly distributed light field. The proposed 3D rendering method is simple, and each pixel processing can be performed independently in parallel computing environments. To prove the effectiveness of our display system, we implemented 31.5" 3D monitor and 10.1" 3D tablet prototypes in which the 3D rendering is processed in the GPU and FPGA board, respectively.

Design and Fabrication of a Three-Dimensional Artificial Compound Eye Using Two-Photon Polymerization.

Microlens arrays have been widely used in the fields of micro-optics because of the advantages of their high diffraction efficiency, high fill factor, and wide operating band. However, the microlens array still has problems with its smaller field of view (FOV) and lower utilization of light energy. In this paper, a 3D compound eye system consisting of a microlens array and a pinhole array was designed according to the optical principle of insect compound eye. The artificial compound eye structure was processed in two-photon polymerization processing technology. Ray tracing and optical system simulation of the designed artificial compound eye structure were performed. The results showed that the artificial compound eye structure had a wider FOV and higher light energy utilization than a conventional 2D microlens array. This thesis may lay a theoretical foundation for the structural optimization design of microlens arrays.

Point of Gaze Estimation Using Corneal Surface Reflection and Omnidirectional Camera Image

We present a human point of gaze estimation system using corneal surface reflection and omni-directional image taken by a fish eye. Only capturing an eye image, our system entables to find where a user is looking in 360P surrounding scene image. We first generates multiple perspective scene images from an equi-rectangular image and perform registration between corneal reflection and perspective images. We then compute the point of gaze using a 3D eye model and project the point to an omni-directional image. We evaluated the robustness of registration and accuracy of PoG estimations using two indoor and five outdoor scenes, and found that gaze mapping error was 5.526[deg] on average. This result shows the potential to the marketing and outdoor training system.

Static Characteristics of a New Three-Dimensional Linear Homeomorphic Saccade Model

A linear homeomorphic saccade model that produces 3D saccadic eye movements consistent with physiological and anatomical evidence is introduced. Central to the model is the implementation of a time-optimal controller with six linear muscles and pulleys that represent the saccade oculomotor plant. Each muscle is modeled as a parallel combination of viscosity [Formula: see text] and series elasticity [Formula: see text] connected to the parallel combination of active-state tension generator [Formula: see text], viscosity element [Formula: see text], and length tension elastic element [Formula: see text]. Additionally, passive tissues involving the eyeball include a viscosity element [Formula: see text], elastic element [Formula: see text], and moment of inertia [Formula: see text]. The neural input for each muscle is separately maintained, whereas the effective pulling direction is modulated by its respective mid-orbital constraint from the pulleys. Initial parameter values for the oculomotor plant are based on anatomical and physiological evidence. The oculomotor plant uses a time-optimal, 2D commutative neural controller, together with the pulley system that actively functions to implement Listing's law during both static and dynamic conditions. In a companion paper, the dynamic characteristics of the saccade model is analyzed using a time domain system identification technique to estimate the final parameter values and neural inputs from saccade data. An excellent match between the model estimates and the data is observed, whereby a total of 20 horizontal, 5 vertical, and 64 oblique saccades are analyzed.

Dynamic Characteristics of a New Three-Dimensional Linear Homeomorphic Saccade Model.

A linear homeomorphic eye movement model that produces 3D saccadic eye movements consistent with anatomical and physiological evidence is introduced in this second part of a two-paper sequence. Central to the model is the implementation of a time-optimal neural control strategy involving six linear muscle models that faithfully represent the dynamic characteristics of 3D saccades. The muscle is modeled as a parallel combination of viscosity [Formula: see text] and series elasticity [Formula: see text], connected to the parallel combination of active-state tension generator [Formula: see text], viscosity element [Formula: see text], and length tension elastic element [Formula: see text]. The neural input for each muscle is separately maintained while the effective pulling direction is modulated by its respective pulley. The results demonstrate that a time-optimal, 2D commutative neural controller, together with the pulley system, actively functions to implement Listing's law during both static and dynamic simulations and provide an excellent match with the experimental data. The parameters and neural input to the muscles are estimated using a time domain system identification technique from saccade data, with an excellent match between the model estimates and the data. A total of 20 horizontal, 5 vertical and 62 oblique saccades are analyzed.

EyePACT

The parallax effect describes the displacement between the perceived and detected touch locations on a touch-enabled surface. Parallax is a key usability challenge for interactive displays, particularly for those that require thick layers of glass between the screen and the touch surface to protect them from vandalism. To address this challenge, we present EyePACT, a method that compensates for input error caused by parallax on public displays. Our method uses a display-mounted depth camera to detect the user's 3D eye position in front of the display and the detected touch location to predict the perceived touch location on the surface. We evaluate our method in two user studies in terms of parallax correction performance as well as multi-user support. Our evaluations demonstrate that EyePACT (1) significantly improves accuracy even with varying gap distances between the touch surface and the display, (2) adapts to different levels of parallax by resulting in significantly larger corrections with larger gap distances, and (3) maintains a significantly large distance between two users' fingers when interacting with the same object. These findings are promising for the development of future parallax-free interactive displays.

Advanced Treatment Planning for Ophthalmic Plaque Brachytherapy of Uveal Melanomas

Brachytherapy (BT) treatments for uveal melanomas using ophthalmic plaques (i.e. 125I COMS and 106Ru eye plaques) achieves excellent clinical results with high tumor control and low toxicity rates. However, in comparison to state of the art external beam therapy and other BT techniques treatment planning of ophthalmologic plaque BT is underdeveloped regarding volumetric dose assessments, automated dose optimization routines and post-implantation quality assurance. The aim of this study was to (i) present a versatile and advanced treatment planning system (TPS) for ophthalmic plaque BT for both 125I COMS and 106Ru plaques and (ii) to demonstrate its potential in improving treatment planning. The TPS enables automated optimization of treatment plans based on Monte Carlo (MC) dose calculation and dose-volume histograms (DVH) as well as handling of multimodal image sets. In a first step, a resizable 3D eye model including all relevant critical structures was established, which enables to define both the tumor location and exact geometry by echography and funduscopy. Patient specific eye model adaptation can be achieved via post-implantation MR scans. An applicator library and respective dose look-up tables were generated by MC methods (MCNP6, Los Alamos, USA) that allow to generate DVHs for the target and all organs-at-risk (OAR). The current library of different plaques includes a 125I COMS plaque (Amersham Health model 6711) as well as several 106Ru plaques. The plaque geometry and 125I seeds were modeled according to AAPM TG 129 recommendations. The novel TPS allows calculating of full 3D dose distributions and DVH based on a cubic grid of 200 μm. The obtained results for the 125I COMS plaque and the values provided by AAPM TG 129 report were in good agreement. A maximum dose difference of 3 % up to a depth of 8 mm and 6 % up to a depth of 10 mm were obtained. Individualized modeling of the 3D eye based on fused funduscopic and post-implantation MR images allows to adjust treatment times and provide more accurate geometric definitions of the tumor and plaque location to improve the accuracy of the obtained DVHs. The implemented optimization routines can be used to define the optimal plaque size and safe eccentric plaque positions with respect to the tumor center to allow significant reductions of OAR doses. So far dose constrains for OARs (e.g. retinal doses of more than 500 Gy as a factor associated with visual acuity loss) were obtained from retrospective studies for treatment plan optimization. An advanced 3D TPS was developed and validated for plaque BT of uveal melanomas, which was designed to overcome the limitations of traditional treatment planning methods. The new TPS allows for a better and individualized treatment by including more image information and automated optimization into the planning process. The plaque library is currently updated to include more plaque types.

眼球画像を用いた視線追跡と物体認識

This paper introduces a method to identify the focused object in eye images captured from a single camera in order to enable intuitive eye-based interactions using wearable devices. The proposed method relies on a 3D eye model reconstruction to evaluate the gaze direction from the eye images. The gaze direction is then used in combination with deep learning algorithms to classify the focused object reflected on the cornea. Experimental results using a wearable prototype demonstrate the potential of the proposed method.

Real-Time 3D Eye Performance Reconstruction for RGBD Cameras.

This paper proposes a real-time method for 3D eye performance reconstruction using a single RGBD sensor. Combined with facial surface tracking, our method generates more pleasing facial performance with vivid eye motions. In our method, a novel scheme is proposed to estimate eyeball motions by minimizing the differences between a rendered eyeball and the recorded image. Our method considers and handles different appearances of human irises, lighting variations and highlights on images via the proposed eyeball model and the -based optimization. Robustness and real-time optimization are achieved through the novel 3D Taylor expansion-based linearization. Furthermore, we propose an online bidirectional regression method to handle occlusions and other tracking failures on either of the two eyes from the information of the opposite eye. Experiments demonstrate that our technique achieves robust and accurate eye performance reconstruction for different iris appearances, with various head/face/eye motions, and under different lighting conditions.

Poster ‐ 40: Treatment Verification of a 3D‐printed Eye Phantom for Proton Therapy

Purpose:Ocular melanoma is a form of eye cancer which is often treated using proton therapy. The benefit of the steep proton dose gradient can only be leveraged for accurate patient eye alignment. A treatment‐planning program was written to plan on a 3D‐printed anatomical eye‐phantom, which was then irradiated to demonstrate the feasibility of verifying in vivo dosimetry for proton therapy using PET imaging.Methods:A 3D CAD eye model with critical organs was designed and voxelized into the Monte‐Carlo transport code FLUKA. Proton dose and PET isotope production were simulated for a treatment plan of a test tumour, generated by a 2D treatment‐planning program developed using NumPy and proton range tables. Next, a plastic eye‐phantom was 3D‐printed from the CAD model, irradiated at the TRIUMF Proton Therapy facility, and imaged using a PET scanner.Results:The treatment‐planning program prediction of the range setting and modulator wheel was verified in FLUKA to treat the tumour with at least 90% dose coverage for both tissue and plastic. An axial isotope distribution of the PET isotopes was simulated in FLUKA and converted to PET scan counts. Meanwhile, the 3D‐printed eye‐phantom successfully yielded a PET signal.Conclusions:The 2D treatment‐planning program can predict required parameters to sufficiently treat an eye tumour, which was experimentally verified using commercial 3D‐printing hardware to manufacture eye‐phantoms. Comparison between the simulated and measured PET isotope distribution could provide a more realistic test of eye alignment, and a variation of the method using radiographic film is being developed.

3D Gaze tracking by combining eye- and facial-gaze vectors

We propose a 3D gaze-tracking method that combines accurate 3D eye- and facial-gaze vectors estimated from a Kinect v2 high-definition face model. Using accurate 3D facial and ocular feature positions, gaze positions can be calculated more accurately than with previous methods. Considering the image resolution of the face and eye regions, two gaze vectors are combined as a weighted sum, allocating more weight to facial-gaze vectors. Hence, the facial orientation mainly determines the gaze position, and eye-gaze vectors then perform minor manipulations. The 3D facial-gaze vector is first defined, and the 3D rotational center of the eyeball is then estimated; together, these define the 3D eye-gaze vector. Finally, the intersection point between the 3D gaze vector and the physical display plane is calculated as the gaze position. Experimental results show that the average gaze estimation root-mean-square error was approximately 23 pixels from the desired position at a resolution of $$1920\times 1080$$1920×1080.

Realtime 3D eye gaze animation using a single RGB camera

This paper presents the first realtime 3D eye gaze capture method that simultaneously captures the coordinated movement of 3D eye gaze, head poses and facial expression deformation using a single RGB camera. Our key idea is to complement a realtime 3D facial performance capture system with an efficient 3D eye gaze tracker. We start the process by automatically detecting important 2D facial features for each frame. The detected facial features are then used to reconstruct 3D head poses and large-scale facial deformation using multi-linear expression deformation models. Next, we introduce a novel user-independent classification method for extracting iris and pupil pixels in each frame. We formulate the 3D eye gaze tracker in the Maximum A Posterior (MAP) framework, which sequentially infers the most probable state of 3D eye gaze at each frame. The eye gaze tracker could fail when eye blinking occurs. We further introduce an efficient eye close detector to improve the robustness and accuracy of the eye gaze tracker. We have tested our system on both live video streams and the Internet videos, demonstrating its accuracy and robustness under a variety of uncontrolled lighting conditions and overcoming significant differences of races, genders, shapes, poses and expressions across individuals.

An Ultrasound Based Eye Tracking System

A novel, non-contact ultrasound device is detailed for recording and analyzing 3D fast eye movements (saccades) and smooth pursuit eye movements. Saccades are studied to gain a better understanding of the human oculomotor plant and neuromuscular systems. Abnormal saccades can be indicators of both neurological disorders and mild traumatic brain injury (MTBI). Limitations in existing saccade measurement devices prevent them from being used to measure saccades immediately after a possible MTBI event or easily outside of the clinical environment. The device proposed is portable allowing saccade measurements in the field to immediately assess neurological dysfunction associated with MTBI. Ease of use and portability allow collection of data at times and places not possible with devices currently available. This increased database of saccades will expand our knowledge of the relationship between saccades and the neurological functioning of the brain. The focus of this paper is the development of a finite element model to establish a starting point for such a design.

カメラを用いた視線追跡のための装着型デバイス

The industry's recent growing interest in virtual reality, augmented reality and smart wearable devices has created a new momentum for eye tracking. Eye movements in particular are viewed as a way to obtain natural user responses from wearable devices alongside gaze information used to analyze interests and behaviors. This paper extends our previous work by introducing a wearable eye-tracking device that enables the reconstruction of 3D eye models of each eye from two RGB cameras. The proposed device is built using high-resolution cameras and a 3D-printed frame attached to a pair of JINS MEME glasses. The 3D eye models reconstructed from the proposed device can be used with any model-based eye-tracking approach. The proposed device is also capable of extracting scene information from the cornea reflections captured by the cameras, detecting blinks from an electrooculography sensor as well as tracking head movements from an accelerometer combined with a gyroscope.

Spatial transformations between superior colliculus visual and motor response fields during head-unrestrained gaze shifts.

We previously reported that visuomotor activity in the superior colliculus (SC)--a key midbrain structure for the generation of rapid eye movements--preferentially encodes target position relative to the eye (Te) during low-latency head-unrestrained gaze shifts (DeSouza et al., 2011). Here, we trained two monkeys to perform head-unrestrained gaze shifts after a variable post-stimulus delay (400-700 ms), to test whether temporally separated SC visual and motor responses show different spatial codes. Target positions, final gaze positions and various frames of reference (eye, head, and space) were dissociated through natural (untrained) trial-to-trial variations in behaviour. 3D eye and head orientations were recorded, and 2D response field data were fitted against multiple models by use of a statistical method reported previously (Keith et al., 2009). Of 60 neurons, 17 showed a visual response, 12 showed a motor response, and 31 showed both visual and motor responses. The combined visual response field population (n = 48) showed a significant preference for Te, which was also preferred in each visual subpopulation. In contrast, the motor response field population (n = 43) showed a preference for final (relative to initial) gaze position models, and the Te model was statistically eliminated in the motor-only population. There was also a significant shift of coding from the visual to motor response within visuomotor neurons. These data confirm that SC response fields are gaze-centred, and show a target-to-gaze transformation between visual and motor responses. Thus, visuomotor transformations can occur between, and even within, neurons within a single frame of reference and brain structure.

Real time gaze estimation with a consumer depth camera

Existing eye-gaze-tracking systems typically require multiple infrared (IR) lights and high-quality cameras to achieve good performance and robustness against head movement. This requirement limits the systems’ potential for broader applications. In this paper, we present a low-cost, non-intrusive, simple-setup gaze estimation system that can estimate the gaze direction under free head movement. In particular, the proposed system only uses a consumer depth camera (Kinect sensor) positioned at a distance from the subject. We develop a simple procedure to calibrate the geometric relationship between the screen and the camera, and subject-specific parameters. A parameterized iris model is then used to locate the center of the iris for gaze feature extraction, which can handle low-quality eye images. Finally, the gaze direction is determined based on a 3D geometric eye model, where the head movement and deviation of the visual axis from the optical axis are taken into consideration. Experimental results indicate that the system can estimate gaze with an accuracy of 1.4–2.7° and is robust against large head movements. Two real-time human–computer interaction (HCI) applications are presented to demonstrate the potential of the proposed system for wide applications.

Computations underlying the visuomotor transformation for smooth pursuit eye movements.

Smooth pursuit eye movements are driven by retinal motion and enable us to view moving targets with high acuity. Complicating the generation of these movements is the fact that different eye and head rotations can produce different retinal stimuli but giving rise to identical smooth pursuit trajectories. However, because our eyes accurately pursue targets regardless of eye and head orientation (Blohm G, Lefèvre P. J Neurophysiol 104: 2103-2115, 2010), the brain must somehow take these signals into account. To learn about the neural mechanisms potentially underlying this visual-to-motor transformation, we trained a physiologically inspired neural network model to combine two-dimensional (2D) retinal motion signals with three-dimensional (3D) eye and head orientation and velocity signals to generate a spatially correct 3D pursuit command. We then simulated conditions of 1) head roll-induced ocular counterroll, 2) oblique gaze-induced retinal rotations, 3) eccentric gazes (invoking the half-angle rule), and 4) optokinetic nystagmus to investigate how units in the intermediate layers of the network accounted for different 3D constraints. Simultaneously, we simulated electrophysiological recordings (visual and motor tunings) and microstimulation experiments to quantify the reference frames of signals at each processing stage. We found a gradual retinal-to-intermediate-to-spatial feedforward transformation through the hidden layers. Our model is the first to describe the general 3D transformation for smooth pursuit mediated by eye- and head-dependent gain modulation. Based on several testable experimental predictions, our model provides a mechanism by which the brain could perform the 3D visuomotor transformation for smooth pursuit.

High-quality capture of eyes

Even though the human eye is one of the central features of individual appearance, its shape has so far been mostly approximated in our community with gross simplifications. In this paper we demonstrate that there is a lot of individuality to every eye, a fact that common practices for 3D eye generation do not consider. To faithfully reproduce all the intricacies of the human eye we propose a novel capture system that is capable of accurately reconstructing all the visible parts of the eye: the white sclera , the transparent cornea and the non-rigidly deforming colored iris . These components exhibit very different appearance properties and thus we propose a hybrid reconstruction method that addresses them individually, resulting in a complete model of both spatio-temporal shape and texture at an unprecedented level of detail, enabling the creation of more believable digital humans. Finally, we believe that the findings of this paper will alter our community's current assumptions regarding human eyes, and our work has the potential to significantly impact the way that eyes will be modelled in the future.

Study of in vitro capillary-like structures in psoriatic skin substitutes.

Angiogenesis is one of the important hallmarks of psoriasis. The extension of the superficial microvascular structure and activated pro-angiogenic mediators in psoriasis seem to be important factors involved in the pathology. According to the changes of superficial microvasculature in psoriatic lesions, anti-angiogenic treatment could be a promising therapeutic strategy for psoriasis. The aim of this study was to construct an in vitro vascularized psoriatic skin substitute for fundamental research. Psoriatic fibroblasts and keratinocytes were isolated from psoriatic plaque biopsies, while healthy fibroblasts and keratinocytes, as well as microvascular endothelial cells, were isolated from healthy skin biopsies of cosmetic breast surgery. Psoriatic and healthy skin substitutes with and without endothelial cells were produced using the self-assembly approach. Afterward the substitutes were examined by histology, immunofluorescence studies, and three-dimensional (3D) confocal microscopy. Histological analysis and immunofluorescence staining of specific markers for endothelial cells (von Willebrand, PECAM-1 [CD31], and VE-cadherin [CD144]) and basement membrane component (collagen IV) demonstrated that endothelial cells have the ability to form capillary-like tubes. Moreover, the 3D branched structure of the capillary-like structures and an eagle eye view of them were observed by confocal microscopy. Also the semiquantification of capillary-like tubes (CLTs) was carried out with a 3D eagle eye view of substitutes, and more CLTs were observed in psoriatic substitutes. These results suggest that it is possible to observe 3D capillary-like structures in the self-assembled psoriatic skin substitutes, which could become a good in vitro testing model for anti-angiogenic drug research, and facilitate the study of this complex pathology, which links angiogenesis to its development.

Three-dimensional imaging improves surgical skill performance in a laparoscopic test model for both experienced and novice laparoscopic surgeons.

The objective of this study was to evaluate and compare the impact of three-dimensional (3D) imaging system on the performance of basic laparoscopic tasks in a test model by novice and experienced surgeons. Three tasks were performed in a test model by 30 surgeons, 15 experienced surgeons, and 15 with minimal laparoscopic experience. The tasks were performed using 2D and 3D vision systems. Canadian Task Force II-1. Fifteen experienced laparoscopic surgeons and fifteen novices with minimal laparoscopic experience. Performance times were recorded using both two-dimensional and 3D imaging system for each task. Performance time for all skills was significantly (P < 0.02) shorter when using 3D imaging system. Performance times were reduced by 18-31% using 3D imaging for all participants. Experienced surgeons performed the tasks faster and showed similar improvement while using 3D imaging system. 3D vision systems allow for significant improvement in performance times of basic laparoscopic tasks in a test model for both inexperienced and advanced laparoscopic surgeons. Experienced surgeons benefit as much as novices from 3D imaging system. This benefit should be weighed against the disadvantages of the 3D vision systems, mainly cost, decreased light, eye strain, headaches, and shorter focal lengths.