3D Eye Position Research Articles

Autostereoscopic 3D Display System for 3D Medical Images

Recent advances in autostereoscopic three-dimensional (3D) display systems have led to innovations in consumer electronics and vehicle systems (e.g., head-up displays). However, medical images with stereoscopic depth provided by 3D displays have yet to be developed sufficiently for widespread adoption in diagnostics. Indeed, many stereoscopic 3D displays necessitate special 3D glasses that are unsuitable for clinical environments. This paper proposes a novel glasses-free 3D autostereoscopic display system based on an eye tracking algorithm and explores its viability as a 3D navigator for cardiac computed tomography (CT) images. The proposed method uses a slit-barrier with a backlight unit, which is combined with an eye tracking method that exploits multiple machine learning techniques to display 3D images. To obtain high-quality 3D images with minimal crosstalk, the light field 3D directional subpixel rendering method combined with the eye tracking module is applied using a user’s 3D eye positions. Three-dimensional coronary CT angiography images were volume rendered to investigate the performance of the autostereoscopic 3D display systems. The proposed system was trialed by expert readers, who identified key artery structures faster than with a conventional two-dimensional display without reporting any discomfort or 3D fatigue. With the proposed autostereoscopic 3D display systems, the 3D medical image navigator system has the potential to facilitate faster diagnoses with improved accuracy.

A novel 3D video oculography system for measuring cross-axis vestibulo-ocular reflex

We present a video oculography (VOG) system with 6-degree-of-freedom (6-DOF) mobility for real-time measurements of the binocular 3D eye position of a small animal. A hybrid hexapod that allowed for multi-axis complex motions with the resolution of the microscopic level was used to control the motion of the animal. The instantaneous eyeball movement of the animal was determined based on two approaches: (1) tracking of marker arrays affixed to the cornea; and (2) tracking the pupil outline. The tracking of the eyeball movement and the motion control of the hexapod were implemented with the LabVIEW virtual instruments. Compared with our previous measurements using a servo-motor-based single-axis VOG system, positional error reduced from more than 4% to less than 0.7%. Validation showed that the tracking errors in three rotational axes are less than 2% for the magnitude and less than 5° for the direction angle. The present VOG system is an effective tool for cross-axis 3D vestibulo-ocular reflex study on small animals.

RGB-D-based gaze point estimation via multi-column CNNs and facial landmarks global optimization

In this work, we utilize a multi-column CNNs framework to estimate the gaze point of a person sitting in front of a display from an RGB-D image of the person. Given that gaze points are determined by head poses, eyeball poses, and 3D eye positions, we propose to infer the three components separately and then integrate them for gaze point estimation. The captured depth images, however, usually contain noises and black holes which prevent us from acquiring reliable head pose and 3D eye position estimation. Therefore, we propose to refine the raw depth for 68 facial keypoints by first estimating their relative depths from RGB face images, which along with the captured raw depths are then used to solve the absolute depth for all facial keypoints through global optimization. The refined depths will provide us reliable estimation for both head pose and 3D eye position. Given that existing publicly available RGB-D gaze tracking datasets are small, we also build a new dataset for training and validating our method. To the best of our knowledge, it is the largest RGB-D gaze tracking dataset in terms of the number of participants. Comprehensive experiments demonstrate that our method outperforms existing methods by a large margin on both our dataset and the Eyediap dataset.

RGBD Based Gaze Estimation via Multi-Task CNN

This paper tackles RGBD based gaze estimation with Convolutional Neural Networks (CNNs). Specifically, we propose to decompose gaze point estimation into eyeball pose, head pose, and 3D eye position estimation. Compared with RGB image-based gaze tracking, having depth modality helps to facilitate head pose estimation and 3D eye position estimation. The captured depth image, however, usually contains noise and black holes which noticeably hamper gaze tracking. Thus we propose a CNN-based multi-task learning framework to simultaneously refine depth images and predict gaze points. We utilize a generator network for depth image generation with a Generative Neural Network (GAN), where the generator network is partially shared by both the gaze tracking network and GAN-based depth synthesizing. By optimizing the whole network simultaneously, depth image synthesis improves gaze point estimation and vice versa. Since the only existing RGBD dataset (EYEDIAP) is too small, we build a large-scale RGBD gaze tracking dataset for performance evaluation. As far as we know, it is the largest RGBD gaze dataset in terms of the number of participants. Comprehensive experiments demonstrate that our method outperforms existing methods by a large margin on both our dataset and the EYEDIAP dataset.

Autostereoscopic 3D display using directional subpixel rendering.

In this paper we present an autostereoscopic 3D display using a directional subpixel rendering algorithm in which clear left-right images are expressed in real time based on a viewer's 3D eye positions. In order to maintain the 3D image quality over a wide viewing range, we designed an optical layer that generates a uniformly distributed light field. The proposed 3D rendering method is simple, and each pixel processing can be performed independently in parallel computing environments. To prove the effectiveness of our display system, we implemented 31.5" 3D monitor and 10.1" 3D tablet prototypes in which the 3D rendering is processed in the GPU and FPGA board, respectively.

EyePACT

The parallax effect describes the displacement between the perceived and detected touch locations on a touch-enabled surface. Parallax is a key usability challenge for interactive displays, particularly for those that require thick layers of glass between the screen and the touch surface to protect them from vandalism. To address this challenge, we present EyePACT, a method that compensates for input error caused by parallax on public displays. Our method uses a display-mounted depth camera to detect the user's 3D eye position in front of the display and the detected touch location to predict the perceived touch location on the surface. We evaluate our method in two user studies in terms of parallax correction performance as well as multi-user support. Our evaluations demonstrate that EyePACT (1) significantly improves accuracy even with varying gap distances between the touch surface and the display, (2) adapts to different levels of parallax by resulting in significantly larger corrections with larger gap distances, and (3) maintains a significantly large distance between two users' fingers when interacting with the same object. These findings are promising for the development of future parallax-free interactive displays.

Vergence Neurons Identified in the Rostral Superior Colliculus Code Smooth Eye Movements in 3D Space

The rostral superior colliculus (rSC) encodes position errors for multiple types of eye movements, including microsaccades, small saccades, smooth pursuit, and fixation. Here we address whether the rSC contributes to the development of neural signals that are suitable for controlling vergence eye movements. We use both single-unit recording and microstimulation techniques in monkey to answer this question. We found that vergence eye movements can be evoked using microstimulation in the rSC. Moreover, among the previously described neurons in rSC, we recorded a novel population of neurons that either increased (i.e., convergence neurons) or decreased (i.e., divergence neurons) their activity during vergence eye movements. In particular, these neurons dynamically encoded changes in vergence angle during vergence tracking, fixation in 3D space and the slow binocular realignment that occurs after disconjugate saccades, but were completely unresponsive during conjugate or the rapid component of disconjugate saccades (i.e., fast vergence) and conjugate smooth pursuit. Together, our microstimulation and single-neuron results suggest that the SC plays a role in the generation of signals required to precisely align the eyes toward targets in 3D space. We propose that accurate maintenance of 3D eye position, critical for the perception of stereopsis, may be mediated via the rSC.

The medial posterior parietal area V6A and the control of arm actions in depth

Event Abstract Back to Event The medial posterior parietal area V6A and the control of arm actions in depth Patrizia Fattori1*, Kostas Hadjidimitrakis1, Rossella Breveglieri1, Annalisa Bosco1 and Claudio Galletti1 1 University of Bologna, Pharmacy and Biotechnology, Italy A crucial region of the brain involved in controlling visually guided arm actions in both human and non-human primates is the medial part of posterior parietal cortex. Within this cortical sector, V6A represents a node of the parieto-frontal network involved in arm movement control. Here we show that V6A neurons are involved in several aspects of ocular and arm action control, together with the encoding of sensory signals dealing with somatosensory and visual monitoring of prensile actions. The encoding of action-in-depth by single cells had not been studied till recently in V6A. Here we show that the activity of many V6A neurons is modulated by vergence eye movements aimed at fixating visual targets in depth. These signals are integrated, often at the level of single cells, with information about the direction of gaze, thus encoding spatial locations in 3D space. Moreover, 3D eye position signals seem to be further exploited at two additional levels of neural processing: a) in determining whether targets are located in the peripersonal or extrapersonal space, and b) in shaping the spatial tuning of arm movement related activity towards reachable targets. In the majority of the cells, a significant effect of both target direction and depth was found in all epochs of an instructed-delay reaching task performed in darkness. Spatial modulations of fixation activity were generally maintained across planning and subsequent reach-related epochs. Spatial preferences were kept across epochs and were evenly distributed throughout the reachable space. These findings are in line with studies in putative homologous regions in human medial posterior parietal cortex and point to a role of this cortical sector in the processing of eye position signals in order to jointly encode spatial location and hand movement information. Acknowledgements This work was supported by European Union Grant FP7-IST-217077-EYESHOTS, by Ministero dell’Universita` e della Ricerca (Italy), and by Fondazione del Monte di Bologna e Ravenna (Italy). Keywords: reaching movements, depth, Eye Movements, planning, movement preparation, Executive Function Conference: 4th Conference of the Mediterrarnean Neuroscience Society, Istanbul, Turkey, 30 Sep - 3 Oct, 2012. Presentation Type: Poster Presentation Topic: Abstracts Citation: Fattori P, Hadjidimitrakis K, Breveglieri R, Bosco A and Galletti C (2013). The medial posterior parietal area V6A and the control of arm actions in depth. Conference Abstract: 4th Conference of the Mediterrarnean Neuroscience Society. doi: 10.3389/conf.fnhum.2013.210.00033 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 25 Jan 2013; Published Online: 11 Apr 2013. * Correspondence: Prof. Patrizia Fattori, University of Bologna, Pharmacy and Biotechnology, Bologna, 40126, Italy, patrizia.fattori@unibo.it Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Patrizia Fattori Kostas Hadjidimitrakis Rossella Breveglieri Annalisa Bosco Claudio Galletti Google Patrizia Fattori Kostas Hadjidimitrakis Rossella Breveglieri Annalisa Bosco Claudio Galletti Google Scholar Patrizia Fattori Kostas Hadjidimitrakis Rossella Breveglieri Annalisa Bosco Claudio Galletti PubMed Patrizia Fattori Kostas Hadjidimitrakis Rossella Breveglieri Annalisa Bosco Claudio Galletti Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

5.3: Active Light Field Rendering in Multi‐View Display Systems

AbstractTo provide clear 3D images regardless of the viewer's 3‐dimensional position, we proposed a light field rendering method in multi‐view displays. In the method, we process each light element individually for the viewer's 3D eye positions obtained by eye detection unit. To deliver clear images to the directions of the eyes, we used the models of the angular visibility responses of the light elements. Combining directional rendering with contents matching for the light field elements, we can provide clear image with realistic parallax on multi‐view displays, even whose number of views are not so large.

Three-dimensional eye position signals shape both peripersonal space and arm movement activity in the medial posterior parietal cortex

Research conducted over the last decades has established that the medial part of posterior parietal cortex (PPC) is crucial for controlling visually guided actions in human and non-human primates. Within this cortical sector there is area V6A, a crucial node of the parietofrontal network involved in arm movement control in both monkeys and humans. However, the encoding of action-in-depth by V6A cells had been not studied till recently. Recent neurophysiological studies show the existence in V6A neurons of signals related to the distance of targets from the eyes. These signals are integrated, often at the level of single cells, with information about the direction of gaze, thus encoding spatial location in 3D space. Moreover, 3D eye position signals seem to be further exploited at two additional levels of neural processing: (a) in determining whether targets are located in the peripersonal space or not, and (b) in shaping the spatial tuning of arm movement related activity toward reachable targets. These findings are in line with studies in putative homolog regions in humans and together point to a role of medial PPC in encoding both the vergence angle of the eyes and peripersonal space. Besides its role in spatial encoding also in depth, several findings demonstrate the involvement of this cortical sector in non-spatial processes.

A robust eye gaze tracking method based on a virtual eyeball model

Gaze positions can provide important cues for natural computer interfaces. In this paper, we describe a new gaze estimation method based on a three dimensional analysis of the human eye which can be used in head-mounted display (HMD) environments. This paper presents four advantages over previous works. First, in order to obtain accurate gaze positions, we used a virtual eyeball model based on the 3D characteristics of the human eyeball. Second, we calculated the 3D position of the virtual eyeball and gaze vector by using a camera and three collimated IR-LEDs. Third, three reference frames (the camera, the monitor and the eye reference frames) were unified, which simplified the complex 3D converting calculations and allowed for calculation of the 3D eye position and gaze position on a HMD monitor. Fourth, a simple user-dependent calibration method was proposed by gazing at one position based on Kappa compensation. Experimental results showed that the eye gaze estimation error of the proposed method was lower than 1°.

Real-Time Facial and Eye Gaze Tracking System

The goal of gaze detection is to locate the position (on a monitor) where a user is looking. Previous researches use one wide view camera, which can capture the user's entire face. However, the image resolution is too low with such a camera and the fine movements of user's eye cannot be exactly detected. So, we propose the new gaze detection system with dual cameras (a wide and a narrow view camera). In order to locate the user's eye position accurately, the narrow-view camera has the functionalities of auto focusing/panning/tilting based on the detected 3D eye positions from the wide view camera. In addition, we use the IR-LED illuminators for wide and narrow view camera, which can ease the detecting of facial features, pupil and iris position. To overcome the problem of specular reflection on glasses by illuminator, we use dual IR-LED illuminators for wide and narrow view camera and detect the accurate eye position, which is not hidden by the specular reflection. Experimental results show that the gaze detection error between the computed positions and the real ones is about 2.89 cm of RMS error.

Convergence reduces ocular counterroll (OCR) during static roll-tilt

When humans are roll-tilted around the naso-occipital axis, both eyes roll or tort in the opposite direction to roll-tilt, a phenomenon known as ocular counterroll (OCR). While the magnitude of OCR is primarily determined by vestibular, somatosensory, and proprioceptive input, direction of gaze also plays a major role. The aim of this study was to measure the interaction between some of these factors in the control of OCR. Videooculography was used to measure 3D eye position during maintained whole body (en bloc) static roll-tilt in darkness, while subjects fixated first on a distant (at 130 cm) and then a near (at 30 cm) head-fixed target aligned with the subject’s midline. We found that while converging on the near target, human subjects displayed a significant reduction in OCR for both directions of roll-tilt––i.e. the interaction between OCR and vergence was not simple addition or subtraction of torsion induced by vergence with torsion induced by roll-tilt. To remove the possibility that the OCR reduction may be associated with the changed horizontal position of the eye in the orbit during symmetric convergence, we ran an experiment using asymmetric convergence in which the distant and near targets were aligned directly in front of one eye. We found the magnitude of OCR in this asymmetric convergence case was also reduced for near viewing by about the same amount as in the symmetric vergence condition, confirming that the convergence command rather than horizontal position of the eye underlies the OCR reduction, since there was no horizontal movement of the aligned eye in the orbit between fixation on the distant and near targets. Increasing vergence from 130 to 30 cm reduced OCR gain by around 35% on average. That reduction was equal in both eyes and occurred in both the symmetric and asymmetric convergence conditions. These results demonstrate the important role vergence plays in determining ocular counterroll during roll-tilt and may support the contention that vergence acts to reduce the conflict facing a stereopsis-generating mechanism.

Three-dimensional eye position during static roll and pitch in humans

We investigated how three-dimensional (3D) eye position is influenced by static head position relative to gravity, a reflex probably mediated by the otolith organs. In monkeys, the torsional component of eye position is modulated by gravity, but little data is available in humans. Subjects were held in different head/body tilts in roll and pitch for 35 s while we measured 3D eye position with scleral coils, and we used methods that reduced torsion artifacts produced by the eyelids pressing on the contact lens and exit wire. 3D eye positions were described by planar fits to the data (Listing's plane), and changes in these planes showed how torsion varied with head position. Similar to findings in monkeys, the eyes counterrolled during roll tilts independent of horizontal and vertical eye position, reaching a maximum torsion of 4.9°. Counterroll was not proportional to the shear force on the macula of the utricles: gain (torsion/sine of the head roll angle) decreased by 50% from near upright to ear down. During pitch forward, torsion increased when subjects looked right, and decreased when they looked left. However, the maximum change of torsion was only −0.06° per degree of horizontal eye position, which is less than reported in monkey. Also in contrast to monkey, we found little change in torsion when subjects were pitched backwards.

Three-dimensional extraocular motoneuron innervation in the rhesus monkey. I: Muscle rotation axes and on-directions during fixation.

The rotation axis for each of the six extraocular muscles was determined in four eyes from three perfused rhesus monkeys. Measurements of the locations of muscle insertions and origins were made in the stereotaxic reference frame with the x-y plane horizontal and the x-z plane sagittal. The computed rotation axes of the horizontal recti were close to being in the x-z plane at an angle of about 15 degrees to the z axis. The rotation axes of the vertical recti and the obliques were close to being in the x-y plane at an angle of about 30 degrees to the y axis. In five alert rhesus monkeys, we simultaneously recorded extraocular motoneuron activity and eye position in three dimensions (3D). The activity of 51 motoneuron axons was obtained from the oculomotor (n=34), trochlear (n=11), and abducens nerve (n=6) during spontaneous eye movements. To extend the torsional range of eye position, the animals were also put in different static roll positions, which induced ocular counterroll without dynamic vestibular stimulation. Periods of 100 ms during fixation or slow eye movements (<10 degrees/s) were chosen for analysis. For each motoneuron, a multiple linear regression was performed between firing frequency and 3D eye position, expressed as a rotation vector, in both stereotaxic and Listing's reference frame. The direction with the highest correlation coefficient (average R=0.94+/-0.07 SD) was taken as the on-direction. Each unit's activity could be unequivocally attributed to one particular muscle. On-directions for each motoneuron were confined to a well-defined cone in 3D. Average on-directions of motoneurons differed significantly from the corresponding anatomically determined muscle rotation axes expressed in the stereotaxic reference frame (range of deviations: 11.9 degrees to 29.0 degrees). This difference was most pronounced for the vertical recti and oblique muscles. The muscle rotation axes of the vertical rectus pair and the oblique muscle pair form an angle of 58.3 degrees, whereas the corresponding angle for paired motoneuron on-directions was 105.6 degrees. On-directions of motoneurons were better aligned with the on-directions of semicircular canal afferents (range of deviation: 9.4-18.9 degrees) or with the anatomically determined sensitivity vectors of the semicircular canals (range of deviation: 3.9-15.9 degrees) than with the anatomically determined muscle rotation axes, but significant differences remain to be explained. The on-directions of motoneurons were arranged symmetrically to Listing's plane, in the sense that the torsional components for antagonistically paired muscles were almost equal, but of opposite sign. Thus, the torsional components of motoneuron on-directions cancel when eye movements are confined to Listing's plane. This arrangement simplifies the neuronal transformations for conjugate head-fixed voluntary eye movements, while the approximate alignment with the semicircular canal reference frame is optimal for generating compensatory eye movements.

Task-dependent changes in the shape and thickness of Listing's plane

We examined the 2D surface formed by 3D eye positions of normal subjects to determine whether the shape and thickness changed in tasks that differed in saccadic directions; random, horizontal, vertical, radial, clockwise and counter-clockwise. Eye positions during the random task did not lie precisely on Listing's plane but on a surface with a small twist. This twist was present before, during, and after saccades. The degree of twist changed with the task; becoming less twisted for horizontal tasks and more twisted in the vertical tasks. The surface thickness changed with the task becoming thicker for multidirectional tasks. This greater thickness may occur because surfaces obtained in multidirectional tasks are the composite of surfaces with slightly different shapes.

システムニューロサイエンス教教育講座 ７． なぜ今Ｌｉｓｔｉｎｇなのか？

The anatomical configuration of the eye ball in the orbit is similar to that of a ball-and-socket joint, and the arrangement of extra-ocular muscles gives the globe three rotational degrees of freedom (DOF). If one describes a three-dimensional (3D) eye position using three consecutive rotations about the horizontal, vertical, and torsional eye-fixed axes, the value of torsion, so-called false torsion, depends on the order of rotation. This is a result of the non-commutativity of rotatory movements. The problem of false torsion can be avoided by representing a 3D eye position by a single axis rotation from a chosen reference position to the current eye position, e. g. rotation vectors (Haustein, 1989).According to Listing's law, all axes about which the eye rotates from the reference position to other positions lie in a plane, so-called Listing's plane, provided the head is erect and stationary (Helmholtz, 1867). If the reference position coincides with the primary position, Listing's plane is perpendicular to the direction of gaze in primary position. Listing's law is a remarkable example, of how a motor system reduces DOF (from 3 to 2) to simplify multidimensional motor control.Characteristic changes in the orientation of Listing's plane have been reported under several conditions: Static tilts in the frontal plane of the head induce ocular counterrolling, i.e. parallel shifts of Listing's plane along the torsional axis. Static tilts in the sagittal plane counter-rotate Listing's plane about the vertical axis. Convergence leads to a temporal rotation of Listing's plane about the horizontal axis. During sleep, Listing's plane is not preserved, which implies neural implementation of Listing's law.