3D Gaze Research Articles

Towards free 3D end-point control for robotic-assisted human reaching using binocular eye tracking.

Eye-movements are the only directly observable behavioural signals that are highly correlated with actions at the task level, and proactive of body movements and thus reflect action intentions. Moreover, eye movements are preserved in many movement disorders leading to paralysis (or amputees) from stroke, spinal cord injury, Parkinson's disease, multiple sclerosis, and muscular dystrophy among others. Despite this benefit, eye tracking is not widely used as control interface for robotic interfaces in movement impaired patients due to poor human-robot interfaces. We demonstrate here how combining 3D gaze tracking using our GT3D binocular eye tracker with custom designed 3D head tracking system and calibration method enables continuous 3D end-point control of a robotic arm support system. The users can move their own hand to any location of the workspace by simple looking at the target and winking once. This purely eye tracking based system enables the end-user to retain free head movement and yet achieves high spatial end point accuracy in the order of 6 cm RMSE error in each dimension and standard deviation of 4 cm. 3D calibration is achieved by moving the robot along a 3 dimensional space filling Peano curve while the user is tracking it with their eyes. This results in a fully automated calibration procedure that yields several thousand calibration points versus standard approaches using a dozen points, resulting in beyond state-of-the-art 3D accuracy and precision.

Remote gaze tracking system for 3D environments.

Eye tracking systems are typically divided into two categories: remote and mobile. Remote systems, where the eye tracker is located near the object being viewed by the subject, have the advantage of being less intrusive, but are typically used for tracking gaze points on fixed two dimensional (2D) computer screens. Mobile systems such as eye tracking glasses, where the eye tracker are attached to the subject, are more intrusive, but are better suited for cases where subjects are viewing objects in the three dimensional (3D) environment. In this paper, we describe how remote gaze tracking systems developed for 2D computer screens can be used to track gaze points in a 3D environment. The system is non-intrusive. It compensates for small head movements by the user, so that the head need not be stabilized by a chin rest or bite bar. The system maps the 3D gaze points of the user onto 2D images from a scene camera and is also located remotely from the subject. Measurement results from this system indicate that it is able to estimate gaze points in the scene camera to within one degree over a wide range of head positions.

Evaluating 3D gaze tracking in virtual space: A computer graphics approach

Increasing usage of stereoscopic 3D technology in virtual reality, video games, entertainment, and visualization has risen concern on development of gaze-based interaction. To develop intuitive and accurate gaze-based interaction, estimation of 3D gaze in virtual space should be validated experimentally. 3D gaze tracking is generally performed in real space with rigid object as validation target. Thus, researchers in virtual reality are constrained on choosing appropriate evaluation method when 3D gaze tracking has to be performed in virtual space. To fill this research gap, we present design and development of a new evaluation method for 3D gaze tracking in virtual space. We have implemented computer graphics technology to develop virtual plane containing virtual 3D object as validation target. Experimental results show that the proposed evaluation method was able to support real experiment by proving the accuracy of our 3D gaze tracking system with average Euclidean error less than 1cm (Mean=0.95cm; S.D=0.55cm) in 74cm depth of workspace. Compared with evaluation method for 3D gaze tracking in real space, the proposed method even can be implemented when space of experiment room is limited by adjusting the distance of virtual plane programmatically.

Calibration method for 3D gaze tracking systems

In this study, a novel method that calibrates 3D gaze tracking systems using multi-camera global calibration is proposed. In the proposed method, a calibration camera placed in front of the computer screen and the system camera constitute a multi-camera system. Video images are acquired by this multi-camera system to estimate the position of the light sources and the computer screen relative to the system camera. The method first evaluates the transition matrix between the system camera and calibration camera, and then computes the relative position of camera and targets such as light sources and monitor. Finally, it determines the target position in the system camera’s coordinate system. Experiments conducted on a system containing one camera and two light sources verify that the proposed calibration method is accurate. The method overcomes the problem of the screen and light sources not being in view of the system camera. Further, it is relatively simple to apply, is very accurate, and provides a new basis for many other similar calibration systems.

Estimating 3D Gaze Directions Using Unlabeled Eye Images via Synthetic Iris Appearance Fitting

Estimating three-dimensional (3D) human eye gaze by capturing a single eye image without active illumination is challenging. Although the elliptical iris shape provides a useful cue, existing methods face difficulties in ellipse fitting due to unreliable iris contour detection. These methods may fail frequently especially with low resolution eye images. In this paper, we propose a synthetic iris appearance fitting (SIAF) method that is model-driven to compute 3D gaze direction from iris shape. Instead of fitting an ellipse based on exactly detected iris contour, our method first synthesizes a set of physically possible iris appearances and then optimizes inside this synthetic space to find the best solution to explain the captured eye image. In this way, the solution is highly constrained and guaranteed to be physically feasible. In addition, the proposed advanced image analysis techniques also help the SIAF method be robust to the unreliable iris contour detection. Furthermore, with multiple eye images, we propose a SIAF-joint method that can further reduce the gaze error by half, and it also resolves the binary ambiguity which is inevitable in conventional methods based on simple ellipse fitting.

3D Gaze tracking by combining eye- and facial-gaze vectors

We propose a 3D gaze-tracking method that combines accurate 3D eye- and facial-gaze vectors estimated from a Kinect v2 high-definition face model. Using accurate 3D facial and ocular feature positions, gaze positions can be calculated more accurately than with previous methods. Considering the image resolution of the face and eye regions, two gaze vectors are combined as a weighted sum, allocating more weight to facial-gaze vectors. Hence, the facial orientation mainly determines the gaze position, and eye-gaze vectors then perform minor manipulations. The 3D facial-gaze vector is first defined, and the 3D rotational center of the eyeball is then estimated; together, these define the 3D eye-gaze vector. Finally, the intersection point between the 3D gaze vector and the physical display plane is calculated as the gaze position. Experimental results show that the average gaze estimation root-mean-square error was approximately 23 pixels from the desired position at a resolution of $$1920\times 1080$$1920×1080.

Realtime 3D eye gaze animation using a single RGB camera

This paper presents the first realtime 3D eye gaze capture method that simultaneously captures the coordinated movement of 3D eye gaze, head poses and facial expression deformation using a single RGB camera. Our key idea is to complement a realtime 3D facial performance capture system with an efficient 3D eye gaze tracker. We start the process by automatically detecting important 2D facial features for each frame. The detected facial features are then used to reconstruct 3D head poses and large-scale facial deformation using multi-linear expression deformation models. Next, we introduce a novel user-independent classification method for extracting iris and pupil pixels in each frame. We formulate the 3D eye gaze tracker in the Maximum A Posterior (MAP) framework, which sequentially infers the most probable state of 3D eye gaze at each frame. The eye gaze tracker could fail when eye blinking occurs. We further introduce an efficient eye close detector to improve the robustness and accuracy of the eye gaze tracker. We have tested our system on both live video streams and the Internet videos, demonstrating its accuracy and robustness under a variety of uncontrolled lighting conditions and overcoming significant differences of races, genders, shapes, poses and expressions across individuals.

Gaze Estimation in the 3D Space Using RGB-D Sensors

We address the problem of 3D gaze estimation within a 3D environment from remote sensors, which is highly valuable for applications in human---human and human---robot interactions. To the contrary of most previous works, which are limited to screen gazing applications, we propose to leverage the depth data of RGB-D cameras to perform an accurate head pose tracking, acquire head pose invariance through a 3D rectification process that renders head pose dependent eye images into a canonical viewpoint, and computes the line-of-sight in the 3D space. To address the low resolution issue of the eye image resulting from the use of remote sensors, we rely on the appearance based gaze estimation paradigm, which has demonstrated robustness against this factor. In this context, we do a comparative study of recent appearance based strategies within our framework, study the generalization of these methods to unseen individual, and propose a cross-user eye image alignment technique relying on the direct registration of gaze-synchronized eye images. We demonstrate the validity of our approach through extensive gaze estimation experiments on a public dataset as well as a gaze coding task applied to natural job interviews.

Embodied salience for gaze analysis in ecologically valid environments

The brain is a dynamical system, mapping sensory inputs to motor actions. This relationship has been widely characterised by reductionist controlled lab experiments. However, with the emergence of mobile eye-tracking, increasing emphasis has been placed on the ecological validity of gaze studies, taking them out of the lab and into the wild (Hayhoe & Ballard, 2005; Kingstone et al., 2003; Land & Tatler, 2009). Here we build on this by capturing, rather than constraining, sensory inputs and motor outputs natural behaviour. We record 90% of sensory inputs using head mounted eye-tracking, scene camera and microphone. Simultaneously, recording 95% of skeletal motor outputs by motion tracking 51 degrees of freedom the body and a total of 40 degrees of freedom the hands. All tracking equipment is markerless and thus allows unconstrained behavioural monitoring in the wild. The eye-tracker data is processed post-hoc to give 3D gaze position relative to the subjects' head and limb endpoints using our GT3D decoding method (Abbott & Faisal, 2011; Abbott & Faisal, 2012) and the motion capture data. This enables us to evaluate classical relationships ecologically valid environments including 3 daily scenarios: breakfast the kitchen, evening activities the home and in-door ambulation. We find that the classical categorisation of gaze data to saccades and fixations is insufficient to capture eye-movement repertoire natural behaviour. We spend the majority of daily life making smooth eye-movements directly coupled to body movements (eg VOR). Classically, allocation of gaze has been attributed to both bottom up scene salience and top down ongoing task demands. We propose a new method for analysing and interpreting eye-movements the wild, by relating them directly to body posture. Thus, the interpretation of gaze and attention is integrated, not disembodied, from the kinematics of motor behaviour. Meeting abstract presented at VSS 2015. Language: en

Real-Time Gaze Estimation with Online Calibration

Gaze-tracking technology is highly valuable in many interactive and diagnostic applications. For many gaze estimation systems, calibration is an unavoidable procedure necessary to determine certain person-specific parameters, either explicitly or implicitly. Recently, several offline implicit calibration methods have been proposed to ease the calibration burden. However, the calibration procedure is still cumbersome, and gaze estimation accuracy needs further improvement. In this article, the authors present a novel 3D gaze estimation system with online calibration. The proposed system is based on a new 3D model-based gaze estimation method using a single consumer depth camera sensor (via Kinect). Unlike previous gaze estimation methods using explicit offline calibration with fixed number of calibration points or implicit calibration, their approach constantly improves person-specific eye parameters through online calibration, which enables the system to adapt gradually to a new user. The experimental results and the human-computer interaction (HCI) application show that the proposed system can work in real time with superior gaze estimation accuracy and minimal calibration burden.

Investigation of visually induced motion sickness in dynamic 3D contents based on subjective judgment, heart rate variability, and depth gaze behavior.

Visually induced motion sickness (VIMS) is an important safety issue in stereoscopic 3D technology. Accompanying subjective judgment of VIMS with objective measurement is useful to identify not only biomedical effects of dynamic 3D contents, but also provoking scenes that induce VIMS, duration of VIMS, and user behavior during VIMS. Heart rate variability and depth gaze behavior are appropriate physiological indicators for such objective observation. However, there is no information about relationship between subjective judgment of VIMS, heart rate variability, and depth gaze behavior. In this paper, we present a novel investigation of VIMS based on simulator sickness questionnaire (SSQ), electrocardiography (ECG), and 3D gaze tracking. Statistical analysis on SSQ data shows that nausea and disorientation symptoms increase as amount of dynamic motions increases (nausea: p<;0.005; disorientation: p<;0.05). To reduce VIMS, SSQ and ECG data suggest that user should perform voluntary gaze fixation at one point when experiencing vertical motion (up or down) and horizontal motion (turn left and right) in dynamic 3D contents. Observation of 3D gaze tracking data reveals that users who experienced VIMS tended to have unstable depth gaze than ones who did not experience VIMS.

How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3D + time) environment during a joint attention induction task with a robot

We aimed to compare, during a joint attention (JA) elicitation task, how children with autism spectrum disorder (ASD) and children with typical development (TD) behave and explore their 4 dimensional (meaning spatial 3D+time) when interacting with a human or with a robotic agent.We built a system that employed a Nao robot and a perception system based on a RGB-D sensor (Kinect) to capture social engagement cues. A JA induction experiment was performed in which children with ASD (N=16) and matched TD children (N=16) had a 3-min interaction with the robot or with a therapist. Nao induced JA by gazing; by gazing and pointing; and by gazing, pointing and vocalizing at pictures. Both groups of children performed well with the therapist. However, with Nao, both groups had lower JA scores, and the children with ASD had a significantly lower score than the TD children. We found that (i) multimodal JA induction was more efficient in both groups; (ii) the 3D spatial world gaze exploration showed less accuracy; and (iii) the trunk position in ASD showed less stability in the 4 dimensions compared to TD controls.We conclude that, in ASD, JA skill depends on the interaction partner, and implies a higher motor and cognitive cost.

GazeVis: Interactive 3D Gaze Visualization for Contiguous Cross-Sectional Medical Images.

Gaze visualization has been used to understand the results from gaze tracking studies in a wide range of fields. In the medical field, diagnoses of medical images have been studied with gaze tracking technology to understand how radiologists read medical images. While prior work were mainly based on diagnosis with a single image, recent work focused on diagnosis with consecutive cross-sectional medical images acquired from preoperative computed tomography (CT) or magnetic resonance imaging (MRI). In the diagnosis, radiologists scroll through a stack of images to get a 3D cognition of organs and lesions. Thus, it is important to understand radiologists' gaze patterns three dimensionally across such contiguous cross-sectional images. However, little has been done to visualize more complicated gaze patterns from the contiguous cross-sectional medical images. To address this problem, we present an interactive 3D gaze visualization tool, GazeVis, where InfoVis and SciVis techniques are harmonized to show the abstract gaze data along with a realistic 3D rendering of the visual stimuli (i.e., organs and lesions). We present case studies with 12 radiologists who use GazeVis to investigate gaze patterns of their colleagues with different levels of expertise, providing empirical evidences about the competence of our gaze visualization system.

Online 3D Gaze Localization on Stereoscopic Displays

This article summarizes our previous work on developing an online system to allow the estimation of 3D gaze depth using eye tracking in a stereoscopic environment. We report on recent extensions allowing us to report the full 3D gaze position. Our system employs a 3D calibration process that determines the parameters of a mapping from a naive depth estimate, based simply on triangulation, to a refined 3D gaze point estimate tuned to a particular user. We show that our system is an improvement on the geometry-based 3D gaze estimation returned by a proprietary algorithm provided with our tracker. We also compare our approach with that of the Parameterized Self-Organizing Map (PSOM) method, due to Essig and colleagues, which also individually calibrates to each user. We argue that our method is superior in speed and ease of calibration, is easier to implement, and does not require an iterative solver to produce a gaze position, thus guaranteeing computation at the rate of tracker acquisition. In addition, we report on a user study that indicates that, compared with PSOM, our method more accurately estimates gaze depth, and is nearly as accurate in estimating horizontal and vertical position. Results are verified on two different 4D eye tracking systems, a high accuracy Wheatstone haploscope and a medium accuracy active stereo display. Thus, it is the recommended method for applications that primarily require gaze depth information, while its ease of use makes it suitable for many applications requiring full 3D gaze position.

Cell-based visual surveillance with active cameras for 3D human gaze computation

Capturing fine resolution and well-calibrated video images with good object visual coverage in a wide space is a tough task for visual surveillance. Although the use of active cameras is an emerging method, it suffers from the problems of online camera calibration difficulty, mechanical delay handling, image blurring from motions, and algorithm un-friendly due to dynamic backgrounds, etc. This paper proposes a cell-based visual surveillance system by using N (N???2) active cameras. We propose the camera scan speed map (CSSM) to deal with the practical mechanical delay problem for active camera system design. We formulate the three mutually-coupled problems of camera layout, surveillance space partition with cell sequence, and camera parameter control, into an optimization problem by maximizing the object resolution while meeting various constraints such as system mechanical delay, full visual coverage, minimum object resolution, etc. The optimization problem is solved by using a full searching approach. The cell-based calibration method is proposed to compute both the intrinsic and exterior parameters of active cameras for different cells. With the proposed system, the foreground object is detected based on motion and appearance features and tracked by dynamically switching the two groups of cameras across different cells. The proposed algorithms and system have been validated by an in-door surveillance experiment, where the surveillance space was partitioned into four cells. We used two active cameras with one camera in one group. The active cameras were configured with the optimized pan, tilt, and zooming parameters for different cells. Each camera was calibrated with the cell-based calibration method for each configured pan, tilt, and zooming parameters. The algorithms and system were applied to monitor freely moving peoples within the space. The system can capture good resolution, well-calibrated, and good visual coverage video images with static background in support of automatic object detection and tracking. The proposed system performed better than traditional single or multiple fixed camera system in term of image resolution, surveillance space, etc. We further demonstrated that advanced 3D features, such as 3D gazes, were successfully computed from the captured good-quality images for intelligent surveillance.

Frames of reference for eye–head gaze shifts evoked during frontal eye field stimulation

The frontal eye field (FEF), in the prefrontal cortex, participates in the transformation of visual signals into saccade motor commands and in eye-head gaze control. The FEF is thought to show eye-fixed visual codes in head-restrained monkeys, but it is not known how it transforms these inputs into spatial codes for head-unrestrained gaze commands. Here, we tested if the FEF influences desired gaze commands within a simple eye-fixed frame, like the superior colliculus (SC), or in more complex egocentric frames like the supplementary eye fields (SEFs). We electrically stimulated 95 FEF sites in two head-unrestrained monkeys to evoke 3D eye-head gaze shifts and then mathematically rotated these trajectories into various reference frames. In theory, each stimulation site should specify a specific spatial goal when the evoked gaze shifts are plotted in the appropriate frame. We found that these motor output frames varied site by site, mainly within the eye-to-head frame continuum. Thus, consistent with the intermediate placement of the FEF within the high-level circuits for gaze control, its stimulation-evoked output showed an intermediate trend between the multiple reference frame codes observed in SEF-evoked gaze shifts and the simpler eye-fixed reference frame observed in SC-evoked movements. These results suggest that, although the SC, FEF and SEF carry eye-fixed information at the level of their unit response fields, this information is transformed differently in their output projections to the eye and head controllers.

Mapping the Navigational Information Content of Insect Habitats

For developing and validating models of insect navigation it is essential to identify the visual input insects experience in their natural habitats. Here we report on the development of methods to reconstruct what insects see when making navigational decisions and critically assess the current limitations of such methods. We used a laser-range finder as well as camera-based methods to capture the 3D structure and the appearance of outdoor environments. Both approaches produce coloured point clouds that allow within the model scale the reconstruction of views at defined positions and orientations. For instance, we filmed bees and wasps with a high-speed stereo camera system to estimate their 3D flight paths and gaze direction. The high-speed system is registered with a 3D model of the same environment, such that panoramic images can be rendered along the insects’ flight paths (see accompanying abstract “Benchmark 3D-models of natural navigation environments @ www.InsectVision.org” by Mair et al.). The laser-range finder (see figure A) is equipped with a rotating camera that provides colour information for the measured 3D points. This system is robust and easy-to-use in the field generating high resolution data (about 50 × 106 points) with large field of view, up to a distance of 80 m at typical acquisition times of about 8 minutes. However, a large number of scans at different locations has to be recorded and registered to account for occlusions. In comparison, data acquisition in camera-based reconstruction from multiple view-points is fast, but model generation is computationally more complex due to bundle adjustment and dense pair-wise stereo computation (see figure B, C for views rendered from a 3D model based on 6 image pairs). In addition it is non-trivial and often time-consuming in the field to ensure the acquisition of sufficient information. We are currently developing the tools that will allow us to combine the results of laser-scanner and camera-based 3D reconstruction methods.

Visual gaze analysis of robotic pedestrians moving in urban space

This study is founded on the idea that an analysis of the visual gaze dynamics of pedestrians can increase our understanding of how important architectural features in urban environments are perceived by pedestrians. The results of such an analysis can lead to improvements in urban design. However, a technical challenge arises when trying to determine the gaze direction of pedestrians recorded on video. High ‘noise’ levels and the subtlety of human gaze dynamics hamper precise calculations. However, as robots can be programmed and analysed more efficiently than humans, this study uses them for developing and training a gaze analysis system with the aim to later apply it to human video data using the machine learning technique of manifold alignment. For this study, a laboratory was set up to become a model street scene in which autonomous humanoid robots of approximately 55 cm in height simulate the behaviour of human pedestrians. The experiments compare the inputs from several cameras as the robot walks down the model street and changes its behaviour upon encountering ‘visually attractive objects’. Overhead recordings and the robot's internal joint signals are analysed after filtering to provide ‘true’ data against which the recorded data can be compared for accuracy testing. A central component of the research is the calculation of a torus-like manifold that represents all the different three-dimensional (3D) head directions of a robot head and allows the ordering of extracted 3D gaze vectors obtained from video sequences. We briefly describe how the obtained multidimensional trajectory data can be analysed by using a temporal behaviour analysis technique based on support vector machines that was developed separately.

Ultra-low-cost 3D gaze estimation: an intuitive high information throughput compliment to direct brain–machine interfaces

Eye movements are highly correlated with motor intentions and are often retained by patients with serious motor deficiencies. Despite this, eye tracking is not widely used as control interface for movement in impaired patients due to poor signal interpretation and lack of control flexibility. We propose that tracking the gaze position in 3D rather than 2D provides a considerably richer signal for human machine interfaces by allowing direct interaction with the environment rather than via computer displays. We demonstrate here that by using mass-produced video-game hardware, it is possible to produce an ultra-low-cost binocular eye-tracker with comparable performance to commercial systems, yet 800 times cheaper. Our head-mounted system has 30 USD material costs and operates at over 120 Hz sampling rate with a 0.5–1 degree of visual angle resolution. We perform 2D and 3D gaze estimation, controlling a real-time volumetric cursor essential for driving complex user interfaces. Our approach yields an information throughput of 43 bits s−1, more than ten times that of invasive and semi-invasive brain–machine interfaces (BMIs) that are vastly more expensive. Unlike many BMIs our system yields effective real-time closed loop control of devices (10 ms latency), after just ten minutes of training, which we demonstrate through a novel BMI benchmark-–the control of the video arcade game ‘Pong’.

Using ultra-low cost 3D gaze tracking as an intuitive, non-invasive, high information throughput alternative to BMIs.

Using ultra-low cost 3D gaze tracking as an intuitive, non-invasive, high information throughput alternative to BMIs.