Gaze Estimation Accuracy Research Articles

I2DNet - Design and real-time evaluation of an appearance-based gaze estimation system

Gaze estimation problem can be addressed using either model-based or appearance-based approaches. Model-based approaches rely on features extracted from eye images to fit a 3D eye-ball model to obtain gaze point estimate while appearance-based methods attempt to directly map captured eye images to gaze point without any handcrafted features. Recently, availability of large datasets and novel deep learning techniques made appearance-based methods achieve superior accuracy than model-based approaches. However, many appearance- based gaze estimation systems perform well in within-dataset validation but fail to provide the same degree of accuracy in cross-dataset evaluation. Hence, it is still unclear how well the current state-of-the-art approaches perform in real-time in an interactive setting on unseen users. This paper proposes I2DNet, a novel architecture aimed to improve subject- independent gaze estimation accuracy that achieved a state-of-the-art 4.3 and 8.4 degree mean angle error on the MPIIGaze and RT-Gene datasets respectively. We have evaluated the proposed system as a gaze-controlled interface in real-time for a 9-block pointing and selection task and compared it with Webgazer.js and OpenFace 2.0. We have conducted a user study with 16 participants, and our proposed system reduces selection time and the number of missed selections statistically significantly compared to other two systems.

Transmission of correct gaze direction in video conferencing using screen-embedded cameras

In this paper, we propose a new video conferencing system that presents correct gaze directions of a remote user by switching among images obtained from multiple cameras embedded in a screen according to a local user’s position. Our proposed method reproduces a situation like that in which the remote user is in the same space as the local user. The position of the remote user to be displayed on the screen is determined so that the positional relationship between the users is reproduced. The system selects one of the embedded cameras whose viewing direction towards the remote user is the closest to the local user’s viewing direction to the remote user’s image on the screen. As a result of quantitative evaluation, we confirmed that, in comparison with the case using a single camera, the accuracy of gaze estimation was improved by switching among the cameras according to the position of the local user.

Automatic pupil detection and gaze estimation using the vestibulo-ocular reflex in a low-cost eye-tracking setup

Automatic eye tracking is of interest for interaction with people suffering from amyotrophic lateral sclerosis, for using the eyes to control a computer mouse, and for controlled radiotherapy of uveal melanoma. It has been speculated that gaze estimation accuracy might be improved by using the vestibulo-ocular reflex. This involuntary reflex results in slow, compensatory eye movements, opposing the direction of head motion. We therefore hypothesised that leaving the head to move freely during eye tracking must produce more accurate results than keeping the head fixed, only allowing the eyes to move. The purpose of this study was to create a low-cost eye tracking system that incorporates the vestibulo-ocular reflex in gaze estimation, by keeping the head freely moving. The instrument used comprised a low-cost head-mounted webcam which recorded a single eye. Pupil detection was fully automatic and in real time with a straightforward hybrid colour-based and model-based algorithm, despite the lower-end webcam used for recording and despite the absence of direct illumination. A model-based algorithm and an interpolation-based algorithm were tested in this study. Based on mean absolute angle difference in the gaze estimation results, we conclude that the model-based algorithm performed better when the head was not moving and equally well when the head was moving. With most deviations of the points of gaze from the target points being less than 1° using either algorithm when the head is moving freely, it can be concluded that our setup performs fully within the 2° benchmark from literature, whereas deviations when the head was not moving exceeded 2°. The algorithms used were not previously tested under passive illumination. This was the first study of a low-cost eye-tracking setup taking into account the vestibulo-ocular reflex.

A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation

Human gaze is essential for various appealing applications. Aiming at more accurate gaze estimation, a series of recent works propose to utilize face and eye images simultaneously. Nevertheless, face and eye images only serve as independent or parallel feature sources in those works, the intrinsic correlation between their features is overlooked. In this paper we make the following contributions: 1) We propose a coarse-to-fine strategy which estimates a basic gaze direction from face image and refines it with corresponding residual predicted from eye images. 2) Guided by the proposed strategy, we design a framework which introduces a bi-gram model to bridge gaze residual and basic gaze direction, and an attention component to adaptively acquire suitable fine-grained feature. 3) Integrating the above innovations, we construct a coarse-to-fine adaptive network named CA-Net and achieve state-of-the-art performances on MPIIGaze and EyeDiap.

An Eye Gaze Tracking Method of Virtual Reality Headset Using A Single Camera and Multi-light Source

For the problems the hardware configuration of multi-camera and multi-light source is complicated and the eye image of little auxiliary light source is too dark in the eye gaze tracking system, A new eye gaze tracking system based on two-dimensional mapping model is proposed, which achieved the transformation from two-dimensional coordinates to three-dimensional coordinates combined attitude angle of virtual reality headset. In this paper, the mapping relationship between visual features and eye gaze points is established by improving the method of pupil image recognition and eye gaze estimation based pupil-corneal, which the accuracy of eye gaze estimation is further improved. The experimental results show that the error of eye gaze tracking system is less than 1.1 °, which achieved the good effect of eye gaze tracking.

Appearance-based Gaze Estimation Using Multi-task Neural Network

In order to improve the gaze estimation accuracy through using the information in the eye image more adequately, an appearance-based gaze estimation neural network EyeNet has been proposed in this article. On the one hand, by decomposing the main task into two subtasks, this network could gain more information during the training stage. On the other hand, EyeNet could extract features in the image more efficiently by adding an auxiliary task which is detecting the iris center and eye ball center of the eye. Experiments show that by using these two tricks, the error decreased from 5.42 degree to 5.16 degree.

How Far Are We From Quantifying Visual Attention in Mobile HCI?

With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a core challenge in mobile human-computer interaction. Encouraged by recent advances in automatic eye contact detection using machine learning and device-integrated cameras, we provide a fundamental investigation into the feasibility of quantifying visual attention during everyday mobile interactions. We identify core challenges and sources of errors associated with sensing attention on mobile devices in the wild, including the impact of face and eye visibility, the importance of robust head pose estimation, and the need for accurate gaze estimation. Based on this analysis, we propose future research directions and discuss how eye contact detection represents the foundation for exciting new applications towards next-generation pervasive attentive user interfaces.

Discriminative Robust Head-Pose and Gaze Estimation Using Kernel-DMCCA Features Fusion

There remain outstanding challenges for improving accuracy of multi-feature information for head-pose and gaze estimation. The proposed framework employs discriminative analysis for head-pose and gaze estimation using kernel discriminative multiple canonical correlation analysis (K-DMCCA). The feature extraction component of the framework includes spatial indexing, statistical and geometrical elements. Head-pose and gaze estimation is constructed by feature aggregation and transforming features into a higher dimensional space using K-DMCCA for accurate estimation. The two main contributions are: Enhancing fusion performance through the use of kernel-based DMCCA, and by introducing an improved iris region descriptor based on quadtree. The overall approach is also inclusive of statistical and geometrical indexing that are calibration free (does not require any subsequent adjustment). We validate the robustness of the proposed framework across a wide variety of datasets, which consist of different modalities (RGB and Depth), constraints (wide range of head-poses, not only frontal), quality (accurately labelled for validation), occlusion (due to glasses, hair bang, facial hair) and illumination. Our method achieved an accurate head-pose and gaze estimation of 4.8∘ using Cave, 4.6∘ using MPII, 5.1∘ using ACS, 5.9∘ using EYEDIAP, 4.3∘ using OSLO and 4.6∘ using UULM datasets.

Accurate Regression-Based 3D Gaze Estimation Using Multiple Mapping Surfaces

Accurate 3D gaze estimation using a simple setup remains a challenging issue for head-mounted eye tracking. Current regression-based gaze direction estimation methods implicitly assume that all gaze directions intersect at one point called the eyeball pseudo-center. The effect of this implicit assumption on gaze estimation is unknown. In this paper, we find that this assumption is approximate based on a simulation of all intersections of gaze directions, and it is conditional based on a sensitivity analysis of the assumption in gaze estimation. Hence, we propose a gaze direction estimation method with one mapping surface that satisfies conditions of the assumption by configuring one mapping surface and achieving a high-quality calibration of the eyeball pseudo-center. This method only adds two additional calibration points outside the mapping surface. Furthermore, replacing the eyeball pseudo-center with an additional calibrated surface, we propose a gaze direction estimation method with two mapping surfaces that further improves the accuracy of gaze estimation. This method improves accuracy on the state-of-the-art method by 20 percent (from a mean error of 1.84 degrees to 1.48 degrees) on a public dataset with a usage range of 1 meter and by 17 percent (from a mean error of 2.22 degrees to 1.85 degrees) on a public dataset with a usage range of 2 meters.

CycleGAN-Based Deblurring for Gaze Tracking in Vehicle Environments

Recently, the need for research on tracking driver gazes is increasing owing to the development of driver convenience systems, such as autonomous driving or intelligent driver monitoring system, to address traffic accidents caused by negligence. A camera is installed in a vehicle to track the driver’s gaze in vehicle environment. The accuracy of estimating driver gazes in vehicle environments reduces if a motion blur of the driver occurs, owing to vehicular vibrations during driving. Most past studies on gaze-tracking of a driver in a vehicle did not consider the motion blurs in their experiments. To address this concern, we propose a method for improving the accuracy of gaze estimation by deblurring the blurred images of a driver from the vehicle. This study is the first attempt to calculate a driver’s gaze by deblurring a motion blurred image with CycleGAN, whereas simultaneously using the image information from the two cameras in the vehicle. In previous studies, multiple deep CNNs were used for obtaining the images of a driver’s eyes and face. In this study, information obtained from the two cameras in the vehicle are integrated into an image with three channels and thereafter deblurred, consequently reducing the time required for training. Whereas in previous studies the gaze position was not calculated for severe blurs by measuring the level of blur from the input image, the gaze position was calculated for all the input images in this study. From the database (Dongguk blurred gaze database (DBGD)) from 26 drivers in actual vehicles and the Columbia gaze dataset (CAVE-DB) that is an open database, the proposed method exhibited greater accuracy than the existing methods.

InvisibleEye

Despite their potential for a range of exciting new applications, mobile eye trackers suffer from several fundamental usability problems. InvisibleEye is an innovative approach for mobile eye tracking that uses millimeter-size RGB cameras, which can be fully embedded into normal glasses frames, as well as appearancebased gaze estimation to directly estimate gaze from the eye images. Through evaluation on three large-scale, increasingly realistic datasets, we show that InvisibleEye can achieve a person-specific gaze estimation accuracy of up to 2.04° using three camera pairs with a resolution of only 3x3 pixels.

XSDL: stroboscopic differential lighting eye tracker with extended temporal support

Eye tracking (ET) for gaze interaction in wearable computing imposes harder constraints on computational efficiency and illumination conditions than remote ET. In this paper we present xSDL, an extended temporal support computer vision algorithm for accurate, robust, and efficient pupil detection and gaze estimation. The robustness and efficiency of xSDL partly come from the use of stroboscopic differential lighting (SDL), an extension of the differential lighting pupil detection technique developed in the 90’s. Due to the erratic behavior of eye movements, traditional computer vision tracking techniques (such as Kalman filters) do not perform well, so most ET techniques simply detect some eye feature (such as the pupil center) at every frame. Extended temporal support uses keyframes selected during eye fixations and a simple translation model of the pupil to further improve the computational performance of SDL. A prototype composed of two independent acquisition systems was developed to evaluate the performance of xSDL and other four state-of-the-art ET techniques under similar conditions. Our results show that xSDL outperforms those four algorithms, both in speed (close to 2000 Hz using 240 line frames) and accuracy.

Performance Evaluation Strategies for Eye Gaze Estimation Systems with Quantitative Metrics and Visualizations.

An eye tracker’s accuracy and system behavior play critical roles in determining the reliability and usability of eye gaze data obtained from them. However, in contemporary eye gaze research, there exists a lot of ambiguity in the definitions of gaze estimation accuracy parameters and lack of well-defined methods for evaluating the performance of eye tracking systems. In this paper, a set of fully defined evaluation metrics are therefore developed and presented for complete performance characterization of generic commercial eye trackers, when they operate under varying conditions on desktop or mobile platforms. In addition, some useful visualization methods are implemented, which will help in studying the performance and data quality of eye trackers irrespective of their design principles and application areas. Also the concept of a graphical user interface software named GazeVisual v1.1 is proposed that would integrate all these methods and enable general users to effortlessly access the described metrics, generate visualizations and extract valuable information from their own gaze datasets. We intend to present these tools as open resources in future to the eye gaze research community for use and further advancement, as a contribution towards standardization of gaze research outputs and analysis.

Accurate gaze estimation based on average‐binary‐connected‐component‐centroid

Iris centre is widely used in feature-based gaze estimation system. However, the performance of various existing iris centre localisation methods is degraded in low-resolution images. To address the problem, an accurate regression-based gaze estimation system is proposed by replacing the iris centre with a novel average-binary-connected-component-centroid (ABCCC) feature. First, ABCCC is obtained by averaging a series of binary connected component centroids, which are calculated by different percentile grey thresholds in the normalised eye region of interest. Then, head pose is estimated by employing the OpenCV iterative algorithm. Thereafter, ABCCC and head pose are mapped to compute the gaze using a linear mapping functions, which are regressed in calibration phase. Compared with traditional methods using iris centre, ABCCC is easier to calculate and shows a stronger correlation with the change of gaze, which improves the accuracy of the gaze estimation system. Experimental results on EYEDIAP database verify the effectiveness of the proposed method.

Calibration free, user-independent gaze estimation with tensor analysis

Human gaze directly signals visual attention, therefore, estimation of gaze has been an important research topic in fields such as human attention modeling and human-computer interaction. Accurate gaze estimation requires user, system and even session dependent parameters, which can be obtained by calibration process. However, this process has to be repeated whenever the parameter changes (head movement, camera movement, monitor movement). This study aims to eliminate the calibration process of gaze estimation by building a user-independent, appearance-based gaze estimation model. The system is ideal for multimodal interfaces, where the gaze is tracked without the cooperation from the users. The main goal is to capture the essential representation of the gaze appearance of the target user. We investigate the tensor analysis framework that decomposes the high dimension gaze data into different factors including individual differences, gaze differences, user-screen distances and session differences. The axis that is representative for a particular subject is automatically chosen in the tensor analysis framework using LASSO regression. The proposed approaches show promising results on capturing the test subject gaze changes. To address the estimation shift caused by the variations in individual heights, or relative position to the monitor, we apply domain adaptation to adjust the gaze estimation, observing further improvements. These promising results suggest that the proposed gaze estimation approach is a feasible and flexible scheme to facilitate gaze-based multimodal interfaces.

Quick Bootstrapping of a Personalized Gaze Model from Real-Use Interactions

Understanding human visual attention is essential for understanding human cognition, which in turn benefits human--computer interaction. Recent work has demonstrated a Personalized, Auto-Calibrating Eye-tracking (PACE) system, which makes it possible to achieve accurate gaze estimation using only an off-the-shelf webcam by identifying and collecting data implicitly from user interaction events. However, this method is constrained by the need for large amounts of well-annotated data. We thus present fast-PACE, an adaptation to PACE that exploits knowledge from existing data from different users to accelerate the learning speed of the personalized model. The result is an adaptive, data-driven approach that continuously “learns” its user and recalibrates, adapts, and improves with additional usage by a user. Experimental evaluations of fast-PACE demonstrate its competitive accuracy in iris localization, validity of alignment identification between gaze and interactions, and effectiveness of gaze transfer. In general, fast-PACE achieves an initial visual error of 3.98 degrees and then steadily improves to 2.52 degrees given incremental interaction-informed data. Our performance is comparable to state-of-the-art, but without the need for explicit training or calibration. Our technique addresses the data quality and quantity problems. It therefore has the potential to enable comprehensive gaze-aware applications in the wild.

HEYEbrid

We introduce hEYEbrid, a calibration-free method for spontaneous and long-term eye gaze tracking, with competitive gaze estimation. It is based on a hybrid concept that combines infrared eye images with corneal imaging. For this, two eye cameras are mounted on a glasses frame. In this way, the pupil can be tracked quickly with high precision. This information is translated into the corneal image, which is used to create a connection to the environment, acting like a scene camera. In a user study with 20 participants, we evaluated our approach against an extended version of the system, called 3C-hEYEbrid, and a state-of-the-art head-mounted Pupil Labs eye tracker. We show that hEYEbrid provides accurate gaze estimation in unconstrained environments and is robust against calibration drift (e.g. caused by taking off and putting on the device). In addition, we present a mobile and wearable implementation of hEYEbrid and 3C-hEYEbrid, that is also usable with a monocular Pupil Labs eye tracker. It connects the head-mounted device to a mobile phone, enabling gaze estimation in real time. hEYEbrid represents a significant step towards pervasive gaze-based interfaces.

InvisibleEye

Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5 × 5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.

TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets

We study gaze estimation on tablets; our key design goal is uncalibrated gaze estimation using the front-facing camera during natural use of tablets, where the posture and method of holding the tablet are not constrained. We collected a large unconstrained gaze dataset of tablet users, labeled Rice TabletGaze dataset. The dataset consists of 51 subjects, each with 4 different postures and 35 gaze locations. Subjects vary in race, gender and in their need for prescription glasses, all of which might impact gaze estimation accuracy. We made three major observations on the collected data and employed a baseline algorithm for analyzing the impact of several factors on gaze estimation accuracy. The baseline algorithm is based on multilevel HoG feature and Random Forests regressor, which achieves a mean error of 3.17 cm. We perform extensive evaluation on the impact of various practical factors such as person dependency, dataset size, race, wearing glasses and user posture on the gaze estimation accuracy.

2D Gaze Estimation Based on Pupil-Glint Vector Using an Artificial Neural Network

Gaze estimation methods play an important role in a gaze tracking system. A novel 2D gaze estimation method based on the pupil-glint vector is proposed in this paper. First, the circular ring rays location (CRRL) method and Gaussian fitting are utilized for pupil and glint detection, respectively. Then the pupil-glint vector is calculated through subtraction of pupil and glint center fitting. Second, a mapping function is established according to the corresponding relationship between pupil-glint vectors and actual gaze calibration points. In order to solve the mapping function, an improved artificial neural network (DLSR-ANN) based on direct least squares regression is proposed. When the mapping function is determined, gaze estimation can be actualized through calculating gaze point coordinates. Finally, error compensation is implemented to further enhance accuracy of gaze estimation. The proposed method can achieve a corresponding accuracy of 1.29°, 0.89°, 0.52°, and 0.39° when a model with four, six, nine, or 16 calibration markers is utilized for calibration, respectively. Considering error compensation, gaze estimation accuracy can reach 0.36°. The experimental results show that gaze estimation accuracy of the proposed method in this paper is better than that of linear regression (direct least squares regression) and nonlinear regression (generic artificial neural network). The proposed method contributes to enhancing the total accuracy of a gaze tracking system.