Gaze Estimation Method Research Articles

2D Gaze Estimation Based on Pupil-Glint Vector Using an Artificial Neural Network

Gaze estimation methods play an important role in a gaze tracking system. A novel 2D gaze estimation method based on the pupil-glint vector is proposed in this paper. First, the circular ring rays location (CRRL) method and Gaussian fitting are utilized for pupil and glint detection, respectively. Then the pupil-glint vector is calculated through subtraction of pupil and glint center fitting. Second, a mapping function is established according to the corresponding relationship between pupil-glint vectors and actual gaze calibration points. In order to solve the mapping function, an improved artificial neural network (DLSR-ANN) based on direct least squares regression is proposed. When the mapping function is determined, gaze estimation can be actualized through calculating gaze point coordinates. Finally, error compensation is implemented to further enhance accuracy of gaze estimation. The proposed method can achieve a corresponding accuracy of 1.29°, 0.89°, 0.52°, and 0.39° when a model with four, six, nine, or 16 calibration markers is utilized for calibration, respectively. Considering error compensation, gaze estimation accuracy can reach 0.36°. The experimental results show that gaze estimation accuracy of the proposed method in this paper is better than that of linear regression (direct least squares regression) and nonlinear regression (generic artificial neural network). The proposed method contributes to enhancing the total accuracy of a gaze tracking system.

Appearance-based gaze estimation under slight head motion

At present a lot of gaze estimation methods can get accurate result under ideal conditions, but some practical issues are still the biggest challenges affect the accuracy such as head motion and eye blinking. Improving the accuracy of gaze estimation and the tolerance of head motion are common tasks in the field of gaze estimation. Therefore, this paper aims to propose an accurate gaze estimation method without fixed head pose. The core problem is how to build the mapping relationship between image features and gaze position, and how to resist the head motion through the training samples. To this end, at first, a new input feature, which can well reflect the change of eye image features with different gaze positions, is proposed and it is based on appearance feature and distance feature. So the number of training samples in the process of calibration is significantly reduced. Then l1-optimization is used to select an optimal set, which represents the mapping relationship between input feature and gaze position. At last, a linear equation is fitted to correct the initial estimation bias which is brought by head motion. In this paper, the experimental results demonstrate that our system achieves accurate result with one camera and a small number of calibration points. The accuracy of final gaze estimation is improved by 22 % through compensation equation. In addition, our system is robustness to eye blink and distance change.

Head Pose‐Invariant Eyelid and Iris Tracking Method

SUMMARYThese days, there is more demand for a camera‐based gaze estimation method for new interfaces and new marketing measurement tools. Considering these applications, the system should track a new user without any operation such as calibration. It should also admit user's natural head pose changes. Previous methods, however, need a calibration procedure before execution and have less accuracy in a head moving situation. In this paper, we propose a method which tracks the user's eyelids and iris automatically and accurately. Our method is a pretreatment of gaze estimation without any calibration and head pose restraint. First of all, we track the facial feature points from an input face image and estimate its head pose, extracting the eye region image. On the eye region image, we track the eyelid shape based on an eyelid shape model generated beforehand from PCA. Finally we track the iris inside the eyelid based on the eyeball model. The eyelid and iris tracking is processed by Particle Filter. An evaluation against a database including head pose changes confirmed that the accuracy of eyelid and iris tracking was improved compared with previous methods.

Appearance-Based Gaze Estimation With Online Calibration From Mouse Operations

This paper presents an unconstrained gaze estimation method using an online learning algorithm. We focus on a desktop scenario, where a user operates a personal computer, and use the mouse-clicked positions to infer, where on the screen the user is looking at. Our method continuously captures the user's head pose and eye images with a monocular camera, and each mouse click triggers learning sample acquisition. In order to handle head pose variations, the samples are adaptively clustered according to the estimated head pose. Then, local reconstruction-based gaze estimation models are incrementally updated in each cluster. We conducted a prototype evaluation in real-world environments, and our method achieved an estimation accuracy of 2.9°.

Robust remote gaze estimation method based on multiple geometric transforms

The remote gaze estimation (RGE) technique has been widely used as a natural interface in consumer electronic devices for decades. Although outstanding outcomes on RGE have been recently reported in the literature, tracking gaze under large head movements is still an unsolved problem. General RGE methods estimate a user’s point of gaze (POG) using a mapping function representing the relationship between several infrared light sources and their corresponding corneal reflections (CRs) in the eye image. However, the minimum number of available CRs required for a valid POG estimation cannot be satisfied in those methods because the CRs often tend to be distorted or disappeared inevitably under the unconstrained eye and head movements. To overcome this problem, a multiple-transform-based method is proposed. In the proposed method, through three different geometric transform-based normalization processes, several nonlinear mapping functions are simultaneously obtained in the calibration process and then used to estimate the POG. The geometric transforms and mapping functions can be alternatively employed according to the number of available CRs even under large head movement. Experimental results on six subjects demonstrate the effectiveness of the proposed method.

Gaze Estimation From Eye Appearance: A Head Pose-Free Method via Eye Image Synthesis.

In this paper, we address the problem of free head motion in appearance-based gaze estimation. This problem remains challenging because head motion changes eye appearance significantly, and thus, training images captured for an original head pose cannot handle test images captured for other head poses. To overcome this difficulty, we propose a novel gaze estimation method that handles free head motion via eye image synthesis based on a single camera. Compared with conventional fixed head pose methods with original training images, our method only captures four additional eye images under four reference head poses, and then, precisely synthesizes new training images for other unseen head poses in estimation. To this end, we propose a single-directional (SD) flow model to efficiently handle eye image variations due to head motion. We show how to estimate SD flows for reference head poses first, and then use them to produce new SD flows for training image synthesis. Finally, with synthetic training images, joint optimization is applied that simultaneously solves an eye image alignment and a gaze estimation. Evaluation of the method was conducted through experiments to assess its performance and demonstrate its effectiveness.

A novel single IR light based gaze estimation method using virtual glints

Among various infrared (IR) image based remote eye gaze tracking (REGT) methods, 2-D interpolation based methods, e.g., the cross ratio-based and homography normalization (HN)-based methods, have been widely researched owing to their highly accurate performance and simplicity with respect to hardware setup. These methods, however, can be employed only when user utilizes more than four IR light sources to obtain multiple corneal reflections projected on the image plane, i.e. glints. In this paper, a novel 2-D interpolation-based REGT method with a single IR light source is proposed, which attains both accuracy and headpose robustness. In the proposed method, the virtual glint (VG), which can substitute for the actual glint, is first estimated by utilizing the mathematical and geometrical principles established between the 3-D location of an IR light source and the corresponding glint in the image plane, and then the point of gaze (POG) is calculated by employing the HN method using the estimated VGs. The experimental results indicate that the proposed REGT method is highly competitive with conventional ones requiring multiple IR light sources, in terms of accuracy and robustness against head movements.

A new gaze estimation method considering external light.

Gaze tracking systems usually utilize near-infrared (NIR) lights and NIR cameras, and the performance of such systems is mainly affected by external light sources that include NIR components. This is ascribed to the production of additional (imposter) corneal specular reflection (SR) caused by the external light, which makes it difficult to discriminate between the correct SR as caused by the NIR illuminator of the gaze tracking system and the imposter SR. To overcome this problem, a new method is proposed for determining the correct SR in the presence of external light based on the relationship between the corneal SR and the pupil movable area with the relative position of the pupil and the corneal SR. The experimental results showed that the proposed method makes the gaze tracking system robust to the existence of external light.

Development of a Method for Gaze Estimation Based on Planar Approximations of Voltage Ratios Calculated from Multiple Electro-oculogram Signals

Development of a Method for Gaze Estimation Based on Planar Approximations of Voltage Ratios Calculated from Multiple Electro-oculogram Signals

Real-Time Gaze Estimation with Online Calibration

Gaze-tracking technology is highly valuable in many interactive and diagnostic applications. For many gaze estimation systems, calibration is an unavoidable procedure necessary to determine certain person-specific parameters, either explicitly or implicitly. Recently, several offline implicit calibration methods have been proposed to ease the calibration burden. However, the calibration procedure is still cumbersome, and gaze estimation accuracy needs further improvement. In this article, the authors present a novel 3D gaze estimation system with online calibration. The proposed system is based on a new 3D model-based gaze estimation method using a single consumer depth camera sensor (via Kinect). Unlike previous gaze estimation methods using explicit offline calibration with fixed number of calibration points or implicit calibration, their approach constantly improves person-specific eye parameters through online calibration, which enables the system to adapt gradually to a new user. The experimental results and the human-computer interaction (HCI) application show that the proposed system can work in real time with superior gaze estimation accuracy and minimal calibration burden.

Evaluation of accurate eye corner detection methods for gaze estimation

Accurate detection of iris center and eye corners appears to be a promising approach for low cost gaze estimation. In this paper we propose novel eye inner corner detection methods. Appearance and feature based segmentation approaches are suggested. All these methods are exhaustively tested on a realistic dataset containing images of subjects gazing at different points on a screen. We have demonstrated that a method based on a neural network presents the best performance even in light changing scenarios. In addition to this method, algorithms based on AAM and Harris corner detector present better accuracies than recent high performance face points tracking methods such as Intraface.

Head-pose Invariant Eyelid and Iris Tracking Method

These days, there is more demand for a camera-based gaze estimation method for new interfaces and new marketing measurement tools. Considering these applications, the system should track a new user without any operation such as calibration. It should also admit user's natural head pose changes. Previous methods, however, need a calibration procedure before execution and have less accuracy in a head moving situation. In this paper, we propose a method which tracks the user's eyelids and iris automatically and accurately. Our method is a pretreatment of gaze estimation without any calibration and head pose restraint. First of all, we track the facial feature points from an input face image and estimate its head pose, extracting the eye region image. On the eye region image, we track the eyelid shape based on an eyelid shape model generated beforehand from PCA. Finally we track the iris inside the eyelid based on the eyeball model. The eyelid and iris tracking is processed by Particle Filter. An evaluation against a database including head pose changes confirmed that the accuracy of eyelid and iris tracking was improved compared with previous methods.

Long-Range Gaze Tracking System for Large Movements

In the vision-based remote gaze tracking systems, the most challenging topics are to allow natural movement of a user and to increase the working volume and distance of the system. Several eye gaze estimation methods considering the natural movement of a user have been proposed. However, their working volume and distance are narrow and close. In this paper, we propose a novel 2-D mapping-based gaze estimation method that allows large-movement of user. Conventional 2-D mapping-based methods utilize mapping function between calibration points on the screen and pupil center corneal reflection (PCCR) vectors obtained in user calibration step. However, PCCR vectors and their associated mapping function are only valid at or near to the position where the user calibration is performed. The proposed movement mapping function, complementing the user's movement, estimates scale factors between two PCCR vector sets: one obtained at the user calibration position and another obtained at the new user position. The proposed system targets a longer range gaze tracking which operates from 1.4 to 3 m. A narrow-view camera mounted on a pan and tilt unit is used by the proposed system to capture high-resolution eye image, providing a wide and long working volume of about 100 cm × 40 cm × 100 cm. The experimental results show that the proposed method successfully compensated the poor performance due to user's large movement. Average angular error was 0.8° and only 0.07° of angular error was increased while the user moved around 81 cm.

Gaze Detection by Wearable Eye-Tracking and NIR LED-Based Head-Tracking Device Based on SVR

In this paper, a gaze estimation method is proposed for use with a large-sized display at a distance. Our research has the following four novelties: this is the first study on gaze-tracking for large-sized displays and large Z (viewing) distances; our gaze-tracking accuracy is not affected by head movements since the proposed method tracks the head by using a near infrared camera and an infrared light-emitting diode; the threshold for local binarization of the pupil area is adaptively determined by using a p-tile method based on circular edge detection irrespective of the eyelid or eyelash shadows; and accurate gaze position is calculated by using two support vector regressions without complicated calibrations for the camera, display, and user's eyes, in which the gaze positions and head movements are used as feature values. The root mean square error of gaze detection is calculated as 0.79° for a 30-inch screen.

Study of Polynomial Mapping Functions in Video-Oculography Eye Trackers

Gaze-tracking data have been used successfully in the design of new input devices and as an observational technique in usability studies. Polynomial-based Video-Oculography (VOG) systems are one of the most attractive gaze estimation methods thanks to their simplicity and ease of implementation. Although the functionality of these systems is generally acceptable, there has been no thorough comparative study to date of how the mapping equations affect the final system response. After developing a taxonomic classification of calibration functions, we examined over 400,000 models and evaluated the validity of several conventional assumptions. Our rigorous experimental procedure enabled us to optimize the calibration process for a real VOG gaze-tracking system and halve the calibration time while avoiding a detrimental effect on the accuracy or tolerance to head movement. Finally, a geometry-based method is implemented and tested. The results and performance is compared with those obtained by the general purpose expressions.

Gaze Estimation Method Involving Corneal Reflection-Based Modeling of the Eye as a General Surface of Revolution about the Optical Axis of the Eye

Gaze Estimation Method Involving Corneal Reflection-Based Modeling of the Eye as a General Surface of Revolution about the Optical Axis of the Eye

A Study On Face, Eye Detection And Gaze Estimation

Face and eye detection is one of the most challenging problems in computer vision area. The goal of this paper is to present a study over the existing literature on face and eye detection and gaze estimation. With the uptrend of systems based on face and eye detection in many different areas of life in recent years, this subject has gained much more attention by academic and industrial area. Many different studies have been performed about face and eye detection. Besides having many challenging problems like, having different lighting conditions, having glasses, facial hair or mustache on face, different orientation pose or occlusion of face, face and eye detection methods performed great progress. In this paper we first categorize face detection models and examine the basic algorithms for face detection. Then we present methods for eye detection and gaze estimation.

Guest Editors' Introduction: Special Section on the ACM Symposium on Virtual Reality and Software Technology (VRST 2009)

THIS special section is dedicated to the three Best Papers of VRST 2009, which was held in Kyoto, Japan from 18 to 20 November 2009. The papers were first selected by a jury based on the referees’ reports and the oral presentation. The authors were then asked to provide an extended version. The extended papers were then reviewed through a strict peer review process. The first paper, “Robust Relocalization and Its Evaluation for Online Environment Map Construction” by Sehwan Kim, Christopher Coffin, and Tobias Hollerer from the University of California in Santa Barbara, presents robust methodology and evaluation for camera orientation relocalization, using virtual keyframes for online environment map construction. In case of tracking loss, incoming camera frames are matched against known-orientation keyframes to reestimate camera orientation. Instead of solely using real keyframes from incoming video, the proposed approach employs virtual keyframes which are distributed strategically within completed portions of an environment map. The authors compare different system variants using three evaluation methods to show that the proposed system is useful in a practical sense. To improve recovery robustness against lighting changes in indoor and outdoor environments, they propose a new approach based on illumination normalization and saturated area removal. In the second paper, “Natural Perspective Projections for Head-Mounted Displays,” Frank Steinicke, Gerd Bruder, Scott Kuhl, Pete Willemsen, Markus Lappe, and Klaus H. Hinrichs from Munster University in Germany analyze the user’s perception of a virtual environment displayed in a Head Mounted Displayed (HMD), which is rendered with different geometric fields of views. They introduce a psychophysical calibration method to determine the HMD’s actual field of view, which may vary from the nominal values specified by the manufacturer. Furthermore, we conducted two experiments to identify perspective projections for HMDs, which are identified as natural by subjects—even if these perspectives deviate from the perspectives that are inherently defined by the display field of view. In the third paper, “A Wide-View Parallax-Free EyeMark Recorder with a Hyperboloidal Half-Silvered Mirror and Appearance-Based Gaze Estimation,” Hiroki Mori, Erika Sumiya, Tomohiro Mashita, Kiyoshi Kiyokawa, and Haruo Takemura from Osaka University propose a wideview parallax-free eye-mark recorder with a hyperboloidal half-silvered mirror and a gaze estimation method suitable for the device. Their eye-mark recorder provides a wide field-of-view video recording of the user’s exact view by positioning the focal point of the mirror at the user’s viewpoint. They implemented and evaluated a gaze estimation method for their eye-mark recorder. They apply principal component analysis and multiple regression analysis to determine the relationship between the captured images and their corresponding gaze points.

Gaze estimation in a gaze tracking system

This article presents a gaze estimation method (GEMHSSO) based on the pupil center cornea reflection (PCCR) technique. Existing PCCR systems suffer from several problems, including the restriction of users’ head movement, and the requirement of individual calibration. This paper presents a head position compensation model using a single camera and a single light source, which realizes the analytic compensation of head motion effects on pupil-glint vectors. We then present a transformation model for individual differences to simplify the calibration process to a one-point calibration. On this basis, we establish a novel gaze estimation method that reduces the minimum hardware requirements for accurate estimation to a single camera (not calibrated) and a single light source. Without the need for a complex system, our method has the ability to estimate gaze during natural head movement, and to substantially simplify user calibration. Each step of the proposed method in this paper makes real-time implementation possible, which provides an effective solution for eye-gaze tracking in human-computer interaction systems.

Real-Time Gaze Estimator Based on Driver's Head Orientation for Forward Collision Warning System

This paper presents a vision-based real-time gaze zone estimator based on a driver's head orientation composed of yaw and pitch. Generally, vision-based methods are vulnerable to the wearing of eyeglasses and image variations between day and night. The proposed method is novel in the following four ways: First, the proposed method can work under both day and night conditions and is robust to facial image variation caused by eyeglasses because it only requires simple facial features and not specific features such as eyes, lip corners, and facial contours. Second, an ellipsoidal face model is proposed instead of a cylindrical face model to exactly determine a driver's yaw. Third, we propose new features—the normalized mean and the standard deviation of the horizontal edge projection histogram—to reliably and rapidly estimate a driver's pitch. Fourth, the proposed method obtains an accurate gaze zone by using a support vector machine. Experimental results from 200 000 images showed that the root mean square errors of the estimated yaw and pitch angles are below 7 under both daylight and nighttime conditions. Equivalent results were obtained for drivers with glasses or sunglasses, and 18 gaze zones were accurately estimated using the proposed gaze estimation method.