Appearance-based Gaze Estimation Research Articles

Gaze Estimation in the 3D Space Using RGB-D Sensors

We address the problem of 3D gaze estimation within a 3D environment from remote sensors, which is highly valuable for applications in human---human and human---robot interactions. To the contrary of most previous works, which are limited to screen gazing applications, we propose to leverage the depth data of RGB-D cameras to perform an accurate head pose tracking, acquire head pose invariance through a 3D rectification process that renders head pose dependent eye images into a canonical viewpoint, and computes the line-of-sight in the 3D space. To address the low resolution issue of the eye image resulting from the use of remote sensors, we rely on the appearance based gaze estimation paradigm, which has demonstrated robustness against this factor. In this context, we do a comparative study of recent appearance based strategies within our framework, study the generalization of these methods to unseen individual, and propose a cross-user eye image alignment technique relying on the direct registration of gaze-synchronized eye images. We demonstrate the validity of our approach through extensive gaze estimation experiments on a public dataset as well as a gaze coding task applied to natural job interviews.

Gaze Estimation From Eye Appearance: A Head Pose-Free Method via Eye Image Synthesis.

In this paper, we address the problem of free head motion in appearance-based gaze estimation. This problem remains challenging because head motion changes eye appearance significantly, and thus, training images captured for an original head pose cannot handle test images captured for other head poses. To overcome this difficulty, we propose a novel gaze estimation method that handles free head motion via eye image synthesis based on a single camera. Compared with conventional fixed head pose methods with original training images, our method only captures four additional eye images under four reference head poses, and then, precisely synthesizes new training images for other unseen head poses in estimation. To this end, we propose a single-directional (SD) flow model to efficiently handle eye image variations due to head motion. We show how to estimate SD flows for reference head poses first, and then use them to produce new SD flows for training image synthesis. Finally, with synthetic training images, joint optimization is applied that simultaneously solves an eye image alignment and a gaze estimation. Evaluation of the method was conducted through experiments to assess its performance and demonstrate its effectiveness.

Adaptive Linear Regression for Appearance-Based Gaze Estimation.

We investigate the appearance-based gaze estimation problem, with respect to its essential difficulty in reducing the number of required training samples, and other practical issues such as slight head motion, image resolution variation, and eye blinking. We cast the problem as mapping high-dimensional eye image features to low-dimensional gaze positions, and propose an adaptive linear regression (ALR) method as the key to our solution. The ALR method adaptively selects an optimal set of sparsest training samples for the gaze estimation via ℓ(1)-optimization. In this sense, the number of required training samples is significantly reduced for high accuracy estimation. In addition, by adopting the basic ALR objective function, we integrate the gaze estimation, subpixel alignment and blink detection into a unified optimization framework. By solving these problems simultaneously, we successfully handle slight head motion, image resolution variation and eye blinking in appearance-based gaze estimation. We evaluated the proposed method by conducting experiments with multiple users and variant conditions to verify its effectiveness.

Guest Editors' Introduction: Special Section on the ACM Symposium on Virtual Reality and Software Technology (VRST 2009)

THIS special section is dedicated to the three Best Papers of VRST 2009, which was held in Kyoto, Japan from 18 to 20 November 2009. The papers were first selected by a jury based on the referees’ reports and the oral presentation. The authors were then asked to provide an extended version. The extended papers were then reviewed through a strict peer review process. The first paper, “Robust Relocalization and Its Evaluation for Online Environment Map Construction” by Sehwan Kim, Christopher Coffin, and Tobias Hollerer from the University of California in Santa Barbara, presents robust methodology and evaluation for camera orientation relocalization, using virtual keyframes for online environment map construction. In case of tracking loss, incoming camera frames are matched against known-orientation keyframes to reestimate camera orientation. Instead of solely using real keyframes from incoming video, the proposed approach employs virtual keyframes which are distributed strategically within completed portions of an environment map. The authors compare different system variants using three evaluation methods to show that the proposed system is useful in a practical sense. To improve recovery robustness against lighting changes in indoor and outdoor environments, they propose a new approach based on illumination normalization and saturated area removal. In the second paper, “Natural Perspective Projections for Head-Mounted Displays,” Frank Steinicke, Gerd Bruder, Scott Kuhl, Pete Willemsen, Markus Lappe, and Klaus H. Hinrichs from Munster University in Germany analyze the user’s perception of a virtual environment displayed in a Head Mounted Displayed (HMD), which is rendered with different geometric fields of views. They introduce a psychophysical calibration method to determine the HMD’s actual field of view, which may vary from the nominal values specified by the manufacturer. Furthermore, we conducted two experiments to identify perspective projections for HMDs, which are identified as natural by subjects—even if these perspectives deviate from the perspectives that are inherently defined by the display field of view. In the third paper, “A Wide-View Parallax-Free EyeMark Recorder with a Hyperboloidal Half-Silvered Mirror and Appearance-Based Gaze Estimation,” Hiroki Mori, Erika Sumiya, Tomohiro Mashita, Kiyoshi Kiyokawa, and Haruo Takemura from Osaka University propose a wideview parallax-free eye-mark recorder with a hyperboloidal half-silvered mirror and a gaze estimation method suitable for the device. Their eye-mark recorder provides a wide field-of-view video recording of the user’s exact view by positioning the focal point of the mirror at the user’s viewpoint. They implemented and evaluated a gaze estimation method for their eye-mark recorder. They apply principal component analysis and multiple regression analysis to determine the relationship between the captured images and their corresponding gaze points.