Gaze Estimation Research Articles

Approach to Quantify Eye Movements to Augment Stroke Diagnosis With a Non-Calibrated Eye-Tracker.

We investigated 1) the need for calibration to measure eye movement symmetry in healthy controls and 2) the potential of eye movement symmetry to distinguish between healthy controls and patients. We analyzed fixations, smooth pursuits, saccades, and conjugacy measured by a Spearman correlation coefficient and utilized a linear mixed-effects model to estimate the effect of calibration. Healthy participants (n = 18) did not differ in correlations between calibrated and non-calibrated conditions for all tests. The calibration condition did not improve the linear mixed effects model (log-likelihood ratio test p = 0.426) in predicting correlation coefficients. Interestingly, the patient group (n = 17) differed in correlations for the DOT (0.844 [95% CI 0.602, 0.920] vs. 0.98 [95% CI 0.976, 0.985]), H (0.903 [95% CI 0.746, 0.958] vs. 0.979 [95% CI 0.971, 0.986]), and OKN (0.898 [95% CI 0.785, 0.958] vs. 0.993 [95% CI 0.987, 0.996]) tests compared to healthy controls along the x-axis. These differences were not observed along the y-axis. This study suggests that automated eye tracking can be deployed without calibration to measure eye movement symmetry. It may be a good discriminator between normal and abnormal eye movement symmetry. Validation of these findings in larger populations is required.

Is your mouse attracted by your eyes: Non-intrusive stress detection in off-the-shelf desktop environments

Increasing number of people work long hours with computers under high cognitive load. This could potentially cause mental stress in workplaces. Prolonged exposure to mental stress contributes to poor working experience and even severe health problems. Despite the growing demand, the existing intelligent stress detection methods are limited when applied to actual workplaces. They often measure physiological and physical signals, via intrusive devices, to detect stress. The intrusiveness hampers their accessibility and applicability in daily life and workplaces. To overcome that, behavior-based methods were proposed. Models that explore mouse and gaze behaviors during computer usages were demonstrated to be particularly effective. However, the current methods rely on using prior knowledge of the user interface (UI) layout to construct models. Their applicability thus is limited, especially in real workplaces where task UI is often dynamic. This paper presents a novel stress detection method to address the challenges. It attains non-intrusiveness and UI-agnostic by modeling the relative movement and coordination of mouse and gaze. The method is evaluated on a dynamic-UI task, namely, web searching. An accuracy of 78.8% is achieved using a commercial eye-tracker for gaze estimation, beating the state-of-the-art approaches by around 20%. We further use webcam to estimate gaze locations substituting for the eye-tracker, to enhance the model accessibility. The method yields 68.6% accuracy of stress detection without using any special devices. Experimental results demonstrate the effectiveness and applicability of our method. It opens up a new avenue for cognitive-aware adaptive user interface, intelligent working environment, and related applications.

Object Grasp Control of a 3D Robot Arm by Combining EOG Gaze Estimation and Camera-Based Object Recognition.

The purpose of this paper is to quickly and stably achieve grasping objects with a 3D robot arm controlled by electrooculography (EOG) signals. A EOG signal is a biological signal generated when the eyeballs move, leading to gaze estimation. In conventional research, gaze estimation has been used to control a 3D robot arm for welfare purposes. However, it is known that the EOG signal loses some of the eye movement information when it travels through the skin, resulting in errors in EOG gaze estimation. Thus, EOG gaze estimation is difficult to point out the object accurately, and the object may not be appropriately grasped. Therefore, developing a methodology to compensate, for the lost information and increase spatial accuracy is important. This paper aims to realize highly accurate object grasping with a robot arm by combining EMG gaze estimation and the object recognition of camera image processing. The system consists of a robot arm, top and side cameras, a display showing the camera images, and an EOG measurement analyzer. The user manipulates the robot arm through the camera images, which can be switched, and the EOG gaze estimation can specify the object. In the beginning, the user gazes at the screen's center position and then moves their eyes to gaze at the object to be grasped. After that, the proposed system recognizes the object in the camera image via image processing and grasps it using the object centroid. The object selection is based on the object centroid closest to the estimated gaze position within a certain distance (threshold), thus enabling highly accurate object grasping. The observed size of the object on the screen can differ depending on the camera installation and the screen display state. Therefore, it is crucial to set the distance threshold from the object centroid for object selection. The first experiment is conducted to clarify the distance error of the EOG gaze estimation in the proposed system configuration. As a result, it is confirmed that the range of the distance error is 1.8-3.0 cm. The second experiment is conducted to evaluate the performance of the object grasping by setting two thresholds from the first experimental results: the medium distance error value of 2 cm and the maximum distance error value of 3 cm. As a result, it is found that the grasping speed of the 3 cm threshold is 27% faster than that of the 2 cm threshold due to more stable object selection.

DynamicRead: Exploring Robust Gaze Interaction Methods for Reading on Handheld Mobile Devices under Dynamic Conditions

Enabling gaze interaction in real-time on handheld mobile devices has attracted significant attention in recent years. An increasing number of research projects have focused on sophisticated appearance-based deep learning models to enhance the precision of gaze estimation on smartphones. This inspires important research questions, including how the gaze can be used in a real-time application, and what type of gaze interaction methods are preferable under dynamic conditions in terms of both user acceptance and delivering reliable performance. To address these questions, we design four types of gaze scrolling techniques: three explicit technique based on Gaze Gesture, Dwell time, and Pursuit; and one implicit technique based on reading speed to support touch-free, page-scrolling on a reading application. We conduct a 20-participant user study under both sitting and walking settings and our results reveal that Gaze Gesture and Dwell time-based interfaces are more robust while walking and Gaze Gesture has achieved consistently good scores on usability while not causing high cognitive workload.

Gaze Estimation via Strip Pooling and Multi-Criss-Cross Attention Networks

Deep learning techniques for gaze estimation usually determine gaze direction directly from images of the face. These algorithms achieve good performance because face images contain more feature information than eye images. However, these image classes contain a substantial amount of redundant information that may interfere with gaze prediction and may represent a bottleneck for performance improvement. To address these issues, we model long-distance dependencies between the eyes via Strip Pooling and Multi-Criss-Cross Attention Networks (SPMCCA-Net), which consist of two newly designed network modules. One module is represented by a feature enhancement bottleneck block based on fringe pooling. By incorporating strip pooling, this residual module not only enlarges its receptive fields to capture long-distance dependence between the eyes but also increases weights on important features and reduces the interference of redundant information unrelated to gaze. The other module is a multi-criss-cross attention network. This module exploits a cross-attention mechanism to further enhance long-range dependence between the eyes by incorporating the distribution of eye-gaze features and providing more gaze cues for improving estimation accuracy. Network training relies on the multi-loss function, combined with smooth L1 loss and cross entropy loss. This approach speeds up training convergence while increasing gaze estimation precision. Extensive experiments demonstrate that SPMCCA-Net outperforms several state-of-the-art methods, achieving mean angular error values of 10.13° on the Gaze360 dataset and 6.61° on the RT-gene dataset.

EM-Gaze: eye context correlation and metric learning for gaze estimation

In recent years, deep learning techniques have been used to estimate gaze—a significant task in computer vision and human-computer interaction. Previous studies have made significant achievements in predicting 2D or 3D gazes from monocular face images. This study presents a deep neural network for 2D gaze estimation on mobile devices. It achieves state-of-the-art 2D gaze point regression error, while significantly improving gaze classification error on quadrant divisions of the display. To this end, an efficient attention-based module that correlates and fuses the left and right eye contextual features is first proposed to improve gaze point regression performance. Subsequently, through a unified perspective for gaze estimation, metric learning for gaze classification on quadrant divisions is incorporated as additional supervision. Consequently, both gaze point regression and quadrant classification performances are improved. The experiments demonstrate that the proposed method outperforms existing gaze-estimation methods on the GazeCapture and MPIIFaceGaze datasets.

LiteGaze: Neural architecture search for efficient gaze estimation.

Gaze estimation plays a critical role in human-centered vision applications such as human-computer interaction and virtual reality. Although significant progress has been made in automatic gaze estimation by deep convolutional neural networks, it is still difficult to directly deploy deep learning based gaze estimation models across different edge devices, due to the high computational cost and various resource constraints. This work proposes LiteGaze, a deep learning framework to learn architectures for efficient gaze estimation via neural architecture search (NAS). Inspired by the once-for-all model (Cai et al., 2020), this work decouples the model training and architecture search into two different stages. In particular, a supernet is trained to support diverse architectural settings. Then specialized sub-networks are selected from the obtained supernet, given different efficiency constraints. Extensive experiments are performed on two gaze estimation datasets and demonstrate the superiority of the proposed method over previous works, advancing the real-time gaze estimation on edge devices.

Monocular 3D gaze estimation using feature discretization and attention mechanism

Monocular 3D gaze estimation using feature discretization and attention mechanism

Smart Exam Proctoring System

Abstract: As the world is shifting towards digitalization, mostof the exams and assessments are being conducted online. These exams must be proctored. Several students are accessing thetest at the same time. It is very difficult to manually look if a student is committing malpractice. This project aims to use face detection and recognition for proctoring exams. Face detectionis the process of detecting faces in a video or image while face recognition is identifying or verifying a face from images orvideos. There are several research studies done on the detectionand recognition of faces owing to the requirement for securityfor economic transactions, authorization, national safety andsecurity, and other important factors. Exam proctoring platformsshould be capable of detecting cheating and malpractices like face is not on the screen, gaze estimation, mobile phone detection,multiple face detection, etc. This project uses face identificationusing HAAR Cascades Algorithm and face recognition using theLocal Binary Pattern Histogram algorithm. This system can beused in the future in corporate offices, schools, and universities.

A field test of computer-vision-based gaze estimation in psychology

Computer-vision-based gaze estimation refers to techniques that estimate gaze direction directly from video recordings of the eyes or face without the need for an eye tracker. Although many such methods exist, their validation is often found in the technical literature (e.g., computer science conference papers). We aimed to (1) identify which computer-vision-based gaze estimation methods are usable by the average researcher in fields such as psychology or education, and (2) evaluate these methods. We searched for methods that do not require calibration and have clear documentation. Two toolkits, OpenFace and OpenGaze, were found to fulfill these criteria. First, we present an experiment where adult participants fixated on nine stimulus points on a computer screen. We filmed their face with a camera and processed the recorded videos with OpenFace and OpenGaze. We conclude that OpenGaze is accurate and precise enough to be used in screen-based experiments with stimuli separated by at least 11 degrees of gaze angle. OpenFace was not sufficiently accurate for such situations but can potentially be used in sparser environments. We then examined whether OpenFace could be used with horizontally separated stimuli in a sparse environment with infant participants. We compared dwell measures based on OpenFace estimates to the same measures based on manual coding. We conclude that OpenFace gaze estimates may potentially be used with measures such as relative total dwell time to sparse, horizontally separated areas of interest, but should not be used to draw conclusions about measures such as dwell duration.

Appearance-based gaze estimation with feature fusion of multi-level information elements

Abstract Gaze estimation is a fundamental task in many applications of cognitive sciences, human–computer interaction, and robotics. The purely data-driven appearance-based gaze estimation methods may suffer from a lack of interpretability, which prevents their applicability to pervasive scenarios. In this study, a feature fusion method with multi-level information elements is proposed to improve the comprehensive performance of the appearance-based gaze estimation model. The multi-level feature extraction and expression are carried out from the originally captured images, and a multi-level information element matrix is established. A gaze conduction principle is formulated for reasonably fusing information elements from the established matrix. According to the gaze conduction principle along with the matrix, a multi-level information element fusion (MIEF) model for gaze estimation is proposed. Then, several input modes and network structures of the MIEF model are designed, and a series of grouping experiments are carried out on a small-scale sub-dataset. Furthermore, the optimized input modes and network structures of the MIEF model are selected for training and testing on the whole dataset to verify and compare model performance. Experimental results show that optimizing the feature combination in the input control module and fine-tuning the computational architecture in the feature extraction module can improve the performance of the gaze estimation model, which would enable the reduction of the model by incorporating the critical features and thus improve the performance and accessibility of the method. Compared with the reference baseline, the optimized model based on the proposed feature fusion method of multi-level information elements can achieve efficient training and improve the test accuracy in the verification experiment. The average error is 1.63 cm on phones on the GazeCapture dataset, which achieves comparable accuracy with state-of-the-art methods.

Contactless Real-Time Eye Gaze-Mapping System Based on Simple Siamese Networks

Human–computer interaction (HCI) is a multidisciplinary field that investigates the interactions between humans and computer systems. HCI has facilitated the development of various digital technologies that aim to deliver optimal user experiences. Gaze recognition is a critical aspect of HCI, as it can provide valuable insights into basic human behavior. The gaze-matching method is a reliable approach that can identify the area at which a user is looking. Early methods of gaze tracking required users to wear glasses with a tracking function and limited tracking to a small monitoring area. Additionally, gaze estimation was restricted to a fixed posture within a narrow range. In this study, we proposed a novel non-contact gaze-mapping system that could overcome the physical limitations of previous methods and be applied in real-world environments. Our experimental results demonstrated an average gaze-mapping accuracy of 92.9% across 9 different test environments. Moreover, we introduced the GIST gaze-mapping (GGM) dataset, which served as a valuable resource for learning and evaluating gaze-mapping techniques.

Person-Specific Gaze Estimation from Low-Quality Webcam Images.

Gaze estimation is an established research problem in computer vision. It has various applications in real life, from human-computer interactions to health care and virtual reality, making it more viable for the research community. Due to the significant success of deep learning techniques in other computer vision tasks-for example, image classification, object detection, object segmentation, and object tracking-deep learning-based gaze estimation has also received more attention in recent years. This paper uses a convolutional neural network (CNN) for person-specific gaze estimation. The person-specific gaze estimation utilizes a single model trained for one individual user, contrary to the commonly-used generalized models trained on multiple people's data. We utilized only low-quality images directly collected from a standard desktop webcam, so our method can be applied to any computer system equipped with such a camera without additional hardware requirements. First, we used the web camera to collect a dataset of face and eye images. Then, we tested different combinations of CNN parameters, including the learning and dropout rates. Our findings show that building a person-specific eye-tracking model produces better results with a selection of good hyperparameters when compared to universal models that are trained on multiple users' data. In particular, we achieved the best results for the left eye with 38.20 MAE (Mean Absolute Error) in pixels, the right eye with 36.01 MAE, both eyes combined with 51.18 MAE, and the whole face with 30.09 MAE, which is equivalent to approximately 1.45 degrees for the left eye, 1.37 degrees for the right eye, 1.98 degrees for both eyes combined, and 1.14 degrees for full-face images.

BoT2L-Net: Appearance-Based Gaze Estimation Using Bottleneck Transformer Block and Two Identical Losses in Unconstrained Environments

As a nonverbal cue, gaze plays a critical role in communication, expressing emotions and reflecting mental activity. It has widespread applications in various fields. Recently, the appearance-based gaze estimation method, which utilizes CNN (convolutional neural networks), has rapidly improved the accuracy and robustness of gaze estimation algorithms. Due to their insufficient ability to capture global relationships, the present accuracy of gaze estimation methods in unconstrained environments, has the potential for improvement. To address this challenge, the focus of this paper is to enhance the accuracy of gaze estimation, which is typically measured by mean angular error. In light of Transformer’s breakthrough in image classification and target detection tasks, and the need for an efficient network, the Transformer-enhanced-CNN method is a suitable choice. This paper proposed a novel model for 3D gaze estimation in unconstrained environments, based on the Bottleneck Transformer block and multi-loss methods. Our designed network (BoT2L-Net), incorporates self-attention through the BoT block, utilizing two identical loss functions to predict the two gaze angles. Additionally, the back-propagation network was combined with classification and regression losses, to improve the network’s accuracy and robustness. Our model was evaluated on two commonly used gaze datasets: Gaze360 and MPIIGaze, achieving mean angular errors of 11.53° and 9.59° for front 180° and front-facing gaze angles, respectively, on the Gaze360 testing set, and a mean angular error of 3.97° on the MPIIGaze testing set, outperforming the CNN-based gaze estimation method. The BoT2L-Net model proposed in this paper performs well on two publicly available datasets, demonstrating the effectiveness of our approach.

Static Laser Feedback Interferometry-Based Gaze Estimation for Wearable Glasses

Fast and robust gaze estimation is a key technology for wearable glasses, as it enables novel methods of user interaction as well as display enhancement applications, such as foveated rendering. State-of-the-art video-based systems lack a high update rate, integrateability, slippage robustness, and low power consumption. To overcome these limitations, we propose a model-based fusion algorithm to estimate gaze from multiple static laser feedback interferometry (LFI) sensors, which are capable of measuring distance toward the eye and the eye’s rotational velocity. The proposed system is ambient light robust and robust to glasses slippage. During evaluation, a gaze accuracy of 1.79° at an outstanding update rate of 1 kHz is achieved, while the sensors consume only a fraction of the power compared with the state-of-the-art video-based system.

A Method of Disentanglement of Latent Factor using Geometric Feature for Gaze Estimation Network Training

Since each human eye has different anatomical features, gaze estimation is a very challenging task. Although numerous studies regarding gaze estimation were proposed, there is a need for improving the preciseness in order to facilitate the application of the method to real-world scenarios. To accomplish this goal, I propose a novel training strategy for gaze representation learning. The proposed training method includes two training phases: the autoencoder-based representation learning phase and the gaze estimation network training phase. The proposed training strategy enforces the trained model to disentangle the gaze-related latent code and produce a more accurate gaze estimation. In addition, I also propose and showcase a real-world application that exploits the proposed method in order to prove the practicality of the proposed method. Through the experiment, it is proven that the proposed method shows an outstanding performance compared to other methods on the Gaze360 dataset.

PerimetryNet: A multiscale fine grained deep network for three‐dimensional eye gaze estimation using visual field analysis

AbstractThree‐dimensional gaze estimation aims to reveal where a person is looking, which plays an important role in identifying users' point‐of‐interest in terms of the direction, attention and interactions. Appearance‐based gaze estimation methods could provide relatively unconstrained gaze tracking from commodity hardware. Inspired by medical perimetry test, we have proposed a multiscale framework with visual field analysis branch to improve estimation accuracy. The model is based on the feature pyramids and predicts vision field to help gaze estimation. In particular, we analysis the effect of the multiscale component and the visual field branch on challenging benchmark datasets: MPIIGaze and EYEDIAP. Based on these studies, our proposed PerimetryNet significantly outperforms state‐of‐the‐art methods. In addition, the multiscale mechanism and visual field branch can be easily applied to existing network architecture for gaze estimation. Related code would be available at public repository https://github.com/gazeEs/PerimetryNet.

Unconstrained human gaze estimation approach for medium-distance scene based on monocular vision

Unconstrained human gaze estimation approach for medium-distance scene based on monocular vision

Glints Detection in Noisy Images Using Graph Matching

Glints detection is important in gaze estimation with virtual reality equipment. In some complex scenarios, the glints detection may fail due to multiple reasons, such as the ambiguous correspondence between the multiple light sources or their reflected glints in the captured image, as well as the distinction between the good glints and the noisy glints caused by the lights reflect on the glasses that the users may wear. In this article, the glints detection is formulated as a graph matching problem between the two graphs constructed from the candidate glints and the rough predicted ones from the information given by the last frame. In addition, the optimal parameters for the matching are estimated using an Expectation Maximum algorithm-like algorithm. Finally, with the optimal parameters, the Kuhn–Munkras algorithm is used to get the optimal matching, and the successfully matched true glints are considered the correctly detected glints. Experiments are performed to validate the effectiveness of the proposed method.

Appearance-Based Gaze Estimation Method Using Static Transformer Temporal Differential Network

Gaze behavior is important and non-invasive human–computer interaction information that plays an important role in many fields—including skills transfer, psychology, and human–computer interaction. Recently, improving the performance of appearance-based gaze estimation, using deep learning techniques, has attracted increasing attention: however, several key problems in these deep-learning-based gaze estimation methods remain. Firstly, the feature fusion stage is not fully considered: existing methods simply concatenate the different obtained features into one feature, without considering their internal relationship. Secondly, dynamic features can be difficult to learn, because of the unstable extraction process of ambiguously defined dynamic features. In this study, we propose a novel method to consider feature fusion and dynamic feature extraction problems. We propose the static transformer module (STM), which uses a multi-head self-attention mechanism to fuse fine-grained eye features and coarse-grained facial features. Additionally, we propose an innovative recurrent neural network (RNN) cell—that is, the temporal differential module (TDM)—which can be used to extract dynamic features. We integrated the STM and the TDM into the static transformer with a temporal differential network (STTDN). We evaluated the STTDN performance, using two publicly available datasets (MPIIFaceGaze and Eyediap), and demonstrated the effectiveness of the STM and the TDM. Our results show that the proposed STTDN outperformed state-of-the-art methods, including that of Eyediap (by 2.9%).