Eye Gaze Tracking Research Articles

What Are You Looking at?

In this paper we present a novel mechanism to obtain enhanced gaze estimation for subjects looking at a scene or an image. The system makes use of prior knowledge about the scene (e.g. an image on a computer screen), to define a probability map of the scene the subject is gazing at, in order to find the most probable location. The proposed system helps in correcting the fixations which are erroneously estimated by the gaze estimation device by employing a saliency framework to adjust the resulting gaze point vector. The system is tested on three scenarios: using eye tracking data, enhancing a low accuracy webcam based eye tracker, and using a head pose tracker. The correlation between the subjects in the commercial eye tracking data is improved by an average of 13.91%. The correlation on the low accuracy eye gaze tracker is improved by 59.85%, and for the head pose tracker we obtain an improvement of 10.23%. These results show the potential of the system as a way to enhance and self-calibrate different visual gaze estimation systems.

Guest editorial: special issue on algorithms and architectures for real-time image and video enhancement

Recent advances in real-time image and video enhancement are enabling innovations in a broad range of applications including biomedicine, intelligent transportation, driving assistance, consumer electronics, telecommunication, robotics and surveillance. These innovations encompass complexity-aware algorithms and new hardware– software (HW–SW) architectures and aims at (1) improved visualization performance; (2) acceleration of processing (real-time instead of off-line computation); and (3) complexity reduction to meet demands involving device size, power consumption, cost of target applications such as battery-powered mobile/wearable devices, or low-cost large volume markets. This special issue presents nine papers covering different real-time algorithms and cost-efficient architectures for several image/video enhancement techniques. The accepted papers are from different international institutions located in North and South America, Europe and Asia. The research on image/video enhancement used to be mainly focused on multimedia, consumer or telecom applications. However, this special issue demonstrates the growing interest for image/video enhancement techniques to fields such as biomedicine (capsule endoscopy and neuroscience test), robotics for automation in agriculture, automotive driving assistance, and aerial surveillance. The discussed techniques include object tracking, image and video compression, edge extraction/detection for image analysis, anomaly detection, lighting conditions improvement, and contrast enhancement. The first paper by Khan et al. presents a subsamplingbased image compressor for capsule endoscopic system which is aimed at reducing the chip area and power consumption, while maintaining an acceptable video quality. A low-complexity algorithm, suitable for VLSI implementation, is developed around some special features of endoscopic images and consists of a differential pulse code modulation followed by Golomb–Rice coding. An image corner clipping algorithm is also presented. The reconstructed images are verified by medical doctors for acceptability. Compared to other transform-based algorithms targeted to capsule endoscopy, the proposed raster-scanbased scheme performs very strongly with a compression ratio of 80% and a very high reconstruction PSNR (over 45 dB). The second paper by Armato et al. also deals with biomedical-related applications. This work is focused on exploring and comparing several photometric normalization techniques to improve eye gaze tracking (EGT) systems during light changes. EGT is developed for scientific exploration in controlled environments where it is used in ophthalmology, neurology, psychology, and related areas to study oculomotor characteristics and abnormalities, and their relation to cognition and mental states. The illumination is one of the most restrictive limitations in EGT, due to the problem of pupil center estimation during illumination S. Saponara (&) Dip. Ingegneria della Informazione, Universita di Pisa, via G. Caruso 16, 56122 Pisa, Italy e-mail: sergio.saponara@iet.unipi.it

Comparitive study on photometric normalization algorithms for an innovative, robust and real-time eye gaze tracker

Eye gaze trackers (EGTs) are generally developed for scientific exploration in controlled environments or laboratories and data have been used in ophthalmology, neurology, psychology, and related areas to study oculomotor characteristics and abnormalities, and their relation to cognition and mental states. The illumination is one of the most restrictive limitation of the EGTs, due to a problem of pupil center estimation during illumination changes. Most of the current systems, indeed, work under controlled illumination conditions either in dark or indoor environments, e.g. using infrared sources or conforming the sources of light to fixed levels or pointing directions. This work is focused on exploring and comparing several photometric normalization techniques to improve EGT systems during light changes. In particular, a new wearable and wireless eye tracking system (HATCAM) is used for testing the different techniques in terms of real-time capability, eye tracking and pupil area detection. Embedding real-time image enhancement into the HATCAM can make it an innovative and robust system for eye tracking in different lighting conditions, i.e. darkness, sunlight, indoor and outdoor environments.

An Appearance based Method for Eye Gaze Tracking

Problem statement: Gaze estimation systems compute the direction of e ye gaze based on observed eye movements. The need for gaze-contingent applications is the basis of the current research work. The gaze pointing systems is a substitute for the existing input devices. Approach: The gaze tracking methods are either feature based or appear ance based. In this study, an appearance based approach for gaze tracking is proposed based on Run Length Coding (RLC). The experiment was conducted considering transitional changes and the class-intervals in iris pixels. The image acquisiti on begins from the center of the screen in anticlockwi se direction. The center of the screen was the pivo t point. Results: Using RLC, the recognition rate of 95% was achieved. The image analysis in different directions determines the gaze point. The direction s was determined with respect to the pivot point. Conclusion: The proposed system provides a robust, less computational gaze tracking method usi ng web camera.

A new input system for disabled users involving eye gaze tracker and scanning interface

PurposeThis paper aims to present a new input interaction system for people with severe disabilities. The new system works based on eye gaze tracking and single switch scanning interaction techniques. It combines eye gaze tracking and scanning in a unique way which is faster than only scanning‐based systems while more comfortable to use than only eye gaze tracking‐based systems, which is also supported by a user study.Design/methodology/approachThe aim was to design a new interaction technique combining eye tracking and scanning technique for severely disabled people. A comparative study with eight users involving only an eye tracking system and the system combining eye tracking and scanning was conducted.FindingsThe participants rated the new system easier and less strenuous to use than the eye gaze tracking‐based system, while the new system did not slow the speed of interaction.Originality/valueThis new method based on eye gaze tracking and single switch scanning interaction techniques has potential to be a new interaction technique combining free source eye tracker and scanning switches. It is hoped users who can move a limb to use a single switch‐based scanning system will be hugely benefitted by this new system.

Gaze estimation in a gaze tracking system

This article presents a gaze estimation method (GEMHSSO) based on the pupil center cornea reflection (PCCR) technique. Existing PCCR systems suffer from several problems, including the restriction of users’ head movement, and the requirement of individual calibration. This paper presents a head position compensation model using a single camera and a single light source, which realizes the analytic compensation of head motion effects on pupil-glint vectors. We then present a transformation model for individual differences to simplify the calibration process to a one-point calibration. On this basis, we establish a novel gaze estimation method that reduces the minimum hardware requirements for accurate estimation to a single camera (not calibrated) and a single light source. Without the need for a complex system, our method has the ability to estimate gaze during natural head movement, and to substantially simplify user calibration. Each step of the proposed method in this paper makes real-time implementation possible, which provides an effective solution for eye-gaze tracking in human-computer interaction systems.

Objective eye-gaze behaviour during face-to-face communication with proficient alaryngeal speakers: a preliminary study

There is a large body of research demonstrating the impact of visual information on speaker intelligibility in both normal and disordered speaker populations. However, there is minimal information on which specific visual features listeners find salient during conversational discourse. To investigate listeners' eye-gaze behaviour during face-to-face conversation with normal, laryngeal and proficient alaryngeal speakers. Sixty participants individually participated in a 10-min conversation with one of four speakers (typical laryngeal, tracheoesophageal, oesophageal, electrolaryngeal; 15 participants randomly assigned to one mode of speech). All speakers were > 85% intelligible and were judged to be 'proficient' by two certified speech-language pathologists. Participants were fitted with a head-mounted eye-gaze tracking device (Mobile Eye, ASL) that calculated the region of interest and mean duration of eye-gaze. Self-reported gaze behaviour was also obtained following the conversation using a 10 cm visual analogue scale. While listening, participants viewed the lower facial region of the oesophageal speaker more than the normal or tracheoesophageal speaker. Results of non-hierarchical cluster analyses showed that while listening, the pattern of eye-gaze was predominantly directed at the lower face of the oesophageal and electrolaryngeal speaker and more evenly dispersed among the background, lower face, and eyes of the normal and tracheoesophageal speakers. Finally, results show a low correlation between self-reported eye-gaze behaviour and objective regions of interest data. Overall, results suggest similar eye-gaze behaviour when healthy controls converse with normal and tracheoesophageal speakers and that participants had significantly different eye-gaze patterns when conversing with an oesophageal speaker. Results are discussed in terms of existing eye-gaze data and its potential implications on auditory-visual speech perception.

장애인을 위한 새로운 감성 인터페이스 연구

Objective: Making a new ergonomic interface system based on camera vision system, which helps the handicapped in home environment. Background: Enabling the handicapped to manipulate the consumer electronics by the proposed interface system. Method: A wearable device for capturing the eye image using a near-infrared(NIR) camera and illuminators is proposed for tracking eye gaze position(Heo et al., 2011). A frontal viewing camera is attached to the wearable device, which can recognize the consumer electronics to be controlled(Heo et al., 2011). And the amount of user's eye fatigue can be measured based on eye blink rate, and in case that the user's fatigue exceeds in the predetermined level, the proposed system can automatically change the mode of gaze based interface into that of manual selection. Results: The experimental results showed that the gaze estimation error of the proposed method was 1.98 degrees with the successful recognition of the object by the frontal viewing camera(Heo et al., 2011). Conclusion: We made a new ergonomic interface system based on gaze tracking and object recognition Application: The proposed system can be used for helping the handicapped in home environment

Association Between Visual Gaze Patterns and Adenoma Detection Rate During Colonoscopy: A Preliminary Investigation

Eye gaze tracking (EGT) technology follows a person's gaze and records the resulting visual gaze pattern (VGP). Adenoma detection rate (ADR) is a validated measure of colonoscopy quality. Higher ADRs are associated with prolonged withdrawal times and other endoscopic maneuvers that allow a better visualization of the mucosa; however, the influence of VGP has yet to be explored. We aim to quantify the VGP for endoscopists observing colonoscopy videos and describe the association between VGP and ADR. Furthermore, we will evaluate the relationship between VGP and the endoscopists' years of experience. Eleven endoscopists watched three videos while their VGP was recorded. The videos corresponded to 3 min of three different colonoscopy withdrawals. We divided the screen into a 3 × 3 grid of nine segments: eight peripheral and one central. We compared percent of gaze time (GT) in the central vs. peripheral segments using a paired t-test. VGP with ADR and years of practice were evaluated using Pearson's test. Subjects spent more GT in the screen's central segment (65 vs. 33%, P<0.001). ADR was significantly associated with increased percentage of central GT (r = 0.67, P = 0.024) and increased mean GT in the central segment (r = 0.70, P = 0.017). There was negative correlation between endoscopists' years of practice and the percentage of central GT (r = -0.67, P = 0.025), but no correlation between years of practice and percentage of peripheral GT (r = 0.24, P = 0.47). This reveals an association between a centrally focused VGP and ADR. Future steps include confirming in a larger sample and exploring if VGP can retrain low ADR endoscopists to perform higher quality colonoscopies.

Webcam-Based 2D Eye Gaze Estimation System By Means of Binary Deformable Eyeball Templates

Eye gaze as a form of input was primarily developed for users who are unable to use usual interaction devices such as keyboard and the mouse; however, with the increasing accuracy in eye gaze detection with decreasing cost of development, it tends to be a practical interaction method for able-bodied users in soon future as well. This paper explores a low-cost, robust, rotation and illumination independent eye gaze system for gaze enhanced user interfaces. We introduce two brand-new algorithms for fast and sub-pixel precise pupil center detection and 2D Eye Gaze estimation by means of deformable template matching methodology. In this paper, we propose a new algorithm based on the deformable angular integral search algorithm based on minimum intensity value to localize eyeball (iris outer boundary) in gray scale eye region images. Basically, it finds the center of the pupil in order to use it in our second proposed algorithm which is about 2D eye gaze tracking. First, we detect the eye regions by means of Intel OpenCV AdaBoost Haar cascade classifiers and assign the approximate size of eyeball depending on the eye region size. Secondly, using DAISMI (Deformable Angular Integral Search by Minimum Intensity) algorithm, pupil center is detected. Then, by using the percentage of black pixels over eyeball circle area, we convert the image into binary (Black and white color) for being used in the next part: DTBGE (Deformable Template based 2D Gaze Estimation) algorithm. Finally, using DTBGE algorithm, initial pupil center coordinates are assigned and DTBGE creates new pupil center coordinates and estimates the final gaze directions and eyeball size. We have performed extensive experiments and achieved very encouraging results. Finally, we discuss the effectiveness of the proposed method through several experimental results.

An integrated head pose and eye gaze tracking approach to non-intrusive visual attention measurement for wide FOV simulators

Eye gaze tracking is very useful for quantitatively measuring visual attention in virtual environments. However, most eye trackers have a limited tracking range, e.g., ±35° in the horizontal direction. This paper proposed a method to combine head pose tracking and eye gaze tracking together to achieve a large range of tracking in virtual driving simulation environments. Multiple parallel multilayer perceptrons were used to reconstruct the relationship between head images and head poses. Head images were represented with the coefficients extracted from Principal Component Analysis. Eye gaze tracking provides precise results on the front view, while head pose tracking is more suitable for tracking areas of interest than for tracking points of interest on the side view.

A NOVEL INFRARED GAZE TRACKING SYSTEM BASED ON NONUNIFORM INTERPOLATION

The gaze tracking system has become an active research field for handicapped persons as well as general people in recent years. The precise mapping method plays an important role in the system. In this paper, a novel infrared gaze tracking system based on nonuniform interpolation is proposed. In this system, the eye images for the computer to analyze are extracted under two infrared light sources and a charge-coupled device camera, and the users do not require wearing any device. First, the integral projection algorithm and canny edge detection are applied to extract the pupil boundary points from the captured eye images, and then the pupil center is computed using an efficient and accurate ellipse-fitting algorithm. Finally, to estimate where the user looks, a novel mapping method based on the nonuniform interpolation algorithm is proposed. In this mapping method, the complicated geometric eyeball model and the nonlinear mapping between the pupil center coordinates and computer monitor screen coordinates do not need to be taken into account. Experimental results show that the proposed mapping method is simple, fast and more accurate. Moreover, our system is the remote eye gaze tracking system. The users do not need to wear any device, which make the users feel more comfortable.

Adaptive regulation of CCD camera for real time eye tracking

In this paper, an eye tracking method and an automatic focusing technique, which combine the regulation of the aperture with the adjustment of the focus of the camera lens, are proposed to acquire clear eye images in the eye-gaze tracking system. Firstly, the aperture of the CCD lens is controlled to adapt the system to external lighting circumstances with the average brightness value of the processed images as the basic value. Secondly, a sum-modulus-difference (SMD) operator is used for rough and quick focusing in a large scale to obtain the eye glints, and for the control of a pan-tilt unit to track and aim at the eye, and for regulation of the lens focus. Finally, a frequency selective weighted median (FSWM) operator is applied in the determined window to focus automatically and acquire clear eye images. If the pupil contour can be extracted from the eye image, then the average gradient value of the pupil's edge points can be used for real-time focusing. Experimental results show that this system can adapt to external lighting changes and to the user's head movements. It can track the eye and acquire clear eye images in real-time.

Eyegaze Detection from Monocular Camera Image for Eyegaze Communication System

An eyegaze interface is one of the key technologies as an input device in the ubiquitous-computing society. In particular, an eyegaze communication system is very important and useful for severely handicapped users such as quadriplegic patients. Most of the conventional eyegaze tracking algorithms require specific light sources, equipment and devices. In this study, a simple eyegaze detection algorithm is proposed using a single monocular video camera. The proposed algorithm works under the condition of fixed head pose, but slight movement of the face is accepted. In our system, we assume that all users have the same eyeball size based on physiological eyeball models. However, we succeed to calibrate the physiologic movement of the eyeball center depending on the gazing direction by approximating it as a change in the eyeball radius. In the gaze detection stage, the iris is extracted from a captured face frame by using the Hough transform. Then, the eyegaze angle is derived by calculating the Euclidean distance of the iris centers between the extracted frame and a reference frame captured in the calibration process. We apply our system to an eyegaze communication interface, and verified the performance through key typing experiments with a visual keyboard on display.

Evaluating Eyegaze Targeting to Improve Mouse Pointing for Radiology Tasks

In current radiologists' workstations, a scroll mouse is typically used as the primary input device for navigating image slices and conducting operations on an image. Radiological analysis and diagnosis rely on careful observation and annotation of medical images. During analysis of 3D MRI and CT volumes, thousands of mouse clicks are performed everyday, which can cause wrist fatigue. This paper presents a dynamic control-to-display (C-D) gain mouse movement method, controlled by an eyegaze tracker as the target predictor. By adjusting the C-D gain according to the distance to the target, the mouse click targeting time is reduced. Our theoretical and experimental studies show that the mouse movement time to a known target can be reduced by up to 15%. We also present an experiment with 12 participants to evaluate the role of eyegaze targeting in the realistic situation of unknown target positions. These results indicate that using eyegaze to predict the target position, the dynamic C-D gain method can improve pointing performance by 8% and reduce the error rate over traditional mouse movement.

Auditory visual prominence

Auditory prominence is defined as when an acoustic segment is made salient in its context. Prominence is one of the prosodic functions that has been shown to be strongly correlated with facial movements. In this work, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study, a speech intelligibility experiment is conducted, speech quality is acoustically degraded and the fundamental frequency is removed from the signal, then the speech is presented to 12 subjects through a lip synchronized talking head carrying head-nods and eyebrows raise gestures, which are synchronized with the auditory prominence. The experiment shows that presenting prominence as facial gestures significantly increases speech intelligibility compared to when these gestures are randomly added to speech. We also present a follow-up study examining the perception of the behavior of the talking heads when gestures are added over pitch accents. Using eye-gaze tracking technology and questionnaires on 10 moderately hearing impaired subjects, the results of the gaze data show that users look at the face in a similar fashion to when they look at a natural face when gestures are coupled with pitch accents opposed to when the face carries no gestures. From the questionnaires, the results also show that these gestures significantly increase the naturalness and the understanding of the talking head.

A novel method for gaze tracking by local pattern model and support vector regressor

This paper presents a novel eye gaze tracking method with allowable head movement based on a local pattern model (LPM) and support vector regressor (SVR). The LPM, a combination of improved pixel-pattern-based texture feature (PPBTF) and local-binary-pattern texture feature (LBP), is employed to calculate texture features from the characteristics of the eyes and a new binocular vision scheme is adopted to detect the spatial coordinates of the eyes. The texture features from LPM and the spatial coordinates together are fed into support vector regressor (SVR) to match a gaze mapping function, and subsequently to track gaze direction under allowable head movement. The experimental results show that the proposed approach results in better accuracy in estimating the gaze direction than the state-of-the-art pupil center corneal reflection (PCCR) method.

INFORMATION ACQUISITION USING EYE-GAZE TRACKING FOR PERSON-FOLLOWING WITH MOBILE ROBOTS

In the effort of developing natural means for human-robot interaction (HRI) significant amount of research has been focusing on Person-Following (PF) for mobile robots. PF, which generally consists of detecting, recognizing and following people, is believed to be one of the required functionalities for most future robots that share their environments with their human companions. Research in this field is mostly directed towards fully automating this functionality, which makes the challenge even more tedious. Focusing on this challenge leads research to divert from other challenges that coexist in any PF system. A natural PF functionality consists of a number of tasks that are required to be implemented in the system. However, in more realistic life scenarios, not all the tasks required for PF need to be automated. Instead, some of these tasks can be operated by human operators and therefore require natural means of interaction and information acquisition. In order to highlight all the tasks that are believed to exist in any PF system, this paper introduces a novel taxonomy for PF. Also, in order to provide a natural means for HRI, TeleGaze is used for information acquisition in the implementation of the taxonomy. TeleGaze was previously developed by the authors as a means of natural HRI for teleoperation through eye-gaze tracking. Using TeleGaze in the aid of developing PF systems is believed to show the feasibility of achieving a realistic information acquisition in a natural way.

An adaptive blink detector to initialize and update a view-basedremote eye gaze tracking system in a natural scenario

A method for blink detection from video sequences gathered with a commercial camera is presented. This is used as a view-based remote eye gaze tracker (REGT) component performing two relevant functions, i.e. initialization and automatic updating in case of tracking failures. The method is based on frame differencing and eyes anthropometric properties. It has been tested on a publicly available database and results have been compared with algorithms found in literature. The obtained average true prediction rate is higher than 95%. The robustness of the automatic tracking failure detection has been tested on a set of experimental trials in different conditions, and yielded detection rates around 98%. The computational cost of the processing allows the blink detection algorithm to work in real time at 30 fps. The obtained results are in favour of combining blink detection with gaze mapping for the development of a robust view-based remote eye-gaze tracker to be introduced in different HCI contexts, specifically in the assistive technology framework.

INTEGRATION OF EMG AND EGT MODALITIES FOR THE DEVELOPMENT OF AN ENHANCED CURSOR CONTROL SYSTEM

While Eye Gaze Tracking (EGT) systems have demonstrated their potential as computer cursor control devices, their application outside the laboratory environment has been less prominent than originally expected. This may be due to limitations in the stability of the cursor controlled by EGT devices and in the potential for unintended selections ("clicks"). To address these EGT limitations, we have developed a cursor control system that integrates electromyogram-based control with EGT-based control. This integrated system allows a user to perform small cursor displacements and selection operations ("left click") by detecting distinct changes in electromyogram (EMG) signals generated during specific facial movements. Specifically, the system monitors and processes, in real time, EMG signals from four electrodes attached to the head of the subject. We compared the performance of this EMG/EGT integrated system, with the performance of a stand-alone EGT system and a hand-held mouse, in point-and-click trials. Statistical analysis of the results indicates that the EMG/EGT system is slower than the EGT system and the mouse, but it also revealed that it is significantly less error-prone than the EGT-only system. The analysis also revealed that the performance of the EMG/EGT system does not seem to be well matched to Fitts' model.