Real-time human–computer interface based on eye gaze estimation from low-quality webcam images: integration of convolutional neural networks, calibration, and transfer learning

Abstract

Listen

Translate article

Abstract Eye gaze estimation represents a well-established research domain within computer vision. It has a wide range of practical applications in numerous fields, including human–computer interaction (HCI) for cursor control, health care, and virtual reality, enhancing its suitability for adoption throughout the scientific community. Different methods have been used for eye gaze estimation, such as model based, feature based, and appearance based. The appearance-based method is mainly used because it directly estimates an individual’s gaze direction from images/videos rather than depending on specific features or geometric models. This article developed an appearance-based, real-time generic eye gaze system for HCI to control the cursor through the eye using the convolutional neural network (CNN), calibration, and transfer learning. The study employed low-quality eye images captured from a conventional desktop webcam, enabling the proposed methodology to be implemented on any computer system equipped with a similar web camera without the need for supplementary hardware. Initially, the labeled dataset of both eyes is collected using the webcam. Then, a CNN model is trained by inputting left and right eye images to predict the gaze coordinate as output. We applied the calibration and transfer learning approach to the trained models to make a generic model for new users. In real-time use, the first step is calibration, where the user’s eye images are captured for various screen coordinates, and transfer learning is employed to fine-tune the pre-trained model according to the user’s eyes. Then, the fine-tuned model is used for eye gaze prediction to control the cursor. The system’s performance is evaluated using a test group of multiple users, and it demonstrated an average visual angle accuracy of 2.08 degrees before calibration, which notably improved to 1.81 degrees after the calibration process.