User's Gaze Research Articles

EyeWheel: Autonomous Wheelchair Mobility Through Eye Movement Control

This paper explores the advancements in assistive technology with gaze-controlled wheelchairs. These wheelchairs utilize eye-tracking sensors and intelligent algorithms to translate a user's gaze patterns into steering commands. This offers an intuitive control method for individuals with severe mobility limitations, such as those with ALS or quadriplegia. By simply looking in the desired direction, users can navigate their surroundings with improved independence and quality of life compared to traditional mobility aids. The paper highlights the benefits of smooth and precise control through eye movements. Additionally, it explores the integration of safety features like obstacle detection and customizable settings. These features contribute to enhanced user confidence and security, making gaze- controlled wheelchairs a transformative solution for individuals with diverse mobility needs. Keywords- eye-tracking, assistive technology, mobility impairment, gaze control, wheelchair navigation

Gaze, Wall, and Racket: Combining Gaze and Hand-Controlled Plane for 3D Selection in Virtual Reality

Raypointing, the status-quo pointing technique for virtual reality, is challenging with many occluded and overlapping objects. In this work, we investigate how eye-tracking input can assist the gestural ray pointing in the disambiguation of targets in densely populated scenes. We explore the concept of Gaze + Plane, where the intersection between the user's gaze and a hand-controlled plane facilitates 3D position specification. In particular, two techniques are investigated: Gaze&Wall, which employs an indirect plane positioned in depth using a hand ray, and Gaze&Racket, featuring a hand-held and rotatable plane. In a first experiment, we reveal the speed-error trade-offs between Gaze + Plane techniques. In a second study, we compared the best techniques to newly designed gesture-only techniques, finding that Gaze&Wall is less error-prone and significantly faster. Our research has relevance for spatial interaction, specifically on advanced techniques for complex 3D tasks.

A shared robot control system combining augmented reality and motor imagery brain–computer interfaces with eye tracking

Objective: Brain-computer interface (BCI) control systems monitor neural activity to detect the user's intentions, enabling device control through mental imagery. Despite their potential, decoding neural activity in real-world conditions poses significant challenges, making BCIs currently impractical compared to traditional interaction methods. This study introduces a novel motor imagery (MI) BCI control strategy for operating a physically assistive robotic arm, addressing the difficulties of MI decoding from electroencephalogram (EEG) signals, which are inherently non-stationary and vary across individuals.
Approach: A proof-of-concept BCI control system was developed using commercially available hardware, integrating MI with eye tracking in an augmented reality (AR) user interface to facilitate a shared control approach. This system proposes actions based on the user's gaze, enabling selection through imagined movements. A user study was conducted to evaluate the system's usability, focusing on its effectiveness and efficiency.
Main results:Participants performed tasks that simulated everyday activities with the robotic arm, demonstrating the shared control system's feasibility and practicality in real-world scenarios. Despite low online decoding performance (mean accuracy: 0.52 9, F1: 0.29, Cohen's Kappa: 0.12), participants achieved a mean success rate of 0.83 in the final phase of the user study when given 15 minutes to complete the evaluation tasks. The success rate dropped below 0.5 when a 5-minute cutoff time was selected.
Significance: These results indicate that integrating AR and eye tracking can significantly enhance the usability of BCI systems, despite the complexities of MI-EEG decoding. While efficiency is still low, the effectiveness of our approach was verified. This suggests that BCI systems have the potential to become a viable interaction modality for everyday applications
in the future.

Impact of Socio-Demographic Attributes and Mutual Gaze of Virtual Humans on Users' Visual Attention and Collision Avoidance in VR.

This study investigated the extent that the non-verbal behaviors of virtual humans (VHs) and their socio-demographic attributes altered users' collision avoidance behaviors in Virtual Reality (VR). Users interacted with VHs representing different levels of ethnicities and gender, exhibiting different conditions of physical movement, and gaze behaviors. The VHs were depicted in three major ethnic conditions namely Asian, Caucasian, and Black. The physical movement states of the VHs were either static in the path of the user or walking toward the user in the opposite direction. The non-verbal gaze behavior of the VHs was either direct gaze or averted gaze. We used an HTC Vive tracking system to track users' performing real walking while we collected objective measures (i,e., continuous gaze, fixation gaze, clearance distance, and travel length), and subjective variables (i.e., game experiences and social presence). The results showed that the ethnicity of the VHs significantly impacted the gaze behavior of the users, and the gender of the VHs affected the user avoidance movement and their reciprocal gaze behavior. Our results revealed that users' physical movement, gaze behaviors, and collision avoidance were moderated by the VHs' perceived ethnicity, gender, and gaze behaviors. Understanding the impact of the socio-demographics attributes of VHs and their gaze behavior on users' collision avoidance is critical for applications in which users are navigating through virtual traffic, crowd, and other inter-personal simulations.

Visual perception of distance in 3D‐augmented reality head‐up displays

AbstractA head‐up display (HUD) is conveniently used to provide information on the display without changing the user's gaze. 3D HUDs, capable of projecting three‐dimensional information beyond the 2D HUDs, enable the projection of augmented reality (AR) objects into the real world. Research on the perception of 3D AR HUDs is crucial for their efficient utilization and secure commercialization. In this study, we examined whether a 3D HUD is more comfortable than a 2D HUD in the context of viewing real‐world environments and augmented reality objects together. Additionally, we analyzed participants' perception of distance and fatigue for AR objects at varying distances from the 3D HUD. The study found that using a 3D HUD in an AR environment resulted in less fatigue than using a 2D HUD, as determined by a Mann–Whitney statistical analysis. Participants were able to match the depth of AR objects in a 3D HUD within a range of 3 to 50 m, with similar diopter and parallax angular distance errors, regardless of the distance of the AR object. Visual fatigue increases with increasing distance from the virtual‐image plane and can be modeled as a quadratic function in the domain of diopter and parallax angles.

Capturing Initial Gaze Attraction in Branded Spaces Through VR Eye-Tracking Technology

This paper leverages recent strides in virtual reality (VR) technology to dissect user gaze patterns and initial responses in a VR-based in-store environment, with a focus on how visual elements impact attention in Human–Computer Interaction (HCI). Specifically, this exploratory study focuses on brand-centric components such as logos and colors, while also considering individual variables such as gender and design background. Employing a mixed-methods approach, the research combines qualitative and quantitative tools. A VR-integrated eye-tracking system records how participants’ gaze behaves in a virtual retail environment. The study collects data from 50 participants in four scenarios: a control store (Case 1), introduction of the brand’s logo signage (Case 2), infusion of the brand’s primary color (Case 3), and a fusion of logo and color (Case 4). Two key eye-tracking metrics—time to first fixation (TTFF) and first fixation duration (FFD)—are harnessed to scrutinize gaze behavior across elements such as wall color, logo signage, wall display fixtures, and media screens. The research unearths statistically significant effects of visual elements on gaze behavior, with particular emphasis on TTFF and FFD. Gender disparities come to light in Cases 2, 3, and 4, affecting FFD on wall color (AOI 1) and TTFF on brand logo signage (AOI 2) in Case 2. Moreover, disparities in design background surface in Cases 1 and 3. Complementing the eye-tracking data and post-experiment interviews capture users’ initial gaze attraction and preferences, offering valuable insights into their rationale. In essence, this study underscores the transformative potential of VR technology and eye-tracking techniques in HCI and furnishes actionable insights for those seeking to elevate brand recognition and consumer engagement within immersive virtual retail settings.

NeighboAR: Efficient Object Retrieval using Proximity- and Gaze-based Object Grouping with an AR System

Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.

Eye-Hand Typing: Eye Gaze Assisted Finger Typing via Bayesian Processes in AR.

Nowadays, AR HMDs are widely used in scenarios such as intelligent manufacturing and digital factories. In a factory environment, fast and accurate text input is crucial for operators' efficiency and task completion quality. However, the traditional AR keyboard may not meet this requirement, and the noisy environment is unsuitable for voice input. In this article, we introduce Eye-Hand Typing, an intelligent AR keyboard. We leverage the speed advantage of eye gaze and use a Bayesian process based on the information of gaze points to infer users' text input intentions. We improve the underlying keyboard algorithm without changing user input habits, thereby improving factory users' text input speed and accuracy. In real-time applications, when the user's gaze point is on the keyboard, the Bayesian process can predict the most likely characters, vocabulary, or commands that the user will input based on the position and duration of the gaze point and input history. The system can enlarge and highlight recommended text input options based on the predicted results, thereby improving user input efficiency. A user study showed that compared with the current HoloLens 2 system keyboard, Eye-Hand Typing could reduce input error rates by 28.31 % and improve text input speed by 14.5%. It also outperformed a gaze-only technique, being 43.05% more accurate and 39.55% faster. And it was no significant compromise in eye fatigue. Users also showed positive preferences.

PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR Experiences.

From education to medicine to entertainment, a wide range of industrial and academic fields now utilize eXtended Reality (XR) technologies. This diversity and growing use are boosting research and leading to an increasing number of XR experiments involving human subjects. The main aim of these studies is to understand the user experience in the broadest sense, such as the user cognitive and emotional states. Behavioral data collected during XR experiments, such as user movements, gaze, actions, and physiological signals constitute precious assets for analyzing and understanding the user experience. While they contribute to overcome the intrinsic flaws of explicit data such as post-experiment questionnaires, the required acquisition and analysis tools are costly and challenging to develop, especially for 6DoF (Degrees of Freedom) XR experiments. Moreover, there is no common format for XR behavioral data, which restrains data-sharing, and thus hinders wide usages across the community, replicability of studies, and the constitution of large datasets or meta-analysis. In this context, we present PLUME, an open-source software toolbox (PLUME Recorder, PLUME Viewer, PLUME Python) that allows for the exhaustive record of XR behavioral data (including synchronous physiological signals), their offline interactive replay and analysis (with a standalone application), and their easy sharing due to our compact and interoperable data format. We believe that PLUME can greatly benefit the scientific community by making the use of behavioral and physiological data available for the greatest, contributing to the reproducibility and replicability of XR user studies, enabling the creation of large datasets, and contributing to a deeper understanding of user experience.

Integrating Egocentric and Robotic Vision for Object Identification Using Siamese Networks and Superquadric Estimations in Partial Occlusion Scenarios.

This paper introduces a novel method that enables robots to identify objects based on user gaze, tracked via eye-tracking glasses. This is achieved without prior knowledge of the objects' categories or their locations and without external markers. The method integrates a two-part system: a category-agnostic object shape and pose estimator using superquadrics and Siamese networks. The superquadrics-based component estimates the shapes and poses of all objects, while the Siamese network matches the object targeted by the user's gaze with the robot's viewpoint. Both components are effectively designed to function in scenarios with partial occlusions. A key feature of the system is the user's ability to move freely around the scenario, allowing dynamic object selection via gaze from any position. The system is capable of handling significant viewpoint differences between the user and the robot and adapts easily to new objects. In tests under partial occlusion conditions, the Siamese networks demonstrated an 85.2% accuracy in aligning the user-selected object with the robot's viewpoint. This gaze-based Human-Robot Interaction approach demonstrates its practicality and adaptability in real-world scenarios.

Attention-Aware Visualization: Tracking and Responding to User Perception Over Time.

We propose the notion of attention-aware visualizations (AAVs) that track the user's perception of a visual representation over time and feed this information back to the visualization. Such context awareness is particularly useful for ubiquitous and immersive analytics where knowing which embedded visualizations the user is looking at can be used to make visualizations react appropriately to the user's attention: for example, by highlighting data the user has not yet seen. We can separate the approach into three components: (1) measuring the user's gaze on a visualization and its parts; (2) tracking the user's attention over time; and (3) reactively modifying the visual representation based on the current attention metric. In this paper, we present two separate implementations of AAV: a 2D data-agnostic method for web-based visualizations that can use an embodied eyetracker to capture the user's gaze, and a 3D data-aware one that uses the stencil buffer to track the visibility of each individual mark in a visualization. Both methods provide similar mechanisms for accumulating attention over time and changing the appearance of marks in response. We also present results from a qualitative evaluation studying visual feedback and triggering mechanisms for capturing and revisualizing attention.

A Study on Mobile Phone Interface Interaction Design for the Elderly Based on Digital Technology

Abstract The rapid development of digital technology significantly impacts the use of cell phone interfaces by the elderly. This paper aims to analyze the interaction design of cell phone interfaces for the elderly to enhance their user experience and ease of use. This paper investigates the evaluation system for evaluating cell phone interface interaction for the elderly, including interface size, Layout, and functionality. A group of elderly people aged 65-75 are invited to evaluate the effects of different design options by observing their performance under different interface sizes (large size 6.0 inches, medium size 5.5 inches, small size 4.5 inches) and layouts (nine-cell grid and combination type). The evaluation metrics are task completion time, number of operation errors, and eye movement data. The 6.0-inch interface experienced an average task completion time of 6.97 seconds, a decrease of approximately 6 seconds and 1 second from the 4.5-inch and 5.5-inch interfaces, respectively. The large-size interface had an average error rate of 1.11 per task, significantly less than the average error rate in the 4.5-inch and 5.5-inch interfaces. The more extensive interface significantly decreased user gaze points and gaze time, indicating more efficient information acquisition. More extensive cell phone interfaces and combined layouts can significantly improve interaction efficiency and reduce operation errors for elderly users, enhancing user experience. The findings of this study offer practical guidance for designing cell phone interfaces for the elderly and aid in promoting cell phone interface design for the elderly.

Measuring digital literacy with eye tracking: an examination of skills and performance based on user gaze

PurposeThis study presents an innovative approach to analyzing user behavior when performing digital tasks by integrating eye-tracking technology. Through the measurement of user scan patterns, gaze and attention during task completion, the authors gain valuable insights into users' approaches and execution of these tasks.Design/methodology/approachIn this research, the authors conducted an observational study that centered on assessing the digital skills of individuals with limited proficiency who enrolled in a computer introductory course. A group of 19 participants were tasked with completing various online assignments both before and after completing the course.FindingsThe study findings indicate a significant improvement in participants' skills, particularly in basic and straightforward applications. However, advancements in more sophisticated utilization, such as mastering efficient search techniques or harnessing the Internet for enhanced situational awareness, demonstrate only marginal enhancement.Originality/valueIn recent decades, extensive research has been conducted on the issue of digital inequality, given its significant societal implications. This paper introduces a novel tool designed to analyze digital inequalities and subsequently employs it to evaluate the effectiveness of “LEHAVA,” the largest government-sponsored program aimed at mitigating these disparities in Israel.

Pupil Position by an Improved Technique of YOLO Network for Eye Tracking Application

This Eye gaze following is the real-time collection of information about a person's eye movements and the direction of their look. Eye gaze trackers are devices that measure the locations of the pupils to detect and track changes in the direction of the user's gaze. There are numerous applications for analyzing eye movements, from psychological studies to human-computer interaction-based systems and interactive robotics controls. Real-time eye gaze monitoring requires an accurate and reliable iris center localization technique. Deep learning technology is used to construct a pupil tracking approach for wearable eye trackers in this study. This pupil tracking method uses deep-learning You Only Look Once (YOLO) model to accurately estimate and anticipate the pupil's central location under conditions of bright, natural light (visible to the naked eye). Testing pupil tracking performance with the upgraded YOLOv7 results in an accuracy rate of 98.50% and a precision rate close to 96.34% using PyTorch.

Continuous Gaze Tracking With Implicit Saliency-Aware Calibration on Mobile Devices

Gaze tracking is a useful human-to-computer interface, which plays an increasingly important role in a range of mobile applications. Gaze calibration is an indispensable component of gaze tracking, which transforms the eye coordinates to the screen coordinates. The existing approaches of gaze tracking either have limited accuracy or require the user's cooperation in calibration and in turn hurt the quality of experience. We in this paper propose vGaze, continuous gaze tracking with implicit saliency-aware calibration on mobile devices. The design of vGaze stems from our insight on the temporal and spatial dependent relation between the visual saliency and the user's gaze. vGaze is implemented as a light-weight software that identifies video frames with "useful" saliency information, sensing the user's head movement, performs opportunistic calibration using only those "useful" frames, and leverages historical information for accelerating saliency detection. We implement vGaze on a commercial mobile device and evaluate its performance in various scenarios. The results show that vGaze can work at real time with video playback applications. The average error of gaze tracking is 1.51 cm (2.884 degree) which decreases to 0.99 cm (1.891 degree) with historical information and 0.57 cm (1.089 degree) with an indicator.

Wheelchair-Centered Omnidirectional Gaze-Point Estimation in the Wild

In this article, we propose a dataset named Wheelchair-OmniGaze and a method of estimating the gaze point of a wheelchair user that can cope with large head motion as the first step for understanding the intention of a wheelchair user. In the proposed method, the wheelchair user's face is observed by two remote cameras, called face cameras, mounted on the two front sides of the wheelchair, while a spherical camera with a 360° field of view, called a spherical scene camera, is mounted to the top rear to observe the surrounding scenes. A user's gaze point at the environment is determined by combining the facial information acquired by the face cameras with the spherical image acquired by the spherical scene camera. Concretely, a convolutional neural network–long short-term memory network is used to learn gaze points of the spherical image from the two face image sequences. Using two face cameras enables a user's face to be observed even for a large head motion. Using a spherical scene camera mounted on a wheelchair, the representation of a user's gaze points at a wheelchair-centered spherical camera coordinate can be obtained, which is independent of a user's head motion. The experimental results show that our method achieved competitive performance in comparison with the state-of-the-art method. Since our problem definition is different from the existing research, the experimental results of this article can be seen as a baseline for our built dataset.

Exploring Gaze-assisted and Hand-based Region Selection in Augmented Reality

Region selection is a fundamental task in interactive systems. In 2D user interfaces, users typically use a rectangle selection tool to formulate a region using a mouse or touchpad. Region selection in 3D spaces, especially in Augmented Reality (AR) Head-Mounted Displays (HMDs) is different and challenging because users need to select an intended region via freehand mid-air gestures or eye-based actions that are touchless interactions. In this work, we aim to fill in the gap in the design of region selection techniques in AR HMDs. We first analyzed and discretized the interaction procedure of region selection and explored design possibilities for each step. We then developed four techniques for region selection in AR HMDs, which leveraged users' hand and gaze for unimodal or multimodal interaction. The techniques were evaluated via a user study with a controlled region selection task. The findings led to three design recommendations and two proof-of-concept application examples.

Reducing VR Sickness by Directing User Gaze to Motion Singularity Point/Region as Effective Rest Frame

Simulation sickness is one major obstacle in proliferation of virtual reality. The sensory mismatch between the visual and vestibular senses about user motion is attributed as the main cause. One effective method has been the use of the rest frame, which refers to the reference object that remains fixed in position with respect to the user. A popular choice of the rest frame is the virtual nose, but it can direct one’s attention away from the main part of the navigation content. We propose to instead use the area within the screen space that represents the least amount of motion in the navigation content, called the Motion Singularity Point/Region (MSP/R) as such a rest frame. Viewing such a region is thus expected to reduce the sensory conflict and ensuing sickness. Such a region can be found by analyzing the content with respect to motion and estimating the region(s) of the image space with the least relative amount of the total optical flow. We experimentally validated the VR sickness reduction effect of looking at the MSP/R compared to that of the virtual nose. In addition, we confirmed that the content agnostic MSP/R to be much coinciding with the user’s natural viewing direction, making it less distracting from the main content. This makes the MSP/R a more practical and viable rest frame object for sickness reduction than e.g. the virtual nose.

Development of a Tourist Information Search Support System that Reflects Interests Based on User's Gaze

In 2014, Tobii Tech and Eye Tribe released the first inexpensive non-contact eye tracking device, and eye tracking technology has been attracting more and more attention. It has long been said that gaze information can be used to infer a person's interests and attention, and the analysis of gaze position by eye tracking technology is also used for Search Engine Optimization (SEO). In addition, with the spread of information and communication technology, the Internet has become the most used media for "obtaining information on hobbies and entertainment," surpassing books, magazines, and television for all age groups except those in their 60s in Japan, The majority of information gathering has shifted to web browsing. As a result, the information gap between those who can use the Internet to collect information and those who cannot use it well, so-called digital divide, has become one of the social problems. Based on these background, we attempted to develop an interaction system that presents search keyword to get related information by using the user's eyes while browsing the web. Based on analysis of user's gaze and accumulation of web pages viewed by each user, this system can provide search keywords that reflect a user's interest, such as Tourism information or history information. We believe that this system can support users' web browsing by assisting them in gathering information on sightseeing and travel, presenting them with unexpected facilities that they did not know about, and making it easy for them to obtain a variety of related information.

Gaze as an Indicator of Input Recognition Errors

Input recognition errors are common in gesture- and touch-based recognition systems, and negatively affect user experience and performance. When errors occur, systems are unaware of them, but the user's gaze following an error may provide valuable cues for error detection. A study was conducted using a manual serial selection task to investigate whether gaze could be used to discriminate user-initiated selections from injected false positive selection errors. Logistic regression models of gaze dynamics could successfully identify injected selection errors as early as 50 milliseconds following a selection, with performance peaking at 550 milliseconds. A two-phase gaze pattern was observed in which users exhibited high gaze motion immediately following errors, and then decreased gaze motion as the error was noticed. Together, these results provide the first demonstration that gaze dynamics can be used to detect input recognition errors, and open new possibilities for systems that can assist with error recovery.