Human Gaze Research Articles

A novel binary hashing for agricultural scenery classification.

In this research, we present PerceptHashing, a technique designed to categorize million-scale agricultural scenic images by incorporating human gaze shifting paths (GSPs) into a hashing framework. For each agricultural image, we identify visually and semantically significant object patches, such as fields, crops, and water bodies. These patches are linked to form a graphlet, establishing a network of spatially adjacent patches, and a GSP is then extracted using an active learning algorithm. The GSP reflects the distribution of human gaze across different regions of each agricultural scene, typically involving fewer than 12 regions, as validated by cross-validation. We then design a binary hashing framework that effectively leverages the semantics encoded in these GSPs. This framework integrates three key elements: (i) refinement of noisy labels, (ii) incorporation of deep image-level agricultural semantics, and (iii) updates to the adaptive data graph. The resulting hash codes from each GSP are converted into a kernelized visual descriptor for classification. To evaluate the influence of GSPs on agricultural image classification both qualitatively and quantitatively, we conducted an extensive user study comparing GSPs from typical observers with those from Alzheimer's patients. The results demonstrate that: (1) the classification accuracy of 1.22 million agricultural aerial images using our approach is significantly higher than those processed by other methods, and (2) GSPs from 33 Alzheimer's patients differ markedly from those of 37 normal observers, leading to notable differences in classification accuracy ( versus ).

Interpreting Bar Charts: Effects of 3D Depth Cues on Human Gaze and User Understanding

Three-dimensional information visualizations are widely used in various fields for their aesthetic appeal. However, using them can sometimes lead to occlusion and distortion, which raises questions about when and why to use them. In this study, we aimed to investigate the effects of three-dimensional visualizations on human gaze and user understanding and analyze the perception process in detail. Our empirical research consisted of a two-part experimental study that involved both subjective and objective evaluation. We specifically focused on bar charts as they are among the most frequently used types of information visualizations. The results suggest that, for bar chart visualizations with varying gap dimensions, there is no statistically significant difference in user understanding between the two-dimensional and three-dimensional versions. Our findings indicate that, in general, three-dimensional bar chart visualizations are as comprehensible as their two-dimensional counterparts for the gap dimensions examined in this research. This study provides empirical insights demonstrating that both 3D and 2D bar charts are equally understandable, particularly when a specific gap depth is used in 3D visualizations. These findings contribute to the ongoing discussion about the effective use of three-dimensional visualizations and highlight areas for further research.

Predictive Gaze Analytics: A Comparative Case Study of the Foretelling Signs of User Performance during Interaction with Visualizations of Ontology Class Hierarchies

The current research landscape in ontology visualization has largely focused on tool development, yielding an extensive array of visualization tools. Although many existing solutions provide multiple ontology visualization layouts, there is limited research in adapting to an individual user’s performance, despite successful applications of adaptive technologies in related fields, including information visualization. In an effort to innovate beyond traditional one-size-fits-all visualizations, this paper contributes one step towards realizing user adaptive visualization by recognizing timely moments when users may potentially need intervention, as real-time adaptation can only occur if it is possible to correctly predict user success and failure during an interaction in the first place. In addition, an open-source, reusable, and extensible software: Beach Environment for the Analytics of Human Gaze (BEACH-Gaze) is made available to the broader scientific community interested in descriptive and predictive gaze analytics. Building on a wealth of research in eye tracking, this paper compares four approaches to predictive gaze analytics through a series of experiments that utilize scheduled gaze digests, irregular gaze events, the last known gaze status, as well as all gaze captured for a user at a given moment in time. The results from a set of experimental trials suggest that irregular gaze events are most informative of early predictions of user performance, whereas cognitive workload appears to be most indicative of overall user performance in the task scenario presented in this paper. These empirical findings highlight the importance of an analytical approach to gaze on user predictions and indicate careful consideration when applying.

Gaze tracking embedded collaborative robots for automated metrology and reverse engineering

Conventional geometric metrology, or three-dimensional (3D) scanning, and reverse engineering heavily rely on the experience of the operators. With an increasing need for automation, robot arms have been adopted for this task. However, due to the large variety of parts and designs, automated path planning could provide a scanning solution that may overlook the critical area, which could potentially deteriorate the scan results. This article explores the integration of collaborative robotics (cobots) with eye-tracking technology to improve the autonomous 3D scanning process. The primary objective of this study is to enhance the accuracy and efficiency of cobots in 3D scanning, particularly in the capture of functionally critical areas, and to provide a detailed description of regions with complex geometric features. The study develops a framework where the scanning path of the robot-driven scanner is partially guided by eye tracking data, that is, calibrated gaze tracking, to improve the automated 3D scanning process. This framework provides an innovative integration of human gaze movement with automatic robot path planning, providing a new way of human-autonomy teaming. Case studies are presented to present and validate the proposed framework to automatically improve the 3D point cloud collection process, specifically in areas that usually require human manual intervention to capture details.

Evaluating User Experience and Data Quality in Gamified Data Collection for Appearance-Based Gaze Estimation

Appearance-based gaze estimation, which uses only a regular camera to estimate human gaze, is important in various application fields. While the technique faces data bias issues, data collection protocol is often demanding, and collecting data from a wide range of participants is difficult. It is an important challenge to design opportunities that allow a diverse range of people to participate while ensuring the quality of the training data. To tackle this challenge, we introduce a novel gamified approach for collecting training data. In this game, two players communicate words via eye gaze through a transparent letter board. Images captured during gameplay serve as valuable training data for gaze estimation models. The game is designed as a physical installation that involves communication between players, and it is expected to attract the interest of diverse participants. We assess the game’s significance on data quality and user experience through a comparative user study.

Learning From Human Attention for Attribute-Assisted Visual Recognition.

With prior knowledge of seen objects, humans have a remarkable ability to recognize novel objects using shared and distinct local attributes. This is significant for the challenging tasks of zero-shot learning (ZSL) and fine-grained visual classification (FGVC), where the discriminative attributes of objects have played an important role. Inspired by human visual attention, neural networks have widely exploited the attention mechanism to learn the locally discriminative attributes for challenging tasks. Though greatly promoted the development of these fields, existing works mainly focus on learning the region embeddings of different attribute features and neglect the importance of discriminative attribute localization. It is also unclear whether the learned attention truly matches the real human attention. To tackle this problem, this paper proposes to employ real human gaze data for visual recognition networks to learn from human attention. Specifically, we design a unified Attribute Attention Network (A 2 Net) that learns from human attention for both ZSL and FGVC tasks. The overall model consists of an attribute attention branch and a baseline classification network. On top of the image feature maps provided by the baseline classification network, the attribute attention branch employs attribute prototypes to produce attribute attention maps and attribute features. The attribute attention maps are converted to gaze-like attentions to be aligned with real human gaze attention. To guarantee the effectiveness of attribute feature learning, we further align the extracted attribute features with attribute-defined class embeddings. To facilitate learning from human gaze attention for the visual recognition problems, we design a bird classification game to collect real human gaze data using the CUB dataset via an eye-tracker device. Experiments on ZSL and FGVC tasks without/with real human gaze data validate the benefits and accuracy of our proposed model. This work supports the promising benefits of collecting human gaze datasets and automatic gaze estimation algorithms learning from human attention for high-level computer vision tasks.

Enhancing social robot's direct gaze expression through vestibulo-ocular movements

Non-verbal communication, especially eye contact, plays a crucial role in human interaction. Integrating eye contact capabilities into robotic behavior presents challenges that require comprehensive interdisciplinary collaboration from the fields of vision science and robotics. This study introduces an innovative methodology to improve the user's perception of the tabletop social robot Haru's direct gaze by explaining how eye and head movements can be managed through simulating visual perception processing. By considering the mechanisms of eye and head movements during visual perception processing, the robot's eye and head gaze behaviors are intended to replicate human gaze movements with Vestibulo-Ocular Movement (VOM) during face-to-face interaction, ultimately enhancing the user's perception of the robot's direct gaze.

Fine-grained gaze estimation based on the combination of regression and classification losses

Human gaze is a crucial cue used in various applications such as human-robot interaction, autonomous driving, and virtual reality. Recently, convolution neural network (CNN) approaches have made notable progress in predicting gaze angels. However, estimating accurate gaze direction in-the-wild is still a challenging problem due to the difficulty of obtaining the most crucial gaze information that exists in the eye area which constitutes a small part of the face images. In this paper, we introduce a novel two-branch CNN architecture with a multi-loss approach to estimate gaze angles (pitch and yaw) from face images. Our approach utilizes separate fully connected layers for each gaze angle prediction, allowing explicit learning of discriminative features and emphasizing the distinct information associated with each gaze angle. Moreover, we adopt a multi-loss approach, incorporating both classification and regression losses. This allows for joint optimization of the combined loss for each gaze angle, resulting in improved overall gaze performance. To evaluate our model, we conduct experiments on three popular datasets collected under unconstrained settings: MPIIFaceGaze, Gaze360, and RT-GENE. Our proposed model surpasses current state-of-the-art methods and achieves state-of-the-art performance on all three datasets, showcasing its superior capability in gaze estimation.

Impact of Socio-Demographic Attributes and Mutual Gaze of Virtual Humans on Users' Visual Attention and Collision Avoidance in VR.

This study investigated the extent that the non-verbal behaviors of virtual humans (VHs) and their socio-demographic attributes altered users' collision avoidance behaviors in Virtual Reality (VR). Users interacted with VHs representing different levels of ethnicities and gender, exhibiting different conditions of physical movement, and gaze behaviors. The VHs were depicted in three major ethnic conditions namely Asian, Caucasian, and Black. The physical movement states of the VHs were either static in the path of the user or walking toward the user in the opposite direction. The non-verbal gaze behavior of the VHs was either direct gaze or averted gaze. We used an HTC Vive tracking system to track users' performing real walking while we collected objective measures (i,e., continuous gaze, fixation gaze, clearance distance, and travel length), and subjective variables (i.e., game experiences and social presence). The results showed that the ethnicity of the VHs significantly impacted the gaze behavior of the users, and the gender of the VHs affected the user avoidance movement and their reciprocal gaze behavior. Our results revealed that users' physical movement, gaze behaviors, and collision avoidance were moderated by the VHs' perceived ethnicity, gender, and gaze behaviors. Understanding the impact of the socio-demographics attributes of VHs and their gaze behavior on users' collision avoidance is critical for applications in which users are navigating through virtual traffic, crowd, and other inter-personal simulations.

Predicting Code Comprehension: A Novel Approach to Align Human Gaze with Code using Deep Neural Networks

Predicting Code Comprehension: A Novel Approach to Align Human Gaze with Code using Deep Neural Networks

3D-Oculography: New Method of Determination of Human Gaze Point Position in Space

INTRODUCTION: Oculography — a method of recording movement of eyeballs of a human or animal by analyzing changes in electrical potentials recorded by two electrodes fixed on skin near the eye socket is used to solve various problems. Among them are the determination of pronounced nystagmus, of characteristic changes of oculomotor reactions in different conditions of observing video images. Of interest is the determination of the position of gaze point (GP) and of the area of increased attention in three-dimensional space. This information is associated with the cognitive system of the observer and is of interest not only to physiologists, but also to specialists of related fields. . AIM: To experimentally prove the effectiveness of the developed method of 3D-oculography providing determination of the GP position in space. MATERIALS AND METHODS: The possibility of restoring GP position based on the analysis of recorded oculograms even in the presence of additive noise induced by the external electromagnetic fields, is proven by the numeric modeling method. Direct experimental studies were conducted with recording oculograms of a human observer on a previously verified equipment. A ZB-2 oculograph was used, the data obtained were verified by the method of average values. RESULTS: The obtained modeling results permitted to determine the maximal values of dispersion of additive noise and amplitude shifts of signals, which make possible satisfactory restoration of the coordinates of the GP and parameters of its movement trajectory. The qualitative correspondence of the experimental results to the results of numerical modeling was proven. The correspondence between trajectories of the observed object moving in space and trajectories of GP movement synthesized from counts of recorded oculograms was confirmed. CONCLUSION: Multichannel record of oculographic signals permits restoring parameters of the GP trajectory. Creating systems for recording these signals requires minimization of the noise level. An increase in dispersion of the noise component of the signal leads to most significant errors in calculating the coordinates of trajectory points.

Basic Study for Grasping the Human Gaze Movement of a Remoting-Ship Operator

Our research focuses on understanding the disparity in risk perception when operating maritime autonomous surface ships (MASS) between onboard and through screens. In our initial step, we conducted experiments using a ship simulator, involving 4 students aspiring to be navigators. They navigated through screens and participated in debriefing sessions. Human gaze movement was utilized to ascertain the ability to comprehend navigational risks solely through observation. Simultaneously, salivary alpha-amylase samples were taken to verify their accurate risk cognition. The key finding indicates that human gaze movement aids in precisely understanding navigational risks, suggesting the feasibility of future ship operations through screens. Our research targets navigators crucial in global logistics, aiming to enhance safety in remote ship operations by demonstrating that risk cognition is achievable through human gaze movement. This breakthrough is vital for the evolving realm of MASS operations.

Perceptive Recommendation Robot: Enhancing Receptivity of Product Suggestions Based on Customers' Nonverbal Cues.

Service robots that coexist with humans in everyday life have become more common, and they have provided customer service in physical shops around the world in recent years. However, their potential in effective sales strategies has not been fully realized due to their low social presence. This study aims to clarify what kind of robot behavior enhances the social presence of service robots and how it affects human-robot interaction and purchasing behavior. We conducted two experiments with a sales robot, Pepper, at a retail shop in Kyoto. In Experiment 1, we showed that the robot's social presence increased and that customers looked at the robot longer when the robot understood human gaze information and was capable of shared attention. In Experiment 2, we showed that the probability of customers picking up products increased when the robot suggested products based on the humans' degree of attention from gaze and posture information. These results indicate that the robot's ability to understand and make utterances about a customer's orientation and attention effectively enhances human-robot communication and purchasing motivation.

35‐3: Modeling Eye Movement and Reflection in Virtual Environments for Eye Tracking

Eye movement tracking plays a pivotal role in use interaction within Virtual Reality (VR) environments. Our research focuses on an advanced eye movement simulation using virtual environment for eye tracking training and eye tracking precision evaluating platform. We developed a real‐eye model, enhanced light refraction and corneal reflections under infrared LED light with eye rotation. Our study also customized screen,light source and cameras placement to capture eye images more faithfully. We also presents a noval testing methology and equipments of spatial accuracy testing of the eye‐tracking systems. The proposed proprietary eye‐tracking test system designed to recreate scenarios of human gaze and provide true values for eye gaze directions. Overall, the integration of Pupil Center Corneal Reflection (PCCR) based dataset generation and testing platform mark a significant advancement, enabling huge database, precise groudtruth training and accurate results testing.

Learning User Embeddings from Human Gaze for Personalised Saliency Prediction

Reusable embeddings of user behaviour have shown significant performance improvements for the personalised saliency prediction task. However, prior works require explicit user characteristics and preferences as input, which are often difficult to obtain. We present a novel method to extract user embeddings from pairs of natural images and corresponding saliency maps generated from a small amount of user-specific eye tracking data. At the core of our method is a Siamese convolutional neural encoder that learns the user embeddings by contrasting the image and personal saliency map pairs of different users. Evaluations on two public saliency datasets show that the generated embeddings have high discriminative power, are effective at refining universal saliency maps to the individual users, and generalise well across users and images. Finally, based on our model's ability to encode individual user characteristics, our work points towards other applications that can benefit from reusable embeddings of gaze behaviour.

Human Cognition for Mitigating the Paradox of AI Explainability: A Pilot Study on Human Gaze-based Text Highlighting

Artificial Intelligence (AI) explainability plays a crucial role in fostering robust Human-AI Interaction (HAI). However, circular reasoning compromises decision robustness due to limitations in existing AI explainability methods. To address this challenge, we propose leveraging human cognition to enhance explainability, aligning with analysis goals without relying on potentially biased labels. By developing text highlighting driven by human gaze patterns, our research demonstrates that human gaze-based text highlighting sig-nificantly reduces decision time for proficient readers, with-out significantly affecting accuracy or bias. This study con-cludes by emphasizing the value of human cognition-based explainability in advancing explainable AI (XAI) and HAI.

Gaze-assisted visual grounding via knowledge distillation for referred object grasping with under-specified object referring

Understanding human referring intention is the key challenge in referred object grasping tasks. Current Referring Expression Comprehension (REC) and Referring Expression Segmentation (RES) methods enable robots to ground the referred object through precise language commands while ignoring the under-specified referring expressions in practical scenarios. In this paper, we propose a gaze-assisted visual grounding network, which efficiently estimates the human gaze via knowledge distillation techniques to provide visual guidance for inferring human referring intention. Specifically, a gaze auxiliary network is introduced to incorporate the human gaze to implicitly eliminate the referring ambiguity. A cross-attention-based multi-modal fusion module is designed with a balanced structure to project the diverse modalities into a common feature space to balance the inter-modal information. A tabletop object dataset with human gaze and under-specified referring expressions was established to evaluate our method. Experimental results show that our method achieves advanced results compared to the state-of-the-art visual grounding method, with a gain of 8.23% on REC precision and 10.44% on RES mIoU (mean Intersection over Union). The effectiveness and superiority of our method are also demonstrated through real-world referred object grasping experiments. The code is now available at https://github.com/robotseu/GVGNet_KD.

Gaze-infused BERT: Do human gaze signals help pre-trained language models?

Gaze-infused BERT: Do human gaze signals help pre-trained language models?

(Re)framing built heritage through the machinic gaze

Built heritage has been both subject and product of a gaze that has been sustained through moments of colonial fixation on ruins and monuments, technocratic examination and representation, and fetishisation by a global tourist industry. We argue that the recent proliferation of machine learning and vision technologies create new scopic regimes for heritage: storing and retrieving existing images from vast digital archives, and further imparting their own distortions upon this gaze. We introduce the term ‘ machinic gaze’ to conceptualise the reconfiguration of heritage representation via artificial intelligence (AI) models. To explore how this gaze reframes heritage, we deploy an image-text-image pipeline that reads, interprets, and resynthesizes images of several UNESCO World Heritage Sites. Employing two concepts from media studies— heteroscopia and anamorphosis—we describe the reoriented perspective that machine vision systems introduce. We propose that the machinic gaze highlights the artifice of the human gaze and its underlying assumptions and practices that combine to form established notions of heritage.

Reconstructing human gaze behavior from EEG using inverse reinforcement learning

Decoding eye movements from non-invasive electroencephalography (EEG) data is a challenging yet vital task for both scientific and practical purposes, especially for identifying neurodegenerative disorders like Alzheimer’s disease (AD). Our research tackles this complexity by adapting inverse reinforcement learning (IRL), a machine learning method, to infer decision-making strategies from observed behaviors. We implement this to understand the processes driving eye direction and movements during diverse cognitive tasks, providing new insights into this field. Our paper begins with a detailed description of the procedures for collecting and preprocessing EEG data related to gaze behavior. We then elaborate on the development of an IRL framework designed to predict the spatial and temporal dynamics of eye movements (scanpaths) in participants engaged in cognitive tasks of varying complexity. Our model is tailored to accommodate the complexities inherent in neural signals and the stochastic nature of human gaze patterns. Our research findings underscore IRL’s effectiveness in precisely forecasting gaze patterns based on a combination of EEG and image data. The correlation between the model’s predictions and the actual gaze behavior observed in controlled experiments reinforces the utility of IRL in cognitive neuroscience research. Notably, our IRL-EEG models demonstrated superior performance, especially in more complex cognitive tasks. We further delve into the implications of our results for enhancing the understanding of neural mechanisms that govern gaze behavior.