Gaze Angle Research Articles

Classification of Internal and External Distractions in an Educational VR Environment Using Multimodal Features.

Virtual reality (VR) can potentially enhance student engagement and memory retention in the classroom. However, distraction among participants in a VR-based classroom is a significant concern. Several factors, including mind wandering, external noise, stress, etc., can cause students to become internally and/or externally distracted while learning. To detect distractions, single or multi-modal features can be used. A single modality is found to be insufficient to detect both internal and external distractions, mainly because of individual variability. In this work, we investigated multi-modal features: eye tracking and EEG data, to classify the internal and external distractions in an educational VR environment. We set up our educational VR environment and equipped it for multi-modal data collection. We implemented different machine learning (ML) methods, including k-nearest-neighbors (kNN), Random Forest (RF), one-dimensional convolutional neural network - long short-term memory (1 D-CNN-LSTM), and two-dimensional convolutional neural networks (2D-CNN) to classify participants' internal and external distraction states using the multi-modal features. We performed cross-subject, cross-session, and gender-based grouping tests to evaluate our models. We found that the RF classifier achieves the highest accuracy over 83% in the cross-subject test, around 68% to 78% in the cross-session test, and around 90% in the gender-based grouping test compared to other models. SHAP analysis of the extracted features illustrated greater contributions from the occipital and prefrontal regions of the brain, as well as gaze angle, gaze origin, and head rotation features from the eye tracking data.

Fine-grained gaze estimation based on the combination of regression and classification losses

Human gaze is a crucial cue used in various applications such as human-robot interaction, autonomous driving, and virtual reality. Recently, convolution neural network (CNN) approaches have made notable progress in predicting gaze angels. However, estimating accurate gaze direction in-the-wild is still a challenging problem due to the difficulty of obtaining the most crucial gaze information that exists in the eye area which constitutes a small part of the face images. In this paper, we introduce a novel two-branch CNN architecture with a multi-loss approach to estimate gaze angles (pitch and yaw) from face images. Our approach utilizes separate fully connected layers for each gaze angle prediction, allowing explicit learning of discriminative features and emphasizing the distinct information associated with each gaze angle. Moreover, we adopt a multi-loss approach, incorporating both classification and regression losses. This allows for joint optimization of the combined loss for each gaze angle, resulting in improved overall gaze performance. To evaluate our model, we conduct experiments on three popular datasets collected under unconstrained settings: MPIIFaceGaze, Gaze360, and RT-GENE. Our proposed model surpasses current state-of-the-art methods and achieves state-of-the-art performance on all three datasets, showcasing its superior capability in gaze estimation.

Mixed reality guided advancement osteotomies in congenital craniofacial malformations

We describe the use of mixed reality intra-operative guides, in patients with congenital craniofacial malformations. Our experience shows the potential MR has as an adjunct and possible replacement for conventional cutting guides and intra-operative navigation. SummaryThis paper describes our experience with mixed reality (MR) intra-operative guides, in patients with congenital craniofacial malformations.The first case was a patient with bilateral hemifacial microsomia. He underwent bilateral mandibular distraction osteogenesis. Pre-operative virtual planning determined the sites of osteotomy. Standard Tessellation Language (STL) files of mandibular 3D models with osteotomy sites were uploaded onto the HoloLens 2® MR glasses (Microsoft®, Washington, USA). The superimposed hologram denoted the osteotomy line. This was validated with a physical cutting guide.The second case was a patient with Crouzon’s syndrome. A modified Lefort 2 advancement was performed to correct his midfacial deficiency. Pre-operative virtual planning was performed to determine the sites of osteotomies. Superimposed hologram using the Hololens 2® denoted the osteotomy sites. These were validated with a conventional intra-operative navigation system.The advantages of using MR include its immediate availability for use; saving time and costs. MR allows surgeons to maintain continuous line-of-sight within the operative field. A robust registration system is required to anchor the hologram onto the patient’s skull, without variations in hologram position from different angles of gaze. MR has the potential to function as an adjunct and possible replacement for conventional cutting guides and intra-operative navigation.

Primate eye tracking with carbon-nanotube-paper-composite based capacitive sensors and machine learning algorithms

BackgroundAccurate real-time eye tracking is crucial in oculomotor system research. While the scleral search coil system is the gold standard, its implantation procedure and bulkiness pose challenges. Camera-based systems are affected by ambient lighting and require high computational and electric power. New MethodThis study presents a novel eye tracker using proximity capacitive sensors made of carbon-nanotube-paper-composite (CPC). These sensors detect femtofarad-level capacitance changes caused by primate corneal movement during horizontal and vertical eye rotations. Data processing and machine learning algorithms are evaluated to enhance the accuracy of gaze angle prediction. ResultsThe system performance is benchmarked against the scleral coil during smooth pursuits, saccades tracking, and fixations. The eye tracker demonstrates up to 0.97 correlation with the coil in eye tracking and is capable of estimating gaze angle with a median absolute error as low as 0.30°. ComparisonThe capacitive eye tracker demonstrates good consistency and accuracy in comparison to the gold-standard scleral search coil method. ConclusionsThis lightweight, non-invasive capacitive eye tracker offers potential as an alternative to traditional coil and camera-based systems in oculomotor research and vision science.

Bubble-graft coverage after Descemet Membrane Endothelial Keratoplasty depending on gaze angle.

In Descemet Membrane Endothelial Keratoplasty (DMEK) a gas bubble is usually used to attach the graft to the host cornea. In this study, we observed the bubble size and bubble-graft coverage at different gaze angles following DMEK. This observational prospective study analyzed 465 images of patients who underwent an uneventful DMEK. Intraoperatively, the anterior chamber was filled up to 90% of its volume with a 20% Sulfur Hexafluoride (SF6) gas-air mixture. Postoperatively, the bubble was photographed daily in different gaze angles ranging from a supine position (0°) to an upright position (90°) and a slightly inclined position (105°). The primary outcomes were bubble-graft coverage and bubble diameters depending on the gaze angle and time after DMEK. The highest bubble-graft coverage was achieved at a 0° gaze angle at all times of measurement. In the first 48 h after DMEK, the mean bubble-graft coverage was over 85% at a gaze angle between 0° and 45°. Starting 72 h after DMEK, the graft coverage declined at all gaze angles. The graft coverage at a 0° gaze angle was 88.61 ± 10.90% after 96 postoperative hours, while the graft coverage was below 85% at all other gaze angles. Our clinical results provide novel insight into variation in bubble-graft coverage as a function of gaze angle and may be used to aid in patient counselling for appropriate body positioning following DMEK to prevent early graft detachment. Maintaining supine positioning seems to be most advantageous starting 48 h after DMEK.

Social Semiotic Analysis of a Pet Protection Post on Instagram

This study applies Kress and Van Leeuwen's Visual Grammar framework to analyze an Instagram post advocating pet protection. The research focuses on representational, interactive, and compositional metafunctions to understand how visual and textual elements combine to convey urgency and empathy. The representational analysis highlights the dog's isolated portrayal and the use of the pronoun "me" to foster an emotional connection. The interactive analysis examines indirect gaze, close-up shots, and low angles to engage viewers and emphasize their role in pet protection. The compositional analysis reveals the strategic placement of verbal and visual elements, with bold text and contrasting colors enhancing the message's salience. This multimodal approach demonstrates the synergy between visual and textual elements, creating a cohesive and persuasive narrative. The study's findings align with existing literature on social semiotics and Visual Grammar, emphasizing the importance of integrated visual strategies in social media advocacy. The research offers practical insights for designing effective pet protection campaigns, highlighting the role of Visual Grammar in promoting responsible pet ownership and animal welfare. Future research should explore the long-term impact of these visual strategies on viewer behavior and the effectiveness of different semiotic modes across various social media platforms.

Measurement of visual function in infantile nystagmus: a systematic review

Background/aimsRecent work has called into question the ability of visual acuity (VA) to accurately represent changes in visual function in infantile nystagmus (IN). This systematic review investigated factors affecting visual...

A Study on the Development of Driver Behavior Simulation Dummy for the Performance Evaluation of Driver Monitoring System

Driver monitoring system (DMS) was mainly developed to prevent accident risks by analyzing facial movements related to drowsiness and carelessness in real time such as driver’s gaze, blink, and head angle through cameras and warning the driver. Recently, the scope has been expanded to monitor passengers, and it has been linked to safety functions such as neglecting children, empty seats, or controlling airbags on seats with people under safety weight. However, evaluation research for algorithm advancement and performance optimization is relatively insufficient. In addition, the verification system is facing limitations such as personal information protection problems caused by the subject’s face data, errors in reproducing the subject’s drowsy and careless behavior, and differences in behavior according to individual differences. Therefore, as the importance of traffic safety is emphasized, an evaluation tool that can more efficiently and systematically evaluate the performance of DMS is needed. In this study, a driver behavior simulation dummy was developed that can quantitatively control the movement of the driver’s face and upper body. The driver behavior simulation dummy was developed in three stages in the order of function and specification definition, design and manufacture according to specifications, and verification through error tests for each function.

Assessment of gaze direction during head and neck irradiation and dosimetric impact on the retina, macula and papilla in a cohort of 240 patients with paraoptic tumors

In a prospective cohort of 240 paraoptic tumors patients treated with protons, there was 10° inter-individual gaze angle variability (up to 30°). In a random 21-patient subset with initial CTs versus and adaptive CTs, 6 (28.57 %) patients had at least twice a 10°-difference in their gaze angle, with > 5 Gy difference on the retina/macula or papilla in 2/21 (9.52 %) and 1/21 (4.76 %) based on cumulative dose from rescans, respectively.

Notion of Beauty in the Selected Pakistani Advertisements: A Multimodal Critical Discourse Analysis

Beauty advertisement is one of the most potent and successful tools for significantly influencing consumer purchasing behavior as it modifies individuals’ views of beauty, social conventions, and gender roles. From a Multimodal Critical Discourse Analysis standpoint, this study looks at the beauty advertisements in a few chosen social media (Instagram) posts. The study’s goal is to concentrate on how the idea of beauty is used in these Pakistani advertisements to control and influence consumers through both language as well as images. For this particular reason, four ads from different Instagram pages have been chosen via purposive sampling and they have undergone both linguistic and visual analysis. Fairclough’s Three-Dimensional Model (1995) serves as the foundation for the linguistic examination while Kress and van Leeuwen’s Grammar of Visual Design (2006) is applied for the visual analysis of these ads. The nature of this study is qualitative. The findings reveal that advertisers have employed a variety of discursive techniques like surreal representation, scientific evidence, celebrity endorsement, & self-representation etc., and linguistic devices, such as pronouns, catchy phrases, adjectives, repetitions, etc. in addition to visual tactics, e.g., modality, gaze, social distance, salience, and camera angles etc. to influence women by portraying an idealized version of beauty in the chosen advertisements. Additionally, it shows how advertisers—who are essentially powerful people with vested interests—marginalize and restrict the status of women in society to propagate the ideology of beauty just for boosting the sales of their products. Thus, the companies and creators of these commercials employ images and language as a means of controlling Pakistan’s female consumer base by luring them to get on their items. It is anticipated that this study will increase awareness of the usefulness of multimodal critical discourse analysis and open the door for future researchers to investigate the language and pictures used in advertisements for a variety of other products.

Improving cognitive-state analysis from eye gaze with synthetic eye-movement data

Eye movements can be used to analyze a viewer’s cognitive capacities or mental state. Neural networks that process the raw eye-tracking signal can outperform methods that operate on scan paths preprocessed into fixations and saccades. However, the scarcity of such data poses a major challenge. We therefore develop SP-EyeGAN, a neural network that generates synthetic raw eye-tracking data. SP-EyeGAN consists of Generative Adversarial Networks; it produces a sequence of gaze angles indistinguishable from human ocular micro- and macro-movements. We explore the use of these synthetic eye movements for pre-training neural networks using contrastive learning. We find that pre-training on synthetic data does not help for biometric identification, while results are inconclusive for the detection of ADHD and gender classification. However, for the eye movement-based assessment of higher-level cognitive skills such general reading comprehension, text comprehension, and the distinction of native from non-native readers, pre-training on synthetic eye-gaze data improves the models’ performance and even advances the state-of-the-art for reading comprehension. The SP-EyeGAN model, pre-trained on GazeBase, along with the code for developing your own raw eye-tracking machine learning model with contrastive learning, is available at https://github.com/aeye-lab/sp-eyegan.

Model-Based 3D Gaze Estimation Using a TOF Camera.

Among the numerous gaze-estimation methods currently available, appearance-based methods predominantly use RGB images as input and employ convolutional neural networks (CNNs) to detect facial images to regressively obtain gaze angles or gaze points. Model-based methods require high-resolution images to obtain a clear eyeball geometric model. These methods face significant challenges in outdoor environments and practical application scenarios. This paper proposes a model-based gaze-estimation algorithm using a low-resolution 3D TOF camera. This study uses infrared images instead of RGB images as input to overcome the impact of varying illumination intensity in the environment on gaze estimation. We utilized a trained YOLOv8 neural network model to detect eye landmarks in captured facial images. Combined with the depth map from a time-of-flight (TOF) camera, we calculated the 3D coordinates of the canthus points of a single eye of the subject. Based on this, we fitted a 3D geometric model of the eyeball to determine the subject's gaze angle. Experimental validation showed that our method achieved a root mean square error of 6.03° and 4.83° in the horizontal and vertical directions, respectively, for the detection of the subject's gaze angle. We also tested the proposed method in a real car driving environment, achieving stable driver gaze detection at various locations inside the car, such as the dashboard, driver mirror, and the in-vehicle screen.

A Digital Camera-Based Eye Movement Assessment Method for NeuroEye Examination

The ability to perform quantitative and automated neurological assessment could enhance diagnosis and treatment in the pre-hospital setting, such as during telemedicine or emergency medical services (EMS) encounters. Such a tool could be developed by adapting clinically significant information such as symmetry of eye movement or conjugate eye movement. Here we describe a digital camera-based eye tracking method “NeuroGaze” to capture the symmetry of eye movement while performing neurological eye examination. The proposed method was developed based on detecting the center of the pupil for both eyes from a given video and measuring eye conjugacy by transforming the pupil center coordinates to relative gaze. The method was tested on healthy volunteers while performing three neurological eye examinations <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> The NeuroEye dataset is made available at <uri>https://www.kaggle.com/datasets/mahassan8/neuroeye</uri> </fn> . We also compared our proposed approach to state-of-the-art digital camera-based eye-tracking methods and commercial off-the-shelf (COTS) eye trackers. NeuroGaze outperformed digital camera-based eye tracking methods by reporting a mean Spearman rank-order correlation coefficient of 0.86 for the H-test, 0.87 for the Dot-test, and 0.56 for the OKN-Test, and shows similarity in trends for the relative gaze trajectories with a noticeable offset in the scale of the relative gaze angle compared to COTS eye tracker (see Fig. 1). The study demonstrates that by using a pupil-center-based eye-tracking method, a digital camera can measure clinically relevant information regarding eye movement.

Evaluating Fixed Single-Point Parameters When Applied to Vestibular Evoked Myogenic Potentials: The Effect of Single Point and Signal Window.

Several studies have applied a common objective detection algorithm (fixed single point [ Fsp ]) for detection of the vestibular evoked myogenic potential (VEMP). However, fundamental parameters of Fsp , such as establishing the location and duration of a signal window, have not been examined. In addition, Fsp criterion values used for response detection have not been established for cervical VEMPs (cVEMPs) or ocular VEMPs (oVEMPs). The purpose of this article was to investigate the effect of various single points and signal windows on Fsp , as well as determining Fsp criteria to determine response presence for cVEMP and oVEMP in a group of young healthy participants. Twenty young healthy adults under the age of 30 and with no history of hearing or balance concerns were enrolled in the study protocol. Air-conducted cVEMPs and oVEMPs were evoked using 500 Hz tone bursts at 123 dB pSPL recorded at a fixed electromyography activation of 50 µV for cVEMPs and 35° gaze angle for oVEMPs. Responses were analyzed off-line using visual and objective detection. Fsp was applied to cVEMPs and oVEMPs using a range of single points and signal windows. Noise variance was lowest for cVEMPs at the latency of P1, and for oVEMPs noise variance was not significantly different across the single-point latencies. On average, extending the length of the signal window lowered the Fsp value in cVEMPs and oVEMPs. An Fsp value of 2.0 was chosen as the criterion cutoff associated with the 95th percentile during no-response conditions using group data for cVEMPs and oVEMPs, respectively. Fsp values for cVEMPs and oVEMPs were not significantly different from each other. This study established single-point latency and time-window parameters for VEMP-related applications of the Fsp detection algorithm. Fsp criteria values were established for cVEMP and oVEMP. Using these parameters, responses were detected in all participants.

Three-Dimensional View Relationship-Based Context-Aware Emotion Recognition.

Context-aware emotion recognition (CAER) leverages comprehensive scene information, including facial expressions, body postures, and contextual background. However, current studies predominantly rely on facial expressions, body postures, and global contextual features; the interaction between the agents (target individuals) and other objects in the scene is usually absent or incomplete. In this article, a three-dimensional view relationship-based CAER (TDRCer) method is proposed, which comprises two branches: the personal emotional branch (PEB) and the contextual emotional branch (CEB). First, PEB is designed for the extraction of facial expression features and body posture features from the agent. A vision transformer (ViT), pretrained by contrastive learning with a novel loss function combining Euclidean distance and cosine similarity, is applied to enhance the robustness of facial expression features. Meanwhile, the human body contour images extracted by semantic segmentation are fed into another ViT to extract body posture features. Second, CEB is constructed for the extraction of global contextual features and interactive relationships among objects in the scene. The images masked by the agents' bodies are fed into a ViT to extract global contextual features. By leveraging both the gaze angle and depth map, a three-dimensional view graph (3DVG) is constructed to represent the interactive relationships between agents and objects in the scene. Then, a graph convolutional network is employed to extract interactive relationship features from the 3DVG. Finally, the multiplicative fusion strategy is applied to fuse the features of two branches, and the fused features are utilized to classify the emotions. TDRCer achieves an accuracy of 89.90% on the CAER-S dataset and a mean average precision (mAP) of 36.02% on the EMOTIons in context (EMOTIC) dataset. The code can be accessed at https://github.com/mengTender/TDRCer.

Unsupervised Transfer Learning Approach With Adaptive Reweighting and Resampling Strategy for Inter-subject EOG-based Gaze Angle Estimation.

Gaze estimation based on electrooculograms (EOGs) has been widely explored. However, the inter-subject variability of EOGs still leaves a significant challenge for practical applications. It contributes to performance degradation when handling inter-subject issues. In this paper, an unsupervised transfer learning approach with an adaptive reweighting and resampling (ARR) strategy to fully consider individual variability is proposed for EOG-based gaze angle estimation. It allows quantifying domain shifts by leveraging the source-target similarities, reweighting and resampling the source data to retain relevant instances and disregard irrelevant instances during adaptation. Specifically, our proposed methodology first assesses the domain shifts via decomposing transformation matrices, which are estimated between the training subjects (denoted as multi-source domains) and the test subject (denoted as target domain). Then, the multi-domain shifts are assigned as weighted indicators to resample the multi-source domains for model training. Comparative experiments with several prevailing transfer learning methods including CORrelation ALignment (CORAL), Geodesic Flow Kernel (GFK), Joint Distribution Adaptation (JDA), Transfer component analysis (TCA), and Balanced distribution adaption (BDA) using two different normalization processes were conducted on a realistic scenario across 18 subjects. Experimental results demonstrate that the ARR strategy can significantly improve performance (mean absolute error (MAE) reduction: 7.0%, root mean square error (RMSE) reduction: 6.3%), outperforming the prevailing methods. Besides, the impacts of data diversity and data size on ARR strategy are further investigated. It exhibits that data size is more important than data diversity for EOG-based gaze angle estimation, and also presents the benefits of the ARR strategy for dealing with practical scenarios.

Real-time continuous EOG-based gaze angle estimation with baseline drift compensation under non-stationary head conditions

Objective:This work addresses an impractical and unnatural constraint that has been generally enforced in state-of-the-art electrooculography (EOG)-based gaze estimation methods, that of maintaining a stationary head pose and position. Specifically, this work proposes an EOG-based gaze angle (GA) estimation method that accommodates natural variations in the user’s head pose and position. The EOG data collected under non-stationary head conditions, which was used in this work to validate the proposed method, is also being made publicly available. Methods:This work generalises a two-eye verging gaze geometrical model to cater for arbitrary head poses and positions, and also models the dynamics of the vestibulo-ocular reflex (VOR), which refers to the eye-head coordination that normally takes place during gaze shifts under unrestrained head conditions. These methods are validated by incorporating them within a published multiple-model framework for GA estimation. Results:When applied to short EOG data segments, a horizontal and vertical GA estimation error of 1.85 ± 0.51° and 2.19 ± 0.62°, respectively, and an eye movement detection and labelling F-score close to 90% were obtained. These results are comparable to those reported previously under stationary head conditions. Conclusion:This work demonstrates that accurate GA estimation and eye movement detection and labelling can be achieved using EOG signals, even when the user’s head is not stationary. Significance:This work eliminates the need for users to maintain a stationary head pose and position, a common constraint in the field, thus introducing an EOG-based GA estimation method that allows users to move their heads naturally.

A distance robust EOG-based feature for gaze trajectory inference

The vast majority of eye controlled applications rely on a mapping function which relates the estimated gaze angle of the user to a pixel coordinate on the screen. In turn, such systems heavily depend on the user maintaining an unnatural, fixed and known distance from the computer screen in order to ensure proper operation of the designed application. This work has investigated the use of trajectory bearing angles as a distance-robust electrooculography (EOG)-based feature which can be used for gaze trajectory inference. The trajectory bearing angles are extracted directly from EOG data and are shown to be robust to the distance of the user from the screen. Three different EOG-based bearing angle estimation methods are investigated and results of a proof-of-concept Hidden Markov Model (HMM) swipe typing application carried out by ten subjects, show that an average top-five rate of over 80% can be achieved across three different user distances from the screen.

Measuring gaze angle changes to maintain fixation upon a small target during motion: 3D motion tracking versus wearable eye-tracker

Recently we demonstrated how changes in gaze angle can be determined without an eye-tracker. The approach uses 3D motion-capture, to track the viewed target in the head’s reference frame and assumes head or target movement causes a gaze-angle change. This study determined the validity of this “assumed-gaze” method. Participants read information presented on a smartphone whilst walking. Changes in gaze angles were simultaneously assessed with an eye-tracker and our assumed-gaze method. The spatial and temporal agreement of the assumed-gaze approach with the eye-tracker were ∼1 deg and ∼0.02 s, respectively, and spatial congruence indicated the direction of changes in the assumed-gaze angle were in accordance with those determined with the eye tracker for ∼81 % of the time. Findings indicate that when the head is moving and gaze is continually directed to a small target, our assumed-gaze approach can determine changes in gaze angle with comparable precision to a wearable eye-tracker.

Neck and shoulder strains under various head flexing positions while standing and sitting with and without back support for male and female smartphone users

This study recruited 30 young participants (15 men and 15 women) to examine the smartphone usage patterns in three postures (standing, supported sitting, and unsupported sitting) and at five head angle (HA) positions (0°-40°). Cervical erector spinae (CES) and upper trapezius (UTZ) muscle activity, neck flexion (NF), gaze angle (GA), viewing distance (VD), and discomfort scores were collected. Results showed that HA and posture almost affected all responses, while CES muscle activity, NF, and VD differed between sexes. Strain in the neck and shoulder region increased with HA increase. Particularly, when the HA exceeded 20°, the discomfort scores considerably increased. Unsupported sitting should be avoided during smartphone use because of relatively poor responses in all variables. However, both standing and supported sitting have their respective benefits. Sex-related differences were typically observed in the standing position, with women tending to have lower NF but higher CES muscle activity compared with men. Practitioner summary: Although smartphones have become daily necessities, the overall quantitative neck and shoulder strain of using smartphones in different postures is rarely evaluated. We suggest that maintaining the HA within 20° is recommended because of relatively low load on the neck and shoulders. An unsupported sitting should be avoided during smartphone use.