Eye Movement Classification Research Articles

Combining EEG and eye-tracking in virtual reality: Obtaining fixation-onset event-related potentials and event-related spectral perturbations.

Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.

Viewing preferences of ASD children on paintings

The eye movement patterns of children with autism spectrum disorder (ASD) based on high-precision, high-sampling-rate professional eye trackers have been widely studied. Still, the equipment used in these studies is expensive and requires skilled operators, and the stimuli are focused on pictures or videos created out of ASD group. We utilized a previously developed eye-tracking device using a tablet, and the double-column paintings with one column from children with ASD, and the other from typically developing (TD) children as stimuli, to investigate the preference of ASD children for the paintings created within their group. This study collected eye movement data from 82 children with ASD and 102 TD children, and an adaptive eye movement classification algorithm was applied to the data aligned by the sampling rate, followed by feature extraction and statistical analysis from the aspects of time, frequency, range, and clustering. Statistical tests indicate that apart from displaying more pronounced non-compliance during the experiment, resulting in a higher data loss rate, children with ASD did not show significant preferences in viewing the two types of paintings compared to TD children. Therefore, we tend to believe that there is no significant difference in preference for ASD and TD paintings showcasing as diptych from the two groups of children using our eye tracking device, and their feature values indicate that they do not have a viewing preference for the paintings.

Cepstral Analysis-Based Artifact Detection, Recognition, and Removal for Prefrontal EEG

This paper proposes to use cepstrum for artifact detection, recognition and removal in prefrontal EEG. This work focuses on the artifact caused by eye movement. A database containing artifact-free EEG and eye movement contaminated EEG from different subjects is established. A cepstral analysis-based feature extraction with support vector machine (SVM) based classifier is designed to identify the artifacts from the target EEG signals. The proposed method achieves an accuracy of 99.62% on the artifact detection task and a 82.79% accuracy on the 6-category eye movement classification task. A statistical value-based artifact removal method is proposed and evaluated on a public EEG database, where an accuracy improvement of 3.46% is obtained on the 3-category emotion classification task. In order to make a confident decision of each 5s EEG segment, the algorithm requires only 0.66M multiplication operations. Compared to the state-of-the-art approaches in artifact detection and removal, the proposed method features higher detection accuracy and lower computational cost, which makes it a more suitable solution to be integrated into a real-time and artifact robust Brain-Machine Interface (BMI).

What can entropy metrics tell us about the characteristics of ocular fixation trajectories?

In this study, we provide a detailed analysis of entropy measures calculated for fixation eye movement trajectories from the three different datasets. We employed six key metrics (Fuzzy, Increment, Sample, Gridded Distribution, Phase, and Spectral Entropies). We calculate these six metrics on three sets of fixations: (1) fixations from the GazeCom dataset, (2) fixations from what we refer to as the "Lund" dataset, and (3) fixations from our own research laboratory ("OK Lab" dataset). For each entropy measure, for each dataset, we closely examined the 36 fixations with the highest entropy and the 36 fixations with the lowest entropy. From this, it was clear that the nature of the information from our entropy metrics depended on which dataset was evaluated. These entropy metrics found various types of misclassified fixations in the GazeCom dataset. Two entropy metrics also detected fixation with substantial linear drift. For the Lund dataset, the only finding was that low spectral entropy was associated with what we call "bumpy" fixations. These are fixations with low-frequency oscillations. For the OK Lab dataset, three entropies found fixations with high-frequency noise which probably represent ocular microtremor. In this dataset, one entropy found fixations with linear drift. The between-dataset results are discussed in terms of the number of fixations in each dataset, the different eye movement stimuli employed, and the method of eye movement classification.

An Eye Movement Classification Method based on Cascade Forest.

Eye tracking technology has become increasingly important in scientific research and practical applications. In the field of eye tracking research, analysis of eye movement data is crucial, particularly for classifying raw eye movement data into eye movement events. Current classification methods exhibit considerable variation in adaptability across different participants, and it is necessary to address the issues of class imbalance and data scarcity in eye movement classification. In the current study, we introduce a novel eye movement classification method based on cascade forest (EMCCF), which comprises two modules: (1) a feature extraction module that employs a multi-scale time window method to extract features from raw eye movement data; (2) a classification module that innovatively employs a layered ensemble architecture, integrating the cascade forest structure with ensemble learning principles, specifically for eye movement classification. Consequently, EMCCF not only enhanced the accuracy and efficiency of eye movement classification but also represents an advancement in applying ensemble learning techniques within this domain. Furthermore, experimental results indicated that EMCCF outperformed existing deep learning-based classification models in several metrics and demonstrated robust performance across different datasets and participants.

Residual self-calibrated network with multi-scale channel attention for accurate EOG-based eye movement classification.

Recently, Electrooculography-based Human-Computer Interaction (EOG-HCI) technology has gained widespread attention in industrial areas, including assistive robots, augmented reality in gaming, etc. However, as the fundamental step of EOG-HCI, accurate eye movement classification (EMC) still faces a significant challenge, where their constraints in extracting discriminative features limit the performance of most existing works. To address this issue, a Residual Self-Calibrated Network with Multi-Scale Channel Attention (RSCA), focusing on efficient feature extraction and enhancement is proposed. The RSCA network first employs three self-calibrated convolution blocks within a hierarchical residual framework to fully extract the discriminative multi-scale features. Then, a multi-scale channel attention module adaptively weights the learned features to screen out the discriminative representation by aggregating the multi-scale context information along the channel dimension, thus further boosting the performance. Comprehensive experiments were performed using 5 public datasets and 7 prevailing methods for comparative validation. The results confirm that the RSCA network outperforms all other methods significantly, establishing a state-of-the-art benchmark for EOG-based EMC. Furthermore, thorough ablation analyses confirm the effectiveness of the employed modules within the RSCA network, providing valuable insights for the design of EOG-based deep models.

GAPses: Versatile smart glasses for comfortable and fully-dry acquisition and parallel ultra-low-power processing of EEG and EOG.

Recent advancements in head-mounted wearable technology are revolutionizing the field of biopotential measurement, but the integration of these technologies into practical, user-friendly devices remains challenging due to issues with design intrusiveness, comfort, reliability, and data privacy. To address these challenges, this paper presents GAPSES, a novel smart glasses platform designed for unobtrusive, comfortable, and secure acquisition and processing of electroencephalography (EEG) and electrooculography (EOG) signals.We introduce a direct electrode-electronics interface within a sleek frame design, with custom fully dry soft electrodes to enhance comfort for long wear. The fully assembled glasses, including electronics, weigh 40 g and have a compact size of 160 mm × 145 mm. An integrated parallel ultra-low-power RISC-V processor (GAP9, Greenwaves Technologies) processes data at the edge, thereby eliminating the need for continuous data streaming through a wireless link, enhancing privacy, and increasing system reliability in adverse channel conditions. We demonstrate the broad applicability of the designed prototype through validation in a number of EEG-based interaction tasks, including alpha waves, steady-state visual evoked potential analysis, and motor movement classification. Furthermore, we demonstrate an EEG-based biometric subject recognition task, where we reach a sensitivity and specificity of 98.87% and 99.86% respectively, with only 8 EEG channels and an energy consumption per inference on the edge as low as 121 μJ. Moreover, in an EOG-based eye movement classification task, we reach an accuracy of 96.68% on 11 classes, resulting in an information transfer rate of 94.78 bit/min, which can be further increased to 161.43 bit/min by reducing the accuracy to 81.43%. The deployed implementation has an energy consumption of 40 μJ per inference and a total system power of only 12.4 mW, of which only 1.61% is used for classification, allowing for continuous operation of more than 22 h with a small 75 mAh battery.

Research on Eye Movement Classification Based on the Deep Learning Model

To improve the recognition accuracy of eye movement direction based on EOGM, the Emotiv Epoc, and EEG acquisition headset was used to collect the EEG signals of the subjects in real time. After the 14-channel signals were filtered by wavelet, the short-time energy calculation method was used to select the two channels most sensitive to eye movement. The long and short-term memory network (LSTM) was then used to classify the eye movement signals. On the same data set, the experimental results show that the classification rates in the upper, lower, left, and right directions are up to 91%, 90%, 83%, and 75%, respectively, and the average classification rate is up to 84%. Compared with the existing classification methods, the classification accuracy of the proposed method is higher and the implementation process of the classification algorithm is simpler, which further verifies the feasibility and effectiveness of using EEG to identify the direction of eye movement.

Automated Eye Movement Classification Based on EMG of EOM Signals Using FBSE-EWT Technique

The accurate automated eye movement classification is gaining importance in the field of human–computer interaction (HCI). The present article aims at the classification of six types of eye movements from electromyogram (EMG) of extraocular muscles (EOM) signals using the Fourier–Bessel series expansion-based empirical wavelet transform (FBSE-EWT) with time and frequency-domain (TAFD) features. The FBSE-EWT of EMG signals results in Fourier–Bessel intrinsic mode functions (FBIMFs), which correspond to the frequency contents in the signal. A hybrid approach is used to select the prominent FBIMFs followed by the statistical and signal complexity-based feature extraction. Furthermore, metaheuristic optimization algorithms are employed to reduce the feature space dimension. The discrimination ability of the reduced feature set is verified by Kruskal–Wallis statistical test. Multiclass support vector machine (MSVM) has been employed for classification. First, the classification has been performed with TAFD features followed by the combination of TAFD and FBSE-EWT-based reduced feature set. The combination of TAFD and FBSE-EWT-based feature set has provided good classification performance. This study demonstrates the efficacy of FBSE-EWT and subsequent metaheuristic feature selection algorithms in classifying the eye movements from EMG of EOM signals. The combination of TAFD and the selected features through salp swarm optimization algorithm has provided maximum classification accuracy of 98.91% with MSVM employing Gaussian and radial basis function kernels. Thus, the proposed approach has the potential to be used in HCI applications involving biomedical signals.

A Wireless Electrooculogram (EOG) Wearable Using Conductive Fiber Electrode

Electrooculography (EOG) is a technique for detecting electrical signals from the extra-ocular muscles. The EOG is a precise method for quantifying eye movements, including drowsiness-induced eye closure, and is also a promising technology for its potential use as a contributing mechanism for brain–computer interface applications. Despite the fact that EOG signals change as humans move their eyes, it is still difficult to monitor eye movement patterns in natural behaviors, such as everyday activity. Wearable convenience is essential for obtaining EOG signals while moving freely. This paper proposes the development and use of semi-dry electrodes with low impedance and excellent wearability, as well as a small, portable device with wireless communication capabilities, to increase the likelihood of use in real-life scenarios. The semi-dry electrode produced by the electrospinning technique had an impedance that was 3.5 times lower than that of the existing dry electrode and demonstrated low impedance drift even after long-term use. Furthermore, three steps of eye motion separation were performed using a signal obtained from the wearable device. It was confirmed that the classification of eye movements was at a meaningful level.

Comparison of LSTM And SVM For Classification of Eye Movements in EOG Signals

People with disabilities suffer from inability to communicate with their surroundings, so Human-Computer Interaction (HCI) technologies are used to have a means of communication for people with disabilities with their surroundings. HCI is an emerging technology in the disciplines of Artificial Intelligence and Biomedical Engineering. To power an external device, HCI technology uses several basic signals such as ECG, EMG, and EEG. Electrooculography (EOG) is a technique for measuring the potential difference between the cornea and the retina located between the front and back of the human eye, and the main application of EOG is to determine the directions of different eye movements. This study aims to assess eye movement for communication by persons with disabilities using electrocardiogram (EOG) data. In this study, the Supporting Vector Machine (SVM) and Long- Short term memory (LSTM) classification techniques was used and two types of features (statistical and time domain features) were used. Classification accuracy was 90.7% and 93.9% when using SVM with statistical domain and time domain features, respectively ,whereas Classification accuracy was 90.1% when using LSTM .

A review on voluntary or involuntary eye movement classification methods based on electro-oculogram and their applications

The eye-computer interaction technology based on electro-oculogram provides the users with a convenient way to control the device, which has great social significance. However, the eye-computer interaction is often disturbed by the involuntary eye movements, resulting in misjudgment, affecting the users' experience, and even causing danger in severe cases. Therefore, this paper starts from the basic concepts and principles of eye-computer interaction, sorts out the current mainstream classification methods of voluntary/involuntary eye movement, and analyzes the characteristics of each technology. The performance analysis is carried out in combination with specific application scenarios, and the problems to be solved are further summarized, which are expected to provide research references for researchers in related fields.

An Overview of Classification of Electrooculography (EOG) Signals by Machine Learning Methods

The distribution of the studies conducted between 2011-2021 in the fields of (Electrooculography) EOG and eye movements, EOG and wheelchair, EOG and eye angle, EOG and sleep state, EOG and mood estimation and EOG and game application was determined according to years, and the most cited studies were examined and presented. The study areas are listed as Eye Movement Classification, Wheelchair, Sleep state, Eye Angle, Mood State and Game Applications from the most to the least number of articles. When we examine in terms of the number of citations, they are listed as Sleeping state, Eye Movement Classification, Wheelchair, Eye Angle, Mood State and Game Applications, from the most to the least. In these studies, it has been tried to make the lives of people who have become disabled in various ways better by using the brain-computer interface with machine learning.

Factors affecting inter-rater agreement in human classification of eye movements: a comparison of three datasets.

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters.

Validation of Somno-Art Software, a novel approach of sleep staging, compared with polysomnography in disturbed sleep profiles.

Integrated analysis of heart rate (electrocardiogram [ECG]) and body movements (actimetry) during sleep in healthy subjects have previously been shown to generate similar evaluation of sleep architecture and continuity with Somno-Art Software compared to polysomnography (PSG), the gold standard. However, the performance of this new approach of sleep staging has not yet been evaluated on patients with disturbed sleep. Sleep staging from 458 sleep recordings from multiple studies comprising healthy and patient population (obstructive sleep apnea [OSA], insomnia, major depressive disorder [MDD]) was obtained from PSG visual scoring using the American Academy of Sleep Medicine rules and from Somno-Art Software analysis on synchronized ECG and actimetry. Inter-rater reliability (IRR), evaluated with 95% absolute agreement intra-class correlation coefficient, was rated as "excellent" (ICCAAAvg95% ≥ 0.75) or "good" (ICCAAAvg95% ≥ 0.60) for all sleep parameters assessed, except non-REM (NREM) and N3 sleep in healthy participants (ICCAAAvg95% = 0.43, ICCAAAvg95% = 0.56) and N3 sleep in OSA patients (ICCAAAvg95% = 0.59) rated as "fair" IRR. Overall sensitivity, specificity, accuracy, and Cohen's kappa coefficient of agreement (κ) on the entire sample were respectively of 93.3%, 69.5%, 87.8%, and 0.65 for wake/sleep classification and accuracy and κ were of 68.5% and 0.55 for W/N1+N2/N3/rapid eye movement (REM) classification. These performances were similar in healthy and patient population. The present results suggest that Somno-Art can be a valid sleep-staging tool in both healthy subjects and patients with OSA, insomnia, or MDD. It could complement existing non-attended techniques measuring sleep-related breathing patterns or be a useful alternative to laboratory-based PSG when this latter is not available.

Eye movement patterns and reading ability in children.

Eye movements are integral to the reading process. This study characterised the eye movement patterns of children differentiated by their reading ability, while completing a saccadic eye movement test with irregular target spacing (Development Eye Movement (DEM) test) using a novel eye movement classification algorithm. Participants included 196 Grade 2 Australian schoolchildren (mean age: 7.9±0.3years) who completed a computerised version of the DEM test, while their eye movements were recorded (Tobii TX300 eye-tracker). Children also completed a standardised reading comprehension test, which categorised them into below average and average or above reading ability groups. A novel eye movement classification algorithm was developed that considered the vertical and horizontal eye movements of each child. Compared to children with average or above reading ability, the below average reading group displayed poor vertical eye movement control, demonstrated by a significantly greater proportion of interline eye movements (vertical eye movements away from the current line) (p<0.001). Differences in horizontal eye movements were also observed, with below average readers demonstrating a smaller proportion of expected forward saccades (p<0.001) (within-line forward saccades with horizontal amplitude between the minimum and maximum horizontal spacing between digits), while this group also displayed longer fixation durations (p=0.001). The below average reading group demonstrated significantly poorer results on all standard DEM metrics than the average or above reading ability group: horizontal subtest time (p<0.001), vertical subtest time (p=0.004) and ratio (p=0.004). Children exhibiting below average reading ability were poorer at maintaining control of vertical (interline), as well as horizontal, eye movements compared to children with average or above reading ability. Future studies should explore the mechanisms underlying these differences, particularly in vertical eye movements, given that reading paragraphs (involving multiple lines of text) requires accurate eye movements in both the vertical and horizontal direction.

ANFIS fusion algorithm for eye movement recognition via soft multi-functional electronic skin

Eye movement detection has attracted increasing attention in the fields of safety driving, eye motion tracking, psychological assessment and telemedicine. Soft multi-functional electronic skin (SMFES) is designed to collect electrooculogram (EOG), skin temperature and sweat signals simultaneously for eye movement detection. Serpentine structure is adopted to ensure the stretchability of SMFES for satisfying large deformation (>30%) of the soft skin surface. The paper demonstrates that EOG, skin temperature and sweat signals are successfully collected under different eye movements. The feature data from EOG, skin temperature and sweat signals are extracted with different eye movements, and the principal component analysis (PCA) method is adopted to reduce the dimensionality of the feature space. The paper also proposes an intelligent data fusion algorithm for eye movement classification whose input vector is represented by the first three principal components. Adaptive neuro fuzzy inference system (ANFIS) is built to classify and recognize the eye movements (Up, Down, Left, and Right). Furthermore, experiments have demonstrated that ANFIS algorithm achieves 90% recognition accuracy of such eye movements. This work demonstrates that SMFES integrated with data fusion algorithm can successfully solve the eye movement tracking problem, with significant impact in safety driving and wearable electronics.

Classification of Eye Movement and Its Application in Driving Based on a Refined Pre-Processing and Machine Learning Algorithm

The eyes are the first channel used by humans to obtain various types of visual information from the outside world and, especially when driving, 80-90% of information is received through the eyes. Eye movement behaviors are generally divided into six types, but attention is often paid to fixation, saccade, and smooth pursuit. Due to their importance, it is essential to classify eye movement behaviors accurately. The classification of eye movements should be a complete process, including the three steps of pre-processing, classification, and post-processing. However, it is very uncommon for all of these steps to be included in the eye-tracking literature when eye movement classification is discussed. Therefore, first, this paper proposes a refined eye movement data pre-processing framework and an improved method consisting of three steps is introduced. Second, an eye movement classification algorithm based on an improved decision tree that is independent of the threshold setting and application environment is proposed, and a post-processing consisting of merging adjacent fixations and discarding short fixations is described. Finally, the application of the classified eye movement behavior in the driving field is described, including the estimation of preview time using fixation and the estimation of time-to-collision using smooth pursuit. Two important results are obtained in this paper. One concerns the classification accuracy of eye movement behavior, the F1-scores of fixation, saccade, and smooth pursuit being respectively 92.63%, 93.46%, and 65.29%, which are higher than the scores of other algorithms. The other relates to the application to driving. On the one hand, the preview time calculated by fixation is mostly distributed around 1-6s, which is closer to reality than the traditional setting of 1s. At the same time, the regression relationship between the preview time and the road turning radius is also quantitatively analyzed and their regression function is obtained. On the other hand, the average estimated error of time-to-collision used by smooth pursuit is 7.37%. These results can play an important role in the development of ADAS and the improvement of traffic safety.

Two hours in Hollywood: A manually annotated ground truth data set of eye movements during movie clip watching.

In this short article we present our manual annotation of the eye movement events in a subset of the large-scale eye tracking data set Hollywood2. Our labels include fixations, saccades, and smooth pursuits, as well as a noise event type (the latter representing either blinks, loss of tracking, or physically implausible signals). In order to achieve more consistent annotations, the gaze samples were labelled by a novice rater based on rudimentary algorithmic suggestions, and subsequently corrected by an expert rater. Overall, we annotated eye movement events in the recordings corresponding to 50 randomly selected test set clips and 6 training set clips from Hollywood2, which were viewed by 16 observers and amount to a total of approximately 130 minutes of gaze data. In these labels, 62.4% of the samples were attributed to fixations, 9.1% – to saccades, and, notably, 24.2% – to pursuit (the remainder marked as noise). After evaluation of 15 published eye movement classification algorithms on our newly collected annotated data set, we found that the most recent algorithms perform very well on average, and even reach human-level labelling quality for fixations and saccades, but all have a much larger room for improvement when it comes to smooth pursuit classification. The data set is made available at https://gin.g-node.org/ioannis.agtzidis/hollywood2_em.

Smooth Pursuit And Saccadic Eye Movements Following Years Of Contact Collision Sports: A Pilot Study

Repetitive head impacts (RHI) are the result of a blow to the head that does not elicit clinical signs or symptoms of a concussion. Recent evidence suggests that RHI from a single season of collegiate football can lead to a reduction in midbrain white matter integrity. The midbrain carries projecting fibers to the trochlear and oculomotor nerves, which if damaged may impair oculomotor control. PURPOSE: The purpose of this study was to evaluate oculomotor function following multiple years of Division I contact sports during a dynamic visual acuity (DVA) task. METHODS: Two NCAA Division I football defensive backs with no diagnosed concussion history (a first-year freshmen [F1; age=18 years], senior [S1; age=21 years]) and a healthy control (CON; age=23 years), all with lower than 20/20 vision, completed a DVA task at pre-season. For the DVA task (optotype spatial range=1.0 to -0.3), participants were asked to complete 60 randomized trials of smooth pursuit (30°/s) and saccades (150°/s). Participants head were stabilized in a chin rest at a distance of 154cm away from the 26° visual field monitor (165Hz, 2560 x 1440 pixel resolution, 300 cd/m2 luminance) while wearing a head-mounted binocular video oculography eye tracker (Eyelink SR research, 500 Hz, Ottawa, CN). Using a 2-up-1-down staircase method, participants tracked a Landolt-C ring that moved across the screen (horizontally left to right) where the size of the gap in the C along with the orientation (left, right, up or down) adjusted based on the correct/incorrect responses during both smooth pursuit and saccadic trials. Smooth pursuit eye movement (SPEM) velocity gain and saccadic peak velocity were calculated using ternary eye movement classification from the transformed spherical coordinates via a custom MATLAB code (MATLAB 2019a, Natick, MA, USA). No statistical analysis were performed given the single-subject design. RESULTS: SPEMs gain is lower for S1 (0.88) when compared to F1 (0.82) and CON (0.92). Similarly, during the saccadic trials, S1 had slower average saccadic peak velocity (S1=279.75°/s; F1 =392.34°/s; CON=491.96°/s). CONCLUSIONS: These results may indicate that engaging in contact collision sport for 2+ years at the Division 1 level may result in less accurate (lower SPEMs gain) and slower saccadic eye movements. Supported by NIH P20GM103650