Gaze-independent Brain-computer Interface Research Articles

Correcting for ERP latency jitter improves gaze-independent BCI decoding

Objective. Patients suffering from heavy paralysis or Locked-in-Syndrome can regain communication using a Brain–Computer Interface (BCI). Visual event-related potential (ERP) based BCI paradigms exploit visuospatial attention (VSA) to targets laid out on a screen. However, performance drops if the user does not direct their eye gaze at the intended target, harming the utility of this class of BCIs for patients suffering from eye motor deficits. We aim to create an ERP decoder that is less dependent on eye gaze. Approach. ERP component latency jitter plays a role in covert visuospatial attention (VSA) decoding. We introduce a novel decoder which compensates for these latency effects, termed Woody Classifier-based Latency Estimation (WCBLE). We carried out a BCI experiment recording ERP data in overt and covert visuospatial attention (VSA), and introduce a novel special case of covert VSA termed split VSA, simulating the experience of patients with severely impaired eye motor control. We evaluate WCBLE on this dataset and the BNCI2014-009 dataset, within and across VSA conditions to study the dependency on eye gaze and the variation thereof during the experiment. Main results. WCBLE outperforms state-of-the-art methods in the VSA conditions of interest in gaze-independent decoding, without reducing overt VSA performance. Results from across-condition evaluation show that WCBLE is more robust to varying VSA conditions throughout a BCI operation session. Significance. Together, these results point towards a pathway to achieving gaze independence through suited ERP decoding. Our proposed gaze-independent solution enhances decoding performance in those cases where performing overt VSA is not possible.

Spatial Decoding for Gaze Independent Brain-Computer Interface Based on Covert Visual Attention Shift Using Electroencephalography.

The gaze-independent brain-computer interface (BCI) device is used to re-establish interaction for individuals who have abnormal eye movement. It may be possible to control the BCI by shifting your attention spatially. However, spatial attention is rarely employed to increase the effectiveness of target detection and is typically used to provide a simple "yes" or "no" response to the target recognition inquiry. To improve the effectiveness of detecting target, it is crucial to take advantage of the possible advantages of spatial attention. N2-posterior-contralateral (N2pc) component reflects correlates of visual spatial attention and is used to determine target position. In this study, a long-short-term memory (LSTM) network is used to answer "yes/no" questions by decoding covert spatial attention based on N2pc characteristics using EEG signals. The proposed LSTM-based model's average decoding accuracy is 92.79%. The target detection efficiency was successfully increased by about 4% when compared to conventional machine learning algorithms. The proposed model is tested on the independent dataset to validate its performance. The results of this work show that N2pc characteristics can be employed in gaze-independent BCIs for tracking covert attention shifts, which may help persons with poor eye mobility to connect with their environment.

Brain-computer interface with rapid serial multimodal presentation using artificial facial images and voice.

Electroencephalography (EEG) signals elicited by multimodal stimuli can drive brain-computer interfaces (BCIs), and research has demonstrated that visual and auditory stimuli can be employed simultaneously to improve BCI performance. However, no studies have investigated the effect of multimodal stimuli in rapid serial visual presentation (RSVP) BCIs. The present study proposed a rapid serial multimodal presentation (RSMP) BCI that incorporates artificial facial images and artificial voice stimuli. To clarify the effect of audiovisual stimuli on the RSMP BCI, scrambled images and masked sounds were applied instead of visual and auditory stimuli, respectively. The findings indicated that the audiovisual stimuli improved performance of the RSMP BCI, and that P300 at Pz contributed to classification accuracy. Online accuracy of the BCI reached 85.7±11.5%. Taken together, these findings may aid in the development of better gaze-independent BCI systems.

Impact of Stimulus Features on the Performance of a Gaze-Independent Brain-Computer Interface Based on Covert Spatial Attention Shifts.

Regaining communication abilities in patients who are unable to speak or move is one of the main goals in decoding brain waves for brain-computer interface (BCI) control. Many BCI approaches designed for communication rely on attention to visual stimuli, commonly applying an oddball paradigm, and require both eye movements and adequate visual acuity. These abilities may, however, be absent in patients who depend on BCI communication. We have therefore developed a response-based communication BCI, which is independent of gaze shifts but utilizes covert shifts of attention to the left or right visual field. We recorded the electroencephalogram (EEG) from 29 channels and coregistered the vertical and horizontal electrooculogram. Data-driven decoding of small attention-based differences between the hemispheres, also known as N2pc, was performed using 14 posterior channels, which are expected to reflect correlates of visual spatial attention. Eighteen healthy participants responded to 120 statements by covertly directing attention to one of two colored symbols (green and red crosses for “yes” and “no,” respectively), presented in the user’s left and right visual field, respectively, while maintaining central gaze fixation. On average across participants, 88.5% (std: 7.8%) of responses were correctly decoded online. In order to investigate the potential influence of stimulus features on accuracy, we presented the symbols with different visual angles, by altering symbol size and eccentricity. The offline analysis revealed that stimulus features have a minimal impact on the controllability of the BCI. Hence, we show with our novel approach that spatial attention to a colored symbol is a robust method with which to control a BCI, which has the potential to support severely paralyzed people with impaired eye movements and low visual acuity in communicating with their environment.

Decoding the covert shift of spatial attention from electroencephalographic signals permits reliable control of a brain-computer interface

Objective. One of the main goals of brain-computer interfaces (BCI) is to restore communication abilities in patients. BCIs often use event-related potentials (ERPs) like the P300 which signals the presence of a target in a stream of stimuli. The P300 and related approaches, however, are inherently limited, as they require many stimulus presentations to obtain a usable control signal. Many approaches depend on gaze direction to focus the target, which is also not a viable approach in many cases, because eye movements might be impaired in potential users. Here we report on a BCI that avoids both shortcomings by decoding spatial target information, independent of gaze shifts. Approach. We present a new method to decode from the electroencephalogram (EEG) covert shifts of attention to one out of four targets simultaneously presented in the left and right visual field. The task is designed to evoke the N2pc component—a hemisphere lateralized response, elicited over the occipital scalp contralateral to the attended target. The decoding approach involves decoding of the N2pc based on data-driven estimation of spatial filters and a correlation measure. Main results. Despite variability of decoding performance across subjects, 22 out of 24 subjects performed well above chance level. Six subjects even exceeded 80% (cross-validated: 89%) correct predictions in a four-class discrimination task. Hence, the single-trial N2pc proves to be a component that allows for reliable BCI control. An offline analysis of the EEG data with respect to their dependence on stimulation time and number of classes demonstrates that the present method is also a workable approach for two-class tasks. Significance. Our method extends the range of strategies for gaze-independent BCI control. The proposed decoding approach has the potential to be efficient in similar applications intended to decode ERPs.

A gaze-independent audiovisual brain-computer Interface for detecting awareness of patients with disorders of consciousness

BackgroundCurrently, it is challenging to detect the awareness of patients who suffer disorders of consciousness (DOC). Brain-computer interfaces (BCIs), which do not depend on the behavioral response of patients, may serve for detecting the awareness in patients with DOC. However, we must develop effective BCIs for these patients because their ability to use BCIs does not as good as healthy users.MethodsBecause patients with DOC generally do not exhibit eye movements, a gaze-independent audiovisual BCI is put forward in the study where semantically congruent and incongruent audiovisual number stimuli were sequentially presented to evoke event-related potentials (ERPs). Subjects were required to pay attention to congruent audiovisual stimuli (target) and ignore the incongruent audiovisual stimuli (non-target). The BCI system was evaluated by analyzing online and offline data from 10 healthy subjects followed by being applied to online awareness detection in 8 patients with DOC.ResultsAccording to the results on healthy subjects, the audiovisual BCI system outperformed the corresponding auditory-only and visual-only systems. Multiple ERP components, including the P300, N400 and late positive complex (LPC), were observed using the audiovisual system, strengthening different brain responses to target stimuli and non-target stimuli. The results revealed the abilities of three of eight patients to follow commands and recognize numbers.ConclusionsThis gaze-independent audiovisual BCI system represents a useful auxiliary bedside tool to detect the awareness of patients with DOC.

A sLORETA study for gaze-independent BCI speller.

EEG-based BCI (brain-computer-interface) speller, especially gaze-independent BCI speller, has become a hot topic in recent years. It provides direct spelling device by non-muscular method for people with severe motor impairments and with limited gaze movement. Brain needs to conduct both stimuli-driven and stimuli-related attention in fast presented BCI paradigms for such BCI speller applications. Few researchers studied the mechanism of brain response to such fast presented BCI applications. In this study, we compared the distribution of brain activation in visual, auditory, and audio-visual combined stimuli paradigms using sLORETA (standardized low-resolution brain electromagnetic tomography). Between groups comparisons showed the importance of visual and auditory stimuli in audio-visual combined paradigm. They both contribute to the activation of brain regions, with visual stimuli being the predominate stimuli. Visual stimuli related brain region was mainly located at parietal and occipital lobe, whereas response in frontal-temporal lobes might be caused by auditory stimuli. These regions played an important role in audio-visual bimodal paradigms. These new findings are important for future study of ERP speller as well as the mechanism of fast presented stimuli.

A gaze independent hybrid-BCI based on visual spatial attention

Objective. Brain-computer interfaces (BCI) use measures of brain activity to convey a user’s intent without the need for muscle movement. Hybrid designs, which use multiple measures of brain activity, have been shown to increase the accuracy of BCIs, including those based on EEG signals reflecting covert attention. Our study examined whether incorporating a measure of the P3 response improved the performance of a previously reported attention-based BCI design that incorporates measures of steady-state visual evoked potentials (SSVEP) and alpha band modulations. Approach. Subjects viewed stimuli consisting of two bi-laterally located flashing white boxes on a black background. Streams of letters were presented sequentially within the boxes, in random order. Subjects were cued to attend to one of the boxes without moving their eyes, and they were tasked with counting the number of target-letters that appeared within. P3 components evoked by target appearance, SSVEPs evoked by the flashing boxes, and power in the alpha band are modulated by covert attention, and the modulations can be used to classify trials as left-attended or right-attended. Main Results. We showed that classification accuracy was improved by including a P3 feature along with the SSVEP and alpha features (the inclusion of a P3 feature lead to a 9% increase in accuracy compared to the use of SSVEP and Alpha features alone). We also showed that the design improves the robustness of BCI performance to individual subject differences. Significance. These results demonstrate that incorporating multiple neurophysiological indices of covert attention can improve performance in a gaze-independent BCI.

Fast detection of covert visuospatial attention using hybrid N2pc and SSVEP features

Objective. Detecting the shift of covert visuospatial attention (CVSA) is vital for gaze-independent brain–computer interfaces (BCIs), which might be the only communication approach for severely disabled patients who cannot move their eyes. Although previous studies had demonstrated that it is feasible to use CVSA-related electroencephalography (EEG) features to control a BCI system, the communication speed remains very low. This study aims to improve the speed and accuracy of CVSA detection by fusing EEG features of N2pc and steady-state visual evoked potential (SSVEP). Approach. A new paradigm was designed to code the left and right CVSA with the N2pc and SSVEP features, which were then decoded by a classification strategy based on canonical correlation analysis. Eleven subjects were recruited to perform an offline experiment in this study. Temporal waves, amplitudes, and topographies for brain responses related to N2pc and SSVEP were analyzed. The classification accuracy derived from the hybrid EEG features (SSVEP and N2pc) was compared with those using the single EEG features (SSVEP or N2pc). Main results. The N2pc could be significantly enhanced under certain conditions of SSVEP modulations. The hybrid EEG features achieved significantly higher accuracy than the single features. It obtained an average accuracy of 72.9% by using a data length of 400 ms after the attention shift. Moreover, the average accuracy reached ∼80% (peak values above 90%) when using 2 s long data. Significance. The results indicate that the combination of N2pc and SSVEP is effective for fast detection of CVSA. The proposed method could be a promising approach for implementing a gaze-independent BCI.

Exploring Combinations of Different Color and Facial Expression Stimuli for Gaze-Independent BCIs.

Background: Some studies have proven that a conventional visual brain computer interface (BCI) based on overt attention cannot be used effectively when eye movement control is not possible. To solve this problem, a novel visual-based BCI system based on covert attention and feature attention has been proposed and was called the gaze-independent BCI. Color and shape difference between stimuli and backgrounds have generally been used in examples of gaze-independent BCIs. Recently, a new paradigm based on facial expression changes has been presented, and obtained high performance. However, some facial expressions were so similar that users couldn't tell them apart, especially when they were presented at the same position in a rapid serial visual presentation (RSVP) paradigm. Consequently, the performance of the BCI is reduced.New Method: In this paper, we combined facial expressions and colors to optimize the stimuli presentation in the gaze-independent BCI. This optimized paradigm was called the colored dummy face pattern. It is suggested that different colors and facial expressions could help users to locate the target and evoke larger event-related potentials (ERPs). In order to evaluate the performance of this new paradigm, two other paradigms were presented, called the gray dummy face pattern and the colored ball pattern.Comparison with Existing Method(s): The key point that determined the value of the colored dummy faces stimuli in BCI systems was whether the dummy face stimuli could obtain higher performance than gray faces or colored balls stimuli. Ten healthy participants (seven male, aged 21–26 years, mean 24.5 ± 1.25) participated in our experiment. Online and offline results of four different paradigms were obtained and comparatively analyzed.Results: The results showed that the colored dummy face pattern could evoke higher P300 and N400 ERP amplitudes, compared with the gray dummy face pattern and the colored ball pattern. Online results showed that the colored dummy face pattern had a significant advantage in terms of classification accuracy (p < 0.05) and information transfer rate (p < 0.05) compared to the other two patterns.Conclusions: The stimuli used in the colored dummy face paradigm combined color and facial expressions. This had a significant advantage in terms of the evoked P300 and N400 amplitudes and resulted in high classification accuracies and information transfer rates. It was compared with colored ball and gray dummy face stimuli.

Endogenous Sensory Discrimination and Selection by a Fast Brain Switch for a High Transfer Rate Brain-Computer Interface.

In this study, we present a novel multi-class brain-computer interface (BCI) for communication and control. In this system, the information processing is shared by the algorithm (computer) and the user (human). Specifically, an electro-tactile cycle was presented to the user, providing the choice (class) by delivering timely sensory input. The user discriminated these choices by his/her endogenous sensory ability and selected the desired choice with an intuitive motor task. This selection was detected by a fast brain switch based on real-time detection of movement-related cortical potentials from scalp EEG. We demonstrated the feasibility of such a system with a four-class BCI, yielding a true positive rate of ∼80% and ∼70%, and an information transfer rate of ∼7 bits/min and ∼5 bits/min, for the movement and imagination selection command, respectively. Furthermore, when the system was extended to eight classes, the throughput of the system was improved, demonstrating the capability of accommodating a large number of classes. Combining the endogenous sensory discrimination with the fast brain switch, the proposed system could be an effective, multi-class, gaze-independent BCI system for communication and control applications.

Towards a gaze-independent hybrid-BCI based on SSVEPs, alpha-band modulations and the P300.

In recent years it has been shown to be possible to create a Brain Computer Interface (BCI) using non-invasive electroencephalographic (EEG) measurements of covert visual spatial attention. For example, that both Steady-State Visual Evoked Potentials (SSVEP) and parieto-occipital alpha band activity have been shown to be sensitive to covert attention and this has been exploited to provide simple communication control without the need for any physical movement. In this study, potential improvements in the speed and accuracy of such a BCI are investigated by exploring the possibility of incorporating a P300 task into an SSVEP covert attention paradigm. Should this be possible it would pave the way for a gaze-independent hybrid BCI based on three somewhat independent EEG signals. Within a well-established SSVEP-based attention paradigm we show that it is possible to make a binary classification of covert attention using just the P300 with an average accuracy of 71% across three subjects. We also validate previously published research by showing robust attention effects on the SSVEP and alpha band activity within this paradigm. In future work, it is hoped that by integrating the three signals into a hybrid BCI a significant improvement in performance will be forthcoming leading to an easily usable real time communication device for patients with severe disabilities such as Locked-In Syndrome (LIS).

A New Gaze-BCI-Driven Control of an Upper Limb Exoskeleton for Rehabilitation in Real-World Tasks

This paper proposes a new multimodal architecture for gaze-independent brain-computer interface (BCI)-driven control of a robotic upper limb exoskeleton for stroke rehabilitation to provide active assistance in the execution of reaching tasks in a real setting scenario. At the level of action plan, the patient's intention is decoded by means of an active vision system, through the combination of a Kinect-based vision system, which can online robustly identify and track 3-D objects, and an eye-tracking system for objects selection. At the level of action generation, a BCI is used to control the patient's intention to move his/her own arm, on the basis of brain activity analyzed during motor imagery. The main kinematic parameters of the reaching movement (i.e., speed, acceleration, and jerk) assisted by the robot are modulated by the output of the BCI classifier so that the robot-assisted movement is performed under a continuous control of patient's brain activity. The system was experimentally evaluated in a group of three healthy volunteers and four chronic stroke patients. Experimental results show that all subjects were able to operate the exoskeleton movement by BCI with a classification error rate of 89.4±5.0% in the robot-assisted condition, with no difference of the performance observed in stroke patients compared with healthy subjects. This indicates the high potential of the proposed gaze-BCI-driven robotic assistance for neurorehabilitation of patients with motor impairments after stroke since the earliest phase of recovery.

Eye-gaze independent EEG-based brain–computer interfaces for communication

The present review systematically examines the literature reporting gaze independent interaction modalities in non-invasive brain–computer interfaces (BCIs) for communication. BCIs measure signals related to specific brain activity and translate them into device control signals. This technology can be used to provide users with severe motor disability (e.g. late stage amyotrophic lateral sclerosis (ALS); acquired brain injury) with an assistive device that does not rely on muscular contraction. Most of the studies on BCIs explored mental tasks and paradigms using visual modality. Considering that in ALS patients the oculomotor control can deteriorate and also other potential users could have impaired visual function, tactile and auditory modalities have been investigated over the past years to seek alternative BCI systems which are independent from vision. In addition, various attentional mechanisms, such as covert attention and feature-directed attention, have been investigated to develop gaze independent visual-based BCI paradigms. Three areas of research were considered in the present review: (i) auditory BCIs, (ii) tactile BCIs and (iii) independent visual BCIs. Out of a total of 130 search results, 34 articles were selected on the basis of pre-defined exclusion criteria. Thirteen articles dealt with independent visual BCIs, 15 reported on auditory BCIs and the last six on tactile BCIs, respectively. From the review of the available literature, it can be concluded that a crucial point is represented by the trade-off between BCI systems/paradigms with high accuracy and speed, but highly demanding in terms of attention and memory load, and systems requiring lower cognitive effort but with a limited amount of communicable information. These issues should be considered as priorities to be explored in future studies to meet users’ requirements in a real-life scenario.

A comparison of classification techniques for a gaze-independent P300-based brain–computer interface

This off-line study aims to assess the performance of five classifiers commonly used in the brain–computer interface (BCI) community, when applied to a gaze-independent P300-based BCI. In particular, we compared the results of four linear classifiers and one nonlinear: Fisher's linear discriminant analysis (LDA), stepwise linear discriminant analysis (SWLDA), Bayesian linear discriminant analysis (BLDA), linear support vector machine (LSVM) and Gaussian supported vector machine (GSVM). Moreover, different values for the decimation of the training dataset were tested. The results were evaluated both in terms of accuracy and written symbol rate with the data of 19 healthy subjects. No significant differences among the considered classifiers were found. The optimal decimation factor spanned a range from 3 to 24 (12 to 94 ms long bins). Nevertheless, performance on individually optimized classification parameters is not significantly different from a classification with general parameters (i.e. using an LDA classifier, about 48 ms long bins).

Gaze independent brain–computer speller with covert visual search tasks

Objective Brain–computer interface (BCI) provides a mean of communication for the patients that are severely disabled by neuromuscular diseases. The performance of the classical P300 speller, however, declines noticeably in the gaze fixation condition. The speller paradigm presented in this paper aims to release the gaze dependency at the cost of an extra covert visual search task. Methods Clusters of characters were presented sequentially in the near-central vision as stimulation. Participants fixed their gaze on the center, searched and recognized the target character with covert shift of attention. Random position (RP) and fixed position (FP) presentation modes designed with different searching set size (6 for RP, and ⩽2 for FP) were examined. Results Online sessions using 10 stimulus sequences achieved character accuracies of 94.4% and 96.3% for RP and FP mode, respectively. For offline overall evaluation, the peak written symbol rate (WSR) of 1.38 symbols/min was obtained, with corresponding accuracies of 87.8% (RP) and 84.1% (FP). The P300 waveform of RP mode has evident longer latency and larger amplitude. Electrooculogram (EOG) analysis indicated that the performance was independent of gaze shift. Conclusions The proposed speller could be operated effectively and gaze independently by healthy participants. Significance The proposed gaze independent BCI approach promises reasonable communication capability for the profoundly paralyzed patients with head or ocular motor impairments.