Hybrid Brain-computer Interface Research Articles

A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching

Objective. In order to increase the number of states classified by a brain–computer interface (BCI), we utilized a motor imagery task where subjects imagined both force and speed of hand clenching. Approach. The BCI utilized simultaneously recorded electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) signals. The time-phase-frequency feature was extracted from EEG, whereas the HbD [the difference of oxy-hemoglobin (HbO) and deoxy-hemoglobin (Hb)] feature was used to improve the classification accuracy of fNIRS. The EEG and fNIRS features were combined and optimized using the joint mutual information (JMI) feature selection criterion; then the extracted features were classified with the extreme learning machines (ELMs). Main results. In this study, the averaged classification accuracy of EEG signals achieved by the time-phase-frequency feature improved by 7%, to 18%, more than the single-type feature, and improved by 15% more than common spatial pattern (CSP) feature. The HbD feature of fNIRS signals improved the accuracy by 1%, to 4%, more than Hb, HbO, or HbT (total hemoglobin). The EEG–fNIRS feature for decoding motor imagery of both force and speed of hand clenching achieved an accuracy of 89% ± 2%, and improved the accuracy by 1% to 5% more than the sole EEG or fNIRS feature. Significance. Our novel motor imagery paradigm improves BCI performance by increasing the number of extracted commands. Both the time-phase-frequency and the HbD feature improve the classification accuracy of EEG and fNIRS signals, respectively, and the hybrid EEG–fNIRS technique achieves a higher decoding accuracy for two-class motor imagery, which may provide the framework for future multi-modal online BCI systems.

Simultaneous detection of P300 and steady-state visually evoked potentials for hybrid brain-computer interface.

ObjectiveWe study the feasibility of a hybrid Brain-Computer Interface (BCI) combining simultaneous visual oddball and Steady-State Visually Evoked Potential (SSVEP) paradigms, where both types of stimuli are superimposed on a computer screen. Potentially, such a combination could result in a system being able to operate faster than a purely P300-based BCI and encode more targets than a purely SSVEP-based BCI.ApproachWe analyse the interactions between the brain responses of the two paradigms, and assess the possibility to detect simultaneously the brain activity evoked by both paradigms, in a series of 3 experiments where EEG data are analysed offline.Main ResultsDespite differences in the shape of the P300 response between pure oddball and hybrid condition, we observe that the classification accuracy of this P300 response is not affected by the SSVEP stimulation. We do not observe either any effect of the oddball stimulation on the power of the SSVEP response in the frequency of stimulation. Finally results from the last experiment show the possibility of detecting both types of brain responses simultaneously and suggest not only the feasibility of such hybrid BCI but also a gain over pure oddball- and pure SSVEP-based BCIs in terms of communication rate.

Multi-objective genetic algorithm as channel selection method for P300 and motor imagery data set

As different region of the brain is associated with different mental activity, channel selection is commonly used to enhance the performance of multi-electrode electroencephalography (EEG) system by removing task-irrelevant and redundant channels. Various channel selection methods are successfully implemented in Brain–Computer Interface (BCI) system that uses one type of brain activity by earlier researchers. Upon realizing the limitation of conventional BCI systems, there has been increasing number of hybrid BCI systems. These hybrid systems use combinations of two brain activity patterns to enhance the functionality of a system. In this paper, three multi-objective genetic algorithms (GAs) are proposed to optimize the number of channels selected and system accuracy. The objective of this research is to investigate the optimal tradeoff between the classification accuracy of a BCI system and the number of selected channels. This tradeoff is important because different BCI applications have different priorities; some implementations prefer minimum number of channels while others favor the classification accuracy. The second objective of this research is to investigate the effectiveness of the GAs adopted as a channel selection method for BCI systems based on different brain activity. Three BCI Competition data sets are used to evaluate the performance of the proposed GAs. Non-parametric Friedman test (p-value=0.635) is also conducted and the result reveals that the significant reduction in number of channels does not have significant impact in the classification accuracy on the evaluation data. This confirms the validity of genetic algorithms as a channel selection method for both P300 and motor imagery data.

SBCI-Headset—Wearable and Modular Device for Hybrid Brain-Computer Interface

Severely disabled people, like completely paralyzed persons either with tetraplegia or similar disabilities who cannot use their arms and hands, are often considered as a user group of Brain Computer Interfaces (BCI). In order to achieve high acceptance of the BCI by this user group and their supporters, the BCI system has to be integrated into their support infrastructure. Critical disadvantages of a BCI are the time consuming preparation of the user for the electroencephalography (EEG) measurements and the low information transfer rate of EEG based BCI. These disadvantages become apparent if a BCI is used to control complex devices. In this paper, a hybrid BCI is described that enables research for a Human Machine Interface (HMI) that is optimally adapted to requirements of the user and the tasks to be carried out. The solution is based on the integration of a Steady-state visual evoked potential (SSVEP)-BCI, an Event-related (de)-synchronization (ERD/ERS)-BCI, an eye tracker, an environmental observation camera, and a new EEG head cap for wearing comfort and easy preparation. The design of the new fast multimodal BCI (called sBCI) system is described and first test results, obtained in experiments with six healthy subjects, are presented. The sBCI concept may also become useful for healthy people in cases where a “hands-free” handling of devices is necessary.

Hybrid P300-Based Brain-Computer Interface to Improve Usability for People With Severe Motor Disability: Electromyographic Signals for Error Correction During a Spelling Task

ObjectiveTo evaluate the impact of a hybrid control on usability of a P300-based brain-computer interface (BCI) system that was designed to control an assistive technology software and was integrated with an electromyographic channel for error correction. DesignProof-of-principle study with a convenience sample. SettingNeurologic rehabilitation hospital. ParticipantsParticipants (N=11) in this pilot study included healthy (n=8) and severely motor impaired (n=3) persons. The 3 people with severe motor disability were identified as potential candidates to benefit from the proposed hybrid BCI system for communication and environmental interaction. InterventionsTo eventually investigate the improvement in usability, we compared 2 modalities of BCI system control: a P300-based and a hybrid P300 electromyographic-based mode of control. Main Outcome MeasuresSystem usability was evaluated according to the following outcome measures within 3 domains: (1) effectiveness (overall system accuracy and P300-based BCI accuracy); (2) efficiency (throughput time and users' workload); and (3) satisfaction (users' satisfaction). We also considered the information transfer rate and time for selection. ResultsFindings obtained in healthy participants were in favor of a higher usability of the hybrid control as compared with the nonhybrid. A similar trend was indicated by the observational results gathered from each of the 3 potential end-users. ConclusionsThe proposed hybrid BCI control modality could provide end-users with severe motor disability with an option to exploit some residual muscular activity, which could not be fully reliable for properly controlling an assistive technology device. The findings reported in this pilot study encourage the implementation of a clinical trial involving a large cohort of end-users.

A Hybrid Brain-Computer Interface Based on the Fusion of P300 and SSVEP Scores.

The present study proposes a hybrid brain-computer interface (BCI) with 64 selectable items based on the fusion of P300 and steady-state visually evoked potential (SSVEP) brain signals. With this approach, row/column (RC) P300 and two-step SSVEP paradigms were integrated to create two hybrid paradigms, which we denote as the double RC (DRC) and 4-D spellers. In each hybrid paradigm, the target is simultaneously detected based on both P300 and SSVEP potentials as measured by the electroencephalogram. We further proposed a maximum-probability estimation (MPE) fusion approach to combine the P300 and SSVEP on a score level and compared this approach to other approaches based on linear discriminant analysis, a naïve Bayes classifier, and support vector machines. The experimental results obtained from thirteen participants indicated that the 4-D hybrid paradigm outperformed the DRC paradigm and that the MPE fusion achieved higher accuracy compared with the other approaches. Importantly, 12 of the 13 participants, using the 4-D paradigm achieved an accuracy of over 90% and the average accuracy was 95.18%. These promising results suggest that the proposed hybrid BCI system could be used in the design of a high-performance BCI-based keyboard.

Enhanced Motor Imagery Training Using a Hybrid BCI With Feedback.

Motor imagery-related mu/beta rhythms, which can be voluntarily modulated by subjects, have been widely used in EEG-based brain computer interfaces (BCIs). Moreover, it has been suggested that motor imagery-specific EEG differences can be enhanced by feedback training. However, the differences observed in the EEGs of naive subjects are typically not sufficient to provide reliable EEG control and thus result in unintended feedback. Such feedback can frustrate subjects and impede training. In this study, a hybrid BCI paradigm combining motor imagery and steady-state visually evoked potentials (SSVEPs) has been proposed to provide effective continuous feedback for motor imagery training. During the initial training sessions, subjects must focus on flickering buttons to evoke SSVEPs as they perform motor imagery tasks. The output/feedback of the hybrid BCI is based on hybrid features consisting of motor imagery- and SSVEP-related brain signals. In this context, the SSVEP plays a more important role than motor imagery in generating feedback. As the training progresses, the subjects can gradually decrease their visual attention to the flickering buttons, provided that the feedback is still effective. In this case, the feedback is mainly based on motor imagery. Our experimental results demonstrate that subjects generate distinguishable brain patterns of hand motor imagery after only five training sessions lasting approximately 1.5 h each. The proposed hybrid feedback paradigm can be used to enhance motor imagery training. This hybrid BCI system with feedback can effectively identify the intentions of the subjects.

A Hybrid BCI speller based on the combination of EMG envelopes and SSVEP

Abstract Recently, much attention has been paid to Hybrid Brain-Computer Interfaces (BCI). In this study, we developed a hybrid BCI speller that simultaneously utilized information from both hand Electromyography (EMG) and SSVEP. This cross-modal BCI speller could increase the target number so as to enhance the information transfer rate (ITR). A 60-target hybrid BCI speller was built in this study. A frame-based sampled sinusoidal stimulation method was used to generate the flickering stimulus on the LCD screen. The 60 targets were equally divided into 4 sections, and each section had the same frequency range. EMG signal was used to distinguish different sections. Subjects were required to repeatedly make a fist from 0 to 3 times when the target was shown in section 1 to section 4. Then by extracting the envelope of the EMG signal and calculating the number of peaks of the envelope, we could know which section the target was in. Canonical Correlation Analysis (CCA) method was used to classify the SSVEP signal. The offline results showed that ITR achieved maximum value when the time window was set to be 2 s. The average classification accuracy of a 2 s time window was 80.5% and information transfer rate was 83.2 bit/min using the proposed hybrid BCI system. While the ITR was 32.7 bit/min for EMG only condition and 58.2 bit/min for SSVEP only condition, which revealed that the hybrid system had better performance than the two single-modal modalities.

Hybrid brain-computer interface and functional electrical stimulation for sensorimotor training in participants with tetraplegia: a proof-of-concept study.

Impaired hand function decreases quality of life in persons with tetraplegia. We tested functional electrical stimulation (FES) controlled by a hybrid brain-computer interface (BCI) for improving hand function in participants with tetraplegia. Two participants with subacute tetraplegia (participant 1: C5 Brown-Sequard syndrome, participant 2: complete C5 lesion) took part in this proof-of-concept study. The goal was to determine whether the BCI system could drive the FES device by accurately classifying participants' intent (open or close the hand). Participants 1 and 2 received 10 sessions and 4 sessions of BCI-FES, respectively. A novel time-switch BCI strategy based on motor imagery was used to activate the FES. In one session, we tested a hybrid BCI-FES based on 2 spontaneously generated brain rhythms: a sensory-motor rhythm during motor imagery to activate a stimulator and occipital alpha rhythms to deactivate the stimulator. Participants received BCI-FES therapy 2 to 3 times a week in addition to conventional therapy. Imagery ability and muscle strength were measured before and after treatment. Visual feedback was associated with a 4-fold increase of brain response during motor imagery in both participants. For participant 1, classification accuracy (open/closed) for motor imagery-based BCI was 83.5% (left hand) and 83.8% (right hand); participant 2 had a classification accuracy of 83.8% for the right hand. Participant 1 had moderate improvement in muscle strength, while there was no change for participant 2. We demonstrated feasibility of BCI-FES, using 2 naturally generated brain rhythms. Studies on a larger number of participants are needed to separate the effects of BCI training from effects of conventional therapy.Video Abstract available. (see Video, Supplemental Digital Content 1, http://links.lww.com/JNPT/A84) for more insights from the authors.

Machine Learning and Brain Computer Interfacing

Machine Learning and Brain Computer Interfacing

Hybrid BCI Coupling EEG and EMG for Severe Motor Disabilities

In this paper, we are studying hybrid Brain-Computer Interfaces (BCI) coupling joystick data, electroencephalogram (EEG – electrical activity of the brain) and electromyogram (EMG – electrical activity of muscles) activities for severe motor disabilities. We are focusing our study on muscular activity as a control modality to interact with an application. We present our data processing and classification technique to detect right and left hand movements. EMG modality is well adapted for DMD patients, because less strength is needed to detect movements in contrast to conventional interfaces like joysticks. Across virtual reality tools, we believe that users will be more able to understand how to interact with such kind of interactive systems. This first part of our study report some very good results concerning the detection of hand movements, according to muscular channel, on healthy subjects.

Hybrid fNIRS-EEG based classification of auditory and visual perception processes.

For multimodal Human-Computer Interaction (HCI), it is very useful to identify the modalities on which the user is currently processing information. This would enable a system to select complementary output modalities to reduce the user's workload. In this paper, we develop a hybrid Brain-Computer Interface (BCI) which uses Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS) to discriminate and detect visual and auditory stimulus processing. We describe the experimental setup we used for collection of our data corpus with 12 subjects. On this data, we performed cross-validation evaluation, of which we report accuracy for different classification conditions. The results show that the subject-dependent systems achieved a classification accuracy of 97.8% for discriminating visual and auditory perception processes from each other and a classification accuracy of up to 94.8% for detecting modality-specific processes independently of other cognitive activity. The same classification conditions could also be discriminated in a subject-independent fashion with accuracy of up to 94.6 and 86.7%, respectively. We also look at the contributions of the two signal types and show that the fusion of classifiers using different features significantly increases accuracy.

Achieving a hybrid brain–computer interface with tactile selective attention and motor imagery

Objective. We propose a new hybrid brain–computer interface (BCI) system that integrates two different EEG tasks: tactile selective attention (TSA) using a vibro-tactile stimulator on the left/right finger and motor imagery (MI) of left/right hand movement. Event-related desynchronization (ERD) from the MI task and steady-state somatosensory evoked potential (SSSEP) from the TSA task are retrieved and combined into two hybrid senses. Approach. One hybrid approach is to measure two tasks simultaneously; the features of each task are combined for testing. Another hybrid approach is to measure two tasks consecutively (TSA first and MI next) using only MI features. For comparison with the hybrid approaches, the TSA and MI tasks are measured independently. Main results. Using a total of 16 subject datasets, we analyzed the BCI classification performance for MI, TSA and two hybrid approaches in a comparative manner; we found that the consecutive hybrid approach outperformed the others, yielding about a 10% improvement in classification accuracy relative to MI alone. It is understood that TSA may play a crucial role as a prestimulus in that it helps to generate earlier ERD prior to MI and thus sustains ERD longer and to a stronger degree; this ERD may give more discriminative information than ERD in MI alone. Significance. Overall, our proposed consecutive hybrid approach is very promising for the development of advanced BCI systems.

Detecting awareness in patients with disorders of consciousness using a hybrid brain–computer interface

Objective. The bedside detection of potential awareness in patients with disorders of consciousness (DOC) currently relies only on behavioral observations and tests; however, the misdiagnosis rates in this patient group are historically relatively high. In this study, we proposed a visual hybrid brain–computer interface (BCI) combining P300 and steady-state evoked potential (SSVEP) responses to detect awareness in severely brain injured patients. Approach. Four healthy subjects, seven DOC patients who were in a vegetative state (VS, n = 4) or minimally conscious state (MCS, n = 3), and one locked-in syndrome (LIS) patient attempted a command-following experiment. In each experimental trial, two photos were presented to each patient; one was the patientʼs own photo, and the other photo was unfamiliar. The patients were instructed to focus on their own or the unfamiliar photos. The BCI system determined which photo the patient focused on with both P300 and SSVEP detections. Main results. Four healthy subjects, one of the 4 VS, one of the 3 MCS, and the LIS patient were able to selectively attend to their own or the unfamiliar photos (classification accuracy, 66–100%). Two additional patients (one VS and one MCS) failed to attend the unfamiliar photo (50–52%) but achieved significant accuracies for their own photo (64–68%). All other patients failed to show any significant response to commands (46–55%). Significance. Through the hybrid BCI system, command following was detected in four healthy subjects, two of 7 DOC patients, and one LIS patient. We suggest that the hybrid BCI system could be used as a supportive bedside tool to detect awareness in patients with DOC.

Towards a gaze-independent hybrid-BCI based on SSVEPs, alpha-band modulations and the P300.

In recent years it has been shown to be possible to create a Brain Computer Interface (BCI) using non-invasive electroencephalographic (EEG) measurements of covert visual spatial attention. For example, that both Steady-State Visual Evoked Potentials (SSVEP) and parieto-occipital alpha band activity have been shown to be sensitive to covert attention and this has been exploited to provide simple communication control without the need for any physical movement. In this study, potential improvements in the speed and accuracy of such a BCI are investigated by exploring the possibility of incorporating a P300 task into an SSVEP covert attention paradigm. Should this be possible it would pave the way for a gaze-independent hybrid BCI based on three somewhat independent EEG signals. Within a well-established SSVEP-based attention paradigm we show that it is possible to make a binary classification of covert attention using just the P300 with an average accuracy of 71% across three subjects. We also validate previously published research by showing robust attention effects on the SSVEP and alpha band activity within this paradigm. In future work, it is hoped that by integrating the three signals into a hybrid BCI a significant improvement in performance will be forthcoming leading to an easily usable real time communication device for patients with severe disabilities such as Locked-In Syndrome (LIS).

Adaptive hybrid brain-computer interaction: ask a trainer for assistance!

In applying mental imagery brain-computer interfaces (BCIs) to end users, training is a key part for novice users to get control. In general learning situations, it is an established concept that a trainer assists a trainee to improve his/her aptitude in certain skills. In this work, we want to evaluate whether we can apply this concept in the context of event-related desynchronization (ERD) based, adaptive, hybrid BCIs. Hence, in a first session we merged the features of a high aptitude BCI user, a trainer, and a novice user, the trainee, in a closed-loop BCI feedback task and automatically adapted the classifier over time. In a second session the trainees operated the system unassisted. Twelve healthy participants ran through this protocol. Along with the trainer, the trainees achieved a very high overall peak accuracy of 95.3 %. In the second session, where users operated the BCI unassisted, they still achieved a high overall peak accuracy of 83.6%. Ten of twelve first time BCI users successfully achieved significantly better than chance accuracy. Concluding, we can say that this trainer-trainee approach is very promising. Future research should investigate, whether this approach is superior to conventional training approaches. This trainer-trainee concept could have potential for future application of BCIs to end users.

A new hybrid BCI paradigm based on P300 and SSVEP

BackgroundP300 and steady-state visual evoked potential (SSVEP) approaches have been widely used for brain–computer interface (BCI) systems. However, neither of these approaches can work for all subjects. Some groups have reported that a hybrid BCI that combines two or more approaches might provide BCI functionality to more users. Hybrid P300/SSVEP BCIs have only recently been developed and validated, and very few avenues to improve performance have been explored. New methodThe present study compares an established hybrid P300/SSVEP BCIs paradigm to a new paradigm in which shape changing, instead of color changing, is adopted for P300 evocation to decrease the degradation on SSVEP strength. ResultThe result shows that the new hybrid paradigm presented in this paper yields much better performance than the normal hybrid paradigm. Comparison with existing methodA performance increase of nearly 20% in SSVEP classification is achieved using the new hybrid paradigm in comparison with the normal hybrid paradigm. All the paradigms except the normal hybrid paradigm used in this paper obtain 100% accuracy in P300 classification. ConclusionsThe new hybrid P300/SSVEP BCIs paradigm in which shape changing, instead of color changing, could obtain as high classification accuracy of SSVEP as the traditional SSVEP paradigm and could obtain as high classification accuracy of P300 as the traditional P300 paradigm. P300 did not interfere with the SSVEP response using the new hybrid paradigm presented in this paper, which was superior to the normal hybrid P300/SSVEP paradigm.

Selecting Filter Range of Hybrid Brain-Computer Interfaces by Mutual Information

The hybrid brain-computer interface (BCI) based on electroencephalography (EEG) become more and more popular. Motor imagery, steady state visual evoked potentials (SSVEPs) and P300 are main training Paradigms. In our previous research, BCI systems based on motor imagery can be extended by speech imagery. However, noise and artifact may be produced by different mental tasks and EEG signals are also different among users, so the classification accuracy can be improved by selecting optimum frequency range for each user. Mutual information (MI) is usually used to choose optimal features. After extracted the features from each narrow frequency range of EEG by common spatial patterns (CSP), the features are assessed by MI. Then, the optimum frequency range can be acquired. The final classification results are calculated by support vector machine (SVM). The average result of optimum frequency range from seven subjects is better than the result of a fixed frequency range.

A brain computer interface-based explorer

In recent years, various applications of brain computer interfaces (BCIs) have been studied. In this paper, we present a hybrid BCI combining P300 and motor imagery to operate an explorer. Our system is mainly composed of a BCI mouse, a BCI speller and an explorer. Through this system, the user can access his computer and manipulate (open, close, copy, paste, and delete) files such as documents, pictures, music, movies and so on. The system has been tested with five subjects, and the experimental results show that the explorer can be successfully operated according to subjects’ intentions.

Neurally and ocularly informed graph-based models for searching 3D environments

Objective. As we move through an environment, we are constantly making assessments, judgments and decisions about the things we encounter. Some are acted upon immediately, but many more become mental notes or fleeting impressions—our implicit ‘labeling’ of the world. In this paper, we use physiological correlates of this labeling to construct a hybrid brain–computer interface (hBCI) system for efficient navigation of a 3D environment. Approach. First, we record electroencephalographic (EEG), saccadic and pupillary data from subjects as they move through a small part of a 3D virtual city under free-viewing conditions. Using machine learning, we integrate the neural and ocular signals evoked by the objects they encounter to infer which ones are of subjective interest to them. These inferred labels are propagated through a large computer vision graph of objects in the city, using semi-supervised learning to identify other, unseen objects that are visually similar to the labeled ones. Finally, the system plots an efficient route to help the subjects visit the ‘similar’ objects it identifies. Main results. We show that by exploiting the subjects’ implicit labeling to find objects of interest instead of exploring naively, the median search precision is increased from 25% to 97%, and the median subject need only travel 40% of the distance to see 84% of the objects of interest. We also find that the neural and ocular signals contribute in a complementary fashion to the classifiers’ inference of subjects’ implicit labeling. Significance. In summary, we show that neural and ocular signals reflecting subjective assessment of objects in a 3D environment can be used to inform a graph-based learning model of that environment, resulting in an hBCI system that improves navigation and information delivery specific to the user’s interests.