Multimodal Brain-computer Interface Research Articles

Multimodal Brain–Computer Interface for In-Vehicle Driver Cognitive Load Measurement: Dataset and Baselines

Multimodal Brain–Computer Interface for In-Vehicle Driver Cognitive Load Measurement: Dataset and Baselines

Exploring Aesthetic Perception in Impaired Aging: A Multimodal Brain-Computer Interface Study.

In the field of neuroscience, brain-computer interfaces (BCIs) are used to connect the human brain with external devices, providing insights into the neural mechanisms underlying cognitive processes, including aesthetic perception. Non-invasive BCIs, such as EEG and fNIRS, are critical for studying central nervous system activity and understanding how individuals with cognitive deficits process and respond to aesthetic stimuli. This study assessed twenty participants who were divided into control and impaired aging (AI) groups based on MMSE scores. EEG and fNIRS were used to measure their neurophysiological responses to aesthetic stimuli that varied in pleasantness and dynamism. Significant differences were identified between the groups in P300 amplitude and late positive potential (LPP), with controls showing greater reactivity. AI subjects showed an increase in oxyhemoglobin in response to pleasurable stimuli, suggesting hemodynamic compensation. This study highlights the effectiveness of multimodal BCIs in identifying the neural basis of aesthetic appreciation and impaired aging. Despite its limitations, such as sample size and the subjective nature of aesthetic appreciation, this research lays the groundwork for cognitive rehabilitation tailored to aesthetic perception, improving the comprehension of cognitive disorders through integrated BCI methodologies.

Activation of a Rhythmic Lower Limb Movement Pattern during the Use of a Multimodal Brain-Computer Interface: A Case Study of a Clinically Complete Spinal Cord Injury.

Brain-computer interfaces (BCIs) that integrate virtual reality with tactile feedback are increasingly relevant for neurorehabilitation in spinal cord injury (SCI). In our previous case study employing a BCI-based virtual reality neurorehabilitation protocol, a patient with complete T4 SCI experienced reduced pain and emergence of non-spastic lower limb movements after 10 sessions. However, it is still unclear whether these effects can be sustained, enhanced, and replicated, as well as the neural mechanisms that underlie them. The present report outlines the outcomes of extending the previous protocol with 24 more sessions (14 months, in total). Clinical, behavioral, and neurophysiological data were analyzed. The protocol maintained or reduced pain levels, increased self-reported quality of life, and was frequently associated with the appearance of non-spastic lower limb movements when the patient was engaged and not experiencing stressful events. Neural activity analysis revealed that changes in pain were encoded in the theta frequency band by the left frontal electrode F3. Examination of the lower limbs revealed alternating movements resembling a gait pattern. These results suggest that sustained use of this BCI protocol leads to enhanced quality of life, reduced and stable pain levels, and may result in the emergence of rhythmic patterns of lower limb muscle activity reminiscent of gait.

Multimodal collaborative brain-computer interfaces aid human-machine team decision-making in a pandemic scenario

Objective. Critical decisions are made by effective teams that are characterized by individuals who trust each other and know how to best integrate their opinions. Here, we introduce a multimodal brain-computer interface (BCI) to help collaborative teams of humans and an artificial agent achieve more accurate decisions in assessing danger zones during a pandemic scenario. Approach. Using high-resolution simultaneous electroencephalography/functional MRI (EEG/fMRI), we first disentangled the neural markers of decision-making confidence and trust and then employed machine-learning to decode these neural signatures for BCI-augmented team decision-making. We assessed the benefits of BCI on the team’s decision-making process compared to the performance of teams of different sizes using the standard majority or weighing individual decisions. Main results. We showed that BCI-assisted teams are significantly more accurate in their decisions than traditional teams, as the BCI is capable of capturing distinct neural correlates of confidence on a trial-by-trial basis. Accuracy and subjective confidence in the context of collaborative BCI engaged parallel, spatially distributed, and temporally distinct neural circuits, with the former being focused on incorporating perceptual information processing and the latter involving action planning and executive operations during decision making. Among these, the superior parietal lobule emerged as a pivotal region that flexibly modulated its activity and engaged premotor, prefrontal, visual, and subcortical areas for shared spatial-temporal control of confidence and trust during decision-making. Significance. Multimodal, collaborative BCIs that assist human-artificial agent teams may be utilized in critical settings for augmented and optimized decision-making strategies.

Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning.

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have potentially complementary characteristics that reflect the electrical and hemodynamic characteristics of neural responses, so EEG-fNIRS-based hybrid brain-computer interface (BCI) is the research hotspots in recent years. However, current studies lack a comprehensive systematic approach to properly fuse EEG and fNIRS data and exploit their complementary potential, which is critical for improving BCI performance. To address this issue, this study proposes a novel multimodal fusion framework based on multi-level progressive learning with multi-domain features. The framework consists of a multi-domain feature extraction process for EEG and fNIRS, a feature selection process based on atomic search optimization, and a multi-domain feature fusion process based on multi-level progressive machine learning. The proposed method was validated on EEG-fNIRS-based motor imagery (MI) and mental arithmetic (MA) tasks involving 29 subjects, and the experimental results show that multi-domain features provide better classification performance than single-domain features, and multi-modality provides better classification performance than single-modality. Furthermore, the experimental results and comparison with other methods demonstrated the effectiveness and superiority of the proposed method in EEG and fNIRS information fusion, it can achieve an average classification accuracy of 96.74% in the MI task and 98.42% in the MA task. Our proposed method may provide a general framework for future fusion processing of multimodal brain signals based on EEG-fNIRS.

Multimodal collaborative BCI system based on the improved CSP feature extraction algorithm

BackgroundAs a novel approach for people to directly communicate with an external device, the study of brain-computer interfaces (BCIs) has become well-rounded. However, similar to the real-world scenario, where individuals are expected to work in groups, the BCI systems should be able to replicate group attributes. MethodsWe proposed a 4-order cumulants feature extraction method (CUM4- CSP) based on the common spatial patterns (CSP) algorithm. Simulation experiments conducted using motion visual evoked potentials (mVEP) EEG data verified the robustness of the proposed algorithm. In addition, to freely choose paradigms, we adopted the mVEP and steady-state visual evoked potential (SSVEP) paradigms and designed a multimodal collaborative BCI system based on the proposed CUM4- CSP algorithm. The feasibility of the proposed multimodal collaborative system framework was demonstrated using a multiplayer game controlling system that simultaneously facilitates the coordination and competitive control of two users on external devices. To verify the robustness of the proposed scheme, we recruited 30 subjects to conduct online game control experiments, and the results were statistically analyzed. ResultsThe simulation results prove that the proposed CUM4-CSP algorithm has good noise immunity. The online experimental results indicate that the subjects could reliably perform the game confrontation operation with the selected BCI paradigm. ConclusionsThe proposed CUM4-CSP algorithm can effectively extract features from EEG data in a noisy environment. Additionally, the proposed scheme may provide a new solution for EEG-based group BCI research.

A multi-modal brain-computer interface based on threshold discrimination and its application in wheelchair control.

In this study, we propose a novel multi-modal brain-computer interface (BCI) system based on the threshold discrimination, which is proposed for the first time to distinguish between SSVEP and MI potentials. The system combines these two heterogeneous signals to increase the number of control commands and improve the performance of asynchronous control of external devices. In this research, an electric wheelchair is controlled as an example. The user can continuously control the wheelchair to turn left/right through motion imagination (MI) by imagining left/right-hand movement and generate another 6 commands for the wheelchair control by focusing on the SSVEP stimulation panel. Ten subjects participated in a MI training session and eight of them completed a mobile obstacle-avoidance experiment in a complex environment requesting high control accuracy for successful manipulation. Comparing with the single-modal BCI-controlled wheelchair system, the results demonstrate that the proposed multi-modal method is effective by providing more satisfactory control accuracy, and show the potential of BCI-controlled systems to be applied in complex daily tasks.

Hemodynamic Response Detection Using Integrated EEG-fNIRS-VPA for BCI

For BCI systems, it is important to have an accurate and less complex architecture to control a device with enhanced accuracy. In this paper, a novel methodology for more accurate detection of the hemodynamic response has been developed using a multimodal brain-computer interface (BCI). An integrated classifier has been developed for achieving better classification accuracy using two modalities. An integrated EEG-fNIRS-based vector-phase analysis (VPA) has been conducted. An open-source dataset collected at the Technische Universität Berlin, including simultaneous electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals of 26 healthy participants during n-back tests, has been used for this research. Instrumental and physiological noise removal has been done using preprocessing techniques followed by individually detecting activity in both modalities. With resting state threshold circle, VPA has been used to detect a hemodynamic response in fNIRS signals, whereas phase plots for EEG signals have been constructed using Hilbert Transform to detect the activity in each trial. Multiple threshold circles are drawn in the vector plane, where each circle is drawn after task completion in each trial of EEG signal. Finally, both processes are integrated into one vector-phase plot to get combined detection of hemodynamic response for activity. Results of this study illustrate that the combined EEG-fNIRS VPA yields considerably higher average classification accuracy, that is 91.35%, as compared to other classifiers such as support vector machine (SVM), convolutional neural networks (CNN), deep neural networks (DNN) and VPA (with dual-threshold circles) with classification accuracies 82%, 89%, 87% and 86% respectively. Outcomes of this research demonstrate that improved classification performance can be feasibly achieved using multimodal VPA for EEG-fNIRS hybrid data.

Towards multimodal BCIs: the impact of peripheral control on motor cortex activity and sense of agency.

In the recent years, brain computer interfaces (BCI) using motor imagery have shown some limitations regarding the quality of control. In an effort to improve this promising technology, some studies intended to develop hybrid BCI with other technologies such as eye tracking which shows more reliability. However, the use of an eye tracker in the control of a robot might affect by itself the sense of agency (SoA) and the brain activity in the regions used for motor imagery (MI). Here, we explore the link between the sense of agency and the activity of the motor cortex. For this purpose, we used of a virtual arm projected on a surface which is either controlled by motion capture or controlled by gaze using an eye tracker. We found out that there is an activity in the motor cortex during the task of control by gaze and that having control over a projected robotic arm presents significant differences with the situation of observing the robot moving.

A multi-modal modified feedback self-paced BCI to control the gait of an avatar

Brain–computer interfaces (BCIs) have been used to control the gait of a virtual self-avatar with a proposed application in the field of gait rehabilitation. Some limitations of existing systems are: (a) some systems use mental imagery (MI) of movements other than gait; (b) most systems allow the user to take single steps or to walk but do not allow both; (c) most function in a single BCI mode (cue-paced or self-paced). Objective. The objective of this study was to develop a high performance multi-modal BCI to control single steps and forward walking of an immersive virtual reality avatar. Approach. This system used MI of these actions, in cue-paced and self-paced modes. Twenty healthy participants participated in this study, which was comprised of four sessions across four different days. They were cued to imagine a single step forward with their right or left foot, or to imagine walking forward. They were instructed to reach a target by using the MI of multiple steps (self-paced switch-control mode) or by maintaining MI of forward walking (continuous-control mode). The movement of the avatar was controlled by two calibrated regularized linear discriminate analysis classifiers that used the µ power spectral density over the foot area of the motor cortex as a feature. The classifiers were retrained after every session. For a subset of the trials, positive modified feedback (MDF) was presented to half of the participants, where the avatar moved correctly regardless of the classification of the participants’ MI. The performance of the BCI was computed on each day, using different control modes. Main results. All participants were able to operate the BCI. Their average offline performance, after retraining the classifiers was 86.0 ± 6.1%, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9 ± 8.4% showing that MDF enhanced BCI performance (p = 0.001). The average performance was 83% at self-paced switch control and 92% at continuous control mode. Significance. This study reports on a first BCI to use motor imagery of the lower limbs in order to control the gait of an avatar with different control modes and different control commands (single steps or forward walking). BCI performance is increased in a novel way by combining three different performance enhancement techniques, resulting in a single high performance and multi-modal BCI system. This study also showed that the improvements due to the effects of MDF lasted for more than one session.

Advances in Multimodal Emotion Recognition Based on Brain-Computer Interfaces.

With the continuous development of portable noninvasive human sensor technologies such as brain–computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.

Development of a ternary hybrid fNIRS-EEG brain–computer interface based on imagined speech

ABSTRACTThere is increasing interest in developing intuitive brain-computer interfaces (BCIs) to differentiate intuitive mental tasks such as imagined speech. Both electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have been used for this purpose. However, the classification accuracy and number of commands in such BCIs have been limited. The use of multi-modal BCIs to address these issues has been proposed for some common BCI tasks, but not for imagined speech. Here, we propose a multi-class hybrid fNIRS-EEG BCI based on imagined speech. Eleven participants performed multiple iterations of three tasks: mentally repeating ‘yes’ or ‘no’ for 15 s or an equivalent duration of unconstrained rest. We achieved an average ternary classification accuracy of 70.45 ± 19.19% which is significantly better than that attained with each modality alone (p < 0.05). Our findings suggest that concurrent measurements of EEG and fNIRS can improve classification accuracy of BCIs based on imagined speech.

Decoding covert speech for intuitive control of brain-computer interfaces based on single-trial EEG: a feasibility study.

For individuals with severe motor deficiencies, controlling external devices such as robotic arms or wheelchairs can be challenging, as many devices require some degree of motor control to be operated, e.g. when controlled using a joystick. A brain-computer interface (BCI) relies only on signals from the brain and may be used as a controller instead of muscles. Motor imagery (MI) has been used in many studies as a control signal for BCIs. However, MI may not be suitable for all control purposes, and several people cannot obtain BCI control with MI. In this study, the aim was to investigate the feasibility of decoding covert speech from single-trial EEG and compare and combine it with MI. In seven healthy subjects, EEG was recorded with twenty-five channels during six different actions: Speaking three words (both covert and overt speech), two arm movements (both motor imagery and execution), and one idle class. Temporal and spectral features were derived from the epochs and classified with a random forest classifier. The average classification accuracy was $67 \pm 9$ % and $75\pm 7$ % for covert and overt speech, respectively; this was 5-10 % lower than the movement classification. The performance of the combined movement-speech decoder was $61 \pm 9$ % and $67\pm 7$ % (covert and overt), but it is possible to have more classes available for control. The possibility of using covert speech for controlling a BCI was outlined; this is a step towards a multimodal BCI system for improved usability.

Quadcopter Flight Control Using a Non-invasive Multi-Modal Brain Computer Interface.

Brain-Computer Interfaces (BCIs) translate neuronal information into commands to control external software or hardware, which can improve the quality of life for both healthy and disabled individuals. Here, a multi-modal BCI which combines motor imagery (MI) and steady-state visual evoked potential (SSVEP) is proposed to achieve stable control of a quadcopter in three-dimensional physical space. The complete information common spatial pattern (CICSP) method is used to extract two MI features to control the quadcopter to fly left-forward and right-forward, and canonical correlation analysis (CCA) is employed to perform the SSVEP classification for rise and fall. Eye blinking is designed to switch these two modes while hovering. Real-time feedback is provided to subjects by a global camera. Two flight tasks were conducted in physical space in order to certify the reliability of the BCI system. Subjects were asked to control the quadcopter to fly forward along the zig-zag pattern to pass through a gate in the relatively simple task. For the other complex task, the quadcopter was controlled to pass through two gates successively according to an S-shaped route. The performance of the BCI system is quantified using suitable metrics and subjects are able to acquire 86.5% accuracy for the complicated flight task. It is demonstrated that the multi-modal BCI has the ability to increase the accuracy rate, reduce the task burden, and improve the performance of the BCI system in the real world.

Transfer Learning for a Multimodal Hybrid EEG-fTCD Brain–Computer Interface

Humans can transfer knowledge previously acquired from a specific task to new and unknown ones. Recently, transfer learning (TL) has been extensively used in brain–computer interface (BCI) research to reduce the training/calibration requirements. BCI systems have been designed to provide alternative communication or control access through computers to individuals with limited speech and physical abilities (LSPA). These systems generally require a calibration session in order to train the BCI before each usage. Such a calibration session may be burdensome for the individuals with LSPA. In this article, we introduce a multimodal hybrid BCI based on electroencephalography (EEG) and functional transcranial Doppler ultrasound (fTCD) and present a TL approach to reduce the calibration requirements. In the hybrid BCI, EEG, and fTCD are used simultaneously to measure the electrical brain activity and cerebral blood velocity, respectively, in response to motor imagery (MI) tasks. Using the data we collected from ten healthy individuals, we perform dimensionality reduction utilizing regularized discriminant analysis (RDA). Using the scores from RDA, we learn class conditional probabilistic distributions for each individual. We use these class conditional distributions to perform TL across different participants. More specifically, in order to reduce the calibration requirements for each individual, we choose the recorded data from other individuals to augment the training data for that specific individual. We choose the data for augmentation based on the probabilistic similarities between the class conditional distributions. For the final classification, we use the RDA scores after TL as features input to three different classifiers: quadratic discriminant analysis (QDA), linear discriminant analysis (LDA), and support vector machines (SVMs). Using our experimental data, we show that TL decreases the calibration requirements up to $87.5\%$ . Also by comparing SVM, LDA, and QDA, we observe that the SVM provides the best classification performance.

Feature Extraction and Classification Algorithm of Brain-computer Interface Based on Human Brain Central Nervous System

For the problems that brain-computer interface is susceptible to environmental noise interference and has low classification accuracy in traditional single-mode electroencephalography (EEG), this paper introduces the technique of function Near Infrared Spectroscopy (fNIRS) based on the study of EEG brain-computer interface in human brain central nervous system, designs and simplifies the experimental paradigm of the EEG-fNIRS multimodal brain-computer interface based on the fisting action. According to the result of single modal feature classification, the fusion feature based on EEG wavelet coefficient and fNIRS slope is proposed. The result shows that the average recognition rate of the fisting action task after feature fusion is 3% to 9% higher than that of the EEG feature and fNIRS alone. The fNIRS can significantly enhance the brain-machine interface performance based on the EEG and is of significance for the improvement of the application of the multimodal brain-machine interface.

Design and experiment of a multi-modal electroencephalogram-near infrared spectroscopy helmet for simultaneously acquiring at the same brain area

Multi-modal brain-computer interface and multi-modal brain function imaging are developing trends for the present and future. Aiming at multi-modal brain-computer interface based on electroencephalogram-near infrared spectroscopy (EEG-NIRS) and in order to simultaneously acquire the brain activity of motor area, an acquisition helmet by NIRS combined with EEG was designed and verified by the experiment. According to the 10-20 system or 10-20 extended system, the diameter and spacing of NIRS probe and EEG electrode, NIRS probes were aligned with C3 and C4 as the reference electrodes, and NIRS probes were placed in the middle position between EEG electrodes to simultaneously measure variations of NIRS and the corresponding variation of EEG in the same functional brain area. The clamp holder and near infrared probe were coupled by tightening a screw. To verify the feasibility and effectiveness of the multi-modal EEG-NIRS helmet, NIRS and EEG signals were collected from six healthy subjects during six mental tasks involving the right hand clenching force and speed motor imagery. These signals may reflect brain activity related to hand clenching force and speed motor imagery in a certain extent. The experiment showed that the EEG-NIRS helmet designed in the paper was feasible and effective. It not only could provide support for the multi-modal motor imagery brain-computer interface based on EEG-NIRS, but also was expected to provide support for multi-modal brain functional imaging based on EEG-NIRS.

Multimodal brain-computer interface combining synchronously electroencephalography and electromyography

Multimodal brain-computer interface combining synchronously electroencephalography and electromyography

Open Access Dataset for EEG+NIRS Single-Trial Classification.

We provide an open access dataset for hybrid brain-computer interfaces (BCIs) using electroencephalography (EEG) and near-infrared spectroscopy (NIRS). For this, we con-ducted two BCI experiments (left vs. right hand motor imagery; mental arithmetic vs. resting state). The dataset was validated using baseline signal analysis methods, with which classification performance was evaluated for each modality and a combination of both modalities. As already shown in previous literature, the capability of discriminating different mental states can be en-hanced by using a hybrid approach, when comparing to single modality analyses. This makes the provided data highly suitable for hybrid BCI investigations. Since our open access dataset also comprises motion artifacts and physiological data, we expect that it can be used in a wide range of future validation approaches in multimodal BCI research.

Multimodal BCIs: Target Detection, Multidimensional Control, and Awareness Evaluation in Patients With Disorder of Consciousness

Despite rapid advances in the study of brain–computer interfaces (BCIs) in recent decades, two fundamental challenges, namely, improvement of target detection performance and multidimensional control, continue to be major barriers for further development and applications. In this paper, we review the recent progress in multimodal BCIs (also called hybrid BCIs), which may provide potential solutions for addressing these challenges. In particular, improved target detection can be achieved by developing multimodal BCIs that utilize multiple brain patterns, multimodal signals, or multisensory stimuli. Furthermore, multidimensional object control can be accomplished by generating multiple control signals from different brain patterns or signal modalities. Here, we highlight several representative multimodal BCI systems by analyzing their paradigm designs, detection/control methods, and experimental results. To demonstrate their practicality, we report several initial clinical applications of these multimodal BCI systems, including awareness evaluation/detection in patients with disorder of consciousness (DOC). As an evolving research area, the study of multimodal BCIs is increasingly requiring more synergetic efforts from multiple disciplines for the exploration of the underlying brain mechanisms, the design of new effective paradigms and means of neurofeedback, and the expansion of the clinical applications of these systems.