Collaborative Brain-computer Interface Research Articles

Single-Trial EEG Classification Using Spatio-Temporal Weighting and Correlation Analysis for RSVP-Based Collaborative Brain Computer Interface.

Since single brain computer interface (BCI) is limited in performance, it is necessary to develop collaborative BCI (cBCI) systems which integrate multi-user electroencephalogram (EEG) information to improve system performance. However, there are still some challenges in cBCI systems, including effective discriminant feature extraction of multi-user EEG data, fusion algorithms, time reduction of system calibration, etc. Methods: This study proposed an event-related potential (ERP) feature extraction and classification algorithm of spatio-temporal weighting and correlation analysis (STC) to improve the performance of cBCI systems. The proposed STC algorithm consisted of three modules. First, source extraction and interval modeling were used to overcome the problem of inter-trial variability. Second, spatio-temporal weighting and temporal projection were utilized to extract effective discriminant features for multi-user information fusion and cross-session transfer. Third, correlation analysis was conducted to match target/non-target templates for classification of multi-user and cross-session datasets. The collaborative cross-session datasets of rapid serial visual presentation (RSVP) from 14 subjects were used to evaluate the performance of the EEG classification algorithm. For single-user/collaborative EEG classification of within-session and cross-session datasets, STC had significantly higher performance than the existing state-of-the-art machine learning algorithms. It was demonstrated that STC was effective to improve the classification performance of multi-user collaboration and cross-session transfer for RSVP-based BCI systems, and was helpful to reduce the system calibration time.

Event-Related Potential-Based Collaborative Brain-Computer Interface for Augmenting Human Performance Using a Low-Cost, Custom Electroencephalogram Hyperscanning Infrastructure

The electroencephalogram (EEG) hyperscanning technique has been demonstrated to facilitate the applicability of a collaborative brain-computer interface (cBCI) to augment human performance with respect to the collective intelligence of multiple brains. However, assembling a hyperscanning platform with commercial products inevitably introduces practical and cost burdens regarding labor and hardware setup, hindering group scalability. This work thus explores how effectively a low-cost, custom-made EEG hyperscanning platform can be achieved by demonstrating a cBCI framework. This work quantifies BCI performance in collaborative and single-brain scenarios applied to the EEG dataset collected from three subjects simultaneously participating in a target-distractor differentiation task over 10 days. This work also compares various brain-fusion scenarios with feature-extraction methods for multiple brains. Given the 30 pseudo brains (i.e., 3 subjects × 10 day sessions), the decision-level committee voting outperformed the single-brain BCI performance and considerably improved by leveraging more pseudo brains. The 27-brain setting achieved the best information transfer rate (ITR) of 116.6±5.6 bits/min, which was a nearly 817% enhancement over the single-brain ITR (12.7±9.2 bits/min). In addition, the cBCI decision augmented the actual button-pressing time by 25 ms. Such a low-cost, custom-made hyperscanning infrastructure economically and practically favors multiple-brain applications in a larger group.

Comparing users' performance and game experience between a competitive and collaborative brain-computer interface

ABSTRACT The recent advancements in Brain-Computer Interfaces (BCIs) technology made possible for its employment in entertainment applications, such as brain-controlled games. However, there are still many challenges to overcome in the design and development of such applications, especially when considering the interaction between multiple users. This paper presents the results of a controlled experiment aimed at comparing the performance and game experience of both competitive and collaborative matches in a multiplayer BCI game. In the experiment, sixteen subjects were divided in pairs into two groups, and each pair played a total of five competitive or collaborative matches of the attention-based game. Quantitative and qualitative data were collected to investigate key features of the subjects' interaction with both the BCI platform and the other players. It was found that there is no statistically significant difference in performance between the subjects of both groups, and between multiplayer and single player modes. Although the group playing competitively reported overall better scores for game experience, there was no statistically significant difference between groups. There were significant correlations between performance and positive aspects of game experience for both modes, and multiplayer matches were significantly shorter than single player ones, although this effect could not be attributed to the mode of interaction alone.

Multimodal collaborative brain-computer interfaces aid human-machine team decision-making in a pandemic scenario

Objective. Critical decisions are made by effective teams that are characterized by individuals who trust each other and know how to best integrate their opinions. Here, we introduce a multimodal brain-computer interface (BCI) to help collaborative teams of humans and an artificial agent achieve more accurate decisions in assessing danger zones during a pandemic scenario. Approach. Using high-resolution simultaneous electroencephalography/functional MRI (EEG/fMRI), we first disentangled the neural markers of decision-making confidence and trust and then employed machine-learning to decode these neural signatures for BCI-augmented team decision-making. We assessed the benefits of BCI on the team’s decision-making process compared to the performance of teams of different sizes using the standard majority or weighing individual decisions. Main results. We showed that BCI-assisted teams are significantly more accurate in their decisions than traditional teams, as the BCI is capable of capturing distinct neural correlates of confidence on a trial-by-trial basis. Accuracy and subjective confidence in the context of collaborative BCI engaged parallel, spatially distributed, and temporally distinct neural circuits, with the former being focused on incorporating perceptual information processing and the latter involving action planning and executive operations during decision making. Among these, the superior parietal lobule emerged as a pivotal region that flexibly modulated its activity and engaged premotor, prefrontal, visual, and subcortical areas for shared spatial-temporal control of confidence and trust during decision-making. Significance. Multimodal, collaborative BCIs that assist human-artificial agent teams may be utilized in critical settings for augmented and optimized decision-making strategies.

Multi-person feature fusion transfer learning-based convolutional neural network for SSVEP-based collaborative BCI.

ObjectiveThe conventional single-person brain–computer interface (BCI) systems have some intrinsic deficiencies such as low signal-to-noise ratio, distinct individual differences, and volatile experimental effect. To solve these problems, a centralized steady-state visually evoked potential collaborative BCI system (SSVEP-cBCI), which characterizes multi-person electroencephalography (EEG) feature fusion was constructed in this paper. Furthermore, three different feature fusion methods compatible with this new system were developed and applied to EEG classification, and a comparative analysis of their classification accuracy was performed with transfer learning-based convolutional neural network (TL-CNN) approach.ApproachAn EEG-based SSVEP-cBCI system was set up to merge different individuals’ EEG features stimulated by the instructions for the same task, and three feature fusion methods were adopted, namely parallel connection, serial connection, and multi-person averaging. The fused features were then input into CNN for classification. Additionally, transfer learning (TL) was applied first to a Tsinghua University (THU) benchmark dataset, and then to a collected dataset, so as to meet the CNN training requirement with a much smaller size of collected dataset and increase the classification accuracy. Ten subjects were recruited for data collection, and both datasets were used to gauge the three fusion algorithms’ performance.Main resultsThe results predicted by TL-CNN approach in single-person mode and in multi-person mode with the three feature fusion methods were compared. The experimental results show that each multi-person mode is superior to single-person mode. Within the 3 s time window, the classification accuracy of the single-person CNN is only 90.6%, while the same measure of the two-person parallel connection fusion method can reach 96.6%, achieving better classification effect.SignificanceThe results show that the three multi-person feature fusion methods and the deep learning classification algorithm based on TL-CNN can effectively improve the SSVEP-cBCI classification performance. The feature fusion method of multi -person parallel feature connection achieves better classification results. Different feature fusion methods can be selected in different application scenarios to further optimize cBCI.

A Collaborative Brain-Computer Interface Framework for Enhancing Group Detection Performance of Dynamic Visual Targets.

The superiority of collaborative brain-computer interface (cBCI) in performance enhancement makes it an effective way to break through the performance bottleneck of the BCI-based dynamic visual target detection. However, the existing cBCIs focus on multi-mind information fusion with a static and unidirectional mode, lacking the information interaction and learning guidance among multiple agents. Here, we propose a novel cBCI framework to enhance the group detection performance of dynamic visual targets. Specifically, a mutual learning domain adaptation network (MLDANet) with information interaction, dynamic learning, and individual transferring abilities is developed as the core of the cBCI framework. MLDANet takes P3-sSDA network as individual network unit, introduces mutual learning strategy, and establishes a dynamic interactive learning mechanism between individual networks and collaborative decision-making at the neural decision level. The results indicate that the proposed MLDANet-cBCI framework can achieve the best group detection performance, and the mutual learning strategy can improve the detection ability of individual networks. In MLDANet-cBCI, the F1 scores of collaborative detection and individual network are 0.12 and 0.19 higher than those in the multi-classifier cBCI, respectively, when three minds collaborate. Thus, the proposed framework breaks through the traditional multi-mind collaborative mode and exhibits a superior group detection performance of dynamic visual targets, which is also of great significance for the practical application of multi-mind collaboration.

An EEG Neurofeedback Interactive Model for Emotional Classification of Electronic Music Compositions Considering Multi-Brain Synergistic Brain-Computer Interfaces.

This paper presents an in-depth study and analysis of the emotional classification of EEG neurofeedback interactive electronic music compositions using a multi-brain collaborative brain-computer interface (BCI). Based on previous research, this paper explores the design and performance of sound visualization in an interactive format from the perspective of visual performance design and the psychology of participating users with the help of knowledge from various disciplines such as psychology, acoustics, aesthetics, neurophysiology, and computer science. This paper proposes a specific mapping model for the conversion of sound to visual expression based on people’s perception and aesthetics of sound based on the phenomenon of audiovisual association, which provides a theoretical basis for the subsequent research. Based on the mapping transformation pattern between audio and visual, this paper investigates the realization path of interactive sound visualization, the visual expression form and its formal composition, and the aesthetic style, and forms a design expression method for the visualization of interactive sound, to benefit the practice of interactive sound visualization. In response to the problem of neglecting the real-time and dynamic nature of the brain in traditional brain network research, dynamic brain networks proposed for analyzing the EEG signals induced by long-time music appreciation. During prolonged music appreciation, the connectivity of the brain changes continuously. We used mutual information on different frequency bands of EEG signals to construct dynamic brain networks, observe changes in brain networks over time and use them for emotion recognition. We used the brain network for emotion classification and achieved an emotion recognition rate of 67.3% under four classifications, exceeding the highest recognition rate available.

Optimization of Task Allocation for Collaborative Brain-Computer Interface Based on Motor Imagery.

ObjectiveCollaborative brain–computer interfaces (cBCIs) can make the BCI output more credible by jointly decoding concurrent brain signals from multiple collaborators. Current cBCI systems usually require all collaborators to execute the same mental tasks (common-work strategy). However, it is still unclear whether the system performance will be improved by assigning different tasks to collaborators (division-of-work strategy) while keeping the total tasks unchanged. Therefore, we studied a task allocation scheme of division-of-work and compared the corresponding classification accuracies with common-work strategy’s.ApproachThis study developed an electroencephalograph (EEG)-based cBCI which had six instructions related to six different motor imagery tasks (MI-cBCI), respectively. For the common-work strategy, all five subjects as a group had the same whole instruction set and they were required to conduct the same instruction at a time. For the division-of-work strategy, every subject’s instruction set was a subset of the whole one and different from each other. However, their union set was equal to the whole set. Based on the number of instructions in a subset, we divided the division-of-work strategy into four types, called “2 Tasks” … “5 Tasks.” To verify the effectiveness of these strategies, we employed EEG data collected from 19 subjects who independently performed six types of MI tasks to conduct the pseudo-online classification of MI-cBCI.Main resultsTaking the number of tasks performed by one collaborator as the horizontal axis (two to six), the classification accuracy curve of MI-cBCI was mountain-like. The curve reached its peak at “4 Tasks,” which means each subset contained four instructions. It outperformed the common-work strategy (“6 Tasks”) in classification accuracy (72.29 ± 4.43 vs. 58.53 ± 4.36%).SignificanceThe results demonstrate that our proposed task allocation strategy effectively enhanced the cBCI classification performance and reduced the individual workload.

Research progress and prospect of collaborative brain-computer interface for group brain collaboration

As the most common active brain-computer interaction paradigm, motor imagery brain-computer interface (MI-BCI) suffers from the bottleneck problems of small instruction set and low accuracy, and its information transmission rate (ITR) and practical application are severely limited. In this study, we designed 6-class imagination actions, collected electroencephalogram (EEG) signals from 19 subjects, and studied the effect of collaborative brain-computer interface (cBCI) collaboration strategy on MI-BCI classification performance, the effects of changes in different group sizes and fusion strategies on group multi-classification performance are compared. The results showed that the most suitable group size was 4 people, and the best fusion strategy was decision fusion. In this condition, the classification accuracy of the group reached 77%, which was higher than that of the feature fusion strategy under the same group size (77.31% vs. 56.34%), and was significantly higher than that of the average single user (77.31% vs. 44.90%). The research in this paper proves that the cBCI collaboration strategy can effectively improve the MI-BCI classification performance, which lays the foundation for MI-cBCI research and its future application.

Performance of a Steady-State Visual Evoked Potential and Eye Gaze Hybrid Brain-Computer Interface on Participants With and Without a Brain Injury

The brain–computer interface (BCI) and the tracking of eye gaze provide modalities for human–machine communication and control. In this article, we provide the evaluation of a collaborative BCI and eye gaze approach, known as a hybrid BCI. The combined inputs interact with a virtual environment to provide actuation according to a four-way menu system. The following two approaches are evaluated: first, steady-state visual evoked potential (SSVEP) BCI with on-screen stimulation; second, hybrid BCI, which combined eye gaze and SSVEP for navigation and selection. A study comprises participants without known brain injury (non-BI, N = 30) and participants with known brain injury (BI, N = 14). A total of 29 out of 30 non-BI participants can successfully control the hybrid BCI, while nine out of the 14 BI participants are able to achieve control, as evidenced by task completion. The hybrid BCI provides a mean accuracy of 99.84% in the cohort of non-BI participants and 99.14% in the cohort of BI participants. Information transfer rates are 24.41 bpm in non-BI participants and 15.87 bpm in BI participants. The research goal is to quantify usage of SSVEP and ET approaches in cohorts of non-BI and BI participants. The hybrid is the preferred interaction modality for most participants for both cohorts. When compared to non-BI participants, it is encouraging that nine out of 14 participants with known BI can use the hBCI technology with equivalent accuracy and efficiency, albeit with slower transfer rates.

Simultaneous EEG Acquisition System for Multiple Users: Development and Related Issues.

Social interaction is one of humans’ most important activities and many efforts have been made to understand the phenomenon. Recently, some investigators have attempted to apply advanced brain signal acquisition systems that allow dynamic brain activities to be measured simultaneously during social interactions. Most studies to date have investigated dyadic interactions, although multilateral interactions are more common in reality. However, it is believed that most studies have focused on such interactions because of methodological limitations, in that it is very difficult to design a well-controlled experiment for multiple users at a reasonable cost. Accordingly, there are few simultaneous acquisition systems for multiple users. In this study, we propose a design framework for an acquisition system that measures EEG data simultaneously in an environment with 10 or more people. Our proposed framework allowed us to acquire EEG data at up to 1 kHz frequency from up to 20 people simultaneously. Details of our acquisition system are described from hardware and software perspectives. In addition, various related issues that arose in the system’s development—such as synchronization techniques, system loads, electrodes, and applications—are discussed. In addition, simultaneous visual ERP experiments were conducted with a group of nine people to validate the EEG acquisition framework proposed. We found that our framework worked reasonably well with respect to less than 4 ms delay and average loss rates of 1%. It is expected that this system can be used in various hyperscanning studies, such as those on crowd psychology, large-scale human interactions, and collaborative brain–computer interface, among others.

Collaborative Brain-Computer Interfaces to Enhance Group Decisions in an Outpost Surveillance Task.

We present a two-layered collaborative Brain-Computer Interface (cBCI) to aid groups making decisions under time constraints in a realistic video surveillance setting - the very first cBCI application of this type. The cBCI first uses response times (RTs) to estimate the decision confidence the user would report after each decision. Such an estimate is then used with neural features extracted from EEG to refine the decision confidence so that it better correlates with the correctness of the decision. The refined confidence is then used to weigh individual responses and obtain group decisions. Results obtained with 10 participants indicate that cBCI-assisted groups are significantly more accurate than groups using standard majority or weighing decisions using reported confidence values. This two-layer architecture allows the cBCI to not only further enhance group performance but also speed up the decision process, as the cBCI does not have to wait for all users to report their confidence after each decision.

Analysis of Collaborative Brain Computer Interface (BCI) based Personalized GUI for Differently Abled

Brain-Computer Interfaces (BCI) use Electroencephalography (EEG) signals recorded from the brain scalp, which enable a communication between the human and the outside world. The present study helps...

Enhancement of Group Perception via a Collaborative Brain-Computer Interface.

We aimed at improving group performance in a challenging visual search task via a hybrid collaborative brain-computer interface (cBCI). Ten participants individually undertook a visual search task where a display was presented for 250ms, and they had to decide whether a target was present or not. Local temporal correlation common spatial pattern (LTCCSP) was used to extract neural features from response- and stimulus-locked EEG epochs. The resulting feature vectors were extended by including response times and features extracted from eye movements. A classifier was trained to estimate the confidence of each group member. cBCI-assisted group decisions were then obtained using a confidence-weighted majority vote. Participants were combined in groups of different sizes to assess the performance of the cBCI. Results show that LTCCSP neural features, response times, and eye movement features significantly improve the accuracy of the cBCI over what we achieved with previous systems. For most group sizes, our hybrid cBCI yields group decisions that are significantly better than majority-based group decisions. The visual task considered here was much harder than a task we used in previous research. However, thanks to a range of technological enhancements, our cBCI has delivered a significant improvement over group decisions made by a standard majority vote. With previous cBCIs, groups may perform better than single non-BCI users. Here, cBCI-assisted groups are more accurate than identically sized non-BCI groups. This paves the way to a variety of real-world applications of cBCIs where reducing decision errors is vital.

Towards the automated localisation of targets in rapid image-sifting by collaborative brain-computer interfaces.

The N2pc is a lateralised Event-Related Potential (ERP) that signals a shift of attention towards the location of a potential object of interest. We propose a single-trial target-localisation collaborative Brain-Computer Interface (cBCI) that exploits this ERP to automatically approximate the horizontal position of targets in aerial images. Images were presented by means of the rapid serial visual presentation technique at rates of 5, 6 and 10 Hz. We created three different cBCIs and tested a participant selection method in which groups are formed according to the similarity of participants’ performance. The N2pc that is elicited in our experiments contains information about the position of the target along the horizontal axis. Moreover, combining information from multiple participants provides absolute median improvements in the area under the receiver operating characteristic curve of up to 21% (for groups of size 3) with respect to single-user BCIs. These improvements are bigger when groups are formed by participants with similar individual performance, and much of this effect can be explained using simple theoretical models. Our results suggest that BCIs for automated triaging can be improved by integrating two classification systems: one devoted to target detection and another to detect the attentional shifts associated with lateral targets.

Collaborative Brain-Computer Interface for People with Motor Disabilities [Research Frontier

This research investigates the effects of collaboration mode, luminance contrast and motor disability on task performance, brain activity and satisfaction of users with motor disabilities who performed a robot control task using a collaborative brain-computer interface (C-BCI) based on steady-state visually evoked potentials (SSVEPs). Users can perform the task by himself/herself (individual mode), working together (simultaneous mode), or taking turns (sequential mode). Fourteen amyotrophic laterals sclerosis (ALS) participants and fourteen able-bodied participants of similar age were recruited from local ALS association and local communities. Results showed that participants in both groups had significantly better task performance and stronger brain activity in collaborative modes than individual mode. High luminance contrast produced significantly better task performance and stronger brain activity than low luminance contrast. There was no significant effect of motor disability between ALS participants and able-bodied participants. The two groups showed very similar task performance and brain activity. These results could provide precious empirical data and invaluable insights to the real-world applicability of the SSVEP-based BCI applications for people with motor disabilities.

A Collaborative Brain-Computer Interface (BCI) for ALS Patients

This study evaluated a SSVEP-based collaborative brain-computer interface (BCI) for people with severe motor disabilities. With ten ALS (amyotrophic lateral sclerosis) patients and 10 age-matched able-bodied participants as control group, effects of collaboration and motor disability were investigated. Participants were requested to control a robot in a predefined path with their brain signals. Two collaboration modes were developed in the study: individual mode and simultaneous mode. Results revealed significantly better performance in simultaneous mode than individual mode, but no significant effect of motor disability. The study showed promising preliminary results for supporting collaborative work between BCI users with severe motor disabilities. It should provide great insights for future research and system development.

Single-Trial Classification of Multi-User P300-Based Brain-Computer Interface Using Riemannian Geometry.

The classification of electroencephalographic (EEG) data recorded from multiple users simultaneously is an important challenge in the field of Brain-Computer Interface (BCI). In this paper we compare different approaches for classification of single-trials Event-Related Potential (ERP) on two subjects playing a collaborative BCI game. The minimum distance to mean (MDM) classifier in a Riemannian framework is extended to use the diversity of the inter-subjects spatio-temporal statistics (MDM-hyper) or to merge multiple classifiers (MDM-multi). We show that both these classifiers outperform significantly the mean performance of the two users and analogous classifiers based on the step-wise linear discriminant analysis. More importantly, the MDM-multi outperforms the performance of the best player within the pair.

Collaborative brain-computer interface for aiding decision-making.

We look at the possibility of integrating the percepts from multiple non-communicating observers as a means of achieving better joint perception and better group decisions. Our approach involves the combination of a brain-computer interface with human behavioural responses. To test ideas in controlled conditions, we asked observers to perform a simple matching task involving the rapid sequential presentation of pairs of visual patterns and the subsequent decision as whether the two patterns in a pair were the same or different. We recorded the response times of observers as well as a neural feature which predicts incorrect decisions and, thus, indirectly indicates the confidence of the decisions made by the observers. We then built a composite neuro-behavioural feature which optimally combines the two measures. For group decisions, we uses a majority rule and three rules which weigh the decisions of each observer based on response times and our neural and neuro-behavioural features. Results indicate that the integration of behavioural responses and neural features can significantly improve accuracy when compared with the majority rule. An analysis of event-related potentials indicates that substantial differences are present in the proximity of the response for correct and incorrect trials, further corroborating the idea of using hybrids of brain-computer interfaces and traditional strategies for improving decision making.

A Collaborative Brain-Computer Interface for Improving Human Performance

Electroencephalogram (EEG) based brain-computer interfaces (BCI) have been studied since the 1970s. Currently, the main focus of BCI research lies on the clinical use, which aims to provide a new communication channel to patients with motor disabilities to improve their quality of life. However, the BCI technology can also be used to improve human performance for normal healthy users. Although this application has been proposed for a long time, little progress has been made in real-world practices due to technical limits of EEG. To overcome the bottleneck of low single-user BCI performance, this study proposes a collaborative paradigm to improve overall BCI performance by integrating information from multiple users. To test the feasibility of a collaborative BCI, this study quantitatively compares the classification accuracies of collaborative and single-user BCI applied to the EEG data collected from 20 subjects in a movement-planning experiment. This study also explores three different methods for fusing and analyzing EEG data from multiple subjects: (1) Event-related potentials (ERP) averaging, (2) Feature concatenating, and (3) Voting. In a demonstration system using the Voting method, the classification accuracy of predicting movement directions (reaching left vs. reaching right) was enhanced substantially from 66% to 80%, 88%, 93%, and 95% as the numbers of subjects increased from 1 to 5, 10, 15, and 20, respectively. Furthermore, the decision of reaching direction could be made around 100–250 ms earlier than the subject's actual motor response by decoding the ERP activities arising mainly from the posterior parietal cortex (PPC), which are related to the processing of visuomotor transmission. Taken together, these results suggest that a collaborative BCI can effectively fuse brain activities of a group of people to improve the overall performance of natural human behavior.