Brain-computer Interaction Research Articles

EEG classification of driver mental states by deep learning.

Driver fatigue is attracting more and more attention, as it is the main cause of traffic accidents, which bring great harm to society and families. This paper proposes to use deep convolutional neural networks, and deep residual learning, to predict the mental states of drivers from electroencephalography (EEG) signals. Accordingly we have developed two mental state classification models called EEG-Conv and EEG-Conv-R. Tested on intra- and inter-subject, our results show that both models outperform the traditional LSTM- and SVM-based classifiers. Our major findings include (1) Both EEG-Conv and EEG-Conv-R yield very good classification performance for mental state prediction; (2) EEG-Conv-R is more suitable for inter-subject mental state prediction; (3) EEG-Conv-R converges more quickly than EEG-Conv. In summary, our proposed classifiers have better predictive power and are promising for application in practical brain-computer interaction .

Brain–computer interaction for online enhancement of visuospatial attention performance

Objective. this study on real-time decoding of visuospatial attention has two objectives: first, to reliably decode self-directed shifts of attention from electroencephalography (EEG) data, and second, to analyze whether this information can be used to enhance visuospatial performance. Visuospatial performance was measured in a target orientation discrimination task, in terms of reaction time, and error rate. Approach. Our experiment extends the Posner paradigm by introducing a new type of ambiguous cues to indicate upcoming target location. The cues are designed so that their ambiguity is imperceptible to the user. This entails endogenous shifts of attention which are truly self-directed. Two protocols were implemented to exploit the decoding of attention shifts. The first ‘adaptive’ protocol uses the decoded locus to display the target. In the second ‘warning’ protocol, the target position is defined in advance, but a warning is flashed when the target mismatches the decoded locus. Main results. Both protocols were tested in an online experiment involving ten subjects. The reaction time improved in both the adaptive and the warning protocol. The error rate was improved in the adaptive protocol only. Significance. This proof of concept study brings evidence that visuospatial brain–computer interfaces (BCIs) can be used to enhance improving human–machine interaction in situations where humans must react to off-center events in the visual field.

Human and Smart Machine Co-Learning: Brain-Computer Interaction at the 2017 IEEE International Conference on Systems, Man, and Cybernetics

Machine learning has become a very popular approach for cybernetics systems, and it has always been considered important research in the Computational Intelligence area. Nevertheless, when it comes to smart machines, it is not just about the methodologies. We need to consider systems and cybernetics as well as include human in the loop. The purpose of this article is as follows: (1) To integrate the open source Facebook AI Research (FAIR) DarkForest program of Facebook with Item Response Theory (IRT), to the new open learning system, namely, DDF learning system; (2) To integrate DDF Go with Robot namely Robotic DDF Go system; (3) To invite the professional Go players to attend the activity to play Go games on site with a smart machine. The research team will apply this technology to education, such as, playing games to enhance the children concentration on learning mathematics, languages, and other topics. With the detected brainwaves, the robot will be able to speak some words that are very much to the point for the students and to assist the teachers in classroom in the future.

Enhancing EEG Signals Recognition Using ROC Curve

Mental tasks, such as calculation, reasoning, motor imagery, etc., can be recognized by the pattern of electroencephalograph (EEG) signals. So EEG signal recognition plays an important role in brain-computer interaction (BCI). In this study, to enhance the ability of classifiers such as support vector machine (SVM), deep neural networks (DNN), k-nearest neighbor method (kNN), decision tree (DT), a feature extraction method is proposed using techniques of fast Fourier transform (FFT) and receiver operating characteristic (ROC) curve. In the proposed method, the raw EEG data was transformed into power spectrum of FFT at first, and then to find frequencies decided by area under curve (AUC) of ROC between the value of spectrums of different classes of metal tasks. Experiment results using benchmark data and BCI competition II data showed the effectiveness of the proposed method for all above classifiers.

Brain Computer Interfaced Single Key Omni Directional Pointing and Command System: a Screen Pointing Interface for Differently-abled Person

Biomedical signals have got very incredible reach to the people outside clinical domain as devices are developed that can be controlled using one’s imagination and this is the very booming area of research nowadays. An effective way of using Human Computer Interaction (HCI) is using the Brain Computer Interaction (BCI) and Electroencephalogram (EEG) which can extend the people’s access to use machines specifically for specially abled persons. Generally, EEG is a costly matter which is not every time affordable by any common person. Here in this proposed system, we tried to explore the possibility of single channel EEG through which a person can communicate to computer according to his/her neurosignal excitement which are classified as Attention, Meditation and Blink strength and then can be interpreted into effective Human Machine Interaction (HMI). The proposed algorithm here uses brainwaves (voluntary blinks as a control tool in BCI) detected using Neurosky Mindwave Mobile headset from the frontal lobe (because it offers EEG clarity since this is the forehead area with minimal hair). This proposed system can be extended for typing, gaming, entertainment as well as for daily life interaction purposes. It can also be extended as a non-verbal communication medium for specially abled persons. The experiment conducted using the proposed system revealed significant development in accessing computer and typing skill. We tried to explore scan keyboard which can be operated by neural signals captured by the Neurosky Mindwave Mobile Headset Single channel EEG Headset.

Learning Emotions EEG-based Recognition and Brain Activity: A Survey Study on BCI for Intelligent Tutoring System

Learners experience emotions in a variety of valence and arousal in learning, which impacts the cognitive process and the success of learning. Learning emotions research has a wide range of benefits from improving learning outcomes and experience in Intelligent Tutoring System (ITS), as well as increasing operation and work productivity. This survey reviews techniques that have been used to measure emotions and theories for modeling emotions. It investigates EEG-based Brain-Computer Interaction (BCI) of general and learning emotion recognition. The induction methods of learning emotions and related issues are also included and discussed. The survey concludes with challenges for further learning emotion research.

TiD—Introducing and Benchmarking an Event-Delivery System for Brain–Computer Interfaces

In this paper, we present and analyze an event distribution system for brain-computer interfaces. Events are commonly used to mark and describe incidents during an experiment and are therefore critical for later data analysis or immediate real-time processing. The presented approach, called Tools for brain-computer interaction interface D (TiD), delivers messages in XML format via a buslike system using transmission control protocol connections or shared memory. A dedicated server dispatches TiD messages to distributed or local clients. The TiD message is designed to be flexible and contains time stamps for event synchronization, whereas events describe incidents, which occur during an experiment. TiD was tested extensively toward stability and latency. The effect of an occurring event jitter was analyzed and benchmarked on a reference implementation under different conditions as gigabit and 100-Mb Ethernet or Wi-Fi with a different number of event receivers. A 3-dB signal attenuation, which occurs when averaging jitter influenced trials aligned by events, is starting to become visible at around 1-2 kHz in the case of a gigabit connection. Mean event distribution times across operating systems are ranging from 0.3 to 0.5ms for a gigabit network connection for 106 events. Results for other environmental conditions are available in this paper. References already using TiD for event distribution are provided showing the applicability of TiD for event delivery with distributed or local clients.

Design of deep convolutional networks for prediction of image rapid serial visual presentation events.

We report in this paper an investigation of convolutional neural network (CNN) models for target prediction in time-locked image rapid serial visual presentation (RSVP) experiment. We investigated CNN models with 11 different designs of convolution filters in capturing spatial and temporal correlations in EEG data. We showed that for both within-subject and cross-subject predictions, the CNN models outperform the state-of-the-art algorithms: Bayesian linear discriminant analysis (BLDA) and xDAWN spatial filtering and achieved >6% improvement. Among the 11 different CNN models, the global spatial filter and our proposed region of interest (ROI) achieved best performance. We also implemented the deconvolution network to show how we can visualize from activated hidden units for target/nontarget events learned by the ROI-CNN. Our study suggests that deep learning is a powerful tool for RSVP target prediction and the proposed model is applicable for general EEG-based classifications in brain computer interaction research. The code of this project is available at https://github.com/ZijingMao/ROICNN.

Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Data

Emotion recognition is an important field of research in Brain Computer Interactions. As technology and the understanding of emotions are advancing, there are growing opportunities for automatic emotion recognition systems. Neural networks are a family of statistical learning models inspired by biological neural networks and are used to estimate functions that can depend on a large number of inputs that are generally unknown. In this paper we seek to use this effectiveness of Neural Networks to classify user emotions using EEG signals from the DEAP (Koelstra et al (2012)) dataset which represents the benchmark for Emotion classification research. We explore 2 different Neural Models, a simple Deep Neural Network and a Convolutional Neural Network for classification. Our model provides the state-of-the-art classification accuracy, obtaining 4.51 and 4.96 percentage point improvements over (Rozgic et al (2013)) classification of Valence and Arousal into 2 classes (High and Low) and 13.39 and 6.58 percentage point improvements over (Chung and Yoon(2012)) classification of Valence and Arousal into 3 classes (High, Normal and Low). Moreover our research is a testament that Neural Networks could be robust classifiers for brain signals, even outperforming traditional learning techniques.

BHCR: RSVP target retrieval BCI framework coupling with CNN by a Bayesian method

To combine the complementary strengths of human vision (HV) and computer vision (CV) in target image retrieval, we proposed a brain-computer interface framework, Bayesian HV-CV Retrieval (BHCR), which couples HV with CV by a Bayesian method to retrieve target images in rapid serial visual presentation (RSVP) sequences. To construct a well-suited electroencephalogram (EEG) decoding module for BHCR, we conducted a comparative inspection on the selection of classification algorithms, and adopted linear discriminant analysis and random forests as a feature extraction method and classification algorithm, respectively. We also introduced a CV system based on convolutional neural network (CNN) as a component of BHCR. A Bayesian brain-computer interaction (BBCI) module was carefully designed so that for each presented image, a Bayesian model that takes HV insight as prior information and CV insights as sample information is built up to present retrieval results. Unlike existing HV-CV coupled works that usually require extra manual labor, BHCR directly enhanced retrieval performance with the help of CV insights. As an auxiliary work and a natural extension of BHCR, we then proposed a probability propagation scheme that incorporates EEG decoding insights to improve the CV system and a one-shot image database retrieval scheme. We demonstrated the effectiveness of BHCR by extensive experiments and simulations on both the entire framework and its sub-components. The results showed the following: (1) The performance of BHCR was significantly better than the EEG-only mechanism in both receiver operating characteristic (ROC) and classification aspects; (2) The robustness of BHCR was ensured by its process flow and the steady performances of its sub-components.

Discrimination of motor imagery patterns by electroencephalogram phase synchronization combined with frequency band energy

Central nerve signal evoked by thoughts can be directly used to control a robot or prosthetic devices without the involvement of the peripheral nerve and muscles. This is a new strategy of human-computer interaction. A method of electroencephalogram U+0028 EEG U+0029 phase synchronization combined with band energy was proposed to construct a feature vector for pattern recognition of brain-computer interaction based on EEG induced by motor imagery in this paper. rhythm and beta rhythm were first extracted from EEG by band pass filter and then the frequency band energy was calculated by the sliding time window; the instantaneous phase values were obtained using Hilbert transform and then the phase synchronization feature was calculated by the phase locking value U+0028 PLV U+0029 and the best time interval for extracting the phase synchronization feature was searched by the distribution of the PLV value in the time domain. Finally, discrimination of motor imagery patterns was performed by the support vector machine U+0028 SVM U+0029. The results showed that the phase synchronization feature more effective in 4s-7s and the correct classification rate was 91.4 U+0025. Compared with the results achieved by a single EEG feature related to motor imagery, the correct classification rate was improved by 3.5 and 4.3 percentage points by combining phase synchronization with band energy. These indicate that the proposed method is effective and it is expected that the study provides a way to improve the performance of the online real-time brain-computer interaction control system based on EEG related to motor imagery.

A Randomized Controlled Study of Neurofeedback for Chronic PTSD.

IntroductionBrain/Computer Interaction (BCI) devices are designed to alter neural signals and, thereby, mental activity. This study was a randomized, waitlist (TAU) controlled trial of a BCI, EEG neurofeedback training (NF), in patients with chronic PTSD to explore the capacity of NF to reduce PTSD symptoms and increase affect regulation capacities.Study Design52 individuals with chronic PTSD were randomized to either NF (n = 28) or waitlist (WL) (n = 24). They completed four evaluations, at baseline (T1), after week 6 (T2), at post-treatment (T3), and at one month follow up (T4). Assessment measures were:1. Traumatic Events Screening Inventory (T1); 2. the Clinician Administered PTSD Scale (CAPS; T1, T3, T4); 3. the Davidson Trauma Scale (DTS; T1-T4) and 4. the Inventory of Altered Self-Capacities (IASC; T1-T4). NF training occurred two times per week for 12 weeks and involved a sequential placement with T4 as the active site, P4 as the reference site.ResultsParticipants had experienced an average of 9.29 (SD = 2.90) different traumatic events. Post-treatment a significantly smaller proportion of NF (6/22, 27.3%) met criteria for PTSD than the WL condition (15/22, 68.2%), χ2 (n = 44, df = 1) = 7.38, p = .007. There was a significant treatment condition x time interaction (b = -10.45, t = -5.10, p< .001). Measures of tension reduction activities, affect dysregulation, and affect instability exhibited a significant Time x Condition interaction. The effect sizes of NF (d = -2.33 within, d = - 1.71 between groups) are comparable to those reported for the most effective evidence based treatments for PTSD.DiscussionCompared with the control group NF produced significant PTSD symptom improvement in individuals with chronic PTSD, as well as in affect regulation capacities. NF deserves further investigation for its potential to ameliorate PTSD and to improve affect regulation, and to clarify its mechanisms of action.

Design and Evaluation of Fusion Approach for Combining Brain and Gaze Inputs for Target Selection

Gaze-based interfaces and Brain-Computer Interfaces (BCIs) allow for hands-free human–computer interaction. In this paper, we investigate the combination of gaze and BCIs. We propose a novel selection technique for 2D target acquisition based on input fusion. This new approach combines the probabilistic models for each input, in order to better estimate the intent of the user. We evaluated its performance against the existing gaze and brain–computer interaction techniques. Twelve participants took part in our study, in which they had to search and select 2D targets with each of the evaluated techniques. Our fusion-based hybrid interaction technique was found to be more reliable than the previous gaze and BCI hybrid interaction techniques for 10 participants over 12, while being 29% faster on average. However, similarly to what has been observed in hybrid gaze-and-speech interaction, gaze-only interaction technique still provides the best performance. Our results should encourage the use of input fusion, as opposed to sequential interaction, in order to design better hybrid interfaces.

Invited Symposium I: Brain Computer Interaction

Invited Symposium I: Brain Computer Interaction

Classification of EEG Signal using Correlation Coefficient among Channels as Features Extraction Method

Objectives: In this paper, we have evaluated the effectiveness of classification of Electroencephalogram (EEG) signals using the correlation between channels as a method of features selection. Methods/Statistical Analysis: First data is broken sample wise, then correlation coefficient between channel pair for each sample is calculated. After that mean of the correlation coefficient of all channel pair for each class over all samples is calculated and in a similar manner, standard deviation from the mean is also calculated. For feature selection we have plotted a pair of the Gaussian curves between channels of two separate classes and choose those channels which give us lower misclassified area as features. Then these features are used for training purpose of Support Vector Machine (SVM). Findings: Most of the previous researches follow either signal processing approach or machine learning approach while we emphasized upon the nature of the signal propagation amongst the neurons. The basic idea behind the feature selection is taken from the way the signals propagate from one neuron to the other. In our work we assume that EEG signals follow the normal distribution and verify the fact using chi-square test. On applying SVM the accuracy of classification on testing data confirms that correlation among channels can be used for feature selection. Application/Improvements: The results can be improved by improving the pre-processing of EEG signals. It can be used to develop a Brain Computer Interaction (BCI) system.

EEG correlates of video game experience and user profile in motor-imagery-based brain–computer interaction

Through the use of brain---computer interfaces (BCIs), neurogames have become increasingly more advanced by incorporating immersive virtual environments and 3D worlds. However, training both the user and the system requires long and repetitive trials resulting in fatigue and low performance. Moreover, many users are unable to voluntarily modulate the amplitude of their brain activity to control the neurofeedback loop. In this study, we are focusing on the effect that gaming experience has in brain activity modulation as an attempt to systematically identify the elements that contribute to high BCI control and to be utilized in neurogame design. Based on the current literature, we argue that experienced gamers could have better performance in BCI training due to enhanced sensorimotor learning derived from gaming. To investigate this, two experimental studies were conducted with 20 participants overall, undergoing 3 BCI sessions, resulting in 88 EEG datasets. Results indicate (a) an effect from both demographic and gaming experience data to the activity patterns of EEG rhythms, and (b) increased gaming experience might not increase significantly performance, but it could provide faster learning for `Hardcore' gamers.

Motor priming in virtual reality can augment motor-imagery training efficacy in restorative brain-computer interaction: a within-subject analysis.

BackgroundThe use of Brain–Computer Interface (BCI) technology in neurorehabilitation provides new strategies to overcome stroke-related motor limitations. Recent studies demonstrated the brain's capacity for functional and structural plasticity through BCI. However, it is not fully clear how we can take full advantage of the neurobiological mechanisms underlying recovery and how to maximize restoration through BCI. In this study we investigate the role of multimodal virtual reality (VR) simulations and motor priming (MP) in an upper limb motor-imagery BCI task in order to maximize the engagement of sensory-motor networks in a broad range of patients who can benefit from virtual rehabilitation training.MethodsIn order to investigate how different BCI paradigms impact brain activation, we designed 3 experimental conditions in a within-subject design, including an immersive Multimodal Virtual Reality with Motor Priming (VRMP) condition where users had to perform motor-execution before BCI training, an immersive Multimodal VR condition, and a control condition with standard 2D feedback. Further, these were also compared to overt motor-execution. Finally, a set of questionnaires were used to gather subjective data on Workload, Kinesthetic Imagery and Presence.ResultsOur findings show increased capacity to modulate and enhance brain activity patterns in all extracted EEG rhythms matching more closely those present during motor-execution and also a strong relationship between electrophysiological data and subjective experience.ConclusionsOur data suggest that both VR and particularly MP can enhance the activation of brain patterns present during overt motor-execution. Further, we show changes in the interhemispheric EEG balance, which might play an important role in the promotion of neural activation and neuroplastic changes in stroke patients in a motor-imagery neurofeedback paradigm. In addition, electrophysiological correlates of psychophysiological responses provide us with valuable information about the motor and affective state of the user that has the potential to be used to predict MI-BCI training outcome based on user’s profile. Finally, we propose a BCI paradigm in VR, which gives the possibility of motor priming for patients with low level of motor control.

The effect of target and non-target similarity on neural classification performance: a boost from confidence.

Brain computer interaction (BCI) technologies have proven effective in utilizing single-trial classification algorithms to detect target images in rapid serial visualization presentation tasks. While many factors contribute to the accuracy of these algorithms, a critical aspect that is often overlooked concerns the feature similarity between target and non-target images. In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes. It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented. This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets. The resulting findings show that behavior is slower and less accurate when targets are presented together with similar non-targets; moreover, single-trial classification yielded high levels of misclassification when infrequent non-targets are included. Furthermore, we present an approach to mitigate the image misclassification. We use confidence measures to assess the quality of single-trial classification, and demonstrate that a system in which low confidence trials are reclassified through a secondary process can result in improved performance.

Bio-hybrid interfaces to study neuromorphic functionalities: New multidisciplinary evidences of cell viability on poly(anyline) (PANI), a semiconductor polymer with memristive properties

The interfacing of artificial devices with biological systems is a challenging field that crosses several disciplines ranging from fundamental research (biophysical chemistry, neurobiology, material and surface science) to frontier technological application (nanotechnology, bioelectronics). The memristor is the fourth fundamental circuit element, whose electrical properties favor applications in signal processing, neural networks, and brain–computer interactions and it represents a new frontier for technological applications in many fields including the nanotechnologies, bioelectronics and the biosensors. Using multidisciplinary approaches, covering surface science, cell biology and electrophysiology, we successfully implemented a living bio-hybrid system constituted by cells adhering to films of poly(aniline) (PANI), a semiconductor polymer having memristive properties assembled with polyelectrolytes. Here we tested whether the PANI devices could support survivor, adhesion and differentiation of several cell lines, including the neuron-like SHSY5Y cells. Moreover, we performed electrophysiology on these cells showing that the biophysical properties are retained with differences occurring in the recorded ion currents. Taken together, the cell viability here reported is the key requirement to design and develop a reliable functional memristor-based bio-hybrid able to mimic neuronal activity and plasticity.

Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective.

In a recent theoretical synthesis on the concept of engagement, Fredricks, Blumenfeld and Paris defined engagement by its multiple dimensions: behavioral, emotional and cognitive. They observed that individual types of engagement had not been studied in conjunction, and little information was available about interactions or synergy between the dimensions; consequently, more studies would contribute to creating finely tuned teaching interventions. Benefiting from the recent technological advances in neurosciences, this paper presents a recently developed methodology to gather and synchronize data on multidimensional engagement during learning tasks. The technique involves the collection of (a) electroencephalography, (b) electrodermal, (c) eye-tracking, and (d) facial emotion recognition data on four different computers. This led to synchronization issues for data collected from multiple sources. Post synchronization in specialized integration software gives researchers a better understanding of the dynamics between the multiple dimensions of engagement. For curriculum developers, these data could provide informed guidelines for achieving better instruction/learning efficiency. This technique also opens up possibilities in the field of brain-computer interactions, where adaptive learning or assessment environments could be developed.