Auditory Brain-computer Interface Research Articles

An auditory brain-computer interface based on selective attention to multiple tone streams.

In this study, we attempted to improve brain-computer interface (BCI) systems by means of auditory stream segregation in which alternately presented tones are perceived as sequences of various different tones (streams). A 3-class BCI using three tone sequences, which were perceived as three different tone streams, was investigated and evaluated. Each presented musical tone was generated by a software synthesizer. Eleven subjects took part in the experiment. Stimuli were presented to each user's right ear. Subjects were requested to attend to one of three streams and to count the number of target stimuli in the attended stream. In addition, 64-channel electroencephalogram (EEG) and two-channel electrooculogram (EOG) signals were recorded from participants with a sampling frequency of 1000 Hz. The measured EEG data were classified based on Riemannian geometry to detect the object of the subject's selective attention. P300 activity was elicited by the target stimuli in the segregated tone streams. In five out of eleven subjects, P300 activity was elicited only by the target stimuli included in the attended stream. In a 10-fold cross validation test, a classification accuracy over 80% for five subjects and over 75% for nine subjects was achieved. For subjects whose accuracy was lower than 75%, either the P300 was also elicited for nonattended streams or the amplitude of P300 was small. It was concluded that the number of selected BCI systems based on auditory stream segregation can be increased to three classes, and these classes can be detected by a single ear without the aid of any visual modality.

The challenge of controlling an auditory BCI in the case of severe motor disability

BackgroundThe locked-in syndrome (LIS), due to a lesion in the pons, impedes communication. This situation can also be met after some severe brain injury or in advanced Amyotrophic Lateral Sclerosis (ALS). In the most severe condition, the persons cannot communicate at all because of a complete oculomotor paralysis (Complete LIS or CLIS). This even prevents the detection of consciousness. Some studies suggest that auditory brain–computer interface (BCI) could restore a communication through a « yes–no» code.MethodsWe developed an auditory EEG-based interface which makes use of voluntary modulations of attention, to restore a yes–no communication code in non-responding persons. This binary BCI uses repeated speech sounds (alternating “yes” on the right ear and “no” on the left ear) corresponding to either frequent (short) or rare (long) stimuli. Users are instructed to pay attention to the relevant stimuli only. We tested this BCI with 18 healthy subjects, and 7 people with severe motor disability (3 “classical” persons with locked-in syndrome and 4 persons with ALS).ResultsWe report online BCI performance and offline event-related potential analysis. On average in healthy subjects, online BCI accuracy reached 86% based on 50 questions. Only one out of 18 subjects could not perform above chance level. Ten subjects had an accuracy above 90%. However, most patients could not produce online performance above chance level, except for two people with ALS who obtained 100% accuracy. We report individual event-related potentials and their modulation by attention. In addition to the classical P3b, we observed a signature of sustained attention on responses to frequent sounds, but in healthy subjects and patients with good BCI control only.ConclusionsAuditory BCI can be very well controlled by healthy subjects, but it is not a guarantee that it can be readily used by the target population of persons in LIS or CLIS. A conclusion that is supported by a few previous findings in BCI and should now trigger research to assess the reasons of such a gap in order to propose new and efficient solutions.Clinical trial registrations: No. NCT02567201 (2015) and NCT03233282 (2013).

An auditory brain-computer interface to detect changes in sound pressure level for automatic volume control

Volume control is necessary to adjust sound levels for a comfortable audio or video listening experience. This study aims to develop an automatic volume control system based on a brain-computer interface (BCI). We thus focused on a BCI using an auditory oddball paradigm, and conducted two types of experiments. In the first experiment, the participant was asked to pay attention to a target sound where the sound level was high (70 dB) compared with the other sounds (60 dB). The brain activity measured by electroencephalogram showed large positive activity (P300) for the target sound, and classification of the target and nontarget sounds achieved an accuracy of 0.90. The second experiment adopted a two-target paradigm where a low sound level (50 dB) was introduced as the second target sound. P300 was also observed in the second experiment, and a value of 0.76 was obtained for the binary classification of the target and nontarget sounds. Further, we found that better accuracy was observed in large sound levels compared to small ones. These results suggest the possibility of using BCI for automatic volume control; however, it is necessary to improve its accuracy for application in daily life.

EEG-based assessment of temporal fine structure and envelope effect in mandarin syllable and tone perception.

Abstract In recent years, speech perception research has benefited from low-frequency rhythm entrainment tracking of the speech envelope. However, speech perception is still controversial regarding the role of speech envelope and temporal fine structure, especially in Mandarin. This study aimed to discuss the dependence of Mandarin syllables and tones perception on the speech envelope and the temporal fine structure. We recorded the electroencephalogram (EEG) of the subjects under three acoustic conditions using the sound chimerism analysis, including (i) the original speech, (ii) the speech envelope and the sinusoidal modulation, and (iii) the fine structure of time and the modulation of the non-speech (white noise) sound envelope. We found that syllable perception mainly depended on the speech envelope, while tone perception depended on the temporal fine structure. The delta bands were prominent, and the parietal and prefrontal lobes were the main activated brain areas, regardless of whether syllable or tone perception was involved. Finally, we decoded the spatiotemporal features of Mandarin perception from the microstate sequence. The spatiotemporal feature sequence of the EEG caused by speech material was found to be specific, suggesting a new perspective for the subsequent auditory brain-computer interface. These results provided a new scheme for the coding strategy of new hearing aids for native Mandarin speakers. Highlights

Conformal in-ear bioelectronics for visual and auditory brain-computer interfaces

Brain-computer interfaces (BCIs) have attracted considerable attention in motor and language rehabilitation. Most devices use cap-based non-invasive, headband-based commercial products or microneedle-based invasive approaches, which are constrained for inconvenience, limited applications, inflammation risks and even irreversible damage to soft tissues. Here, we propose in-ear visual and auditory BCIs based on in-ear bioelectronics, named as SpiralE, which can adaptively expand and spiral along the auditory meatus under electrothermal actuation to ensure conformal contact. Participants achieve offline accuracies of 95% in 9-target steady state visual evoked potential (SSVEP) BCI classification and type target phrases successfully in a calibration-free 40-target online SSVEP speller experiment. Interestingly, in-ear SSVEPs exhibit significant 2nd harmonic tendencies, indicating that in-ear sensing may be complementary for studying harmonic spatial distributions in SSVEP studies. Moreover, natural speech auditory classification accuracy can reach 84% in cocktail party experiments. The SpiralE provides innovative concepts for designing 3D flexible bioelectronics and assists the development of biomedical engineering and neural monitoring.

Brain–computer interface in an inter-individual approach using spatial coherence: Identification of better channels and tests repetition using auditory selective attention

Auditory Brain–Computer Interfaces (BCIs) have been studied with the main purpose of improving the quality of life of totally paralyzed people. BCIs based on Auditory Selective Attention (ASA) may have distinct features, such as the number of sound sources and most approaches presented in literature use individualized settings. This setup requires more training of individuals or adjustments of the internal classifiers. In this context, a generalized approach can be an interesting alternative for an interface tailored to application and without the need for exhaustive training. The present work investigates ASA using stimuli with AM modulation and spatial coherence to evaluate its value in the modulating frequency and the electrodes position. The objective is to identify better combinations of electrodes and evaluate the number of repetitions and the intervals between them in an inter-individual approach. The best result obtained using the modular classifier, proposed in an earlier work, reached average hit rates of 75% and information transfer rate (ITR) of 2.217 bits/min, considering three windows (5.1 s). This result was obtained using AM stimuli (500 Hz and 2 kHz carriers) and a combination of mostly frontal and prefrontal electrodes and considering 5 s interval between repetitions. With these results, an implementation of a vision-free BCI that covers inter-individual differences could allow users to communicate through selective auditory attention without previous training of subjects.

Using Mental Shadowing Tasks to Improve the Sound-Evoked Potential of EEG in the Design of an Auditory Brain–Computer Interface

This study proposed an auditory stimulation protocol based on Shadowing Tasks to improve the sound-evoked potential in an EEG and the efficiency of an auditory brain–computer interface system. We use stories as auditory stimulation to enhance users’ motivation and presented the sound stimuli via headphones to enable the user to concentrate better on the keywords in the stories. The protocol presents target stimuli with an oddball P300 paradigm. To decline mental workload, we shift the usual Shadowing Tasks paradigm: Instead of loudly repeating the auditory target stimuli, we ask subjects to echo the target stimuli mentally as it occurs. Twenty-four healthy participants, not one of whom underwent a BCI use or training phase before the experimental procedure, ran twenty trials each. We analyzed the effect of the auditory stimulation based on the Shadowing Tasks theory with the performance of the auditory BCI system. We also evaluated the judgment effectiveness of the three ERPs components (N2P3, P300, and N200) from five chosen electrodes. The best average accuracy of post-analysis was 78.96%. Using component N2P3 to distinguish between target and non-target can improve the efficiency of the auditory BCI system and give it good practicality. We intend to persist in this study and involve the protocol in an aBCI-based home care system (HCS) for target patients to provide daily assistance.

Sound Target Detection Under Noisy Environment Using Brain-Computer Interface.

As an important means of environmental reconnaissance and regional security protection, sound target detection (STD) has been widely studied in the field of machine learning for a long time. Considering the shortcomings of the robustness and generalization performance of existing methods based on machine learning, we proposed a target detection method by an auditory brain-computer interface (BCI). We designed the experimental paradigm according to the actual application scenarios of STD, recorded the changes in Electroencephalogram (EEG) signals during the process of detecting target sound, and further extracted the features used to decode EEG signals through the analysis of neural representations, including Event-Related Potential (ERP) and Event-Related Spectral Perturbation (ERSP). Experimental results showed that the proposed method achieved good detection performance under noisy environment. As the first study of BCI applied to STD, this study shows the feasibility of this scheme in BCI and can serve as the foundation for future related applications.

Hybrid Brain–Computer Interface System Using Auditory Stimulation and Speech Imagination Paradigms for Assistive Technology

In the case of paralysis with visual and tactile impairments, brain–computer interfaces (BCIs) based on auditory and mental imagery paradigms are alternative methods for controlling external devices. This study demonstrates the use of a hybrid BCI via auditory stimulation and speech imagination for assistive technology. The proposed auditory BCI using Thai vowel and numeral stimulus patterns as well as multi-loudspeaker position settings for multi-command BCI are investigated. To avoid auditory stimulation during resting periods, a speech imagery method is used to enable an audio stimulator. We observe the classification efficiency of speech imagery and auditory BCIs from selected electroencephalogram channels using the proposed algorithms. We examine the efficiency of using the proposed BCI in the presence of background noise (speech). One command is created using the proposed speech-imagination paradigm. Four commands are created using the proposed auditory stimulation paradigm. We design an experiment to verify the proposed BCI paradigm and classification algorithms for real-time processing. The results show that the average classification accuracy of the proposed auditory BCI using numeral stimuli and scatter patterns without speech noise is 72.2% to 83.3%, respectively. The efficiency under background noise is approximately two times lower than that without background noise.

Simultaneous multiple-stimulus auditory brain–computer interface with semi-supervised learning and prior probability distribution tuning

Objective. Auditory brain–computer interfaces (BCIs) enable users to select commands based on the brain activity elicited by auditory stimuli. However, existing auditory BCI paradigms cannot increase the number of available commands without decreasing the selection speed, because each stimulus needs to be presented independently and sequentially under the standard oddball paradigm. To solve this problem, we propose a double-stimulus paradigm that simultaneously presents multiple auditory stimuli. Approach. For addition to an existing auditory BCI paradigm, the best discriminable sound was chosen following a subjective assessment. The new sound was located on the right-hand side and presented simultaneously with an existing sound from the left-hand side. A total of six sounds were used for implementing the auditory BCI with a 6 × 6 letter matrix. We employ semi-supervised learning (SSL) and prior probability distribution tuning to improve the accuracy of the paradigm. The SSL method involved updating of the classifier weights, and their prior probability distributions were adjusted using the following three types of distributions: uniform, empirical, and extended empirical (e-empirical). The performance was evaluated based on the BCI accuracy and information transfer rate (ITR). Main results. The double-stimulus paradigm resulted in a BCI accuracy of 67.89 ± 11.46% and an ITR of 2.67 ± 1.09 bits min−1, in the absence of SSL and with uniform distribution. The proposed combination of SSL with e-empirical distribution improved the BCI accuracy and ITR to 74.59 ± 12.12% and 3.37 ± 1.27 bits min−1, respectively. The event-related potential analysis revealed that contralateral and right-hemispheric dominances contributed to the BCI performance improvement. Significance. Our study demonstrated that a BCI based on multiple simultaneous auditory stimuli, incorporating SSL and e-empirical prior distribution, can increase the number of commands without sacrificing typing speed beyond the acceptable level of accuracy.

Design considerations for the auditory brain computer interface speller

Brain-computer interface (BCI) systems are designed to translate the measured brain signals into a set of instructions, with a purpose of interaction with outside world. Most common application of such systems is to provide aid to patients with limited physical capabilities, where the visual stimulation based BCI are most commonly used. However, in the most extreme situations, e.g. for patients with amyotrophic lateral sclerosis, visual stimulation approach is insufficient for high-level interaction due to the patient's limited eye movement capabilities. To overcome these limitations, we propose a novel design of the two-stage auditory BCI system which exposes a subject to a stream of letter utterances, such that subject is directly stimulated with letter pronunciation, decreasing the mental effort required by the subject. This is contrary to frequently used auditory spellers, which use various non-related sounds representing a specific letter (i.e. various instruments, natural sounds, etc.). Letter discrimination between targets and non-targets is performed with convolutional neural network, with convolutional spatial and temporal filters capable of efficient extraction of event related brain activity features. Second stage of the target letter discrimination is proposed in order to increase the accuracy and usability in the real-world situations, including the reduced stimuli set based on results from the first classification stage. The proposed BCI has been tested on ten healthy subjects, achieving on average spelling accuracy of 30% (max. 100%) and high information transfer rate of 2.38 bits/min (max. 8.14 bits/min), outperforming the state-of-the-art auditory BCI spellers.

Extracellular voltage thresholds for maximizing information extraction in primate auditory cortex: implications for a brain computer interface

Objective. Research by Oby (2016 J. Neural. Eng. 13 036009) demonstrated that the optimal threshold for extracting information from visual and motor cortices may differ from the optimal threshold for identifying single neurons via spike sorting methods. The optimal threshold for extracting information from auditory cortex has yet to be identified, nor has the optimal temporal scale for representing auditory cortical activity. Here, we describe a procedure to jointly optimize the extracellular threshold and bin size with respect to the decoding accuracy achieved by a linear classifier for a diverse set of auditory stimuli. Approach. We used linear multichannel arrays to record extracellular neural activity from the auditory cortex of awake squirrel monkeys passively listening to both simple and complex sounds. We executed a grid search of the coordinate space defined by the voltage threshold (in units of standard deviation) and the bin size (in units of milliseconds), and computed decoding accuracy at each point. Main results. The optimal threshold for information extraction was consistently near two standard deviations below the voltage trace mean, which falls significantly below the range of three to five standard deviations typically used as inputs to spike sorting algorithms in basic research and in brain-computer interface (BCI) applications. The optimal binwidth was minimized at the optimal voltage threshold, particularly for acoustic stimuli dominated by temporally dynamic features, indicating that permissive thresholding permits readout of cortical responses with temporal precision on the order of a few milliseconds. Significance. The improvements in decoding accuracy we observed for optimal readout parameters suggest that standard thresholding methods substantially underestimate the information present in auditory cortical spiking patterns. The fact that optimal thresholds were relatively low indicates that local populations of cortical neurons exhibit high temporal coherence that could be leveraged in service of future auditory BCI applications.

Speech stream segregation to control an ERP-based auditory BCI

Objective. The use of natural sounds in auditory brain–computer interfaces (BCI) has been shown to improve classification results and usability. Some auditory BCIs are based on stream segregation, in which the subjects must attend one audio stream and ignore the other(s); these streams include some kind of stimuli to be detected. In this work we focus on event-related potentials (ERP) and study whether providing intelligible content to each audio stream could help the users to better concentrate on the desired stream and so to better attend the target stimuli and to ignore the non-target ones. Approach. In addition to a control condition, two experimental conditions, based on the selective attention and the cocktail party effect, were tested using two simultaneous and spatialized audio streams: (a) the condition A2 consisted of an overlap of auditory stimuli (single syllables) on a background consisting of natural speech for each stream, (b) in condition A3, brief alterations of the natural flow of each speech were used as stimuli. Main results. The two experimental proposals improved the results of the control condition (single words as stimuli without a speech background) both in a cross validation analysis of the calibration part and in the online test. The analysis of the ERP responses also presented better discriminability for the two proposals in comparison to the control condition. The results of subjective questionnaires support the better usability of the first experimental condition. Significance. The use of natural speech as background improves the stream segregation in an ERP-based auditory BCI (with significant results in the performance metrics, the ERP waveforms, and in the preference parameter in subjective questionnaires). Future work in the field of ERP-based stream segregation should study the use of natural speech in combination with easily perceived but not distracting stimuli.

Effect of Distracting Background Speech in an Auditory Brain-Computer Interface.

Studies so far have analyzed the effect of distractor stimuli in different types of brain–computer interface (BCI). However, the effect of a background speech has not been studied using an auditory event-related potential (ERP-BCI), a convenient option when the visual path cannot be adopted by users. Thus, the aim of the present work is to examine the impact of a background speech on selection performance and user workload in auditory BCI systems. Eleven participants tested three conditions: (i) auditory BCI control condition, (ii) auditory BCI with a background speech to ignore (non-attentional condition), and (iii) auditory BCI while the user has to pay attention to the background speech (attentional condition). The results demonstrated that, despite no significant differences in performance, shared attention to auditory BCI and background speech required a higher cognitive workload. In addition, the P300 target stimuli in the non-attentional condition were significantly higher than those in the attentional condition for several channels. The non-attentional condition was the only condition that showed significant differences in the amplitude of the P300 between target and non-target stimuli. The present study indicates that background speech, especially when it is attended to, is an important interference that should be avoided while using an auditory BCI.

Semi-Supervised Learning for Auditory Event-Related Potential-Based Brain–Computer Interface

A brain-computer interface (BCI) is a communication tool that analyzes neural activity and relays the translated commands to carry out actions. In recent years, semi-supervised learning (SSL) has attracted attention for visual event-related potential (ERP)-based BCIs and motor-imagery BCIs as an effective technique that can adapt to the variations in patterns among subjects and trials. The applications of the SSL techniques are expected to improve the performance of auditory ERP-based BCIs as well. However, there is no conclusive evidence supporting the positive effect of SSL techniques on auditory ERP-based BCIs. If the positive effect could be verified, it will be helpful for the BCI community. In this study, we assessed the effects of SSL techniques on two public auditory BCI datasets-AMUSE and PASS2D-using the following machine learning algorithms: step-wise linear discriminant analysis, shrinkage linear discriminant analysis, spatial temporal discriminant analysis, and least-squares support vector machine. These backbone classifiers were firstly trained by labeled data and incrementally updated by unlabeled data in every trial of testing data based on SSL approach. Although a few data of the datasets were negatively affected, most data were apparently improved by SSL in all cases. The overall accuracy was logarithmically increased with every additional unlabeled data. This study supports the positive effect of SSL techniques and encourages future researchers to apply them to auditory ERP-based BCIs.

Convolutional Neural Network Transfer Learning Applied to the Affective Auditory P300-Based BCI

Brain-computer interface (BCI) enables us to interact with the external world via electroencephalography (EEG) signals. Recently, deep learning methods have been applied to the BCI to reduce the time required for recording training data. However, more evidence is required due to lack of comparison. To reveal more evidence, this study proposed a deep learning method named time-wise convolutional neural network (TWCNN), which was applied to a BCI dataset. In the evaluation, EEG data from a subject was classified utilizing previously recorded EEG data from other subjects. As a result, TWCNN showed the highest accuracy, which was significantly higher than the typically used classifier. The results suggest that the deep learning method may be useful to reduce the recording time of training data.

Decoding auditory attention from single-trial EEG for a high-efficiency brain-computer interface.

Brain-computer interface (BCI) systems enable humans to communicate with a machine in a non-verbal and covert way. Many past BCI designs used visual stimuli, due to the robustness of neural signatures evoked by visual input. However, these BCI systems can only be used when visual attention is available. This study proposes a new BCI design using auditory stimuli, decoding spatial attention from electroencephalography (EEG). Results show that this new approach can decode attention with a high accuracy (>75%) and has a high information transfer rate (>10 bits/min) compared to other auditory BCI systems. It also has the potential to allow decoding that does not depend on subject-specific training.

Auditory attention tracking states in a cocktail party environment can be decoded by deep convolutional neural networks

Objective. A deep convolutional neural network (CNN) is a method for deep learning (DL). It has a powerful ability to automatically extract features and is widely used in classification tasks with scalp electroencephalogram (EEG) signals. However, the small number of samples and low signal-to-noise ratio involved in scalp EEG with low spatial resolution constitute a limitation that might restrict potential brain-computer interface (BCI) applications that are based on the CNN model. In the present study, a novel CNN model with source-spatial feature images (SSFIs) as the input is proposed to decode auditory attention tracking states in a cocktail party environment. Approach. We first extract SSFIs using rhythm entropy and weighted minimum norm estimation. Next, we develop a CNN model with three convolutional layers. Furthermore, we estimate the performance of the proposed model via generalized performance, alternative models that deleted or replaced a model’s component, and loss curves. Finally, we use a deep transfer model with fine-tuning for a low (poor) behavioral performance group (L-group). Main results. Based on cortical activity reconstructions from the scalp EEGs, the classification accuracy (CA) of the proposed model is 80.4% (chance level: 52.5%), which is superior to that achieved by scalp EEG. Additionally, the performance of the proposed model is more stable when compared to alternative models that delete or replace specific model components. The proposed model identifies the difference between two auditory attention tracking states (successful versus unsuccessful) at an early stage with a short time window (250 ms after target offset). Furthermore, we propose a deep transfer learning model to improve the classification for the L-group. With this model, the CA of the L-group significantly increase by 5.3%. Significance. Our proposed model improves the performance of a decoder for auditory attention tracking, which could be suitable for relieving the difficulty with the attentional modulation of individual’s neural responses. It provides a novel communication channel with auditory cognitive BCI for patients with attention and hearing impairment.

How Does the Degree of Valence Influence Affective Auditory P300-Based BCIs?

A brain-computer interface (BCI) translates brain signals into commands for the control of devices and for communication. BCIs enable persons with disabilities to communicate externally. Positive and negative affective sounds have been introduced to P300-based BCIs; however, how the degree of valence (e.g., very positive or positive) influences the BCI has not been investigated. To further examine the influence of affective sounds in P300-based BCIs, we applied sounds with five degrees of valence to the P300-based BCI. The sound valence ranged from very negative to very positive, as determined by Scheffe's method. The effect of sound valence on the BCI was evaluated by waveform analyses, followed by the evaluation of offline stimulus-wise classification accuracy. As a result, the late component of P300 showed significantly higher point-biserial correlation coefficients in response to very positive and very negative sounds than in response to the other sounds. The offline stimulus-wise classification accuracy was estimated from a region-of-interest. The analysis showed that the very negative sound achieved the highest accuracy and the very positive sound achieved the second highest accuracy, suggesting that the very positive sound and the very negative sound may be required to improve the accuracy.

Neural mechanisms of training an auditory event-related potential task in a brain-computer interface context.

Effective use of brain-computer interfaces (BCIs) typically requires training. Improved understanding of the neural mechanisms underlying BCI training will facilitate optimisation of BCIs. The current study examined the neural mechanisms related to training for electroencephalography (EEG)-based communication with an auditory event-related potential (ERP) BCI. Neural mechanisms of training in 10 healthy volunteers were assessed with functional magnetic resonance imaging (fMRI) during an auditory ERP-based BCI task before (t1) and after (t5) three ERP-BCI training sessions outside the fMRI scanner (t2, t3, and t4). Attended stimuli were contrasted with ignored stimuli in the first-level fMRI data analysis (t1 and t5); the training effect was verified using the EEG data (t2-t4); and brain activation was contrasted before and after training in the second-level fMRI data analysis (t1 vs. t5). Training increased the communication speed from 2.9 bits/min (t2) to 4 bits/min (t4). Strong activation was found in the putamen, supplementary motor area (SMA), and superior temporal gyrus (STG) associated with attention to the stimuli. Training led to decreased activation in the superior frontal gyrus and stronger haemodynamic rebound in the STG and supramarginal gyrus. The neural mechanisms of ERP-BCI training indicate improved stimulus perception and reduced mental workload. The ERP task used in the current study showed overlapping activations with a motor imagery based BCI task from a previous study on the neural mechanisms of BCI training in the SMA and putamen. This suggests commonalities between the neural mechanisms of training for both BCI paradigms.