Speech Brain-computer Interface Research Articles

Imagined speech increases the hemodynamic response and functional connectivity of the dorsal motor cortex

Objective. Decoding imagined speech from brain signals could provide a more natural, user-friendly way for developing the next generation of the brain–computer interface (BCI). With the advantages of non-invasive, portable, relatively high spatial resolution and insensitivity to motion artifacts, the functional near-infrared spectroscopy (fNIRS) shows great potential for developing the non-invasive speech BCI. However, there is a lack of fNIRS evidence in uncovering the neural mechanism of imagined speech. Our goal is to investigate the specific brain regions and the corresponding cortico-cortical functional connectivity features during imagined speech with fNIRS. Approach. fNIRS signals were recorded from 13 subjects’ bilateral motor and prefrontal cortex during overtly and covertly repeating words. Cortical activation was determined through the mean oxygen–hemoglobin concentration changes, and functional connectivity was calculated by Pearson’s correlation coefficient. Main results. (a) The bilateral dorsal motor cortex was significantly activated during the covert speech, whereas the bilateral ventral motor cortex was significantly activated during the overt speech. (b) As a subregion of the motor cortex, sensorimotor cortex (SMC) showed a dominant dorsal response to covert speech condition, whereas a dominant ventral response to overt speech condition. (c) Broca’s area was deactivated during the covert speech but activated during the overt speech. (d) Compared to overt speech, dorsal SMC(dSMC)-related functional connections were enhanced during the covert speech. Significance. We provide fNIRS evidence for the involvement of dSMC in speech imagery. dSMC is the speech imagery network’s key hub and is probably involved in the sensorimotor information processing during the covert speech. This study could inspire the BCI community to focus on the potential contribution of dSMC during speech imagery.

Analysis and Classification of Speech Imagery EEG Based on Chinese Initials

Brain-computer interfaces (BCI) can provide external information communication for people with normal thinking but impaired motor functions. For patients with language disorders, speech imagery BCIs make it possible to communicate normally. However, there are few studies on Chinese speech imagery at present. Almost all studies employed fixed experimental content, without considering the diversity in subjects. With the purpose of improving the effect of a Chinese speech imagery BCI system, a novel experiment of Chinese initials imagery was designed. The experiment is divided into two parts. A preliminary experiment used to select content for subjects. Formal experiment-specific experimental content was designed for subjects. After preprocessing, feature extraction was carried out by common spatial patterns (CSP) and discrete wavelet transform (DWT), and then a support vector machine (SVM) and extreme learning machine (ELM) were used for classification. Finally, the best performance was obtained by the model using DWT and ELM with a highest accuracy of 73.04%. This study shows that the novel experiment is feasible and can potentially extend the capability of utilizing speech imagery in future BCI applications.

EEG based direct speech BCI system using a fusion of SMRT and MFCC/LPCC features with ANN classifier

Brain computer interface (BCI) technology has a great deal of scientific interest with various application systems. An advancement that is increasingly relevant in the BCI is communicating with a speech in mind. Thus, the paper aims to develop a direct speech BCI (DS-BCI) system using short time-based features and an ANN classifier. Signal processing methods like Mel frequency cepstral coefficients (MFCC), Linear predictive cepstral coefficients (LPCC) and Sequency mapped real transform (SMRT) are utilized on a short time bases to extract base level features. Statistical parameters are then determined based on the ensemble average (EA) and time average (TA) to extract two reduced vectors in each method. Hybrid feature vectors like MLC, SMC and SLC are prepared by fusion of features from MFCC & LPCC, SMRT & MFCC and SMRT & LPCC, respectively, in both EA & TA analysis. Principal component analysis (PCA) is performed on hybrid feature vectors to derive uncorrelated components. The proposed method is evaluated on imagined EEG (EEG-i) & vocalized EEG (EEG-v) signals from the ‘Kara one’ database and presented classification accuracy of individual methods & hybrid methods. The results show that hybrid features SMC & SLC enhance the classification accuracy compared to the unique features. PCA analysis helps improve accuracy and reduce feature dimension. TA-based SMC features with PCA provide maximum accuracy as 77.37% and 62.52% for EEG-i and EEG-v signals, respectively. The proposed method outperforms the state-of-the-art algorithms discussed in the paper.

Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus

Objective. To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the ‘hand knob’ area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. Approach. Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode’s binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the ‘Brain-to-Speech’ pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes’ onset times. Main results. A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. Significance. The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.

Multiclass covert speech classification using extreme learning machine.

The objective of the proposed research is to classify electroencephalography (EEG) data of covert speech words. Six subjects were asked to perform covert speech tasks i.e mental repetition of four different words i.e 'left', 'right', 'up' and 'down'. Fifty trials for each word recorded for every subject. Kernel-based Extreme Learning Machine (kernel ELM) was used for multiclass and binary classification of EEG signals of covert speech words. We achieved a maximum multiclass and binary classification accuracy of (49.77%) and (85.57%) respectively. The kernel ELM achieves significantly higher accuracy compared to some of the most commonly used classification algorithms in Brain-Computer Interfaces (BCIs). Our findings suggested that covert speech EEG signals could be successfully classified using kernel ELM. This research involving the classification of covert speech words potentially leading to real-time silent speech BCI research.

Control and Ownership of Neuroprosthetic Speech

Implantable brain-computer interfaces (BCIs) are being developed to restore speech capacity for those who are unable to speak. Patients with locked-in syndrome or amyotrophic lateral sclerosis could be able to use covert speech – vividly imagining saying something without actual vocalisation – to trigger neural controlled systems capable of synthesising speech. User control has been identified as particularly pressing for this type of BCI. The incorporation of machine learning and statistical language models into the decoding process introduces a contribution to (or ‘shaping of’) the output that is beyond the user’s control. Whilst this type of ‘shared control’ of BCI action is not unique to speech BCIs, the automated shaping of what a user ‘says’ has a particularly acute ethical dimension, which may differ from parallel concerns surrounding automation in movement BCIs. This paper provides an analysis of the control afforded to the user of a speech BCI of the sort under development, as well as the relationships between accuracy, control, and the user’s ownership of the speech produced. Through comparing speech BCIs with BCIs for movement, we argue that, whilst goal selection is the more significant locus of control for the user of a movement BCI, control over process will be more significant for the user of the speech BCI. The design of the speech BCI may therefore have to trade off some possible efficiency gains afforded by automation in order to preserve sufficient guidance control necessary for users to express themselves in ways they prefer. We consider the implications for the speech BCI user’s responsibility for produced outputs and their ownership of token outputs. We argue that these are distinct assessments. Ownership of synthetic speech concerns whether the content of the output sufficiently represents the user, rather than their morally relevant, causal role in producing that output.

Motor-Induced Suppression of the N100 Event-Related Potential During Motor Imagery Control of a Speech Synthesizer Brain-Computer Interface.

Purpose Speech motor control relies on neural processes for generating sensory expectations using an efference copy mechanism to maintain accurate productions. The N100 auditory event-related potential (ERP) has been identified as a possible neural marker of the efference copy with a reduced amplitude during active listening while speaking when compared to passive listening. This study investigates N100 suppression while controlling a motor imagery speech synthesizer brain-computer interface (BCI) with instantaneous auditory feedback to determine whether similar mechanisms are used for monitoring BCI-based speech output that may both support BCI learning through existing speech motor networks and be used as a clinical marker for the speech network integrity in individuals without severe speech and physical impairments. Method The motor-induced N100 suppression is examined based on data from 10 participants who controlled a BCI speech synthesizer using limb motor imagery. We considered listening to auditory target stimuli (without motor imagery) in the BCI study as passive listening and listening to BCI-controlled speech output (with motor imagery) as active listening since audio output depends on imagined movements. The resulting ERP was assessed for statistical significance using a mixed-effects general linear model. Results Statistically significant N100 ERP amplitude differences were observed between active and passive listening during the BCI task. Post hoc analyses confirm the N100 amplitude was suppressed during active listening. Conclusion Observation of the N100 suppression suggests motor planning brain networks are active as participants control the BCI synthesizer, which may aid speech BCI mastery.

Towards reconstructing intelligible speech from the human auditory cortex

Auditory stimulus reconstruction is a technique that finds the best approximation of the acoustic stimulus from the population of evoked neural activity. Reconstructing speech from the human auditory cortex creates the possibility of a speech neuroprosthetic to establish a direct communication with the brain and has been shown to be possible in both overt and covert conditions. However, the low quality of the reconstructed speech has severely limited the utility of this method for brain-computer interface (BCI) applications. To advance the state-of-the-art in speech neuroprosthesis, we combined the recent advances in deep learning with the latest innovations in speech synthesis technologies to reconstruct closed-set intelligible speech from the human auditory cortex. We investigated the dependence of reconstruction accuracy on linear and nonlinear (deep neural network) regression methods and the acoustic representation that is used as the target of reconstruction, including auditory spectrogram and speech synthesis parameters. In addition, we compared the reconstruction accuracy from low and high neural frequency ranges. Our results show that a deep neural network model that directly estimates the parameters of a speech synthesizer from all neural frequencies achieves the highest subjective and objective scores on a digit recognition task, improving the intelligibility by 65% over the baseline method which used linear regression to reconstruct the auditory spectrogram. These results demonstrate the efficacy of deep learning and speech synthesis algorithms for designing the next generation of speech BCI systems, which not only can restore communications for paralyzed patients but also have the potential to transform human-computer interaction technologies.

The Potential for a Speech Brain-Computer Interface Using Chronic Electrocorticography.

A brain-computer interface (BCI) is a technology that uses neural features to restore or augment the capabilities of its user. A BCI for speech would enable communication in real time via neural correlates of attempted or imagined speech. Such a technology would potentially restore communication and improve quality of life for locked-in patients and other patients with severe communication disorders. There have been many recent developments in neural decoders, neural feature extraction, and brain recording modalities facilitating BCI for the control of prosthetics and in automatic speech recognition (ASR). Indeed, ASR and related fields have developed significantly over the past years, and many lend many insights into the requirements, goals, and strategies for speech BCI. Neural speech decoding is a comparatively new field but has shown much promise with recent studies demonstrating semantic, auditory, and articulatory decoding using electrocorticography (ECoG) and other neural recording modalities. Because the neural representations for speech and language are widely distributed over cortical regions spanning the frontal, parietal, and temporal lobes, the mesoscopic scale of population activity captured by ECoG surface electrode arrays may have distinct advantages for speech BCI, in contrast to the advantages of microelectrode arrays for upper-limb BCI. Nevertheless, there remain many challenges for the translation of speech BCIs to clinical populations. This review discusses and outlines the current state-of-the-art for speech BCI and explores what a speech BCI using chronic ECoG might entail.

Online classification of imagined speech using functional near-infrared spectroscopy signals

Objective. Most brain–computer interfaces (BCIs) based on functional near-infrared spectroscopy (fNIRS) require that users perform mental tasks such as motor imagery, mental arithmetic, or music imagery to convey a message or to answer simple yes or no questions. These cognitive tasks usually have no direct association with the communicative intent, which makes them difficult for users to perform. Approach. In this paper, a 3-class intuitive BCI is presented which enables users to directly answer yes or no questions by covertly rehearsing the word ‘yes’ or ‘no’ for 15 s. The BCI also admits an equivalent duration of unconstrained rest which constitutes the third discernable task. Twelve participants each completed one offline block and six online blocks over the course of two sessions. The mean value of the change in oxygenated hemoglobin concentration during a trial was calculated for each channel and used to train a regularized linear discriminant analysis (RLDA) classifier. Main results. By the final online block, nine out of 12 participants were performing above chance (p < 0.001 using the binomial cumulative distribution), with a 3-class accuracy of 83.8% ± 9.4%. Even when considering all participants, the average online 3-class accuracy over the last three blocks was 64.1 % ± 20.6%, with only three participants scoring below chance (p < 0.001). For most participants, channels in the left temporal and temporoparietal cortex provided the most discriminative information. Significance. To our knowledge, this is the first report of an online 3-class imagined speech BCI. Our findings suggest that imagined speech can be used as a reliable activation task for selected users for development of more intuitive BCIs for communication.

Key considerations in designing a speech brain-computer interface

Key considerations in designing a speech brain-computer interface

Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication

The ultimate goal of brain-computer-interface (BCI) research on speech restoration is to develop devices which will be able to reconstruct spontaneous, naturally spoken language from the underlying neuronal signals. From this it follows that thorough understanding of brain activity and its functional dynamics during real-world speech will be required. Here, we review current developments in intracranial neurolinguistic and BCI research on speech production under increasingly naturalistic conditions. With an example of neurolinguistic data from our ongoing research, we illustrate the plausibility of neurolinguistic investigations in non-experimental, out-of-the-lab conditions of speech production. We argue that interdisciplinary endeavors at the interface of neuroscience and linguistics can provide valuable insight into the functional significance of speech-related neuronal data. Finally, we anticipate that work with neurolinguistic corpora composed of real-world language samples and simultaneous neuronal recordings, together with machine-learning methodology accounting for the specifics of the neurolinguistic material, will improve the functionality of speech BCIs.

Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces.

Restoring natural speech in paralyzed and aphasic people could be achieved using a Brain-Computer Interface (BCI) controlling a speech synthesizer in real-time. To reach this goal, a prerequisite is to develop a speech synthesizer producing intelligible speech in real-time with a reasonable number of control parameters. We present here an articulatory-based speech synthesizer that can be controlled in real-time for future BCI applications. This synthesizer converts movements of the main speech articulators (tongue, jaw, velum, and lips) into intelligible speech. The articulatory-to-acoustic mapping is performed using a deep neural network (DNN) trained on electromagnetic articulography (EMA) data recorded on a reference speaker synchronously with the produced speech signal. This DNN is then used in both offline and online modes to map the position of sensors glued on different speech articulators into acoustic parameters that are further converted into an audio signal using a vocoder. In offline mode, highly intelligible speech could be obtained as assessed by perceptual evaluation performed by 12 listeners. Then, to anticipate future BCI applications, we further assessed the real-time control of the synthesizer by both the reference speaker and new speakers, in a closed-loop paradigm using EMA data recorded in real time. A short calibration period was used to compensate for differences in sensor positions and articulatory differences between new speakers and the reference speaker. We found that real-time synthesis of vowels and consonants was possible with good intelligibility. In conclusion, these results open to future speech BCI applications using such articulatory-based speech synthesizer.

Key considerations in designing a speech brain-computer interface

Restoring communication in case of aphasia is a key challenge for neurotechnologies. To this end, brain-computer strategies can be envisioned to allow artificial speech synthesis from the continuous decoding of neural signals underlying speech imagination. Such speech brain-computer interfaces do not exist yet and their design should consider three key choices that need to be made: the choice of appropriate brain regions to record neural activity from, the choice of an appropriate recording technique, and the choice of a neural decoding scheme in association with an appropriate speech synthesis method. These key considerations are discussed here in light of (1) the current understanding of the functional neuroanatomy of cortical areas underlying overt and covert speech production, (2) the available literature making use of a variety of brain recording techniques to better characterize and address the challenge of decoding cortical speech signals, and (3) the different speech synthesis approaches that can be considered depending on the level of speech representation (phonetic, acoustic or articulatory) envisioned to be decoded at the core of a speech BCI paradigm.

Neural distance amplification of lexical tone in human auditory cortex.

In tonal languages, like Chinese, lexical tone serves as a key feature to provide contrast in word meaning. Behavior studies suggest that Mandarin Chinese tone is categorically perceived. However, the neural mechanism underlying Mandarin tone perception is still poorly understood. In this study, an Oddball paradigm was designed by selecting two standard-deviant stimulus pairs with same physical distance but different category labels, among the synthesized tones with continuously varying pitch contours. Using electrocorticography (ECoG) recording over human auditory cortex, high temporal and spatial resolution cortical neural signals were used for the first time to investigate the cortical processing of lexical tone. Here, we found different neural responses to the two standard-deviant tone pairs, and the difference increased from low to high level along the hierarchy of human auditory cortex. In the two dimensional neural space, cross-category neural distance of lexical tones is selectively amplified on those high level electrodes. These findings support a hierarchical and categorical model of Mandarin tone perception, and favor the using of high-level electrodes for a better performance of lexical tone discrimination in speech brain computer interface.