Imagined speech can be decoded from low- and cross-frequency intracranial EEG features

Timothée Proix,Brian N Pasley,Xing Tian,Werner K Doyle,Jaime Delgado Saa,Andy Christen,Luc H Arnal,Anne-Lise Giraud,Stephanie Martin,Pierre Mégevand,David Poeppel,Orrin Devinsky,Robert T Knight

doi:10.1038/s41467-021-27725-3

Abstract

Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits. While decoding overt speech has progressed, decoding imagined speech has met limited success, mainly because the associated neural signals are weak and variable compared to overt speech, hence difficult to decode by learning algorithms. We obtained three electrocorticography datasets from 13 patients, with electrodes implanted for epilepsy evaluation, who performed overt and imagined speech production tasks. Based on recent theories of speech neural processing, we extracted consistent and specific neural features usable for future brain computer interfaces, and assessed their performance to discriminate speech items in articulatory, phonetic, and vocalic representation spaces. While high-frequency activity provided the best signal for overt speech, both low- and higher-frequency power and local cross-frequency contributed to imagined speech decoding, in particular in phonetic and vocalic, i.e. perceptual, spaces. These findings show that low-frequency power and cross-frequency dynamics contain key information for imagined speech decoding.

Highlights

Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits
While the motor hypothesis posits that imagined speech is essentially an attenuated version of overt speech with a wellspecified articulatory plan, the abstraction hypothesis proposes that it arises from higher-level linguistic representations that can be evoked without an explicit motor plan[10,26,27,28,29,30]
We examined the neural processes underlying the production of overt and imagined speech, in order to identify features that could be used for decoding imagined speech, having in sight a potential future application to severe speech production deficits

Summary

Introduction

Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits. One approach to decode imagined speech is to train algorithms on neural activity corresponding to articulatory motor commands produced during overt or silently articulated speech, hoping that the learned features could be transferred to patients who are unable to speak[2,3,4,5] Potentially interesting, this hypothesis is limited in scope as it can only apply to cases where language and cortical motor commands are preserved (such as in motor neuron disease), i.e. a minority of the patients with severe speech production deficits[6,7]. An important argument for the flexible abstraction hypothesis is that silently articulated speech exhibits the phonemic similarity effect (i.e. errors involving more similar phonemes are more likely), whereas imagined speech without explicit mouthing does not[26] These theories suggest that perceptual spaces, in particular auditory/phonetics, deserve as much attention as the articulatory dimension in imagined speech decoding. These frequencyspecific neural signals could be of particular importance for intended speech decoding, as focal articulatory signals indexed by BHA are expected to be notably weaker during imagined speech

Methods

Results

Conclusion