Neural Speech Processing Research Articles

Hemodynamic responses to emotional speech in two-month-old infants imaged using diffuse optical tomography

Emotional speech is one of the principal forms of social communication in humans. In this study, we investigated neural processing of emotional speech (happy, angry, sad and neutral) in the left hemisphere of 21 two-month-old infants using diffuse optical tomography. Reconstructed total hemoglobin (HbT) images were analysed using adaptive voxel-based clustering and region-of-interest (ROI) analysis. We found a distributed happy > neutral response within the temporo-parietal cortex, peaking in the anterior temporal cortex; a negative HbT response to emotional speech (the average of the emotional speech conditions < baseline) in the temporo-parietal cortex, neutral > angry in the anterior superior temporal sulcus (STS), happy > angry in the superior temporal gyrus and posterior superior temporal sulcus, angry < baseline in the insula, superior temporal sulcus and superior temporal gyrus and happy < baseline in the anterior insula. These results suggest that left STS is more sensitive to happy speech as compared to angry speech, indicating that it might play an important role in processing positive emotions in two-month-old infants. Furthermore, happy speech (relative to neutral) seems to elicit more activation in the temporo-parietal cortex, thereby suggesting enhanced sensitivity of temporo-parietal cortex to positive emotional stimuli at this stage of infant development.

Proactive neural processing of native and non-native speech

The impact of attention and language-experience in neural speech processing is typically assessed using sentences, words, and syllables that, when presented in isolation, may not engage the cortical speech network as well as more realistic continuous speech. Here, we explore the neuromodulatory effects of attention and language experience using continuous speech. We recorded electroencephalographic responses from native speakers of English and late Chinese-English bilinguals while listening to a story recorded in English. The story was mixed with a tone sequence, and listeners were instructed to focus either on the speech (attended speech condition) or the tone sequence (ignored speech condition). We used the multivariate temporal response function and the accuracy of a machine-learning based brain-to-speech decoder to quantify differences in cortical entrainment and speech-sound category processing. Our analyses revealed a more robust, context-independent neural encoding of speech-sound categories when they were attended and native. Interestingly, while cortical entrainment to speech was also enhanced by attention, the enhancement was stronger among non-native speakers. Our results suggest that while listeners can manage attention to improve the neural parsing of continuous speech via cortical entrainment, the benefits of attention for speech-sound category processing can be attenuated when the sounds are not native.The impact of attention and language-experience in neural speech processing is typically assessed using sentences, words, and syllables that, when presented in isolation, may not engage the cortical speech network as well as more realistic continuous speech. Here, we explore the neuromodulatory effects of attention and language experience using continuous speech. We recorded electroencephalographic responses from native speakers of English and late Chinese-English bilinguals while listening to a story recorded in English. The story was mixed with a tone sequence, and listeners were instructed to focus either on the speech (attended speech condition) or the tone sequence (ignored speech condition). We used the multivariate temporal response function and the accuracy of a machine-learning based brain-to-speech decoder to quantify differences in cortical entrainment and speech-sound category processing. Our analyses revealed a more robust, context-independent neural encoding of speech-sound categories when t...

Aging and hearing loss effects on neural speech processing

The effects of age-related hearing loss are pervasive, resulting in declines in social, emotional, and cognitive function. As the world becomes a quieter place, the decrease in sensory input from the auditory periphery may trigger homeostatic mechanisms to preserve a stable rate of neural firing at higher levels of the auditory system from brainstem to cortex. For example, decreased inhibitory neurotransmission increases neural excitability, preserving the sensation of loudness for moderate to moderately loud conversational speech levels. However, this change in the balance of excitatory and inhibitory neurotransmission may disrupt the brain’s ability to follow the rapid acoustic changes that are characteristic of running speech. Age-related disruptions in auditory processing of synthesized syllables and naturally-produced words and sentences have been demonstrated using electrophysiology (EEG) and magnetoencephalography (MEG). This presentation will review a series of EEG and MEG studies demonstrating effects of aging and/or hearing loss that vary depending on factors associated with type of hearing loss, stimulus choice, and primary neural source (midbrain versus cortex). Clinical implications for hearing loss management will be discussed. [Work supported by NIH-NIDCD, R21 DC015843-01.]

Restoration and Efficiency of the Neural Processing of Continuous Speech Are Promoted by Prior Knowledge.

Sufficiently noisy listening conditions can completely mask the acoustic signal of significant parts of a sentence, and yet listeners may still report the perception of hearing the masked speech. This occurs even when the speech signal is removed entirely, if the gap is filled with stationary noise, a phenomenon known as perceptual restoration. At the neural level, however, it is unclear the extent to which the neural representation of missing extended speech sequences is similar to the dynamic neural representation of ordinary continuous speech. Using auditory magnetoencephalography (MEG), we show that stimulus reconstruction, a technique developed for use with neural representations of ordinary speech, works also for the missing speech segments replaced by noise, even when spanning several phonemes and words. The reconstruction fidelity of the missing speech, up to 25% of what would be attained if present, depends however on listeners’ familiarity with the missing segment. This same familiarity also speeds up the most prominent stage of the cortical processing of ordinary speech by approximately 5 ms. Both effects disappear when listeners have no or little prior experience with the speech segment. The results are consistent with adaptive expectation mechanisms that consolidate detailed representations about speech sounds as identifiable factors assisting automatic restoration over ecologically relevant timescales.

The Cortical Maps of Hierarchical Linguistic Structures during Speech Perception.

The hierarchical nature of language requires human brain to internally parse connected-speech and incrementally construct abstract linguistic structures. Recent research revealed multiple neural processing timescales underlying grammar-based configuration of linguistic hierarchies. However, little is known about where in the whole cerebral cortex such temporally scaled neural processes occur. This study used novel magnetoencephalography source imaging techniques combined with a unique language stimulation paradigm to segregate cortical maps synchronized to 3 levels of linguistic units (i.e., words, phrases, and sentences). Notably, distinct ensembles of cortical loci were identified to feature structures at different levels. The superior temporal gyrus was found to be involved in processing all 3 linguistic levels while distinct ensembles of other brain regions were recruited to encode each linguistic level. Neural activities in the right motor cortex only followed the rhythm of monosyllabic words which have clear acoustic boundaries, whereas the left anterior temporal lobe and the left inferior frontal gyrus were selectively recruited in processing phrases or sentences. Our results ground a multi-timescale hierarchical neural processing of speech in neuroanatomical reality with specific sets of cortices responsible for different levels of linguistic units.

Linguistic effect on speech perception observed at the brainstem

Linguistic experience affects speech perception from early infancy, as previously evidenced by behavioral and brain measures. Current research focuses on whether linguistic effects on speech perception can be observed at an earlier stage in the neural processing of speech (i.e., auditory brainstem). Brainstem responses reflect rapid, automatic, and preattentive encoding of sounds. Positive experiential effects have been reported by examining the frequency-following response (FFR) component of the complex auditory brainstem response (cABR) in response to sustained high-energy periodic portions of speech sounds (vowels and lexical tones). The current study expands the existing literature by examining the cABR onset component in response to transient and low-energy portions of speech (consonants), employing simultaneous magnetoencephalography (MEG) in addition to electroencephalography (EEG), which provide complementary source information on cABR. Utilizing a cross-cultural design, we behaviorally measured perceptual responses to consonants in native Spanish- and English-speaking adults, in addition to cABR. Brain and behavioral relations were examined. Results replicated previous behavioral differences between language groups and further showed that individual consonant perception is strongly associated with EEG-cABR onset peak latency. MEG-cABR source analysis of the onset peaks complimented the EEG-cABR results by demonstrating subcortical sources for both peaks, with no group differences in peak locations. Current results demonstrate a brainstem-perception relation and show that the effects of linguistic experience on speech perception can be observed at the brainstem level.

Speech Intelligibility Predicted from Neural Entrainment of the Speech Envelope.

Speech intelligibility is currently measured by scoring how well a person can identify a speech signal. The results of such behavioral measures reflect neural processing of the speech signal, but are also influenced by language processing, motivation, and memory. Very often, electrophysiological measures of hearing give insight in the neural processing of sound. However, in most methods, non-speech stimuli are used, making it hard to relate the results to behavioral measures of speech intelligibility. The use of natural running speech as a stimulus in electrophysiological measures of hearing is a paradigm shift which allows to bridge the gap between behavioral and electrophysiological measures. Here, by decoding the speech envelope from the electroencephalogram, and correlating it with the stimulus envelope, we demonstrate an electrophysiological measure of neural processing of running speech. We show that behaviorally measured speech intelligibility is strongly correlated with our electrophysiological measure. Our results pave the way towards an objective and automatic way of assessing neural processing of speech presented through auditory prostheses, reducing confounds such as attention and cognitive capabilities. We anticipate that our electrophysiological measure will allow better differential diagnosis of the auditory system, and will allow the development of closed-loop auditory prostheses that automatically adapt to individual users.

Neural Processing of Congruent and Incongruent Audiovisual Speech in School‐Age Children and Adults

AbstractSeeing articulatory gestures enhances speech perception. Perception of auditory speech can even be changed by incongruent visual gestures, which is known as the McGurk effect (e.g., dubbing a voice saying /mi/ onto a face articulating /ni/, observers often hear /ni/). In children, the McGurk effect is weaker than in adults, but no previous knowledge exists about the neural‐level correlates of the McGurk effect in school‐age children. Using brain event‐related potentials, we investigated change detection responses to congruent and incongruent audiovisual speech in school‐age children and adults. We used an oddball paradigm with a congruent audiovisual /mi/ as the standard stimulus and a congruent audiovisual /ni/ or McGurk A/mi/V/ni/ as the deviant stimulus. In adults, a similar change detection response was elicited by both deviant stimuli. In children, change detection responses differed between the congruent and the McGurk stimulus. This reflects a maturational difference in the influence of visual stimuli on auditory processing.

Mild Cognitive Impairment Is Characterized by Deficient Brainstem and Cortical Representations of Speech.

Mild cognitive impairment (MCI) is recognized as a transitional phase in the progression toward more severe forms of dementia and is an early precursor to Alzheimer's disease. Previous neuroimaging studies reveal that MCI is associated with aberrant sensory-perceptual processing in cortical brain regions subserving auditory and language function. However, whether the pathophysiology of MCI extends to speech processing before conscious awareness (brainstem) is unknown. Using a novel electrophysiological approach, we recorded both brainstem and cortical speech-evoked brain event-related potentials (ERPs) in older, hearing-matched human listeners who did and did not present with subtle cognitive impairment revealed through behavioral neuropsychological testing. We found that MCI was associated with changes in neural speech processing characterized as hypersensitivity (larger) brainstem and cortical speech encoding in MCI compared with controls in the absence of any perceptual speech deficits. Group differences also interacted with age differentially across the auditory pathway; brainstem responses became larger and cortical ERPs smaller with advancing age. Multivariate classification revealed that dual brainstem-cortical speech activity correctly identified MCI listeners with 80% accuracy, suggesting its application as a biomarker of early cognitive decline. Brainstem responses were also a more robust predictor of individuals' MCI severity than cortical activity. Our findings suggest that MCI is associated with poorer encoding and transfer of speech signals between functional levels of the auditory system and advance the pathophysiological understanding of cognitive aging by identifying subcortical deficits in auditory sensory processing mere milliseconds (<10 ms) after sound onset and before the emergence of perceptual speech deficits.SIGNIFICANCE STATEMENT Mild cognitive impairment (MCI) is a precursor to dementia marked by declines in communication skills. Whether MCI pathophysiology extends below cerebral cortex to affect speech processing before conscious awareness (brainstem) is unknown. By recording neuroelectric brain activity to speech from brainstem and cortex, we show that MCI hypersensitizes the normal encoding of speech information across the hearing brain. Deficient neural responses to speech (particularly those generated from the brainstem) predicted the presence of MCI with high accuracy and before behavioral deficits. Our findings advance the neurological understanding of MCI by identifying a subcortical biomarker in auditory-sensory processing before conscious awareness, which may be a precursor to declines in speech understanding.

Can we Measure Speech Perception Ability Objectively in Young Children using Cochlear Implant?

Speech perception plays a significant role in the development of speech and language ability in children with hearing loss. There are various speech perception tests with the specific protocol to administer and score. Auditory evoked potential is an objective measure used for assessing hearing acuity in young children. Higher cortical evoked potential test provides an effective indication of the physiological status of auditory cortex. CEAP can be obtained in young children with assistive devices. The present research study aims to investigate whether latency and amplitude of aided cortical evoked potential can predict Speech perception score in cochlear implant users. 52 cochlear implant users and 102 typical children were tested. The mean age for children using cochlear implant was 8.05 years, standard deviation 1.4. The normal hearing group had a mean age of 8.12 years and standard deviation 1.46. Cochlear implant users were having at least 15 active electrodes. Speech stimulus was used with 100 ms duration, 20 ms rise/fall, 20 ms plateau tones. The inter-stimulus interval was kept at 1125 ms. During statistical analysis of data, CEAP P1 latencies and speech perception scores were found to be related. A further regression equation was obtained. The present study reveals speech stimuli can evoke distinct neural response patterns from auditory cortex. The present research study also helps in understanding the neural processing of speech in individuals with hearing impairment using Cochlear implant. Statistical analysis of data CEAP P1 latency and speech perception score regression equation were obtained. The result reveals that speech stimuli can evoke distinct neural response patterns from auditory cortex. CEAP finding support speech perception ability and auditory evoked potential help to the study of neural processing of speech in individuals with the cochlear implant. The evoked potential and speech perception ability has a significant relation between them.

Neurophysiological aspects of brainstem processing of speech stimuli in audiometric-normal geriatric population.

Poor auditory speech perception in geriatrics is attributable to neural de-synchronisation due to structural and degenerative changes of ageing auditory pathways. The speech-evoked auditory brainstem response may be useful for detecting alterations that cause loss of speech discrimination. Therefore, this study aimed to compare the speech-evoked auditory brainstem response in adult and geriatric populations with normal hearing. The auditory brainstem responses to click sounds and to a 40 ms speech sound (the Hindi phoneme |da|) were compared in 25 young adults and 25 geriatric people with normal hearing. The latencies and amplitudes of transient peaks representing neural responses to the onset, offset and sustained portions of the speech stimulus in quiet and noisy conditions were recorded. The older group had significantly smaller amplitudes and longer latencies for the onset and offset responses to |da| in noisy conditions. Stimulus-to-response times were longer and the spectral amplitude of the sustained portion of the stimulus was reduced. The overall stimulus level caused significant shifts in latency across the entire speech-evoked auditory brainstem response in the older group. The reduction in neural speech processing in older adults suggests diminished subcortical responsiveness to acoustically dynamic spectral cues. However, further investigations are needed to encode temporal cues at the brainstem level and determine their relationship to speech perception for developing a routine tool for clinical decision-making.

Individual differences in speech-in-noise perception parallel neural speech processing and attention in preschoolers

From bustling classrooms to unruly lunchrooms, school settings are noisy. To learn effectively in the unwelcome company of numerous distractions, children must clearly perceive speech in noise. In older children and adults, speech-in-noise perception is supported by sensory and cognitive processes, but the correlates underlying this critical listening skill in young children (3–5 year olds) remain undetermined. Employing a longitudinal design (two evaluations separated by ∼12 months), we followed a cohort of 59 preschoolers, ages 3.0–4.9, assessing word-in-noise perception, cognitive abilities (intelligence, short-term memory, attention), and neural responses to speech. Results reveal changes in word-in-noise perception parallel changes in processing of the fundamental frequency (F0), an acoustic cue known for playing a role central to speaker identification and auditory scene analysis. Four unique developmental trajectories (speech-in-noise perception groups) confirm this relationship, in that improvements and declines in word-in-noise perception couple with enhancements and diminishments of F0 encoding, respectively. Improvements in word-in-noise perception also pair with gains in attention. Word-in-noise perception does not relate to strength of neural harmonic representation or short-term memory. These findings reinforce previously-reported roles of F0 and attention in hearing speech in noise in older children and adults, and extend this relationship to preschool children.

Native language shapes automatic neural processing of speech

The development of the phoneme inventory is driven by the acoustic-phonetic properties of one's native language. Neural representation of speech is known to be shaped by language experience, as indexed by cortical responses, and recent studies suggest that subcortical processing also exhibits this attunement to native language. However, most work to date has focused on the differences between tonal and non-tonal languages that use pitch variations to convey phonemic categories. The aim of this cross-language study is to determine whether subcortical encoding of speech sounds is sensitive to language experience by comparing native speakers of two non-tonal languages (French and English). We hypothesized that neural representations would be more robust and fine-grained for speech sounds that belong to the native phonemic inventory of the listener, and especially for the dimensions that are phonetically relevant to the listener such as high frequency components. We recorded neural responses of American English and French native speakers, listening to natural syllables of both languages. Results showed that, independently of the stimulus, American participants exhibited greater neural representation of the fundamental frequency compared to French participants, consistent with the importance of the fundamental frequency to convey stress patterns in English. Furthermore, participants showed more robust encoding and more precise spectral representations of the first formant when listening to the syllable of their native language as compared to non-native language. These results align with the hypothesis that language experience shapes sensory processing of speech and that this plasticity occurs as a function of what is meaningful to a listener.

The relationship between spoken language and speech and nonspeech processing in children with autism: a magnetic event-related field study.

It has been proposed that language impairments in children with Autism Spectrum Disorders (ASD) stem from atypical neural processing of speech and/or nonspeech sounds. However, the strength of this proposal is compromised by the unreliable outcomes of previous studies of speech and nonspeech processing in ASD. The aim of this study was to determine whether there was an association between poor spoken language and atypical event-related field (ERF) responses to speech and nonspeech sounds in children with ASD (n=14) and controls (n=18). Data from this developmental population (ages 6-14) were analysed using a novel combination of methods to maximize the reliability of our findings while taking into consideration the heterogeneity of the ASD population. The results showed that poor spoken language scores were associated with atypical left hemisphere brain responses (200 to 400ms) to both speech and nonspeech in the ASD group. These data support the idea that some children with ASD may have an immature auditory cortex that affects their ability to process both speech and nonspeech sounds. Their poor speech processing may impair their ability to process the speech of other people, and hence reduce their ability to learn the phonology, syntax, and semantics of their native language.

Musical intervention enhances infants’ neural processing of temporal structure in music and speech

Individuals with music training in early childhood show enhanced processing of musical sounds, an effect that generalizes to speech processing. However, the conclusions drawn from previous studies are limited due to the possible confounds of predisposition and other factors affecting musicians and nonmusicians. We used a randomized design to test the effects of a laboratory-controlled music intervention on young infants' neural processing of music and speech. Nine-month-old infants were randomly assigned to music (intervention) or play (control) activities for 12 sessions. The intervention targeted temporal structure learning using triple meter in music (e.g., waltz), which is difficult for infants, and it incorporated key characteristics of typical infant music classes to maximize learning (e.g., multimodal, social, and repetitive experiences). Controls had similar multimodal, social, repetitive play, but without music. Upon completion, infants' neural processing of temporal structure was tested in both music (tones in triple meter) and speech (foreign syllable structure). Infants' neural processing was quantified by the mismatch response (MMR) measured with a traditional oddball paradigm using magnetoencephalography (MEG). The intervention group exhibited significantly larger MMRs in response to music temporal structure violations in both auditory and prefrontal cortical regions. Identical results were obtained for temporal structure changes in speech. The intervention thus enhanced temporal structure processing not only in music, but also in speech, at 9 mo of age. We argue that the intervention enhanced infants' ability to extract temporal structure information and to predict future events in time, a skill affecting both music and speech processing.

Music-based rehabilitation in cochlear implant users: Insights from brain-plasticity research

Music-based rehabilitation in cochlear implant users: Insights from brain-plasticity research

Interaction matters: A perceived social partner alters the neural processing of human speech

Mounting evidence suggests that social interaction changes how communicative behaviors (e.g., spoken language, gaze) are processed, but the precise neural bases by which social-interactive context may alter communication remain unknown. Various perspectives suggest that live interactions are more rewarding, more attention-grabbing, or require increased mentalizing—thinking about the thoughts of others. Dissociating between these possibilities is difficult because most extant neuroimaging paradigms examining social interaction have not directly compared live paradigms to conventional “offline” (or recorded) paradigms. We developed a novel fMRI paradigm to assess whether and how an interactive context changes the processing of speech matched in content and vocal characteristics. Participants listened to short vignettes—which contained no reference to people or mental states—believing that some vignettes were prerecorded and that others were presented over a real-time audio-feed by a live social partner. In actuality, all speech was prerecorded. Simply believing that speech was live increased activation in each participant’s own mentalizing regions, defined using a functional localizer. Contrasting live to recorded speech did not reveal significant differences in attention or reward regions. Further, higher levels of autistic-like traits were associated with altered neural specialization for live interaction. These results suggest that humans engage in ongoing mentalizing about social partners, even when such mentalizing is not explicitly required, illustrating how social context shapes social cognition. Understanding communication in social context has important implications for typical and atypical social processing, especially for disorders like autism where social difficulties are more acute in live interaction.

Engagement in community music classes sparks neuroplasticity and language development in children from disadvantaged backgrounds.

Children from disadvantaged backgrounds often face impoverished auditory environments, such as greater exposure to ambient noise and fewer opportunities to participate in complex language interactions during development. These circumstances increase their risk for academic failure and dropout. Given the academic and neural benefits associated with musicianship, music training may be one method for providing auditory enrichment to children from disadvantaged backgrounds. We followed a group of primary-school students from gang reduction zones in Los Angeles, CA, USA for 2 years as they participated in Harmony Project. By providing free community music instruction for disadvantaged children, Harmony Project promotes the healthy development of children as learners, the development of children as ambassadors of peace and understanding, and the development of stronger communities. Children who were more engaged in the music program—as defined by better attendance and classroom participation—developed stronger brain encoding of speech after 2 years than their less-engaged peers in the program. Additionally, children who were more engaged in the program showed increases in reading scores, while those less engaged did not show improvements. The neural gains accompanying music engagement were seen in the very measures of neural speech processing that are weaker in children from disadvantaged backgrounds. Our results suggest that community music programs such as Harmony Project provide a form of auditory enrichment that counteracts some of the biological adversities of growing up in poverty, and can further support community-based interventions aimed at improving child health and wellness.

Neural processing of speech in children is influenced by extent of bilingual experience

Language experience fine-tunes how the auditory system processes sound. Bilinguals, relative to monolinguals, have more robust evoked responses to speech that manifest as stronger neural encoding of the fundamental frequency (F0) and greater across-trial consistency. However, it is unknown whether such enhancements increase with increasing second language experience. We predict that F0 amplitude and neural consistency scale with dual-language experience during childhood, such that more years of bilingual experience leads to more robust F0 encoding and greater neural consistency. To test this hypothesis, we recorded auditory brainstem responses to the synthesized syllables ‘ba’ and ‘ga’ in two groups of bilingual children who were matched for age at test (8.4±0.67 years) but differed in their age of second language acquisition. One group learned English and Spanish simultaneously from birth (n=13), while the second group learned the two languages sequentially (n=15), spending on average their first four years as monolingual Spanish speakers. We find that simultaneous bilinguals have a larger F0 response to ‘ba’ and ‘ga’ and a more consistent response to ‘ba’ compared to sequential bilinguals and we demonstrate that these neural enhancements track with years of bilingual experience. These findings support the notion that bilingualism enhances subcortical auditory processing.

Auditory learning through active engagement with sound: biological impact of community music lessons in at-risk children.

The young nervous system is primed for sensory learning, facilitating the acquisition of language and communication skills. Social and linguistic impoverishment can limit these learning opportunities, eventually leading to language-related challenges such as poor reading. Music training offers a promising auditory learning strategy by directing attention to meaningful acoustic elements of the soundscape. In light of evidence that music training improves auditory skills and their neural substrates, there are increasing efforts to enact community-based programs to provide music instruction to at-risk children. Harmony Project is a community foundation that has provided free music instruction to over 1000 children from Los Angeles gang-reduction zones over the past decade. We conducted an independent evaluation of biological effects of participating in Harmony Project by following a cohort of children for 1 year. Here we focus on a comparison between students who actively engaged with sound through instrumental music training vs. students who took music appreciation classes. All children began with an introductory music appreciation class, but midway through the year half of the children transitioned to the instrumental training. After the year of training, the children who actively engaged with sound through instrumental music training had faster and more robust neural processing of speech than the children who stayed in the music appreciation class, observed in neural responses to a speech sound /d/. The neurophysiological measures found to be enhanced in the instrumentally-trained children have been previously linked to reading ability, suggesting a gain in neural processes important for literacy stemming from active auditory learning. Despite intrinsic constraints on our study imposed by a community setting, these findings speak to the potential of active engagement with sound (i.e., music-making) to engender experience-dependent neuroplasticity and may inform the development of strategies for auditory learning.