Noise-vocoded Speech Research Articles

Mapping cortical responses to speech using high-density diffuse optical tomography

The functional neuroanatomy of speech processing has been investigated using positron emission tomography (PET) and functional magnetic resonance imaging (fMRI) for more than 20years. However, these approaches have relatively poor temporal resolution and/or challenges of acoustic contamination due to the constraints of echoplanar fMRI. Furthermore, these methods are contraindicated because of safety concerns in longitudinal studies and research with children (PET) or in studies of patients with metal implants (fMRI). High-density diffuse optical tomography (HD-DOT) permits presenting speech in a quiet acoustic environment, has excellent temporal resolution relative to the hemodynamic response, and provides noninvasive and metal-compatible imaging. However, the performance of HD-DOT in imaging the brain regions involved in speech processing is not fully established. In the current study, we use an auditory sentence comprehension task to evaluate the ability of HD-DOT to map the cortical networks supporting speech processing. Using sentences with two levels of linguistic complexity, along with a control condition consisting of unintelligible noise-vocoded speech, we recovered a hierarchically organized speech network that matches the results of previous fMRI studies. Specifically, hearing intelligible speech resulted in increased activity in bilateral temporal cortex and left frontal cortex, with syntactically complex speech leading to additional activity in left posterior temporal cortex and left inferior frontal gyrus. These results demonstrate the feasibility of using HD-DOT to map spatially distributed brain networks supporting higher-order cognitive faculties such as spoken language.

The effects of hearing loss on neural processing and plasticity.

The effects of hearing loss on neural processing and plasticity.

Voice emotion recognition by cochlear-implanted children and their normally-hearing peers

Despite their remarkable success in bringing spoken language to hearing impaired listeners, the signal transmitted through cochlear implants (CIs) remains impoverished in spectro-temporal fine structure. As a consequence, pitch-dominant information such as voice emotion, is diminished. For young children, the ability to correctly identify the mood/intent of the speaker (which may not always be visible in their facial expression) is an important aspect of social and linguistic development. Previous work in the field has shown that children with cochlear implants (cCI) have significant deficits in voice emotion recognition relative to their normally hearing peers (cNH). Here, we report on voice emotion recognition by a cohort of 36 school-aged cCI. Additionally, we provide for the first time, a comparison of their performance to that of cNH and NH adults (aNH) listening to CI simulations of the same stimuli. We also provide comparisons to the performance of adult listeners with CIs (aCI), most of whom learned language primarily through normal acoustic hearing. Results indicate that, despite strong variability, on average, cCI perform similarly to their adult counterparts; that both groups' mean performance is similar to aNHs' performance with 8-channel noise-vocoded speech; that cNH achieve excellent scores in voice emotion recognition with full-spectrum speech, but on average, show significantly poorer scores than aNH with 8-channel noise-vocoded speech. A strong developmental effect was observed in the cNH with noise-vocoded speech in this task. These results point to the considerable benefit obtained by cochlear-implanted children from their devices, but also underscore the need for further research and development in this important and neglected area.This article is part of a Special Issue entitled <Lasker Award>.

Information-bearing acoustic changes are important for understanding vocoded speech in a simulation of cochlear implant processing strategies

Information-bearing acoustic changes (IBACs) in the speech signal are important for understanding speech. This was demonstrated with cochlea-scaled entropy for cochlear implants (CSECI), which measures perceptually significant intervals of noise-vocoded speech (Stilp et al. , 2013 JASA; Stilp, 2014 JASA; Stilp & Goupell, 2014 ASA). However, vocoding does not necessarily mimic CI processing. Some CI processing strategies present acoustic information in all channels at all times (e.g., CIS) while others present only the n-highest-amplitude channels out of m at any time (e.g., ACE). Here, IBACs were explored in a simulation of ACE processing. Sentences were divided into 22 channels spanning 188–7938 Hz and noise-vocoded. In each 1-ms interval (simulating 1000 pulses/second stimulation rate), only the eight highest-amplitude channels were retained. CSECI was calculated between 1-ms or 16-ms sentence segments, then summed into 80-ms intervals. High-CSECI or low-CSECI intervals were replaced by speech-shaped no...

Relationship between perceptual learning in speech and statistical learning in younger and older adults.

Within a few sentences, listeners learn to understand severely degraded speech such as noise-vocoded speech. However, individuals vary in the amount of such perceptual learning and it is unclear what underlies these differences. The present study investigates whether perceptual learning in speech relates to statistical learning, as sensitivity to probabilistic information may aid identification of relevant cues in novel speech input. If statistical learning and perceptual learning (partly) draw on the same general mechanisms, then statistical learning in a non-auditory modality using non-linguistic sequences should predict adaptation to degraded speech. In the present study, 73 older adults (aged over 60 years) and 60 younger adults (aged between 18 and 30 years) performed a visual artificial grammar learning task and were presented with 60 meaningful noise-vocoded sentences in an auditory recall task. Within age groups, sentence recognition performance over exposure was analyzed as a function of statistical learning performance, and other variables that may predict learning (i.e., hearing, vocabulary, attention switching control, working memory, and processing speed). Younger and older adults showed similar amounts of perceptual learning, but only younger adults showed significant statistical learning. In older adults, improvement in understanding noise-vocoded speech was constrained by age. In younger adults, amount of adaptation was associated with lexical knowledge and with statistical learning ability. Thus, individual differences in general cognitive abilities explain listeners' variability in adapting to noise-vocoded speech. Results suggest that perceptual and statistical learning share mechanisms of implicit regularity detection, but that the ability to detect statistical regularities is impaired in older adults if visual sequences are presented quickly.

The eye as a window to the listening brain: Neural correlates of pupil size as a measure of cognitive listening load

An important aspect of hearing is the degree to which listeners have to deploy effort to understand speech. One promising measure of listening effort is task-evoked pupil dilation. Here, we use functional magnetic resonance imaging (fMRI) to identify the neural correlates of pupil dilation during comprehension of degraded spoken sentences in 17 normal-hearing listeners. Subjects listened to sentences degraded in three different ways: the target female speech was masked by fluctuating noise, by speech from a single male speaker, or the target speech was noise-vocoded. The degree of degradation was individually adapted such that 50% or 84% of the sentences were intelligible. Control conditions included clear speech in quiet, and silent trials.The peak pupil dilation was larger for the 50% compared to the 84% intelligibility condition, and largest for speech masked by the single-talker masker, followed by speech masked by fluctuating noise, and smallest for noise-vocoded speech. Activation in the bilateral superior temporal gyrus (STG) showed the same pattern, with most extensive activation for speech masked by the single-talker masker. Larger peak pupil dilation was associated with more activation in the bilateral STG, bilateral ventral and dorsal anterior cingulate cortex and several frontal brain areas. A subset of the temporal region sensitive to pupil dilation was also sensitive to speech intelligibility and degradation type. These results show that pupil dilation during speech perception in challenging conditions reflects both auditory and cognitive processes that are recruited to cope with degraded speech and the need to segregate target speech from interfering sounds.

Top-down influences of written text on perceived clarity of degraded speech.

An unresolved question is how the reported clarity of degraded speech is enhanced when listeners have prior knowledge of speech content. One account of this phenomenon proposes top-down modulation of early acoustic processing by higher-level linguistic knowledge. Alternative, strictly bottom-up accounts argue that acoustic information and higher-level knowledge are combined at a late decision stage without modulating early acoustic processing. Here we tested top-down and bottom-up accounts using written text to manipulate listeners’ knowledge of speech content. The effect of written text on the reported clarity of noise-vocoded speech was most pronounced when text was presented before (rather than after) speech (Experiment 1). Fine-grained manipulation of the onset asynchrony between text and speech revealed that this effect declined when text was presented more than 120 ms after speech onset (Experiment 2). Finally, the influence of written text was found to arise from phonological (rather than lexical) correspondence between text and speech (Experiment 3). These results suggest that prior knowledge effects are time-limited by the duration of auditory echoic memory for degraded speech, consistent with top-down modulation of early acoustic processing by linguistic knowledge.

Lexico-semantic and acoustic-phonetic processes in the perception of noise-vocoded speech: implications for cochlear implantation

Noise-vocoding is a transformation which, when applied to speech, severely reduces spectral resolution and eliminates periodicity, yielding a stimulus that sounds “like a harsh whisper” (Scott et al., 2000, p. 2401). This process simulates a cochlear implant, where the activity of many thousand hair cells in the inner ear is replaced by direct stimulation of the auditory nerve by a small number of tonotopically-arranged electrodes. Although a cochlear implant offers a powerful means of restoring some degree of hearing to profoundly deaf individuals, the outcomes for spoken communication are highly variable (Moore and Shannon, 2009). Some variability may arise from differences in peripheral representation (e.g., the degree of residual nerve survival) but some may reflect differences in higher-order linguistic processing. In order to explore this possibility, we used noise-vocoding to explore speech recognition and perceptual learning in normal-hearing listeners tested across several levels of the linguistic hierarchy: segments (consonants and vowels), single words, and sentences. Listeners improved significantly on all tasks across two test sessions. In the first session, individual differences analyses revealed two independently varying sources of variability: one lexico-semantic in nature and implicating the recognition of words and sentences, and the other an acoustic-phonetic factor associated with words and segments. However, consequent to learning, by the second session there was a more uniform covariance pattern concerning all stimulus types. A further analysis of phonetic feature recognition allowed greater insight into learning-related changes in perception and showed that, surprisingly, participants did not make full use of cues that were preserved in the stimuli (e.g., vowel duration). We discuss these findings in relation cochlear implantation, and suggest auditory training strategies to maximize speech recognition performance in the absence of typical cues.

The effects of noise vocoding on speech quality perception

Speech perception depends on access to spectral and temporal acoustic cues. Temporal cues include slowly varying amplitude changes (i.e. temporal envelope, TE) and quickly varying amplitude changes associated with the center frequency of the auditory filter (i.e. temporal fine structure, TFS). This study quantifies the effects of TFS randomization through noise vocoding on the perception of speech quality by parametrically varying the amount of original TFS available above 1500Hz. The two research aims were: 1) to establish the role of TFS in quality perception, and 2) to determine if the role of TFS in quality perception differs between subjects with normal hearing and subjects with sensorineural hearing loss. Ratings were obtained from 20 subjects (10 with normal hearing and 10 with hearing loss) using an 11-point quality scale. Stimuli were processed in three different ways: 1) A 32-channel noise-excited vocoder with random envelope fluctuations in the noise carrier, 2) a 32-channel noise-excited vocoder with the noise-carrier envelope smoothed, and 3) removal of high-frequency bands. Stimuli were presented in quiet and in babble noise at 18dB and 12dB signal-to-noise ratios. TFS randomization had a measurable detrimental effect on quality ratings for speech in quiet and a smaller effect for speech in background babble. Subjects with normal hearing and subjects with sensorineural hearing loss provided similar quality ratings for noise-vocoded speech.

Evidence for cognitive restoration of time-reversed speech by a language-trained chimpanzee (Pan troglodytes)

Previously, we reported on the ability of Panzee, a language-trained chimpanzee, to identify sine-wave and noise-vocoded speech by attending to the amplitude and frequency modulations in the altered signals. Here, we report on her ability to perceive phoneme-length information in words reproduced in time-reversed form. While this manipulation preserves the amplitude of frequency components, it reverses the pattern of energy changes within each reversal window. Listeners easily recognize speech at reversal windows up to 100-ms length, but at longer reversal lengths unintelligibility begins to occur (Saberi and Perrott, 1999). The theoretical interpretation is that individual phonetic segments range from 50 to 100 ms (Crystal and House, 1988), and reversal-windows less than 100 ms provide for restoration of phoneme perception. Hypothesizing that Panzee also perceives speech based on phonemic segments, we tested her and humans with words in eight reversal forms ranging from 25 to 200 ms. Results revealed time-reversal window length significantly predicted percentage-correct word identification for Panzee and the humans. Additionally, window lengths exceeding 100 ms produced partial word intelligibility, with 50% intelligibility occurring at approximately 130 ms for both species. We therefore conclude that Panzee attends to phoneme-related cues in time-reversed speech, and hence in natural speech.

Cognitive control and its impact on recovery from aphasic stroke

Aphasic deficits are usually only interpreted in terms of domain-specific language processes. However, effective human communication and tests that probe this complex cognitive skill are also dependent on domain-general processes. In the clinical context, it is a pragmatic observation that impaired attention and executive functions interfere with the rehabilitation of aphasia. One system that is important in cognitive control is the salience network, which includes dorsal anterior cingulate cortex and adjacent cortex in the superior frontal gyrus (midline frontal cortex). This functional imaging study assessed domain-general activity in the midline frontal cortex, which was remote from the infarct, in relation to performance on a standard test of spoken language in 16 chronic aphasic patients both before and after a rehabilitation programme. During scanning, participants heard simple sentences, with each listening trial followed immediately by a trial in which they repeated back the previous sentence. Listening to sentences in the context of a listen–repeat task was expected to activate regions involved in both language-specific processes (speech perception and comprehension, verbal working memory and pre-articulatory rehearsal) and a number of task-specific processes (including attention to utterances and attempts to overcome pre-response conflict and decision uncertainty during impaired speech perception). To visualize the same system in healthy participants, sentences were presented to them as three-channel noise-vocoded speech, thereby impairing speech perception and assessing whether this evokes domain general cognitive systems. As expected, contrasting the more difficult task of perceiving and preparing to repeat noise-vocoded speech with the same task on clear speech demonstrated increased activity in the midline frontal cortex in the healthy participants. The same region was activated in the aphasic patients as they listened to standard (undistorted) sentences. Using a region of interest defined from the data on the healthy participants, data from the midline frontal cortex was obtained from the patients. Across the group and across different scanning sessions, activity correlated significantly with the patients’ communicative abilities. This correlation was not influenced by the sizes of the lesion or the patients’ chronological ages. This is the first study that has directly correlated activity in a domain general system, specifically the salience network, with residual language performance in post-stroke aphasia. It provides direct evidence in support of the clinical intuition that domain-general cognitive control is an essential factor contributing to the potential for recovery from aphasic stroke.

The Contribution of Matched Envelope Dynamic Range to the Binaural Benefits in Simulated Bilateral Electric Hearing

This study examined the effects of envelope dynamic-range mismatch on the intelligibility of Mandarin speech in noise by simulated bilateral electric hearing. Noise-vocoded Mandarin speech, corrupted by speech-shaped noise at 5 and 0 dB signal-to-noise ratios, was presented unilaterally or bilaterally to 10 normal-hearing listeners for recognition. For unilateral conditions, the right ear was presented with the 8-channel noise-vocoded stimuli generated using a 15-dB envelope dynamic range (DR). To simulate the envelope DR mismatch between the 2 ears, the left ear was presented with the 8-channel noise-vocoded stimuli generated using a 5-, 10-, or 15-dB envelope DR, respectively. Significant binaural summation benefits for Mandarin speech recognition were observed only with matched envelope DR between the 2 ears. With reduced DR, the performance of tone identification was more consistent in the steady-state speech-shaped noise than that of sentence recognition. Consistent with previous findings, the present results suggest that Mandarin speech-perception performance of bilateral electric listening in noise is affected by the difference of envelope DR between the 2 implanted ears, and the binaural summation benefits are maximized when DR mismatch is minimized between the 2 implanted ears.

Speech-Specific Tuning of Neurons in Human Superior Temporal Gyrus

How the brain extracts words from auditory signals is an unanswered question. We recorded approximately 150 single and multi-units from the left anterior superior temporal gyrus of a patient during multiple auditory experiments. Against low background activity, 45% of units robustly fired to particular spoken words with little or no response to pure tones, noise-vocoded speech, or environmental sounds. Many units were tuned to complex but specific sets of phonemes, which were influenced by local context but invariant to speaker, and suppressed during self-produced speech. The firing of several units to specific visual letters was correlated with their response to the corresponding auditory phonemes, providing the first direct neural evidence for phonological recoding during reading. Maximal decoding of individual phonemes and words identities was attained using firing rates from approximately 5 neurons within 200 ms after word onset. Thus, neurons in human superior temporal gyrus use sparse spatially organized population encoding of complex acoustic-phonetic features to help recognize auditory and visual words.

Disruptive effect of unattended noise-vocoded speech on recall of visually presented digits: Interaction between the number of frequency bands and languages

To assess the effects of degraded irrelevant speech on the serial recall of visually presented digits, noise-vocoded speech was generated in Japanese and German. Effects of the participants' native language were also examined by studying 40 Japanese and 40 German listeners. The number of frequency bands used in vocoding and the language (native or not) the irrelevant sound was derived from affected performance significantly. The participants' native language had a greater disruptive effect than the non-native language, particularly in conditions in which intelligibility was moderate. Speech sounds appear to have been processed automatically although the participant was instructed to neglect them. This must have required some amount of cognitive resources, which could have been used for the recall task otherwise. This automatic interference was stronger when the native language was used, probably because it contained perceptual cues that were more difficult to degrade.

Spatial release from masking for noise-vocoded speech

Spatially separating a speech target from interfering masker(s) generally improves target intelligibility; an effect known as spatial release from masking (SRM). This study assessed the contribution of envelope cues to SRM. Target speech was presented from the front (0° azimuth) and speech maskers were either colocated or symmetrically separated from the target in azimuth (±15°, ±30°, ±45° and ±90°) using KEMAR head-related transfer functions. The target and maskers were presented either as natural speech or as noise-vocoded speech. For the vocoded speech, intelligibility was conveyed only by the envelopes from M frequency bands. Experiment 1 examined the effects of varying the number of frequency bands for the vocoder, and the degree of target-masker spatial separation, on SRM. Experiment 2 examined the effects of low-pass filtering the envelopes of the vocoded speech bands on SRM. Preliminary results for Experiment 1 indicated that SRM improved as the number of spectral channels providing independent envelope cues increased for all spatial separations. Preliminary results for Experiment 2 showed no difference in SRM between low and high envelope-frequency cutoffs. Potential implications for studying hearing-impaired and cochlear-implant subjects will be discussed.

Toddlers' comprehension of noise-vocoded speech and sine-wave analogs to speech

A great deal of research has investigated listeners’ ability to compensate for degraded speech signals such as noise-vocoded speech (a signal with reduced spectral structure but intact amplitude envelope information) and sine-wave analogs to speech (a signal that maintains the global dynamic spectral structure of the signal at the expense of amplitude envelope information). Nittrouer and colleagues found developmental changes in the ability to comprehend such signals, reporting that while adults perform more accurately with sine-wave analogs than with noise-vocoded speech, school-aged children show the opposite pattern [e.g., Nittrouer Lowenstein and Packer (2009)]. In a series of studies, we tested toddler’s comprehension of these degraded signals. Twenty-seven-month-old children saw two images on each trial (e.g., cat, dog), and heard a voice instructing them which image to look at (“Find the cat!”). Sentences were presented either in full speech or were degraded. Toddlers (n = 24 per condition) looked at the appropriate object equally long with vocoded speech of 24 channels (60.2%) or 8 channels (62.4%) as with full speech (62.6%), but performed barely above chance with 4 channels (53.6%) and at chance for 2 channels (49.8%). Preliminary results suggest that performance with sine-wave analogs is poorer than 8-channel vocoded speech (56.1%), but testing is ongoing.

Audiovisual perception of noise vocoded speech in dyslexic and non-dyslexic adults: The role of low-frequency visual modulations

Visual speech inputs can enhance auditory speech information, particularly in noisy or degraded conditions. The natural statistics of audiovisual speech highlight the temporal correspondence between visual and auditory prosody, with lip, jaw, cheek and head movements conveying information about the speech envelope. Low-frequency spatial and temporal modulations in the 2–7Hz range are of particular importance. Dyslexic individuals have specific problems in perceiving speech envelope cues. In the current study, we used an audiovisual noise-vocoded speech task to investigate the contribution of low-frequency visual information to intelligibility of 4-channel and 16-channel noise vocoded speech in participants with and without dyslexia. For the 4-channel speech, noise vocoding preserves amplitude information that is entirely congruent with dynamic visual information. All participants were significantly more accurate with 4-channel speech when visual information was present, even when this information was purely spatio-temporal (pixelated stimuli changing in luminance). Possible underlying mechanisms are discussed.

Speech perception in simulated electric hearing exploits information-bearing acoustic change

Stilp and Kluender [(2010). Proc. Natl. Acad. Sci. U.S.A. 107(27), 12387-12392] reported measures of sensory change over time (cochlea-scaled spectral entropy, CSE) reliably predicted sentence intelligibility for normal-hearing listeners. Here, implications for listeners with atypical hearing were explored using noise-vocoded speech. CSE was parameterized as Euclidean distances between biologically scaled spectra [measured before sentences were noise vocoded (CSE)] or between channel amplitude profiles in simulated cochlear-implant processing [measured after vocoding (CSE(CI))]. Sentence intelligibility worsened with greater amounts of information replaced by noise; patterns of performance did not differ between CSE and CSE(CI). Results demonstrate the importance of information-bearing change for speech perception in simulated electric hearing.

Toddlers' recognition of noise-vocoded speech.

Despite their remarkable clinical success, cochlear-implant listeners today still receive spectrally degraded information. Much research has examined normally hearing adult listeners' ability to interpret spectrally degraded signals, primarily using noise-vocoded speech to simulate cochlear implant processing. Far less research has explored infants' and toddlers' ability to interpret spectrally degraded signals, despite the fact that children in this age range are frequently implanted. This study examines 27-month-old typically developing toddlers' recognition of noise-vocoded speech in a language-guided looking study. Children saw two images on each trial and heard a voice instructing them to look at one item ("Find the cat!"). Full-spectrum sentences or their noise-vocoded versions were presented with varying numbers of spectral channels. Toddlers showed equivalent proportions of looking to the target object with full-speech and 24- or 8-channel noise-vocoded speech; they failed to look appropriately with 2-channel noise-vocoded speech and showed variable performance with 4-channel noise-vocoded speech. Despite accurate looking performance for speech with at least eight channels, children were slower to respond appropriately as the number of channels decreased. These results indicate that 2-yr-olds have developed the ability to interpret vocoded speech, even without practice, but that doing so requires additional processing. These findings have important implications for pediatric cochlear implantation.

Relative contributions of spectral and temporal resolutions to the perception of syllables, words, and sentences in noise-vocoded speech

In this study, we investigated how the temporal envelopes contribute to the recognition of isolated syllables, words, and sentences in noise-vocoded speech, under comparison with the influence of spectral resolution. The spectral and temporal resolutions of speech materials were systematically manipulated by a noise-vocoding technique. Japanese monomoraic syllables, meaningful and meaningless words, and sentences were used as test speech materials. The original speech sound was spectrally separated by a filter bank, and each spectral band was replaced with band noise, preserving its original temporal envelope. Spectral resolution was controlled by varying the number of spectral bands as 4, 8, or 16 bands. Temporal resolution was altered by smoothing the amplitude envelope using a low-pass filter with cut-off frequencies at 4, 8, or 16 Hz. Results demonstrated an increased dependence on temporal resolution in comparison with spectral resolution for the recognition of speech materials at higher structural levels (i.e., monosyllables, words, and sentences). This increased temporal dependence suggests that the temporal envelope in noise-vocoded sentences contained a greater amount of suprasegmental information, such as coarticulations, prosodies, and rhythms, in the original speech sound.