Human Speech Perception Research Articles

Streams of processing and hemispheric asymmetries in speech perception

Studies in non-human primates have indicated that, as in the visual system, there are (at least) two streams of processing in the auditory system. These pathways are associated with different types of auditory processes - an anterior 'what' pathway and posterior 'how/where' pathway(s). In this talk I will use these neurophysiological theories as a framework for interpreting findings from a range of PET and fMRI studies of human speech perception and production, and present evidence that the anterior 'what' pathway in humans shows hierarchical processing of the speech signal, reflecting a move from acoustic/phonetic processing to a more abstract representation in the anterior superior temporal sulcus. In contrast, posterior auditory areas in humans are associated with sensory/motor interactions in speech, and with aspects of working memory processing. I will address how these systems are differentially recruited when speech perception is made difficult, due to different types of masking noise. Finally, I will outline differences in the processing of speech in left and right auditory areas.

Phoneme representation and classification in primary auditory cortex

A controversial issue in neurolinguistics is whether basic neural auditory representations found in many animals can account for human perception of speech. This question was addressed by examining how a population of neurons in the primary auditory cortex (A1) of the naive awake ferret encodes phonemes and whether this representation could account for the human ability to discriminate them. When neural responses were characterized and ordered by spectral tuning and dynamics, perceptually significant features including formant patterns in vowels and place and manner of articulation in consonants, were readily visualized by activity in distinct neural subpopulations. Furthermore, these responses faithfully encoded the similarity between the acoustic features of these phonemes. A simple classifier trained on the neural representation was able to simulate human phoneme confusion when tested with novel exemplars. These results suggest that A1 responses are sufficiently rich to encode and discriminate phoneme classes and that humans and animals may build upon the same general acoustic representations to learn boundaries for categorical and robust sound classification.

Coding of communication calls in the subcortical and cortical structures of the auditory system

The processing of species-specific communication signals in the auditory system represents an important aspect of animal behavior and is crucial for its social interactions, reproduction, and survival. In this article the neuronal mechanisms underlying the processing of communication signals in the higher centers of the auditory system--inferior colliculus (IC), medial geniculate body (MGB) and auditory cortex (AC)--are reviewed, with particular attention to the guinea pig. The selectivity of neuronal responses for individual calls in these auditory centers in the guinea pig is usually low--most neurons respond to calls as well as to artificial sounds; the coding of complex sounds in the central auditory nuclei is apparently based on the representation of temporal and spectral features of acoustical stimuli in neural networks. Neuronal response patterns in the IC reliably match the sound envelope for calls characterized by one or more short impulses, but do not exactly fit the envelope for long calls. Also, the main spectral peaks are represented by neuronal firing rates in the IC. In comparison to the IC, response patterns in the MGB and AC demonstrate a less precise representation of the sound envelope, especially in the case of longer calls. The spectral representation is worse in the case of low-frequency calls, but not in the case of broad-band calls. The emotional content of the call may influence neuronal responses in the auditory pathway, which can be demonstrated by stimulation with time-reversed calls or by measurements performed under different levels of anesthesia. The investigation of the principles of the neural coding of species-specific vocalizations offers some keys for understanding the neural mechanisms underlying human speech perception.

Spectral structure across the syllable specifies final-stop voicing for adults and children alike

Traditional accounts of speech perception generally hold that listeners use isolable acoustic "cues" to label phonemes. For syllable-final stops, duration of the preceding vocalic portion and formant transitions at syllable's end have been considered the primary cues to voicing decisions. The current experiment tried to extend traditional accounts by asking two questions concerning voicing decisions by adults and children: (1) What weight is given to vocalic duration versus spectral structure, both at syllable's end and across the syllable? (2) Does the naturalness of stimuli affect labeling? Adults and children (4, 6, and 8 years old) labeled synthetic stimuli that varied in vocalic duration and spectral structure, either at syllable's end or earlier in the syllable. Results showed that all listeners weighted dynamic spectral structure, both at syllable's end and earlier in the syllable, more than vocalic duration, and listeners performed with these synthetic stimuli as listeners had performed previously with natural stimuli. The conclusion for accounts of human speech perception is that rather than simply gathering acoustic cues and summing them to derive strings of phonemic segments, listeners are able to attend to global spectral structure, and use it to help recover explicitly phonetic structure.

Neural representation of spectral and temporal features of song in the auditory forebrain of zebra finches as revealed by functional MRI

Song perception in songbirds, just as music and speech perception in humans, requires processing the spectral and temporal structure found in the succession of song-syllables. Using functional magnetic resonance imaging and synthetic songs that preserved exclusively either the temporal or the spectral structure of natural song, we investigated how vocalizations are processed in the avian forebrain. We found bilateral and equal activation of the primary auditory region, field L. The more ventral regions of field L showed depressed responses to the synthetic songs that lacked spectral structure. These ventral regions included subarea L3, medial-ventral subarea L and potentially the secondary auditory region caudal medial nidopallium. In addition, field L as a whole showed unexpected increased responses to the temporally filtered songs and this increase was the largest in the dorsal regions. These dorsal regions included L1 and the dorsal subareas L and L2b. Therefore, the ventral region of field L appears to be more sensitive to the preservation of both spectral and temporal information in the context of song processing. We did not find any differences in responses to playback of the bird's own song vs other familiar conspecific songs. We also investigated the effect of three commonly used anaesthetics on the blood oxygen level-dependent response: medetomidine, urethane and isoflurane. The extent of the area activated and the stimulus selectivity depended on the type of anaesthetic. We discuss these results in the context of what is known about the locus of action of the anaesthetics, and reports of neural activity measured in electrophysiological experiments.

Variability in cortical representations of speech sound perception

Recent brain mapping studies have provided new insights into the cortical systems that mediate human speech perception. Electrocortical stimulation mapping (ESM) is a brain mapping method that is used clinically to localize cortical functions in neurosurgical patients. Recent ESM studies have yielded new insights into the cortical systems that mediate speech perception and how these systems vary as a function of individual differences. ESM methods are described and findings from recent ESM studies of speech perception are reviewed. The clinical implications of these findings are discussed as they relate to current understanding of how individual differences in listening abilities are reflected in the underlying cortical representations.

Oreja: A MATLAB environment for the design of psychoacoustic stimuli

The Oreja software package (available from www.liv.ac.uk/psychology/Downloads/Oreja.htm) was designed to study speech intelligibility. It is a tool that allows manipulation of speech signals to facilitate study of human speech perception. A feature of this package is that it uses a high-level interpreted scripting environment (MATLAB), allowing the user to load, break down into different channels, analyze, and select the parts of the signal(s) of interest (e.g., attenuation of the amplitude of the selected channels, addition of noise, etc.).

Rhesus macaques spontaneously perceive formants in conspecific vocalizations

We provide a direct demonstration that nonhuman primates spontaneously perceive changes in formant frequencies in their own species-typical vocalizations, without training or reinforcement. Formants are vocal tract resonances leading to distinctive spectral prominences in the vocal signal, and provide the acoustic determinant of many key phonetic distinctions in human languages. We developed algorithms for manipulating formants in rhesus macaque calls. Using the resulting computer-manipulated calls in a habituation/dishabituation paradigm, with blind video scoring, we show that rhesus macaques spontaneously respond to a change in formant frequencies within the normal macaque vocal range. Lack of dishabituation to a "synthetic replica" signal demonstrates that dishabituation was not due to an artificial quality of synthetic calls, but to the formant shift itself. These results indicate that formant perception, a significant component of human voice and speech perception, is a perceptual ability shared with other primates.

A feature extraction method using subband based periodicity and aperiodicity decomposition with noise robust frontend processing for automatic speech recognition

This paper proposes a frontend processing technique that employs a speech feature extraction method called Subband based Periodicity and Aperiodicity DEcomposition (SPADE), and examines its validity for automatic speech recognition in noisy environments. SPADE divides speech signals into subband signals, which are then decomposed into their periodic and aperiodic features, and uses both features as speech feature parameters. SPADE employs independent periodicity estimation within each subband and periodicity–aperiodicity decomposition design based on a parallel distributed processing technique motivated by the human speech perception process. Unlike other speech features, this decomposition of speech into two characteristics provides information about periodicities and aperiodicities, and thus allows the utilization of the robustness exhibited by periodic features without losing certain essential information included in aperiodic features. This paper first introduces an implementation of SPADE that operates in the frequency domain, and then examines the validity of combining SPADE with speech enhancement methods. For this examination, we combine SPADE with noise compensation methods that operate in the frequency domain and cepstral normalization methods. In addition, we employ an energy parameter calculation method based on the SPADE framework. An evaluation with the AURORA-2J noisy continuous digit speech recognition database (Japanese AURORA-2) shows that SPADE combined with adaptive Wiener filtering, cepstral normalization, and the energy parameter achieves average word accuracy rates of 82.58% with clean training and 92.55% with multicondition training. These rates are higher than those achieved with ETSI WI008 advanced DSR frontend processing (77.98% and 91.01%, respectively) whose speech feature parameter is based on conventional Mel-frequency cepstral coefficients. By comparison with ETSI WI008 advanced DSR frontend, the proposed method reduces word error rates by 20.9% with clean training and 17.2% with multicondition training. These results confirmed that SPADE combined with noise reduction methods can increase robustness in the presence of noise.

Pitch-based monaural segregation of reverberant speech

In everyday listening, both background noise and reverberation degrade the speech signal. Psychoacoustic evidence suggests that human speech perception under reverberant conditions relies mostly on monaural processing. While speech segregation based on periodicity has achieved considerable progress in handling additive noise, little research in monaural segregation has been devoted to reverberant scenarios. Reverberation smears the harmonic structure of speech signals, and our evaluations using a pitch-based segregation algorithm show that an increase in the room reverberation time causes degraded performance due to weakened periodicity in the target signal. We propose a two-stage monaural separation system that combines the inverse filtering of the room impulse response corresponding to target location and a pitch-based speech segregation method. As a result of the first stage, the harmonicity of a signal arriving from target direction is partially restored while signals arriving from other directions are further smeared, and this leads to improved segregation. A systematic evaluation of the system shows that the proposed system results in considerable signal-to-noise ratio gains across different conditions. Potential applications of this system include robust automatic speech recognition and hearing aid design.

The psychology of songbird acoustic communication: Open-ended categorization and its role in songbird perception

Songbirds are used as a model system for understanding vocal learning, production, and perception. Over the past several years one particular aspect of songbird communication has been examined, vocal perception that has parallels with human speech perception, namely, the perception of natural acoustic categories in songbird vocalizations. A series of studies will be presented highlighting the role of open-ended categorization in the perception of vocalizations in songbirds. Attempts will be made to make the case that open-ended categorization is a fundamental proximate mechanism for acoustic communication in both humans and songbirds, that it is used in a highly analogous manner in each, and future directions will attempt to integrate these two areas of research and, ultimately, lead to a more complete understanding of acoustic communication in animals.

Grouping in auditory temporal perception and vocal production is mutually adapted: the case of wriggling calls of mice

Auditory Gestalt perception by grouping of species-specific vocalizations to a perceptual stream with a defined meaning is typical for human speech perception but has not been studied in non-human mammals so far. Here we use synthesized models of vocalizations (series of wriggling calls) of mouse pups (Mus domesticus) and show that their mothers perceive the call series as a meaningful Gestalt for the release of instinctive maternal behavior, if the inter-call intervals have durations of 100-400 ms. Shorter or longer inter-call intervals significantly reduce the maternal responsiveness. We also show that series of natural wriggling calls have inter-call intervals mainly in the range of 100-400 ms. Thus, series of natural wriggling calls of pups match the time-domain auditory filters of their mothers in order to be optimally perceived and recognized. A similar time window exists for the production of human speech and the perception of series of sounds by humans. Neural mechanisms for setting the boundaries of the time window are discussed.

Perceptual Fusion and Stimulus Coincidence in the Cross-Modal Integration of Speech

Human speech perception is profoundly influenced by vision. Watching a speaker's mouth movements significantly improves comprehension, both for normal listeners in noisy environments and especially for the hearing impaired. A number of brain regions have been implicated in audiovisual speech tasks, but little evidence distinguishes them functionally. In an event-related functional magnetic resonance imaging study, we differentiate neural systems that evaluate cross-modal coincidence of the physical stimuli from those that mediate perceptual binding. Regions consistently involved in perceptual fusion per se included Heschl's gyrus, superior temporal sulcus, middle intraparietal sulcus, and inferior frontal gyrus. Successful fusion elicited activity biased toward the left hemisphere, although failed cross-modal binding recruited regions in both hemispheres. A broad network of other areas, including the superior colliculus, anterior insula, and anterior intraparietal sulcus, were more involved with evaluating the spatiotemporal correspondence of speech stimuli, regardless of a subject's perception. All of these showed greater activity to temporally offset stimuli than to audiovisually synchronous stimuli. Our results demonstrate how elements of the cross-modal speech integration network differ in their sensitivity to physical reality versus perceptual experience.

Endothelial nitric oxide synthase upregulation in the cochlea of the guinea pig after intratympanic gentamicin injection

Single-shot transtympanic gentamicin therapy has become a popular treatment modality for Meniere's disease despite the known possible ototoxic properties of this drug. It was shown recently that NO production and iNOS were upregulated after gentamicin application, which was interpreted as a possible effect of ototoxicity. In this study we analyzed the expression of eNOS after gentamicin application to determine a possible correlation of this enzyme with gentamicin-induced ototoxicity. We compared eNOS expression in gentamicin-treated and non-treated guinea pigs in the second turn of the cochlea, an area corresponding to speech perception in humans. Gentamicin (4 mg) was injected intratympanically into the middle ear of guinea pigs ( n =3) and the reduction of the hearing threshold level was determined by recording acoustic-evoked potentials (AEP) before and 5 days after gentamicin application. Morphological alterations in the organ of Corti were analyzed by light and electron microscopy. Gold-labeled anti-eNOS antibodies were counted in eight different cell areas for quantification of eNOS expression. Seven animals were analyzed as controls. After gentamicin application, a deterioration of hearing level was observed varying from 10 to 30 dB. A high degree of vacuolization was identified in the third row of outer hair cells. At the subcellular level, the subsurface cisterns in outer hair cells were dissociated from the basolateral cell membrane, and the mitochondrial membranes were frequently damaged. Statistically significant upregulation of eNOS was observed in all cell types analyzed. Depending on the various cell types the amount of gold-labeled eNOS antibodies was 2.5 to 5.7 times higher after gentamicin application. We observed significant eNOS upregulation after gentamicin application in the cochlea, in conjunction with cellular damages and decreased hearing.

“Soundmorphing”: A new approach to studying speech perception in humans

A problem in current studies of brain activation in speech perception is that most studies use stimuli that are sampled from different categories. This study presents a new approach in creating acoustic stimuli by filtering real words in different ways, thus this enables one to ‘morph’ the sounds gradually from something like tones to real words while preserving temporal characteristics (“soundmorphing”). This will make it possible to explore steps between tone and speech processing on a gradual scale. This study aimed to investigate more closely the network, involved in the perception of speech and the decoding of auditory (speech-)stimuli. Sets of auditory stimuli were created by dividing real words into several frequency-bands and creating a randomised rearrangement of these components. Four different sets of stimuli were used, containing 1, 2, 3 or 4 frequency bands (total range: 0–2500 Hz). Only the latter one contained intelligible words. During the fMRI session, these four trial types were presented pseudo-randomised. In all conditions, significant activations of auditory cortex were observed. The number of activated voxels and their significance increased over the four trial types. In addition, Broca's area, SMA, left thalamus, and right cerebellum were activated in the most complex and most speech-like condition. Comparing these complexes with the tone-like trials, only the left auditory cortex, left SMA, and cingulate gyrus became significant. These activations reflect the bilateral processing of the auditory stimuli, but only the left temporal areas demonstrated an increasing BOLD response with an increasing number of presented frequency components.

Haptic information enhances auditory speech perception

Studies of tactile enhancement effects on auditory speech perception [Reed et al., JSHR, 1978, 1982, 1989] have traditionally used experienced, pre-trained subjects. These studies have left unanswered the basic questions of whether tactile enhancement of speech is a basic component of human speech perception or a learned association [Fowler, JPhon, 1986; Diehl and Kluender, Ecol. Psych., 1989], and which aspects of the signal are enhanced through this modality. The present study focuses exclusively on tactile enhancement effects available to naive speakers. In a speech-masked environment, naive subjects were tasked with identifying consonants using the Tadoma method. Half of the stimuli were presented with tactile input, and half without. The results show that all subjects gained considerably from tactile information, although which part of the tactile signal was most helpful varied by subject. The features that contributed most to consonant identification were aspiration and voicing. [Work supported by NSERC.]

Modulation enhancement of speech by a pre-processing algorithm for improving intelligibility in reverberant environments

Most listeners have difficulty understanding speech in reverberant conditions. The purpose of this study is to investigate whether it is possible to reduce the degree of degradation of speech intelligibility in reverberation through the development of an algorithm. The modulation spectrum is the spectral representation of the temporal envelope of the speech signal. That of clean speech is dominated by components between 1 and 16 Hz centered at 4 Hz which is the most important range for human perception of speech. In reverberant conditions, the modulation spectrum of speech is shifted toward the lower end of the modulation frequency range. In this study, we proposed to enhance the important modulation spectral components prior to distortion of speech by reverberation. Word intelligibility in a carrier sentence was tested with the newly developed algorithm including two different filter designs in three reverberant conditions. The reverberant speech was simulated by convoluting clean speech with impulse responses measured in the actual halls. The experimental results show that modulation filtering incorporated into a pre-processing algorithm improves intelligibility for normal hearing listeners when (1) the modulation filters are optimal for a specific reverberant condition (i.e., T 60 = 1.1 s), and (2) consonants are preceded by highly powered segments. Under shorter (0.7 s) and longer (1.6 s) reverberation times, the modulation filtering in the current experiments, an Empirically-Designed (E-D) filter and a Data-Derived (D-D) filter, caused a slight performance decrement respectively. The results of this study suggest that further gains in intelligibility may be accomplished by re-design of the modulation filters suitable for other reverberant conditions.

Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study

During social interaction speech is perceived simultaneously by audition and vision. We studied interactions in the processing of auditory (A) and visual (V) speech signals in the human brain by comparing neuromagnetic responses to phonetically congruent audiovisual (AV) syllables with the arithmetic sum of responses to A and V syllables. Differences between AV and A+V responses were found bilaterally in the auditory cortices 150–200 ms and in the right superior temporal sulcus (STS) 250–600 ms after stimulus onset, showing that both sensory-specific and multisensory regions of the human temporal cortices are involved in AV speech processing. Importantly, our results suggest that AV interaction in the auditory cortex precedes that in the multisensory STS region.

Techniques for handling convolutional distortion with `missing data' automatic speech recognition

In this study we describe two techniques for handling convolutional distortion with `missing data' speech recognition using spectral features. The missing data approach to automatic speech recognition (ASR) is motivated by a model of human speech perception, and involves the modification of a hidden Markov model (HMM) classifier to deal with missing or unreliable features. Although the missing data paradigm was proposed as a means of handling additive noise in ASR, we demonstrate that it can also be effective in dealing with convolutional distortion. Firstly, we propose a normalisation technique for handling spectral distortions and changes of input level (possibly in the presence of additive noise). The technique computes a normalising factor only from the most intense regions of the speech spectrum, which are likely to remain intact across various noise conditions. We show that the proposed normalisation method improves performance compared to a conventional missing data approach with spectrally distorted and noise contaminated speech, and in conditions where the gain of the input signal varies. Secondly, we propose a method for handling reverberated speech which attempts to identify time-frequency regions that are not badly contaminated by reverberation and have strong speech energy. This is achieved by using modulation filtering to identify `reliable' regions of the speech spectrum. We demonstrate that our approach improves recognition performance in cases where the reverberation time T 60 exceeds 0.7 s, compared to a baseline system which uses acoustic features derived from perceptual linear prediction and the modulation-filtered spectrogram.

Detection of speech landmarks: use of temporal information.

Studies by Shannon et al. [Science, 270, 303-304 (1995)], Van Tasell et al. [J. Acoust. Soc. Am. 82, 1152-1161 (1987)], and others show that human listeners can understand important aspects of the speech signal when spectral shape has been significantly degraded. These experiments suggest that temporal information is particularly important in human speech perception when the speech signal is heavily degraded. In this study, a system is developed that extracts linguistically relevant temporal information that can be used in the front end of an automatic speech recognition system. The parameters targeted include energy onset and offsets (computed using an adaptive algorithm) and measures of periodic and aperiodic content; together these are used to find abrupt acoustic events which signify landmarks. Overall detection rates for strongly robust events, robust events, and weak events in a portion of the TIMIT test database are 98.9%, 94.7%, and 52.1%, respectively. Error rates increase by less than 5% when the speech signals are spectrally impoverished. Use of the four temporal parameters as the front end of a hidden Markov model (HMM)-based system for the automatic recognition of the manner classes "sonorant," "fricative," "stop," and "silence" results in the same recognition accuracy achieved when the standard 39 cepstral-based parameters are used, 70.1%. The combination of the temporal parameters and cepstral parameters results in an accuracy of 74.8%.