Theories Of Speech Perception Research Articles

Production-Oriented Models for Speech Recognition

Acoustic modeling in speech recognition uses very little knowledge of the speech production process. At many levels our models continue to model speech as a surface phenomenon. Typically, hidden Markov model (HMM) parameters operate primarily in the acoustic space or in a linear transformation thereof; state-to-state evolution is modeled only crudely, with no explicit relationship between states, such as would be afforded by the use of phonetic features commonly used by linguists to describe speech phenomena, or by the continuity and smoothness of the production parameters governing speech. This survey article attempts to provide an overview of proposals by several researchers for improving acoustic modeling in these regards. Such topics as the controversial Motor Theory of Speech Perception, work by Hogden explicitly using a continuity constraint in a pseudo-articulatory domain, the Kalman filter based Hidden Dynamic Model, and work by many groups showing the benefits of using articulatory features instead of phones as the underlying units of speech, will be covered.

The motor theory of speech perception: Its history, progress, and perspective

The motor theory of speech perception: Its history, progress, and perspective

Differentiation of speech and nonspeech processing within primary auditory cortex

Primary auditory cortex (PAC), located in Heschl's gyrus (HG), is the earliest cortical level at which sounds are processed. Standard theories of speech perception assume that signal components are given a representation in PAC which are then matched to speech templates in auditory association cortex. An alternative holds that speech activates a specialized system in cortex that does not use the primitives of PAC. Functional magnetic resonance imaging revealed different brain activation patterns in listening to speech and nonspeech sounds across different levels of complexity. Sensitivity to speech was observed in association cortex, as expected. Further, activation in HG increased with increasing levels of complexity with added fundamentals for both nonspeech and speech stimuli, but only for nonspeech when separate sources (release bursts/fricative noises or their nonspeech analogs) were added. These results are consistent with the existence of a specialized speech system which bypasses more typical processes at the earliest cortical level.

The syntactic motor system

The human brain has mechanisms that can support production and perception of language. We ground the evolution of these mechanisms in primate systems that support manual dexterity, especially the mirror system that integrates execution and observation of hand movements. We relate the motor theory of speech perception to the mirror system hypothesis for language and evolution; explore links between manual actions and speech; contrast “language” in apes with language in humans; show in what sense the “syntax” implemented in Broca’s area is a “motor syntax” far more general than the syntax of linguistics; and relate communicative goals to sentential form.

Enhancement of visual perception of speech via tactile input

Motor theories of speech perception predict that perceptual information from modalities other than sound enhance speech perception directly by informing the perceiver of the speaker’s gestures [A. Lieberman and I. Mattingly, Cognition 21, 1–36 (1985)]. Acoustic theories predict that perceptual information from modalities other than sound will only enhance speech perception if there is a learned mapping between the acoustic speech signal speech and that modality [R. Diehl and K. Kluender, Ecol. Psych. 1, 121–144 (1989)]. As normal subjects are unlikely to have learned a mapping between visual and tactile speech information, this study tests whether and how tactile input enhances visual speech perception. In the control condition, perceivers in noise repeat syllables pronounced by a speaker who they can see clearly. Accuracy is judged on the basis of the repeated syllables. In the experimental condition, subjects additionally have their hand on the speaker’s face in the Tadoma position. Results show that subjects are significantly more accurate at perceiving speech when they have both visual and tactile input then when they have visual input alone. In particular, tactile input enhances perceptual accuracy of voice and manner features. [Work supported by NSERC.]

Control and representations in speech production

In this paper the issue of the nature of the representations of the speech production task in the speaker's brain is addressed in a production-perception interaction framework. Since speech is produced to be perceived, it is hypothesized that its production is associated for the speaker with the generation of specific physical characteristics that are for the listeners the objects of speech perception. Hence, in the first part of the paper, four reference theories of speech perception are presented, in order to guide and to constrain the search for possible correlates of the speech production task in the physical space: the Acoustic Invariance Theory, the Adaptive Variability Theory, the Motor Theory and the Direct-Realist Theory. Possible interpretations of these theories in terms of representations of the speech production task are proposed and analyzed. In a second part, a few selected experimental studies are presented, which shed some light on this issue. In the conclusion, on the basis of the joint analysis of theoretical and experimental aspects presented in the paper, it is proposed that representations of the speech production task are multimodal, and that a hierarchy exists among the different modalities, the acoustic modality having the highest level of priority. It is also suggested that these representations are not associated with invariant characteristics, but with regions of the acoustic, orosensory and motor control spaces.

The syntactic motor system

The human brain has mechanisms that can support production and perception of language. We ground the evolution of these mechanisms in primate systems that support manual dexterity, especially the mirror system that integrates execution and observation of hand movements. We relate the motor theory of speech perception to the mirror system hypothesis for language and evolution; explore links between manual actions and speech; contrast “language” in apes with language in humans; show in what sense the “syntax” implemented in Broca’s area is a “motor syntax” far more general than the syntax of linguistics; and relate communicative goals to sentential form.

Perceptual overshoot in listeners with cochlear implants

Perceptual overshoot (PO), a phenomenon in which the boundary for perceived vowel categories shifts as a result of preceding formant transitions, has been demonstrated previously with synthetic vowels in CV contexts in normal-hearing individuals. In one of the previous studies by the author, the same phenomenon was tested with sinewave analogs that mimicked the synthetic vowels. The results demonstrated that PO could be elicited for sinewave analogs after training on categorization of sinewave steady states. These findings suggest that PO may be partly mediated by general processes in the auditory system. In the current study, subjects with cochlear implants were presented with synthetic vowel and sinewave continua with and without transitions. It was predicted that PO would be greatly diminished or absent for this population because of the degraded nature of the spectral input. Implications of the results for theories of speech perception will be presented.

Specification of cross-modal source information in isolated kinematic displays of speech.

Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions. The results showed that isolated kinematic displays provide enough information to match the source of an utterance across sensory modalities. Furthermore, isolated kinematic displays can be integrated to yield better word recognition performance under audiovisual conditions than under auditory-alone conditions. The results are discussed in terms of their implications for describing the nature of speech information and current theories of speech perception and spoken word recognition.

Identification and discrimination of Spanish front vowels

The idea that vowels are perceived less categorically than consonants is widely accepted. Ades [Psychol. Rev. 84, 524–530 (1977)] tried to explain this fact on the basis of the Durlach and Braida [J. Acoust. Soc. Am. 46, 372–383 (1969)] theory of intensity resolution. Since vowels seem to cover a broader perceptual range, context-coding noise for vowels should be greater than for consonants leading to a less categorical performance on the vocalic segments. However, relatively recent work by Macmillan et al. [J. Acoust. Soc. Am. 84, 1262–1280 (1988)] has cast doubt on the assumption of different perceptual ranges for vowels and consonants even though context variance is acknowledged to be greater for the former. A possibility is that context variance increases as number of long-term phonemic categories also increases. To test this hypothesis we focused on Spanish as the target language. Spanish has less vowel categories than English and the implication is that Spanish vowels will be more categorically perceived. Identification and discrimination experiments were conducted on a synthetic /i/-/e/ continuum and the obtained functions were studied to assess whether Spanish vowels are more categorically perceived than English vowels. The results are discussed in the context of different theories of speech perception.

Speech perception in rats: use of duration and rise time cues in labeling of affricate/fricative sounds.

The voiceless affricate/fricative contrast has played an important role in developing auditory theories of speech perception. This type of theory draws some of its support from experimental data on animals. However, nothing is known about differential responding of affricate/fricative continua by animals. In the current study, the ability of hooded rats to "label" an affricate/fricative continuum was tested. Transfer (without retraining) to analogous nonspeech continua was also tested. The nonspeech continua were chosen so that if transfer occurred, it would indicate whether the animals had learned to use rise time or duration cues to differentiate affricates from fricatives. The data from 9 of 10 rats indicated that rats can discriminate between these cues and do so in a similar manner to human subjects. The data from 9 of 10 rats also demonstrated that the rise time of the stimulus was the basis of the discrimination; the remaining rat appeared to use duration.

Les illusions McGurk dans la parole : 25 ans de recherches

Summary : The McGurk illusions in speech: 25years of research. When presented with an auditory /b/ dubbed onto a visual /g/, listeners perceive sometimes a fused phoneme like /d/ whereas with the reverse presentation, they experience a combination such as /bg/. These two kinds of illusions were reported by McGurk and MacDonald (1976). The aim of this paper is to review research on the McGurk effect and on the underlying cognitive processes. Different experimental manipulations that are likely to modulate the size of the effect are examined, such as spatial separation and temporal desynchronisation between the signals, vocalic and consonantic environment, articulation rate, stimulus quality, cognitive influences and cross-linguistic differences. Datafrom studies bearing on the issue of the level of processing at which audiovisual integration occurs relative to phonetic categorization are considered. The place of the McGurk effect in current theories of speech perception is also discussed, as well as the development of the effect during childhood, and finally the possible cerebral bases. Key words : audiovisual integration, speech reading, speech perception

Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory

Objectives: The McGurk–MacDonald illusory percept is obtained by dubbing an incongruent articulatory movement on an auditory phoneme. This type of audiovisual speech perception contributes to the assessment of theories of speech perception. The mismatch negativity (MMN) reflects the detection of a deviant stimulus within the auditory short-term memory and besides an acoustic component, possesses, under certain conditions, a phonetic one. The present study assessed the existence of an MMN evoked by McGurk–MacDonald percepts elicited by audiovisual stimuli with constant auditory components. Methods: Cortical evoked potentials were recorded using the oddball paradigm on 8 adults in 3 experimental conditions: auditory alone, visual alone and audiovisual stimulation. Obtaining illusory percepts was confirmed in an additional psychophysical condition. Results: The auditory deviant syllables and the audiovisual incongruent syllables elicited a significant MMN at F z. In the visual condition, no negativity was observed either at F z, or at O z. Conclusions: An MMN can be evoked by visual articulatory deviants, provided they are presented in a suitable auditory context leading to a phonetically significant interaction. The recording of an MMN elicited by illusory McGurk percepts suggests that audiovisual integration mechanisms in speech take place rather early during the perceptual processes.

Speech listening specifically modulates the excitability of tongue muscles: a TMS study.

The precise neural mechanisms underlying speech perception are still to a large extent unknown. The most accepted view is that speech perception depends on auditory-cognitive mechanisms specifically devoted to the analysis of speech sounds. An alternative view is that, crucial for speech perception, it is the activation of the articulatory (motor) gestures that generate these sounds. The listener understands the speaker when his/her articulatory gestures are activated (motor theory of speech perception). Here, by using transcranial magnetic stimulation (TMS), we demonstrate that, during speech listening, there is an increase of motor-evoked potentials recorded from the listeners' tongue muscles when the presented words strongly involve, when pronounced, tongue movements. Although these data do not prove the motor theory of speech perception, they demonstrate for the first time that word listening produces a phoneme specific activation of speech motor centres.

Speech Science: An Integrated Approach to Theory and Clinical Practice

Speech Science: An Integrated Approach to Theory and Clinical Practice

The relationship between perception and acoustics for a high-low vowel contrast produced by speakers with dysarthria.

This study was designed to explore the relationship between perception of a high-low vowel contrast and its acoustic correlates in tokens produced by persons with motor speech disorders. An intelligibility test designed by Kent, Weismer, Kent, and Rosenbek (1989a) groups target and error words in minimal-pair contrasts. This format allows for construction of phonetic error profiles based on listener responses, thus allowing for a direct comparison of the acoustic characteristics of vowels perceived as the intended target with those heard as something other than the target. The high-low vowel contrast was found to be a consistent error across clinical groups and therefore was selected for acoustic analysis. The contrast was expected to have well-defined acoustic measures or correlates, derived from the literature, that directly relate to a listeners' responses for that token. These measures include the difference between the second and first formant frequency (F2-F1), the difference between F1 and the fundamental frequency (FO), and vowel duration. Results showed that the acoustic characteristics of tongue-height errors were not clearly differentiated from the acoustic characteristics of targets. Rather, the acoustic characteristics of errors often looked like noisy (nonprototypical) versions of the targets. Results are discussed in terms of the test from which the errors were derived and within the framework of speech perception theory.

Event coding as feature guessing: The lessons of the motor theory of speech perception

The claim that perception and action are commonly coded because they are indistinguishable at the distal level is crucial for theories of cognition. However, the consequences of this claim run deep, and the Theory of Event Coding (TEC) is not up to the challenge it poses. We illustrate why through a brief review of the evidence that led to the motor theory of speech perception.

Context-dependent neural coding in the chinchilla cochlear nucleus

One of the most challenging dilemmas for theories of speech perception is the lack of invariance between acoustic signal and perception. Due to physical constraints upon articulators, there is a good deal of context-dependency both in speech production and in the resulting acoustic speech signal. Consequently, the acoustic pattern most closely related to a given speech sound varies dramatically depending on context. Yet, by some means, the perceptual system perceives these unique acoustic events as linguistically equivalent. This phenomenon is observed experimentally as perceptual context effects whereby adjacent speech can modulate perceived identity of a given speech sound. Recent perceptual results suggest that this phenomenon may be governed by general auditory mechanisms [Holt et al., 1996; Lotto et al., 1997; Lotto and Kluender, 1998; Holt, 1999] rather than speech-specific processes. In the present study, we sought to explore how such context-dependencies might be encoded. We recorded responses of ventral cochlear nucleus (VCN) neurons of anesthetized chinchillas to nonspeech stimulus targets with adjacent context stimuli that varied in spectral content. Results demonstrate context-dependent effects of frequency and intensity.

The importance of parity; its implications for understanding speech perception

Early in his career, Alvin Liberman obtained experimental findings that he describes in his book, Speech: A Special Code, as ‘‘an epiphany.’’ The findings revealed that listeners track speakers’ articulations. From these findings and subsequent others, Liberman developed his motor theory of speech perception. In addition, however, he set out to understand why listeners track articulation. His explanation was that humans evolved a phonetic system jointly responsible for producing coarticulated speech and for recovering intended phonetic segments from coarticulated signals. Underlying the claim that production and perception are linked in this way is the idea of parity. In Liberman’s interpretation, parity is the requirement that, for language to serve as a major component of human communication systems, listeners and talkers must agree on what set of perceivable human actions can count as components of a linguistic message. In addition, and more concretely, in their conversational interactions, listeners and talkers must typically achieve a relation of parity between messages sent and received. This paper will focus on how this parity requirement rationalizes the body of findings that listeners track articulatory gestures.

Alvin Liberman, infant research, and the speech module

In 1957, Alvin Liberman offered a highly controversial statement based on the experimental work at Haskins Laboratories, known as ‘‘the motor theory of speech perception.’’ Over time, the theory evolved to presume that the mechanisms underlying speech processing were biologically determined and specialized for the perception and production of speech. Speech perception was in effect not an auditory process, at least beyond the initial reception of speech, but rather part of a species-specific processing system that made language possible. We consider this view in light of selected aspects of our research on infant speech perception, which was driven in large part by the work of Liberman and his associates. We believe that the evidence in its entirety supports Liberman’s idea of a species-specific processor or module that permits the acquisition and use of language, including much of the relatively early processing of speech.