Theories Of Speech Perception Research Articles

Hearing smiles and frowns in normal and whisper registers

Two experiments measured listeners' abilities to detect facial expression in unfamiliar speech in normal and whisper registers. Acoustic differences between speech produced with neutral or marked facial expression were also assessed. Experiment 1 showed that in a forced-choice identification task, listeners could accurately select frowned speech as such, and neutral speech as happier sounding than frowned speech in the same speakers. Listeners were able to judge frowning in the same speakers' whispered speech. Relative to neutral speech, frowning lowers formant frequencies and increases syllable duration. In both registers, judgments of frowning and its relative happiness were significantly poorer for lip-rounded vowels, suggesting that listeners may recover lip protrusion in making judgments. Experiment 2 replicated the finding [V. Tartter, Percept. Psychophys. 27, 24-27 (1980)] that listeners can select speech produced with a smile as happier sounding than neutral speech in normal register, and extended the findings to whisper register. Relative to neutral, smiling increased second formant frequency. Results are discussed with respect to nonverbal auditory emotion prototypes and with respect to the direct realist theory of speech perception.

The role of talker-specific information in memory for spoken sentences

Contrary to most traditional theories of speech perception, talker-specific information is not filtered out or ‘‘lost’’ during perceptual analysis, but is retained in the processing and representation of spoken language. Effects of talker-specific information have been found for words and phonemes spoken in isolation [J. W. Mullennix and D. B. Pisoni, J. Acoust. Soc. Am. 85, 365–378 (1990); Goldinger etal., J. Exp. Psychol.: Learn., Mem. Cog. 17, 152–162 (1991)]. However, the processing of spoken sentences may be quite different from isolated words due to coarticulation across words, suprasegmental information, and syntactic and semantic constraints. The present experiment investigates whether the effects of talker-specific information on the processing of sentences are similar to those found for the processing of isolated words and phonemes. Subjects first transcribed lists of sentences in either a single-talker or multiple-talker presentation format; they were given a probed recall test to assess memory for the sentences. While transcription accuracy was higher in the single-talker than the multiple-talker condition, cued recall performance for correctly transcribed sentences was not significantly different between the two presentation conditions. Implications of these findings for current theories of spoken language processing will be discussed in the context of recent episodic and multiple-trace models of human memory.

Speech perception by second language learners

Cross-language studies of speech perception by adults have shown ‘‘language-specific’’ patterns of perception of phonetic categories and contrasts. In general, phonetic categories that are distinctive (phonemic) in the listener’s native language are differentiated easily and effortlessly, while non-native phonetic categories/contrasts present perceptual difficulties. Thus learners of a second language (L2) often have persistant difficulty learning to perceive (and produce) ‘‘foreign’’ consonants and vowels. However, recent research has shown that not all non-native phonetic categories and contrasts are equally difficult to differentiate perceptually. Current theories that attempt to predict and explain these relative perceptual difficulties in terms of the relationship between native language (L1) and L2 phonetic categories will be discussed. In addition, results of perceptual training experiments with L2 learners which explore the effects of subject, stimulus, and task variables on perception of non-native phonetic categories will be summarized. Finally, implications of this research for general theories of speech perception and perceptual learning will be suggested. [Work supported by NIDCD.]

Alphabet-related biases in psycholinguistic enquiries: considerations for direct theories of speech production and perception

In recent issues of the Journal of Phonetics , proponents of a dynamical model of speech production and of a direct theory of speech perception appear to disagree on the need to define units in speech which are commensurate with phonological segments. The disagreement stems from varying interpretations of transcribed speech error data. This paper argues that deciding on basic units by referring to such data is questionable for three reasons. (1) The belief that transcriptions constitute a valid means of investigating units such as phoneme-like segments—which were originally inferred from alphabetic symbols—implies the unfounded assumption that the invention of alphabets arose spontaneously given a natural cognitive predisposition for deriving segments. (2) The suggested compatibility between aspects of speech errors and phonological segments or “gestures” can be questioned on the basis of some inviolate patterns in error sequences. (3) The presumed accuracy of transcriptions is shown to be doubtful by observations of errors captured on X-ray films; in fact, aspects that are crucial to a definition of basic units of seriation may be misrepresented by alphabetic notations. In considering certain findings relating to these three points, it is suggested that, by restricting the debate to psycholinguistic evidence, authors are overlooking other segmentations which may more aptly define direct relationships between production, perception and units of speech processing.

Modularity and the Motor Theory of Speech Perception

Modularity and the Motor Theory of Speech Perception

Conventional, Biological and Environmental Factors in Speech Communication: A Modulation Theory

Speech signals contain various types of information that can be grouped under the headings phonetic, affective, personal and transmittal. Listeners are capable of distinguishing these. Previous theories of speech perception have not considered this fully. They have mainly been concerned with problems relating exclusively to phonetic quality. The theory presented in this paper considers speech signals as the result of allowing conventional gestures to modulate a carrier signal that has the personal characteristics of the speaker, which implies that in general the conventional information can only be retrieved by demodulation.

Speech perception without audition

Most knowledge about speech perception is within the framework of studies concerned with acoustic-phonetic stimulus attributes. Vision is known to affect speech perception but is typically considered only as a supplement to hearing. The literature contains assertions that only 30% of words are recognized in lipreading/speechreading and that hearing subjects are more accurate than deaf subjects. Observations of expert visual speech perception among some profoundly deaf subjects [L. E. Bernstein et al., J. Acoust. Soc. Am. 90, 2971–2984 (1991)] promoted questioning of the conventional wisdom. A normative study of lipreading was therefore conducted involving 96 hearing and 72 deaf young adults. The two populations differed significantly across nonsense syllable, word, and sentence identification measures. The most accurate subjects were from within the deaf population. Results that lipreading in some deaf subjects is comparable to listening to speech in noise by hearing subjects. Expert lipreading demonstrates that speech perception does not require audition. Speech perception appears to be fundamentally the perception of linguistic entities not acoustic-phonetic attributes. Theories of speech perception for which generality is claimed need either to account also for visual speech perception or need to limit their claims to generality.

The perception of speech gestures

An experiment was conducted to evaluate the hypothesis that speakers perceive the intended gestures of the talker, not merely the sound that is carried to their ears. This is a central assumption of the revised motor theory of speech perception [Liberman and Mattingly, Cognition 21, 1–36 (1985)]. Of relevance to this hypothesis is how listeners perceive instances of consonant gestures that show extreme overlap in casual speech and that may, therefore, lack the expected correlates in the acoustic signal (so-called ‘‘hidden’’ gestures). To test this, utterances that showed varying degrees of actual gestural overlap were selected from an x-ray microbeam study investigating the production of consonant sequences. These were presented to listeners in a phoneme monitoring task. Results show that as the temporal distance between centers of overlapping gestures decreases, detection also decreases. That is, as the target gesture becomes more and more hidden by the subsequent gesture, subjects report hearing it less and less. This result is qualified by the finding that the magnitude of the gesture also decreases when a gesture is hidden. This research is a first step towards using natural utterances with known articulatory properties to investigate theories of speech perception.

Changing the deployment of attention to phonetic structure

Previous research on perceptual learning of synthetic speech and non-native phonetic contrasts suggests that listeners shift the focus of their attention from one set of acoustic cues to another, although this hypothesis has never been tested directly. The present study was carried out to determine whether training could alter perception of stop-consonant place of articulation of shifting attention from a transition cue to the spectral shape of the release burst. CV syllables were synthesized with transitions specifying one place of articulation and release burst shape specifying another. One group of subjects was given feedback consistent with the category specified by the transitions; the other group was given feedback consistent with the bursts. Although subjects initially tended to respond based on transition cues, feedback was effective for both groups. Furthermore, for subjects who showed learning, attentional changes also generalized to the same phonetic categories in new vowel contexts. The implications of these results for theories of speech perception and perceptual learning will be discussed.

Visual influence on heard speech syllables with a reduced visual image

Visual information of a speaker’s articulations can influence heard speech syllables [H. McGurk and J. McDonald, Nature 264, 746–748 (1976)]. The strength of this so-called McGurk effect was tested using a highly reduced visual image. A point-light technique was adopted whereby an actor’s face was darkened and reflective dots were arranged on various parts of the actor’s lips, teeth, tongue, and jaw. The actor was videotaped producing syllables in the dark. These reduced visual stimuli were dubbed onto discrepant auditory syllables in order to test their visual influence. Although subjects could not identify a frozen frame of these stimuli as a face, dynamic presentations resulted in a significant visual influence on syllable identifications. These results suggest that ‘‘pictorial’’ facial features are not necessary for audiovisual integration in speech perception. The results will be discussed in terms of the ecological approach, the fuzzy logical model, and the motor theory of speech perception.

A simulation study of the relation between vocal tract movement and energy changes in restricted frequency bands

A four-tube model of the vocal tract was used to investigate the relationship between systematic variations in tube-segment diameter and changes in energy within different frequency regions. The hypothesized near linear relation between changes in displacement and changes in energy was observed for some frequency regions. This result suggests that tracking of energy changes within select frequency bands might provide rather direct information on the kinematics (and, perhaps, the kinetics) of the articulators. Implications of the result for the evaluation of articulatory abnormalities as well as for theories of speech perception will be discussed.

Labeling of the voiceless affricate/fricative contrast and an analogous nonspeech continuum by hooded rats

The voiceless affricate/fricative contrast has played an important role in auditory theories of speech perception. This type of theory draws its support from, among others, experimental data on animals. However, nothing is known about differential responding of affricate/fricative continua by animals. Preliminary results were obtained regarding the ability of hooded rats to ‘‘label’’ affricate–fricative and an analogous nonspeech continua where the nonspeech continuum was created by randomly flipping the polarity of each sample in the digitized speech waveforms with a probability 1/2. The rats were trained on the endpoints of a ‘‘cha’’–‘‘sha’’continuum (using positive reinforcement in a two-level choice experiment) cued by covariations in rise time and frication duration (0–80 ms rise time). Differential labeling was similar to humans. The animals were switched to the nonspeech continuum without further training. Once again, performance was similar to humans. The implications for auditory theories are discussed. [Work supported by MRC and CRF.]

Context effects in a double-weak theory of speech perception.

The present study provides an elaboration of the "double-weak" theory of speech perception proposed by Nearey (1990, 1991). In this framework, the objects of speech perception (and production) are viewed as neither primarily auditory nor primarily gestural; rather, they are abstract, symbolic elements lawfully constrained to map onto relatively simple (but not entirely transparent) patterns in both domains. Speech cannot be understood unless both articulation and acoustics are considered: Many production strategies appear to be directed at achieving acoustically-oriented goals, yet most context effects in speech perception seem to be motivated by the consequences of gestural overlap. The double-weak framework suggests that speech perception and speech production are less-than-perfect inverses of each other. Despite long-term accommodation of each for the demands of the other, real-time production and perception may operate as autonomous subsystems. A family of perceptual models is discussed that provides varying degrees of approximation to "ideal solutions" (in the sense of minimizing error rate) to classifying production data exhibiting contextual interactions. Members of this family that provide substantial, yet incomplete, compensation for the consequences of gestural overlap appear to be adequate to account for the results of many speech perception experiments. Such partial perceptual compensation allows, in principle, for the kind of imperfect "error correction" discussed by Ohala (1981, 1990) in conjunction with hypo- and hypercorrection phenomena.

Modeling acoustic transitions in speech by state-interpolation hidden Markov models

The authors present a new type of hidden Markov model (HMM) for vowel-to-consonant (VC) and consonant-to-vowel (CV) transitions based on the locus theory of speech perception. The parameters of the model can be trained automatically using the Baum-Welch algorithm and the training procedure does not require that instances of all possible CV and VC pairs be present. When incorporated into an isolated word recognizer with a 75000 word vocabulary it leads to the modest improvement in recognition rates. The authors give recognition results for the state interpolation HMM and compare them to those obtained by standard context-independent HMMs and generalized triphone models.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Modularity and The Motor Theory of Speech Perception: Proceedings of a Conference to Honor Alvin M. Liberman

Modularity and The Motor Theory of Speech Perception: Proceedings of a Conference to Honor Alvin M. Liberman

Integrating Articulations in the Perception of Vowel Height

Abstract In vowels contrasting for height, a large number of articulations covary with tongue height, which is supposed to be the principal bearer of the contrast. However, attempts to link these covarying articulations to tongue movement physiologically have been largely unsuccessful, and the particular pattern of covariation appears to make more sense as a concerted effort to influence the perceived height of F&lt;sub&gt;1&lt;/sub&gt;. The experiments reported here used the Garner interference paradigm, modified to assess the perceptual primacy of stimulus dimensions, to show that the acoustic effects of two of these covarying articulations, velum height (nasalization) and rate of vocal fold vibration (pitch) are integrated perceptually with the acoustic effects of varying tongue height. This perceptual integration suggests that the different articulations are not independently perceived, contrary to the predictions of direct realist theories of speech perception, that articulatory events covary so as to enhance each other’s perceptual effects, and that the surface phonological or initial phonetic representation of vowels might be quite richly specified (contrary to claims of phonetic underspecification).

Audiovisual investigation of the loudness-effort effect for speech and nonspeech events.

There is some evidence that loudness judgments of speech are more closely related to the degree of vocal effort induced in speech production than to the speech signal's surface-acoustic properties such as intensity. Other researchers have claimed that speech loudness can be rationalized simply by considering the acoustic complexity of the signal. Because vocal effort can be specified optically as well as acoustically, a study to test the effort-loudness hypothesis was conducted that used conflicting audiovisual presentations of a speaker that produced consonant-vowel syllables with different efforts. It was predicted that if loudness judgments are constrained by effort perception rather than by simple acoustic parameters, then judgments ought to be affected by visual as well as auditory information. It is shown that loudness judgments are affected significantly by visual information even when subjects are instructed to base their judgments only on what they hear. A similar (though less pronounced) patterning of results is shown for a nonspeech "clapping" event, which attests to the generality of the loudness-effort effect previously thought to be special to speech. Results are discussed in terms of auditory, fuzzy logical, motor, and ecological theories of speech perception.

A nonspeech replication of the McGurk effect using pluck and bow stimuli

A replication of the McGurk effect was conducted using nonspeech stimuli. The McGurk effect is a well‐known phenomenon in speech perception that demonstrates how visual information for articulation can change an auditory speech percept [H. McGurk and J. McDonald, Nature 264, 746–748 (1976)]. The McGurk effect is often used to support the notion that a specialized speech module exists in the nervous system, which processes all information related to speech regardless of modality. In the current study subjects were presented with both consistent and discrepant audio‐visual tokens of plucks and bows on a cello. Audio stimuli consisted of a continuum of tokens that ranged from plucks to bows. The tokens were dubbed synchronously on to a video image of a musician either plucking or bowing a cello. It was observed that subjects' auditory judgments could be affected by the visual stimuli suggesting that the McGurk effect might not be special to speech. These results will be discussed in terms of auditory, fuzzy logical model, motor, and ecological theories of speech perception.

A note on hidden factors in vowel perception experiments

The article ‘‘Static, dynamic, and relational properties in vowel perception,’’ by Terrance M. Nearey [J. Acoust. Soc. Am. 85, 2088–2113 (1989)], contains a report on a vowel perception experiment that reproduces the previous finding that the perceptual identification of vowels is influenced by both extrinsic, contextual factors and by intrinsic factors such as F0. The effect of F0 is, however, interpreted to be smaller than claimed in previous research. This appears to be due to an incorrect analysis of the role of context. It is shown how the results are to be interpreted taking into account how the listener’s expectations are affected by a preceding context. The results are compatible with predictions on the basis of a relational theory of speech perception for cases in which intrinsic and extrinsic factors do not counteract each other.

Calling a mirage a mirage: direct perception of speech produced without a tongue

I address Morrish’s contention that the compensatory articulations of a speaker without a tongue disconfirm a direct-realist theory of speech perception. Listeners’ successful identifications of the speaker’s utterances and their mistaken identifications pattern in ways that suggest to Morrish that listeners perceive acoustic signals, not their articulatory causes. I point out that the interpretation is inconsistent with other evidence in speech perception and with interpretations of analogous situations (mirages) in visual perception. Morrish argues that the better performance of listeners identifying the speaker’s intended words in connected speech than in isolation reveals the central role of “top-down” processes in speech perception. I point out that she fails to show that the top-down influences are on perception, rather than on judgment, I suggest reasons whey we should hope that they are on judgment, and I point out that top-down information, acquired as it is by perceptual experience, would be itself impoverished were perception impoverished in the absence of top-down help. Acquiring useful top-down information presupposes veridical perception. Morrish (1990) offers challenges to two claims of a direct-realist theory: its claim that perceptual objects are articulatory and its claim that perception is direct (that is, unmediatcd by “top-down” processes). I will consider each challenge in turn and then address some more particular criticisms that Morrish raises against the theory.