Non-speech Signals Research Articles

On processing speech and non-speech signals in acoustics

In a nonspeech communication, the pitch, duration, formants arrangement, and other acoustic features are bearers of coded information. Certain meanings are assigned to them by convention. The processing of nonspeech signals, consists in distinguishing and, subsequently, interpreting them. Speech oral messages are not, by contrast, amenable to such one-average ‘‘technology.’’ Each oral utterance betrays a two-layer organization. Here, true information carriers are coded articulary changes within the speech channel. Such gestures may be successfully ‘‘read’’ directly in TADOMA communication (by the hand placed against the face of the speaker), or in the ‘‘inner speech’’ (through bioimpulses announcing rudimentary movements of our speech organs). Mostly, however, the invisible articulations need applying certain ‘‘echo-effects’’ to become perceivable. Usually, the ‘‘echoes’’ of the voice, noise, whistle, and light (cf. ‘‘lip-reading’’) are used. Accordingly, the acoustician’s mission in speech processing ought to be reduced to restoring—from the sound characteristics of utterances—the invisible changes within the throat of the speaker (through an improved ‘‘inverse mapping, the directional hearing methodology, etc.’’?). Starting from these data, physiologists and linguists could decipher messages using ‘‘lexicon of speech gestures.’’

Speech recognition

The authors have designed, successfully trained and tested an Arabic speech recognition system. This system was implemented using C++ programming language on Windows 95. It can be partitioned into five main modules. These modules are the front-end, feature extraction, training, pattern recognition and decision making and display. The front-end module functions as signal preparation and calibration. This includes: setting the signal sampling rate, removing the DC component from the signal, setting the scaling factor of the signal and detecting the endpoints of the utterance. The endpoint task removes the non-speech signal portions created by the speaker's pauses. This reduces the system computation time needed and the memory requirements. The feature extraction module is mainly a digital signal processor. The training module is the one that finds the best templates for every word or sound (phonemes) in the system's database. In short, this module needs to be executed only one time before users can utilize the system. The next module is the pattern recognition module. Its function is to compare the given utterance (test utterance) to all the stored templates (the reference module). The decision and display module functions as an interface between the user and the hidden system modules. In other words, after getting the recognition module results, this module displays the best candidate(s) and/or their likelihood percentage. The error rates are computed and displayed in this module.

Auditory Perception

Auditory perception involves the human listener's awareness or apprehension of auditory stimuli in the environment. Auditory stimuli, which include speech communications as well as non-speech signals, occur in the presence and absence of environmental noise. Non-speech auditory signals range from simple pure tones to complex signals found in three-dimensional auditory displays. Special hearing protection device (HPD) designs, as well as additions to conventional protectors, have been developed to improve speech communication and auditory perception capabilities of those exposed to noise. The thoughtful design of auditory stimuli and the proper design, selection, and use of HPDs within the environment can improve human performance and reduce accidents. The purpose of this symposium will be to discuss issues in auditory perception and to describe methods to improve the perception of auditory stimuli in environments with and without noise. The issues of interest include the perception of non-speech auditory signals and the improvement of auditory perception capabilities of persons exposed to noise. The first three papers of this symposium describe the perception of non-speech auditory signals. Ellen Haas defines the extent to which certain signal elements affect the perceived urgency of auditory warning signals. Michael D. Good and Dr. Robert H. Gilkey investigate free-field masking as a function of the spatial separation between signal and masker sounds within the horizontal and median planes. Jeffrey M. Gerth explores the discrimination of complex auditory signal components that differ by sound category, temporal pattern, density, and component manipulation. The fourth paper of this symposium focuses upon the improvement of auditory perception capabilities of persons exposed to hazardous noise, and who must wear hearing protection. Special HPD designs, as well as additions to conventional protectors, have been developed to improve speech communication and auditory perception capabilities of persons exposed to noise. Dr. John G. Casali reviews several new HPD technologies and describes construction features, empirical performance data, and applications of each device. These papers illustrate current research issues in the perception of auditory signals. The issues are all relevant to the human factors engineering of auditory signals and personal protective gear. The perception of auditory stimuli can be improved by the thoughtful human factors design of auditory stimuli and by the proper use of HPDs.

On the Objects of Speech Perception

From both direct-realist and motor-theoretic perspectives, the objects of speech perception are construed as articulatory events. We offer the alternative view that it is the auditory properties of phonetic events to which listeners are primarily sensitive. Evidence for this alternative view comes from a variety of sources, including studies of phonetic and phonological universals, comparisons of the perception of speech and nonspeech signals, cross-language perceptual experi- ments, and studies of phonetic categorization by nonhuman animals.

Effect of vowel duration on the perception of syllable‐initial /∫/ and /t∫/

The duration of an adjacent vowel has been demonstrated to affect the judgment of consonant duration and, hence, phonetic identity. For example, syllable‐initial stops can be distinguished from glides on the basis of transition duration and, when the following vowel is relatively long, longer transitions are required in order for a glide to be perceived. Apparently, the vowel provides a source of durational contrast, whereby a longer vowel makes the adjacent consonant seem shorter. This effect has been demonstrated for both speech and nonspeech signals [L. Diehl and M. A. Walsh, J. Acoust. Soc. Am. Suppl. 1 80, S125 (1986)], and is presumably grounded in a general auditory mechanism. In this study, the effect of vowel duration on perception of syllable‐initial consonants is also demonstrated for fricatives and affricates. Frication duration is an important cue to this distinction, with longer durations yielding more /∫/ responses. Two series of edited natural tokens of /∫a/ and /t∫a/ were created in which frication duration varied from 100 to 210 ms. The duration of the following vowel was 204 ms for one series, and 358 ms for the other. With the longer vowel context, significantly longer frication was required to yield a /∫/ percept. Consistent with earlier findings for other consonantal distinctions, durational contrast occurred between the vowel and fricative. [Work supported by NINCDS.]

Auditory signals in military aircraft: ergonomics principles versus practice

The complete ensembles of auditory signals in selected USAF aircraft (the F-4D, F-15, two models of the F-16, the C-5, and the C-141) are described and evaluated. Human factors research related to the design of speech and non-speech auditory signals is reviewed. Major findings are: that auditory signals are not well standardised among the aircraft, even between those with similar combat roles; that a relatively large number of non-speech auditory signals are used, which may make it difficult for the aircrew to recall the meanings of all the signals; that some non-speech signals are sufficiently similar that they may be confused, particularly in high workload and stressful conditions; and that the criticality of the warnings is not reliably indicated by any characteristic of the signals. Four problem areas requiring further research are discussed: reduction of signal loudness, enhancement of the distinctiveness and masking resistance of non-speech signals, effects of concurrent warning signals on aircrew performance, and additional uses of auditory information.

Speech transformations based on a sinusoidal representation

In this paper a new speech analysis/synthesis technique is presented which provides the basis for a general class of speech transformations including time-scale modification, frequency scaling, and pitch modification. These modifications can be performed with a time-varying change, permitting continuous adjustment of a speaker's fundamental frequency and rate of articulation. The method is based on a sinusoidal representation of the speech production mechanism which has been shown to produce synthetic speech that preserves the wave-form shape and is perceptually indistinguishable from the original. Although the analysis/synthesis system was originally designed for single-speaker signals, it is also capable of recovering and modifying nonspeech signals such as music, multiple speakers, marine biologic sounds, and speakers in the presence of interferences such as noise and musical backgrounds.

Continuous speech recognition

An improved speech recognition method and apparatus for recognizing keywords in a continuous audio signal are disclosed. The keywords, generally either a word or a string of words, are each represented by an element template defined by a plurality of target patterns. Each target pattern is represented by a plurality of statistics describing the expected behavior of a group of spectra selected from plural short-term spectra generated by processing of the incoming audio. The incoming audio spectra are processed to enhance the separation between the spectral pattern classes during later analysis. The processed audio spectra are grouped into multi-frame spectral patterns and are compared, using likelihood statistics, with the target patterns of the element templates. Each multi-frame pattern is forced to contribute to each of a plurality of pattern scores as represented by the element templates. The method and apparatus use speaker independent word models during the training stage to generate, automatically, improved target patterns. The apparatus and method further employ grammatical syntax during the training stage for identifying the beginning and ending boundaries of unknown keywords. Recognition is further improved by use of a plurality of templates representing "silence" or non-speech signals, for example, hum. Also, memory and computation load is reduced by use of modified (collapsed or folded) syntax flow graph logic, implemented by additional (augment) control numbers. A concatenation technique is employed, using dynamic programming techniques, to determine the correct identity of the word string.

Application of smoothed Wigner distribution (WD) to speech signals

The WD is a powerful tool for the simultaneous time‐frequency analysis of time‐varying signals [Martin and Flandrin, IEEE Trans. Acoust. Speech Signal Process. ASSP‐33 (Dec. 1985)]. For signals with simple time‐frequency structures (e.g., frequency‐modulated signals), the WD provides excellent resolution in the time‐frequency (t − ω) plane, as compared with conventional short‐time spectral analysis techniques. The interpretation of the WD becomes more difficult as the time‐frequency relationship of the signal gets more complex (e.g., in speech), because of the presence of cross terms and of regions below (i.e., negative) the t − ω plane. This disadvantage can be overcome by smoothing the WD in the t and ω directions over a region TΩ > 1 (T and Ω are the “smear” values in time and frequency) [Janssen and Claasen, IEEE Trans. Acoust. Speech Signal Process. ASSP‐33 (Aug. 1985)]. Such a smoothing process, however, makes the WD and the spectrogram equivalent in terms of the time and frequency resolutions. This presentation is directed towards comparing the performances of the WD (with partial smoothing, i.e., T < 1) and the spectrogram. Nonspeech signals as well as speech signals will be used to demonstrate the application of WD to the analysis of speech signals. [Work supported by NSERC, Canada.]

Speech–nonspeech detector-classifier

The classification of a signal on a telephone line can be based on its short term average energy level; nonspeech (data or signalling) as high level; speech as medium level, and noise as low level. A common use for speech detectors occurs with a time assignment speech interpolation (TASI) system. A speech detector extends a trunk activity signal (TAS) for indicating whether energy or noise is detected on a trunk. Responsive to the TAS signal, a processor processes a signal on the trunk. On occasion, it may be desirable to process different energy signals according to different methodologies. For example, during overload, a TASI system may truncate bits of a speech signal and not truncate bits of a nonspeech signal. For classifying energy signals as between speech signals and nonspeech signals, the instant classifier includes an arrangement for extracting at least one feature, or property, such as the short term power of the signal to be classified. Responsive to the extracted property, an estimator provides a coarse estimate as to whether the signal to be classified is a speech signal or a nonspeech signal. Responsive to the estimate, a six-state, sequency machine provides a fine, classifying signal for classifying the energy signal.

Noncategorical identification of rise time.

The view that speech and certain nonspeech signals such as music are perceived categorically has been contradicted by many studies. In most studies, subjects identified many different stimuli as belonging to one of two classes (e.g., “pluck” and “bow”). To determine whether subjects can identify exactly each of a number of different stimuli, subjects heard a sawtooth wave contained within six different amplitude envelopes. In one task, subjects learned the envelopes by rise-time value (10, 20, 30, 40, 50, and 60 msec), and later identified them by pressing labeled keys. Analysis of responses showed that accuracy of estimating envelope rise time is not worse than ± 10 msec. It is concluded that subjects trained in auditory perception do not classify a continuum of sawtooth waveforms varying in rise time into two sharply defined categories, but are able to resolve rise-time values with much greater accuracy than is achieved by simply dividing the 10–60 msec range into two categories such as “pluck” and “bow.”

Perception of duration of medial silent intervals in speech and nonspeech signals

Vowels are longer before voiced consonants than before voiceless consonants, and a number of studies have demonstrated that perception of voicing, in syllable‐final and intervocalic positions, is in accordance with this production regularity. We present a study, using nonspeech stimuli, that suggests that this agreement between production and perception for segmental duration is grounded in auditory processing of complex stimuli in general. Two types of square‐wave analogs, modeled after a speech series ranging from “aba” to “apa” were constructed. The two nonspeech stimulus series varied in duration of a silent interval from 20–110 ms bounded by frequency transitions out of and into steady‐state segments. One series had an initial segment duration of 155 ms, whereas, the initial segment duration of the other series was 245 ms. Duration of the final segment was 188 ms for both series. Listeners judged significantly more of these nonspeech stimuli to be similar to the short silent interval endpoint when the initial segment was long than when the initial segment was short. This result parallels the finding for speech that listeners perceived more voiced /b/ than voiceless /p/ consonants following longer vowels. These results call into question explanations of this production/perception concordance that suggest either mediation of perception by knowledge of production regularities, or direct perception or articulation. [Work supported by NICHHD.]

Infants' recognition of cross-modal correspondences for speech: Is it based on physics or phonetics

Four-month-old infants recognize cross-modal correspondences between the auditory and visual concomitants of speech (Kuhl and Meltzoff, 1982). In those experiments, infants were shown a filmed display of two faces, one producing the articulatory movements corresponding to /a/ and the other producing the articulatory movements corresponding to /i/. One of the sound tracks (either /a/ or /i/) was played in synchrony with the faces. Infants' visual fixations to the faces were scored. Results demonstrated that infants looked longer at the face that matched the sound being presented. Infants who heard /a/ looked longer at the /a/ face, while infants who heard /i/ looked longer at the /i/ face. We now report an experiment designed to dissociate two potential explanations for this cross-modal effect, one based on “physics,” the other on “phonetics.” We replicated the experiment, using the same visual stimuli but altered auditory stimuli. The auditory signals were pure tones that followed the first three formant frequencies of the /a/ and /i/ vowels. Results showed that listening to the nonspeech signals produced a complete reversal of the effect. When listening to the pure-tone analog of /a/, infants looked longer at /i/. When listening to the pure-tone analog of /i/, infants looked longer at /a/. [Supported by NSF and NICHD.]

Susceptibility to the effects of delayed auditory feedback

Speech performance deteriorates if a delay is introduced between the time when speakers speak a sound and when they hear it (DAF). Most disturbance occurs when speech is delayed by about 200 msec. Not all speakers are equally susceptible to the effects of DAF. The hypothesis examined here is that deterioration in performing a serially organized behavior (such as speech)occurs when a rhythmic event goes on concurrent with but out of synchrony with the activity. Different susceptibility to the effects of DAF can be accounted for by this hypothesis because the out-of-synchrony signal would differ in loudness, depending on the amount by which speakers raise their voices under DAF. Two experiments are reported which are consistent with this hypothesis. In Experiment 1, it is shown that changes in voice level brought about by voluntary or electronic means increase the disturbance observed. In Experiment 2, it is shown that the out-of-synchrony signal does not have to be speech to produce problems for speech control, because equivalent disruption occurs when a nonspeech signal at the same level as the speech is substituted for the delayed speech sound.

Infants' discrimination of the duration of a rapid spectrum change in nonspeech signals.

Two-month-old infants discriminated complex sinusoidal patterns that varied in the duration of their initial frequency transitions. Discrimination of these nonspeech sinusoidal patterns was a function of both the duration of the transitions and the total duration of the stimulus pattern. This contextual effect was observed even though the information specifying stimulus duration occurred after the transitional information. These findings parallel those observed with infants for perception of synthetic speech stimuli. Specialized speech processing capacities are thus not required to account for infants' sensitivity to contextual effects in acoustic signals, whether speech or nonspeech.

Perception of the duration of rapid spectrum changes in speech and nonspeech signals.

For a number of years, investigators have studied the effects one acoustic segment has on the perception of other acoustic segments. In one recent study, Miller and Liberman (Perception & Psychophysics, 1979,25, 457–465) reported that overall syllable duration influences the location of the labeling boundary between the stop [bl and the semivowel [w]. They interpreted this “context effect” as reflecting a form of perceptual normalization whereby the listener readjusts his perceptual apparatus to take account of the differences in rate of articulation of the talker. In the present paper, we report the results of several comparisons between speech and nonspeech control signals. We observed comparable context effects for perception of the duration of rapid spectrum changes as a function of overall duration of the stimulus with both speech and nonspeech signals. The results with nonspeech control signals therefore call into question the earlier claims of Miller and Liberman by demonstrating clearly that context effects are not peculiar to the perception of speech signals or to normalization of speaking rate. Rather, such context effects may simply reflect general psychophysical principles that influence the perceptual categorization and discrimination of all acoustic signals, whether speech or nonspeech.

Effect and artifact in the auditory discrimination of rise and decay time: Speech and nonspeech

The just noticeable difference (JND) for abruptness so far has not been studied in speech-like signals, and earlier studies have confounded decay time and overall signal duration. We therefore established JND for rise and decay time in a 10–80-msec range in a series of experiments involving an adjustment method using various speech and nonspeech signal types. Decay time did or did not vary independently of duration. Results showed that JND is in the order of 25%–30% of the reference value, with no essential difference between rise- and decay-time discrimination if these parameters are varied independently of duration. Decay-time discrimination turned out to be more accurate for wide-band signal types (noise) in the upper part of the stimulus range than it did for narrow-band signals (tones and complex harmonic signals). The data suggest that rise- and decay-time discrimination is too poor to reliably cue more than two categories in spite of the wide range of values found in speech sounds.

Perception of the duration of rapid spectrum changes: Evidence for context effects with speech and nonspeech signals

For a number of years investigators have focused attention on the effects one acoustic segment has on the perception of other acoustic segments. In one study, Miller and Liberman [Percept. Psychophys. 25, 457–465 (1979)] reported that overall syllable duration influences the location of the labeling boundary between the stop /b/ and the semivowel /w/. They claim that this “context effect” reflects a form of perceptual normalization whereby the listener somehow readjusts his perceptual apparatus to take account of the difference in rate of articulation of the talker. More recently, Eimas and Miller [Science (in press)] have reported that prelinguistic infants also show similar context effects in discrimination of these stimuli. The inference drawn from the latter study is that infants perceive these speech sounds like adults and the context effects reflect the operation of perceptual mechanisms that underlie a phonetic‐like mode of processing specific to speech. In the present study, we carried out several critical comparisons between speech and nonspeech signals and observed comparable context effects for perception of the duration of rapid spectrum changes as a function of overall duration of the stimulus. Our results with nonspeech signals falsify both of the earlier claims by demonstrating clearly that these sorts of context effects are not peculiar to the perception of speech signals or to normalization of speaking rate. [Work supported by NIMH Research Grant MH‐24027‐06.]

Perceptual discrimination of rise and decay time in speech‐ and nonspeech signals

In a number of languages, differences in the decay time of vowels have been claimed to contribute to the perception of phonemic contrasts. In contrast with many other acoustic parameters relevant to speech perception, the perceptual acuity for abruptness phenomena has hardly been subjected to systematic psychophysical investigation. Previous experiments [V. J. J.P. van Heuven and M. P. R. van den Broecke, J. Acoust. Soc. Am. 66, 1308–1315 (1979)] have demonstrated that the JND is about 25% for rise and decay time differences in tone and noise bursts. Further experiments on vowel‐like signals seemed to demonstrate a superior accuracy for offsets as compared to onsets. However, interaction of these offsets with overall duration may have played a role. These experiments have been replicated in order to create full comparability of rise‐ and decay‐time signal conditions with respect to overall signal duration. Results indicate that perceptual acuity for rise and decay time in complex harmonic signals, including vowel‐like sounds, is essentially symmetrical and in the same magnitude of those earlier established for tone and noise bursts. Data will be presented and implications with respect to our earlier findings will be discussed.

Adaptation of the relative onset time of two-component tones

The results of three selective adaptation experiments employing nonspeech signals that differed in temporal onset are reported. In one experiment, adaptation effects were observed when both the adapting and test stimuli were selected from the same nonspeech test continuum. This result was interpreted as evidence for selective processing of temporal order information in nonspeech signals. Two additional experiments tested for the presence of cross-series adaptation effects from speech to nonspeech and then from nonspeech to speech. Both experiments failed to show any evidence of cross-series adaptation effects, implying a possible dissociation between perceptual classes of speech and nonspeech signals in processing temporal order information. Despite the absence of cross-series effects, it is argued that the ability of the auditory system to process temporal order information may still provide a possible basis for explaining the perception of voicing in stops that differ in VOT. The results of the present experiments, taken together with earlier findings on the perception of temporal onset in nonspeech signals, were viewed as an example of the way spoken language has exploited the basic sensory capabilities of the auditory system to signal phonetic differences.