Spectrogram Reading Research Articles

Towards an expert system for automatic stop consonant identification

The adequacies of three sets of acoustic measurements for stop consonant recognition were assessed by examining the performance of a trained human spectrogram reader in a stop identification task. The data sets consisted principally of(1) gray‐level pictures (wideband spectrograms); (2) two‐dimensional line drawings (LPC spectra and short‐time energy contours); and (3) tables of numerical measurements (e.g., frequencies, amplitudes, and bandwidths of spectral peaks). The reader's task was to identify the initial consonant in /CVb/ syllables selected from a corpus containing all combinations of the six stop consonants /b,d,g,p,t,k/ with the six vowels /i,e,ae,a,o,u/ as spoken by three male and three female talkers. The objective of the experiment was to identify an appropriate set of measurements for use in a knowledge‐based expert system for automatic stop consonant identification. Results of the experiment will be presented and discussed in the context of this objective.

The identification of prosodic information from acoustic records by phoneticians

Previous work [S. G. Svensson (1974). Prosody and grammar in speech perception. Monographs Instit. Ling. Univ. Stockholm] on spectrogram reading using Swedish utterances drawn from a very large corpus indicates that highly improved scores can be obtained if subjects are provided with (1) a grammar- and prosody-based decoding strategy, and (2) an error-free specification of the prosodic structure (stress and word accent information) of each test utterance spectrographically portrayed. The present study attempts to examine the realism of the assumption that the prosodic parameters can be extracted bottom-up independently of the interpretation of an utterance. Test phrases were recorded in a fixed sentence frame and the following variables were controlled: phrase length (2–4 syllables), three degrees of stress, word accent (grave versus acute), acoustic salience of syllable boundaries (sonorants versus voice-less obstruents), and semantic content (meaningful versus non-sense phrase). Subjects had considerable experience from research on the acoustic features of Swedish prosody and responded for each item to a set of displays (F0, intensity, duplex and microphone signals) by indicating: the number of syllables, syllable stress, and word accents. The aim of this paper is to report on the identification of these aspects as a function of the experimental variables.

The use of context in spectrogram reading

In order to assess the role of syntactic, semantic, and discourse knowledge in spectrogram reading, three short stories were recorded and speech spectrograms were made of the individual sentences of each story. The stories were presented one spectrogram at a time to an expert spectrogram reader who is instructed to read each story word-by-word without writing down segment labels. The three stories contained a total of 670 words, and 612 (91%) were correctly identified. The median reading time per sentence across the three stories was about 40 s, or about 20 times real time. However, syllable-by-syllable analysis of reading times in one story reveal that this value was inflated by a small subset of syllables and words that took a long time to decode—sometimes over a minute. The modal (most frequent) reading time per syllable fell between 1–2 s, or between 3 and 6 times real-time. Further analysis reveal that many common syllables were immediately recognized as complete patterns (e.g., “ment”, “tion”), and the use of context to recognize words from partial information was evident in many cases. Implications of the results for real-time spectrogram reading will be discussed.

Performance of an expert spectrogram reader

The subject in our study has spent between 2000–2500 h learning to read speech spectrograms. We tested the subject's ability to identify the phonetic content of broadband speech spectrograms of unknown utterances during eight separate sessions of about 4 h each. The expert was presented with 23 spectrograms of English sentences and sequences of words and nonsense words, and 45 English words embedded in a known carrier phrase. The phonetic labels produced by the expert agreed with the phonetic labels produced by trained phoneticians (who listened to the speech) between 80% and 90% of the time, depending upon the scoring method used. When presented with words in a known carrier phrase, labeling performance improved to about 93%. A linguist who was presented with the phonetic transcriptions produced by the spectrogram reader was unable to identify all of the words in ten of 15 utterances, and missed only a single work in each of the remaining five. A film will be shown of the expert segmenting and labeling two spectrograms of unknown utterances.

Performance of an expert spectrogram reader

The subject in our study has spent between 2000 and 2500 h learning to read speech spectrograms. We tested the subject's ability to identify the phonetic content of broadband speech spectrograms of unknown utterances during eight separate sessions of about 4 h each. The expert was presented with 23 spectrograms of English sentences and sequences of words and nonsense words, and 50 English words embedded in a known carrier phrase. The phonetic labels produced by the expert agreed with the phonetic labels produced by trained phoneticians (who listened to the speech) between 80% and 90% of the time, depending upon the scoring method used. When presented with words in a known carrier phrase, labeling performance improved to about 95%. A linguist who was presented with the phonetic transcriptions produced by the spectrogram reader was able to identify all of the words in 10 of 15 utterances, and missed only a single word in each of the remaining five. A film will be shown of the expert segmenting and labeling two spectrograms of unknown utterances.

Performance of an expert spectrogram reader. II

Analysis of the videotaped performance of an expert spectrogram reader revealed the control strategy and sources of knowledge used by the expert while reading spectrograms. Spectrograms were read in two stages: segmentation and labeling. During segmentation, which was usually performed in a single pass from left to right, segment markers were placed on the spectrogram to indicate roughly phone‐sized units. Segmentation was accomplished through the detection of spectral discontinuities, although the precise placement of segment markers was guided by a detailed knowledge of acoustic phonetics. Labeling was not performed left to fight, and was usually guided by an “easiest segments first” strategy. The vast majority of segments were identified through the recognition of characteristic patterns and salient features. The identification of other segments required the application of knowledge about coarticulation and phonological rules. Several experiments revealed that the expert did not use higher‐order knowledge (e.g., lexical, syntactic, or semantic) during labeling—performance was slightly better on nonsense utterances than English sentences. Implications of the expert's performance will be discussed for theories of speech perception, machine recognition of speech, and the development of aids for the speech‐ and hearing‐impaired.

Interaction between segmental and nonsegmental factors in speech recognition

The present study demonstrates that spectrograms of Swedish utterances can be read with great accuracy under nontrivial conditions. This result is to be attributed primarily to the development of a formalized strategy that was designed to make it possible for spectrogram readers to derive information on certain grammatical features of an utterance such as word class, word boundaries, endings, and function elements. The input to this strategy consists of segmental phonetic features that the subjects extract from the spectrographic display and of information on prosodic features such as stress and tonal accent. The latter information is specified on the spectrogram for each syllable. An experimental situation is thus created that differs from the informal recognition of unknown utterances from spectrograms. A subject can base his final identification of lexical items not only on segmental phonetic features but also on an error-free specification of prosodic features and, in so far as he has been able to use the strategy successfully, on grammatical information. Experimental results are reported indicating that subjects improve their performance markedly with the aid of the strategy. In conclusion, attention is drawn to the important role that grammar and prosody appear to play in the present experiments and to the implications of the findings for future work on automatic speech recognition and speech perception.

Organizing a System for Continuous Speech Understanding

In spectrogram reading experiments [Klatt and Stevens, Boston Speech Conference (1972)], the performance obtained by human experts for phonetic segmentation and labeling was approximately 75% partially or completely labeled but correctly specified, 15% mislabeled, and 10% segments missed. When syntactic, semantic, and vocabulary constraints were applied, the success rate in identifying words was 96%. How does one design a computer system to match this performance at word identification? This paper describes the BBN speech understanding system, which is being designed to extract the meaning from continuous speech inputs in a limited semantic environment by using syntactic, semantic, and vocabulary information to overcome vagueness and errors in phonetic segmentation and labeling. The system contains components for acoustic feature extraction, phonetic segmentation and labeling, word retrieval from partial phonetic information, evaluation of word-match quality, and syntactic and semantic evaluation of partial utterances and proposal of words from context. A control component embodies the strategy for applying different components at appropriate times and places depending on conditions which will vary from point to point in an analysis.