Spectrogram Reading Research Articles

An integrated approach for teaching speech spectrogram analysis to engineering students.

Spectrogram analysis is a vital skill for learning speech acoustics. Spectrograms are necessary for visualizing cause-effect relationships between speech articulator movements and the resulting sound produced. However, many interpretation techniques needed to read spectrograms are counterintuitive to engineering students who have been taught to use more rigid mathematical formulas. As a result, spectrogram reading is often challenging for these students who do not have prior background in acoustic phonetics. In this paper, a structured, inclusive framework for teaching spectrogram reading to students of engineering backgrounds is presented. Findings from the implementation of these teaching methods in undergraduate and graduate engineering courses at University of California, Los Angeles are also unveiled.

Artikulatorische und akustische Phonetik (review)

Reviewed by: Artikulatorische und akustische Phonetik by Henning Reetz Karen Steffen Chung Artikulatorische und akustische Phonetik. By Henning Reetz. Trier: Wissenschaftlicher Verlag Trier, 1999. Pp. 200. ISBN 3884763377. €23.50. This book is designed as a university-level textbook for German-speaking students of phonetics who come from liberal arts backgrounds and are relatively weak in math and physics. While somewhat similar in coverage and tone to Peter Ladefoged’s Elements of acoustic phonetics (2nd edn., Chicago: University of Chicago Press, 1996), it includes materials not in Elements or most other books of this type, such as detailed information on the hearing mechanism. Reetz manages to make difficult material extremely accessible, clear, and even fun to go through and learn. It is easy to understand how he earned a teaching award in 2001. A listing of major topics covered in the two main parts of the text, Parts 2 and 3, provides a clear idea of what to expect from this volume. Part 2 explores sound waves, speed of sound, wave types, microphones, waveforms, frequency, wave length, amplitude, RMS amplitude, decibels, SPL, phase, resonance and damping, filtering and digitalization of acoustic signals, sampling rates, speech analysis and processing, Fourier transforms, spectra, harmonics, windowing, spectrogram reading, cepstra, LPC, and the mel, Bark, and ERB pitch scales. R provides useful definitions of five different types of sounds: Ton ‘tone’, Klang ‘complex signal’, Geräusch ‘sound’, Rauschen ‘noise’, and Knall ‘impulse’. English equivalents are given for each key term and usually their etymologies as well, a thoughtful touch for language-oriented readers. The text abounds in apt analogies taken from everyday life, which are often accompanied by clear and frequently charming illustrations, such as the Katzenjammer Kids-type drawings used to explain the propagation of transverse and longitudinal waves (6, 9). Reference to the Alphorn (35) to illustrate tube acoustics adds a touch of local color from the author’s home region. Part 3 covers the anatomy, physiology, and dynamics of the speech and hearing organs, the Bernoulli effect, interactions between the vocal folds and vocal tract, the evolution of human hearing, and various theories of how the hearing mechanism works, including some of the most recent findings in the field. Formants and the acoustic structure of individual sounds are also covered here. R describes with great clarity the problem of how a biological system with definite reaction and transmission speed limits can successfully perceive high frequencies (170). The three appendices include background in basic physics (mass, weight, and pressure; energy, power, and intensity) for nonscience students, definitions of physical terms with English and sometimes Latin equivalents, and average formant and fundamental frequency values for men, women, and children. There is an English and a German index of key concepts, a bibliography, and recommendations for further reading. A few dozen typos were spotted, mostly superfluous stray verbs. These may be corrected in the newest edition (2003). This book offers a rare treat for anyone who happens to like both German and phonetics. An English version should soon be available from Blackwell if it is not already. Karen Steffen Chung National Taiwan University Copyright © 2005 Linguistic Society of America

Forensic study of a case involving SMS text-to-speech conversion

The process of decoding the content of a message involving Short Message-to-speech conversion is described in the form of a forensic case study. The intelligibility of the message was low, primarily due to the fact that the written input to the text-to-speech system was inTurkish although the system was designed for text-to-speech conversion in German. This created some form of synthesised Turkish with a German “accent”. It is shown how “spectrogram reading” can be employed in the task of deriving phonetic categories from the signal of the synthetic speech message. In order to obtain a maximally accurate estimate of the input typed by the sender, one disputed passage of the SMS-to-speech message was subjected to a simulation. It is described how a full match between the case material and the simulated passage was obtained, despite typographical errors and incorrect Turkish orthography on the part of the sender.

Speechreading aids based on automatic speech recognition: Prospects for the automatic generation of cued speech

Current progress in the development of automatic speech recognition (ASR) systems may soon permit discrete symbolic speechreading supplements to be derived from the speech signal. Such supplements could be similar to those used in manual cued speech, in which the talker uses discrete hand positions and shapes to provide distinctions between constants and vowels that are often confused in speechreading. Highly trained receivers of manual cued speech can achieve nearly perfect reception of everyday connected speech materials at normal speaking rates through the visual sense alone. To understand the accuracy that might be achieved with automatically generated cues, we measured how well trained spectrogram readers and an automatic speech recognizer could assign cues for various cue systems. A model of audiovisual integration was then applied to these measurements and data on human recognition of consonant and vowel segments via speechreading was published. This analysis suggests that with cues derived from current recognizers, consonant and vowel segments can be received with accuracies in excess of 80%, roughly equivalent to the segment reception accuracy required to account for observed levels of manual cued speech reception. To provide guidance for the development of automatic cueing systems, we describe techniques for determining optimum cue groups for a given recognizer and speechreader, and estimate the cueing performance that might be achieved if the performance of current recognizers were improved.

A knowledge-based system for stop consonant identification based on speech spectrogram reading

In order to formalize the information used in spectrogram reading, a knowledge-based system for identifying spoken stop consonants was developed. Speech spectrogram reading involves interpreting the acoustic patterns in the image to determine the spoken utterance. One must selectively attend to many different acoustic cues, interpret their significance in light of other evidence, and make inferences based on information from multiple sources. The evidence, obtained from both spectrogram reading experiments and from teaching spectrogram reading, indicates that the process can be modeled with rules. Formalizing spectrogram reading entails refining the language used to describe acoustic events in the spectrogram, selecting a set of relevant acoustic events that distinguish among phonemes, and developing rules which map these acoustic attributes into phonemes. One way to assess how well the knowledge used by experts has been captured is by embedding the rules in a computer program. A knowledge-based system was selected because the expression and use of knowledge are explicit. The emphasis was in capturing the acoustic descriptions and modeling the reasoning used by human spectrogram readers. In this paper, the knowledge acquisition and knowledge representation, in terms of descriptions and rules, are described. A performance evaluation and error analysis are also provided, and the performance of an HMM-based phone recognizer on the same test data is given for comparison.

A semiautomated methodology for knowledge elicitation

A semiautomated methodology for eliciting knowledge from an expert is described. The methodology suggests the use of a manual interview technique and the application of a machine learning program to structure the elicited information. This way the experts are free to express their knowledge in a familiar format. Automated processing helps the knowledge engineer formalize knowledge. This interaction between manual and automatic techniques is particularly useful when the expert meets with difficulties in formulating general rules and prefers, instead, to reason by cases. The application of the methodology to an image interpretation task, namely speech spectrogram reading, is described as a test case.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Morphological approaches to the automatic extraction of phonetic features

An experiment in speech analysis in which a a parallel processor uses morphological algorithms to extract phonetic features from a spectrogram and performs an initial segmentation and labeling is described. Experiments in spectrogram reading by R. A. Cole et al. (1980) have suggested that more information is present in, and furthermore, this information may reside in the spectrogram, i.e., the speech image. The spectrogram readings of V. Zue et al. (1980) are automated by using image processing techniques in an image processor. A very fast and powerful parallel pipeline image processor, the cytocomputer, is used. The cytocomputer contains a serial pipeline of programmable processing stages, where each stage performs a single cellular transformation on the entire image.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Answers to spectrograms in 20.1

In the previous issue of the Journal, on page 52, there were reproductions of two spectrograms that readers were invited to interpret. Both had been tested by presenting them to a group of about half a dozen experienced spectrogram readers. This group discussed the reproductions, arguing for different interpretations until they had reached a consensus on what phrases must have been spoken. The following are some comments that readers might wish to consider, based on the group's procedure in interpreting the first phrase, which is reproduced again below. Remember that the spectrograms contain English sentences spoken in a British accent close to the RP standard.

Phoneme segmentation expert system using spectrogram reading knowledge

AbstractA human expert is able to determine phoneme boundaries through spectrogram reading with high accuracy using one's knowledge and strategy. The authors developed a phoneme segmentation expert system which simulates a human expert spectrogram reading process using knowledge and strategy. This expert system detects phonemes from a continuous speech spectrogram and determines phoneme boundaries with their coarsely classified phoneme according to human expert knowledge and strategy which is described as rules. An assumption‐based inference with certainty factors and top‐down acoustic feature extraction under phoneme context hypotheses are incorporated into the expert system to simulate the phoneme contextual fuzzy knowledge and strategy used by a human expert. This framework makes it possible to deal with phoneme variations caused by coarticulation, and to extract accurate acoustic features on a spectrogram for phoneme segmentation. Japanese consonant segmentation knowledge currently is incorporated into the system which detected 90.8 percent of the phonemes correctly. In particular, the average alignment error of phoneme boundaries detected by the system was 5.8 ms, which is as accurate as those detected by a human expert.

Phoneme recognition expert system using spectrogram reading knowledge and neural networks

AbstractA human expert spectrogram reader is able to recognize phonemes in high‐accuracy performing phoneme segmentation and phoneme identification simultaneously using one's spectrogram reading knowledge. Spectrogram reading knowledge consists mainly of two parts: a strategic part for phoneme segmentation and identification and a pattern‐matching part. There are several knowledge‐based approaches in which all knowledge is implemented as rules. However, it is especially difficult to describe the pattern matching part of the knowledge as rules and to extract acoustic features automatically for phoneme identification. Here, we construct a phoneme recognition expert system which consists of two parts: (1) rule‐based phoneme segmentation, and (2) neural network‐based phoneme identification for knowledge such as pattern matching. This paper presents the architecture of the phoneme recognition expert system with its experimental result tested on Japanese consonants. The experimental result shows that 90.8 percent of the phonemes were segmented correctly and 92.4 percent of the phonemes were identified correctly within the correct segments, which means 83.9 percent of the phonemes were correctly recognized both in segmentation and identification.

Dennis Klatt's contribution to automatic speech recognition

Over the past 20 years, Dennis Klatt has made enormous contributions to the field of automatic speech recognition through his research, writing, and student supervision. In the early 70's, he served as a member of the steering committee of the ARPA Speech Understanding Research (SUR) program, providing leadership and guidance to the research community. He also participated actively in speech recognition research, first performing a set of spectrogram reading experiments assessing the role of various sources of knowledge, and later investigating the use of synthesis‐by‐rule techniques for word verification. Out of this involvement with the ARPA‐SUR program came the landmark paper reviewing its technical achievement. as well as several publications describing his own proposals, LAFS and SCRIBER, for human and machine speech recognition. Over the past 10 years, Dennis directed his attention to the design of signal representation front‐ends, as well as the investigation of perceptually motivated distance metrics in order to implement his speech recognition models. This talk pays tribute to Dennis's incredibly active research life by examining the legacy he left behind in speech recognition.

Speaker-independent phone recognition using hidden Markov models

Hidden Markov modeling is extended to speaker-independent phone recognition. Using multiple codebooks of various linear-predictive-coding (LPC) parameters and discrete hidden Markov models (HMMs) the authors obtain a speaker-independent phone recognition accuracy of 58.8-73.8% on the TIMIT database, depending on the type of acoustic and language models used. In comparison, the performance of expert spectrogram readers is only 69% without use of higher level knowledge. The authors introduce the co-occurrence smoothing algorithm, which enables accurate recognition even with very limited training data. Since the results were evaluated on a standard database, they can be used as benchmarks to evaluate future systems.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Speaker‐independent phoneme recognition using hidden Markov models

In this paper, the currently popular hidden Markov modeling to speaker‐independent phoneme recognition is extended. Using multiple code books of various LPC‐derived parameters and discrete HMMs, speaker‐independent phoneme recognition accuracy of 58.8%–73.8% on the DARPA TIMIT database, depending on the type of acoustic and language models used, is obtained. In comparison, the performance of expert spectrogram readers is only 69% without use of higher level knowledge. The co‐occurrence smoothing algorithm that enables accurate recognition with only a few training examples of each phone is also introduced. Since these results were evaluated on a standard database, they can be used as benchmarks to evaluate future systems. [Work supported by DARPA.]

Relaxation-based speech labeling

It was revealed that a trained human spectrogram reader could perform accurate speech labeling, and that accuracy was based on the flexibility of his/her decision process using many kinds of spectrographic features [S. Katagiri, SP87-115, IEICE Tech. Rep. (1988)]. In this paper, a new flexible speech labeling system that simulates the trained reader capability is proposed. The main task of the system is to apply the trial-and-error process used in a human reader's labeling work. Therefore, a relaxation method is adopted here [S. Katagiri, 2-1-19, A. S. J. Spring Meeting (1988)]. The system consists of three parts: an acoustic analyzer, a verifier, and a supervisor. In the acoustic analyzer, many kinds of acoustic feature candidates, e.g., formant and pitch frequencies, are calculated. In the verifier, possible speech labels are verified. The supervisor, with a behavior principle based on the relaxation method, controls the whole system. Experimental results show that the system's performance is comparable to a human reader's performance and that very accurate labels are automatically created.

Recognition of continuously spoken letters by listeners and spectrogram readers

Because of acoustic similarities between some letters of the alphabet, automatic recognition of continuously spoken letters is a difficult task. The goal of this study is to determine and compare how well listeners and spectrogram readers can recognize continuously spoken letter strings from multiple speakers. The interest in spectrogram reading results is motivated by the belief that this procedure may help to identify acoustic attributes and decision strategies that are useful for system implementation. Listening and spectrogram reading tests involving eight listeners and six spectrogram readers, respectively, were conducted using a corpus of 1000 wordlike strings designed to minimize the use of lexical knowledge. Results show that listeners' performance was better than readers' (98.4% vs 91.0%). In both experiments, string lengths were determined very accurately (98.1% and 96.2%), presumably due to the large number of glottal stops inserted at letter boundaries to facilitate segmentation. Most of the errors were due to substitution of one letter for another (68% and 92%), and they generally fall into two categories. Asymmetric errors can often be attributed to subjects' disregard for contextual influence, whereas symmetric errors are largely due to acoustic similarities between certain letter pairs. Subsequent acoustic study of four of the most confusable letter pairs has resulted in the identification of a number of distinguishing acoustic attributes. Using these attributes, overall recognition performance better than that of the readers was achieved. [Work supported by NSF and DARPA under contract N00014-82-K-0727, monitored through the Office of Naval Research.]

APHODEX, AN ACOUSTIC-PHONETIC DECODING EXPERT SYSTEM

The acoustic-phonetic decoding of sentences is a major bottleneck in continuous speech recognition. Our group has been working in this area for the past twelve years. In order to improve the accuracy of present phonetic decoders, we are now developing an expert system called APHODEX that should emulate the competence of an experienced spectrogram reader. After a brief description of the methodology used to elucidate the expert's knowledge and strategies, we focus on the expert's approach and results. Finally we explain how this expertise has been implemented; some results from the present experimental version of our system are given.

Some results of speech analysis using a parallel image processor

Special purpose parallel pipelined image processors are not only faster for images than general purpose computers but also allow one to investigate the applications of the latest image processing algorithms and techniques to continuous speech. This is particularly valuable when trying to automate the spectrogram reading capabilities of Victor Zue [Proc. IEEE 73, 1602–1615 (1985)]. Several sentences of continuous speech were sampled and converted to images using the FFT. These images were analyzed on a parallel cellular image processor called the Cytocomputer™. Suitable structuring elements were selected to smooth the inherently bumpy FFT spectra and to remove noise. Several of the distinctive features of speech are described in mathematical morphology terms and these features were tracked. Specifically, stop consonants, fricatives, vowels, and nasals were located and labeled. Examples of intermediate image transformations and resultant labeling were recorded by photographic process. [Work supported by Ohi...

Automatic speech recognition based on spectrogram reading

An approach to the problem of automatic speech recognition based on spectrogram reading is described. Firstly, the process of spectrogram reading by humans is discussed, and experimental findings presented which confirm that it is possible to learn to carry out such a process with some success. Secondly, a knowledge-engineering approach to the automation of the linguistic transcription of spectrograms is described and some results are presented. It is concluded that the approach described here offers the promise of progress towards the automatic recognition of multi-speaker continuous speech.

Knowledge‐based recognition of voiceless stop consonants

Recent experiments [M. Bush et al., Proc. 1983 IEEE ICASSP] have demonstrated the ability of a trained spectrogram reader to identify initial stops in /CVb/ syllables from a table of numerical acoustic measurements with approximately 80% accuracy. This paper discusses an automatic system for discriminating between the voiceless plosives (/p,t,k/)which is based on the features and rules identified in these experiments. Ten binary features are extracted from two linear prediction spectra which are computed during the 35 ms following the consonant release. Typical features include “back‐k‐release‐spectrum” and “compact‐release‐spectrum.” The features are detected by examining the frequencies and amplitudes of the local maxima and minima of the two LPC spectra, in a manner motivted by the actions of the human spectrogram reader. A simple statistical classifier is used to combine the outputs of the ten feature detectors. The classifier was trained on the 108 /p,t,k/ tokens of the multi‐speaker corpus used in the spectrogram reading experiment. When tested on the training data, the system achieved 96% correct recognition. When tested on two additional data sets of similar composition, the system achieved scores of 94% and 92%, respectively.

Identification of visual displays of speech: Comparisons of naive and trained observers

Two groups of subjects were presented with spectrograms of 50 words they had never seen before and were asked to provide a single monosyllabic English word for each spectrogram. One group had learned to identify a limited set of speech spectrograms after 15 h of training using a study‐test procedure which stressed wholistic word identification. A subgroup of participants in a formal course on spectrogram reading at MIT served as a second group of subjects. These subjects learned specific acoustic and phonetic principles and strategies for interpreting spectrograms. Subjects in the first group correctly identified 33% of the possible phonetic segments in spectrograms they had never seen before. The second group of subjects correctly identified 40% of the possible segments in the same set of spectrograms given to the first group. When the data were scored for correct manner class of consonants only, the two groups did not differ significantly. Detailed descriptions of the identification results will be presented. Implications of these findings for developing visual aids for hearing impaired persons and improved phonetic recognition strategies will be discussed. [Supported by NSF.]