Human Speech Recognition Research Articles

Using self-organizing maps to recognize acoustic units associated with information content in animal vocalizations

Kohonen self-organizing neural networks, also called self-organizing maps (SOMs), have been used successfully to recognize human phonemes and in this way to aid in human speech recognition. This paper describes how SOMS also can be used to associate specific information content with animal vocalizations. A SOM was used to identify acoustic units in Gunnison's prairie dog alarm calls that were vocalized in the presence of three different predator species. Some of these acoustic units and their combinations were found exclusively in the alarm calls associated with a particular predator species and were used to associate predator species information with individual alarm calls. This methodology allowed individual alarm calls to be classified by predator species with an average of 91% accuracy. Furthermore, the topological structure of the SOM used in these experiments provided additional insights about the acoustic units and their combinations that were used to classify the target alarm calls. An important benefit of the methodology developed in this paper is that it could be used to search for groups of sounds associated with information content for any animal whose vocalizations are composed of multiple simultaneous frequency components.

Acoustic detection and classification of microchiroptera using machine learning: Lessons learned from automatic speech recognition

Current automatic acoustic detection and classification of microchiroptera utilize global features of individual calls (i.e., duration, bandwidth, frequency extrema), an approach that stems from expert knowledge of call sonograms. This approach parallels the acoustic phonetic paradigm of human automatic speech recognition (ASR), which relied on expert knowledge to account for variations in canonical linguistic units. ASR research eventually shifted from acoustic phonetics to machine learning, primarily because of the superior ability of machine learning to account for signal variation. To compare machine learning with conventional methods of detection and classification, nearly 3000 search-phase calls were hand labeled from recordings of five species: Pipistrellus bodenheimeri, Molossus molossus, Lasiurus borealis, L. cinereus semotus, and Tadarida brasiliensis. The hand labels were used to train two machine learning models: a Gaussian mixture model (GMM) for detection and classification and a hidden Markov model (HMM) for classification. The GMM detector produced 4% error compared to 32% error for a baseline broadband energy detector, while the GMM and HMM classifiers produced errors of 0.6 +/- 0.2% compared to 16.9 +/- 1.1% error for a baseline discriminant function analysis classifier. The experiments showed that machine learning algorithms produced errors an order of magnitude smaller than those for conventional methods.

Further intelligibility results from human listening tests using the short-time phase spectrum

State-of-the-art automatic speech recognition systems (ASRs) use only the short-time magnitude spectrum for feature extraction; the short-time phase spectrum is generally ignored in these systems. Results from our recent human listening tests indicate that the short-time phase spectrum can significantly contribute to speech intelligibility over small window durations (i.e., 20–40 ms). This is an interesting result, indicating the possible usefulness of the short-time phase spectrum for ASR, which commonly employs small window durations of 20–40 ms for spectral analysis. In this paper, we continue our investigation of the short-time phase spectrum. We explore the use of partial short-time phase spectrum information, in the absence of all the short-time magnitude spectrum information, for intelligible signal reconstruction. We create two types of stimuli; one in which its frequency-derivative (i.e., group delay function, GDF) is preserved and another in which its time-derivative (i.e., instantaneous frequency distribution, IFD) is preserved. We do this to determine the contribution that each of these derivatives provides toward intelligibility. Reconstructing stimuli from knowledge of only the GDF or only the IFD results in poor intelligibility. However, when we create stimuli using knowledge of both the GDF and the IFD, reasonable intelligibility is obtained. In light of these results, we conclude that both the GDF and IFD components of the short-time phase spectrum are needed to reconstruct an intelligible signal. In addition, we also perform some experiments to quantify the intelligibility of stimuli reconstructed from the short-time phase and magnitude spectra of noisy speech. The intelligibility of stimuli constructed from either the short-time magnitude spectrum or the short-time phase spectrum degrades at a similar rate under increasing noise levels. The intelligibility of the original signals under noisy conditions also degrades with increased noise, but in all cases the intelligibility is superior to that provided by the stimuli constructed from the separate short-time components. Therefore, we argue that knowledge of both short-time magnitude and phase spectrum information results in superior human speech recognition performance.

How Should a Speech Recognizer Work?

Although researchers studying human speech recognition (HSR) and automatic speech recognition (ASR) share a common interest in how information processing systems (human or machine) recognize spoken language, there is little communication between the two disciplines. We suggest that this lack of communication follows largely from the fact that research in these related fields has focused on the mechanics of how speech can be recognized. In Marr's (1982) terms, emphasis has been on the algorithmic and implementational levels rather than on the computational level. In this article, we provide a computational-level analysis of the task of speech recognition, which reveals the close parallels between research concerned with HSR and ASR. We illustrate this relation by presenting a new computational model of human spoken-word recognition, built using techniques from the field of ASR that, in contrast to current existing models of HSR, recognizes words from real speech input.

Automatic detection of microchiroptera echolocation calls from field recordings using machine learning algorithms

The authors have recently presented experimental results of applying machine learning algorithms, used extensively in human automatic speech recognition research (ASR), to automatic species identification of echolocating bats [Skowronski and Harris, J. Acoust. Soc. Am. 116, 2639 (2004)]. The results of those experiments demonstrated that frame-based classification, preferred in ASR, out-performs holistic classification typically employed in automatic echolocating species identification. The authors have extended the paradigm of machine learning algorithms to the related problem of bat call detection. A robust automatic bat call detection algorithm, to replace hand labeling, is required for two reasons: (1) for real-time species identification in the field, and (2) because hand labeling is subjective, tedious, slow, and error-prone. The current experiments compare various frame-based features (log energy, pitch estimates, pitch slopes) with several models of detection (matched filters, Gaussian mixtures, decision trees). Detector sensitivity and specificity are quantified for comparison using hand-labeled calls, with considerations of classification requirements for detected calls. That is, a detector is not penalized for including a short segment of background signal before and after a hand-labeled call. The results demonstrate the superior performance of the frame-based features and machine learning detection algorithms compared to conventional features and detection algorithms.

Automatic classification and speaker identification of African elephant (Loxodonta africana) vocalizations

A hidden Markov model (HMM) system is presented for automatically classifying African elephant vocalizations. The development of the system is motivated by successful models from human speech analysis and recognition. Classification features include frequency-shifted Mel-frequency cepstral coefficients (MFCCs) and log energy, spectrally motivated features which are commonly used in human speech processing. Experiments, including vocalization type classification and speaker identification, are performed on vocalizations collected from captive elephants in a naturalistic environment. The system classified vocalizations with accuracies of 94.3% and 82.5% for type classification and speaker identification classification experiments, respectively. Classification accuracy, statistical significance tests on the model parameters, and qualitative analysis support the effectiveness and robustness of this approach for vocalization analysis in nonhuman species.

Articulation and Intelligibility

Immediately following the Second World War, between 1947 and 1955, several classic papers quantified the fundamentals of human speech information processing and recognition. In 1947 French and Steinberg published their classic study on the articulation index. In 1948 Claude Shannon published his famous work on the theory of information. In 1950 Fletcher and Galt published their theory of the articulation index, a theory that Fletcher had worked on for 30 years, which integrated his classic works on loudness and speech perception with models of speech intelligibility. In 1951 George Miller then wrote the first book Language and Communication, analyzing human speech communication with Claude Shannon's just published theory of information. Finally in 1955 George Miller published the first extensive analysis of phone decoding, in the form of confusion matrices, as a function of the speech-to-noise ratio. This work extended the Bell Labs' speech articulation studies with ideas from Shannon's Information theory. Both Miller and Fletcher showed that speech, as a code, is incredibly robust to mangling distortions of filtering and noise.

Statistical automatic species identification of microchiroptera from echolocation calls: Lessons learned from human automatic speech recognition

Current automatic species identification of microchiroptera through echolocation search calls utilizes holistic features, such as call duration, bandwidth, and frequency extrema, which stem from expert knowledge of sonograms of individual calls. This approach parallels the early acoustic-phonetic paradigm of human automatic speech recognition (ASR), which relied heavily on expert knowledge to account for variations in canonical linguistic units of interest. Acoustic-phonetic ASR gave way to the statistical paradigm of ASR primarily because of the superior ability of machine learning to account for variations in the acoustic signal (noise, speaker characteristics) [Juang and Furui, Proc. IEEE 88 (8), 1142–1165 (2000)]. In the current work, machine learning methods from human ASR, hidden Markov models (HMM) and dynamic time warping (DTW), are considered for the problem of automatic species identification of microchiroptera from echolocation calls. The extensive history of ASR provides valuable lessons, which are highlighted and applied to species recognition from field-recorded acoustic data. Frame-based features of the acoustic signal (fundamental frequency, log energy, temporal derivatives) are incorporated into HMM and DTW classifiers, and experimental results indicate the superior approach of statistical automatic species identification compared to current techniques.

Fifty years of progress in speech recognition

Human level speech recognition has proved to be an elusive goal because of the many sources of variability that affect speech: from stationary and dynamic noise, microphone variability, and speaker variability to variability at phonetic, prosodic, and grammatical levels. Over the past 50 years, Jim Flanagan has been a continuous source of encouragement and inspiration to the speech recognition community. While early isolated word systems primarily used acoustic knowledge, systems in the 1970s found mechanisms to represent and utilize syntactic (e.g., information retrieval) and semantic knowledge (e.g., Chess) in speech recognition systems. As vocabularies became larger, leading to greater ambiguity and perplexity, we had to explore the use task specific and context specific knowledge to reduce the branching factors. As the need arose for systems that can be used by open populations using telephone quality speech, we developed learning techniques that use very large data sets and noise adaptation methods. We still have a long way to go before we can satisfactorily handle unrehearsed spontaneous speech, speech from non-native speakers, and dynamic learning of new words, phrases, and grammatical forms.

Grouping Miller–Nicely by linear vector space rotations

Human speech recognition has been studied using response to CV speech stimuli. Miller and Nicely (1955) studied such data in the form of confusion matrices to obtain insight into the psychological structure of the phone in noise. Here, the confusion matrices are modeled as phone coordinates in a high dimensional perceptual vector space. The model generalizes to an eigenvalue decomposition (EVD) [Allen (2004)]. This is followed by agglomerative hierarchical clustering of the transformed data, and an automated process is used to identify the main clusters. The resulting EVD clustering is very similar to other Miller–Nicely groupings, based on both production and MDS derived features, but is more model based. It was found that there is a gradual and highly consistent change in the clustering of sounds, independent of cluster size and configuration. By examining the change in similarity between various speech sounds, it is hoped that perceptual features may be uniquely identified.

A Holistic Approach to the Introduction of Automatic Speech Recognition Technology in Ground Combat Vehicles

Recent advances in speech recognition technology prompt consideration of using machine recognition of human speech in a wider range of applications. Communication using speech recognition has a number of benefits. One area where this technology has only recently been considered is in military armored fighting vehicles during combat. A holistic approach to considering speech recognition in this form of transport is reported here by viewing this technology within the context of the cockpit resource management paradigm. This facilitated development of a function-based crew communication model. It is within this framework that the use of speech recognition in armored vehicles is evaluated. The use of speech technology led to a reduction in the total amount of communication taking place in the vehicle.

Auditory-evoked magnetic field codes place of articulation in timing and topography around 100 milliseconds post syllable onset

This study demonstrates by means of magnetic source imaging how consonants and vowels that constitute a syllable differently affect the neural processing within the auditory cortex. We recently identified a topographically separate processing for mutually exclusive place features in isolated vowels (Obleser et al., in press). Does this mapping principle also hold for stop consonants with differing places of articulation? How is the N100m response to consonant–vowel (CV) syllables affected by the congruency of place information in the consonant and the vowel? Moreover, how is the N100m affected by coarticulation, i.e., the spreading of place features to adjacent phonemes? By systematically varying phonological information in the consonant as well as in the vowel of CV syllables, we were able to reveal a difference in N100m syllable source location along the anterior–posterior axis due to mutually exclusive places of articulation in the vowel of the syllable. We also found a change in source orientation rather than source location due to the same mutually exclusive features in the onset of the syllable. Furthermore, the N100m time course of the brain response delivered important complementary information to identify the phonological features present in the speech signal. Responses to all syllable categories originated in the perisylvian region anterior to the source of a band-passed noise stimulus. The systematic variation of both consonantal and vocalic place features and the study of their interaction on auditory processing proves to be a valuable method to gain more insight into the elusive phenomenon of human speech recognition.

Voice recognition.

Voice recognition represents one of the new technologies that are changing the practice of radiology. Thirty percent of radiology practices are either currently or plan to have voice recognition (VR) systems. VR software encompasses 4 core processes: spoken recognition of human speech, synthesis of human readable characters into speech, speaker identification and verification, and comprehension. Many software packages are available offering VR. All these packages should contain an interface with the radiology information system. The benefits include decreased turnaround time and cost savings. Its advantages include the transfer of secretarial duties to the radiologist with a result in decreased productivity.

Human factor cepstral coefficients

Automatic speech recognition (ASR) is an emerging field with the goal of creating a more natural man/machine interface. The single largest obstacle to widespread use of ASR technology is robustness to noise. Since human speech recognition greatly outperforms current ASR systems in noisy environments, ASR systems seek to improve noise robustness by drawing on biological inspiration. Most ASR front ends employ mel frequency cepstral coefficients (mfcc) which is a filter bank-based algorithm whose filters are spaced on a linear-log frequency scale. Although center frequency is based on a perceptually motivated frequency scale, filter bandwidth is set by filter spacing and not through biological motivation. The coupling of filter bandwidth to other filter bank parameters (frequency range, number of filters) has led to variations of the original algorithm with different filter bandwidths. In this work, a novel extension to mfcc is introduced which decouples filter bandwidth from the rest of the filter bank parameters by employing the relationship between filter center frequency and critical bandwidth of the human auditory system. The new algorithm, called human factor cepstral coefficients (hfcc), is shown to outperform the original mfcc and two popular variations in several ASR experiments and noise sources.

Recognition of word and nonsense syllables in noise

In the recognition of human speech, listeners use sensory information from the speech signal as well as signal-independent information, such that acoustic-phonetic salience, lexical status, frequency of usage, and neighborhood density interact in speech recognition. This paper presents detailed results from a replication of a study of context effects reported by Boothroyd and Nittrouer [J. Acoust. Soc. Am. 84, 101–108 (1988)]. 240 phonetically matched word and nonsense CVC syllables at different SNRs were presented to 32 listeners for identification. The results are consistent with the original study, with greater accuracy for words than nonsense items as quantified by a j-factor analysis according to lexical status. Online response collection enabled detailed analyses not reported in the original study. Values for the j-factor according to usage frequency and phonetic confusability are presented. Confusion matrices of the phonemes are presented, and are largely consistent with previous studies for the initial consonants and vowels. Accuracy in the final consonants of nonsense syllables is substantially lower than the corresponding phonemes of words or than the initial consonants of nonsense syllables. Final nasals were particularly difficult to identify.

Detecting stop consonants in continuous speech.

The problem of implementing a detector for stop consonants in continuously spoken speech is considered. The problem is posed as one of finding an optimal filter (linear or nonlinear) that operates on a particular appropriately chosen representation, and ideally outputs a 1 when a stop occurs and 0 otherwise. The performance of several variants of a canonical stop detector is discussed and its implications for human and machine speech recognition is considered.

Feature extraction in human speech recognition

In 1955 George Miller and Patricia Nicely explored the human speech code and discovered that a five-channel feature code is used to extract phonemes from speech. Their mutual information analysis of the confusion matrices of filtered noisy speech showed that the five channels were virtually independent of each other, accounting for the robustness of speech communication quantified by Miller, Heise, and Lichten in 1951. While the 1955 analysis was a major step forward, it did not lead to practical results, since there are no known methods for extracting these features from the speech. This weakness is the basis of the present research program, to define the signal processing required to robustly extract features from narrow bands of speech. Our emphasis is to work in narrow bands, starting from critical bands. In this paper we discuss the first step of this process, which is to measure (a) the jnd for the detection of speech in noise and (b) the jnd for residue pitch given narrow bands of speech in noise. We have been investigating these features with the ultimate goal of trying to build a speech recognition machine based on distinctive feature signal processing.

Analysis-by-synthesis and acoustic landmarks

In the 1950s and 1960s Ken Stevens and Morris Halle proposed a model for human speech recognition that incorporated analysis-by-synthesis. In the more recent past the notion of acoustic landmarks has arisen from the speech group at MIT as part of a model for human lexical access from features by human listeners. Here, it is proposed that these ideas are fruitful ones for incorporation into a method (constraint and refinement) for training a machine with its own vocal tract to imitate human speech, while learning the relation between articulation and acoustic output. The machine would be trained in stages starting with the attempts to imitate the sequence of relatively open articulations and closed articulations. Landmarks, based on acoustic criteria, would be placed near the peaks in the amplitude envelope, allowing the machine to constrain the type of articulatory activations. This constraining step would involve, initially, an overly restrictive set of phonotatic rules, such as all sequences are obstruent-vowel sequences. The refining step would be the analysis-by-synthesis with the machine’s vocal tract. As the stages progress, the machine would attend to more acoustic properties of the speech signal with different kinds of landmarks and increase the generality of the phonotatic rules.

Time-interval information in the auditory representation of speech sounds

In the auditory system, the primary fibers that encode mechanical motion of the basilar partition are phase-locked to that motion, and this information is preserved, to varying degrees, up to the inferior colliculus. It is known that this timing-interval information is used in localization, and it is probably also used to separate sources from diffuse background noise. The time intervals are on the order of milliseconds, and so traditional speech preprocessors (like MFCC systems) with frames on the order of 15 ms, remove the time-interval information from the representation. The performance of these systems deteriorates badly when the speaker is in a noisy environment. This suggests that time-interval processing will eventually need to be integrated into speech recognition systems if they are to achieve the kind of noise resistance characteristic of human speech recognition. An auditory image model (AIM) will be presented that is designed to stabilize repeating time-interval patterns like those produced by voiced speech, and results from experiments where AIM has been used as a preprocessor for automatic speech recognition. [Work supported by UK MRC (G9703469).]

Adaptive fusion of acoustic and visual sources for automatic speech recognition

Among the various methods proposed to improve the accuracy and the robustness of automatic speech recognition (ASR), the use of additional knowledge sources is a successful one. In particular, a recent method proposes supplementing the acoustic information with visual data mostly derived from the speaker's lip shape. Perceptual studies support this approach by emphasising the importance of visual information for speech recognition in humans. This paper describes a method we have developed for adaptive integration of acoustic and visual information in ASR. Each modality is involved in the recognition process with a different weight, which is dynamically adapted during this process mainly according to the signal-to-noise ratio provided as a contextual input. We tested this method on continuous hidden Markov model-based systems developed according to direct identification (DI), separate identification (SI) and hybrid identification (DI + SI) strategies. Experiments performed under various noise-level conditions show that the DI + SI based system is the most promising one when compared to both DI and SI-based systems for a speaker-dependent continuous-spelling of French letters recognition task. They also confirm that using adaptive modality weights instead of fixed weights allows for performance improvement and that weight estimation could benefit from using visemes as decision units for the visual recogniser in SI and DI + SI based systems.