Telephone Speech Research Articles

Getting closer: tailored human–computer speech dialog

This paper presents an advanced call center, which adapts presentation and interaction strategy to properties of the caller such as age, gender, and emotional state. User studies on interactive voice response (IVR) systems have shown that these properties can be used effectively to “tailor” services to users or user groups who do not maintain personal preferences, e.g., because they do not use the service on a regular basis. The adopted approach to achieve individualization of services, without being able to personalize them, is based on the analysis of a caller’s voice. This paper shows how this approach benefits service providers by being able to target entertainment and recommendation options. It also shows how this analysis at the same time benefits the customer, as it can increase accessibility of IVR systems to user segments which have particular expectations or which do not cope well with a “one size fits all” system. The paper summarizes the authors’ current work on component technologies, such as emotion detection, age and gender recognition on telephony speech, and presents results of usability and acceptability tests as well as an architecture to integrate these technologies in future multi-modal contact centers. It is envisioned that these will eventually serve customers with an avatar representation of an agent and tailored interaction strategies, matching powerful output capabilities with advanced analysis of the user’s input.

A spectral/temporal method for robust fundamental frequency tracking

In this paper, a fundamental frequency (F(0)) tracking algorithm is presented that is extremely robust for both high quality and telephone speech, at signal to noise ratios ranging from clean speech to very noisy speech. The algorithm is named "YAAPT," for "yet another algorithm for pitch tracking." The algorithm is based on a combination of time domain processing, using the normalized cross correlation, and frequency domain processing. Major steps include processing of the original acoustic signal and a nonlinearly processed version of the signal, the use of a new method for computing a modified autocorrelation function that incorporates information from multiple spectral harmonic peaks, peak picking to select multiple F(0) candidates and associated figures of merit, and extensive use of dynamic programming to find the "best" track among the multiple F(0) candidates. The algorithm was evaluated by using three databases and compared to three other published F(0) tracking algorithms by using both high quality and telephone speech for various noise conditions. For clean speech, the error rates obtained are comparable to those obtained with the best results reported for any other algorithm; for noisy telephone speech, the error rates obtained are lower than those obtained with other methods.

Acoustic-phonetic features for stop consonant place detection in clean and telephone speech

This work classifies voiceless stop consonant place in CV tokens of English using burst release cues for clean (TIMIT) and telephone speech (NTIMIT). We compared the performance of cepstral coefficients to acoustic phonetics-motivated features such as center of gravity, burst amplitude and relative difference of formant amplitudes. In clean speech, cepstral coefficients resulted in better classification. However, for test data from NTIMIT, acoustic phonetic-based features outperformed cepstral coefficients, particularly if models were trained on clean speech. In addition, augmenting cepstral coefficients with acoustic phonetic-based measurements resulted in the best performance. These findings suggest that cepstral coefficients are able to model speech in a given environment in finer detail, whereas acoustic phonetic-based features are more robust to changes in environment, so that combining both types of measurements leads to the best performance.

Large-margin minimum classification error training: A theoretical risk minimization perspective

Large-margin discriminative training of hidden Markov models has received significant attention recently. A natural and interesting question is whether the existing discriminative training algorithms can be extended directly to embed the concept of margin. In this paper, we give this question an affirmative answer by showing that the sigmoid bias in the conventional minimum classification error (MCE) training can be interpreted as a soft margin. We justify this claim from a theoretical classification risk minimization perspective where the loss function associated with a non-zero sigmoid bias is shown to include not only empirical error rates but also a margin-bound risk. Based on this perspective, we propose a practical optimization strategy that adjusts the margin (sigmoid bias) incrementally in the MCE training process so that a desirable balance between the empirical error rates on the training set and the margin can be achieved. We call this modified MCE training process large-margin minimum classification error (LM-MCE) training to differentiate it from the conventional MCE. Speech recognition experiments have been carried out on two tasks. First, in the TIDIGITS recognition task, LM-MCE outperforms the state-of-the-art MCE method with 17% relative digit-error reduction and 19% relative string-error reduction. Second, on the Microsoft internal large vocabulary telephony speech recognition task (with 2000 h of training data and 120 K words in the vocabulary), significant recognition accuracy improvement is achieved, demonstrating that our formulation of LM-MCE can be successfully scaled up and applied to large-scale speech recognition tasks.

Combining Spectral Representations for Large-Vocabulary Continuous Speech Recognition

In this paper, we investigate the combination of complementary acoustic feature streams in large-vocabulary continuous speech recognition (LVCSR). We have explored the use of acoustic features obtained using a pitch-synchronous analysis, Straight, in combination with conventional features such as Mel frequency cepstral coefficients. Pitch-synchronous acoustic features are of particular interest when used with vocal tract length normalization (VTLN) which is known to be affected by the fundamental frequency. We have combined these spectral representations directly at the acoustic feature level using heteroscedastic linear discriminant analysis (HLDA) and at the system level using ROVER. We evaluated this approach on three LVCSR tasks: dictated newspaper text (WSJCAM0), conversational telephone speech (CTS), and multiparty meeting transcription. The CTS and meeting transcription experiments were both evaluated using standard NIST test sets and evaluation protocols. Our results indicate that combining conventional and pitch-synchronous acoustic feature sets using HLDA results in a consistent, significant decrease in word error rate across all three tasks. Combining at the system level using ROVER resulted in a further significant decrease in word error rate.

Maximizing cochlear implant patients’ performance with advanced speech training procedures

Advances in implant technology and speech processing have provided great benefit to many cochlear implant patients. However, some patients receive little benefit from the latest technology, even after many years’ experience with the device. Moreover, even the best cochlear implant performers have great difficulty understanding speech in background noise, and music perception and appreciation remain major challenges. Recent studies have shown that targeted auditory training can significantly improve cochlear implant patients’ speech recognition performance. Such benefits are not only observed in poorly performing patients, but also in good performers under difficult listening conditions (e.g., speech noise, telephone speech, music, etc.). Targeted auditory training has also been shown to enhance performance gains provided by new implant devices and/or speech processing strategies. These studies suggest that cochlear implantation alone may not fully meet the needs of many patients, and that additional auditory rehabilitation may be needed to maximize the benefits of the implant device. Continuing research will aid in the development of efficient and effective training protocols and materials, thereby minimizing the costs (in terms of time, effort and resources) associated with auditory rehabilitation while maximizing the benefits of cochlear implantation for all recipients.

Unsupervised intra-speaker variability compensation based on Gestalt and model adaptation in speaker verification with telephone speech

In this paper, an unsupervised intra-speaker variability compensation (ISVC) method based on Gestalt is proposed to address the problem of limited enrolling data and noise robustness in text-dependent speaker verification (SV). Experiments with two databases show that: ISVC can lead to reductions in EER as high as 20% or 40% and ISCV provides reductions in the integral below the ROC curve between 30% and 60%. Also, the observed improvements are independent of the number of enrolling utterances. In contrast to model adaptation methods, ISVC is memoryless with respect to previous verification attempts. As shown here, unsupervised model adaptation can lead to substantial improvements in EER but is highly dependent on the sequence of client/impostor verification events. In adverse scenarios, such as massive impostor attacks and verification from alternated telephone line, unsupervised model adaptation might even provide reductions in verification accuracy when compared with the baseline system. In those cases, ISVC can even outperform adaptation schemes. It is worth emphasizing that ISVC and unsupervised model adaptation are compatible and the combination of both methods always improves the performance of model adaptation. The combination of both schemes can lead to improvements in EER as high as 34%. Due to the restrictions of commercially available databases for text-dependent SV research, the results presented here are based on local databases in Spanish. By doing so, the visibility of research in Iberian Languages is highlighted.

An Interactive Open-Vocabulary Chinese Name Input System Using Syllable Spelling and Character Description Recognition Modules for Error Correction

The open-vocabulary name recognition technique is one of the most challenging tasks in the application of automatic Chinese speech recognition technology. It can be used as the free name input method for telephony speech applications and automatic directory assistance systems. A Chinese name usually has two to three characters, each of which is pronounced as a single tonal syllable. Obviously, it is very confusing to recognize a three-syllable word from millions to billions of possible candidates. A novel interactive automatic-speech-recognition system is proposed to resolve this highly challenging task. This system was built as an open-vocabulary Chinese name recognition system using character-based approaches. Two important character-input speech-recognition modules were designed as backoff approaches in this system to complete the name input or to correct any misrecognized characters. Finite-state networks were compiled from regular grammar of syllable spellings and character descriptions for these two speech recognition modules. The possible candidate names cover more than five billions. This system has been tested publicly and proved a robust way to interact with the speaker. An 86.7% name recognition success rate was achieved by the interactive open-vocabulary Chinese name input system.

MPE-based discriminative linear transforms for speaker adaptation

In this paper, the use of discriminative linear transforms (DLT) is investigated to construct speaker adaptive speech recognition systems, where a discriminative criterion rather than ML is used for transform parameter estimation. The minimum phone error (MPE) criterion is investigated for DLT estimation, by making use of a so-called weak-sense auxiliary function to derive the estimation formulae. An implementation based on lattices is used for DLT statistics accumulation, where the use of a weakened language model allows more confusion data to be included. To improve DLT estimation for unsupervised adaptation, a method of incorporating word correctness information of the supervision into transform estimation is developed. The confidence scores calculated by confusion network decoding are used to represent the word correctness and weight the numerator statistics during DLT estimation. This makes the DLT estimation less sensitive to errors in the supervision. Experiments on transcription of read newspaper data and on conversational telephone speech transcription have shown the improvements of DLT over MLLR for both supervised and unsupervised adaptation, and the effectiveness of confidence scores for improving both normal and DLT-based MLLR adaptation.

A Joint Factor Analysis Approach to Progressive Model Adaptation in Text-Independent Speaker Verification

This paper addresses the issue of speaker variability and session variability in text-independent Gaussian mixture model (GMM)-based speaker verification. A speaker model adaptation procedure is proposed which is based on a joint factor analysis approach to speaker verification. It is shown in this paper that this approach facilitates the implementation of a progressive unsupervised adaptation strategy which is able to produce an improved model of speaker identity while minimizing the influence of channel variability. The paper also deals with the interaction between this model adaptation approach and score normalization strategies which act to reduce the variation in likelihood ratio scores. This issue is particularly important in establishing decision thresholds in practical speaker verification systems since the variability of likelihood ratio scores can increase as a result of progressive model adaptation. These adaptation methods have been evaluated under the adaptation paradigm defined under the NIST 2005 Speaker Recognition Evaluation Plan, which is based on conversation sides derived from telephone speech utterances. It was found that when target speaker models were trained from a single conversation, an equal error rate (EER) of 4.5% was obtained under the NIST unsupervised speaker adaptation scenario.

Mapping Speech Spectra from Throat Microphone to Close-Speaking Microphone: A Neural Network Approach

Speech recorded from a throat microphone is robust to the surrounding noise, but sounds unnatural unlike the speech recorded from a close-speaking microphone. This paper addresses the issue of improving the perceptual quality of the throat microphone speech by mapping the speech spectra from the throat microphone to the close-speaking microphone. A neural network model is used to capture the speaker-dependent functional relationship between the feature vectors (cepstral coefficients) of the two speech signals. A method is proposed to ensure the stability of the all-pole synthesis filter. Objective evaluations indicate the effectiveness of the proposed mapping scheme. The advantage of this method is that the model gives a smooth estimate of the spectra of the close-speaking microphone speech. No distortions are perceived in the reconstructed speech. This mapping technique is also used for bandwidth extension of telephone speech.

Wideband Speech Coding Advances in VMR-WB Standard

This paper presents novel techniques for source-controlled variable-rate wideband speech coding. These techniques have been used in the variable-rate multimode wideband (VMR-WB) speech codec recently selected by the Third-Generation Partnership Project 2 (3GPP2) for wideband (WB) speech telephony, streaming, and multimedia messaging services in the cdma2000 third-generation wireless system. The codec utilizes efficient coding modes optimized for different classes of speech signal including generic coding based on AMR-WB for transients and onsets, voiced coding optimized for stable voiced signals, unvoiced coding optimized for unvoiced segments, and comfort noise generation for inactive segments. Several innovations enable very good performance at average bit rates below 8 kb/s for active speech coding. The article presents an overview of the codec and describes in detail some of the codec novel features: Robust pitch tracking algorithm, coding-mode dependent prediction of linear prediction (LP) filter quantization, and novel frame erasure concealment techniques including supplementary information for reconstruction of lost onsets and improving decoder convergence. Selected results from the Selection and Characterization tests of the codec illustrate its performance

Telephone Speech Perception by Mandarin-Speaking Cochlear Implantees

To evaluate Mandarin-speaking cochlear implant patients' understanding of telephone speech. Telephone speech was simulated by band-limiting broadband speech stimuli (300-3200 Hz) and adding Gaussian noise (35 dB signal-to-noise ratio). Recognition of multitalker vowels, consonants, voice gender, and Chinese tones was measured for both simulated telephone speech and broadband speech in fifteen Mandarin-speaking cochlear implant patients. Results showed no significant difference in Chinese tone recognition scores between broadband and telephone speech. However, mean recognition scores for vowels, consonants and voice gender were significantly lower with telephone speech. The effect of the limited telephone bandwidth on speech recognition was highly variable among subjects. Some subjects were more sensitive to high-frequency speech cues, resulting in a significant drop in performance with band-limited telephone speech, while other subjects were less sensitive to high-frequency speech cues, resulting in similar performance between broadband and band-limited telephone speech. These results suggest that the limited bandwidth negatively affects cochlear implant patients' understanding of telephone speech. Because the effect of band-limited speech was highly variable among subjects, the results also suggest that the contribution of high frequency information to speech recognition may vary significantly among cochlear implant patients. For patients who receive little benefit from high-frequency speech cues, speech processor adjustments may be necessary to access the additional cues provided in broadband speech.

Telephony Speech Enhancement by Data Hiding

The current public switched telephone network (PSTN) is only able to deliver analog signals in a relatively narrow frequency band, about 200-3500 Hz. Such a limited bandwidth causes the typical sound of the narrowband telephone speech. In order to improve intelligibility and perceived quality of telephone speech, we propose using data hiding to extend the PSTN channel bandwidth. Based on the perceptual masking principle, the inaudible spectrum components within the telephone bandwidth can be removed without degrading the speech quality, providing a hidden channel to transmit extra information. The audible components outside the PSTN bandwidth, which are spread out by using orthogonal pseudo-noise codes, are embedded into this hidden channel and then transmitted through the PSTN channel. While this hidden signal is not audible to the human ear, it can be extracted at the receiver end. It results in a final speech signal with a wider bandwidth than the normal PSTN channel. Using both theoretical and simulation analysis, it is shown that the proposed approach is robust to quantization errors and channel noises. Although we cannot physically extend the transmission bandwidth of PSTN, the telephony speech quality can be significantly improved by using the proposed data hiding technique

Training Wideband Acoustic Models Using Mixed-Bandwidth Training Data for Speech Recognition

One serious difficulty in the deployment of wideband speech recognition systems for new tasks is the expense in both time and cost of obtaining sufficient training data. A more economical approach is to collect telephone speech and then restrict the application to operate at the telephone bandwidth. However, this generally results in suboptimal performance compared to a wideband recognition system. In this paper, we propose a novel expectation-maximization (EM) algorithm in which wideband acoustic models are trained using a small amount of wideband speech and a larger amount of narrowband speech. We show how this algorithm can be incorporated into the existing training schemes of hidden Markov model (HMM) speech recognizers. Experiments performed using wideband speech and telephone speech demonstrate that the proposed mixed-bandwidth training algorithm results in significant improvements in recognition accuracy over conventional training strategies when the amount of wideband data is limited

Recognition of Simulated Telephone Speech by Cochlear Implant Users

To evaluate cochlear implant users' understanding of telephone speech. Telephone speech was simulated by band-limiting broadband speech stimuli. Multitalker vowel, consonant, and sentence recognition was measured for both simulated telephone speech and broadband speech in 10 postlingually deafened adult cochlear implant users. The study was approved by the St. Vincent's Hospital institutional review board, and signed, informed consent was obtained from all participants. There was no significant difference in vowel recognition scores between broadband and telephone speech. However, mean consonant and sentence recognition scores were significantly poorer with telephone speech. The limited telephone bandwidth significantly reduced cochlear implant users' understanding of telephone speech. The effect of band-limited speech was highly variable, suggesting that the contribution of high-frequency information to speech recognition varied significantly among the cochlear implant users.

Bandwidth Extension of Telephone Speech Aided by Data Embedding

A system for bandwidth extension of telephone speech, aided by data embedding, is presented. The proposed system uses the transmitted analog narrowband speech signal as a carrier of the side information needed to carry out the bandwidth extension. The upper band of the wideband speech is reconstructed at the receiving end from two components: a synthetic wideband excitation signal, generated from the narrowband telephone speech and a wideband spectral envelope, parametrically represented and transmitted as embedded data in the telephone speech. We propose a novel data embedding scheme, in which the scalar Costa scheme is combined with an auditory masking model allowing high rate transparent embedding, while maintaining a low bit error rate. The signal is transformed to the frequency domain via the discrete Hartley transform (DHT) and is partitioned into subbands. Data is embedded in an adaptively chosen subset of subbands by modifying the DHT coefficients. In our simulations, high quality wideband speech was obtained from speech transmitted over a telephone line (characterized by spectral magnitude distortion, dispersion, and noise), in which side information data is transparently embedded at the rate of 600 information bits/second and with a bit error rate of approximately . In a listening test, the reconstructed wideband speech was preferred (at different degrees) over conventional telephone speech in of the test utterances.

Language and gender differences in speech overlaps in conversation

Language and gender differences in speech overlaps in conversation were investigated, using the LDC CallHome telephone speech corpora of six languages: Arabic, English, German, Japanese, Mandarin, and Spanish. To automatically obtain the speech overlaps between two sides in a conversation, each side was segmented into pause and speaking segments, and the overlap segments during which both sides were speaking were time stamped. Two types of speech overlaps are distinguished: (1) One side takes over the turn before the other side finishes (turn-taking type). (2) One side speaks in the middle of the other side’s turn (backchannel type). It was found that Japanese conversations have more short (less than 500 ms) turn-taking type of overlap segments than the other languages. The average number of such segments per 10 min of conversation for Japanese was 62.6 whereas the average numbers for the other languages ranged from 37.9 to 43.3. There were, however, no significant differences among the languages on the backchannel type of overlaps. Cross-linguistically, conversations between two females contained more speech overlaps (both types) than those between a male and a female or between two males, and there was no significant difference between the latter two.

Enriching speech recognition with automatic detection of sentence boundaries and disfluencies

Effective human and automatic processing of speech requires recovery of more than just the words. It also involves recovering phenomena such as sentence boundaries, filler words, and disfluencies, referred to as structural metadata. We describe a metadata detection system that combines information from different types of textual knowledge sources with information from a prosodic classifier. We investigate maximum entropy and conditional random field models, as well as the predominant hidden Markov model (HMM) approach, and find that discriminative models generally outperform generative models. We report system performance on both broadcast news and conversational telephone speech tasks, illustrating significant performance differences across tasks and as a function of recognizer performance. The results represent the state of the art, as assessed in the NIST RT-04F evaluation

Advances in transcription of broadcast news and conversational telephone speech within the combined EARS BBN/LIMSI system

This paper describes the progress made in the transcription of broadcast news (BN) and conversational telephone speech (CTS) within the combined BBN/LIMSI system from May 2002 to September 2004. During that period, BBN and LIMSI collaborated in an effort to produce significant reductions in the word error rate (WER), as directed by the aggressive goals of the Effective, Affordable, Reusable, Speech-to-text [Defense Advanced Research Projects Agency (DARPA) EARS] program. The paper focuses on general modeling techniques that led to recognition accuracy improvements, as well as engineering approaches that enabled efficient use of large amounts of training data and fast decoding architectures. Special attention is given on efforts to integrate components of the BBN and LIMSI systems, discussing the tradeoff between speed and accuracy for various system combination strategies. Results on the EARS progress test sets show that the combined BBN/LIMSI system achieved relative reductions of 47% and 51% on the BN and CTS domains, respectively