TIMIT Corpus Research Articles

Automatic classification of English fricatives using cepstral coefficients

We use a classification tool previously tested on Romanian fricatives to categorize the front (non-sibilant) fricatives of English by place of articulation. Labiodental and interdental fricatives are difficult to distinguish acoustically, posing problems even for human perception. Prior classification work with English front fricatives has not been very successful with this contrast, with correct classification rates ranging from 40 to 60%. The feature set we use for coding the acoustic properties of the fricatives and their following vowels comprise the first six cepstral coefficients (c0–c5). The acoustic features are measured at 10-ms intervals across each segment; the measures obtained are then binned into three contiguous intervals for both the fricative and the vowel, representing the onset, steady state, and offset of each segment. The boundaries between regions are set by using a hidden Markov model to determine three internally uniform regions with respect to their acoustic properties. The mean value of each acoustic feature within each region is obtained; thus, each CV production yields six measurements per coefficient. We are testing this model on fricatives from the TIMIT corpus and classifying them using multinomial logistic regression models. While this investigation is underway, we expect higher correct classification rates than previous work.

Physiologically Motivated Feature Extraction for Robust Automatic Speech Recognition

In this paper, a new method is presented to extract robust speech features in the presence of the external noise. The proposed method based on two-dimensional Gabor filters takes in account the spectro-temporal modulation frequencies and also limits the redundancy on the feature level. The performance of the proposed feature extraction method was evaluated on isolated speech words which are extracted from TIMIT corpus and corrupted by background noise. The evaluation results demonstrate that the proposed feature extraction method outperforms the classic methods such as Perceptual Linear Prediction, Linear Predictive Coding, Linear Prediction Cepstral coefficients and Mel Frequency Cepstral Coefficients.

A Doubly Hierarchical Dirichlet Process Hidden Markov Model with a Non-Ergodic Structure

Nonparametric Bayesian models use a Bayesian framework to learn model complexity automatically from the data, eliminating the need for a complex model selection process. A Hierarchical Dirichlet Process Hidden Markov Model (HDPHMM) is the nonparametric Bayesian equivalent of a hidden Markov model (HMM), but is restricted to an ergodic topology that uses a Dirichlet Process Model to achieve a mixture distribution-like model. For applications involving ordered sequences (e.g., speech recognition), it is desirable to impose a left-to-right structure on the model. In this paper, we introduce a model based on HDPHMM that: 1) shares data points between states, 2) models non-ergodic structures, and 3) models non-emitting states. The first point is particularly important because Gaussian mixture models, which support such sharing, have been very effective at modeling modalities in a signal (e.g., speaker variability). Further, sharing data points allows models to be estimated more accurately, an important consideration for applications such as speech recognition in which some mixture components occur infrequently. We demonstrate that this new model produces a 20% relative reduction in error rate for phoneme classification and an 18% relative reduction on a speech recognition task on the TIMIT Corpus compared to a baseline system consisting of a parametric HMM.

Speaker height estimation from speech: Fusing spectral regression and statistical acoustic models.

Estimating speaker height can assist in voice forensic analysis and provide additional side knowledge to benefit automatic speaker identification or acoustic model selection for automatic speech recognition. In this study, a statistical approach to height estimation that incorporates acoustic models within a non-uniform height bin width Gaussian mixture model structure as well as a formant analysis approach that employs linear regression on selected phones are presented. The accuracy and trade-offs of these systems are explored by examining the consistency of the results, location, and causes of error as well a combined fusion of the two systems using data from the TIMIT corpus. Open set testing is also presented using the Multi-session Audio Research Project corpus and publicly available YouTube audio to examine the effect of channel mismatch between training and testing data and provide a realistic open domain testing scenario. The proposed algorithms achieve a highly competitive performance to previously published literature. Although the different data partitioning in the literature and this study may prevent performance comparisons in absolute terms, the mean average error of 4.89 cm for males and 4.55 cm for females provided by the proposed algorithm on TIMIT utterances containing selected phones suggest a considerable estimation error decrease compared to past efforts.

Convex weighting criteria for speaking rate estimation.

Speaking rate estimation directly from the speech waveform is a long-standing problem in speech signal processing. In this paper, we pose the speaking rate estimation problem as that of estimating a temporal density function whose integral over a given interval yields the speaking rate within that interval. In contrast to many existing methods, we avoid the more difficult task of detecting individual phonemes within the speech signal and we avoid heuristics such as thresholding the temporal envelope to estimate the number of vowels. Rather, the proposed method aims to learn an optimal weighting function that can be directly applied to time-frequency features in a speech signal to yield a temporal density function. We propose two convex cost functions for learning the weighting functions and an adaptation strategy to customize the approach to a particular speaker using minimal training. The algorithms are evaluated on the TIMIT corpus, on a dysarthric speech corpus, and on the ICSI Switchboard spontaneous speech corpus. Results show that the proposed methods outperform three competing methods on both healthy and dysarthric speech. In addition, for spontaneous speech rate estimation, the result show a high correlation between the estimated speaking rate and ground truth values.

Augmenting acoustic phonetics with articulatory features for phone recognition

In articulatory phonetics, a phoneme’s identity is specified by its articulator-free (manner) and articulator-bound (place) features. Previous studies have shown that acoustic-phonetic features (APs) can be used to segment speech into broad classes determined by the manner of articulation of speech sounds; compared to MFCCs, however, APs perform poorly in determining place of articulation. This study explores the combination of APs with vocal Tract constriction Variables (TVs) to distinguish phonemes according to their place of articulation for stops, fricatives and nasals. TVs were estimated from acoustics using speech inversion systems trained on the XRMB database with pellet trajectories converted into TVs. TIMIT corpus sentences were first segmented into broad classes using a landmark based broad class segmentation algorithm. Each stop, fricative and nasal speech segment was further classified according to its place of articulation: stops were classified as bilabial (/P/, /B/), alveolar (/T/, /D/) or velar (/K/, /G/); fricatives were classified as labiodental (/F/, /V/), alveolar (/TH/, /DH/, /S/, /Z/), palatal (/SH/, /ZH/) or glottal (/HH/); and nasals were classified as bilabial (/M/), alveolar (/N/) or velar (/NG/). Polynomial kernel support vector machines were trained on APs concatenated with vocal tract constriction features. Results showed that combining acoustic and articulatory features leads to reliable recognition of manner and place of articulation, and improves phone recognition.

Blind phone segmentation based on spectral change detection using Legendre polynomial approximation.

Phone segmentation involves partitioning a continuous speech signal into discrete phone units. In this paper, a method for automatic phone segmentation without prior knowledge of speech content is proposed. The signal spectrum was represented by band-energies. A segment of the band-energy curve was approximated using Legendre polynomial expansion, allowing Legendre polynomial coefficients to describe the properties of the segment. The spectral changes, which imply phone boundaries in the speech signal, were then detected by monitoring the variations of Legendre polynomial coefficients. A two-step algorithm for detecting phone boundaries was derived. The first step was to detect phone boundaries using first-order and second-order coefficients of the Legendre polynomial approximation. The second step was to locate slow spectral changes in the regions of concatenated voiced phones using zero-order coefficients of the Legendre polynomial approximation. This enabled the phone boundaries missed during the first step to be recovered. An evaluation using the TIMIT corpus indicated that the proposed method is comparable to or more accurate than previous methods.

Deep Neural Networks with Multistate Activation Functions.

We propose multistate activation functions (MSAFs) for deep neural networks (DNNs). These MSAFs are new kinds of activation functions which are capable of representing more than two states, including the N-order MSAFs and the symmetrical MSAF. DNNs with these MSAFs can be trained via conventional Stochastic Gradient Descent (SGD) as well as mean-normalised SGD. We also discuss how these MSAFs perform when used to resolve classification problems. Experimental results on the TIMIT corpus reveal that, on speech recognition tasks, DNNs with MSAFs perform better than the conventional DNNs, getting a relative improvement of 5.60% on phoneme error rates. Further experiments also reveal that mean-normalised SGD facilitates the training processes of DNNs with MSAFs, especially when being with large training sets. The models can also be directly trained without pretraining when the training set is sufficiently large, which results in a considerable relative improvement of 5.82% on word error rates.

Corpus testing a fricative discriminator: Or, just how invariant is this invariant?

Acoustic cues to the distinction between sibilant fricatives are claimed to be invariant across languages. Evers et al. (1998) present a method for distinguishing automatically between [s] and [ʃ], using the slope of regression lines over separate frequency ranges within a DFT spectrum. They report accuracy rates in excess of 90% for fricatives extracted from recordings of minimal pairs in English, Dutch and Bengali. These findings are broadly replicated by Maniwa et al. (2009), using VCV tokens recorded in the lab. We tested the algorithm from Evers et al. (1998) against tokens of fricatives extracted from the TIMIT corpus of American English read speech, and the Kiel corpora of German. We were able to achieve similar accuracy rates to those reported in previous studies, with the following caveats: (1) the measure relies on being able to perform a DFT for frequencies from 0 to 8 kHz, so that a minimum sampling rate of 16 kHz is necessary for it to be effective, and (2) although the measure draws a similarly clear distinction between [s] and [ʃ] to those found in previous studies, the threshold value between the two sounds is sensitive to the dynamic range of the input signal.

Selection and enhancement of Gabor filters for automatic speech recognition

Motivated by neurophysiological studies, the use of Gabor filters as acoustic feature extractors for speech recognition purposes has received increasing attention in the new millenium. As the optimal parametrization of these filters is not obvious, many researchers employ different feature selection methods to find the best filter set. In this study, however, we argue that these kinds of feature selection methods cannot fulfill this task, as we demonstrate this with results obtained from experiments. We show that one can easily construct a better filter set manually, using simple heuristic rules. Then, as an alternative to the usual filter selection methods, we propose a training method that can jointly optimize the spectro-temporal filters and the neural net acoustic model built on them. In this special neural network achitecture, the filters are incorporated into the network and employed as the lowest layer of it. This allows us to tune the filters using backpropagation, and to manipulate them directly and not through their parameters. This method also has the advantage of reducing the task of filter set enhancement to that of a simple neural net training. Next, we show that we can enhance our manually selected filter set with this novel neural net architecture using the filter coefficients as initial values for the backpropagation training. The resulting filter sets were evaluated on the phone recognition task of the TIMIT corpus, using both clean and noise contaminated data; while cross-database phone recognition performance was evaluated on the “Szeged” Hungarian broadcast news database. The results we get demonstrate that the proposed filter optimization algorithm can outperform the usual feature selection-based methods, and that the filter set obtained by fine tuning the manual filters with the neural net algorithm performs even better, beating all the other methods in terms of performance.

A corpus-based analysis of English segments produced by Korean learners

This study investigated English segments produced by Korean learners using a large speech corpus. The study focused on capturing only the most noticeable segmental variations produced by learners and that were not found in smaller-scale studies. Analysis of the Korean learners' most noticeable variation patterns was performed by introducing a method of comparing the variation matrices of both Korean learners and native speakers. This comparison, based on manual transcriptions of a large speech corpus of learners and the TIMIT corpus transcriptions, was made to examine the properties of the learners' segmental production of English. The results demonstrated that Korean learners showed patterns of segmental variation that differed from those of native speakers; the learners were greatly influenced by orthography for vowels, while they tended to be influenced by their native language for consonants. These results illustrate the usefulness of our method in capturing the most noticeable variations of learners that differentiate them from native speakers.

A hierarchical model of vision (HMAX) can also recognize speech

HMAX is a well-known computational model of visual recognition in cortex consisting of just two computational operations – a “template match” and non-linear pooling – alternating in a feedforward hierarchy in which receptive fields exhibit increasing specificity and invariance [1]. Interestingly, auditory recognition problems (such as speech recognition) share similar computational requirements, and recent work in auditory neuroscience suggests that auditory and visual cortex share similar anatomical and functional organization. Based on these similarities, we tested whether HMAX could support an auditory recognition task (specifically, word spotting). To test HMAX on word spotting, recorded speech samples from the TIMIT corpus [2] were first converted into time-frequency spectrograms using a computational model of the auditory periphery [3]. These spectrograms were then split into 750 ms frames and input to a standard HMAX model [4]. Based on observed similarities between the receptive fields in primary auditory cortex (spectro-temporal receptive fields, or STRFs) and primary visual cortex (typically modeled as oriented Gabor filters), we used S1 filters identical to those used in vision [4]. Similarly, S2 “patches” were randomly selected from C1 representations of speech sounds drawn from an independent speech corpus. One vs. all linear support vector machines (SVMs) were then trained to discriminate frames that contain a target word from those that did not. These SVMs were then tested on a novel set of test sentences using a sliding frame approach (750 ms frame size, 20 ms step size). For each frame in a sentence, the SVM produced a distance from the hyperplane, and a threshold value was applied to produce a binary classification whether or not the target word was present in the sentence. When tested on target words that appeared in a fixed context (i.e. SA sentences in TIMIT), performance was highly robust, with ROC areas consistently above 0.9. When tested on target words that appeared in variable contexts (i.e., SI sentences in TIMIT), performance was somewhat decreased with ROC areas around 0.8. This decrease in performance is likely due to the inclusion of “clutter” (i.e., target irrelevant features) within the frame, also commonly observed when HMAX is applied to visual object recognition tasks [1]. These results are novel in that they provide support for the hypothesis that the simple computational framework implemented in HMAX – consisting of a feedforward hierarchy of only two alternating computational operations – may generalize beyond vision to support auditory recognition as well. It is possible that such a representation could give rise to stable neural encodings that are invariant to behaviorally irrelevant characteristics as seen in higher order visual and auditory cortices [5,6]. While it is likely that this auditory version of the HMAX model would benefit from the use of more auditory-specific filters based on STRF models [7], the Gabor features used here are largely compatible with previous computational models based on STRFs up to the level of primary auditory cortex [8]. Additional benefit may also be gained by learning sparse representations from natural sounds, at both the S1 and S2 levels [9].

Rapid speech keyword spotting method based on template matching

When dealing with keywords detection without training samples, template matching-based keyword spotting can still be able to spot compared with the traditional method. However, template matching-based method is time-consuming,because it uses frame-by-frame move method to calculate the local minimum distance. The extreme points of the local minimum distance are usually near phoneme segmentation points. A fast template matching method can come out by combining their positions with interpolation idea. By using interpolation to generate the local minimum distance between phoneme segmentation points, this method can greatly reduce the calculation time. When running on the TIMIT and CASIA corpus, the improved method approximately is 2. 8 times faster than the conventional template matching-based keyword spotting.

Classification of affricate burst place in consonant-vowel contexts in English

This study investigates characteristics of affricate burst place of articulation compared with the bursts for the three places of articulation for stops (labial, alveolar, and velar) in English. The data comprise consonant-vowel tokens in the TIMIT corpus. To assess which stop place of articulation may be used to simultaneously model affricate bursts, Jensen-Shannon divergence measures are found for probability distributions of acoustic-phonetic features. In addition, we conduct classification experiments using combinations of acoustic-phonetic features and Mel-frequency cepstral coefficients (MFCCs), to see how well affricate burst place is classified using models for the three stop places. The experimental results show that although affricate place is similar to the alveolar place of articulation for stops, a separate post-alveolar place for affricate burst provides a better model. The results suggest that a separate affricate place model will be useful in a feature-based speech recognition system that explicitly detects place of articulation for consonants.

Perceptual and phonotactic effects in loanword adaptation: English postvocalic stops in Taiwan Mandarin

When an English word with a postvocalic stop is borrowed into Taiwan Mandarin (TM), because TM allows only nasal coda consonants, an often-used strategy to repair the illegality is vowel-insertion (e.g., Wood → [wu.tv̥]). Based on a corpus study of 335 English borrowed names, this trend is confirmed (76%). Among the deleted cases, an asymmetry of different places of articulation was found: Coronal stops are deleted most often (15%, e.g., Hollywood → [hau.lai.wu]), and dorsals more often than labials (12%, e.g., Titanic → [thiɛ.ta.ni] vs 0%, e.g., Jeep → [tɕ. phu]). Following Kang’s (2003) perceptual explanation, the tendency of coronal-deletion can be explained by the fact that postvocalic coronals are often unreleased and thus less perceptually salient to TM speakers. According to TIMIT corpus, the release rate of coronals, labials, and dorsals stops is 37%, 52%, and 83%, respectively (Kang, 2003). However, this cannot explain the reversed pattern of dorsals and labials. I propose that this is due to another factor: Labial coda is marked in TM since only [n] and [ŋ], but not [m], can occur in coda position. In other words, the deletion of postvocalic stops depends on the saliency that considers both perceptual and phonotactic factors.

Environment dependent noise tracking for speech enhancement

Numerous efforts have focused on the problem of reducing the impact of noise on the performance of various speech systems such as speech recognition, speaker recognition, and speech coding. These approaches consider alternative speech features, improved speech modeling, or alternative training for acoustic speech models. This study presents an alternative viewpoint by approaching the same problem from the noise perspective. Here, a framework is developed to analyze and use the noise information available for improving performance of speech systems. The proposed framework focuses on explicitly modeling the noise and its impact on speech system performance in the context of speech enhancement. The framework is then employed for development of a novel noise tracking algorithm for achieving better speech enhancement under highly evolving noise types. The first part of this study employs a noise update rate in conjunction with a target enhancement algorithm to evaluate the need for tracking in many enhancement algorithms. It is shown that noise tracking is more beneficial in some environments than others. This is evaluated using the Log-MMSE enhancement scheme for a corpus of four noise types consisting of Babble (BAB), White Gaussian (WGN), Aircraft Cockpit (ACN), and Highway Car (CAR) using the Itakura-Saito (IS) (Gray et al. in IEEE Trans. Acoust. Speech Signal Process. 28:367---376, 1980) quality measure. A test set of 200 speech utterances from the TIMIT corpus are used for evaluations. The new Environmentally Aware Noise Tracking (EA-NT) method is shown to be superior in comparison with the contemporary noise tracking algorithms. Evaluations are performed for speech degraded using a corpus of four noise types consisting of: Babble (BAB), Machine Gun (MGN), Large Crowd (LCR), and White Gaussian (WGN). Unlike existing approaches, this study provides an effective foundation for addressing noise in speech by emphasizing noise modeling so that available resources can be used to achieve more reliable overall performance in speech systems.

Low-Variance Multitaper MFCC Features: A Case Study in Robust Speaker Verification

In speech and audio applications, short-term signal spectrum is often represented using mel-frequency cepstral coefficients (MFCCs) computed from a windowed discrete Fourier transform (DFT). Windowing reduces spectral leakage but variance of the spectrum estimate remains high. An elegant extension to windowed DFT is the so-called multitaper method which uses multiple time-domain windows (tapers) with frequency-domain averaging. Multitapers have received little attention in speech processing even though they produce low-variance features. In this paper, we propose the multitaper method for MFCC extraction with a practical focus. We provide, first, detailed statistical analysis of MFCC bias and variance using autoregressive process simulations on the TIMIT corpus. For speaker verification experiments on the NIST 2002 and 2008 SRE corpora, we consider three Gaussian mixture model based classifiers with universal background model (GMM-UBM), support vector machine (GMM-SVM) and joint factor analysis (GMM-JFA). Multitapers improve MinDCF over the baseline windowed DFT by relative 20.4% (GMM-SVM) and 13.7% (GMM-JFA) on the interview-interview condition in NIST 2008. The GMM-JFA system further reduces MinDCF by 18.7% on the telephone data. With these improvements and generally noncritical parameter selection, multitaper MFCCs are a viable candidate for replacing the conventional MFCCs.

Co-registration of articulographic and real-time magnetic resonance imaging data for multimodal analysis of rapid speech

We propose a method for co-registrating speech articulatory/acoustic data from two modalities that provide complementary advantages. Electromagnetic Articulography (EMA) provides high temporal resolution (100 samples/second in WAVE system) and flesh-point tracking, while real-time Magnetic Resonance Imaging, rtMRI, (23 frames/second) offers a complete midsagittal view of the vocal tract, including articulated structures and the articulatory environment. Co-registration was achieved through iterative alignment in the acoustic and articulatory domains. Acoustic signals were aligned temporally using Dynamic Time Warping, while articulatory signals were aligned variously by minimization of mean total error between articulatometry data and estimated corresponding flesh points and by using mutual information derived from articulatory parameters for each sentence. We demonstrate our method on a subset of the TIMIT corpus elicited from a male and a female speaker of American English, and illustrate the benefits of co-registered multi-modal data in the study of liquid and fricative consonant production in rapid speech. [Supported by NIH and NSF grants.]

Keyword spotting exploiting Long Short-Term Memory

We investigate various techniques for keyword spotting which are exclusively based on acoustic modeling and do not presume the existence of an in-domain language model. Since adequate context modeling is nevertheless necessary for word spotting, we show how the principle of Long Short-Term Memory (LSTM) can be incorporated into the decoding process. We propose a novel technique that exploits LSTM in combination with Connectionist Temporal Classification in order to improve performance by using a self-learned amount of contextual information. All considered approaches are evaluated on read speech as contained in the TIMIT corpus as well as on the SEMAINE database which consists of spontaneous and emotionally colored speech. As further evidence for the effectiveness of LSTM modeling for keyword spotting, results on the CHiME task are shown.

Genetic Programming for Robust Text Independent Speaker Verification

Robust Automatic Speaker Verification has become increasingly desirable in recent years with the growing trend toward remote security verification procedures for telephone banking, bio-metric security measures and similar applications. While many approaches have been applied to this problem, Genetic Programming offers inherent feature selection and solutions that can be meaningfully analyzed, making it well suited for this task. This article introduces a Genetic Programming system to evolve programs capable of speaker verification and evaluates its performance with the publicly available TIMIT corpora. Also presented are the effects of a simulated telephone network on classification results which highlight the principal advantage, namely robustness to both additive and convolutive noise.