Formant Tracking Research Articles

Automated Measurement of Vowel Formants in the Buckeye Corpus

In recent years, corpus phonetics has become a rapidly expanding field. However, the lack of appropriate tools for automatic acoustic analysis hinders further development of the field. In this paper, we present a methodological study on the automatic extraction of vowel formants using both robust linear predictive coding (RLPC; Lee, 1988) and dynamic formant tracking (Talkin, 1987). Acoustic data were taken from the Buckeye corpus of English conversations. We varied two aspects of the analysis - preemphasis and LPC order - to optimize formant tracking results by speaker and vowel. We also show, based on the optimal results, the distribution of ten English vowels in the F1/F2 space in conversational speech.

Formant Tracking Linear Prediction Model using HMMs for Noisy Speech Processing

This paper presents a formant-tracking linear prediction (FTLP) model for speech processing in noise. The main focus of this work is the detection of formant trajectory based on Hidden Markov Models (HMM), for improved formant estimation in noise. The approach proposed in this paper provides a systematic framework for modelling and utilization of a timesequence of peaks which satisfies continuity constraints on parameter; the within peaks are modelled by the LP parameters. The formant tracking LP model estimation is composed of three stages: (1) a pre-cleaning multi-band spectral subtraction stage to reduce the effect of residue noise on formants (2) estimation stage where an initial estimate of the LP model of speech for each frame is obtained (3) a formant classification using probability models of formants and Viterbi-decoders. The evaluation results for the estimation of the formant tracking LP model tested in Gaussian white noise background, demonstrate that the proposed combination of the initial noise reduction stage with formant tracking and LPC variable order analysis, results in a significant reduction in errors and distortions. The performance was evaluated with noisy natual vowels extracted from international french and English vocabulary speech signals at SNR value of 10dB. In each case, the estimated formants are compared to reference formants. Keywords—Formants Estimation, HMM, Multi Band Spectral Subtraction, Variable order LPC coding, White Gauusien Noise.

Formant‐frequency trajectories as acoustic correlates to speech perception.

Formant trajectories are excellent vowel discriminants; within vowel, they are nearly constant across speaker size, age, and sex, and across consonantal contexts. However, this model assumes that formant peaks are perceptually important and that human listeners track formant‐frequency changes across time. Speech‐recognition applications have avoided formant frequencies due to the difficulty of reliable formant tracking. In addition, it is not actually known whether human listeners do indeed follow formants perceptually across time. This paper presents results from several studies that examine the relationship between changing formant frequencies and perception. Alternative perceptual representations of vowels, such as global spectral shape, are precluded by evidence that individual formant amplitudes are largely ignored in vowel perception. In addition, where other spectral properties appear to have a perceptual effect, it is because stimuli have used formants that do not change. When formants are changing, perceptual effects of spectral shape properties disappear. In terms of human formant tracking, perceptual extrapolation of a formant sweep is mostly dependent on peak frequency and not other properties related to spectral shape. This demonstrates that listeners do indeed follow formant‐frequency changes as auditory objects. Further research on formant frequency perception will be described.

An acoustical comparison of English tense and lax vowels.

Several studies on the pronunciation of English vowels point out that Korean learners have difficulty producing English tense and lax vowel pairs. The acoustic comparisons of those studies are mostly based on the formant measurement at one time point of a given vowel section. However, the English lax vowels usually show dynamic changes across their syllable peaks and subjects’ English levels account for various conflicting results. The purposes of this paper are to compare the temporal duration and dynamic formant tracks of English tense and lax vowel pairs produced by five Korean and five American males. Results showed that both the Korean and American males produced the vowels with comparable durations. The duration of the front tense‐lax vowel pair was longer than that of the back vowel pair. From the formant track comparisons, the American males produced the tense and lax pairs much more distinctly than the Korean males. The results suggest that the Korean males should pay attention to the F1 and F2 movements, i.e., the jaw and tongue movements, in order to match those of the American males. Further studies are recommended on the auditorily acceptable ranges of F2 variation for the lax vowels.

Measuring Norwegian dialect distances using acoustic features

Levenshtein distance has become a popular tool for measuring linguistic dialect distances, and has been applied to Irish Gaelic, Dutch, German and other dialect groups. The method, in the current state of the art, depends upon phonetic transcriptions, even when acoustic differences are used the number of segments in the transcriptions is used for speech rate normalization. The goal of this paper is to find a fully acoustic measure which approximates the quality of semi-acoustic measures that rely on tagged speech. We use a set of 15 Norwegian dialect recordings and test the hypothesis that the use of the acoustic signal only, without transcriptions, is sufficient for obtaining results which largely agree with both traditional Norwegian dialectology and the perception of the speakers themselves. We use formant trajectories and consider both the Hertz and the Bark scale. We experiment with an approach in which z-scores per frame are used instead of the original frequency values. Besides formant tracks, we also consider zero crossing rates: the number of times per interval that the amplitude waveform crosses the zero line. The zero crossing rate is sensitive to the difference between voiced and unvoiced speech sections. When using the fully acoustic measure on the basis of the combined representation with normalized frequency values, we obtained results comparable with the results obtained with the semi-acoustic measure. We applied cluster analysis and multidimensional scaling to distances obtained with this method and found results which largely agree with both the results of traditional Norwegian dialectology and with the perception of the speakers. When scaling to three dimensions, we found the first dimension responsible for gender differences. However, when leaving out this dimension, dialect specific information is lost as well.

Speaker Identification Using Instantaneous Frequencies

This paper presents an experimental evaluation of different features for use in speaker identification. The features are tested using speech data provided by the chains corpus, in a closed-set speaker identification task. The main objective of the paper is to present a novel parametrization of speech that is based on the AM-FM representation of the speech signal and to assess the utility of these features in the context of speaker identification. In order to explore the extent to which different instantaneous frequencies due to the presence of formants and harmonics in the speech signal may predict a speaker's identity, this work evaluates three different decompositions of the speech signal within the same AM-FM framework: a first setup has been used previously for formant tracking, a second setup is designed to enhance familiar resonances below 4000 Hz, and a third setup is designed to approximate the bandwidth scaling of the filters conventionally used in the extraction of Mel-fequency cepstral coefficients (MFCCs). From each of the proposed setups, parameters are extracted and used in a closed-set text-independent speaker identification task. The performance of the new featural representation is compared with results obtained adopting MFCC and RASTA-PLP features in the context of a generic Gaussian mixture model (GMM) classification system. In evaluating the novel features, we look selectively at information for speaker identification contained in the frequency range 0-4000 Hz and 4000-8000 Hz, as the instantaneous frequencies revealed by the AM-FM approach suggest the presence of structures not well known from conventional spectrographic analyses. Accuracy results obtained using the new parametrization perform as well as conventional MFCC parameters within the same reference system, when tested and trained on modally voiced speech which is mismatched in both channel and style. When the testing material is whispered speech, the new parameters provide better results than any of the other features tested, although they remain far from ideal in this limiting case.

Bayesian formant tracking using conditionally linear Gaussian models

Formants play a central role in the perception and analysis of speech. In this presentation we describe Bayesian approaches to estimating vocal tract resonances from speech waveforms, formulated as a statistical model‐based tracking problem. In particular, approaches by Deng and colleagues have shown the promise of an extended Kalman filtering approach based on a robust linearization of the formant‐to‐cepstrum map. We describe recent extensions to model inter‐ as well as intra‐formant correlation, and detail efficient inference schemes that preserve conditional Gaussianity. A database of formant trajectories provides a notion of ground truth by which estimator performance can be evaluated, and which demonstrates the efficacy of our approach relative to contemporary benchmark tools for formant analysis.

Kalman tracking of linear predictor and harmonic noise models for noisy speech enhancement

This paper presents a speech enhancement method based on the tracking and denoising of the formants of a linear prediction (LP) model of the spectral envelope of speech and the parameters of a harmonic noise model (HNM) of its excitation. The main advantages of tracking and denoising the prominent energy contours of speech are the efficient use of the spectral and temporal structures of successive speech frames and a mitigation of processing artefact known as the ‘musical noise’ or ‘musical tones’. The formant-tracking linear prediction (FTLP) model estimation consists of three stages: (a) speech pre-cleaning based on a spectral amplitude estimation, (b) formant-tracking across successive speech frames using the Viterbi method, and (c) Kalman filtering of the formant trajectories across successive speech frames. The HNM parameters for the excitation signal comprise; voiced/unvoiced decision, the fundamental frequency, the harmonics’ amplitudes and the variance of the noise component of excitation. A frequency-domain pitch extraction method is proposed that searches for the peak signal to noise ratios (SNRs) at the harmonics. For each speech frame several pitch candidates are calculated. An estimate of the pitch trajectory across successive frames is obtained using a Viterbi decoder. The trajectories of the noisy excitation harmonics across successive speech frames are modeled and denoised using Kalman filters. The proposed method is used to deconstruct noisy speech, de-noise its model parameters and then reconstitute speech from its cleaned parts. Experimental evaluations show the performance gains of the formant tracking, pitch extraction and noise reduction stages.

Analysis and Synthesis of Formant Spaces of British, Australian, and American Accents

In this paper, the probability distribution functions (pdf's) of the formant spaces of three major accents of the English language, namely, British Received Pronunciation (RP), General American, and Broad Australian, are modeled and compared. The statistical differences across the formant spaces of these accents are employed for accent conversion. An improved formant tracking method, based on linear prediction (LP) feature analysis and a two-dimensional hidden Markov model (2-D-HMM) of format trajectories, is used for estimation of the formant trajectories of vowels and diphthongs of each accent. Comparative analysis of the formant spaces of the three accents indicates that these accents are partly conveyed by the differences of the formants of vowels. The estimates of the probability distributions of the formants for each accent are used in a speech synthesis system for accent conversion. Accent synthesis, through modification of the acoustic parameters of speech, provides a means of assessing the perceptual contribution of each formant parameter on conveying an accent. The results of perceptual evaluations of accent conversion illustrate that formants play an important role in conveying accents

Formant tracking linear prediction model using HMMs and Kalman filters for noisy speech processing

This paper presents a formant tracking linear prediction (LP) model for speech processing in noise. The main focus of this work is on the utilization of the correlation of the energy contours of speech, along the formant tracks, for improved formant and LP model estimation in noise. The approach proposed in this paper provides a systematic framework for modelling and utilization of the inter-frame correlation of speech parameters across successive speech frames; the within frame correlations are modelled by the LP parameters. The formant tracking LP model estimation is composed of three stages: (1) a pre-cleaning spectral amplitude estimation stage where an initial estimate of the LP model of speech for each frame is obtained, (2) a formant classification and estimation stage using probability models of formants and Viterbi-decoders and (3) an inter-frame formant de-noising and smoothing stage where Kalman filters are used to model the formant trajectories and reduce the effect of residue noise on formants. The adverse effects of car and train noise on estimates of formant tracks and LP models are investigated. The evaluation results for the estimation of the formant tracking LP model demonstrate that the proposed combination of the initial noise reduction stage with formant tracking and Kalman smoothing stages, results in a significant reduction in errors and distortions.

Compensation following real-time manipulation of formants in isolated vowels

Auditory feedback influences human speech production, as demonstrated by studies using rapid pitch and loudness changes. Feedback has also been investigated using the gradual manipulation of formants in adaptation studies with whispered speech. In the work reported here, the first formant of steady-state isolated vowels was unexpectedly altered within trials for voiced speech. This was achieved using a real-time formant tracking and filtering system developed for this purpose. The first formant of vowel /epsilon/ was manipulated 100% toward either /ae/ or /I/, and participants responded by altering their production with average Fl compensation as large as 16.3% and 10.6% of the applied formant shift, respectively. Compensation was estimated to begin <460 ms after stimulus onset. The rapid formant compensations found here suggest that auditory feedback control is similar for both F0 and formants.

Initialization, Training, and Context-Dependency in HMM-Based Formant Tracking

This paper presents an algorithm for formant tracking using HMMs and analyzes the influence of HMM initialization, training and context-dependency on the accuracy of the formant tracks obtained with the HMMs. Formant trackers usually include two different phases: one in which the speech is analyzed and formant candidates are obtained, and another in which, by imposing different constraints, the most likely formants are chosen. While the first stage usually relies on standard spectrum estimation techniques, the second stage has evolved notably in the recent years. Traditionally the second phase tries to impose continuity constraints on the formant selection process. Lately there has been ongoing research to include phonemic knowledge in the second stage to make formant tracking more reliable. In order to incorporate phonemic knowledge newer approaches make use of the orthographic transcription of the speech utterance. From the orthographic transcription, the phonemic transcription is obtained, and from this and the speech itself a phonemic segmentation can be obtained. This phonemic segmentation, along with the phonemic transcription and some knowledge of the nominal formant positions for the different phonemes provides extra information that can be used to obtain more accurate formant tracks. This paper presents a complete HMM-based data-driven algorithm for formant tracking suitable to combine different levels of acoustic and phonemic information. A detailed analysis on the performance of this algorithm is discussed for: different initialization strategies using different levels of knowledge, different degrees of training, and context-independent and dependent HMMs. Experimental speaker-dependent results show that the efficient use of phonemic information in HMM training and context-dependent modeling significantly reduces the formant tracking error rate especially for formants $ F_2$ and $ F_3$ .

Robust Formant Tracking for Continuous Speech With Speaker Variability

Several algorithms have been developed for tracking formant frequency trajectories of speech signals, however most of these algorithms are either not robust in real-life noise environments or are not suitable for real-time implementation. The algorithm presented in this paper obtains formant frequency estimates from voiced segments of continuous speech by using a time-varying adaptive filterbank to track individual formant frequencies. The formant tracker incorporates an adaptive voicing detector and a gender detector for formant extraction from continuous speech, for both male and female speakers. The algorithm has a low signal delay and provides smooth and accurate estimates for the first four formant frequencies at moderate and high signal-to-noise ratios. Thorough testing of the algorithm has shown that it is robust over a wide range of signal-to-noise ratios for various types of background noises.

A state-space model with neural-network prediction for recovering vocal tract resonances in fluent speech from Mel-cepstral coefficients

In this paper, we present a state-space formulation of a neural-network-based hidden dynamic model of speech whose parameters are trained using an approximate EM algorithm. This efficient and effective training makes use of the output of an off-the-shelf formant tracker (for the vowel segments of the speech signal), in addition to the Mel-cepstral observations, to simplify the complex sufficient statistics that would be required in the exact EM algorithm. The trained model, consisting of the state equation for the target-directed vocal tract resonance (VTR) dynamics on all classes of speech sounds (including consonant closure and constriction) and the observation equation for mapping from the VTR to Mel-cepstral acoustic measurement, is then used to recover the unobserved VTR based on the extended Kalman filter. The results demonstrate accurate estimation of the VTR, especially during rapid consonant–vowel or vowel–consonant transitions and during consonant closure when the acoustic measurement alone provides weak or no information to infer the VTR values. The practical significance of correctly identifying the VTRs during consonantal closure or constriction is that they provide target frequency values for the VTR or formant transitions from adjacent sounds. Without such target values, the VTR transitions from vowel to consonant or from consonant to vowel are often very difficult to extract accurately by the previous formant tracking techniques. With the use of the new technique reported in this paper, the consonantal VTRs and the related transitions become more reliably identified from the speech signal.

Formant tracking using context-dependent phonemic information

A new formant-tracking algorithm using phoneme information is proposed. Conventional formant-tracking algorithms obtain formant tracks by analyzing the acoustic speech signal using continuity constraints without any additional information. The formant-tracking error rate of the conventional methods is reportedly in the range of 10%-20%. In this paper, we show that if text or phoneme transcription of speech utterances is available, the error rate can be significantly reduced. The basic idea behind this approach is that given the phoneme identity, formant-tracking algorithms can have a better clue of where to look for formants. The algorithm consists of three phases: 1) analysis, 2) segmentation and alignment, and 3) formant tracking by the Viterbi searching algorithm. In the analysis phase, formant candidates are obtained for each analysis frame by solving the linear prediction polynomial. In the segmentation and alignment phase, the text corresponding to the input speech utterance is converted into a sequence of phoneme symbols. Then, the phoneme sequence is time aligned with the speech utterance. A hidden Markov model (HMM) based automatic segmentation algorithm is used for forced-time alignment. For each phoneme segment, nominal formant frequencies are assigned at the center of each phoneme segment. Then nominal formant tracks for the entire utterance are obtained by interpolating the nominal formant frequencies. In order to compensate for the coarticulation effect, different interpolation methods are used depending on the phonemic context. The interpolation process makes the formant-tracking algorithm robust to possible segmentation errors made by the HMM-based segmentation algorithm. As a result, the proposed formant-tracking algorithm does not require highly accurate alignment/segmentation. Finally, a set of formants is chosen from the formant candidates in such a way that the resulting formant tracks come close to the nominal formant tracks while satisfying the continuity constraints. The algorithm is tested using natural speech utterances and the performance is compared against formant tracks obtained by the conventional method using continuity constraints only. The new algorithm significantly reduces the formant-tracking error rate (5.03% for male and 3.73% for female) over the conventional formant-tracking algorithm (13.00% for male and 15.82% for female).

Knowledge-based formant tracking with confidence measure using dynamic programming

In this study, we report on refinements to a formant tracking technique originally reported in [Xia et al., ICSLP (2000)]. The formant tracker operates in two phases. First, it finds optimal formant track estimates in oral sonorant regions by imposing frequency continuity constraints using dynamic programming. Second, post-processing is performed to make the estimates more robust and accurate, and to extend formant tracks in nasal and obstruent regions. In recent work, we have improved on our initial estimates of the formants by combining the outputs from a 12th-order LPC analysis and a 16th-order LPC analysis. Additionally, we have added a confidence measure for each formant track in each frame. The confidence measure is based on formant continuity, competing formants, short-time energy, context information, and formant information over the entire utterance. The experiments show that most of the tracking errors are associated with a low confidence value, while the correct formants have high confidence values. The performance of the algorithm in the sonorant regions was tested using randomly selected male and female speech from the TIMIT database. [Work supported by NSF Grant No. BCS0236707.]

Turning speech into music in a two-dimensional space by varying the bandwidth and rate of tone pulses placed along formant tracks

A two-dimensional space is introduced which changes sounds from speech-like to music-like by varying the rate and the bandwidth (decay-time) of individual tones placed along formant contours. At one extreme of a very fast pulse rate and a long decay time, the sounds are equivalent to sine-wave speech. With a more moderate tone rate and a short decay time, the sounds become monotone (at the tone-rate) speech. For tone rates around 10/s with long decay times, individual musical tones are heard at the formant frequencies. Intelligibility is poor for such sounds because of the low sampling rate of the formant contours. Increasing the tone rate and decreasing tone decay time increases intelligibility and makes the sounds less music like. Intelligibility data together with speech-like and music-like ratings of sounds from this space will be presented.

Measuring Norwegian Dialect Distances Using Acoustic Features

Levenshtein distance has become a popular tool for measuring linguistic dialect distances, and has been applied to Irish Gaelic, Dutch, German and other dialect groups. The method, in the current state of the art, depends upon phonetic transcriptions, even when acoustic differences are used the number of segments in the transcriptions is used for speech rate normalization. The goal of this paper is to find a fully acoustic measure which approximates the quality of semi-acoustic measures that rely on tagged speech. We use a set of 15 Norwegian dialect recordings and test the hypothesis that the use of the acoustic signal only, without transcriptions, is sufficient for obtaining results which largely agree with both traditional Norwegian dialectology and the perception of the speakers themselves. We use formant trajectories and consider both the Hertz and the Bark scale. We experiment with an approach in which z-scores per frame are used instead of the original frequency values. Besides formant tracks, we also consider zero crossing rates: the number of times per interval that the amplitude waveform crosses the zero line. The zero crossing rate is sensitive to the difference between voiced and unvoiced speech sections. When using the fully acoustic measure on the basis of the combined representation with normalized frequency values, we obtained results comparable with the results obtained with the semi-acoustic measure. We applied cluster analysis and multidimensional scaling to distances obtained with this method and found results which largely agree with both the results of traditional Norwegian dialectology and with the perception of the speakers. When scaling to three dimensions, we found the first dimension responsible for gender differences. However, when leaving out this dimension, dialect specific information is lost as well.

Formant frequency estimation of high-pitched speech by homomorphic prediction

The conventional model of the linear prediction analysis suffers from difficulties in estimating vocal tract characteristics of high-pitched speakers. This is because the autocorrelation function used by the autocorrelation method of linear prediction for estimating autoregressive coefficients is actually an aliased version of that of the vocal tract impulse response. This aliasing occurs due to the periodic nature of voiced speech. Generally it is accepted that homomorphic filtering can be used to obtain an estimate of vocal tract impulse response which is free from periodicity. Thus linear prediction of the resulting vocal tract impulse response (referred to as homomorphic prediction) is expected to be free from variations of fundamental frequencies. To our knowledge any experimental study, however, has not yet appeared on the suitability of this method for analyzing high-pitched speech. This paper presents a detail study on the prospects of homomorphic prediction as a formant tracking tool especially for high-pitched speech where linear prediction fails to obtain accurate estimation. The formant frequencies estimated using the proposed method are found to be accurate by more than an order of magnitude compared to the conventional procedure. The accuracy of formant estimation is verified on synthetic vowels for a wide range of pitch periods covering typical male and high-pitched female speakers. The validity of the proposed method is also examined by inspecting the spectral envelopes of natural speech spoken by high-pitched female speakers. We noticed that almost all the previous methods dealing with this limitation of linear prediction are based on the covariance technique where the obtained AR filter can be unstable. The solutions obtained by the current method are guaranteed to be stable which makes it superior for many speech analysis applications.

Objective analysis of the singing voice as a training aid

A new tool for robust tracking of fundamental frequency is proposed, along with an objective measure of main singing voice parameters, such as vibrato rate, vibrato extent, and vocal intonation. High-resolution Power Spectral Density estimation is implemented, based on AutoRegressive models of suitable order, allowing reliable formant tracking also in vocalizations characterized by highly varying values. The proposed techniques are applied to about 1000 vocalizations, coming from both professional and non-professional singers, and show better performance as compared to classical Fourier-based approaches. If properly implemented, and with a user-friendly interface, the new tool would allow real-time analysis of singing voice. Hence, it could be of help in giving non-professional singers and singing teachers reliable measures of possible improvements during and after training.