Formant Tracking Research Articles

An Approach to Explaining Formants

Purpose: This tutorial is a description of a possible approach to teaching the concept of formants to students in a speech science course, at either the undergraduate or graduate level. The approach is to explain formants as prominent regions of energy in the output spectrum envelope radiated at the lips, and how they arise as the superposition of vocal tract resonances on a source signal. Standing waves associated with vocal tract resonances are briefly explained and standing wave animations are provided. Animations of the temporal variation of the vocal tract, vocal tract resonances, spectra, and spectrograms, along with audio samples are included to provide dynamic demonstrations of the concept of formants. Conclusions: The explanations, accompanying demonstrations, and suggested activities are intended to provide a launching point for understanding formants and how they can be measured, analyzed, and interpreted. As a result, participants should be able to describe the meaning of the term “formant” as it relates to a spectrum and a spectrogram, explain the difference between formants and vocal tract resonances, explain how vocal tract resonances combined with the voice source generate formants, and identify formants in both narrow-band and wide-band spectrograms and track their time-varying patterns with a formant tracking algorithm. Supplemental Material: https://doi.org/10.23641/asha.25168202

Vowel distinctiveness as a concurrent predictor of expressive language function in autistic children.

Speech ability may limit spoken language development in some minimally verbal autistic children. In this study, we aimed to determine whether an acoustic measure of speech production, vowel distinctiveness, is concurrently related to expressive language (EL) for autistic children. Syllables containing the vowels [i] and [a] were recorded remotely from 27 autistic children (4;1-7;11) with a range of spoken language abilities. Vowel distinctiveness was calculated using automatic formant tracking software. Robust hierarchical regressions were conducted with receptive language (RL) and vowel distinctiveness as predictors of EL. Hierarchical regressions were also conducted within a High EL and a Low EL subgroup. Vowel distinctiveness accounted for 29% of the variance in EL for the entire group, RL for 38%. For the Low EL group, only vowel distinctiveness was significant, accounting for 38% of variance in EL. Conversely, in the High EL group, only RL was significant and accounted for 26% of variance in EL. Replicating previous results, speech production and RL significantly predicted concurrent EL in autistic children, with speech production being the sole significant predictor for the Low EL group and RL the sole significant predictor for the High EL group. Further work is needed to determine whether vowel distinctiveness longitudinally, as well as concurrently, predicts EL. Findings have important implications for the early identification of language impairment and in developing language interventions for autistic children.

Applying the Lombard Effect to Speech-in-Noise Communication

This study explored how the Lombard effect, a natural or artificial increase in speech loudness in noisy environments, can improve speech-in-noise communication. This study consisted of several experiments that measured the impact of different types of noise on synthesizing the Lombard effect. The main steps were as follows: first, a dataset of speech samples with and without the Lombard effect was collected in a controlled setting; then, the frequency changes in the speech signals were detected using the McAulay and Quartieri algorithm based on a 2D speech representation; next, an average formant track error was computed as a metric to evaluate the quality of the speech signals in noise. Three image assessment methods, namely the SSIM (Structural SIMilarity) index, RMSE (Root Mean Square Error), and dHash (Difference Hash) were used for this purpose. Furthermore, this study analyzed various spectral features of the speech signals in relation to the Lombard effect and the noise types. Finally, this study proposed a method for automatic noise profiling and applied pitch modifications to neutral speech signals according to the profile and the frequency change patterns. This study used an overlap-add synthesis in the STRAIGHT vocoder to generate the synthesized speech.

Refining a deep learning-based formant tracker using linear prediction methods

In this study, formant tracking is investigated by refining the formants tracked by an existing data-driven tracker, DeepFormants, using the formants estimated in a model-driven manner by linear prediction (LP)-based methods. As LP-based formant estimation methods, conventional covariance analysis (LP-COV) and the recently proposed quasi-closed phase forward–backward (QCP-FB) analysis are used. In the proposed refinement approach, the contours of the three lowest formants are first predicted by the data-driven DeepFormants tracker, and the predicted formants are replaced frame-wise with local spectral peaks shown by the model-driven LP-based methods. The refinement procedure can be plugged into the DeepFormants tracker with no need for any new data learning. Two refined DeepFormants trackers were compared with the original DeepFormants and with five known traditional trackers using the popular vocal tract resonance (VTR) corpus. The results indicated that the data-driven DeepFormants trackers outperformed the conventional trackers and that the best performance was obtained by refining the formants predicted by DeepFormants using QCP-FB analysis. In addition, by tracking formants using VTR speech that was corrupted by additive noise, the study showed that the refined DeepFormants trackers were more resilient to noise than the reference trackers. In general, these results suggest that LP-based model-driven approaches, which have traditionally been used in formant estimation, can be combined with a modern data-driven tracker easily with no further training to improve the tracker’s performance.

The challenges and uncertainties when hand measuring F1/F2 values in child speech: Towards an automated approach

Vowels serve a critical role in understanding the acoustic properties of speech. Since they appear early in speech development, they can be used to assess important speech milestones. Prior literature provides the empirical foundation for understanding expected F1/F2 values for typically developing children. Although the traditional measurement of the first and second formant frequencies is a widely accepted practice, its clinical use is limited because of the challenges and uncertainties that are encountered during the process. Automated tools have been developed to address these limitations, however many of these tools have been reported to have inconsistencies when identifying F1/F2 values for child speech. This study explores the use of three semi-automated measurement methods: Wavesurfer Linear Predictive Coding, Praat formant tracking, and a lab-developed approach. The speech of six, native English-speaking, four-year-old children with no history of hearing or speech disorders was analyzed. 48 words, including eight American English vowels, were measured using the three approaches by two researchers, an undergraduate researcher, and an experienced research mentor. The challenges, procedural observations, and accuracy of the methods are discussed. This project was supported by the SURIEA summer research internship program.

Adjustable deterministic pseudonymization of speech

While public speech resources become increasingly available, there is a growing interest to preserve the privacy of the speakers, through methods that anonymize the speaker information from speech while preserving the spoken linguistic content. In this paper, a method for pseudonymization (reversible anonymization) of speech is presented, that allows to obfuscate the speaker identity in untranscribed running speech. The approach manipulates the spectro-temporal structure of the speech to simulate a different length and structure of the vocal tract by modifying the formant locations, as well as by altering the pitch and speaking rate. The method is deterministic and partially reversible, and the changes are adjustable on a continuous scale. The method has been evaluated in terms of (i) ABX listening experiments, and (ii) automatic speaker verification and speech recognition. ABX experimental results indicate that the speaker identifiability among forced choice pairs reduced from over 90% to less than 70% through pseudonymization, and that de-pseudonymization was partially effective. An evaluation on the VoicePrivacy 2020 challenge data showed that the proposed approach performs better than the signal processing based baseline method that uses McAdams coefficient and performs slightly worse than the neural source filtering based baseline method. Further analysis showed that the proposed approach: (i) is comparable to the neural source filtering baseline based method in terms of phone posterior feature based objective intelligibility measure, (ii) preserves formant tracks better than the McAdams based method, and (iii) preserves paralinguistic aspects such as dysarthria in several speakers.

Reconsidering variability in child speech production

Child speech is highly variable between and within speakers. This pattern has traditionally been explained by a lack of motor routine and articulatory control in the developing speech-motor apparatus. But the assumption that child speech is inherently variable—and the result of motor development and control—may be premature. Recent work suggests that some child speech variation may instead be attributable to errors in measurement: children's high fundamental frequencies make many traditional acoustic measures, such as formant tracking, unreliable and variable. Additionally, child speech variation can often be explained by experiential factors, such as the size of the lexicon or the type and quantity of language exposure. This talk will report on these and other recent findings that suggest a need to reconsider the prevalence of, and reasons for, variability in young children's speech.

Formant Tracking Using Quasi-Closed Phase Forward-Backward Linear Prediction Analysis and Deep Neural Networks

Formant tracking is investigated in this study by using trackers based on dynamic programming (DP) and deep neural nets (DNNs). Using the DP approach, six formant estimation methods were first compared. The six methods include linear prediction (LP) algorithms, weighted LP algorithms and the recently developed quasi-closed phase forward-backward (QCP-FB) method. QCP-FB gave the best performance in the comparison. Therefore, a novel formant tracking approach, which combines benefits of deep learning and signal processing based on QCP-FB, was proposed. In this approach, the formants predicted by a DNN-based tracker from a speech frame are refined using the peaks of the all-pole spectrum computed by QCP-FB from the same frame. Results show that the proposed DNN-based tracker performed better both in detection rate and estimation error for the lowest three formants compared to reference formant trackers. Compared to the popular Wavesurfer, for example, the proposed tracker gave a reduction of 29%, 48% and 35% in the estimation error for the lowest three formants, respectively.

It's About Time: Minimizing Hardware and Software Latencies in Speech Research With Real-Time Auditory Feedback.

PurposeVarious aspects of speech production related to auditory–motor integration and learning have been examined through auditory feedback perturbation paradigms in which participants' acoustic speech output is experimentally altered and played back via earphones/headphones “in real time.” Scientific rigor requires high precision in determining and reporting the involved hardware and software latencies. Many reports in the literature, however, are not consistent with the minimum achievable latency for a given experimental setup. Here, we focus specifically on this methodological issue associated with implementing real-time auditory feedback perturbations, and we offer concrete suggestions for increased reproducibility in this particular line of work.MethodHardware and software latencies as well as total feedback loop latency were measured for formant perturbation studies with the Audapter software. Measurements were conducted for various audio interfaces, desktop and laptop computers, and audio drivers. An approach for lowering Audapter's software latency through nondefault parameter specification was also tested.ResultsOft-overlooked hardware-specific latencies were not negligible for some of the tested audio interfaces (adding up to 15 ms). Total feedback loop latencies (including both hardware and software latency) were also generally larger than claimed in the literature. Nondefault parameter values can improve Audapter's own processing latency without negative impact on formant tracking.ConclusionsAudio interface selection and software parameter optimization substantially affect total feedback loop latency. Thus, the actual total latency (hardware plus software) needs to be correctly measured and described in all published reports. Future speech research with “real-time” auditory feedback perturbations should increase scientific rigor by minimizing this latency.

Time-varying quasi-closed-phase analysis for accurate formant tracking in speech signals

In this paper, we propose a new method for the accurate estimation and tracking of formants in speech signals using time-varying quasi-closed-phase (TVQCP) analysis. Conventional formant tracking methods typically adopt a two-stage estimate-and-track strategy wherein an initial set of formant candidates are estimated using short-time analysis (e.g., 10–50 ms), followed by a tracking stage based on dynamic programming or a linear state-space model. One of the main disadvantages of these approaches is that the tracking stage, however good it may be, cannot improve upon the formant estimation accuracy of the first stage. The proposed TVQCP method provides a single-stage formant tracking that combines the estimation and tracking stages into one. TVQCP analysis combines three approaches to improve formant estimation and tracking: (1) it uses temporally weighted quasi-closed-phase analysis to derive closed-phase estimates of the vocal tract with reduced interference from the excitation source, (2) it increases the residual sparsity by using the $L_1$ optimization and (3) it uses time-varying linear prediction analysis over long time windows (e.g., 100–200 ms) to impose a continuity constraint on the vocal tract model and hence on the formant trajectories. Formant tracking experiments with a wide variety of synthetic and natural speech signals show that the proposed TVQCP method performs better than conventional and popular formant tracking tools, such as Wavesurfer and Praat (based on dynamic programming), the KARMA algorithm (based on Kalman filtering), and DeepFormants (based on deep neural networks trained in a supervised manner). Matlab scripts for the proposed method can be found at: https://github.com/njaygowda/ftrack

Prosodic Analysis and Enhancement of Dysarthric Speech

Dysarthria is a disorder that is caused in the nervous system. It is caused by damage in some parts of the brain such as cerebellum. Because of the damage in brain it causes weakness in muscles used for speech therefore it happens as in mumbled, slurred or slow speech that human and the machine find it difficult to understand such slurred speech. The Automatic Speech Recognizers which were designed for speech intelligibility perform poorly on dysarthric speech recognition. This paper focuses on the transformation of voice for dysarthria to enhance its intelligibility Formant tracking, pitch and energy estimation with durational cues from dysarthric speech facilitate the modification of these trajectories to more closely approximate the desired intelligible target speech. The transformation of the speech is done by using formant re-synthesis, pitch change and duration morphing. The results of such transformation results indicate that the transformation of the pitch and duration step enhances the intelligibility of dysarthric speech and make it easy to understand for humans and machines.

Novel acoustic measures of coarticulation reveal morphological planning in child speech

How do children plan and represent morphologically complex words throughout childhood? We employed coarticulation as a lexical planning metric in speakers of South Bolivian Quechua (SBQ), a highly agglutinating language with over 200 suffixes. 10 adult and 50 child (age 5–10) SBQ speakers completed a real word repetition task. V-C coarticulation was measured in the sequences [ap] and [am] in two contexts: within root morphemes (e.g., papa ‘potato’) or at morpheme boundaries (e.g., llama-pi ‘llama-locative’). Acoustic analysis of coarticulation using formant tracking is complicated in child speech because of children’s short vocal tracts and high f0. Following Gerosa et al. (in IEEE International Conference on Acoustics Speed and Signal Processing Proceedings (2006), Vol. 1, pp. 393–396), we quantified the degree and time course of coarticulation in Mel-frequency spectral differences. Results show that adults distinguished acoustically between phones more at morpheme boundaries than within morphemes, while children did not distinguish. This conclusion is further evidence that adults decompose words and plan them online. The lack of distinction in child speech, however, suggests that children may be accessing these complex words holistically, even in this highly agglutinating language.

Effects of focus and tone on vowel space in Chongming Chinese

The formant frequencies of vowels are prone to change according to contexts, but the effects of focus on vowel space are rarely studied. This study attempts to investigate how focus and tone may influence the vowel space in Chongming Chinese with a production experiment. Chongming Chinese is a Chinese dialect with a complex tonal system. Twelve syllables with the vowel /ae/ were chosen and the syllables spread over eight lexical tones (two level tones, four contour tones, and two checked tones). The target syllables were embedded in carrier sentences with different preceding and following syllables. Four focus conditions were manipulated: broad focus, initial focus, medial focus, and final focus. During the experiment, twelve participants were elicited with precursor questions to produce sentences with different focus locations. The frequencies of the first two formants (F1 and F2) were extracted over the middle 50 ms of the vowel interval for each target syllable. Formant tracks were calculated with the Burg algorithm in Praat. F1 and F2 values under each focus condition and tone will be compared. It is hypothesized that F1 and F2 will be increased under focus because focus might make the vowels more open and front. The formant frequencies of vowels are prone to change according to contexts, but the effects of focus on vowel space are rarely studied. This study attempts to investigate how focus and tone may influence the vowel space in Chongming Chinese with a production experiment. Chongming Chinese is a Chinese dialect with a complex tonal system. Twelve syllables with the vowel /ae/ were chosen and the syllables spread over eight lexical tones (two level tones, four contour tones, and two checked tones). The target syllables were embedded in carrier sentences with different preceding and following syllables. Four focus conditions were manipulated: broad focus, initial focus, medial focus, and final focus. During the experiment, twelve participants were elicited with precursor questions to produce sentences with different focus locations. The frequencies of the first two formants (F1 and F2) were extracted over the middle 50 ms of the vowel interval for each target syllable. Formant tracks were calculated with the B...

Automated formant tracking using reassigned spectrograms

The measurement and tracking of formant frequencies is usually accomplished with a linear prediction spectrum estimate, combined with a heuristic-based tracking algorithm. Despite the broad acceptance of this procedure over decades, there is a known bias towards the nearest harmonic, and formant estimates are less accurate for speech with higher fundamental frequency. Previous research has established the superior accuracy of formant measurement using the reassigned spectrogram; however, the measurements must be done by hand, severely limiting its usefulness. Here we present a technique that can locate formants automatically in a reassigned spectrogram. First, the reassigned spectrogram of the signal must be heavily “pruned” so that only a few points remain which highlight the detected frequency components, which are the presumed formants. Second, we apply a simple ridge-finding routine to the pruned spectrogram, to determine the formant tracks automatically. This process was tested on speech samples—/hVd/-words—by four speakers previously used in a comparison of 5 automatic algorithms and manually-measured reassigned spectrograms, and on breathy and falling tones in a speaker of Hmong. The results of this process are generally as good or better than the manual measurement when the optimal parameters for ridge-finding are used.

Transcription and forced alignment of the digital archive of southern speech

We describe transcription and forced alignment of the Digital Archive of Southern Speech (DASS), a project that will provide a large corpus of historical, semi-spontaneous Southern speech for acoustic analysis. 372 hours of recordings (64 interviews) comprise a subset of the Linguistic Atlas of the Gulf States, an extensive dialect study of 1121 speakers conducted across eight southern U.S. states from 1968 to 1983. Manual orthographic transcription of full DASS interviews is carried out according to in-house guidelines that ensure consistency across files and transcribers. Separate codes are used for the interviewee, interviewer, non-speech, overlapping, and unintelligible speech. Transcriber output is converted to Praat TextGrids using LaBB-CAT, a tool for maintaining large speech corpora. TextGrids containing only the interviewee’s speech are generated, and subjected to forced alignment by DARLA, which accommodates the levels of variation and noise in the DASS files with a high degree of success. Toward acoustic analysis, we evaluate three methods for vowel formant extraction: the native output of DARLA, a local implementation of FAVE-Extract, and a Praat-based extractor that incorporates separate formant tracks for different regions of the vowel space. We present this workflow of transcription and analysis to benefit other projects of similar size and scope.

Piecewise-Linear Frequency Shifting Algorithm for Frequency Resolution Enhancement in Digital Hearing Aids

In human hearing, frequency resolution is a term used to determine how well the ear can separate and distinguish two sounds that are close in frequency. This capability of breaking speech sounds into various frequency components plays a key role in processing and understanding speech information. In this paper, a piecewise-linear frequency shifting algorithm for digital hearing aids is proposed. The algorithm specifically aims at improving the frequency resolution capability. In the first step, frequency discrimination thresholds are processed by a computer testing software. Then, the input signal is parsed through the proposed piecewise-linear frequency shifting algorithm, which comprises of linearly stretching and compressing the frequency content at different frequency ranges. Experimental results showed that by using the proposed frequency shifting algorithm, the separation of formant tracks was increased in the stretching region and slightly squeezed in the adjacent compression region. Subjective assessment on six hearing-impaired persons with V-shaped audiograms demonstrated that nearly a 10% improvement of speech discrimination score was achieved for monosyllabic word lists tested in a quiet acoustic setting. In addition, the speech reception threshold was also improved by 2~8 dB when disyllabic word listswere tested in a noisy acoustic scenario.

The beauty in a beast: Minimising the effects of diverse recording quality on vowel formant measurements in sociophonetic real-time studies

Sociophonetic real-time studies of vowel variation and change rely on acoustic analyses of sound recordings made at different times, often using different equipment and data collection procedures. The circumstances of a recording are known to affect formant tracking and may therefore compromise the validity of conclusions about sound changes made on the basis of real-time data. In this paper, a traditional F1/F2-analysis using linear predictive coding (LPC) was applied to the vowels /i u a/ extracted from spontaneous speech corpora of Glaswegian vernacular, that were recorded in the 1970s and 2000s. We assessed the technical quality of each recording, concentrating on the average levels of noise and the properties of spectral balance, and showed that the corpus comprised of mixed quality data. A series of acoustic vowel analyses subsequently unveiled that formant measurements using LPC were sensitive to the technical specification of a recording, with variable magnitudes of the effects for vowels of different qualities. We evaluated the performance of three commonly used formant normalisation procedures (Lobanov, Nearey and Watt-Fabricius) as well as normalisations by a distance ratio metric and statistical estimation, and compared these results to raw Bark-scaled formant data, showing that some of the approaches could ameliorate the impact of technical issues better than the others. We discuss the implications of these results for sociophonetic research that aims to minimise extraneous influences on recorded speech data while unveiling gradual, potentially small-scale sound changes across decades.

Automatic selection of the number of poles for different gender and age groups in steady-state isolated vowels

Formant frequency estimation using a linear prediction (LPC) algorithm is based on the assumption of age- and gender-specific number of poles. However, when visually crosschecking the calculated formant frequencies along with a spectrogram, investigators often change the parameter because of a lack of correspondence. The misprediction is mainly due to a high variation within the calculated formant tracks, or tracks not matching with spectral peaks, possibly combined with an unexpected low or high number of occurring formants (e.g., formant merging, spurious formants). To solve the problem of changing the number of filter poles, we propose a new method which addresses the first aspect of the constancy of formant tracks. For a given vowel sound, the first three formant frequencies for three different settings (number of poles = 10, 12, and 14 for a frequency range 0-5.5 kHz) are calculated. The standard deviation of the formant tracks is used to find a Euclidean distance for three settings separately. The algorithm chooses the setting that produces least variability (minimum Euclidean distance) in steady-state vowel nuclei. We tested the method on vowel sounds of standard German /i, y, e, ø, ɛ, a, o, u/ produced by 14 men, 14 women, and 8 children.

Speech sound naturalness alters compensation in response to transformed auditory feedback

Articulatory compensations in response to real-time formant perturbation have revealed that auditory feedback plays an important role in speech production. However, these compensatory responses were at most 40% for formant shifts and varied depending on vowel type and subjects. Although previous formant perturbation studies have been done using linear predictive coding (LPC), it is known that the estimation accuracy for low vowels and female speech would be degraded due to a glottal source-vocal tract interaction. To improve the accuracy, we have developed a real-time robust formant tracking system using phase equalization-based autoregressive exogenous (PEAR) model which utilizes the glottal source signals measured by electroglottography. In this study, we compared compensatory responses to real-time formant perturbation using PEAR and LPC. Eleven Japanese subjects (seven females) read a Japanese mora (/hi/ or /he/) with headphones. The first two formant frequencies were altered. Results showed that compensatory responses using PEAR were significantly larger than LPC. Moreover, naturalness of altered speech sounds was improved by PEAR. This indicates that improving speech sound naturalness by PEAR led to larger compensatory responses. Therefore, our system would be useful to understand the auditory feedback mechanisms in more detail.

Articulator movement contributions to formant trajectories in diphthongs

Our purpose was to learn about the relative contributions of lip aperture and protrusion as well as tongue height and advancement to the trajectories of F1 and F2 during diphthongs produced by 20 speakers. Audio and kinematic signals were recorded with an NDI Wave system. Audio segments were analyzed with PRAAT to extract the F1 and F2 histories during the diphthongs. These were time-aligned with the kinematic records in MATLAB. Because time-series data violate the assumptions of traditional correlation and regression analyses, novel methods were developed to represent the relative contributions of individual articulator movements to F1 and F2 changes. All movement and formant tracks during diphthong transitions were converted to z-scores and overlaid on a common time scale. Absolute difference scores between kinematic (predictor) variables and acoustic (dependent) variables were summed along each plot to reflect the strength of the contribution of each movement to the acoustic outcome. The lowest difference sums reflected the closest alignment between a movement and a formant track. Results revealed differences in the relative contributions of the tongue and lips for each diphthong, suggesting that formant patterns during diphthongs may be difficult to interpret in a straightforward way in terms of articulator movement.