Articulatory Trajectories Research Articles

Decode, Move and Speak! Self-supervised Learning of Speech Units, Gestures, and Sound Relationships Using Vocal Imitation

Abstract Speech learning encompasses mastering a complex motor system to produce speech sounds from articulatory gestures while simultaneously uncovering discrete units that provide entry to the linguistic system. Remarkably, children acquire these associations between speech sounds, articulatory gestures, and linguistic units in a weakly supervised manner, without the need for explicit labeling of auditory inputs or access to target articulatory gestures. This study uses self-supervised deep learning to investigate the respective roles of sounds, gestures, and linguistic units in speech acquisition and control. In a first experiment, we analyzed the quantized representations learned by vector-quantized variational autoencoders (VQ-VAE) from ground truth acoustic and articulatory data using ABX tests. We show an interesting complementarity between acoustic and articulatory modalities that may help in the discovery of phonemes. In a second experiment, we introduce a computational agent that repeats auditory speech inputs by controlling a virtual vocal apparatus. This agent integrates an articulatory synthesizer capable of reproducing diverse speech stimuli from interpretable parameters, along with two internal models implementing the articulatory-to-acoustic (forward) and acoustic-to-articulatory (inverse) mapping, respectively. Additionally, two inductive biases are used to regularize the ill-posed acoustic-to-articulatory inverse mapping. In line with the first experiment, we explore the complementarity between the auditory input and the articulatory parameters inferred by the agent. We also evaluate the impact of discretizing auditory inputs using VQ-VAE. While the majority of the agent’s productions are intelligible (according to perceptual evaluations), our analysis highlights inconsistencies in the underlying articulatory trajectories. In particular, we show that the agent’s productions only partially reproduce the complementarity between the auditory and articulatory modalities observed in humans.

Is the Articulatory Trajectory of Changing Syllables Important for Achieving Higher Syllable Rates Compared to Repeated Syllables?

The production of speech-like sequences composed of varying syllables has been reported to achieve higher syllable rates than the production of repeated syllables (commonly designed as sequential motion rate [SMR] and alternating motion rate [AMR] tasks, respectively). The faster rate for SMR relative to AMR sequences is explained by different interpretative hypotheses, which remain empirically unexplored. In the present study, we aimed to investigate whether the high syllable rates of SMR sequences are due to the specific sequences used in most studies that involve front-to-back articulatory movements. Syllable rates of SMR sequences composed of different articulatory trajectories (i.e., labial-alveolar-velar [/pateko/], alveolar-velar-labial [/tekopa/], and velar-labial-alveolar [/kopate/]) were compared with those of the AMR sequences /papapa/, /tetete/, and /kokoko/ in 20 participants. The results show higher syllable rates for each of the three SMR sequences as compared to AMR, suggesting that the trajectory of the articulatory movements in the sequential sequences is not the key to achieving higher syllable rates. The syllable rate advantage for SMR over AMR sequences is not explained by the articulatory trajectories included in the former task, indicating that the front-to-back movements generally included in SMR sequences (i.e., /pataka/) are not decisive in achieving a higher syllable rate.

Overt speech decoding from cortical activity: a comparison of different linear methods.

Speech BCIs aim at reconstructing speech in real time from ongoing cortical activity. Ideal BCIs would need to reconstruct speech audio signal frame by frame on a millisecond-timescale. Such approaches require fast computation. In this respect, linear decoder are good candidates and have been widely used in motor BCIs. Yet, they have been very seldomly studied for speech reconstruction, and never for reconstruction of articulatory movements from intracranial activity. Here, we compared vanilla linear regression, ridge-regularized linear regressions, and partial least squares regressions for offline decoding of overt speech from cortical activity. Two decoding paradigms were investigated: (1) direct decoding of acoustic vocoder features of speech, and (2) indirect decoding of vocoder features through an intermediate articulatory representation chained with a real-time-compatible DNN-based articulatory-to-acoustic synthesizer. Participant's articulatory trajectories were estimated from an electromagnetic-articulography dataset using dynamic time warping. The accuracy of the decoders was evaluated by computing correlations between original and reconstructed features. We found that similar performance was achieved by all linear methods well above chance levels, albeit without reaching intelligibility. Direct and indirect methods achieved comparable performance, with an advantage for direct decoding. Future work will address the development of an improved neural speech decoder compatible with fast frame-by-frame speech reconstruction from ongoing activity at a millisecond timescale.

Modeling trajectories of human speech articulators using general Tau theory

This paper presents an application of general Tau theory to the modeling and analysis of articulatory trajectories in speech. We evaluated the model using electromagnetic articulometry data from 12 native speakers of English reading a common text, where trajectories of the following sensors were fitted: lower and upper lips, jaw, and three tongue sensors. Additionally, we analyzed trajectories of the lip aperture signal. Our experiments show that the general Tau theory model gives a better fit than existing (i) methods based on critically damped oscillators, and (ii) a method based on sequential target approximation. These findings support the hypothesis of Tau-guided movements of articulators during speech production. In the second part of the paper, our Tau theory analysis shows that articulatory movements follow similar velocity profile distributions across speakers. In particular, the value of the shape parameter κ of the Tau theory equation is identically distributed across speakers, following a unimodal distribution. The statistical mode of the distribution corresponds to the value of κ that generates a symmetric velocity profile. The analysis of the statistical distribution of κ values also reveals that its variance decreases when greater articulatory effort is required, such that produced articulatory effort remains close to that predicted by the theoretical minimal cost function based on forces acting on the moving articulator. This provides new evidence that articulatory effort is optimized during speech production.

Modeling speaker-specific vocal tract kinematics from gestural scores

The theory of Task Dynamics provides a method of predicting articulatory kinematics from a discrete phonologically-relevant representation (“gestural score”). However, because the implementations of that model (e.g., Nam et al., 2004) have generally used a simplified articulatory geometry (Mermelstein et al., 1981) whose forward model (from articulator to constriction coordinates) can be analytically derived, quantitative predictions of the model for individual human vocal tracts have not been possible. Recently, methods of deriving individual speaker forward models from real-time MRI data have been developed (Sorensen et al., 2019). This has further allowed development of task dynamic models for individual speakers, which make quantitative predictions. Thus far, however, these models (Alexander et al., 2019) could only synthesize limited types of utterances due to their inability to model temporally overlapping gestures. An updated implementation is presented, which can accommodate overlapping gestures and incorporates an optimization loop to improve the fit of modeled articulatory trajectories to the observed ones. Using an analysis-by-synthesis approach, the updated implementation can be utilized: (1) to refine the hypothesized speaker-general gestural parameters (target, stiffness) for individual speakers; (2) to test different degrees of temporal overlapping among multiple gestures such as a CCVC syllable. [Work supported by NSF, Grant 1908865.]

Using high-order derivatives of articulatory trajectories to identify gesture onsets

Articulatory trajectories are useful for measuring speech properties, but come with complications: a given trajectory may be influenced by multiple articulators and/or multiple overlapping gestures. Standard procedure is to identify landmarks from the trajectory and its velocity. I present a new procedure: higher order derivatives can be used to identify gesture onsets. High-order derivatives are generally associated with signal noise, and therefore not considered to be meaningful. More broadly, however, they reflect the jaggedness or un-smoothness of a signal, which can derive from sources other than random noise. I demonstrate that such jaggedness at gesture onsets is both predicted by the task dynamic model of speech production (Saltzman & Munhall 1989) and observed in real EMA data. Articulator trajectories generated with the Task Dynamics Application (TADA) Matlab implementation of this model (Nam & Goldstein 2004) show a pattern of increasing activity around changes in gestural activation and decreasing activity elsewhere as successive derivatives are taken. This was confirmed in EMA data on glides, demonstrating not only that the effect is real, but that it can be measurable above other noise. Activation noise can be isolated with targeted filtering, and may yield new insights into speech motor control.

Analyzing dynamic phonetic data using generalized additive mixed modeling: A tutorial focusing on articulatory differences between L1 and L2 speakers of English

In phonetics, many datasets are encountered which deal with dynamic data collected over time. Examples include diphthongal formant trajectories and articulator trajectories observed using electromagnetic articulography. Traditional approaches for analyzing this type of data generally aggregate data over a certain timespan, or only include measurements at a fixed time point (e.g., formant measurements at the midpoint of a vowel). This paper discusses generalized additive modeling, a non-linear regression method which does not require aggregation or the pre-selection of a fixed time point. Instead, the method is able to identify general patterns over dynamically varying data, while simultaneously accounting for subject and item-related variability. An advantage of this approach is that patterns may be discovered which are hidden when data is aggregated or when a single time point is selected. A corresponding disadvantage is that these analyses are generally more time consuming and complex. This tutorial aims to overcome this disadvantage by providing a hands-on introduction to generalized additive modeling using articulatory trajectories from L1 and L2 speakers of English within the freely available R environment. All data and R code is made available to reproduce the analysis presented in this paper.

The EMA Study on the Inter-individual Variability and Differences in Articulation between Polish Oral and Nasalised Vowels

The electromagnetic articulography (EMA) is a relatively exact and efficient method used in study on speech production physiology. It allows to precisely estimate movement trajectories of speech articulators like tongue, lips, jaw etc. by tracking position of sensors fixed to the articulators. This paper presents results of EMA research on Polish oral and nasalised vowels in orthography represented by the graphemes <e>, <o>, <ę>, <ą>. Inter-individual variability of tongue and lips position in X-axis direction during realization of the same phoneme has been estimated. Differences between oral and nasalised vowels in terms of movement of articulators in X-axis direction have been assessed too.

Quantifying kinematic aspects of reduction in a contrasting rate production task

Electromagnetic articulometry (EMA) was used to record the 720 phonetically balanced Harvard sentences (IEEE, 1969) from multiple speakers at normal and fast production rates. Participants produced each sentence twice, first at their preferred “normal” speaking rate followed by a “fast” production (for a subset of the sentences two normal rate productions were elicited). They were instructed to produce the “fast” repetition as quickly as possible without making errors. EMA trajectories were obtained at 100 Hz from sensors placed on the tongue, lips, and mandible, corrected for head movement and aligned to the occlusal plane. Synchronized audio was recorded at 22050 Hz. Comparison of normal to fast acoustic durations for paired utterances showed a mean 67% length reduction, and assessed using Mermelstein's method (1975), two fewer syllables on average. A comparison of inflections in vertical jaw movement between paired utterances showed an average of 2.3 fewer syllables. Cross-recurrence analysis of distance maps computed on paired sensor trajectories comparing corresponding normal:normal to normal:fast utterances showed systematically lower determinism and entropy for the cross-rate comparisons, indicating that rate effects on articulator trajectories are not uniform. Examples of rate-related differences in gestural overlap that might account for these differences in predictability will be presented. [Work supported by NSF.]

Generalized additive modeling to analyze dynamic phonetic data: a tutorial focusing on articulatory differences between L1 and L2 speakers of English

In phonetics many data sets are encountered which deal with dynamic data collected over time. Examples include dynamic formant trajectories, or tongue position trajectories obtained via electromagnetic articulography. Traditional approaches to analyzing this type of data generally aggregate data over a certain timespan, or only include measurements at a certain fixed time point (e.g., formant measurements at the midpoint of a vowel). While these types of analyses are relatively easy to understand and conduct, I argue in this paper for a more elaborate approach, generalized additive modeling, which is able to take into account the non-linear patterns over time while simultaneously taking into account subject and item-related variability. I will illustrate its use in this tutorial using articulatory trajectories from L1 and L2 speakers of English. All data and R code is made available for readers to replicate the analysis presented in this paper.

Investigating dialectal differences using articulography

The present study uses electromagnetic articulography, by which the position of tongue and lips during speech is measured, for the study of dialect variation. By using generalized additive modeling to analyze the articulatory trajectories, we are able to reliably detect aggregate group differences, while simultaneously taking into account the individual variation of dozens of speakers. Our results show that two Dutch dialects show clear differences in their articulatory settings, with generally a more anterior tongue position in the dialect from Ubbergen in the southern half of the Netherlands than in the dialect of Ter Apel in the northern half of the Netherlands. A comparison with formant-based acoustic measurements further reveals that articulography is able to reveal interesting structural articulatory differences between dialects which are not visible when only focusing on the acoustic signal.

Durational and spectral differences in Thai diphthongs and final glides

Acoustic analysis was conducted to compare Thai monophthongs /i, ii, Ɯ, ƜƜ, u, uu, a, aa/, diphthongs /ia, Ɯa, ua/, and vowel-to-glides (vowel + /j/or /w/) in terms of duration, formant frequency, and spectral rate of change (TLroc). Preliminary results from multiple repetitions of 30 target monosyllabic words (from 3 males) show that durational values of long monophthongs are approximately two times as long as the short counterparts. The diphthong onsets (/i/, /Ɯ/, /u/) and offsets (/a/) as part of diphthongs appear to be more centralized than when they occur in monophthongs. Syllables with final glides appear to be longer than those with other final consonants, reducing length differences between syllables with short and long vowels. Interestingly, not only that Thai diphthongs and vowel-to-glides differ in their articulatory trajectories, but they appear to differ in terms of duration and TLroc values. Average duration of diphthongs is shorter than that of vowel-to-glides. TLroc of diphthongs is on average higher than that of long-vowel-to-glides, but lower than short-vowel-to-glides. Taken together, our preliminary results suggest and agree with the notion that transition of diphthongs is usually slower and more gradual than that of vowel-to-glides.

Articulation Artifacts During Overt Language Production in Event-Related Brain Potentials: Description and Correction

Overt articulation produces strong artifacts in the electroencephalogram and in event-related potentials (ERPs), posing a serious problem for investigating language production with these variables. Here we describe the properties of articulation-related artifacts and propose a novel correction procedure. Experiment 1 co-recorded ERPs and trajectories of the articulators with an electromagnetic articulograph from a single participant. The generalization of the findings from the single participant to standard picture naming was investigated in Experiment 2. Both experiments provided evidence that articulation-induced artifacts may start up to 300ms or more prior to voice onset or voice key onset-depending on the specific measure; they are highly similar in topography across many different phoneme patterns and differ mainly in their time course and amplitude. ERPs were separated from articulation-related artifacts with residue iteration decomposition (RIDE). After obtaining the artifact-free ERPs, their correlations with the articulatory trajectories dropped near to zero. Artifact removal with independent component analysis was less successful; while correlations with the articulatory movements remained substantial, early components prior to voice onset were attenuated in reconstructed ERPs. These findings offer new insights into the nature of articulation artifacts; together with RIDE as method for artifact removal the present report offers a fresh perspective for ERP studies requiring overt articulation.

Speaker-Adaptive Acoustic-Articulatory Inversion Using Cascaded Gaussian Mixture Regression

This paper addresses the adaptation of an acoustic-articulatory model of a reference speaker to the voice of another speaker, using a limited amount of audio-only data. In the context of pronunciation training, a virtual talking head displaying the internal speech articulators (e.g., the tongue) could be automatically animated by means of such a model using only the speaker’s voice. In this study, the articulatory-acoustic relationship of the reference speaker is modeled by a gaussian mixture model (GMM). To address the speaker adaptation problem, we propose a new framework called cascaded Gaussian mixture regression (C-GMR), and derive two implementations. The first one, referred to as Split-C-GMR, is a straightforward chaining of two distinct GMRs: one mapping the acoustic features of the source speaker into the acoustic space of the reference speaker, and the other estimating the articulatory trajectories with the reference model. In the second implementation, referred to as Integrated-C-GMR, the two mapping steps are tied together in a single probabilistic model. For this latter model, we present the full derivation of the exact EM training algorithm, that explicitly exploits the missing data methodology of machine learning. Other adaptation schemes based on maximum-a posteriori (MAP), maximum likelihood linear regression (MLLR) and direct cross-speaker acoustic-to-articulatory GMR are also investigated. Experiments conducted on two speakers for different amount of adaptation data show the interest of the proposed C-GMR techniques.

Speaker verification based on the fusion of speech acoustics and inverted articulatory signals

We propose a practical, feature-level and score-level fusion approach by combining acoustic and estimated articulatory information for both text independent and text dependent speaker verification. From a practical point of view, we study how to improve speaker verification performance by combining dynamic articulatory information with the conventional acoustic features. On text independent speaker verification, we find that concatenating articulatory features obtained from measured speech production data with conventional Mel-frequency cepstral coefficients (MFCCs) improves the performance dramatically. However, since directly measuring articulatory data is not feasible in many real world applications, we also experiment with estimated articulatory features obtained through acoustic-to-articulatory inversion. We explore both feature level and score level fusion methods and find that the overall system performance is significantly enhanced even with estimated articulatory features. Such a performance boost could be due to the inter-speaker variation information embedded in the estimated articulatory features. Since the dynamics of articulation contain important information, we included inverted articulatory trajectories in text dependent speaker verification. We demonstrate that the articulatory constraints introduced by inverted articulatory features help to reject wrong password trials and improve the performance after score level fusion. We evaluate the proposed methods on the X-ray Microbeam database and the RSR 2015 database, respectively, for the aforementioned two tasks. Experimental results show that we achieve more than 15% relative equal error rate reduction for both speaker verification tasks.

Dual electromagnetic articulometer observation of head movements coordinated with articulatory gestures for interacting talkers in synchronized speech tasks

Previous research has demonstrated that speakers readily entrain to one another in synchronized speech tasks (e.g., Cummins 2002, Vatikiotis-Bateson et al. 2014, Natif et al. 2014), but the mixture of auditory and visual cues they use to achieve such alignment remains unclear. In this work, we extend the dual-EMA paradigm of Tiede et al. (2012) to observe the speech and coordinated head movements of speaker pairs interacting face-to-face during synchronized production in three experimental tasks: the “Grandfather” passage, repetition of short rhythmically consistent sentences, and competing alternating word pairs (e.g., “topper-cop” vs. “copper-top"). The first task was read with no eye contact, the second was read and then produced with eye contact, and the third required continuous eye contact. Head movement was characterized using the tracked position of the upper incisor reference sensor. Prosodic prominence was identified using F0 and amplitude contours from the acoustics, and gestural stiffness on articulator trajectories. Preliminary results show that frequency and amplitude of synchronized head movement increased with task eye contact, and that this was coordinated systematically with both acoustic and articulatory prosodic prominence. [Work supported by NIH.]

Data driven articulatory synthesis with deep neural networks

The conventional approach for data-driven articulatory synthesis consists of modeling the joint acoustic-articulatory distribution with a Gaussian mixture model (GMM), followed by a post-processing step that optimizes the resulting acoustic trajectories. This final step can significantly improve the accuracy of the GMM frame-by-frame mapping but is computationally intensive and requires that the entire utterance be synthesized beforehand, making it unsuited for real-time synthesis. To address this issue, we present a deep neural network (DNN) articulatory synthesizer that uses a tapped-delay input line, allowing the model to capture context information in the articulatory trajectory without the need for post-processing. We characterize the DNN as a function of the context size and number of hidden layers, and compare it against two GMM articulatory synthesizers, a baseline model that performs a simple frame-by-frame mapping, and a second model that also performs trajectory optimization. Our results show that a DNN with a 60-ms context window and two 512-neuron hidden layers can synthesize speech at four times the frame rate – comparable to frame-by-frame mappings, while improving the accuracy of trajectory optimization (a 9.8% reduction in Mel Cepstral distortion). Subjective evaluation through pairwise listening tests also shows a strong preference toward the DNN articulatory synthesizer when compared to GMM trajectory optimization.

On smoothing articulatory trajectories obtained from Gaussian mixture model based acoustic-to-articulatory inversion

It is well-known that the performance of acoustic-to-articulatory inversion improves by smoothing the articulatory trajectories estimated using Gaussian mixture model (GMM) mapping (denoted by GMM + Smoothing). GMM + Smoothing also provides similar performance with GMM mapping using dynamic features, which integrates smoothing directly in the mapping criterion. Due to the separation between smoothing and mapping, what objective criterion GMM + Smoothing optimizes remains unclear. In this work a new integrated smoothness criterion, the smoothed-GMM (SGMM), is proposed. GMM + Smoothing is shown, both analytically and experimentally, to be identical to the asymptotic solution of SGMM suggesting GMM + Smoothing to be a near optimal solution of SGMM.

An episodic memory-based solution for the acoustic-to-articulatory inversion problem

This paper presents an acoustic-to-articulatory inversion method based on an episodic memory. An episodic memory is an interesting model for two reasons. First, it does not rely on any assumptions about the mapping function but rather it relies on real synchronized acoustic and articulatory data streams. Second, the memory inherently represents the real articulatory dynamics as observed. It is argued that the computational models of episodic memory, as they are usually designed, cannot provide a satisfying solution for the acoustic-to-articulatory inversion problem due to the insufficient quantity of training data. Therefore, an episodic memory is proposed, called generative episodic memory (G-Mem), which is able to produce articulatory trajectories that do not belong to the set of episodes the memory is based on. The generative episodic memory is evaluated using two electromagnetic articulography corpora: one for English and one for French. Comparisons with a codebook-based method and with a classical episodic memory (which is termed concatenative episodic memory) are presented in order to evaluate the proposed generative episodic memory in terms of both its modeling of articulatory dynamics and its generalization capabilities. The results show the effectiveness of the method where an overall root-mean-square error of 1.65 mm and a correlation of 0.71 are obtained for the G-Mem method. They are comparable to those of methods recently proposed.

Detailed study of articulatory kinematics of critical articulators and dependent articulators of emotional speech

This study investigates the articulatory kinematics of critical articulators and dependent articulators as a function of emotion. Our hypothesis is that critical articulators and dependent articulators are utilized differently for achieving distinctive emotion goals that overlay linguistic goals. For example, speakers may use variability of dependent articulators distinctively to that of critical articulators in achieving emotion goals. Distinctive articulatory movements for different emotions have been observed (Lee etal. 2005). This study uses a database of three speakers (2 female and 1 male) collected with electromagnetic articulography to collect kinematic information. Articulatory trajectories are aligned by dynamic time warping. Linguistically identical syllable-level segments are analyzed based on detailed aspects of articulatory movements (e.g., position, velocity and phase), after sampling with a 20-ms window and 10-ms shifting. The emotion-specific patterns that emerge for critical (i.e., goal-directed) articulators are compared to those for articulators that are not under overt control for a particular phone. [Work supported by NIH and NSF Grants.]