Articulatory Model Research Articles

Phoneme sequence-to-speech conversion in dynamic phonological models: A survey

This survey is devoted to dynamic models that model how a discrete phoneme sequence becomes converted to the corresponding continuous flow of articulations. The key concepts of modern dynamic models are discussed: articulatory model, articulatory parameters, goals and gestures, pronunciation effort economy principle, etc. The results of research conducted by specialists from Haskins Laboratories (articulatory phonology, task dynamic model), as well as by Japanese scientists (mostly from Waseda University) are presented. The survey is illustrated by both model examples and real articulatory X-ray microbeam measurements.

A shared model-based linguistic space for transmitting our thoughts from brain to brain in natural conversations

Effective communication hinges on a mutual understanding of word meaning in different contexts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We developed a model-based coupling framework that aligns brain activity in both speaker and listener to a shared embedding space from a large language model (LLM). The context-sensitive LLM embeddings allow us to track the exchange of linguistic information, word by word, from one brain to another in natural conversations. Linguistic content emerges in the speaker's brain before word articulation and rapidly re-emerges in the listener's brain after word articulation. The contextual embeddings better capture word-by-word neural alignment between speaker and listener than syntactic and articulatory models. Our findings indicate that the contextual embeddings learned by LLMs can serve as an explicit numerical model of the shared, context-rich meaning space humans use to communicate their thoughts to one another.

Bilinguals from Larynx to Lips: Exploring Bilingual Articulatory Strategies with Anatomic MRI Data.

The goal of this article is to illustrate the use of MRI for exploring bi- and multi-lingual articulatory strategies. One male and one female speaker recorded sets of static midsagittal MRIs of the whole vocal tract, producing vowels as well as consonants in various vowel contexts in either the male's two or the female's three languages. Both speakers were native speakers of English (American and Australian English, respectively), and both were fluent L2 speakers of French. In addition, the female speaker was a heritage speaker of Croatian. Articulatory contours extracted from the MRIs were subsequently used at three progressively more compact and abstract levels of analysis. (1) Direct comparison of overlaid contours was used to assess whether phones analogous across L1 and L2 are similar or dissimilar, both overall and in specific vocal tract regions. (2) Consonant contour variability along the vocal tract due to vowel context was determined using dispersion ellipses and used to explore the variable resistance to coarticulation for non-analogous rhotics and analogous laterals in Australian, French, and Croatian. (3) Articulatory modeling was used to focus on specific articulatory gestures (tongue position and shape, lip protrusion, laryngeal height, etc.) and then to explore the articulatory strategies in the speakers' interlanguages for production of the French front rounded vowel series. This revealed that the Australian and American speakers used different strategies to produce the non-analogous French vowel series. We conclude that MRI-based articulatory data constitute a very rich and underused source of information that amply deserves applications to the study of L2 articulation and bilingual and multi-lingual speech.

An MRI-based articulatory analysis of the Kannada dental-retroflex contrast

AbstractThis paper investigates the production of dental and retroflex stops, fricatives, nasals, and laterals in the Dravidian language Kannada. This is done using articulatory contours extracted from an extensive midsagittal MRI corpus of two female Kannada speakers’ static vocal tract postures intended to capture key aspects of phonemic articulations. Articulatory modelling was used to determine a set of components responsible for the implementation of place and manner contrasts (/t̪ s̪ n̪ l̪/ vs. /ʈ ʂ ɳ ɭ/). These components included both lingual and non-lingual articulatory parameters. Constriction location and length were also determined based on articulatory contours. The results showed that the two speakers produced non-fricative retroflexes with a retracted tongue tip making a constriction behind the alveolar ridge and a characteristic convex tongue shape, yet without a retraction of the posterior portion of the tongue. Apart from the lingual parameters, place differences were also manifested by the vertical position of the larynx (lower for retroflexes). The realisation of the place contrast in sibilant fricatives was different, as /ʂ/ appeared to be produced by both speakers with a laminal alveolopalatal constriction. Manner differences were captured by various non-lingual parameters, yet being also manifested in constriction locations (more anterior for stops). These findings are discussed in the context of previous descriptive and articulatory accounts of dental-retroflex contrasts.

Computer-Implemented Articulatory Models for Speech Production: A Review.

Modeling speech production and speech articulation is still an evolving research topic. Some current core questions are: What is the underlying (neural) organization for controlling speech articulation? How to model speech articulators like lips and tongue and their movements in an efficient but also biologically realistic way? How to develop high-quality articulatory-acoustic models leading to high-quality articulatory speech synthesis? Thus, on the one hand computer-modeling will help us to unfold underlying biological as well as acoustic-articulatory concepts of speech production and on the other hand further modeling efforts will help us to reach the goal of high-quality articulatory-acoustic speech synthesis based on more detailed knowledge on vocal tract acoustics and speech articulation. Currently, articulatory models are not able to reach the quality level of corpus-based speech synthesis. Moreover, biomechanical and neuromuscular based approaches are complex and still not usable for sentence-level speech synthesis. This paper lists many computer-implemented articulatory models and provides criteria for dividing articulatory models in different categories. A recent major research question, i.e., how to control articulatory models in a neurobiologically adequate manner is discussed in detail. It can be concluded that there is a strong need to further developing articulatory-acoustic models in order to test quantitative neurobiologically based control concepts for speech articulation as well as to uncover the remaining details in human articulatory and acoustic signal generation. Furthermore, these efforts may help us to approach the goal of establishing high-quality articulatory-acoustic as well as neurobiologically grounded speech synthesis.

Temporal Convolution Network Based Joint Optimization of Acoustic-to-Articulatory Inversion

Articulatory features are proved to be efficient in the area of speech recognition and speech synthesis. However, acquiring articulatory features has always been a difficult research hotspot. A lightweight and accurate articulatory model is of significant meaning. In this study, we propose a novel temporal convolution network-based acoustic-to-articulatory inversion system. The acoustic feature is converted into a high-dimensional hidden space feature map through temporal convolution with frame-level feature correlations taken into account. Meanwhile, we construct a two-part target function combining prediction’s Root Mean Square Error (RMSE) and the sequences’ Pearson Correlation Coefficient (PCC) to jointly optimize the performance of the specific inversion model from both aspects. We also further conducted an analysis on the impact of the weight between the two parts on the final performance of the inversion model. Extensive experiments have shown that our, temporal convolution networks (TCN) model outperformed the Bi-derectional Long Short Term Memory model by 1.18 mm in RMSE and 0.845 in PCC with 14 model parameters when optimizing evenly with RMSE and PCC aspects.

Articulatory characterization of the alveolar sibilant fricatives of Basque: Part one

An articulatory analysis of the two alveolar sibilant fricatives of Basque (<z> and <s>) is presented, based on fifteen collections of 2D and 3D magnetic resonance images. By applying a set of phonetic, qualitative and quantitative parameters, two articulatory models for <z> and three for <s> are identified, and the procedures used to distinguish between them are examined. The main models are: dorsal denti-alveolar for <z> and apico-alveolar for <s>. Due to its length, the work is presented in two separate parts.

Motor representations underlie the reading of unfamiliar letter combinations

Silent reading is a cognitive operation that produces verbal content with no vocal output. One relevant question is the extent to which this verbal content is processed as overt speech in the brain. To address this, we acquired sound, eye trajectories and lips’ dynamics during the reading of consonant-consonant-vowel (CCV) combinations which are infrequent in the language. We found that the duration of the first fixations on the CCVs during silent reading correlate with the duration of the transitions between consonants when the CCVs are actually uttered. With the aid of an articulatory model of the vocal system, we show that transitions measure the articulatory effort required to produce the CCVs. This means that first fixations during silent reading are lengthened when the CCVs require a greater laryngeal and/or articulatory effort to be pronounced. Our results support that a speech motor code is used for the recognition of infrequent text strings during silent reading.

Timing Evidence for Symbolic Phonological Representations and Phonology-Extrinsic Timing in Speech Production.

The goals of this paper are (1) to discuss the key features of existing articulatory models of speech production that govern their approaches to timing, along with advantages and disadvantages of each, and (2) to evaluate these features in terms of several pieces of evidence from both the speech and nonspeech motor control literature. This evidence includes greater timing precision at movement endpoints compared to other parts of movements, suggesting the separate control of the timing of movement endpoints compared to other parts of movement. This endpoint timing precision challenges models in which all parts of a movement trajectory are controlled by the same equation of motion, but supports models in which (a) abstract, symbolic phonological representations map onto spatial and temporal characteristics of the part(s) of movement most closely related to the goal of producing a planned set of acoustic cues to signal the phonological contrast (often the endpoint), (b) movements are coordinated primarily based on the goal-related part of movement, and (c) speakers give priority to the accurate implementation of the part(s) of movement most closely related to the phonological goals. In addition, this paper presents three types of evidence for phonology-extrinsic timing, suggesting that surface duration requirements are represented during speech production. Phonology-extrinsic timing is also supported by greater timing variability for repetitions of longer intervals, assumed to be due to noise in a general-purpose (and phonology-extrinsic) timekeeping process. The evidence appears to be incompatible with models that have a unified Phonology/Phonetics Component, that do not represent the surface timing of phonetic events, and do not represent, specify and track timing by general-purpose timekeeping mechanisms. Taken together, this evidence supports an alternative approach to modeling speech production that is based on symbolic phonological representations and general-purpose, phonology-extrinsic, timekeeping mechanisms, rather than on spatio-temporal phonological representations and phonology-specific timing mechanisms. Thus, the evidence suggests that models in that alternative framework should be developed, so they can be tested with the same rigor as have models based on spatio-temporal phonological representations with phonology-intrinsic timing.

A modular architecture for articulatory synthesis from gestural specification.

This paper proposes a modular architecture for articulatory synthesis from a gestural specification comprising relatively simple models for the vocal tract, the glottis, aero-acoustics, and articulatory control. The vocal tract module combines a midsagittal statistical analysis articulatory model, derived by factor analysis of air-tissue boundaries in real-time magnetic resonance imaging data, with an αβ model for converting midsagittal section to area function specifications. The aero-acoustics and glottis models were based on a software implementation of classic work by Maeda. The articulatory control module uses dynamical systems, which implement articulatory gestures, to animate the statistical articulatory model, inspired by the task dynamics model. Results on synthesizing vowel-consonant-vowel sequences with plosive consonants, using models that were built on data from, and simulate the behavior of, two different speakers are presented.

Toward cross-speaker articulatory modeling

The recent increasing availability of comprehensive real-time MRI data of the vocal tract and concomitant progress in air-tissue boundary segmentation present novel opportunities for articulatory modeling. PCA-based articulatory models represent vocal tract configurations as weighted linear combinations of articulatory components that characterize vocal tract shaping patterns. Historically, most such models have been developed using data from a single speaker, and their direct application to data from multiple speakers may provide components that are not comparable across speakers. A technique that can address this issue is PARAFAC, which has been applied in the past to analyze data corresponding to tongue contour X-ray tracings of 10 English vowels from 5 speakers. PARAFAC introduces an additional level of weighting of the articulatory components that is constant for, and therefore characteristic of, each speaker. We revisited PARAFAC and ran a successful pilot on real-time MRI air tissue-boundaries of the entire vocal tract from 4 speakers, each uttering two repetitions of a set of 4 Shibboleth sentences, yielding 289 phones per speaker. Application on a much larger and diverse real-time MRI dataset already collected by our team will provide crucial progress toward true cross-speaker articulatory modeling. [Work supported by NIH and NSF.]

Toward a Theory of the Mechanism of High-Velocity, Low-Amplitude Technique: A Literature Review.

This review seeks to integrate the current literature to create a more unified and inclusive theory regarding the therapeutic mechanism of high-velocity, low-amplitude (HVLA) technique. The authors review the literature currently available regarding the physiologic effects of HVLA. The progression from an articulatory model to a neuromuscular one is discussed, and the body of work demonstrating that HVLA has a centralized mechanism of action, rather than just a local one, is described.

Characterization of inter-speaker articulatory variability: A two-level multi-speaker modelling approach based on MRI data.

Speech communication relies on articulatory and acoustic codes shared between speakers and listeners despite inter-individual differences in morphology and idiosyncratic articulatory strategies. This study addresses the long-standing problem of characterizing and modelling speaker-independent articulatory strategies and inter-speaker articulatory variability. It explores a multi-speaker modelling approach based on two levels: statistically-based linear articulatory models, which capture the speaker-specific articulatory variability on the one hand, are in turn controlled by a speaker model, which captures the inter-speaker variability on the other hand. A low dimensionality speaker model is obtained by taking advantage of the inter-speaker correlations between morphology and strategy. To validate this approach, contours of the vocal tract articulators were manually segmented on midsagittal MRI data recorded from 11 French speakers uttering 62 vowels and consonants. Using these contours, multi-speaker models with 14 articulatory components and two morphology and strategy components led to overall variance explanations of 66%-69% and root-mean-square errors of 0.36-0.38 cm obtained in leave-one-out procedure over the speakers. Results suggest that inter-speaker variability is more related to the morphology than to the idiosyncratic strategies and illustrate the adaptation of the articulatory components to the morphology.

Reference speaker selection for kinematic-independent acoustic-to-articulatory-inversion

This paper investigates the most effective reference speaker set for the Parallel Reference Speaker Weighting (PRSW) algorithm for kinematic-independent acoustic-to-articulatory inversion. To obtain the adaptation weights for estimating the articulatory model, different reference speaker accent types and quantities have been acquired. The reference speaker sets have been selected not only based on their performance in speaker-dependent kinematic-inversion but also based on the type of accent. The experiments have been conducted on parallel Acoustic-Articulatory data, the Marquette Electromagnetic Articulography corpus of Mandarin Accented English (EMA-MAE) consisting of 20 native English speakers and 19 native Mandarin speakers of English. A comparison is made between different types of target speakers and reference speakers with results indicating that the accuracy of the adapted model increases when we select balanced distributed accents of English and lower number of speakers. This paper investigates the most effective reference speaker set for the Parallel Reference Speaker Weighting (PRSW) algorithm for kinematic-independent acoustic-to-articulatory inversion. To obtain the adaptation weights for estimating the articulatory model, different reference speaker accent types and quantities have been acquired. The reference speaker sets have been selected not only based on their performance in speaker-dependent kinematic-inversion but also based on the type of accent. The experiments have been conducted on parallel Acoustic-Articulatory data, the Marquette Electromagnetic Articulography corpus of Mandarin Accented English (EMA-MAE) consisting of 20 native English speakers and 19 native Mandarin speakers of English. A comparison is made between different types of target speakers and reference speakers with results indicating that the accuracy of the adapted model increases when we select balanced distributed accents of English and lower number of speakers.

Recenzja monografii Piotr Rybki „Samogłoski w gwarach Górnego Śląska i polszczyźnie ogólnej. Porównanie z wykorzystaniem danych akustycznych i artykulacyjnych”

The review is devoted to the latest book written by Piotr Rybka, The Vowels in the Dialects of Upper Silesia and in the contemporary Polish language. Comparison with the Use of Acoustic and Articulation Data, published in 2018 as part of the work of the Institute of the Polish Language of the Polish Academy of Sciences. This is an experimental study in the field of dialectal phonetics. The author has developed acoustic and articulatory models that made it possible to compare features of the vowels in the speech of Upper Silesian residents with Polish standard pronunciation.

Automatic segmentation of speech articulators from real-time midsagittal MRI based on supervised learning

Speech production mechanisms can be characterized at a peripheral level by both their acoustic and articulatory traces along time. Researchers have thus developed very large efforts to measure articulation. Thanks to the spectacular progress accomplished in the last decade, real-time Magnetic Resonance Imaging (RT-MRI) offers nowadays the advantages of frame rates closer than before to those achieved by electromagnetic articulography or ultrasound echography while providing very detailed geometrical information about the whole vocal tract. RT-MRI has thus become inescapable for the study of speech articulators’ movements. However, making efficient use of large sets of images to characterize and model speech tasks implies the development of automatic methods to segment the articulators from these images with sufficient accuracy. The present article describes our approach to develop, based on supervised machine learning techniques, an automatic segmentation method that offers various useful features such as (1) capability of dealing with individual articulators independently, (2) ensuring hard palate, jaw and hyoid bone to be adequately tracked as rigid structures, (3) delivering contours for a full set of articulators, including the epiglottis and the back of the larynx, which partly reflects the vocal fold abduction / adduction state, (4) dealing more explicitly and thus more accurately with contact between articulators, and (5) reaching an accuracy better than one millimeter.The main contributions of this work are the following. We have recorded the first large database of high quality RT-MRI midsagittal images for a French speaker. We have manually segmented the main speech articulators (jaw, lips, tongue, velum, hyoid, larynx, etc.) for a small training set of about 60 images selected by hierarchical clustering to represent the whole corpus as faithfully as possible. We have used these data to train various image and contour models for developing automatic articulatory segmentation methods. The first method, based on Multiple Linear Regression, allows to predict the contour coordinates from the image pixel intensities with a Mean Sum of Distances (MSD) segmentation error over all articulators of 0.91 mm, computed with a Leave-One-Out Cross Validation procedure on the training set. Another method, based on Shape Particle Filtering, reaches an MSD error of 0.66 mm. Finally the modified version of Active Shape Models (mASM) explored in this study gives an MSD error of a mere 0.55 mm (0.68 mm for the tongue). These results demonstrate that this mASM approach performs better than state-of-the-art methods, though at the cost of the manual segmentation of the training set. The same method used on other MRI data leads to similar errors, which testifies to its robustness. The large quantity of contour data that can be obtained with this automatic segmentation method opens the way to various fruitful perspectives in speech: establishing more elaborate articulatory models, analyzing more finely coarticulation and articulatory variability or invariance, implementing machine learning methods for articulatory speaker normalization or adaptation, or illustrating adequate or prototypical articulatory gestures for application in the domains of speech therapy and of second language pronunciation training.

Articulatory modeling of human and non-human vocal production

This paper reviews two of the many questions that might be addressed by adapting articulatory models of vocal production across primate species. The first concerns vocal-fold shaping at phonation. The human folds have been modeled to have uniform cross-sectional shapes with round edges along the longitudinal axis, whereas MRI slices reveal an obvious vertical prominence of the mucosa near each vocal process. This shape, due to depressed vocal processes in adduction, somewhat resembles the pointed edges of the vocal membranes in nonhuman primates, which are incorporated in some models of chaotic phonation in those species. This suggests a model of vocal-fold vibration propagating back to the posterior glottis, or coupled oscillation for irregularities in human voices. The second concerns tongue-lip coordination. In human vowel inventories, tongue retraction typically accompanies rounding. In a case report, a trained Chimpanzee was able to produce back vowels /a, o, u/ in whispering, but could not coordinate these articulatory positions with voiced phonation. This suggests an innate neural circuit for patterned tongue-lip coordination as seen in sucking, and such a basic neural unit may be active in humans to simplify the articulatory control of back rounded vowels. [Work supported by NSFC No. 61573254.]

Which Language R You Speaking? /r/ as a Language Marker in Tyrolean and Italian Bilinguals.

Across languages of the world the /r/ sound is known for its variability. This variability has been investigated using articulatory models as well as in sociolinguistic studies. The current study investigates to what extent /r/ is a marker of a bilingual's dominant language. To this end, a reading task was carried out by bilingual speakers from South Tyrol, who produce /r/ differently according to whether they dominantly speak Tyrolean or Italian. The recorded reading data were subsequently used in a perception experiment to investigate whether South Tyrolean bilingual listeners are able to identify the dominant language of the speaker. Results indicate that listeners use /r/ as a cue to determine the dominant language of the speaker whilst relying on articulatory distinctions between the variants. It is furthermore shown that /r/ correlates with three interdependent variables: the sociolinguistic background of the speakers, their speech production, and how their speech is perceived.

Investigations on Mandarin Aspiratory Animations Using an Airflow Model

Various three-dimensional (3-D) talking heads have been developed lately for language learning, with both external and internal articulatory movements being visualized to guide learning. Mandarin pronunciation animation is challenging due to its confusable stops and affricates with similar places of articulation. Until now, less attention has been paid to the biosignal information of aspiratory airflow, which is essential in distinguishing Mandarin consonants. This study fills a research gap by presenting the quantitative analyses of airflow, and then designing an airflow model for a 3-D pronunciation system. The airflow information was collected by Phonatory Aerodynamic System, so that confusable consonants in Mandarin could be discerned by mean airflow rate, peak airflow rate, airflow duration, and peak time. Based on the airflow parameters, an airflow model using the physical equation of fluid flow was proposed and solved, which was then combined and synchronized with the existing 3-D articulatory model. Therefore, the new multimodal system was implemented to synchronously exhibit the airflow motions and articulatory movements of uttering Mandarin syllables. Both an audio-visual perception test and a pronunciation training study were conducted to assess the effectiveness of our system. Perceptual results indicated that identification accuracy was improved for both native and nonnative groups with the help of airflow motions, while native perceivers exhibited higher accuracy due to long-term language experience. Moreover, our system helped Japanese learners of Mandarin enhance their production skills of Mandarin aspirated consonants, reflected by higher gain values of voice onset time after training.

Speaker-Independent Silent Speech Recognition from Flesh-Point Articulatory Movements Using an LSTM Neural Network.

Silent speech recognition (SSR) converts non-audio information such as articulatory movements into text. SSR has the potential to enable persons with laryngectomy to communicate through natural spoken expression. Current SSR systems have largely relied on speaker-dependent recognition models. The high degree of variability in articulatory patterns across different speakers has been a barrier for developing effective speaker-independent SSR approaches. Speaker-independent SSR approaches, however, are critical for reducing the amount of training data required from each speaker. In this paper, we investigate speaker-independent SSR from the movements of flesh points on tongue and lip with articulatory normalization methods that reduce the inter-speaker variation. To minimize the across-speaker physiological differences of the articulators, we propose Procrustes matching-based articulatory normalization by removing locational, rotational, and scaling differences. To further normalize the articulatory data, we apply feature-space maximum likelihood linear regression and i-vector. In this paper, we adopt a bidirectional long short term memory recurrent neural network (BLSTM) as an articulatory model to effectively model the articulatory movements with long-range articulatory history. A silent speech data set with flesh points was collected using an electromagnetic articulograph (EMA) from twelve healthy and two laryngectomized English speakers. Experimental results showed the effectiveness of our speaker-independent SSR approaches on healthy as well as laryngectomy speakers. In addition, BLSTM outperformed standard deep neural network. The best performance was obtained by BLSTM with all the three normalization approaches combined.