Electromagnetic Articulometry Research Articles

Co-speech head nods are used to enhance prosodic prominence at different levels of narrow focus in French.

Previous research has shown that prosodic structure can regulate the relationship between co-speech gestures and speech itself. Most co-speech studies have focused on manual gestures, but head movements have also been observed to accompany speech events by Munhall, Jones, Callan, Kuratate, and Vatikiotis-Bateson [(2004). Psychol. Sci. 15(2), 133-137], and these co-verbal gestures may be linked to prosodic prominence, as shown by Esteve-Gibert, Borrás-Comes, Asor, Swerts, and Prieto [(2017). J. Acoust. Soc. Am. 141(6), 4727-4739], Hadar, Steiner, Grant, and Rose [(1984). Hum. Mov. Sci. 3, 237-245], and House, Beskow, and Granström [(2001). Lang. Speech 26(2), 117-129]. This study examines how the timing and magnitude of head nods may be related to degrees of prosodic prominence connected to different focus conditions. Using electromagnetic articulometry, a time-varying signal of vertical head movement for 12 native French speakers was generated to examine the relationship between head nod gestures and F0 peaks. The results suggest that speakers use two different alignment strategies, which integrate both temporal and magnitudinal aspects of the gesture. Some evidence of inter-speaker preferences in the use of the two strategies was observed, although the inter-speaker variability is not categorical. Importantly, prosodic prominence itself is not the cause of the difference between the two strategies, but instead magnifies their inherent differences. In this way, the use of co-speech head nod gestures under French focus conditions can be considered as a method of prosodic enhancement.

Modeling trajectories of human speech articulators using general Tau theory

This paper presents an application of general Tau theory to the modeling and analysis of articulatory trajectories in speech. We evaluated the model using electromagnetic articulometry data from 12 native speakers of English reading a common text, where trajectories of the following sensors were fitted: lower and upper lips, jaw, and three tongue sensors. Additionally, we analyzed trajectories of the lip aperture signal. Our experiments show that the general Tau theory model gives a better fit than existing (i) methods based on critically damped oscillators, and (ii) a method based on sequential target approximation. These findings support the hypothesis of Tau-guided movements of articulators during speech production. In the second part of the paper, our Tau theory analysis shows that articulatory movements follow similar velocity profile distributions across speakers. In particular, the value of the shape parameter κ of the Tau theory equation is identically distributed across speakers, following a unimodal distribution. The statistical mode of the distribution corresponds to the value of κ that generates a symmetric velocity profile. The analysis of the statistical distribution of κ values also reveals that its variance decreases when greater articulatory effort is required, such that produced articulatory effort remains close to that predicted by the theoretical minimal cost function based on forces acting on the moving articulator. This provides new evidence that articulatory effort is optimized during speech production.

Spatiotemporal coordination in word-medial stop-lateral and s-stop clusters of American English.

This paper is concerned with the relation between syllabic organization and intersegmental spatiotemporal coordination using Electromagnetic Articulometry recordings from seven speakers of American English (henceforth, English). Whereas previous work on English has focused on word-initial clusters (preceding a vowel whose identity was not systematically varied), the present work examined word-medial clusters /pl, kl, sp, sk/ in the context of three different vowel heights (high, mid, low). Our results provide evidence for a global organization for the segments involved in these cluster-vowel combinations. This is reflected in a number of ways: compression of the prevocalic consonant and reduction of CV timing in the word-medial cluster case compared to its singleton paired word in both stop-lateral and s-stop clusters, early vowel initiation (as permitted by the clusters' phonetic properties), and presence of compensatory relations between phonetic properties of different segments or intersegmental transitions within each cluster. In other words, we find that the global organization presiding over the segments partaking in these word-medial tautosyllabic CCVs is pleiotropic, that is, simultaneously expressed in multiple phonetic exponents rather than via a privileged metric such as c-center stability or any other such given single measure employed in previous works.

Global organization in Spanish onsets

This paper addresses the relation between syllable structure and inter-segmental temporal coordination. The data examined are Electromagnetic Articulometry recordings from six speakers of Central Peninsular Spanish (henceforth, Spanish), producing words beginning with the clusters /pl, bl, kl, gl, pɾ, kɾ, tɾ/ as well as corresponding unclustered sonorant-initial words in three vowel contexts /a, e, o/. In our results, we find evidence for a global organization of the segments involved in these combinations. This is reflected in a number of ways: shortening of the prevocalic sonorant in the cluster-initial case compared to the unclustered case, reorganization of the relative timing of the internal CV subsequence (in a CCV) in the obstruent-lateral context, early vowel initiation, and a strong compensatory relation between the duration of the obstruent-to-lateral transition and the duration of the lateral. In other words, we find that the global organization presiding over the segments partaking in these tautosyllabic CCVs is pleiotropic, that is, simultaneously expressed over a set of different phonetic parameters rather than via a privileged metric such as c-center stability or any other such given single measure (employed in prior works).

Pause postures: The relationship between articulation and cognitive processes during pauses

Studies examining articulatory characteristics of pauses have identified language-specific postures of the vocal tract in inter-utterance pauses and different articulatory patterns in grammatical and non-grammatical pauses. Pause postures—specific articulatory movements that occur during pauses at strong prosodic boundaries—have been identified for Greek and German. However, the cognitive function of these articulations has not been examined so far. We start addressing this question by investigating the effect of (1) utterance type and (2) planning on pause posture occurrence and properties in American English. We first examine whether pause postures exist in American English. In an electromagnetic articulometry study, seven participants produced sentences varying in linguistic structure (stress, boundary, sentence type). To determine the presence of pause postures, as well as to lay the groundwork for their future automatic annotation and detection, a Support Vector Machine Classifier was built to identify pause postures. Results show that pause postures exist for all speakers in this study but that the frequency of occurrence is speaker dependent. Across participants, we find that there is a stable relationship between the pause posture and other events (boundary tones and vowels) at prosodic boundaries, parallel to previous work in Greek. We find that the occurrence of pause postures is not systematically related to utterance type. Lastly, pause postures increase in frequency and duration as utterance length increases, suggesting that pause postures are at least partially related to speech planning processes.

An electromagnetic articulography-facilitated deep neural network model of tongue contour detection from ultrasound images

Ultrasound imaging is a non-invasive technique for the measurement of the tongue in speech. Recent advancements in analytical edge detection algorithms and deep learning methods have improved tongue contour segmentation. However, most edge detection algorithms require user input as initialization “seeds” and accuracy can drift as segmentation subsequently proceeds. Deep learning in ultrasound tongue contour tracking, on the other hand, requires large, manually labelled training data, has poor spatial resolution and does not generalize well to images acquired by ultrasound machines outside the training set. Here, we demonstrate an approach combining both edge detection and deep learning in automatic tongue contour tracking in ultrasound images of the tongue, aided by synchronously acquired electromagnetic articulometry (EMA) data. A deep learning model was trained by edge detection seeded by tongue locations from EMA with minimal human intervention. Spatial and temporal constraints for tongue contours were learned simultaneously using a three-dimensional convolutional neural network. Finally, the deep neural network inferred tongue contour in low resolution was passed through additional edge detection to refine the contour in higher resolution. Our preliminary results demonstrate the proposed architecture improved accuracy compared to using analytical edge detection or deep learning alone.

Fitts' Law in Tongue Movements of Repetitive Speech.

Fitts' law, perhaps the most celebrated law of human motor control, expresses a relation between the kinematic property of speed and the non-kinematic, task-specific property of accuracy. We aimed to assess whether speech movements obey this law using a metronome-driven speech elicitation paradigm with a systematic speech rate control. Specifically, using the paradigm of repetitive speech, we recorded via electromagnetic articulometry speech movement data in sequences of the form /CV…/ from 6 adult speakers. These sequences were spoken at 8 distinct rates ranging from extremely slow to extremely fast. Our results demonstrate, first, that the present paradigm of extensive metronome-driven manipulations satisfies the crucial prerequisites for evaluating Fitts' law in a subset of our elicited rates. Second, we uncover for the first time in speech evidence for Fitts' law at the faster rates and specifically beyond a participant-specific critical rate. We find no evidence for Fitts' law at the slowest metronome rates. Finally, we discuss implications of these results for models of speech.

Temporal Aspects of Word Initial Single Consonants and Consonants in Clusters in Spanish

We examined gestural coordination in C1C2 (C1 stop, C2 lateral or tap) word initial clusters using articulatory (electromagnetic articulometry) and acoustic data from six speakers of Standard Peninsular Spanish. We report on patterns of voice onset time (VOT), gestural plateau duration of C1, C2, and their overlap. For VOT, as expected, place of articulation is a major factor, with velars exhibiting longer VOTs than labials. Regarding C1 plateau duration, voice and place effects were found such that voiced consonants are significantly shorter than voiceless consonants, and velars show longer duration than labials. For C2 plateau duration, lateral duration was found to vary as a function of onset complexity (C vs. CC). As for overlap, unlike in French, where articulatory data for clusters have also been examined, clusters where both C1 and C2 are voiced show more overlap than where voicing differs. Further, overlap was affected by the C2 such that clusters where C2 is a tap show less overlap than clusters where C2 is a lateral. We discuss these results in the context of work aiming to uncover phonetic (e.g., articulatory or perceptual) and phonological forces (e.g., syllabic organization) on timing.

The speed-curvature power law in tongue movements of repetitive speech.

The speed-curvature power law is a celebrated law of motor control expressing a relation between the kinematic property of speed and the geometric property of curvature. We aimed to assess whether speech movements obey this law just as movements from other domains do. We describe a metronome-driven speech elicitation paradigm designed to cover a wide range of speeds. We recorded via electromagnetic articulometry speech movements in sequences of the form /CV…/ from nine speakers (five German, four English) speaking at eight distinct rates. First, we demonstrate that the paradigm of metronome-driven manipulations results in speech movement data consistent with earlier reports on the kinematics of speech production. Second, analysis of our data in their full three-dimensions and using advanced numerical differentiation methods offers stronger evidence for the law than that reported in previous studies devoted to its assessment. Finally, we demonstrate the presence of a clear rate dependency of the power law’s parameters. The robustness of the speed-curvature relation in our datasets lends further support to the hypothesis that the power law is a general feature of human movement. We place our results in the context of other work in movement control and consider implications for models of speech production.

Pause postures in American English

Recent work examining articulation during pauses has found that articulatory patterns distinguish grammatical from ungrammatical pauses (Ramanarayanan et al. 2009). For Greek, Katsika et al. (2014) have identified pause postures (specific configurations of the vocal tract at prosodic boundaries) which are triggered by π-gestures with high activation levels and consequently occur at strong prosodic boundaries. This study investigates pause postures in American English, specifically, whether they occur, and how they are coordinated with other events at prosodic boundaries. In an electromagnetic articulometry (EMA) study, seven speakers produced 7-8 repetitions of 42 types of sentences varying in linguistic structure (stress, boundary, phrasing, and sentence type). Results from two speakers analyzed to date indicate that pause postures exist but that their presence might be speaker dependent. Analyses of gesture lags indicate a stable relationship between the boundary tone and pause posture, that the boundary tone gesture starts later when stress is on the second syllable, and that the π-gesture is shifted towards the stressed syllable, parallel to findings in Katsika et al. (2014). We discuss these results in relation to models of prosodic structure and to speech planning processes. [Work supported by NIH and NSF.]

Quantifying kinematic aspects of reduction in a contrasting rate production task

Electromagnetic articulometry (EMA) was used to record the 720 phonetically balanced Harvard sentences (IEEE, 1969) from multiple speakers at normal and fast production rates. Participants produced each sentence twice, first at their preferred “normal” speaking rate followed by a “fast” production (for a subset of the sentences two normal rate productions were elicited). They were instructed to produce the “fast” repetition as quickly as possible without making errors. EMA trajectories were obtained at 100 Hz from sensors placed on the tongue, lips, and mandible, corrected for head movement and aligned to the occlusal plane. Synchronized audio was recorded at 22050 Hz. Comparison of normal to fast acoustic durations for paired utterances showed a mean 67% length reduction, and assessed using Mermelstein's method (1975), two fewer syllables on average. A comparison of inflections in vertical jaw movement between paired utterances showed an average of 2.3 fewer syllables. Cross-recurrence analysis of distance maps computed on paired sensor trajectories comparing corresponding normal:normal to normal:fast utterances showed systematically lower determinism and entropy for the cross-rate comparisons, indicating that rate effects on articulator trajectories are not uniform. Examples of rate-related differences in gestural overlap that might account for these differences in predictability will be presented. [Work supported by NSF.]

Aeroacoustic consequences of tongue troughs in labiodentals

It has long been accepted that the main constriction for fricatives /f, v/ is formed by the lower lip pressing against the upper teeth, thus allowing the tongue to freely coarticulate with preceding and following segments. Here, electromagnetic articulometry data were obtained from 5 subjects in a study of tongue troughs, defined as a discontinuity in anticipatory coarticulation, such as when the tongue drops during a bilabial consonant in /ii/ context. The corpus included /f/ and /v/ in VC(C)V contexts, where V = /i/ for C= /v/, and V = {/i a u/} for C= /f/. The tongue moved down and back for /f/ in all vowel contexts; in /iC(C)i/ context, the troughs were deeper for long (VCCV) than short labial consonants, as predicted, and deeper for /f/ than for /p, b, v, m/, which was unexpected. There seems to be a secondary gesture specifying that the tongue be down during labiodental fricatives. Our hypothesis is that lowering the tongue for /f/ ensures that the airflow resistance will be due only to the labiodental constriction, thus providing an aeroacoustic advantage for the onset of turbulence. This is supported by comparison of asymmetric vowel contexts (e.g., /uffi/-/iffu/ tokens); tongue dorsum and blade sensors show the tongue moving quickly away from /i/ to /f/ position, in contrast to movement from /u/ to /f/. Aerodynamic considerations thus appear to be actively incorporated into the speech motor plan. [Work supported by NIH grant DC-002717.]

A Kinematic Study of Prosodic Structure in Articulatory and Manual Gestures: Results from a Novel Method of Data Collection.

The primary goal of this work is to examine prosodic structure as expressed concurrently through articulatory and manual gestures. Specifically, we investigated the effects of phrase-level prominence (Experiment 1) and of prosodic boundaries (Experiments 2 and 3) on the kinematic properties of oral constriction and manual gestures. The hypothesis guiding this work is that prosodic structure will be similarly expressed in both modalities. To test this, we have developed a novel method of data collection that simultaneously records speech audio, vocal tract gestures (using electromagnetic articulometry) and manual gestures (using motion capture). This method allows us, for the first time, to investigate kinematic properties of body movement and vocal tract gestures simultaneously, which in turn allows us to examine the relationship between speech and body gestures with great precision. A second goal of the paper is thus to establish the validity of this method. Results from two speakers show that manual and oral gestures lengthen under prominence and at prosodic boundaries, indicating that the effects of prosodic structure extend beyond the vocal tract to include body movement.1

Mind the gap: Electromagnetic articulometer observation of speech articulation in conversational turn-taking

Joe Perkell pioneered the use of electromagnetic articulometry (EMA) for the observation and quantification of the kinematics of speech articulator movements. In the spirit of his research, we have extended these methods to EMA observation of two facing speakers interacting in conversation. The gaps or pauses between turns in speaking are known from acoustic measurement to be relatively short in duration, about 200 ms or the length of a syllable on average, and this gap duration has been shown to be consistent across widely diverse languages and cultures (Stivers et al., 2009). However, because the cognitive latencies for producing a response are much longer than this interval its planning must occur during the incoming turn. Here we provide evidence for this planning from articulator movements that anticipate speech. Movements of sensors attached to the tongue, jaw and lips have been tracked for each of 12 speaker pairs. Gaps are measured as the difference between the acoustic end of one speaker's turn t...

Kinematic properties of concurrently recorded speech and body gestures and their relationship to prosodic structure

Previous research has shown that body gesturing, such as hand and torso movements, is related to prosodic structure (e.g., Esteve-Gibert & Prieto 2013). However, the details of the relationship are not well understood and specifically the effects of prosodic structure on gesturing duration have received little attention. We present an experiment investigating the effects of stress and prosodic boundaries on the temporal properties of body gesturing and gestures of the vocal tract. Two speakers read 6 repetitions of 12 sentences examining the effect of boundary (no boundary, ip boundary, IP boundary), stress (no stress, stress) and their interaction, phrase finally and phrase initially (144 sentences total). They were observed through concurrent recordings of speech audio, vocal tract gestures using electromagnetic articulometry, and body gesturing using Vicon motion capture. We test the hypothesis that manual movements lengthen under prominence and at boundaries and that the lengthening at boundaries is cumulative, parallel to the behavior of gestures of the vocal tract. Analyzing speech prosody and body gesturing in this manner will allow us to investigate whether prosodic control extends to body gesturing and to identify coordinative patterns that are relevant to fields such as language pedagogy and speech pathology. [Work supported by NIH.]

A gestural account of Mandarin tone sandhi

Recently tones have been analyzed as articulatory gestures, which may be coordinated with segmental gestures. Our data from electromagnetic articulometry (EMA) show that purported neutralized phonological contrast can nonetheless exhibit coordinative difference. We develop a model based on gestural coupling to account for observed patterns. Mandarin Third Tone Sandhi (e.g., Tone3 → T3S /_ Tone3) is perceptually neutralizing in that the sandhi output (T3S) shares great similarity with Tone2. Despite both tones having rising pitch contours, there exist subtle acoustic differences. However, the difference in underlying representation between T3S and Tone2 remains unclear. By presenting evidence from the alignment pattern between tones and segments, we show that the acoustic differences between Tone2 and T3S arises out of the difference in gestural organizations. The temporal lag between the initiation of the Vowel gesture and that of Tone gesture in T3S is shorter than that in Tone2. We further argue that underlying Tone3 is the source of incomplete neutralization between the Tone2 and T3S. That is, despite the surface similarity, T3S is stored in the mental lexicon as Tone3.

Temporal alignment between head gesture and prosodic prominence in naturally occurring conversation: An electromagnetic articulometry study

Studies of the relationship between speech events and gesticulation have suggested that the peak of the prosodic pitch accent serves as a target with which body gestures may be coordinated (Roth, 2002; Loehr, 2004). While previous work has relied on controlled speech elicitation generally restricted to nonrepresentational extension/retraction (Leonard and Cummins, 2011) or iconic (Kelly et al., 2008) gestures, here we examine the kinematics of the speech articulators and associated head movements from pairs of individuals engaged in spontaneous conversation. Age and gender matched native speakers of American English seated 2 m apart were recorded using two electromagnetic articulometer (EMA) devices (Tiede and Mooshammer, 2013). Head movements were characterized by the centroid of reference sensors placed on the left and right mastoid processes and the upper incisors. Pitch accents were coded following the ToBI implementation of Pierrehumbert's intonational framework following Beckman and Elam (1997). Preliminary findings show that the apex (point of maximum excursion) of head movements within an IP in general precedes the peak of the associated pitch accent, and is consistently aligned with co-occurring articulatory events within the syllable. [Work supported by NIH NIDCD-DC-012350.]

Speaking tongues are always braced

Bracing the tongue against rigid vocal tract surfaces (i.e., teeth or palate) has been suggested to be important in facilitating certain kinds of tongue movements [Stone, J. Acoust. Soc. Am. 81, 2207–2218 (1990)]. However, previous studies have generally sought bracing in only a narrow range of phonetic contexts, resulting in a widespread view of bracing as an occasional state, peculiar to specific sounds or sound combinations. The present study uses electropalatography (EPG) as well as ultrasound imaging and electromagnetic articulometry (EMA) to describe tongue bracing in continuous speech passages, finding that the tongue is almost constantly braced against lateral surfaces during running speech. Analysis of archival data from the male and female speakers of American English in the KayPENTAX Palatometer Database (Model 4333) shows that they brace the tongue continuously, except during a small percentage of low vowels, and during a larger percentage of instances of /l/. Additional measures using all three devices, as well as biomechanical simulations using ArtiSynth (www.artisynth.org), provide further insight, indicating that the tongue also braces against the central palate and/or lower jaw, and that bracing points slide anteroposteriorly across speech sounds. These results suggest that bracing is a constant and necessary aspect of tongue motor control.

Coarticulatory effects of lateral tongue bracing in first and second language English speakers

This study uses electromagnetic articulometry (EMA) to examine the coarticulatory effects of tongue bracing in L1 and L2 English speakers. The tongue is hydrostatic, so we brace it against our teeth for added control, and this bracing is an important part of pronunciation. The amount of bracing may differ across languages (and may be part of one’s articulatory setting), so understanding these differences could be a key to L2 pronunciation learning. Although lingual coarticulation has been examined using acoustics and midsagittal views of the vocal tract, not much focus has been placed on the coronal view. We collected EMA point-tracking data from two native speakers of North American English and looked at the movement of a lateral tongue marker. As stimuli, we choose the nursery rhyme “Mary had a Little Lamb” because of the variation in vowels, and also the /l/ and /r/ phonemes, which are absent in Japanese. Initial results show differences between vowels that occur next to /l/ and those that occur next to /r/ and stops. Results will also be presented for Japanese speakers of both their L1 (Japanese) and L2 English. If we find crosslinguistic differences in bracing, this fact will be important for pedagogical purposes.

Imaging and quantification of glottal kinematics with ultrasound during speech

Most examinations of glottal abduction and adduction during speech have employed laryngoscopic video or transillumination (Hoole, 2006). While these provide accurate information about timing of glottal movements, they are invasive and cannot provide absolute measurements about glottal width. At the same time, recent medical studies have used ultrasonic imaging to accurately capture glottal movements and laryngeal anatomy [Hu, J. Ultrasound Med. (2010); Jadcherla etal., Dysphagia (2006)]. We demonstrate novel methods of using ultrasound to measure both temporal and spatial aspects of glottal movements during speech. While previous work on glottal ultrasound has been limited by the need to manually analyze each acquired frame, we present methods to automatically quantify glottal aperture in ultrasound images. Finally, since glottal ultrasound does not interfere with the acquisition of supra-laryngeal articulatory data, we present the results of preliminary experiments that record laryngeal and supra-laryngeal speech movements simultaneously using ultrasound concurrently with electromagnetic articulometry. This allows the analysis of relative timing between movements in the two systems as well as examination of changes in timing or magnitude due to variables such as prosodic structure or speech rate. [Work supported by NIH.]