Lip Kinematics Research Articles

A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online

Recent advances in automated face-tracking have made it possible for laboratory phonologists to measure lip motion using technology no more advanced than a common webcam. The present paper introduces a lip-tracking approach specifically designed for use in web-based research. The central innovation is a custom extension written for jsPsych, an existing JavaScript framework for running behavioral experiments online. This extension gathers data from the participant’s webcam and processes them through FaceMesh, an open-source, JavaScript face-tracker. Face-tracking happens on the fly inside the participant’s browser. Only time-stamped vertical or horizontal lip apertures are saved to the experiment server. That is, this extension allows experiments implemented in jsPsych to collect de-identified lip kinematics from participants seated at their own home computers. After explaining the core functionality of the extension, this paper presents two validation experiments. The first establishes that utterances with different lip kinematics result in very different lip aperture trajectories, and that the timing of a key kinematic landmark agrees with the timing of acoustic landmarks obtained for the same utterances. The second experiment replicates a vowel-priming result previously demonstrated using a prior face-tracking system and saved facial video. All associated scripts have been made publicly available.

Compensatory articulatory behaviors after tongue reconstruction in production of English plosives: Establishing the range of lip movement patterns for control speakers

Head and neck cancer can have a devastating impact on speech and swallowing function. In particular, a tumor in the tongue can reduce the ability to produce articulatory gestures typical for English plosives. Previous case studies suggest that the lower lip can compensate for tongue tip gestures in speech after tongue reconstruction. The goal of this study is to establish typical lower lip movement patterns for alveolar and velar plosives for English speakers towards identifying compensatory lip movements in cancer speakers. Ten participants were recruited with no reported hearing or speech problems. A list of 40 minimal pairs beginning with the target plosives were created and embedded in carrier sentences. The participants read sentences in random order with and without babble noise in a standing posture. Lip motion was captured using a custom app on an iPhone 11 capturing the perioral surface area with the TrueDepth infrared camera. Preliminary results suggest that this app-based approach to lip tracking is a viable tool to capture typical lip movement patterns, ultimately enabling clinical research in more accessible settings. Mixed-effects linear regression analyses of the lip kinematics will be discussed.

Auditory startle disrupts speech coordination

Abstract Speech production requires temporal coordination between the actions of different functional groupings of muscles in the human body. Crucially, such functionally organized units, or “modules”, may be susceptible to disruption by an external stimulus such as a startling auditory stimulus (SAS; >120dB), enabling a possible window into the internal structure of learned speech movements. Following on the observation that SAS is known to accelerate the release of pre-planned actions, the current study examines lip kinematics in SAS-induced responses during speech movements to test whether this accelerated release applies on the scale of entire syllables or on the scale of smaller functional units. Production measures show that SAS-elicited bilabial movements in [ba] syllables are prone to disruption as measured by discontinuity in velocity profiles. We use a 3D finite element method (FEM) biomechanical model to simulate the temporal interaction between muscle groupings in speech. Simulation results indicate that this discontinuity can be accounted for as an instance of temporally decoupled coordination across neuromuscular modules. In such instances, the muscle groupings controlling lip compression and jaw opening, which normally fire sequentially, appear more likely to be activated synchronously.

Effects of utterance rate and length on the spatiotemporal index in Parkinson’s disease

Purpose: To characterise labial articulatory pattern variability using the spatiotemporal index (STI) in speakers with idiopathic Parkinson’s disease (PD) across different speaking rates and syllable-sentence conditions compared to age- and sex-matched healthy controls.Method: Ten speakers with mild–severe idiopathic PD and 10 controls produced “pa” and the Rainbow Passage at slow, typical and fast speech rates. Upper lip and lower lip kinematics were digitised during a motion capture system. Data were analysed using linear mixed modelling.Result: Regardless of the participant group, a high STI value was observed in the fast speech rate for the “pa” syllable condition, particularly for movements of the lower lip. As utterance rate increased, the control group showed the highest variability, followed by PD OFF and PD ON conditions. Syllable “pa” showed a greater STI value compared to both the first and second utterance of Rainbow Passage.Conclusion: PD manifests sufficient residual capacity to achieve near-normal motor compensation to preserve the consistency of lower lip movements during speech production. The lack of a significant difference in lip STI values between ON-OFF medication states suggests that dopaminergic treatment does not influence stability of speech for individuals with mild-moderate stage PD.

High-Resolution, Non-Invasive Imaging of Upper Vocal Tract Articulators Compatible with Human Brain Recordings.

A complete neurobiological understanding of speech motor control requires determination of the relationship between simultaneously recorded neural activity and the kinematics of the lips, jaw, tongue, and larynx. Many speech articulators are internal to the vocal tract, and therefore simultaneously tracking the kinematics of all articulators is nontrivial—especially in the context of human electrophysiology recordings. Here, we describe a noninvasive, multi-modal imaging system to monitor vocal tract kinematics, demonstrate this system in six speakers during production of nine American English vowels, and provide new analysis of such data. Classification and regression analysis revealed considerable variability in the articulator-to-acoustic relationship across speakers. Non-negative matrix factorization extracted basis sets capturing vocal tract shapes allowing for higher vowel classification accuracy than traditional methods. Statistical speech synthesis generated speech from vocal tract measurements, and we demonstrate perceptual identification. We demonstrate the capacity to predict lip kinematics from ventral sensorimotor cortical activity. These results demonstrate a multi-modal system to non-invasively monitor articulator kinematics during speech production, describe novel analytic methods for relating kinematic data to speech acoustics, and provide the first decoding of speech kinematics from electrocorticography. These advances will be critical for understanding the cortical basis of speech production and the creation of vocal prosthetics.

Effects of aspiration and vowel context on lip and jaw kinematics in Cantonese bilabial plosive production

The present study examined the possible effects of aspiration and vowel context on lip and jaw movements during production of Cantonese bilabial plosives /p/ and /ph/. VCV syllables were randomly produced by 30 adult native Cantonese speakers. Movement of upper lip, lower lip, and jaw during speech production was recorded using electromagnetic articulography. Kinematic analysis of lip and jaw movements during the opening phase of bilabial plosive production revealed a significant aspiration effect, especially on vertical displacement and velocity of the jaw. Such effects were consistent across all vowel contexts. Similarly, a significant aspiration effect was observed in vertical displacement of the lower lip. These findings suggest that muscular force is the main driving force in the production of unaspirated bilabial plosives.

Gesture and word analysis: the same or different processes?

The present study aimed at determining whether elaboration of communicative signals (symbolic gestures and words) is always accompanied by integration with each other and, if present, this integration can be considered in support of the existence of a same control mechanism. Experiment 1 aimed at determining whether and how gesture is integrated with word. Participants were administered with a semantic priming paradigm with a lexical decision task and pronounced a target word, which was preceded by a meaningful or meaningless prime gesture. When meaningful, the gesture could be either congruent or incongruent with word meaning. Duration of prime presentation (100, 250, 400 ms) randomly varied. Voice spectra, lip kinematics, and time to response were recorded and analyzed. Formant 1 of voice spectra, and mean velocity in lip kinematics increased when the prime was meaningful and congruent with the word, as compared to meaningless gesture. In other words, parameters of voice and movement were magnified by congruence, but this occurred only when prime duration was 250 ms. Time to response to meaningful gesture was shorter in the condition of congruence compared to incongruence. Experiment 2 aimed at determining whether the mechanism of integration of a prime word with a target word is similar to that of a prime gesture with a target word. Formant 1 of the target word increased when word prime was meaningful and congruent, as compared to meaningless congruent prime. Increase was, however, present for whatever prime word duration. Experiment 3 aimed at determining whether symbolic prime gesture comprehension makes use of motor simulation. Transcranial Magnetic Stimulation was delivered to left primary motor cortex 100, 250, 500 ms after prime gesture presentation. Motor Evoked Potential of First Dorsal Interosseus increased when stimulation occurred 100 ms post-stimulus. Thus, gesture was understood within 100 ms and integrated with the target word within 250 ms. Experiment 4 excluded any hand motor simulation in order to comprehend prime word. Thus, the same type of integration with a word was present for both prime gesture and word. It was probably successive to understanding of the signal, which used motor simulation for gesture and direct access to semantics for words.

The effect of focus marking on supralaryngeal articulation – Is it mediated by accentuation?

In this study we explore the effects of focus-background structure on accentuation (i.e. whether a word bears a pitch accent or not) and supralaryngeal articulation, measured in terms of acoustic durations (syllable and foot durations) and lip kinematics (parameters relating to the opening gesture: duration, displacement, peak velocity and stiffness). Although words in focus were accented and those out of focus were not, there were few supralaryngeal differences between accented words when they were produced in the broad focus context and unaccented target words (out of focus). Thus, accentuation per se did not appear to lead to supralaryngeal modifications. However, there was a clear distinction between the supralaryngeal articulation of words in broad focus and those in contrastive focus. We conclude that supralaryngeal articulation – in terms of acoustic duration and lip kinematics – is related directly to the expression of focus structure and contrastivity, and is not, contrary to conclusions drawn in previous studies, mediated by the presence or absence of accent.

Producing whole speech events: Anticipatory lip compression in bilabial stops

Bilabial stops /b/, /p/, and /m/ ostensibly share a common lip constriction. Recent evidence shows that different bilabial stops involve distinct facial muscle activations, suggesting that oral speech movements anticipate aerodynamic conditions [Gick et al. Proc. Acoust. (2012) 2pSC1]. The present study investigates how the lips themselves behave in whole speech events. Existing models of speech production governing only articulatory motions predict that lip compression would respond to changes in aerodynamic conditions rather than anticipating such changes; a model that includes whole events predicts anticipatory activation of lip muscles with concomitant kinematic lip compression, but only in cases where a real increase in air pressure is expected. Lip kinematics were recorded using OptoTrak to trace lip movements of bilabial stops in response to imperative acoustic stimuli. Results show consistent anticipatory lip compression in spoken /b/ and /p/, but not in non-speech jaw opening movements and only sporadic compression in mouthed /b/ and /p/, where air pressure is not expected to increase. Biomechanical simulation using an orofacial model developed within the Artisynth simulation toolkit (www.artisynth.org) confirms anticipatory muscle activations. These findings support a model of speech tasks wherein coordinated body-level muscular systems govern whole speech events.

FOXP2 gene and language development: the molecular substrate of the gestural-origin theory of speech?

OPINION article Front. Behav. Neurosci., 05 August 2013Sec. Pathological Conditions https://doi.org/10.3389/fnbeh.2013.00099

Speech‐like and non‐speech lip kinematics and coordination in aphasia

In addition to the well-known linguistic processing impairments in aphasia, oro-motor skills and articulatory implementation of speech segments are reported to be compromised to some degree in most types of aphasia. This study aimed to identify differences in the characteristics and coordination of lip movements in the production of a bilabial closure gesture between speech-like and non-speech tasks in individuals with aphasia and healthy control subjects. Upper and lower lip movement data were collected for a speech-like and a non-speech task using an AG 100 EMMA system from five individuals with aphasia and five age- and gender-matched control subjects. Each task was produced at two rate conditions (normal and fast), and in a familiar and a less familiar manner. Single articulator kinematic parameters (peak velocity, amplitude, duration and cyclic spatio-temporal index) and multi-articulator coordination indices (average relative phase and variability of relative phase) were measured to characterize lip movements. The results showed that when the two lips had similar task goals (bilabial closure) in speech-like and non-speech task, kinematic and coordination characteristics were not found to be different. However, when changes in rate were imposed on the bilabial gesture, only speech-like task showed functional adaptations, indicated by a greater decrease in amplitude and duration at fast rates. In terms of group differences, individuals with aphasia showed smaller amplitudes and longer movement durations for upper lip, higher spatio-temporal variability for both lips, and higher variability in lip coordination than the control speakers. Rate was an important factor in distinguishing the two groups, and individuals with aphasia were limited in implementing the rate changes. The findings support the notion of subtle but robust differences in motor control characteristics between individuals with aphasia and the control participants, even in the context of producing bilabial closing gestures for a relatively simple speech-like task. The findings also highlight the functional differences between speech-like and non-speech tasks, despite a common movement coordination goal for bilabial closure.

Effects of utterance length on lip kinematics in aphasia

Most existing models of language production and speech motor control do not explicitly address how language requirements affect speech motor functions, as these domains are usually treated as separate and independent from one another. This investigation compared lip movements during bilabial closure between five individuals with mild aphasia and five age and gender-matched control speakers when the linguistic characteristics of the stimuli were varied by increasing the number of syllables. Upper and lower lip movement data were collected for mono-, bi- and tri-syllabic nonword sequences using an AG 100 EMMA system. Each task was performed under both normal and fast rate conditions. Single articulator kinematic parameters (peak velocity, amplitude, duration, and cyclic spatio-temporal index) were measured to characterize lip movements. Results revealed that compared to control speakers, individuals with aphasia showed significantly longer movement duration and lower movement stability for longer items (bi- and tri-syllables). Moreover, utterance length affected the lip kinematics, in that the monosyllables had smaller peak velocities, smaller amplitudes and shorter durations compared to bi- and trisyllables, and movement stability was lowest for the trisyllables. In addition, the rate-induced changes (smaller amplitude and shorter duration with increased rate) were most prominent for the short items (i.e., monosyllables). These findings provide further support for the notion that linguistic changes have an impact on the characteristics of speech movements, and that individuals with aphasia are more affected by such changes than control speakers.

Bite-block perturbation in people who stutter: Immediate compensatory and delayed adaptive processes

This exploratory study investigated sensory-motor mechanisms in five people who stutter (PWS) and five people who do not (PNS). Lip kinematic and coordination data were recorded as they produced bi-syllabic nonwords at two rates (normal and fast) in three conditions (jaw-free, immediately after insertion of a bite-block, and after a 10-min accommodation period). At normal speech rates, effects of bite-blocks on lip kinematics were similar for both PWS and PNS speakers showing larger amplitudes, peak velocities, shorter durations and more stable movement cycle patterns. However, at fast speech rates upper lip responses of PWS exhibited larger amplitudes and peak velocities. At both speech rates, the presence of a bite-block changed movement coordination patterns only for PNS. However, at fast speech rates bite-blocks decreased variability of coordination patterns for both groups. No adaptive changes in movement stability were found for either group, but a practice-related increase in lower lip peak velocity was found at normal speech rates. These findings indicate that bite-block perturbation did not exacerbate any hypothesized limitation or difficulty in controlling individual articulatory movements or their coordination in PWS. The results also support the position that specific motor control strategies are used by PWS as compared to PNS to compensate for bite-block perturbations under increased speech rate demands. Learning outcomes: The reader will be able to: (1) distinguish between compensatory and adaptive responses to bite-block perturbation; (2) explain the measurement of articulatory stability; (3) summarize the potential role of motor control strategies in people who stutter; and (4) discuss the assumptions of the motor skills approach to stuttering.

When the hands speak

In the present review we will summarize evidence that the control of spoken language shares the same system involved in the control of arm gestures. Studies of primate premotor cortex discovered the existence of the so-called mirror system as well as of a system of double commands to hand and mouth. These systems may have evolved initially in the context of ingestion, and later formed a platform for combined manual and vocal communication. In humans, manual gestures are integrated with speech production, when they accompany speech. Lip kinematics and parameters of voice spectra during speech production are influenced by executing or observing transitive actions (i.e. guided by an object). Manual actions also play an important role in language acquisition in children, from the babbling stage onwards. Behavioural data reported here even show a reciprocal influence between words and symbolic gestures and studies employing neuroimaging and repetitive transcranial magnetic stimulation (rTMS) techniques suggest that the system governing both speech and gesture is located in Broca’s area.

Articulator Movement Associated With the Development of Prosodic Control in Children

This study explored the relationship between articulator movement and prosody in children at different developmental ages. Jaw, lower lip, and upper lip kinematics were examined in 4-, 7-, and 11-year-old children as they produced the declarative and interrogative forms of utterances "Show Bob a bot" and "Show Pop a pot." Articulator movement differences were found between declaratives and interrogatives during the production of vowel and consonant targets. The 7- and 11-year-olds differentiated the duration and displacement of their articulator movements to distinguish between declaratives and interrogatives. Several age-related differences in movement duration between the 4-year-olds and the older participants were associated with the production of these targets. Productions of declaratives and interrogatives were not associated with any significant differences in articulator movement variability, although variability did decrease significantly with maturation. The results suggest that children modify their articulator movements to meet the prosodic and linguistic demands of the task. While 4-year-olds appear to be able to modify lip and jaw movement to mark the declarative-interrogative contrast, refinement of these movements continues throughout childhood.

Imitation during phoneme production

Does listening to and observing the speaking interlocutor influence phoneme production? In two experiments female participants were required to recognize and, then, to repeat the string-of-phonemes /aba/ presented by actors visually, acoustically and audiovisually. In experiment 1 a male actor presented the string-of-phonemes and the participants’ lip kinematics and voice spectra were compared with those of a reading control condition. In experiment 2 female and male actors presented the string-of-phonemes and the lip kinematics and the voice spectra of the participants’ responses to the male actors were compared with those to the female actors (control condition). In both experiments 1 and 2, the lip kinematics in the visual presentations and the voice spectra in the acoustical presentations changed in the comparison with the control conditions approaching the male actors’ values, which were different from those of the female participants and actors. The variation in lip kinematics induced changes also in voice formants but only in the visual presentation. The data suggest that both features of the lip kinematics and of the voice spectra tend to be automatically imitated when repeating a string-of-phonemes presented by a visible and/or audible speaking interlocutor. The use of imitation, in place of the usual lip kinematics and vocal features, suggests an automatic and unconscious tendency of the perceiver to interact closely with the interlocutor. This is in accordance with the idea that resonant circuits are activated by the activity of the mirror system, which relates observation to execution of arm and mouth gestures.

Automatic audiovisual integration in speech perception

Two experiments aimed to determine whether features of both the visual and acoustical inputs are always merged into the perceived representation of speech and whether this audiovisual integration is based on either cross-modal binding functions or on imitation. In a McGurk paradigm, observers were required to repeat aloud a string of phonemes uttered by an actor (acoustical presentation of phonemic string) whose mouth, in contrast, mimicked pronunciation of a different string (visual presentation). In a control experiment participants read the same printed strings of letters. This condition aimed to analyze the pattern of voice and the lip kinematics controlling for imitation. In the control experiment and in the congruent audiovisual presentation, i.e. when the articulation mouth gestures were congruent with the emission of the string of phones, the voice spectrum and the lip kinematics varied according to the pronounced strings of phonemes. In the McGurk paradigm the participants were unaware of the incongruence between visual and acoustical stimuli. The acoustical analysis of the participants' spoken responses showed three distinct patterns: the fusion of the two stimuli (the McGurk effect), repetition of the acoustically presented string of phonemes, and, less frequently, of the string of phonemes corresponding to the mouth gestures mimicked by the actor. However, the analysis of the latter two responses showed that the formant 2 of the participants' voice spectra always differed from the value recorded in the congruent audiovisual presentation. It approached the value of the formant 2 of the string of phonemes presented in the other modality, which was apparently ignored. The lip kinematics of the participants repeating the string of phonemes acoustically presented were influenced by the observation of the lip movements mimicked by the actor, but only when pronouncing a labial consonant. The data are discussed in favor of the hypothesis that features of both the visual and acoustical inputs always contribute to the representation of a string of phonemes and that cross-modal integration occurs by extracting mouth articulation features peculiar for the pronunciation of that string of phonemes.

Lip kinematics in long and short stop and fricative consonants

This paper examines lip and jaw kinematics in the production of labial stop and fricative consonants where the duration of the oral closure/constriction is varied for linguistic purposes. The subjects were speakers of Japanese and Swedish, two languages that have a contrast between short and long consonants. Lip and jaw movements were recorded using a magnetometer system. Based on earlier work showing that the lips are moving at a high velocity at the oral closure, it was hypothesized that speakers could control closure/constriction duration by varying the position of a virtual target for the lips. According to this hypothesis, the peak vertical position of the lower lip during the oral closure/constriction should be higher for the long than for the short consonants. This would result in the lips staying in contact for a longer period. The results show that this is the case for the Japanese subjects and one Swedish subject who produced non-overlapping distributions of closure/ constriction duration for the two categories. However, the peak velocity of the lower lip raising movement did not differ between the two categories. Thus if the lip movements in speech are controlled by specifying a virtual target, that control must involve variations in both the position and the timing of the target.

Lip Kinematics for /p/ and /b/ Production during Whispered and Voiced Speech

In the absence of voicing, the discrimination of ‘voiced’ and ‘voiceless’ stop consonants in whispered speech relies on such acoustic cues as burst duration and amplitude, and formant transition characteristics. The articulatory processes that generate these features of whispered speech remain speculative. This preliminary investigation examines the articulatory kinematics differences between whispered /p/ and /b/, which may underlie the acoustic differences previously reported for these sounds. Computerized video-tracking methods were used to evaluate kinematic differences between voiced and voiceless stops. Seven subjects produced the target utterances ‘my papa puppy’ and ‘my baba puppy’ in voiced and whispered speech modes. The results revealed that mean peak opening and closing velocities for /b/ were significantly greater than those for /p/ during whispered speech. No differences in peak velocity for either oral closing or opening were observed during voiced speech. The maximum distance between the lips for oral opening for /b/ was significantly greater than for /p/ during whisper, whereas no difference was observed during voiced speech. These data supported the suggestion that whispered speech and voiced speech rely on distinct motor control processes.

Influences of stop consonant voicing on tongue, lip, and jaw movement kinematics

Lip closing movements for bilabial stops have been reported to be faster and of shorter duration for voiceless than for voiced stops. The experimental evidence is conflicting, however, and recordings have mostly been limited to lip and jaw movements in a single dimension. The present study examines articulatory kinematics in /aCV/ sequences, where the consonant is one of the set /p,b,t,d,k,g/ and the second vowel one of /i,a,u/. A magnetometer system was used to track vertical and horizontal movements of receivers placed on the lips and the jaw, and on four points on the tongue. Tangential velocity was used to define movement onsets and offsets. Movement amplitude was calculated as the path of the receiver from movement onset to offset. Preliminary results from two subjects suggest the possibility that the effects of consonant voicing on movement kinematics vary for different articulators. Tongue body movements towards consonantal closure had consistently higher velocity, larger amplitude and longer duration for voiced than for voiceless velar stops. Tongue tip and lip and jaw closing movements showed less robust differences between voiced and voiceless alveolar and labial stops. [Work supported by NIH.]