Lip Gestures Research Articles

Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration

This research introduces an innovative approach to enhance lip reading-based text extraction and translation through the integration of a double Convolutional Neural Network (CNN) coupled with Recurrent Neural Network (RNN) architecture. The proposed model aims to leverage the strengths of both CNN and RNN to achieve superior accuracy in lip movement interpretation and subsequent text extraction. The methodology involves training the double CNN+RNN model on extensive datasets containing synchronized lip movements and corresponding linguistic expressions. The initial layers of the model utilize CNNs to effectively capture spatial features from the visual input of lip images. The extracted features are then fed into RNN layers, allowing the model to grasp temporal dependencies and contextual information crucial for accurate lip reading. The trained model showcases its proficiency in extracting textual content from spoken words, demonstrating an advanced capability to decipher nuances in lip gestures. Furthermore, the extracted text undergoes a translation process, enabling the conversion of spoken language into various target languages. This research not only contributes to the advancement of lip reading technologies but also establishes a robust foundation for real-world applications such as accessibility solutions for individuals with hearing impairments, real-time multilingual translation services, and improved communication in challenging acoustic environments. The abstract concludes with a discussion on the potential impact of the double CNN+RNN model in pushing the boundaries of human-computer interaction, emphasizing the synergy between deep learning, lip reading, and translation technologies

Lip Movement Recognition using Histogram of Oriented Gradient (HOG) and Support Machine Vector (SVM) for Arabic Word

This research aims to develop a lip gesture recognition system in Arabic words by utilizing Histogram of Oriented Gradient (HOG) feature extraction and Support Vector Machine (SVM) classification. The evaluation was conducted on a dataset of 1749 videos with male and female participation using Modern Standard Arabic. The 10 cross-fold validation method was used to measure the performance of the system. By applying a polynomial kernel, this study achieved an accuracy rate of 95.63%, while the word recognition rate reached 96%. These results confirm the system's ability to recognize lip movements with precision, confirming the effectiveness of the approach used in visual recognition for Arabic.

Gestural timing in naïve listener imitation of Cantonese vowels

Studies of nonnative vowel production have analyzed sounds predominantly in terms of acoustics, while the underlying articulatory gestures, as well as their timing, remain understudied. This study examines nonnative production of backness and rounding contrasts in the Cantonese vowel sets /i y u/ and /ɛ œ ɔ/. 15 native Cantonese speakers and 15 native English speakers participated. Native English speakers read a list of English words containing the vowels [i u ɪ ʊ e o ɛ ɔ], and then imitated Cantonese words containing [i y u ɪ ʊ ɛœ ɵ ɔ], presented auditorily. L1 Cantonese speakers produced the same set of Cantonese words, presented orthographically. Dynamic acoustic, ultrasound, and lip video data were collected and analyzed using GAMMs. In L1 Cantonese, target tongue and lip positions are achieved at vowel onset and maintained throughout the vowel duration, yielding acoustically monophthongal vowels. In contrast, L1 English [u, o] exhibit peak rounding during the vowel offglide, while magnitude of tongue and F2 displacement vary according to consonantal context. Lip gestures show similar late timing in non-native imitations of Cantonese [y, u], yielding English-like F2 trajectories, although the two vowels differed in tongue position. These results suggest that acoustically non-native-like vowel productions may result in part from transfer of L1 gestural timing. [Work supported by Hong Kong RGC, No. 27614421.]

Analyzing lower half facial gestures for lip reading applications: Survey on vision techniques

Lip reading has gained popularity due to the proliferation of emerging real-world applications. This article provides a comprehensive review of benchmark datasets available for lip-reading applications and pioneering works that analyze lower facial cues for lip-reading applications. A comprehensive review of lip reading applications is broadly classified into five distinct applications: Lip Reading Biometrics (LRB), Audio Visual Speech Recognition (AVSR), Silent Speech Recognition (SSR), Voice from Lips, and Lip HCI (Human–computer interaction). LRB entails extensive research in the fields of authentication and liveness detection. AVSR covers key findings that have contributed significantly to applications such as voice assistants, video-to-text transcription, hearing aids, and pronunciation-correcting systems. SSR analyzes the efforts made for silent-video-to-text transcription and surveillance camera applications. The voice from lips section discusses applications such as voice for the voiceless and vision-infused speech inpainting. In lip HCI, LR-HCI for smartphones, smart TVs, computers, robots, and musical instruments is reviewed in detail. Comprehensive coverage is given to cutting-edge techniques in computer vision, signal processing, machine learning, and deep learning. The advancements that aid the system in learning to lip-read and authenticate lip gestures, generate text transcription, synthesize voice based on lip movements, and control systems via lip movements (lip HCI) are covered. The work concludes by highlighting the limitations of existing frameworks, the road maps of each application illustrating the evolution of techniques employed over time, and future research avenues in lip-reading applications.

Electromyogram‐Based Lip‐Reading via Unobtrusive Dry Electrodes and Machine Learning Methods (Small 17/2023)

Electromyogram-Based Lip-Reading Systems In article number 2205058, Petar M. Djurić, Shanshan Yao, and co-workers present the design of a truly natural and robust electromyogram-based lip-reading system that can capture speech-relevant lip gestures and decode lip movements for speech. The system allows for a mechanically imperceptible, skin-friendly, visually unobtrusive platform for tracking and interpreting lip movements with minimal interference.

Vowel space in hypokinetic dysarthria

The paper discusses acoustic and articulatory data on the use of vowel space by speaker affected by Parkinson’s Disease who developed hypokinetic dysarthria. Two experiments involving pathological subjects and matching controls are described, whose general aim is to better understand if the vowel space in Parkinson’s Disease dysarthric subjects is always and homogeneously reduced. In the first investigation, acoustic and kinematic data are collected and analyzed to test if pathological speakers always use a reduced vowel space compared to control subjects, and if they adopt different articulatory strategies depending on the axis of the speech gesture (vertical vs horizontal). In the second investigation, various articulatory metrics are used to better investigate the dimension and position of the acoustic vowel space, and if they change in Parkinson’s Disease subjects compared to controls. Results show that reduction takes place, but some subjects appear to compensate, widening their tongue gestures on the horizontal axis even though the lip gesture is not necessarily undershot. Nevertheless, metrics used in the second experiment do not allow to capture a reduction, even though, in line with results of the first experiment, they point to an asymmetry in the vowel space used depending on the axis considered.

Neuromotor Control of Speech and Speechlike Tasks: Implications From Articulatory Gestures

Purpose This study aimed to provide a preliminary examination of the articulatory control of speech and speechlike tasks based on a gestural framework and identify shared and task-specific articulatory factors in speech and speechlike tasks. Method Ten healthy participants performed two speechlike tasks (i.e., alternating motion rate [AMR] and sequential motion rate [SMR]) and three speech tasks (i.e., reading of “clever Kim called the cat clinic” at the regular, fast, and slow rates) that varied in phonological complexity and rate. Articulatory kinematics were recorded using an electromagnetic kinematic tracking system (Wave, Northern Digital Inc.). Based on the gestural framework for articulatory phonology, the gestures of tongue body and lips were derived from the kinematic data. These gestures were subjected to a fine-grained analysis, which extracted (a) four gestural features (i.e., range of magnitude [ROM], frequency [Freq], acceleration time, and maximum speed [maxSpd]) for the tongue body gesture; (b) three intergestural measures including the peak intergestural coherence (InterCOH), frequency at which the peak intergestural coherence occurs (Freq_InterCOH), and the mean absolute relative phase between the tongue body and lip gestures; and (c) three intragestural (i.e., interarticulator) measures including the peak intragestural coherence (IntraCOH), Freq_IntraCOH, and mean absolute relative phase between the tongue body and the jaw, which are the component articulators that underlie the tongue body gesture. In addition, the performance rate for each task was also derived. The effects of task and sex on all the articulatory and behavioral measures were examined using mixed-design analysis of variance followed by post hoc pairwise comparisons across tasks. Results Task had a significant effect on performance rate, ROM, Freq, maxSpd, InterCOH, Freq_InterCOH, IntraCOH, and Freq_IntraCOH. Compared to the speech tasks, the AMR task showed a decrease in ROM and increases in Freq, InterCOH, Freq_InterCOH, IntraCOH, and Freq_IntraCOH. The SMR task showed similar ROM, Freq, maxSpd, InterCOH, and IntraCOH as the fast and regular speech tasks. Conclusions The simple phonological structure and demand for rapid syllable rate for the AMR task may elicit a distinct articulatory control mechanism. Despite being a rapid nonsense syllable repetition task, the relatively complex phonological structure of the SMR task appeared to elicit a similar articulatory control mechanism as that of speech production. Based on these shared and task-specific articulatory features between speech and speechlike tasks, the clinical implications for articulatory assessment were discussed.

RFID Tattoo

This paper presents an RF-based assistive technology for voice impairments (i.e., dysphonia), which occurs in an estimated 1% of the global population. We specifically focus on acquired voice disorders where users continue to be able to make facial and lip gestures associated with speech. Despite the rich literature on assistive technologies in this space, there remains a gap for a solution that neither requires external infrastructure in the environment, battery-powered sensors on skin or body-worn manual input devices. We present RFTattoo, which to our knowledge is the first wireless speech recognition system for voice impairments using batteryless and flexible RFID tattoos. We design specialized wafer-thin tattoos attached around the user's face and easily hidden by makeup. We build models that process signal variations from these tattoos to a portable RFID reader to recognize various facial gestures corresponding to distinct classes of sounds. We then develop natural language processing models that infer meaningful words and sentences based on the observed series of gestures. A detailed user study with 10 users reveals 86% accuracy in reconstructing the top-100 words in the English language, even without the users making any sounds.

Alternative Visual Units for an Optimized Phoneme-Based Lipreading System

Lipreading is understanding speech from observed lip movements. An observed series of lip motions is an ordered sequence of visual lip gestures. These gestures are commonly known, but as yet are not formally defined, as ‘visemes’. In this article, we describe a structured approach which allows us to create speaker-dependent visemes with a fixed number of visemes within each set. We create sets of visemes for sizes two to 45. Each set of visemes is based upon clustering phonemes, thus each set has a unique phoneme-to-viseme mapping. We first present an experiment using these maps and the Resource Management Audio-Visual (RMAV) dataset which shows the effect of changing the viseme map size in speaker-dependent machine lipreading and demonstrate that word recognition with phoneme classifiers is possible. Furthermore, we show that there are intermediate units between visemes and phonemes which are better still. Second, we present a novel two-pass training scheme for phoneme classifiers. This approach uses our new intermediary visual units from our first experiment in the first pass as classifiers; before using the phoneme-to-viseme maps, we retrain these into phoneme classifiers. This method significantly improves on previous lipreading results with RMAV speakers.

Asymmetric discrimination of nonspeech tonal analogues of vowels.

Directional asymmetries reveal a universal bias in vowel perception favoring extreme vocalic articulations, which lead to acoustic vowel signals with dynamic formant trajectories and well-defined spectral prominences because of the convergence of adjacent formants. The present experiments investigated whether this bias reflects speech-specific processes or general properties of spectral processing in the auditory system. Toward this end, we examined whether analogous asymmetries in perception arise with nonspeech tonal analogues that approximate some of the dynamic and static spectral characteristics of naturally produced /u/ vowels executed with more versus less extreme lip gestures. We found a qualitatively similar but weaker directional effect with 2-component tones varying in both the dynamic changes and proximity of their spectral energies. In subsequent experiments, we pinned down the phenomenon using tones that varied in 1 or both of these 2 acoustic characteristics. We found comparable asymmetries with tones that differed exclusively in their spectral dynamics, and no asymmetries with tones that differed exclusively in their spectral proximity or both spectral features. We interpret these findings as evidence that dynamic spectral changes are a critical cue for eliciting asymmetries in nonspeech tone perception, but that the potential contribution of general auditory processes to asymmetries in vowel perception is limited. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Comparing heterogeneous visual gestures for measuring the diversity of visual speech signals

Visual lip gestures observed whilst lipreading have a few working definitions, the most common two are: ‘the visual equivalent of a phoneme’ and ‘phonemes which are indistinguishable on the lips’. To date there is no formal definition, in part because to date we have not established a two-way relationship or mapping between visemes and phonemes. Some evidence suggests that visual speech is highly dependent upon the speaker. So here, we use a phoneme-clustering method to form new phoneme-to-viseme maps for both individual and multiple speakers. We test these phoneme to viseme maps to examine how similarly speakers talk visually and we use signed rank tests to measure the distance between individuals. We conclude that broadly speaking, speakers have the same repertoire of mouth gestures, where they differ is in the use of the gestures.

Three-dimensional ultrasound images of Polish high front vowels

The 3-D ultrasound method has been applied to collect data on Polish high front vowels. In particular, Polish has one unambiguous high front vowel and another one that in the phonological literature is variously referred to as high central or back unrounded and transcribed as [ɨ]. While there exists a sizeable body of articulatory research on Polish, including X-rays from as early as the 50s and 60s, the ultrasound data reveal more detail about the position of the tongue center and tongue root. The data evaluated so far support the view that the vowel transcribed as [ɨ] is a front vowel. The two front vowels differ in the position of the tongue root, relative raising of the tongue, and extent of lip gesture, but do not differ substantially with regard to tongue body advancement on the font-back axis. The data also capture the temporal aspect of speech, and together with time-aligned audio recordings and video recordings of the lips, allow for fine-grained analysis of the acoustic effects of these articulatory gestures.

Imitation of Non-Speech Oral Gestures by 8-Month-Old Infants.

This study investigates the oral gestures of 8-month-old infants in response to audiovisual presentation of lip and tongue smacks. Infants exhibited more lip gestures than tongue gestures following adult lip smacks and more tongue gestures than lip gestures following adult tongue smacks. The findings, which are consistent with predictions from Articulatory Phonology, imply that 8-month-old infants are capable of producing goal-directed oral gestures by matching the articulatory organ of an adult model.

Phonological Features in the Qur’anic Seven Modes of Reading (Qira’āt)

The paper investigates the phonological features of the seven modes of reading the Qur’an. The source of the modes of reading is attributed primarily to the Prophet Muhammad. However, the feeling of who is more accurate in his reading than the others among Muslim Qur’an reciters has led to the abrogation of all personal codices which are replaced by the official Uthmanic canonical codex. The marathon journey of the modes of reading has started during the 1 st /7 th century in Medina, Makkah, Kufah, Basra, and Damascus. Variant modes of reading are either phonetically-oriented, or semantically-oriented. The phonetically-oriented mode of reading involves vocalic and diacritic differences. However, the semantically-oriented mode of reading focuses on within-the-text exegetical expressions. The major differences among the Qur’an reciters can be classified into: (i) differences in consonants, and (ii) differences in vowels. Differences in consonants involve assimilation, almutamāthilain (two adjacent sounds) , tashīl alhamza (pronouncing the glottal stop between glottal stop and the glottal fricative), alibdal (substitution), alqalb almakāni (metathesis), alnaql (the process of pronouncing the glottal stop [Ɂ] as an open vowel [a]), alta’āt (the voiceless alveolar plosive) [t]), alra’āt (the voiced alveolar trill [r] and the voiced alveolar tap [ɾ]), and allāmāt (the voiced alveolar laterals). However, differences in vowels include alimālah (the deflection of particular vowels from the fathah towards the kasrah or alif towards the yaa' sound), al-ishmām (the rounding of lips without any sound at the same time of producing the unrounded vowel), al-ikhtilās (the close vowel is reduced to mid-close vowel), alrawm (the process of lip gesture alluding to the vowel in the pause position with an extra short vowel [ɪ], [a], or [ŭ]), vowel lengthening, vowel elision, and vowel change.

Lip and Head Gesture Recognition Based PC Interface using Image Processing

Lip and Head Gesture Recognition Based PC Interface using Image Processing

A Biomechanical Modeling Study of the Effects of the Orbicularis Oris Muscle and Jaw Posture on Lip Shape

The authors' general aim is to use biomechanical models of speech articulators to explore how possible variations in anatomical structure contribute to differences in articulatory strategies and phone systems across human populations. Specifically, they investigated 2 issues: (a) the link between lip muscle anatomy and variability in lip gestures and (b) the constraints of coupled lip/jaw biomechanics on jaw posture in labial sounds. The authors used a model coupling the jaw, tongue, and face. First, the influence of the orbicularis oris (OO) anatomical implementation was analyzed by assessing how changes in depth (from epidermis to the skull) and peripheralness (proximity to the lip horn center) affected lip shaping. Second, the capability of the lip/jaw system to generate protrusion and rounding, or labial closure, was evaluated for different jaw heights. Results showed that a peripheral and moderately deep OO implementation is most appropriate for protrusion and rounding; a superficial implementation facilitates closure; protrusion and rounding require a high jaw position; and closure is achievable for various jaw heights. Models provide objective information regarding possible links between anatomical and speech production variability across humans. Comparisons with experimental data will illustrate how motor control and cultural factors cope with these constraints.

Anticipatory lip gestures: A validation of the Movement Expansion Model in congenitally blind speakers

In this paper, anticipatory co-articulation of the lip protrusion and constriction gestures is investigated in speakers with visual deprivation. Audio-visual recordings of 11 congenitally blind French speakers producing [V-roundC-roundV+round] sequences were measured with a lip shape tracking system. Lip protrusion and constriction values and their relative timings were analyzed. Results show that despite the reduced magnitude of lip protrusion and constriction area in blind speakers, the timing of the anticipatory gestures can be appropriately modeled by the Movement Expansion Model [from Abry and Lallouache (1995a). Bul. de la Comm. Parlée 3, 85-99; (1995b). Proceedings of ICPHS, pp. 152-155; Noiray et al. (2011). J. Acoust. Soc. Am. 129, 340-349], which predicts lawful anticipatory behavior expanding linearly as the intervocalic consonant interval increases.

The effect of lip-reading on primary stream segregation

Lip-reading has been shown to improve the intelligibility of speech in multitalker situations, where auditory stream segregation naturally takes place. This study investigated whether the benefit of lip-reading is a result of a primary audiovisual interaction that enhances the obligatory streaming mechanism. Two behavioral experiments were conducted involving sequences of French vowels that alternated in fundamental frequency. In Experiment 1, subjects attempted to identify the order of items in a sequence. In Experiment 2, subjects attempted to detect a disruption to temporal isochrony across alternate items. Both tasks are disrupted by streaming, thus providing a measure of primary or obligatory streaming. Visual lip gestures articulating alternate vowels were synchronized with the auditory sequence. Overall, the results were consistent with the hypothesis that visual lip gestures enhance segregation by affecting primary auditory streaming. Moreover, increases in the naturalness of visual lip gestures and auditory vowels, and corresponding increases in audiovisual congruence may potentially lead to increases in the effect of visual lip gestures on streaming.

Effects of audio–visual integration on the detection of masked speech and non-speech sounds

Integration of simultaneous auditory and visual information about an event can enhance our ability to detect that event. This is particularly evident in the perception of speech, where the articulatory gestures of the speaker’s lips and face can significantly improve the listener’s detection and identification of the message, especially when that message is presented in a noisy background. Speech is a particularly important example of multisensory integration because of its behavioural relevance to humans and also because brain regions have been identified that appear to be specifically tuned for auditory speech and lip gestures. Previous research has suggested that speech stimuli may have an advantage over other types of auditory stimuli in terms of audio–visual integration. Here, we used a modified adaptive psychophysical staircase approach to compare the influence of congruent visual stimuli (brief movie clips) on the detection of noise-masked auditory speech and non-speech stimuli. We found that congruent visual stimuli significantly improved detection of an auditory stimulus relative to incongruent visual stimuli. This effect, however, was equally apparent for speech and non-speech stimuli. The findings suggest that speech stimuli are not specifically advantaged by audio–visual integration for detection at threshold when compared with other naturalistic sounds.

Face models based on a guided PCA of motion-capture data: speaker dependent variability in /s/-/R/ contrast production

We measure face deformations during speech production using a motion capture system, which provides 3D coordinate data of about 60 markers glued on the speaker's face. An arbitrary orthogonal factor analysis followed by a principal component analysis (together called a guided PCA) of the data has showed that the first 6 factors explain about 90% of the variance, for each of our 3 speakers. The 6 derived factors, therefore, allow us to efficiently analyze or to reconstruct with a reasonable accuracy the observed face deformations. Since these factors can be interpreted in articulatory terms, they can reveal underlying articulatory organizations. The comparison of lip gestures in terms of data derived factors suggests that these speakers differently maneuver the lips to achieve contrast between /s/ and /R/. Such inter-speaker variability can occur because the acoustic contrast of these fricatives is shaped not only by the lip tube but also by cavities inside the mouth such as the sublingual cavity. In other words, these tube and cavity can acoustically compensate each other to produce their required acoustic properties.