Lipreading Research Articles

Persuasion in Social Speeches

The study of the political speeches has drawn the attention of scholars since its revelation up to the present day. Many scholars have attempted to study specific features of it. One of these features that has been at the core of interest is persuasion. This research was aimed to elaborate on the concept of persuasion in the political speeches and to find out the persuasive strategies used in the speeches of queen Rania (queen of the Hashemite kingdom of Jordan) The study has confined itself to three main objectives. These are discussing the concept of persuasion as manifested in the political speeches, Identifying the persuasive strategies used, and naming the most common and the least common ones. By using a model based on persuasive communication theory and other related concepts, researcher analyzed the data using quantitative-qualitative research method. The analysis started with a comprehensive reading of queen Rania speech and then identifying the tokens of persuasive strategies. The results were put in numerical form and discussed both quantitively and qualitatively. The study findings show that persuasion is one of the basic concepts in the political speeches due to the nature of this text which aimed at persuading people to follow the righteous course of action and belief. It is also found that all the persuasive strategies included in the model appeared in the text under study. Finally, the study revealed that metaphor is the most commonly used persuasive strategies while personification is the least.

Intelligent event-based lip reading word classification with spiking neural networks using spatio-temporal attention features and triplet loss

Lip reading is a visual alternative to enhance the intelligence of traditional speech recognition, which can benefit from retina-like event cameras that focus on dynamic movements. Spiking Neural Networks (SNNs) are inherently well-suited to cooperate with event cameras. However, employing SNN for event-based lip reading presents significant challenges, particularly in effectively extracting spatio-temporal features from input events and distinguishing between phonetically similar elements. To address these challenges, this paper proposes a novel event-based lip-reading model that leverages the SNN framework, enriched by a designed Spatial-Temporal Attention Block (STAB) and a triplet loss. Specifically, STAB comprises both spatial and temporal attention branches, dynamically emphasizing those spatial and temporal characteristics most reflective of lip movements. STAB also includes a fusion mechanism to integrate these spatial and temporal insights, providing a comprehensive and focused representation of lip movements. In addition, we enhance the model by incorporating triplet loss into the SNN training process, further improving the model's ability to distinguish between visually similar words. Experimental results show the superior performance of our SNN and validate the effect of STAB and triplet loss. We also conduct the energy consumption analysis to affirm the model's energy efficiency.

Lip Reading using computer vision techniques and deep learning algorithms for Deaf and Dumb People

Lip Reading using computer vision techniques and deep learning algorithms for Deaf and Dumb People

Long-term Acoustic Effects of Gender-Affirming Voice Training in Transgender Women

ObjectivesOne role of a speech-language pathologist (SLP) is to help transgender clients in developing a healthy, gender-congruent communication. Transgender women frequently approach SLPs to train their voices to sound more feminine, however, long-term acoustic effects of the training needs to be rigorously examined in effectiveness studies. The aim of this study was to investigate the long-term effects (follow-up 1: 3 months and follow-up 2: 1 year after last session) of gender-affirming voice training for transgender women, in terms of acoustic parameters. Study designThis study was a randomized sham-controlled trial with a cross-over design. Methods26 transgender women were included for follow-up 1 and 18 for follow-up 2. All participants received 14 weeks of gender-affirming voice training (4 weeks sham training, 10 weeks of voice feminization training: 5 weeks pitch elevation training (PET) and 5 weeks articulation-resonance training (ART)), but in a different order. Speech samples were recorded with Praat at 4 different time points (pre, post, follow-up 1, follow-up 2). Acoustic analysis included fo of sustained vowel /a:/, reading and spontaneous speech. Formant frequencies (F1-F2-F3) of vowels /a/, /i/ and /u/ were determined and vowel space was calculated. A linear mixed model was used to compare the acoustic voice measurements between measurements (pre – post, pre – follow-up 1, pre – follow-up 2, post – follow-up 1, post – follow-up 2, follow-up 1 – follow-up 2). ResultsMost of the fo measurements and formant frequencies that increased immediately after the intervention, were stable at both follow-up measurements. The median fo during the sustained vowel, reading and spontaneous speech stayed increased at both follow-ups compared to the pre measurement. However, a decrease of 16 Hz/1.7 ST (reading) and 12 Hz/1.5 ST (spontaneous speech) was detected between the post measurement (169 Hz for reading, 144 Hz for spontaneous speech) and one year after the last session (153 Hz and 132 Hz respectively). The lower limit of fo did not change during reading and spontaneous speech, both directly after the intervention and during both follow-ups. F1-2 of vowel /a/ and the vowel space increased after the intervention and both follow-ups. Individual analyses showed that more aspects should be controlled after the intervention, such as exercises that were performed at home, or the duration of extra gender-affirming voice training sessions. ConclusionsAfter ten sessions of voice feminization training and follow-up measurements after three months and one year, stable increases were found for some formant frequencies and fo measurements, but not all of them. More time should be spent on increasing the fifth percentile of fo, as the lower limit of fo also contributes to the perception of more feminine voice.

Acoustic-Based Lip Reading for Mobile Devices: Dataset, Benchmark and a Self Distillation-Based Approach

Acoustic-Based Lip Reading for Mobile Devices: Dataset, Benchmark and a Self Distillation-Based Approach

Daffawi: self-Orientalism and identity work among Palestinians in Israel

ABSTRACT This ethnographic paper explores attitudes and perceptions of Palestinian citizens of Israel (PCI) regarding Palestinians from the West Bank (PWB). It focuses on the semantic complex of ḍaffāwi (‘who comes from the West Bank’), which entails a combination of cultural superiority, distrust, romanticizing, and ambivalence. Treating the ḍaffāwi as a speech act and reading it as a cultural text that is rooted in PCI’s collective political unconscious, we argue that their mixed attitudes of superiority and idealization with regard to WBP, and by implication their self-Orientalism, reflect a deep sense of deadlock that characterizes their identity work, as indigenous citizens of an ethnocratic state that has been colonizing their people.

SilentInterpreter: Analysis of Lip Movement and Extracting Speech Using Deep Learning

Lip reading presents a captivating avenue for advancing speech recognition algorithms, leveraging visual cues from lip movements to recognise spoken words. This paper introduces a novel method employing deep neural networks to convert lip motions into textual representations. The methodology integrates convolutional neural networks for visual feature extraction, recurrent neural networks to capture temporal context, and the Connectionist Temporal Classification loss function for aligning lip features with phonemes. Additionally, dynamic learning rate scheduling and a unique callback mechanism for training visualization are incorporated into the process. Post-training on a sizeable dataset, the model demonstrates notable convergence, showcasing its ability to discern intricate temporal correlations. Comprehensive evaluations, combining quantitative metrics and qualitative assessments, validate the model's effectiveness. Visual inspections of lip reading capabilities and standard speech recognition criteria evaluation highlight its performance. The study delves into the impact of various model topologies and hyperparameters on performance, providing valuable insights for future research directions. This research contributes a deep learning framework for accurate and efficient speech recognition, expanding the landscape of lip reading technologies. The findings open paths for further refinement and deployment across diverse domains, including assistive technologies, audio-visual communication systems, and human-computer interaction. Keywords: lip reading, deep learning, convolutional neural networks, recurrent neural networks, CTC loss, speech recognition

Survey on Silentinterpreter : Analysis of Lip Movement and Extracting Speech using Deep Learning

Lip reading is a complex but interesting path for the growth of speech recognition algorithms. It is the ability of deciphering spoken words by evaluating visual cues from lip movements. In this study, we suggest a unique method for lip reading that converts lip motions into textual representations by using deep neural networks. Convolutional neural networks are used in the methodology to extract visual features, recurrent neural networks are used to simulate temporal context, and the Connectionist Temporal Classification loss function is used to align lip features with corresponding phonemes. The study starts with a thorough investigation of data loading methods, which include alignment extraction and video preparation. A well selected dataset with video clips and matching phonetic alignments is presented. We select relevant face regions, convert frames to grayscale, then standardize the resulting data so that it can be fed into a neural network. The neural network architecture is presented in depth, displaying a series of bidirectional LSTM layers for temporal context understanding after 3D convolutional layers for spatial feature extraction. Careful consideration of input shapes, layer combinations, and parameter selections forms the foundation of the model's design. To train the model, we align predicted phoneme sequences with ground truth alignments using the CTC loss. Dynamic learning rate scheduling and a unique callback mechanism for training visualization of predictions are integrated into the training process. After training on a sizable dataset, the model exhibits remarkable convergence and proves its capacity to understand intricate temporal correlations. Through the use of both quantitative and qualitative evaluations, the results are thoroughly assessed. We visually check the model's lip reading abilities and assess its performance using common speech recognition criteria. It is explored how different model topologies and hyperparameters affect performance, offering guidance for future research. The trained model is tested on external video samples to show off its practical application. Its accuracy and resilience in lip-reading spoken phrases are demonstrated. By providing a deep learning framework for precise and effective speech recognition, this research adds to the rapidly changing field of lip reading devices. The results offer opportunities for additional development and implementation in various fields, such as assistive technologies, audio-visual communication systems, and human-computer interaction.

Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration

This research introduces an innovative approach to enhance lip reading-based text extraction and translation through the integration of a double Convolutional Neural Network (CNN) coupled with Recurrent Neural Network (RNN) architecture. The proposed model aims to leverage the strengths of both CNN and RNN to achieve superior accuracy in lip movement interpretation and subsequent text extraction. The methodology involves training the double CNN+RNN model on extensive datasets containing synchronized lip movements and corresponding linguistic expressions. The initial layers of the model utilize CNNs to effectively capture spatial features from the visual input of lip images. The extracted features are then fed into RNN layers, allowing the model to grasp temporal dependencies and contextual information crucial for accurate lip reading. The trained model showcases its proficiency in extracting textual content from spoken words, demonstrating an advanced capability to decipher nuances in lip gestures. Furthermore, the extracted text undergoes a translation process, enabling the conversion of spoken language into various target languages. This research not only contributes to the advancement of lip reading technologies but also establishes a robust foundation for real-world applications such as accessibility solutions for individuals with hearing impairments, real-time multilingual translation services, and improved communication in challenging acoustic environments. The abstract concludes with a discussion on the potential impact of the double CNN+RNN model in pushing the boundaries of human-computer interaction, emphasizing the synergy between deep learning, lip reading, and translation technologies

Automatic speaker and age identification of children from raw speech using sincNet over ERB scale

This paper presents the newly developed non-native children’s English speech (NNCES) corpus to reveal the findings of automatic speaker and age recognition from raw speech. Convolutional neural networks (CNN), which have the ability to learn low-level speech representations, can be fed directly with raw speech signals instead of using traditional hand-crafted features. Moreover, the filters that were learned using standard CNNs appeared to be noisy because they consider all elements of each filter. In contrast, sincNet can be able to generate more meaningful filters simply by replacing the first convolutional layer by a sinc-layer in standard CNNs. The low and high cutoff frequencies of the rectangular band-pass filter are the only parameters that can be learned in sincNet, which has the potential to extract significant speech cues from the speaker, such as pitch and formants. In this work, the sincNet model is significantly changed by switching from baseline Mel scale initializations to equivalent rectangular bandwidth (ERB) initializations, which has the added benefit of allocating additional filters in the lower region of the spectrum. Additionally, it needs to be highlighted that the novel sincNet model is well suited to identify the age of the children. The investigations on both read and spontaneous speech tasks in speaker identification, gender independent & dependent age-group identification of children outperform the baseline models with varying relative improvements in terms of accuracy.

Analysis of Model Progression and Parity in Personalized Modeling of Lipsync Decoder to Cater Hearing Impaired Individuals

The Analysis of model progression and parity in personalized modeling of Lipsync decoder to cater hearing impaired individuals aims to enhance communication accessibility for hearing-impaired individuals by understanding flaws in and helping bring light to them translating visual cues from lip movements into intelligible speech through advancements in machine learning and computer vision. This paper delves into the analysis of model progression and parity in personalized modeling of Lipsync decoder technology to enhance communication accessibility for hearing-impaired individuals. By implementing the LipNet model through meticulous data collection, customization, and rigorous testing, the study aims to address challenges in real-time lip sync probe decoding. Results reveal a variation in word error rate when tested upon different video formats, highlighting the consistency, resilience, and potential for personalized modeling to improve communication accessibility for individuals with hearing impairments. The research underscores the critical importance of accurate lip reading and real-time processing in achieving seamless translation of lip movements into understandable speech, emphasizing the necessity for precision and efficiency in communication technologies for the hearing-impaired community. Keywords: LipSyncProbe Decoder, Hearing-impaired individuals, real-time feed, LipNet model, datasets.

Exposure to Canadian French Cued Speech Improves Consonant Articulation in Children With Cochlear Implants: Acoustic and Articulatory Data.

One of the strategies that can be used to support speech communication in deaf children is cued speech, a visual code in which manual gestures are used as additional phonological information to supplement the acoustic and labial speech information. Cued speech has been shown to improve speech perception and phonological skills. This exploratory study aims to assess whether and how cued speech reading proficiency may also have a beneficial effect on the acoustic and articulatory correlates of consonant production in children. Eight children with cochlear implants (from 5 to 11 years of age) and with different receptive proficiency in Canadian French Cued Speech (three children with low receptive proficiency vs. five children with high receptive proficiency) are compared to 10 children with typical hearing (from 4 to 11 years of age) on their production of stop and fricative consonants. Articulation was assessed with ultrasound measurements. The preliminary results reveal that cued speech proficiency seems to sustain the development of speech production in children with cochlear implants and to improve their articulatory gestures, particularly for the place contrast in stops as well as fricatives. This work highlights the importance of studying objective data and comparing acoustic and articulatory measurements to better characterize speech production in children.

Speech Fluency Production and Perception in L1 (Slovak) and L2 (English) Read Speech.

Research on fluency in native (L1) and non-native (L2) speech production and perception helps us understand how individual L1 speaking style might affect perceived L2 fluency and how this relationship might be reflected in L1 versus L2 oral assessment. While the relationship between production and perception of fluency in spontaneous speech has been studied, the information provided by reading has been overlooked. We argue that reading provides a direct and controlled way to assess language proficiency that might complement information gained from spontaneous speaking. This work analyzes the relationship between speech fluency production and perception in passages of L1 (Slovak) and L2 (English) read by 57 undergraduate Slovak students of English and rated for fluency by 15 English teachers who are Slovak natives. We compare acoustic production measures between L1 and L2 and analyze how their effect on perceived fluency differs for the two languages. Our main finding is that the articulation rate, the overall number of pauses, and the number of between-clause and mid-clause pauses predict ratings differently in L1 Slovak versus L2 English. The speech rate and durations of pauses predict ratings similarly in both languages. The contribution of our results to understanding fluency aspects of spontaneous and read speech, the relationship between L1 and L2, the relationship between production and perception, and to the teaching of L2 English are discussed.

Differences in prominence-related acoustic measurements in L2 read speech as a function of L1 framing typology

Speech production in a non-native language is often influenced by their first language’s (L1) phonology. One factor that might affect their production prosodically is typological differences in event construal, specifically motion events. Building on this concept, prior studies on gestures and spontaneous speech show that how speakers frame motion events in their L1 can transfer to their second language (L2) through grammatical patterns. However, no study has looked at how L1-L2 differences in framing motion events may affect speakers’ L2 production prosodically. To explore whether those differences affect how L2 speakers prosodically produce motion events, this study collected read speech from two spoken corpora of L2 English learners. We chose L2 learners from different L1 backgrounds that differ in the way they grammatically construct motion events. English production from Korean and Turkish L1 speakers (which are typologically similar) were compared to that of L1 German speakers (which is typologically similar to English), along with productions by L1 English speakers. Duration and various acoustic measurements of pitch and intensity were extracted and analyzed. Results indicated some similarities among learners of the same typology along with differences between the L1 and L2 English speakers. Implications on conceptual transfer will be discussed.

Preceding word information for predicting speech errors in English as foreign language speech

Speech errors, including disfluency errors (e.g., filled pauses (“uh”, “um”), repetition (“I me-mean right now.”)), and mispronunciation of speech segments (e.g., “think” as /sɪŋk/) are natural occurrences in speech production and they can affect speech fluency and proficiency. Detecting these errors is important, especially in assessing second language (L2) learners. Non-native speakers often produce speech errors, even in read speech, due to increased cognitive load when simultaneously producing the current word and processing the upcoming word. By analyzing two L2 speech corpora having different types of errors (disfluency and mispronunciation), the study aims to assess the effectiveness of incorporating preceding word information in speech error prediction. The results indicate that longer syllable duration in the word preceding the mispronunciation error correlates with error production. The study extends its investigation to identify acoustic and lexical features influenced by the presence of speech errors. The study found that speech errors increase duration of speech. The study also found that words with more syllables may cause L2 speakers to produce speech errors.

Auditory brainstem implantation: surgical experience and audiometric outcomes in the pediatric population.

Pediatric data regarding treatment via an auditory brainstem implant (ABI) remains sparse. The authors aimed to describe their experience at their institution and to delineate associated demographic data, audiometric outcomes, and surgical parameters. An IRB-approved, retrospective chart review was conducted among the authors' pediatric patients who had undergone auditory brainstem implantation between 2012 and 2021. Demographic information including sex, age, race, coexisting syndrome(s), history of cochlear implant placement, average duration of implant use, and follow-up outcomes were collected. Surgical parameters collected included approach, intraoperative findings, number of electrodes activated, and complications. A total of 19 pediatric patients had an ABI placed at the authors' institution, with a mean age at surgery of 4.7 years (range 1.5-17.8 years). A total of 17 patients (89.5%) had bilateral cochlear nerve aplasia/dysplasia, 1 (5.3%) had unilateral cochlear nerve aplasia/dysplasia, and 1 (5.3%) had a hypoplastic cochlea with ossification. A total of 11 patients (57.9%) had a history of cochlear implants that were ineffective and required removal. The mean length of implant use was 5.31 years (0.25-10 years). Two patients (10.5%) experienced CSF-related complications requiring further surgical intervention. The most recent audiometric outcomes demonstrated that 15 patients (78.9%) showed improvement in their hearing ability: 5 with sound/speech awareness, 5 able to discriminate among speech and environmental sounds, and 5 able to understand common phrases/conversation without lip reading. Nine patients (47.4%) are in a school for the deaf and 7 (36.8%) are in a mainstream school with support. The authors' surgical experience with a multidisciplinary team demonstrates that the retrosigmoid approach for ABI placement in children with inner ear pathologies and severe sensorineural hearing loss is a safe and effective treatment modality. Audiometric outcome data showed that nearly 79% of these patients had an improvement in their environmental and speech awareness. Further multicenter collaborations are necessary to improve these outcomes and potentially standardize/enhance electrode placement.

Comparison of automated and manual measures of speaking rate in individuals with dysarthria

Speech patterns in speakers with dysarthria exhibit a broad range of prosodic and articulatory variations, potentially posing a significant challenge for automated speech recognition systems. This study seeks to address a gap in the literature by exploring a fully automated approach to determine speaking rates in English-speaking individuals with dysarthria. This research aims to elucidate whether a syllable-based automated method yields results akin to manual processes when calculating speaking rates across diverse groups [i.e., individuals with dysarthria secondary to multiple sclerosis (MS) or Parkinson’s disease (PD), and healthy controls]. Moreover, this study examines the performance of an automated method across three distinct speech tasks: isolated sentences, paragraph reading, and spontaneous speech. The methodology involves adjusting parameters within a version of a published script specifically designed for calculating speaking rate through syllable detection (Praat Script Syllable Nuclei; de Jong et al., 2021). Using a sample of 60 speakers (20 MS, 20 PD, and 20 healthy controls), this study aims to evaluate the accuracy and validity of an automated script in calculating speaking rate. Ultimately, these findings are intended to guide speaking rate measurement of dysarthria in research and clinical practice.

Comparing face-tracking action units with electromyography during speech

Video face-tracking software, such as OpenFace 2.0, can be used to make inferences about facial muscle activation [Baltrušaitis et al. IEEE 13, 59–66 (2018)]. However, the accuracy of these inferences based on the facial action units (FAUs) calculated by OpenFace 2.0 compared to corresponding muscle activity during speech is unclear. A previous study investigated muscle activation when both smile and speech occur simultaneously, focusing on the zygomaticus major (ZM) muscle [Liu et al. ISSP, 130–133 (2021)], but only presented data for a single speaker and did not compare FAU and EMG results. The present study compares OpenFace 2.0 action units with surface electromyography (EMG) data during speech in order to assess the validity of these inferences about facial muscle activation. We compare ZM activity and the lip corner puller FAU intensity results from a dataset collected for the previously mentioned [Liu et al. 2021] study. Data include four speakers producing read speech in smile conditions; 2 s will be extracted from EMG and FAU data before and after each utterance. Results will be reported on the relative accuracy of the FAU and EMG data. Implications will be discussed for speech communication research. [Work supported by NSERC.]

Acknowledging the Social Bases of Speech Intelligibility: A Key to Improving Equity in Psychological and Physiological Acoustics

A major endeavor in psychological and physiological (P&P) acoustics is understanding the causes and consequences of hearing loss. One of the most common complaints of individuals with hearing loss is difficulty perceiving speech. Hence, many studies in P&P compare psychophysical or physiological measures of auditory perception with performance on speech perception tasks like sentence intelligibility. Sentence intelligibility tasks often use read speech samples produced by a small number of demographically unspecified talkers, with only limited exceptions (i.e., McCloy, Wright, & Souza, 2015; Wright & Souza, 2012). This talk reviews findings from the author’s and others’ research groups showing that sentence intelligibility varies as a function of talkers’ social identities, including a talker’s actual or perceived racial identity (Babel and Russell, 2015; McGowan, 2015; Tripp, Lyons, and Munson, 2022). These findings reinforce recent calls for P&P to revise the intelligibility measures used to characterize the consequences of hearing loss (i.e., Beechey, 2022a, 2022b). In particular, this talk argues for a collective research program developing entirely new speech intelligibility measures that reflect diverse ways of speaking, and the diverse functions of spoken language. [Work funded by NIH grant R21 DC018070.]

Social experiences of deaf and hard‐of‐hearing students at a Saudi Arabian university: A qualitative study

AbstractThis study explored the social experiences of 12 deaf and hard‐of‐hearing female university students. Semi‐structured interviews identified four key themes: university environment, peer interactions, communication and personal barriers. The majority of participants described their social experiences as positive and better than their experiences at school. The participants experienced collaboration, support, peer acceptance, friendship and high awareness of hearing disability among their hearing peers. Communication with peers was facilitated by using applications, sign language and lip reading. However, a lack of understanding from faculty members, isolation, and ignorance exhibited by some peers resulted in negative experiences for a few participants. Some participants reported personal barriers, such as personal characteristics, past negative experiences, and fear of disclosing their disability. Given the importance of social support and inclusion for students' success and well‐being, universities must prioritise the social experiences of deaf and hard‐of‐hearing students as much as their academic experiences.