Lipreading Research Articles

A multi-purpose audio-visual corpus for multi-modal Persian speech recognition: The Arman-AV dataset

Automatic lip reading has advanced significantly in recent years. However, these methods need large-scale datasets that are scarce for many low-resource languages. In this paper, we introduce a new multipurpose audio-visual dataset for Persian. The dataset contains approximately 220 h of videos from 1760 speakers. The dataset can be used for multiple tasks, such as lip reading, automatic speech recognition, audio-visual speech recognition, and speaker recognition. It is also the first large-scale lip reading dataset in this language. We provide a baseline method for each task and propose a technique to identify visemes (visual units of speech) in Persian. The visemes obtained by this technique improve the accuracy of the lip reading task by 7% relatively compared to the previously proposed visemes, which can be generalized to other languages as well.

Social perception in deaf individuals: A meta-analysis of neuroimaging studies.

Deaf individuals may report difficulties in social interactions. However, whether these difficulties depend on deafness affecting social brain circuits is controversial. Here, we report the first meta-analysis comparing brain activations of hearing and (prelingually) deaf individuals during social perception. Our findings showed that deafness does not impact on the functional mechanisms supporting social perception. Indeed, both deaf and hearing control participants recruited regions of the action observation network during performance of different social tasks employing visual stimuli, and including biological motion perception, face identification, action observation, viewing, identification and memory for signs and lip reading. Moreover, we found increased recruitment of the superior-middle temporal cortex in deaf individuals compared with hearing participants, suggesting a preserved and augmented function during social communication based on signs and lip movements. Overall, our meta-analysis suggests that social difficulties experienced by deaf individuals are unlikely to be associated with brain alterations but may rather depend on non-supportive environments.

Responsiveness, Sensitivity and Clinical Utility of Timing-Related Speech Biomarkers for Remote Monitoring of ALS Disease Progression.

In this study, we describe the responsiveness of timing-related measures extracted from read speech in persons with ALS (pALS) collected via a remote patient monitoring platform in an effort to quantify how long it takes to detect a clinically-meaningful change associated with disease progression. We found that the timing alignment of pALS speech relative to a canonical elicitation of the same prompt is the most responsive measure, of the ones considered in this study, at detecting such change in both pALS with bulbar (n = 35) and non-bulbar onset (n = 94). We further evaluated the sensitivity of speech metrics in tracking disease progression in pALS while their ALSFRS-R speech score remained unchanged at 3 out of a total possible score of 4. We observed that timing-related speech metrics showed significant longitudinal changes even after accounting for learning effects. The findings of this study have the potential to inform disease prognosis and functional outcomes of clinical trials.

Questioning questions - the perception of f0 scaling in German questions between categorical function and continuous attitude.

In previous studies comparing the intonation of questions and statements in German, greater f0 excursions of phrase-final rises have been associated with questions in both read speech and spontaneous speech. This holds for production studies as well as perception studies. However, a major question remains whether these differences are perceived categorically or continuously. Furthermore, we ask whether the differences in f0 scaling correspond to categorical linguistic functions or rather an attitudinal continuum. We conducted three different perception experiments: a classical categorical perception task, an imitation task, and a semantic evaluation task. The results suggest that f0 scaling in phrase-final rises is perceived as a phonetic continuum rather than in phonological categories. Furthermore, the gradual increase of the final rise is associated with a gradual increase in perceived questioning. Lastly, the phonetic cues to this degree of questioning are distinct from those to the other investigated meanings surprise and uncertainty. Accordingly, this study supports the assumption that questioning constitutes an attitudinal meaning in its own right.

Work-Related Vocal Challenges of Online Teaching: A Study of Female College Professors

College professors are at an elevated risk of developing voice disorders due to increased vocal demands associated with their working conditions, including high background noise levels, long reverberation times, prolonged voice use duration, and large class sizes. However, the impact of online classes on vocal health among college professors remains unclear. This longitudinal study aimed to characterize voice changes associated with online teaching among female Colombian college professors. Thirteen female college professors teaching online classes completed an online survey that included questions about voice functioning, working conditions, the Spanish translation of the Vocal Fatigue Index (VFI), and the Spanish translation of the Voice-Related Quality of Life. Instrumental measurements were also performed, including environmental background noise levels, hearing screening, and voice recordings before and after a synchronous online class. This study found a decreased fundamental frequency in connected speech (reading) associated with the number of students per class, the number of classes per week, and background noise levels measured with a sound level meter. L1L0 was statistically smaller among experienced teachers with higher background noise levels compared with less-experienced colleagues in quieter classes. 1558kHz was statistically associated with the average number of students, whereas HNR was slightly higher for professors with more students and more classes per week. The findings suggest that online teaching has important implications for voice production among participating teachers. This study highlights the importance of considering working conditions during online classes when designing workplace prevention programs for college professors.

Topic-based variation as both cognitive and agentive

Abstract Topic-based style-shifting refers to the variation pattern that, when people talk about a topic, they shift to a linguistic style which is associated with the topic. Most of the research on topic-based variation in read speech have not taken stance-taking into consideration. This study argues that stance-taking needs to be included in the analysis of topic-based variation, for reading something aloud is a practice where individuals engage with the message communicated in a text. This study looks at the socially salient variable /ʂ/ in Taiwan Mandarin, and how deaf speakers exploit this variable to perform their stances towards a passage concerned with the political relationship between hearing people and deaf signers. The findings show that participants who demonstrate a stance of deaf solidarity diverge from standard speech styles in their repertoires when reading the deaf passage.

Regional variation in articulation rate in French spoken in Canada

This study examines articulation rate in three varieties of Canadian French and includes consideration of speaking style (reading vs. spontaneous), speaker’s age and gender, and length of inter-pause intervals. The varieties are spoken in different geographic areas of Canada – Quebec City (Quebec), Tracadie (New Brunswick), and Windsor (Ontario) – where there are different degrees of French–English contact. The main research question asks how these different contact situations are related to variation in articulation rate. Results show that in both reading and spontaneous speech articulation rates were faster among Quebec City speakers, where French is in a low-contact setting, and slower among speakers from Tracadie and Windsor, where there are greater degrees of contact. The effects of other factors are the same across the three regions: AR was faster in spontaneous productions than in reading; AR decreased with age in the reading task; AR was faster as the length of the inter-pausal intervals increased. The discussion points to similarities and differences with varieties of French spoken in Europe and underscores the importance of language contact in accounting for variation in articulation rate.

The realisation and phonetic features of the glottal stop in Bahdini Kurdish


 
 
 In Bahdini Kurdish (henceforth, BK), the realisation and the phonetic features of the glottal stop (GS) has not been extensively investigated and there is a debate about its phonemic status. In one hand, it is considered a phoneme that occurs only in a word-initial position. On the other hand, it is not con- sidered a phoneme but optionally produced at the beginning of vowel-initial words and does not affect meaning. The study aims at identifying the realisation of this sound experimentally, analyses its acous- tic features and whether it is affected by parameters such as vowel type, vowel position in the word and stress. The study is based on read speech produced by 10 BK native speakers. Then it is analysed phonetically using Praat to identify the presence/absence of the GS. Then the number of occurrences of the GSs are compared across vowel types, vowel position and stress. The results show that the GS is an epenthetic sound in BK that is inserted by the speakers to avoid onsetless syllables and to avoid vowel clusters. Different phonetic variations of the sound are realized in BK: A clear closure of the glottal stop, intermittent vocal folds vibrations during the GS closure, vocal folds vibrations start sim- ultaneously with the GS release. It appears not to be affected by stress but by position, as it is found more medially in V+V contexts than initially, and vowel quality as it is produced more before low vowels than non-low vowels.
 
 

Life Skills Approach in Arabic Language Learning at Islamic Boarding School-Based Madrasah Aliyah

This study aimed to find a life skills approach to learning Arabic in Islamic boarding schools based on Islamic high schools. This research is field research using qualitative methods. The location of this research is the Islamic boarding school-based Madrasah Aliyah. Using the form of observation, interviews and documentation in collecting data. As for analyzing the data, researchers carried out several stages of analysis, starting from data collection, reduction, review and evaluation. The results of this study cover two aspects, namely first, the Arabic language learning curriculum implemented in MA Salafiyah and MA Darussangadah is an integrated curriculum between the curriculum of the Ministry of Religion and the curriculum of Islamic boarding schools, based on classical studies and modern books with a language learning orientation. Arabic language actively or passively. Second, the Arabic language learning model at Madrasah Aliyah is based on Islamic boarding schools in Kebumen Regency, namely (1) Arabic learning at MA Salafiyah Wonoyoso is more oriented towards maharah lughawiyah (istima', kalam, qira'ah, and kitabah) as the basis of student competence. The process of learning Arabic is taught by integrating the four maharah lugawiyah through learning Arabic subjects, religious subjects, and linguistic practices in Islamic boarding schools; (2) Learning Arabic at MA Darussangadah develops learning Arabic that is oriented towards active and passive communication skills. This active communication skill is the skill of using spoken and written language. At the same time, the passive ability is the skill of understanding Arabic reading and other people's speech.

Voice Acoustic Instability During Spontaneous Speech in Parkinson’s Disease

In people with Parkinson's disease (PwPD), both motor and cognitive deficits influence voice and other aspects of communication. PwPD demonstrate vocal instability, but acoustic declines over the course of speaking are not well characterized and the role of cognition on these declines is unknown. We examined voice acoustics related to speech motor instability by comparing the first and the last utterances within a speech task. Our objective was to determine if mild cognitive impairment (MCI) status was associated with different patterns of acoustic change during these tasks. Participants with PD (n=44) were enrolled at University of Massachusetts Chan Medical School and classified by gold-standard criteria as normal cognition (PD-NC) or mild cognitive impairment (PD-MCI). The speech was recorded during the Rainbow Passage and a picture description task (Cookie Theft). We calculated the difference between first and last utterances in fo mean and standardized semitones (STSD), cepstral peak prominence-smoothed (CPPS), and low to high ratio (LH). We used t-tests to compare the declines in acoustic parameters between the task types and between participants with PD-NC versus PD-MCI. Mean fo, fo variability (STSD) and CPPS declined from the first to the last utterance in both tasks, but there was no significant difference in these declines between the PD-NC and PD-MCI groups. Those with PD-MCI demonstrated lower fo variability on the whole in both tasks and lower CPPS in the picture description task, compared to those with PD-NC. Mean and STSD fo as well as CPPS may be sensitive to PD-MCI status in reading and spontaneous speech tasks. Speech motor instability can be observed in these voice acoustic parameters over brief speech tasks, but the degree of decline does not depend on cognitive status. These findings will inform the ongoing development of algorithms to monitor speech and cognitive function in PD.

Lip Movement Detection Using 3D Convolution and Resnet

Abstract: Recognition of Lip movements has become one of the most challenging tasks and has crucial applicationsin the contemporary scenario. Being able to see speech helps people communicate better, especially in challenging listening environments like when there is a background noise and video surveillance when there is no audio. Lip reading is a technique primarily used by deaf people or those who have some form of hearing impairment. It’s a way of understanding and communicating with others who might not be familiar with another form of inclusive communication, such as sign language. Lip-reading technology mainly includes face detection, lip localization, feature extraction, training the classifier and finally recognising the word or sentence through lip movement. Many developments have taken place in this growing field using various deep learning-based techniques. An intelligent system will be trained by giving users lip-movement frames sequences as input and will identify lip movement and the said word using 3D convolution and ResNet . This project does analysis over various deep learning models and other datasets. This study also aims to find out the optimal architecture suitable for building a new model with high accuracy for lip movement detection.

English Learning Assistance Using Interactive Media for Children with Special Needs to Improve Growth and Development

Background. Children with special needs (ABK) are children who are in several ways different from other children in general. Among the crew members are Special Children (ALB), which consists of children who are blind, deaf, mentally retarded, quadriplegic, mentally disabled and double disabled. Some of the main things that need to be considered in the learning process for ALB are teachers, learning methods, learning approaches, infrastructure and learning support media (teaching aids). Purpose. The purpose of this community service activity is to solve the problems faced by partners in the English learning process, namely when a disorder results in disruption in daily functioning, especially in learning, the student requires special services (children with special educational needs) and requires specific learning methods in addition to appropriate and interactive learning media. Method. The solution offered to overcome this problem is the optimization of teaching methods. The recommended approach in the English language assistance activities for the Cilacap State Polytechnic PkM Team is in the form of prompts and demonstrations. Meanwhile, teaching English can use total physical response (TPR) by maximizing lip reading technique in addition to maximizing the use of flash cards to attract students' interest and focus. Results. The results obtained from this community service activity are increasing the ability of children with special needs to say a few simple words in English. The growth and development of children with special needs increase after the community service activities are completed. This is shown from the evaluation results carried out by the service team by conducting a post-test on ABK. Conclusion. To get significant results, namely increasing the growth and development of ABK, especially in the pronunciation of words in English, it is better if this activity is carried out regularly in the future.

A novel non-word speech preparation task to increase stuttering frequency in experimental settings for longitudinal research

PurposeThe variable and intermittent nature of stuttering makes it difficult to consistently elicit a sufficient number of stuttered trials for longitudinal experimental research. This study tests the efficacy of using non-word pairs that phonetically mimic English words with no associated meaning, to reliably elicit balanced numbers of stuttering and fluent trials over multiple sessions. The study also evaluated the effect of non-word length on stuttering frequency, the consistency of stuttering frequency across sessions, and potential carry-over effects of increased stuttering frequency in the experimental task to conversational and reading speech after the task. MethodsTwelve adults who stutter completed multiple sessions (mean of 4.8 sessions) where they were video-recorded during pre-task reading and conversation, followed by an experimental task where they read 400 non-word pairs randomized for each session, and then a post-task reading and conversation sample. ResultsOn average, across sessions and participants, non-word pairs consistently yielded a balanced distribution of fluent (60.7%) and stuttered (39.3%) trials over five sessions. Non-word length had a positive effect on stuttering frequency. No carryover effects from experimental to post-task conversation and reading were found. ConclusionsNon-word pairs effectively and consistently elicited balanced proportions of stuttered and fluent trials. This approach can be used to gather longitudinal data to better understand the neurophysiological and behavioral correlates of stuttering.

Comparison of intensity-based methods for automatic speech rate computation

Automatic computation of speech rate is a necessary task in a wide range of applications that require this prosodic feature, in which a manual transcription and time alignments are not available. Several tools have been developed to this end, but not enough research has been conducted yet to see to what extent they are scalable to other languages.
 In the present work, we take two off-the- shelf tools designed for automatic speech rate computation and already tested for Dutch and English (v1, which relies on intensity peaks preceded by an intensity dip to find syllable nuclei and v3, which relies on intensity peaks surrounded by dips) and we apply them to read and spontaneous Spanish speech. Then, we test which of them offers the best performance. The results obtained with precision and normalized mean squared error metrics showed that v3 performs better than v1. However, recall measurement shows a better performance of v1, which suggests that a more fine-grained analysis on sensitivity and specificity is needed to select the best option depending on the application we are dealing with.

Speech Prosody and Reading Comprehension in Chinese-English Bilingual Children: The Mediating Role of Syntactic Awareness.

Although children's prosodic sensitivity links with their reading comprehension, the factors affecting this link remain unclear. By simultaneously measuring first language (L1) Chinese and second language (L2) English prosodic sensitivity and reading comprehension, this study examined the mediating role of syntactic awareness on prosody-reading comprehension among Hong Kong Chinese-English bilingual children. A group of 227 Hong Kong Chinese-English bilingual fourth graders completed L1 and L2 prosodic sensitivity (Cantonese lexical tone awareness and English prosodic sensitivity), syntactic awareness, and reading comprehension and control measures of cognitive (nonverbal IQ, short-term memory, and working memory), metalinguistic (phonological awareness and morphological awareness), linguistic (vocabulary knowledge), and word reading skills. The within-language analyses showed a partial mediation effect of Chinese syntactic awareness on the relation between Cantonese lexical tone awareness and Chinese reading comprehension, but a full mediation effect of English syntactic awareness on the relation between English prosodic sensitivity and English reading comprehension. The cross-language analyses revealed a significant direct effect of Cantonese lexical tone awareness on English reading comprehension and a significant indirect effect of English prosodic sensitivity on Chinese reading comprehension via Chinese syntactic awareness. These findings suggest that, despite the language-independent mediating role of syntactic awareness in bridging prosody and reading comprehension, the degree of this mediation is shaped by language-specific prosody and its relations with other linguistic structures, including semantics.

End-to-End Video-to-Speech Synthesis Using Generative Adversarial Networks.

Video-to-speech is the process of reconstructing the audio speech from a video of a spoken utterance. Previous approaches to this task have relied on a two-step process where an intermediate representation is inferred from the video and is then decoded into waveform audio using a vocoder or a waveform reconstruction algorithm. In this work, we propose a new end-to-end video-to-speech model based on generative adversarial networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm. Our model consists of an encoder-decoder architecture that receives raw video as input and generates speech, which is then fed to a waveform critic and a power critic. The use of an adversarial loss based on these two critics enables the direct synthesis of the raw audio waveform and ensures its realism. In addition, the use of our three comparative losses helps establish direct correspondence between the generated audio and the input video. We show that this model is able to reconstruct speech with remarkable realism for constrained datasets such as GRID, and that it is the first end-to-end model to produce intelligible speech for Lip Reading in the Wild (LRW), featuring hundreds of speakers recorded entirely "in the wild." We evaluate the generated samples in two different scenarios-seen and unseen speakers-using four objective metrics which measure the quality and intelligibility of artificial speech. We demonstrate that the proposed approach outperforms all previous works in most metrics on GRID and LRW.

Efficient DNN Model for Word Lip-Reading

This paper studies various deep learning models for word-level lip-reading technology, one of the tasks in the supervised learning of video classification. Several public datasets have been published in the lip-reading research field. However, few studies have investigated lip-reading techniques using multiple datasets. This paper evaluates deep learning models using four publicly available datasets, namely Lip Reading in the Wild (LRW), OuluVS, CUAVE, and Speech Scene by Smart Device (SSSD), which are representative datasets in this field. LRW is one of the large-scale public datasets and targets 500 English words released in 2016. Initially, the recognition accuracy of LRW was 66.1%, but many research groups have been working on it. The current the state of the art (SOTA) has achieved 94.1% by 3D-Conv + ResNet18 + {DC-TCN, MS-TCN, BGRU} + knowledge distillation + word boundary. Regarding the SOTA model, in this paper, we combine existing models such as ResNet, WideResNet, WideResNet, EfficientNet, MS-TCN, Transformer, ViT, and ViViT, and investigate the effective models for word lip-reading tasks using six deep learning models with modified feature extractors and classifiers. Through recognition experiments, we show that similar model structures of 3D-Conv + ResNet18 for feature extraction and MS-TCN model for inference are valid for four datasets with different scales.

Analyzing lower half facial gestures for lip reading applications: Survey on vision techniques

Lip reading has gained popularity due to the proliferation of emerging real-world applications. This article provides a comprehensive review of benchmark datasets available for lip-reading applications and pioneering works that analyze lower facial cues for lip-reading applications. A comprehensive review of lip reading applications is broadly classified into five distinct applications: Lip Reading Biometrics (LRB), Audio Visual Speech Recognition (AVSR), Silent Speech Recognition (SSR), Voice from Lips, and Lip HCI (Human–computer interaction). LRB entails extensive research in the fields of authentication and liveness detection. AVSR covers key findings that have contributed significantly to applications such as voice assistants, video-to-text transcription, hearing aids, and pronunciation-correcting systems. SSR analyzes the efforts made for silent-video-to-text transcription and surveillance camera applications. The voice from lips section discusses applications such as voice for the voiceless and vision-infused speech inpainting. In lip HCI, LR-HCI for smartphones, smart TVs, computers, robots, and musical instruments is reviewed in detail. Comprehensive coverage is given to cutting-edge techniques in computer vision, signal processing, machine learning, and deep learning. The advancements that aid the system in learning to lip-read and authenticate lip gestures, generate text transcription, synthesize voice based on lip movements, and control systems via lip movements (lip HCI) are covered. The work concludes by highlighting the limitations of existing frameworks, the road maps of each application illustrating the evolution of techniques employed over time, and future research avenues in lip-reading applications.

Overview of Lip Reading Methods: Issues, Current Developments, and Future Prospects

Overview of Lip Reading Methods: Issues, Current Developments, and Future Prospects

Speech rehabilitation in children with cochlear implants using a multisensory (French Cued Speech) or a hearing-focused (Auditory Verbal Therapy) approach.

Early exposure to a rich linguistic environment is essential as soon as the diagnosis of deafness is made. Cochlear implantation (CI) allows children to have access to speech perception in their early years. However, it provides only partial acoustic information, which can lead to difficulties in perceiving some phonetic contrasts. This study investigates the contribution of two spoken speech and language rehabilitation approaches to speech perception in children with CI using a lexicality judgment task from the EULALIES battery. Auditory Verbal Therapy (AVT) is an early intervention program that relies on auditory learning to enhance hearing skills in deaf children with CI. French Cued Speech, also called Cued French (CF), is a multisensory communication tool that disambiguates lip reading by adding a manual gesture. In this study, 124 children aged from 60 to 140 months were included: 90 children with typical hearing skills (TH), 9 deaf children with CI who had participated in an AVT program (AVT), 6 deaf children with CI with high Cued French reading skills (CF+), and 19 deaf children with CI with low Cued French reading skills (CF-). Speech perception was assessed using sensitivity (d') using both the hit and false alarm rates, as defined in signal-detection theory. The results show that children with cochlear implants from the CF- and CF+ groups have significantly lower performance compared to children with typical hearing (TH) (p < 0.001 and p = 0.033, respectively). Additionally, children in the AVT group also tended to have lower scores compared to TH children (p = 0.07). However, exposition to AVT and CF seems to improve speech perception. The scores of the children in the AVT and CF+ groups are closer to typical scores than those of children in the CF- group, as evidenced by a distance measure. Overall, the findings of this study provide evidence for the effectiveness of these two speech and language rehabilitation approaches, and highlight the importance of using a specific approach in addition to a cochlear implant to improve speech perception in children with cochlear implants.