Visual Speech Research Articles

Effects of Visual Speech Envelope on Audiovisual Speech Perception in Multitalker Listening Environments.

Purpose This study investigated the effects of visually presented speech envelope information with various modulation rates and depths on audiovisual speech perception in noise. Method Forty adults (21.25 ± 1.45 years) participated in audiovisual sentence recognition measurements in noise. Target speech sentences were auditorily presented in multitalker babble noises at a -3 dB SNR. Acoustic amplitude envelopes of target signals were extracted through low-pass filters with different cutoff frequencies (4, 10, and 30 Hz) and a fixed modulation depth at 100% (Experiment 1) or extracted with various modulation depths (0%, 25%, 50%, 75%, and 100%) and a fixed 10-Hz modulation rate (Experiment 2). The extracted target envelopes were synchronized with the amplitude of a spherical-shaped ball and presented as visual stimuli. Subjects were instructed to attend to both auditory and visual stimuli of the target sentences and type down their answers. The sentence recognition accuracy was compared between audio-only and audiovisual conditions. Results In Experiment 1, a significant improvement in speech intelligibility was observed when the visual analog (a sphere) synced with the acoustic amplitude envelope modulated at a 10-Hz modulation rate compared to the audio-only condition. In Experiment 2, the visual analog with 75% modulation depth resulted in better audiovisual speech perception in noise compared to the other modulation depth conditions. Conclusion An abstract visual analog of acoustic amplitude envelopes can be efficiently delivered by the visual system and integrated online with auditory signals to enhance speech perception in noise, independent of particular articulation movements.

Audiovisual speech is more than the sum of its parts: Auditory-visual superadditivity compensates for age-related declines in audible and lipread speech intelligibility.

Multisensory input can improve perception of ambiguous unisensory information. For example, speech heard in noise can be more accurately identified when listeners see a speaker's articulating face. Importantly, these multisensory effects can be superadditive to listeners' ability to process unisensory speech, such that audiovisual speech identification is better than the sum of auditory-only and visual-only speech identification. Age-related declines in auditory and visual speech perception have been hypothesized to be concomitant with stronger cross-sensory influences on audiovisual speech identification, but little evidence exists to support this. Currently, studies do not account for the multisensory superadditive benefit of auditory-visual input in their metrics of the auditory or visual influence on audiovisual speech perception. Here we treat multisensory superadditivity as independent from unisensory auditory and visual processing. In the current investigation, older and younger adults identified auditory, visual, and audiovisual speech in noisy listening conditions. Performance across these conditions was used to compute conventional metrics of the auditory and visual influence on audiovisual speech identification and a metric of auditory-visual superadditivity. Consistent with past work, auditory and visual speech identification declined with age, audiovisual speech identification was preserved, and no age-related differences in the auditory or visual influence on audiovisual speech identification were observed. However, we found that auditory-visual superadditivity improved with age. The novel findings suggest that multisensory superadditivity is independent of unisensory processing. As auditory and visual speech identification decline with age, compensatory changes in multisensory superadditivity may preserve audiovisual speech identification in older adults. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Face and voice perception: Monkey see, monkey hear

Primate brains contain specialized areas for perceiving social cues. New research shows that only some of these areas integrate visual faces with auditory voices.

The multisensory cocktail party problem in adults: Perceptual segregation of talking faces on the basis of audiovisual temporal synchrony

Social interactions often involve a cluttered multisensory scene consisting of multiple talking faces. We investigated whether audiovisual temporal synchrony can facilitate perceptual segregation of talking faces. Participants either saw four identical or four different talking faces producing temporally jittered versions of the same visible speech utterance and heard the audible version of the same speech utterance. The audible utterance was either synchronized with the visible utterance produced by one of the talking faces or not synchronized with any of them. Eye tracking indicated that participants exhibited a marked preference for the synchronized talking face, that they gazed more at the mouth than the eyes overall, that they gazed more at the eyes of an audiovisually synchronized than a desynchronized talking face, and that they gazed more at the mouth when all talking faces were audiovisually desynchronized. These findings demonstrate that audiovisual temporal synchrony plays a major role in perceptual segregation of multisensory clutter and that adults rely on differential scanning strategies of a talker's eyes and mouth to discover sources of multisensory coherence.

The use of broad vs restricted regions of interest in functional near-infrared spectroscopy for measuring cortical activation to auditory-only and visual-only speech

As an alternative to fMRI, functional near-infrared spectroscopy (fNIRS) is a relatively new tool for observing cortical activation. However, spatial resolution is reduced compared to fMRI and often the exact locations of fNIRS optodes and specific anatomical information is not known. The aim of this study was to explore the location and range of specific regions of interest that are sensitive to detecting cortical activation using fNIRS in response to auditory- and visual-only connected speech.Two approaches to a priori region-of-interest selection were explored. First, broad regions corresponding to the auditory cortex and occipital lobe were analysed. Next, the fNIRS Optode Location Decider (fOLD) tool was used to divide the auditory and visual regions into two subregions corresponding to distinct anatomical structures. The Auditory-A and -B regions corresponded to Heschl's gyrus and planum temporale, respectively. The Visual-A region corresponded to the superior occipital gyrus and the cuneus, and the Visual-B region corresponded to the middle occipital gyrus. The experimental stimulus consisted of a connected speech signal segmented into 12.5-sec blocks and was presented in either an auditory-only or visual-only condition.Group-level results for eight normal-hearing adult participants averaged over the broad regions of interest revealed significant auditory-evoked activation for both the left and right broad auditory regions of interest. No significant activity was observed for any other broad region of interest in response to any stimulus condition. When divided into subregions, there was a significant positive auditory-evoked response in the left and right Auditory-A regions, suggesting activation near the primary auditory cortex in response to auditory-only speech. There was a significant positive visual-evoked response in the Visual-B region, suggesting middle occipital gyrus activation in response to visual-only speech. In the Visual-A region, however, there was a significant negative visual-evoked response. This result suggests a significant decrease in oxygenated hemoglobin in the superior occipital gyrus as well as the cuneus in response to visual-only speech. Distinct response characteristics, either positive or negative, in adjacent subregions within the temporal and occipital lobes were fairly consistent on the individual level.Results suggest that temporal regions near Heschl's gyrus may be the most advantageous location in adults for identifying hemodynamic responses to complex auditory speech signals using fNIRS. In the occipital lobe, regions corresponding to the facial processing pathway may prove advantageous for measuring positive responses to visual speech using fNIRS.

Modified Geometrical Method For Visual Lip Extraction From Color Images

Lip Reading or Speech recognition is a visual way of communicating. It depends on watching the speaker's mouth; Especially his lips to understand the letters and words of the speaker. Through the visual image (the form of the expression of the mouth) and the movement of the lips, visual recognition of spoken letters, words, and sentences is made; Gestures and expressions on the face. the position of the mouth and its extracted features help to understand the visual speech more clearly. Researchers are always trying to find the new methods to increase the efficiency of the lip reading systems. for this reason, This paper presents a lip localization. based face detection &features extraction strategy to segment mouth region from input video frames. we have used RGB &HSV color model along with modified channel factor and then applying more edge detection techniques and morphological operations to be a new one of the visual speech recognition tools.

Genetic Testing Contributes to Diagnosis in Cerebral Palsy: Aicardi-Goutières Syndrome as an Example.

Cerebral palsy (CP) is a non-progressive neurodevelopmental disorder characterized by motor impairments, often accompanied by co-morbidities such as intellectual disability, epilepsy, visual and hearing impairment and speech and language deficits. Despite the established role of hypoxic–ischemic injury in some CP cases, several studies suggest that birth asphyxia is actually an uncommon cause, accounting for <10% of CP cases. For children with CP in the absence of traditional risk factors, a genetic basis to their condition is increasingly suspected. Several recent studies indeed confirm copy number variants and single gene mutations with large genetic heterogeneity as an etiology in children with CP. Here, we report three patients with spastic cerebral palsy and a genetically confirmed diagnosis of Aicardi-Goutières syndrome (AGS), with highly variable phenotypes ranging from clinically suggestive to non-specific symptomatology. Our findings suggest that AGS may be a rather common cause of CP, that frequently remains undiagnosed without additional genetic testing, as in only one case a clinical suspicion of AGS was raised. Our data show that a diagnosis of AGS must be considered in cases with spastic CP, even in the absence of characteristic brain abnormalities. Importantly, a genetic diagnosis of AGS may have significant therapeutic consequences, as targeted therapies are being developed for type 1 interferonopathies, the group of diseases to which AGS belongs. Our findings demonstrate the importance of next generation sequencing in CP patients without an identifiable cause, since targeted diagnostic testing is hampered by the often non-specific presentation.

Rethinking the McGurk effect as a perceptual illusion.

Visual speech cues play an important role in speech recognition, and the McGurk effect is a classic demonstration of this. In the original McGurk & Macdonald (Nature 264, 746-748 1976) experiment, 98% of participants reported an illusory "fusion" percept of /d/ when listening to the spoken syllable /b/ and watching the visual speech movements for /g/. However, more recent work shows that subject and task differences influence the proportion of fusion responses. In the current study, we varied task (forced-choice vs. open-ended), stimulus set (including /d/ exemplars vs. not), and data collection environment (lab vs. Mechanical Turk) to investigate the robustness of the McGurk effect. Across experiments, using the same stimuli to elicit the McGurk effect, we found fusion responses ranging from 10% to 60%, thus showing large variability in the likelihood of experiencing the McGurk effect across factors that are unrelated to the perceptual information provided by the stimuli. Rather than a robust perceptual illusion, we therefore argue that the McGurk effect exists only for some individuals under specific task situations.Significance: This series of studies re-evaluates the classic McGurk effect, which shows the relevance of visual cues on speech perception. We highlight the importance of taking into account subject variables and task differences, and challenge future researchers to think carefully about the perceptual basis of the McGurk effect, how it is defined, and what it can tell us about audiovisual integration in speech.

Selective attention to the mouth of a talker in Japanese-learning infants and toddlers: Its relationship with vocabulary and compensation for noise

Infants increasingly gaze at the mouth of talking faces during the latter half of the first postnatal year. This study investigated mouth-looking behavior of 120 full-term infants and toddlers (6 months–3 years) and 12 young adults (21–24 years) from Japanese monolingual families. The purpose of the study included: (1) Is such an attentional shift to the mouth in infancy similarly observed in Japanese environment where contribution of visual speech is known to be relatively weak? (2) Can noisy conditions increase mouth-looking behavior of Japanese young children? (3) Is the mouth-looking behavior related to language acquisition? To this end, movies of a talker speaking short phrases were presented while manipulating signal-to-noise ratio (SNR: Clear, SN+4, and SN-4). Expressive vocabulary of toddlers was obtained through their parents. The results indicated that Japanese infants initially have a strong preference for the eyes to mouth which is weakened toward 10 months, but the shift was later and in a milder fashion compared to known results for English-learning infants. Even after 10 months, no clear-cut preference for the mouth was observed even in linguistically challenging situations with strong noise until 3 years of age. In the Clear condition, there was a return of the gaze to the eyes as early as 3 years of age, where they showed increasing attention to the mouth with increasing noise level. In addition, multiple regression analyses revealed a tendency that 2- and 3-year-olds with larger vocabulary increasingly look at the eyes. Overall, the gaze of Japanese-learning infants and toddlers was more biased to the eyes in various aspects compared to known results of English-learning infants. The present findings shed new light on our understanding of the development of selective attention to the mouth in non-western populations.

Weaker McGurk Effect for Rubin's Vase-Type Speech in People With High Autistic Traits.

While visual information from facial speech modulates auditory speech perception, it is less influential on audiovisual speech perception among autistic individuals than among typically developed individuals. In this study, we investigated the relationship between autistic traits (Autism-Spectrum Quotient; AQ) and the influence of visual speech on the recognition of Rubin's vase-type speech stimuli with degraded facial speech information. Participants were 31 university students (13 males and 18 females; mean age: 19.2, SD: 1.13 years) who reported normal (or corrected-to-normal) hearing and vision. All participants completed three speech recognition tasks (visual, auditory, and audiovisual stimuli) and the AQ-Japanese version. The results showed that accuracies of speech recognition for visual (i.e., lip-reading) and auditory stimuli were not significantly related to participants' AQ. In contrast, audiovisual speech perception was less susceptible to facial speech perception among individuals with high rather than low autistic traits. The weaker influence of visual information on audiovisual speech perception in autism spectrum disorder (ASD) was robust regardless of the clarity of the visual information, suggesting a difficulty in the process of audiovisual integration rather than in the visual processing of facial speech.

A NOVEL TASK-ORIENTED APPROACH TOWARD AUTOMATED LIP-READING SYSTEM IMPLEMENTATION

Abstract. Visual information plays a key role in automatic speech recognition (ASR) when audio is corrupted by background noise, or even inaccessible. Speech recognition using visual information is called lip-reading. The initial idea of visual speech recognition comes from humans’ experience: we are able to recognize spoken words from the observation of a speaker's face without or with limited access to the sound part of the voice. Based on the conducted experimental evaluations as well as on analysis of the research field we propose a novel task-oriented approach towards practical lip-reading system implementation. Its main purpose is to be some kind of a roadmap for researchers who need to build a reliable visual speech recognition system for their task. In a rough approximation, we can divide the task of lip-reading into two parts, depending on the complexity of the problem. First, if we need to recognize isolated words, numbers or small phrases (e.g. Telephone numbers with a strict grammar or keywords). Or second, if we need to recognize continuous speech (phrases or sentences). All these stages disclosed in detail in this paper. Based on the proposed approach we implemented from scratch automatic visual speech recognition systems of three different architectures: GMM-CHMM, DNN-HMM and purely End-to-end. A description of the methodology, tools, step-by-step development and all necessary parameters are disclosed in detail in current paper. It is worth noting that for the Russian speech recognition, such systems were created for the first time.

Improving the Recognition Performance of Lip Reading Using the Concatenated Three Sequence Keyframe Image Technique

This paper proposes a lip reading method based on convolutional neural networks applied to Concatenated Three Sequence Keyframe Image (C3-SKI), consisting of (a) the Start-Lip Image (SLI), (b) the Middle-Lip Image (MLI), and (c) the End-Lip Image (ELI) which is the end of the pronunciation of that syllable. The lip area’s image dimensions were reduced to 32×32 pixels per image frame and three keyframes concatenate together were used to represent one syllable with a dimension of 96×32 pixels for visual speech recognition. Every three concatenated keyframes representing any syllable are selected based on the relative maximum and relative minimum related to the open lip’s width and height. The evaluation results of the model’s effectiveness, showed accuracy, validation accuracy, loss, and validation loss values at 95.06%, 86.03%, 4.61%, and 9.04% respectively, for the THDigits dataset. The C3-SKI technique was also applied to the AVDigits dataset, showing 85.62% accuracy. In conclusion, the C3-SKI technique could be applied to perform lip reading recognition.

Neurophysiological Indices of Audiovisual Speech Processing Reveal a Hierarchy of Multisensory Integration Effects.

Seeing a speaker's face benefits speech comprehension, especially in challenging listening conditions. This perceptual benefit is thought to stem from the neural integration of visual and auditory speech at multiple stages of processing, whereby movement of a speaker's face provides temporal cues to auditory cortex, and articulatory information from the speaker's mouth can aid recognizing specific linguistic units (e.g., phonemes, syllables). However, it remains unclear how the integration of these cues varies as a function of listening conditions. Here, we sought to provide insight on these questions by examining EEG responses in humans (males and females) to natural audiovisual (AV), audio, and visual speech in quiet and in noise. We represented our speech stimuli in terms of their spectrograms and their phonetic features and then quantified the strength of the encoding of those features in the EEG using canonical correlation analysis (CCA). The encoding of both spectrotemporal and phonetic features was shown to be more robust in AV speech responses than what would have been expected from the summation of the audio and visual speech responses, suggesting that multisensory integration occurs at both spectrotemporal and phonetic stages of speech processing. We also found evidence to suggest that the integration effects may change with listening conditions; however, this was an exploratory analysis and future work will be required to examine this effect using a within-subject design. These findings demonstrate that integration of audio and visual speech occurs at multiple stages along the speech processing hierarchy.SIGNIFICANCE STATEMENT During conversation, visual cues impact our perception of speech. Integration of auditory and visual speech is thought to occur at multiple stages of speech processing and vary flexibly depending on the listening conditions. Here, we examine audiovisual (AV) integration at two stages of speech processing using the speech spectrogram and a phonetic representation, and test how AV integration adapts to degraded listening conditions. We find significant integration at both of these stages regardless of listening conditions. These findings reveal neural indices of multisensory interactions at different stages of processing and provide support for the multistage integration framework.

Research on the Function of Visual Phonetic Software Praat in Vocational English Phonetics Teaching

Phonetics teaching is the most difficult part of English majors in higher vocational colleges. Many students are affected by their mother tongue pronunciation habits and the lack of systematic voice training at the middle school stage, resulting in weak phonetic foundations. There are problems such as inability to correctly distinguish between long and short vowels, voiced consonants, emphasized and light pronunciation, and intonation. To this end, the introduction of the visual speech software Praat, using its visual and intuitive image characteristics can assist teachers in class, help students find the pronunciation errors in time and correct them in time. Based on this, this article will combine the status quo of English phonetics teaching in higher vocational colleges and analyze the role of Praat phonetics learning software in phonetics teaching and its specific application implementation.

Synchronous facial action binds dynamic facial features

We asked how dynamic facial features are perceptually grouped. To address this question, we varied the timing of mouth movements relative to eyebrow movements, while measuring the detectability of a small temporal misalignment between a pair of oscillating eyebrows—an eyebrow wave. We found eyebrow wave detection performance was worse for synchronous movements of the eyebrows and mouth. Subsequently, we found this effect was specific to stimuli presented to the right visual field, implicating the involvement of left lateralised visual speech areas. Adaptation has been used as a tool in low-level vision to establish the presence of separable visual channels. Adaptation to moving eyebrows and mouths with various relative timings reduced eyebrow wave detection but only when the adapting mouth and eyebrows moved asynchronously. Inverting the face led to a greater reduction in detection after adaptation particularly for asynchronous facial motion at test. We conclude that synchronous motion binds dynamic facial features whereas asynchronous motion releases them, allowing adaptation to impair eyebrow wave detection.

Example-Based Facial Animation of Virtual Reality Avatars Using Auto-Regressive Neural Networks.

This article presents a hybrid animation approach that combines example-based and neural animation methods to create a simple, yet powerful animation regime for human faces. Example-based methods usually employ a database of prerecorded sequences that are concatenated or looped in order to synthesize novel animations. In contrast to this traditional example-based approach, we introduce a light-weight auto-regressive network to transform our animation-database into a parametric model. During training, our network learns the dynamics of facial expressions, which enables the replay of annotated sequences from our animation database as well as their seamless concatenation in new order. This representation is especially useful for the synthesis of visual speech, where coarticulation creates interdependencies between adjacent visemes, which affects their appearance. Instead of creating an exhaustive database that contains all viseme variants, we use our animation-network to predict the correct appearance. This allows realistic synthesis of novel facial animation sequences like visual-speech but also general facial expressions in an example-based manner.

Recent update on reading disability (dyslexia) focused on neurobiology.

Reading disability (dyslexia) refers to an unexpected difficulty with reading for an individual who has the intelligence to be a much better reader. Dyslexia is most commonly caused by a difficulty in phonological processing (the appreciation of the individual sounds of spoken language), which affects the ability of an individual to speak, read, and spell. In this paper, I describe reading disabilities by focusing on their underlying neurobiological mechanisms. Neurobiological studies using functional brain imaging have uncovered the reading pathways, brain regions involved in reading, and neurobiological abnormalities of dyslexia. The reading pathway is in the order of visual analysis, letter recognition, word recognition, meaning (semantics), phonological processing, and speech production. According to functional neuroimaging studies, the important areas of the brain related to reading include the inferior frontal cortex (Broca’s area), the midtemporal lobe region, the inferior parieto-temporal area, and the left occipitotemporal region (visual word form area). Interventions for dyslexia can affect reading ability by causing changes in brain function and structure. An accurate diagnosis and timely specialized intervention are important in children with dyslexia. In cases in which national infant development screening tests have been conducted, as in Korea, if language developmental delay and early predictors of dyslexia are detected, careful observation of the progression to dyslexia and early intervention should be made.

Visual Speech Recognition using Convolutional Neural Network

Visual speech Recognition or Lip reading is used for teaching differently abled persons to communicate with others. It has been determining speech by looking at the movement of the lips. There are many practical difficulties in traditional lip-reading recognition systems like complicated image processing, difficult to teach classifiers and recognition processes will take a long time. In our paper, we proposed the use of convolutional neural networks- Hidden Markov model (CNN-HMM) in lip reading. Since CNN will assign importance to an input image, it is easier to see a difference among the images. HMM used to handle the dynamics of the image sequence. First we convert the incoming video into images and these images selected for further operation. HMM provides a highly reliable way of speech recognition.

Effect of impoverished information on multisensory integration in judgments of learning.

According to the principle of inverse effectiveness (PIE), weaker responses to information in one modality (i.e., unisensory) benefit more from additional information in a second modality (i.e., multisensory; Meredith & Stein, 1986). We suggest that the PIE may also inform whether perceptual fluency affects judgments of learning (JOLs). If JOLs follow the PIE, the differences in JOLs for multisensory and unisensory items should increase as the unisensory study items become harder to perceive. That is, an influence of perceptual fluency should prompt a similar, interactive pattern across perceptual responses and JOLs. In 3 experiments, we systematically varied the signal intensity or noise in 1 modality to examine how responses might change with the inclusion of information in a second modality. In Experiment 1, written words in several font sizes were sometimes accompanied by spoken equivalents. In Experiments 2 and 3, spoken words in various background noise levels were sometimes accompanied by visual speech articulations. Consistent with the PIE, the multisensory benefits in response time and/or correct identification increased as responses to unisensory information decreased. Also, the multisensory formats received higher JOLs than the unisensory formats; however, unlike the predictions from PIE, this difference did not increase as study items became harder to perceive. Experiment 3 extended this finding to participants' explicit beliefs. In multisensory settings, JOLs may rely more on theory- than data-driven processes. We suggest that broadly defined processing fluency may always contribute to JOLs, but, regarding perceptual information, JOLs appear to track perceptual attributes rather than perceptual fluency. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Audiovisual Speech Perception in Children with Autism Spectrum Disorders: Evidence from Visual Phonemic Restoration.

Children with autism spectrum disorders have been reported to be less influenced by a speaker's face during speech perception than those with typically development. To more closely examine these reported differences, a novel visual phonemic restoration paradigm was used to assess neural signatures (event-related potentials [ERPs]) of audiovisual processing in typically developing children and in children with autism spectrum disorder. Video of a speaker saying the syllable /ba/ was paired with (1) a synthesized /ba/ or (2) a synthesized syllable derived from /ba/ in which auditory cues for the consonant were substantially weakened, thereby sounding more like /a/. The auditory stimuli are easily discriminable; however, in the context of a visual /ba/, the auditory /a/ is typically perceived as /ba/, producing a visual phonemic restoration. Only children with ASD showed a large /ba/-/a/ discrimination response in the presence of a speaker producing /ba/, suggesting reduced influence of visual speech.