Silent Video Research Articles

Neural Mechanisms of Social Interaction Perception: Observing Interpersonal Synchrony Modulates Action Observation Network Activation and Is Spared in Autism.

How the temporal dynamics of social interactions are perceived arguably plays an important role in how one engages in social interactions and how difficulties in establishing smooth social interactions may occur. One aspect of temporal dynamics in social interactions is the mutual coordination of individuals' behaviors during social interaction, otherwise known as behavioral interpersonal synchrony (IPS). Behavioral IPS has been studied increasingly in various contexts, such as a feature of the social interaction difficulties inherent to autism. To fully understand the temporal dynamics of social interactions, or reductions thereof in autism, the neural basis of IPS perception needs to be established. Thus, the current study's aim was twofold: to establish the basic neuro-perceptual processing of IPS in social interactions for typical observers and to test whether it might differ for autistic individuals. In a task-based fMRI paradigm, participants viewed short, silent video vignettes of humans during social interactions featuring a variation of behavioral IPS. The results show that observing behavioral IPS modulates the Action Observation Network (AON). Interestingly, autistic participants showed similar neural activation patterns as non-autistic participants which were modulated by the behavioral IPS they observed in the videos, suggesting that the perception oftemporal dynamics of social interactions is spared and may not underly reduced behavioral IPS often observed in autism. Nevertheless, a general difference in processing social interactions was found in autistic observers, characterized by decreased neural activation in the right middle frontal gyrus, angular gyrus, and superior temporal areas. These findings demonstrate that although the autistic and non-autistic groups indeed differed in the neural processing of social interaction perception, the temporal dynamics of these social interactions were not the reason for these differences in social interaction perception in autism. Hence, spared recruitment of the AON for processing temporal dynamics of social interactions in autism does not account for the widely reported attenuation of IPS in autism and for the widely reported and presently observed differences in social interaction perception in autism.

Effects of attention on the processing of physical and abstract auditory regularities: An exploratory MMN study

The mismatch-negativity (MMN) component of the auditory event-related potential (ERP) reflects preattentive memory mechanisms encoding various types of regularities in the auditory environment. In an oddball paradigm, two types of deviant stimuli (in separate blocks) were presented among frequent standard stimuli: physical deviants (higher in pitch than the standards) and more complex, “abstract” deviants (tone pairs descending in pitch, presented among ascending standard tone pairs). The attentional load of the participant was manipulated under three conditions, where the participants either (1) watched a silent video, (2) played a computer game (Tetris) or (3) attended to the auditory stimuli and tried to detect infrequent target stimuli which were of lower intensity than the standard stimuli. The goal was to find out, whether the possible attention effects (suppression/enhancement) on the MMN are similar or different to stimuli requiring the extraction of either physical or abstract invariances. Both the physical and abstract deviants elicited in all conditions MMNs but no statistically significant amplitude differences between the conditions were found. The N2b and P3b components were elicited only in the attend condition and only by the soft target tones. The results further confirm that the MMN is a robust response to various types of regularity violations, showing no major effects of attentional manipulations. The results also suggest that the most commonly used primary task in MMN experiments, watching a silent video, usually keeps the participants' attention well enough directed away from the auditory stimuli. However, in cases where a cognitively more demanding but still participant-friendly primary task is needed, a simple computer game such as Tetris can be used, enabling better control of the participants’ attention and vigilance.

Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language.

When comprehending speech, listeners can use information encoded in visual cues from a face to enhance auditory speech comprehension. For example, prior work has shown that the mouth movements reflect articulatory features of speech segments and durational information, while pitch and speech amplitude are primarily cued by eyebrow and head movements. Little is known about how the visual perception of segmental and prosodic speech information is influenced by linguistic experience. Using eye-tracking, we studied how perceivers' visual scanning of different regions on a talking face predicts accuracy in a task targeting both segmental versus prosodic information, and also asked how this was influenced by language familiarity. Twenty-four native English perceivers heard two audio sentences in either English or Mandarin (an unfamiliar, non-native language), which sometimes differed in segmental or prosodic information (or both). Perceivers then saw a silent video of a talking face, and judged whether that video matched either the first or second audio sentence (or whether both sentences were the same). First, increased looking to the mouth predicted correct responses only for non-native language trials. Second, the start of a successful search for speech information in the mouth area was significantly delayed in non-native versus native trials, but just when there were only prosodic differences in the auditory sentences, and not when there were segmental differences. Third, (in correct trials) the saccade amplitude in native language trials was significantly greater than in non-native trials, indicating more intensely focused fixations in the latter. Taken together, these results suggest that mouth-looking was generally more evident when processing a non-native versus native language in all analyses, but fascinatingly, when measuring perceivers' latency to fixate the mouth, this language effect was largest in trials where only prosodic information was useful for the task.

Sub-visemic discrimination and the effect of visual resemblance on silent lip-reading

ABSTRACT Relevant visual information is available in speakers’ faces during face-to-face interactions that improve speech perception. There is an ongoing debate, however, about how phonemes and their visual counterparts, visemes are mapped. An influential hypothesis claims that several phonemes can be mapped into a single visemic category (many-to-one phoneme-viseme mapping). In contrast, recent findings have challenged this view, reporting evidence for sub-visemic syllable discrimination. We aim to investigate whether Spanish words from the same visemic category can be identified or not. We designed a lip-reading task in which participants had to identify target words presented in silent video clips among 3 distractors differing in their visual resemblance from the target. Target words were identified above chance and significantly more than distractors from the same visemic category. Moreover, the error rate for distractors significantly decreased with decreasing visemic resemblance to the target. These results challenge the many-to-one phoneme-viseme mapping hypothesis.

Korean Lip-Reading: Data Construction and Sentence-Level Lip-Reading

Lip-reading is the task of inferring the speaker’s utterance from silent video based on learning of lip movements. It is very challenging due to the inherent ambiguities present in the lip movement such as different characters that produce the same lip appearances. Recent advances in deep learning models such as Transformer and Temporal Convolutional Network have led to improve the performance of lip-reading. However, most previous works deal with English lip-reading which has limitations in directly applying to Korean lip-reading, and moreover, there is no a large scale Korean lip-reading dataset. In this paper, we introduce the first large-scale Korean lip-reading dataset with more than 120 k utterances collected from TV broadcasts containing news, documentary and drama. We also present a preprocessing method which uniformly extracts a facial region of interest and propose a transformer-based model based on grapheme unit for sentence-level Korean lip-reading. We demonstrate that our dataset and model are appropriate for Korean lip-reading through statistics of the dataset and experimental results.

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos

The goal of this work is to reconstruct high quality speech from lip motions alone, a task also known as lip-to-speech. A key challenge of lip-to-speech systems is the one-to-many mapping caused by (1) the existence of homophenes and (2) multiple speech variations, resulting in a mispronounced and over-smoothed speech. In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives. Specifically, we incorporate (1) self-supervised speech representations to disambiguate homophenes, and (2) acoustic variance information to model diverse speech styles. Additionally, to better solve the aforementioned problem, we employ a flow based post-net which captures and refines the details of the generated speech. We perform extensive experiments on two datasets, and demonstrate that our method achieves the generation quality close to that of real human utterance, outperforming existing methods in terms of speech naturalness and intelligibility by a large margin. Synthesised samples are available at our demo page: https://mm.kaist.ac.kr/projects/LTBS.

Gender-expansive listeners invite us to reconsider voice categorization as a simple matter of pitch and timbre

Past literature has focused on pitch and timbre cues as the main determinants of the voice of an actor (a speaker or a singer). But in real life, judges rely on non-auditory cues such as the actor’s facial features, height, body size and shape. Resisting these visual biases could provide more accurate assessment of a voice but requires a stronger dissociation between auditory attributes and physical appearance, an ability hypothetically acquired by gender-expansive participants. In three online studies, we examine Faching (allocating voices into traditional categories). In study 1, 166 participants (85 cis, 81 gender-expansive) rated 144 audio (A) samples from 18 different actors (3 from each major Fach category, excluding countertenor) along a slider labeled low/dark to high/bright. Participants then guessed the voice types of the same actors in silent videos (V), before rating them in AV combinations. The cis group exhibited 30% more visual bias than the gender-expansive group. To further understand how gender-expansive participants obtain this benefit, we manipulated the fundamental frequency (in study 2) or the vocal tract length (in study 3) by ± 3 semitones from the original stimuli. Preliminary findings suggest that the two groups differ in the face of timbre but not pitch manipulations.

Impact of visual enhancement and color conversion algorithms on remote sound recovery from silent videos

AbstractThe visual microphone is a technique for remote sound recovery that extracts sound information from tiny pixel‐scale vibrations in a video. Despite having demonstrated success in sound recovery, the impact of various visual enhancement and color conversion algorithms applied on the video before the sound recovery process has not been explored. Thus, it is important to investigate these effects have on the recovered sound quality, as the vibrations are so small the effects play an important role. This work experimented with different color to grayscale conversions and visual enhancement algorithms on 576 videos, and found that the recovered sound quality is indeed greatly affected by the choice of algorithms. The best conversion algorithms were found to be the average of the red, green and blue color channels and the perceptual lightness in the CIELAB color space, improving the recovered sound quality by up to 23.22%. Furthermore, visual enhancement techniques such as gamma correction have been found to corrupt vibration information, leading to a 22.47% drop in recovered sound quality in one of the tested videos. Therefore, it is advisable to avoid or minimize the use of visual enhancement techniques for remote sound recovery to prevent the elimination of useful subtle vibrations.

Examination of medical students' opinions on multimedia learning materials according to social cues: focusing on sound principles.

Although interest in various forms of learning media is increasing due to the coronavirus disease 2019 (COVID-19) pandemic there is relatively little research on influencing student motivation by intervening in cognitive processing. The purpose of this study was to present the optimal form of learning materials provided to medical students. This study provided learning materials in class at a level according to social cues (script, video [artificial intelligence (AI) voice], video [professor voice]) based on the principle of voices among the principles of personalization, voices, image, and embodiment of social cues in multimedia learning, and surveyed students' opinions. There was no statistically significant difference according to social clues in satisfaction and learning help, but both appeared in the order of silent videos containing the professor's voice, followed by videos containing the AI voice. This study is significant in that there is no research on the impact of student motivation on the provision of learning materials for medical school education in Korea, and we hope that it will help provide learning materials for self-directed learning of medical students in the post-COVID-19.

Factors Affecting Visual-Only Speech Recognition in Individuals With Cochlear Implants.

Objectives: Several reports have underlined the benefits of speechreading (visual-only speech recognition) on speech recognition in individuals with hearing loss after cochlear implantation (CI). However, the factors that would affect the ability of speechreading are unclear. The aim of the present study is to assess the factors that affect speechreading abilities in CI users. Methods: A total of 104 participants were enrolled in this retrospective study. They viewed silent videos of sentences being spoken by a model and were tasked with repeating what they thought had been said. They were tested under audio-only and visual-only conditions. Factors (such as the age, the age of CI; <3 years old vs 3-8 years old; male vs female, etc) believed to affect speechreading abilities were analyzed. Results: The age range of the participants is 8 to 34 years. CI users showed significantly different speechreading abilities among themselves. The authors found that age and hearing loss at 3 to 8 years of age were positively related to superior speechreading recognition scores. Conclusions: CI users followed a more complex method of perceptual compensation. Those who have suffered hearing loss between 3 and 8 years of age are more sensitive to developing an advantage in speech recognition by using speechreading. Older age positively affects speechreading abilities; thus, the more experience CI has, the greater speechreading ability they may exhibit.

Cross-cultural Differences in Using Nonverbal Behaviors to Identify Indirect Replies

The ability to decode nonverbal cues is essential for effective cross-cultural communication. Despite the significance of nonverbal communication, research in this area has primarily focused on spoken language. This is particularly problematic during indirect communication, where there is a discrepancy between the surface meaning and the true intention (e.g., “Well, there is still room for improvement in your writing skills”). Misinterpretation of nonverbal cues during indirect communication can impede the decoding of true intention and potentially create hostile situations. The present study investigated cross-cultural differences in the use of nonverbal cues in decoding indirect messages. British and Chinese raters watched silent video clips and identified the types of replies from models of their own and the other culture. Results revealed that British raters were able to recognize indirect replies from British models but not from Chinese models above chance level, whereas Chinese raters were able to recognize indirect replies from models of both cultures above chance level. Furthermore, British raters showed higher accuracy and confidence in identifying indirect replies from British models than Chinese models. In contrast, Chinese raters were equally skilled and confident in identifying indirect replies from both British and Chinese models. Additionally, British and Chinese raters employed different nonverbal cues to recognize indirect replies from models of their respective cultures. These findings underscore the importance of cross-cultural differences in identifying indirect replies through nonverbal communication and provide insights to enhance intercultural communication between British and Chinese individuals.

A representation of abstract linguistic categories in the visual system underlies successful lipreading

There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Much of the ambiguity stems from a lack of behavioral grounding and neurophysiological analyses that cannot disentangle high-level linguistic and phonetic/energetic contributions from visual speech. To address this, we recorded EEG from human observers as they watched silent videos, half of which were novel and half of which were previously rehearsed with the accompanying audio. We modeled how the EEG responses to novel and rehearsed silent speech reflected the processing of low-level visual features (motion, lip movements) and a higher-level categorical representation of linguistic units, known as visemes. The ability of these visemes to account for the EEG – beyond the motion and lip movements – was significantly enhanced for rehearsed videos in a way that correlated with participants’ trial-by-trial ability to lipread that speech. Source localization of viseme processing showed clear contributions from visual cortex, with no strong evidence for the involvement of auditory areas. We interpret this as support for the idea that the visual system produces its own specialized representation of speech that is (1) well-described by categorical linguistic features, (2) dissociable from lip movements, and (3) predictive of lipreading ability. We also suggest a reinterpretation of previous findings of auditory cortical activation during silent speech that is consistent with hierarchical accounts of visual and audiovisual speech perception.

Language learning using muted or wordless videos - A creativity-based edutainment learning forum

From the teacher’s perspective, English language enrichment is more applicable and easier when using videos, resulting in a surge in the student’s receptivity. This paper attempts to bridge the gap by using silent videos to stimulate creative writing with a constructivism paradigm. Conducted in an Engineering college in the rural part of South India, two classes with 60 students each (both male and female) from first-year engineering (heterogeneous classes) at CEFR B1 level were chosen for the study. The researchers were the course instructors themselves, with fifteen-minute silent sports videos used for both groups. A sports video with audio was used for the controlled group, whereas one without audio was used for the target group. The controlled group tape-scripted the video content as they listened, while the experimental group created the script for the video on their own. The scripts were assessed for language quality based on vocabulary usage, sentence formation, and choice of words. The assessment details demonstrated that the experimental group students had demonstrated better scriptwriting skills compared to the control group students, who had relied on the audio and tried to paraphrase the words they had heard, leading to unclear scripting. This research showed that silent videos also help in grasping the English language by ESL learners, especially in creative writing and script drafting, evidentially proving that silent videos stimulate autonomous writing among students as they do not depend on audio for tape scripting. It further enhanced their writing skills with creative ability, and, further, the students preferred silent videos over audio videos due to better outcomes.

Eye tracking as feedback tool in physics teacher education

The ability to direct pupils’ attention to relevant information during the experimental process is relevant for all science teachers. The aim of this article is to investigate the effects of training the ability of prospective physics teachers to direct attention during the presentation of experiments with eye tracking visualizations of pupils’ visual attention as a feedback tool. Many eye tracking studies in the field of learning use eye movement recordings to investigate the effectiveness of an instructional design by varying cues or the presentation format. Another important line of research relates to study the teacher’s gaze in a real classroom setting (mobile eye tracking). Here we use eye tracking in a new and innovative way: Eye tracking is used as a feedback tool for prospective teachers, showing them the effects of their verbal moderations when trying to direct their pupils’ attention. The study is based on a mixed methods approach and is designed as a single factor quasi-experiment with pre-post measurement. Pre- and post-test are identical. Prospective teachers record their verbal moderations on a “silent” experimental video. The quality of the moderation is rated by several independent physics educators. In addition, pupils’ eye movements while watching the videos are recorded using eye tracking. The resulting eye movements are used by the lecturer to give individual feedback to the prospective teachers, focusing on the ability to control attention in class. The effect of this eye tracking feedback on the prospective teachers is recorded in interviews. Between the pre-test and the post-test, the results show a significant improvement in the quality of the moderations of the videos. The results of the interviews show that the reason for this improvement is the perception of one’s own impact on the pupils’ attention through eye tracking feedback. The overall training program of moderating “silent videos” including eye tracking as a feedback tool allows for targeted training of the verbal guidance of the pupils’ attention during the presentation of experiments.

Semantic reconstruction of continuous language from non-invasive brain recordings.

A brain-computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, non-invasive language decoders can only identify stimuli from among a small set of words or phrases. Here we introduce a non-invasive decoder that reconstructs continuous language from cortical semantic representations recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech and even silent videos, demonstrating that a single decoder can be applied to a range of tasks. We tested the decoder across cortex and found that continuous language can be separately decoded from multiple regions. As brain-computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation and found that subject cooperation is required both to train and to apply the decoder. Our findings demonstrate the viability of non-invasive language brain-computer interfaces.

Should Preoperative Information Before Impacted Third Molar Extraction Be Visual, Verbal, or Both?

Offering preoperative information is one option to help patients cope with the anxiety associated with oral and maxillofacial surgical procedures. Although this information is commonly delivered verbally, written information, audio recordings, and videos are also recommended. This study aimed to evaluate the effects of different preoperative information techniques on patients' anxiety levels before and after third molar tooth extraction. A single-blind randomized controlled clinical trial with parallel groups was performed. 92 patients who fulfilled the inclusion criteria were randomly assigned to one of four groups. Group 1 was provided information via a silent video with subtitles. Group 2 was informed via a video with background audio. Group 3 was provided a written information brochure. Group 4, designated as the control group, was informed verbally. The Spielberger state anxiety inventory (STAI-S), dental fear scale (DFS), modified dental anxiety scale (MDAS), and visual analog scale (VAS) were used pre- and postoperatively to evaluate the dental anxiety of patients. The primary predictor variable was the preoperative information method. The primary outcome variables were the scores of STAI-S, DFS, MDAS, and VAS. Gender was also set as a covariate variable. Data analysis was performed with the IBM SPSS Statistics 25.0 software program. Unless otherwise stated, P<.05 was considered statistically significant. A total of 86 patients scheduled for impacted third molar extractions under local anesthesia were included in the study. There was no statistically significant difference among the groups in the mean age (P=.275) or distribution of gender (P=.070). Compared to preoperative values, the postoperative scores of MDAS and VAS were decreased significantly in group 1 (P=.002, P=.003, respectively) and group 3 (P=.007; P=.009, respectively). No statistically significant differences were found between the preoperative and postoperative scores of STAI-S, DFS, MDAS, and VAS in group 2 (P=.300, P=.028, P=.063, P=.038, respectively) and group 4 (P=.127, P=.587, P=.072, P=.016, respectively). The findings indicate that visual or written material, without audio or oral delivery, substantially reduced the anxiety associated with oral and maxillofacial surgery. Therefore, clinicians should preferentially provide information via written methods to control patients' anxiety before impacted third molar surgery.

Effects of sex and sex-related facial traits on trust and trustworthiness: An experimental study.

The ability to trust others, including strangers, is a prerequisite for human cooperation. Economically it is not rational to trust strangers, as trust can be easily exploited. Still, generally, the level of trust toward strangers is relatively high. Trust is closely related to trustworthiness: when trusting others, one expects them to reciprocate. Some individuals elicit more trust than others. Apparently, humans use subtle cues for judging the trustworthiness of their interaction partners. Here, we report on an experiment that investigates trust and trustworthiness in a population of 176 mainly Dutch students. The aims of our study were: (1) to investigate how the sex of interaction partners and their facial appearance (femininity/masculinity) affect the degree of trust and trustworthiness, compared to fully anonymous conditions; (2) to test whether individuals who elicit trust in their interaction partners are trustworthy themselves. Each subject of our experiment played five one-shot Trust Games: one with an anonymous interaction partner, and four "personalized" games after seeing a 20 s silent video of their interaction partner (twice same-sex, and twice opposite-sex). The degree of facial sexual dimorphism was investigated with geometric morphometrics based on full-face photographs. Our results revealed that, despite the already high level of trust in the anonymous setting, the personalization of interactions had a clear effect on behavior. Females elicited more trust in partners of both sexes. Interestingly, females with more feminine faces elicited less trust in both male and female partners, while males with more masculine facial shape were more trusted by females, but less trusted by males. Neither sex nor facial femininity/masculinity predicted trustworthiness. Our results demonstrate that (1) sex and sex-related facial traits of interaction partners have a clear effect on eliciting trust in strangers. However, (2) these cues are not reliable predictors of actual trustworthiness.

The EEG Study on Addictive Inducement of Game Disorder in Undergraduate Students

Game disorder can reduce the ability of cognitive control and lead to negative consequence in many aspects of the individual life. To explore the addictive inducement of game disorder, the EEG evoked by video, flash or audio of game video was analysed. The power spectral density on δ, θ, α, β, γ and high γ frequencies was employed as quantitative features. The results reveal that the addicted and non-addicted individuals may be more sensitive to audio and flash of game video, respectively. The silent video and flash of game video can active brain activities by suppressing β and γ frequencies in all the individuals. Besides, the addicted ones appear extra suppressed brain activities evoked by audio on θ and α waves of frontal area and on θ waves of visual area. This may be the remarkable electrophysiological marker between the addicted and non-addicted individuals. Furthermore, this study also demonstrates that impulsive and self-disciplined may be the underlying personality factors of game disorder for multiplayer online battle arena games. Moreover, this kind of game may not increase the risk of autism spectrum disorder.

Cross-Modal Language Modeling in Multi-Motion-Informed Context for Lip Reading

We observe that for lip reading, the language is locally transformed, instead of globally transformed, i.e., speaking and writing follow the same basic grammar rules. In this work, we present a cross-modal language model to tackle the lip-reading challenge on silent videos. Compared to previous works, we consider multi-motion-informed contexts composed of multiple lip-motion representations from different subspaces to guide decoding via the source-target attention mechanism. We present a piece-wise pre-training strategy inspired by multi-task learning to pre-train a visual module to generate multi-motioninformed contexts for cross-modality and pre-train a decoder to generate texts for language modeling. Our final large-scale model outperforms baseline models on four datasets: LRS2, LRS3, LRW, and GRID. We will open our source code on GitHub.

Dual-flow Spatio-temporal Separation Network for Lip Reading

Lip reading is a task of predicting the corresponding language information in a silent video, which has attracted a lot of attention in recent years. Its key is to capture temporal and spatial features from lip motion videos and decode them. In the past, lip reading methods based on deep learning mostly adopt the form of spatio-temporal series connection, which first extracts spatial features, and then carries out global time-domain modeling on this basis. The spatial information extracted by the current approach is insufficient. To get more abundant spatio-temporal video representation and fully integrate the features from different viewpoints, this paper proposes a novel lip motion feature extraction framework, Dual-flow Spatio-temporal Separation Network (DSSN). Specifically, we adopt an end-to-end double tower structure to model the temporal information and spatial information respectively, and carry out feature fusion through collaborative learning. Finally, we evaluate our proposed model on the OuluVS2 lip reading dataset. Experiments show that our method outperforms baseline models.