Mute Video Research Articles

Language for Specific Purposes and Audio Description Tasks: A Case Study

In the Foreign Language (FL) classroom, video description and dubbing have recently been introduced for the development of communicative skills in various types of didactic methods (Vermeulen and Ibáñez 2014, 2017; Talaván 2020; Ávila-Cabrera and Rodríguez-Arancón 2021). This paper offers an innovative didactic proposal of Audio Description (AD) in the English for Tourism (ET) classroom. It supports the hypothesis that authentic tasks enhance communicative skills in LSP (Language for Specific Purposes) education (Melnichuk et al. 2017). It presents a quasi-experimental study of student performance in two different productive tasks (writing and speaking). The study follows a Task-Based Learning (TBL) approach and addresses a group of 119 students within an undergraduate program of ET (B2 level) at a Spanish distance-education university (UNED). As a post-task, a twofold rubric, designed in a previous teaching project, scored and analyzed student writing and speaking responses to an AD script of a mute video about a touristic place of their choice. Results show that implementing an AD task in the LSP classroom had a positive effect on students’ marks in their final test. In addition, the number of students who completed the writing task was around 50% greater than those who completed the speaking task. Introducing ICT tools in the LSP classroom seems to be positive, and it will be argued that developing digital technologies, such as mobile apps, may be a necessary step to foster future professionals’ language performance and competences.

Audiovisual Perception of Lexical Stress: Beat Gestures and Articulatory Cues.

Human communication is inherently multimodal. Auditory speech, but also visual cues can be used to understand another talker. Most studies of audiovisual speech perception have focused on the perception of speech segments (i.e., speech sounds). However, less is known about the influence of visual information on the perception of suprasegmental aspects of speech like lexical stress. In two experiments, we investigated the influence of different visual cues (e.g., facial articulatory cues and beat gestures) on the audiovisual perception of lexical stress. We presented auditory lexical stress continua of disyllabic Dutch stress pairs together with videos of a speaker producing stress on the first or second syllable (e.g., articulating VOORnaam or voorNAAM). Moreover, we combined and fully crossed the face of the speaker producing lexical stress on either syllable with a gesturing body producing a beat gesture on either the first or second syllable. Results showed that people successfully used visual articulatory cues to stress in muted videos. However, in audiovisual conditions, we were not able to find an effect of visual articulatory cues. In contrast, we found that the temporal alignment of beat gestures with speech robustly influenced participants' perception of lexical stress. These results highlight the importance of considering suprasegmental aspects of language in multimodal contexts.

The effect of candidates and assessors culture on nonverbal expression and nonverbal judgments in the job interview

Interviewing candidates poses various challenges when the candidate and the assessor come from different cultural backgrounds. A subset of these challenges derives from cultural differences in producing and interpreting nonverbal behaviors. The present study analyzed the nonverbal behaviors of 379 British and 313 Asian candidates who completed personality assessments and video-interviews for graduate positions in a financial institution. Interviewees produced significantly different types and rates of facial expressions across cultures; British candidates changed their facial expressions more often, and Asian candidates exhibited more positive emotions. The relationships between personality traits and nonverbal behaviors also varied across cultures, indicating that personality is expressed differently. Furthermore, muted videos of 304 candidates were rated by 3 British/Irish assessors based in London (UK) or Loughrea (Ireland) and 3 Chinese assessors based in Shanghai (China) on job fit, attractiveness, and personality. Candidates received higher ratings on job fit and attractiveness from assessors from their own culture. Overall, Candidates were assessed somewhat accurately (average r = 0.23). Candidates were judged more accurately in the London/Loughrea office condition. British candidates were judged more accurately than Asian candidates in the London/Loughrea condition. The implications of these findings for the fair and accurate selection of employees in cross-cultural settings are discussed.

Embodying Similarity and Difference: The Effect of Listing and Contrasting Gestures During U.S. Political Speech.

Public speakers like politicians carefully craft their words to maximize the clarity, impact, and persuasiveness of their messages. However, these messages can be shaped by more than words. Gestures play an important role in how spoken arguments are perceived, conceptualized, and remembered by audiences. Studies of political speech have explored the ways spoken arguments are used to persuade audiences and cue applause. Studies of politicians' gestures have explored the ways politicians illustrate different concepts with their hands, but have not focused on gesture's potential as a tool of persuasion. Our paper combines these traditions to ask first, how politicians gesture when using spoken rhetorical devices aimed at persuading audiences, and second, whether these gestures influence the ways their arguments are perceived. Study 1 examined two rhetorical devices-contrasts and lists-used by three politicians during U.S. presidential debates and asked whether the gestures produced during contrasts and lists differ. Gestures produced during contrasts were more likely to involve changes in hand location, and gestures produced during lists were more likely to involve changes in trajectory. Study 2 used footage from the same debates in an experiment to ask whether gesture influenced the way people perceived the politicians' arguments. When participants had access to gestural information, they perceived contrasted items as more different from one another and listed items as more similar to one another than they did when they only had access to speech. This was true even when participants had access to only gesture (in muted videos). We conclude that gesture is effective at communicating concepts of similarity and difference and that politicians (and likely other speakers) take advantage of gesture's persuasive potential.

To trust or not to trust? Face and voice modulation of virtual avatars

Introduction: This study explores the graduated perception of apparent social traits in virtual characters by experimental manipulation of perceived affiliation with the aim to validate an existing predictive model in animated whole-body avatars.Methods: We created a set of 210 animated virtual characters, for which facial features were generated according to a predictive statistical model originally developed for 2D faces. In a first online study, participants (N = 34) rated mute video clips of the characters on the dimensions of trustworthiness, valence, and arousal. In a second study (N = 49), vocal expressions were added to the avatars, with voice recordings manipulated on the dimension of trustworthiness by their speakers.Results: In study one, as predicted, we found a significant positive linear (p < 0.001) as well as quadratic (p < 0.001) trend in trustworthiness ratings. We found a significant negative correlation between mean trustworthiness and arousal (τ = −.37, p < 0.001), and a positive correlation with valence (τ = 0.88, p < 0.001). In study two, wefound a significant linear (p < 0.001), quadratic (p < 0.001), cubic (p < 0.001), quartic (p < 0.001) and quintic (p = 0.001) trend in trustworthiness ratings. Similarly, to study one, we found a significant negative correlation between mean trustworthiness and arousal (τ = −0.42, p < 0.001) and a positive correlation with valence (τ = 0.76, p < 0.001).Discussion: We successfully showed that a multisensory graduation of apparent social traits, originally developed for 2D stimuli, can be applied to virtually animated characters, to create a battery of animated virtual humanoid male characters. These virtual avatars have a higher ecological validity in comparison to their 2D counterparts and allow for a targeted experimental manipulation of perceived trustworthiness. The stimuli could be used for social cognition research in neurotypical and psychiatric populations.

Evaluation of veterinary students’ suture performance according to three different instructional modalities

Background: Higher education attempts to ameliorate learning experience through match between learning subjects and students' learning styles. Aim: This study evaluates efficacy of three different instructional modalities aimed at teaching veterinary students how to make simple suture knots. Methods: A cohort of 43 fourth-year students were split into 3 groups and provided with different instructional modalities: presentation with pictures and descriptions, hard copy text, and muted video. Student’s surgical simulation performance was evaluated. Then, they answered a 23 question-survey, Fleming VARK questionnaire based, investigating their learning profile. Kruskal Wallis test evaluated different instructional modalities effect on student’s performance. Chi square test assessed differences between instructional modalities and learning profiles, profile self evaluation, and training session comprehension. Results: Students shown auditory unimodal VARK profile (16/43), did not know their learning profile (26/43), and favoured personalised teaching strategies (43/43). No differences were found for: surgical performance, except for forceps handling; and between instructional modalities: either for learning profiles (p-value=0.43), or profile self evaluation (p-value=0.42). Differences were found between instructional modalities and training session comprehension. As limitations, auditory instructional modalities, participants’ age and gender were not recorded or evaluated. Conclusion: Our study provides feedback on modern teaching modalities in which students play a kye role. Participants showed a variety of learning profiles although displaying no significant performance differences.

Effect of loudspeaker position and model on cognitive load, quality, clarity, and sentence recognition performance

We sought to determine the effects of classroom loudspeaker position, model of loudspeaker, and talker accent on mental effort or cognitive load, word clarity, sound quality, and word recognition. Thirty-one typically-developed university students whose native language is English rated and transcribed recorded sentences produced by a native English talker and a native Dutch talker speaking English. The listeners were seated in acoustically pre-calibrated locations in a classroom and were presented 10 sentences at conversational level in each of four randomized conditions: from a conventional classroom loudspeaker in front or mounted above them, and from a loudspeaker optimized for speech reproduction placed in front or mounted above them. At the front of the classroom was a muted video recording of one of the talkers delivering an online lecture. Results showed front-located speakers yielded less cognitive effort, better sound quality, and better word clarity—showing importance of congruence of loudspeaker/talker position.

Inferential emotion tracking (IET) reveals the critical role of context in emotion recognition.

The ability to recognize others' emotions is critical for social interactions. It is widely assumed that recognizing facial expressions predominantly determines perceived categorical emotion, and contextual information only coarsely modulates or disambiguates interpreted faces. Using a novel method, inferential emotion tracking, we isolated and quantified the contribution of visual context versus face and body information in dynamic emotion recognition. Even when faces and bodies were blurred out in muted videos, observers inferred the emotion of invisible characters accurately and in high agreement based solely on visual context. Our results further show that the presence of visual context can override interpreted emotion categories from face and body information. Strikingly, we find that visual context determines perceived emotion nearly as much and as often as face and body information does. Visual context is an essential and indispensable element of emotion recognition: Without context, observers can misperceive a person's emotion over time. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Horses form cross-modal representations of adults and children.

Recently, research on domestic mammals' sociocognitive skills toward humans has been prolific, allowing us to better understand the human-animal relationship. For example, horses have been shown to distinguish human beings on the basis of photographs and voices and to have cross-modal mental representations of individual humans and human emotions. This leads to questions such as the extent to which horses can differentiate human attributes such as age. Here, we tested whether horses discriminate human adults from children. In a cross-modal paradigm, we presented 31 female horses with two simultaneous muted videos of a child and an adult saying the same neutral sentence, accompanied by the sound of an adult's or child's voice speaking the sentence. The horses looked significantly longer at the videos that were incongruent with the heard voice than at the congruent videos. We conclude that horses can match adults' and children's faces and voices cross-modally. Moreover, their heart rates increased during children's vocalizations but not during adults'. This suggests that in addition to having mental representations of adults and children, horses have a stronger emotional response to children's voices than adults' voices.

Cortical tracking of formant modulations derived from silently presented lip movements and its decline with age.

The integration of visual and auditory cues is crucial for successful processing of speech, especially under adverse conditions. Recent reports have shown that when participants watch muted videos of speakers, the phonological information about the acoustic speech envelope, which is associated with but independent from the speakers' lip movements, is tracked by the visual cortex. However, the speech signal also carries richer acoustic details, for example, about the fundamental frequency and the resonant frequencies, whose visuophonological transformation could aid speech processing. Here, we investigated the neural basis of the visuo-phonological transformation processes of these more fine-grained acoustic details and assessed how they change as a function of age. We recorded whole-head magnetoencephalographic (MEG) data while the participants watched silent normal (i.e., natural) and reversed videos of a speaker and paid attention to their lip movements. We found that the visual cortex is able to track the unheard natural modulations of resonant frequencies (or formants) and the pitch (or fundamental frequency) linked to lip movements. Importantly, only the processing of natural unheard formants decreases significantly with age in the visual and also in the cingulate cortex. This is not the case for the processing of the unheard speech envelope, the fundamental frequency, or the purely visual information carried by lip movements. These results show that unheard spectral fine details (along with the unheard acoustic envelope) are transformed from a mere visual to a phonological representation. Aging affects especially the ability to derive spectral dynamics at formant frequencies. As listening in noisy environments should capitalize on the ability to track spectral fine details, our results provide a novel focus on compensatory processes in such challenging situations.

Language Learning Under Varied Conditions: Neural Indices of Speech Perception in Bilingual Turkish-German Children and in Monolingual Children With Developmental Language Disorder (DLD).

Lateral temporal measures of the auditory evoked potential (AEP) including the T-complex (positive Ta and negative Tb), as well as an earlier negative peak (Na) index maturation of auditory/speech processing. Previous studies have shown that these measures distinguish neural processing in children with typical language development (TD) from those with disorders and monolingual from bilingual children. In this study, bilingual children with Turkish as L1 and German as L2 were compared with monolingual German-speaking children with developmental language disorder (DLD) and monolingual German-speaking children with TD in order to disentangle effects of limited language input vs. reduced perceptual abilities in the processing of speech and non-speech stimuli. Sensory processing reflected by the T-complex (or from lateral temporal electrode sites) was compared in response to a German vowel and a sine-wave tone in the three groups of children, ages 5 through 6 years. Stimuli were presented while children watched a muted video. Auditory evoked potentials (AEPs) were time-locked to the vowels and tones. AEPs to the frequent (standard) stimuli within an oddball paradigm were analyzed at the left (T7) and right (T8) temporal electrode sites.The results revealed language status (monolingual, bilingual, and DLD), stimulus (vowel and tone), and language test measures (receptive and expressive) all influenced the T-complex amplitudes. Particularly, the peak amplitude of Ta was modulated by language status and stimulus type. Bilingual children had significantly more negative Ta responses than the monolingual children with TD for both vowels and tones while DLD children differed from TD children only for the vowel stimulus. The amplitude of the T-complex was overall more negative at the left than at the right site. The Na peak latency was longer for the bilingual group than that observed for the two monolingual groups. The Tb latency was shorter for DLD and bilingual groups than that for TD children in the vowel condition, but no such latency difference between DLD and bilingual children was found. We suggest that the attenuated T-complex for bilingual children indicates continued plasticity of the auditory cortex to allow for learning of novel, second-language speech sounds.

Using Muted-Video Enactments to Develop Sociolinguistic Awareness

Silent films have long been shown during language classes to ease contextual and pragmatic acquisition. Equally empowering, muted videos are commonly used to help students to learn culture-laden and fixed expressions. They report to appreciate and enjoy using visual cues to understand and then appropriately use unfamiliar language ,and especially to cope with novel situations such as job interviews. This study analyses students’ voiceover responses to a series of muted videos shown during their participation in an oral communication skills course intended to develop their sociolinguistic awareness. The researchers wanted to know the extent to which students benefited from the muted video task. A mixed method design was used to identify the most common challenges learners face with when using culture-laden expressions. Both self-developed, online pre and post-test, and focus group interviews were used as data collection tools. The participants, selected according to the convenience sampling method, were students studying at an English-medium state university in Turkey and taking a compulsory oral communications-oriented English course. Afterwards, participants were seen to have developed sensitivity towards using context-bound fixed phrases. The research data was also useful in helping to pinpoint common challenges for learners when using contextually fixed phrases in English.

Generalization of sustained neurophysiological effects of short-term auditory 13-Hz stimulation to neighbouring frequency representation in humans.

A fuller understanding of the effects of auditory tetanization in humans would inform better language and sensory learning paradigms; however, there are still unanswered questions. Here, we probe sustained changes in the event‐related potentials (ERPs) to 1020‐ and 980‐Hz tones following a rapid presentation of 1020‐Hz tone (every 75 ms, 13.3 Hz, tetanization). Consistent with some previous studies, we revealed the increase in the P2 ERP component after tetanization. Contrary to some other studies, we did not observe the expected N1 increase after tetanization even in the identical experimental sequence. We detected a significant N1 decrease after tetanization. Expanding previous research, we showed that P2 increase and N1 decrease are not specific to the stimulus type (tetanized 1020 Hz and non‐tetanized 980 Hz), suggesting the generalizability of tetanization effect to the not‐stimulated auditory tones, at least to those of the neighbouring frequency. The ERPs' tetanization effects were observed for at least 30 min—the most prolonged interval examined, consistent with the duration of long‐term potentiation, LTP. In addition, the tetanization effects were detectable in the blocks where the participants watched muted videos, an experimental setting that can be easily used in children and other challenging groups. Thus, auditory 13‐Hz stimulation affects brain processing of tones including those of neighbouring frequencies.

Identifying Racial Minorities' Nationality: Non-verbal Accent as a Cue to Cultural Group Membership.

Historically, racial appearance has been a common source of information upon which we categorize others, as have verbal accents. Enculturated non-verbal accents which are detected in facial expressions of emotion, hairstyle, and everyday behaviors, have also been found to exist. We investigated the effects of non-verbal accent on categorization and stereotyping when people are exposed to thin slices of behavior. The effects of racial essentialism, which inclines people to categorize and assess others by race, were also tested. In three studies, Australian participants were shown short, muted videos of target individuals performing everyday behaviors. The targets were of a minority (Asian) racial appearance, but half had been interracially adopted as babies and grew up in the Australian mainstream. The other half were foreign nationals who grew up in Asia. In Studies 1 and 2, Australian participants rated each target as Australian or foreign. In both studies, they correctly identified the targets at above chance levels. In Study 3, participants rated the targets on Australian and Asian stereotype traits. They were not told that some targets were Australian and some were foreign, but they nonetheless rated the congruent stereotypes more strongly. Lay theory of race moderated the effect of non-verbal accent, with a weaker effect among participants who endorsed racial essentialism. These preliminary findings reveal subtle effects of non-verbal accent as a cue to cultural group membership and invite further work into the effects of non-verbal accent on person perception and categorization processes.

Using a smartphone camera to analyse rotating and vibrating systems: Feedback on the SURVISHNO 2019 contest

A smartphone is a low-cost pocket wireless multichannel multiphysical data acquisition system: the use of such a device for noise and vibration analysis is a challenging task. To what extent is it possible to carry out relevant analysis from it? The Survishno conference, held in Lyon in July 2019, proposed a contest to participants based on this subject. Two challenges were proposed, wherein each a mute video showing an object moving/excited at different frequencies was provided. Due to the frequencies set and the video sampling characteristics, special effects occurred and are visible on both videos. From the first video, participants were asked to estimate the Instantaneous Angular Speed (IAS) of a rotating fan. From the second video, they were asked to perform the modal analysis of a cantilever beam. This paper gathers the interesting ideas proposed by the contestants and proposes a global method to solve these two problems. One major point of the paper might be the advantageous use of the rolling shutter effect, a well-known artefact of smartphone videos, to perform advanced mechanical analyses: the consideration of the unavoidable slight phase shift between the acquisition of each pixel opens up the possibility to perform a dynamic analysis at frequencies that are much higher than the video frame rate.

Accuracy in Judging Others' Personalities: The Role of Emotion Recognition, Emotion Understanding, and Trait Emotional Intelligence.

The ability to accurately judge others’ personality and the ability to accurately recognize others’ emotions are both part of the broader construct of interpersonal accuracy (IPA). However, little research has examined the association between these two IPA domains. Little is also known about the relationship between personality judgment accuracy and other socio-emotional skills and traits. In the present study, 121 participants judged eight traits (Big Five, intelligence, cooperativeness, and empathy) in each of 30 targets who were presented either in a photograph, a muted video, or a video with sound. The videos were 30 second excerpts from negotiations that the targets had engaged in. Participants also completed standard tests of emotion recognition ability, emotion understanding, and trait emotional intelligence. Results showed that personality judgment accuracy, when indexed as trait accuracy and distinctive profile accuracy, positively correlated with emotion recognition ability and was unrelated to emotion understanding and trait emotional intelligence. Female participants were more accurate in judging targets’ personality than men. These results provide support for IPA as a set of correlated domain-specific skills and encourage further research on personality judgment accuracy as a meaningful individual difference variable.

Language is the missing link in action-perception coupling: an EEG study

The paper reports an electrophysiological (EEG) study investigating how language is involved in perception–action relations in musically trained and untrained participants. Using an original backward priming paradigm, participants were exposed to muted point-light videos of violinists performing piano or forte nuances followed by a congruent vs. incongruent word. After the video presentation, participants were asked to decide whether the musician was playing a piano or forte musical nuance. EEG results showed a greater P200 event-related potential for trained participants at the occipital site, and a greater N400 effect for untrained participants at the central site. Musically untrained participants were more accurate when the word was semantically congruent with the gesture than when it was incongruent. Overall, language seems to influence the performance of untrained participants, for which perception–action couplings are less automatized.

"Are You Telling the Truth?" - Testing Individuals' Ability to Differentiate Between Truth and Deceit in Soccer.

In the present paper, we tested the ability of individuals to judge correctly whether athletes are lying or telling the truth. For this purpose, we first generated 28 videos as stimulus material: in half of the videos, soccer players were telling the truth, while in the other half, the same soccer players were lying. Next, we tested the validity of these video clips by asking N = 65 individuals in a laboratory experiment (Study 1a) and N = 52 individuals in an online experiment (Study 1b) to rate the level of veracity of each video clip. Results suggest that participants can distinguish between true and false statements, but only for some clips and not for others, indicating that some players were better at deceiving than others. In Study 2, participants again had to make veracity estimations, but we manipulated the level of information given, as participants (N = 145) were randomly assigned to one of three conditions (regular video clips, mute video clips, and only the audio stream of each statement). The results revealed that participants from the mute condition were less accurate in their veracity ratings. The theoretical and practical implications of these findings are discussed.

The Look of (Un)confidence: Visual Markers for Inferring Speaker Confidence in Speech

Evidence suggests that observers can accurately perceive a speaker’s static confidence level, related to their personality and social status, by only assessing their visual cues. However, less is known about the visual cues that speakers produce to signal their transient confidence level in the content of their speech. Moreover, it is unclear what visual cues observers use to accurately perceive a speaker’s confidence level. Observers are hypothesized to use visual cues in their social evaluations based on the cue’s level of perceptual salience and/or their beliefs about the cues that speakers with a given mental state produce. We elicited high and low levels of confidence in the speech content by having a group of speakers answer general knowledge questions ranging in difficulty while their face and upper body were video recorded. A group of observers watched muted videos of these recordings to rate the speaker’s confidence and report the face/body area(s) they used to assess the speaker’s confidence. Observers accurately perceived a speaker’s confidence level relative to the speakers’ subjective confidence, and broadly differentiated speakers as having low compared to high confidence by using speakers’ eyes, facial expressions, and head movements. Our results argue that observers use a speaker’s facial region to implicitly decode a speaker’s transient confidence level in a situation of low-stakes social evaluation, although the use of these cues differs across speakers. The effect of situational factors on speakers’ visual cue production and observers’ utilisation of these visual cues are discussed, with implications for improving how observers in real world contexts assess a speaker’s confidence in their speech content.

Interaction of Self-Regulation and Contextual Effects on Pre-attentive Auditory Processing: A Combined EEG/ECG Study

Environmental changes are not always within the focus of our attention, and sensitive reactions (i.e., quicker and stronger responses) can be essential for an organism's survival and adaptation. Here we report that neurophysiological responses to sound changes that are not in the focus of attention are related to both ambient acoustic contexts and regulation ability. We assessed electroencephalograph (EEG) mismatch negativity (MMN) latency and amplitude in response to sound changes in two contexts: ascending and descending pitch sequences while participants were instructed to attend to muted videos. Prolonged latency and increased amplitude of MMN at fronto-central region occurred in ascending pitch sequences relative to descending sequences. We also assessed how regulation related to the contextual effects on MMN. Reactions to changes in the ascending sequence were observed with the attention control (frontal EEG theta/beta ratio) indicating speed of reaction, and the autonomous regulation (heart-rate variability) indicating intensity of reaction. Moreover, sound changes in the ascending context were associated with more activation of anterior cingulate cortex and insula, suggesting arousal effects and regulation processes. These findings suggest that the relation between speed and intensity is not fixed and may be modified by contexts and self-regulation ability. Specifically, cortical and cardiovascular indicators of self-regulation may specify different aspects of response sensitivity in terms of speed and intensity.