Audiovisual Modalities Research Articles

Cross-Linguistic Recognition of Irony Through Visual and Acoustic Cues.

To avoid misunderstandings, ironic speakers may accompany their ironic remarks with a particular intonation and specific facial expressions that signal that the message should not be taken at face value. The acoustic realization of the ironic tone of voice differs from language to language, whereas the ironic face manifests the speaker's negative stance and might thus have a universal basis. We conducted a study on 574 participants speaking 6 different languages (French, German, Dutch, English, Mandarin, and Italian-the control group) to verify whether they could recognize ironic remarks uttered in Italian in three different modalities: watching muted videos, listening to audio tracks, and when both cues were present. We found that speakers of other languages could overall recognize irony uttered in Italian when all the markers were present, and they relied mostly on visual cues: In all these language groups, accuracy in the audio-only modality was always lower than accuracy in the video-only modality, although this trend was significant only for Chinese and Dutch participants. Moreover, the rate of recognition in the audio-visual modality was always significantly higher compared to the audio-only, while the difference between the audio-visual and the video-only modality was significant only for the English group. Overall, these results speak in favor of the hypothesis of a common basis for the visual expression of irony, whereas the acoustic markers alone do not constitute a reliable cue for the cross-linguistic recognition of irony.

Investigating Media Accessibility for the Deaf and Hard of Hearing in Iraq

Media accessibility, (MA) for short, is still largely uncharted research territory in the Translation Studies scene in Iraq. To examine the status quo of media accessibility for the (DHH) community, this study has three aims to address: first, to subject the Iraqi constitution to a thematic analysis to highlight the legal status of media accessibility; second, to examine the real actions taken by the government and non-governmental organizations to ensure that the d/Deaf and hard of hearing (DHH) can access this service; third, to probe the DHH community’s reception and their level of expectations towards the available (MA) service. The findings suggest that the Constitution of the Republic of Iraq contains Articles that refer directly to rights of people with special needs to access the public service, but no reference to the right of media accessibility (MA) is identified. The study also shows that sign language interpreting (SLI), though scarcely employed in very few TV channels, has been the only audiovisual modality available for DHH viewers to access TV news whereas subtitling for the d/Deaf and hard of hearing (SDH) modality has never been implemented.

Multimodal meaning making in news communication about immigration: using the NewsScape corpus to explore co-verbal images in TV news

The communication of news relies on semiotic resources besides language, including various audiovisual modes of representation. Owing to the difficulties associated with obtaining televisual data, the vast majority of research addressing multimodality in the news has been targeted at print news media, where various strategies in visual representation and patterns of interaction between verbal and visual modes have been discerned. Where televisual data has been interrogated, this has been based on a very limited number of data points. In this study, I exploit the NewsScape library – a massive multimodal corpus of news communication – to investigate multimodal representations of immigration in television news. Accessible via CQPWeb, the corpus is searched for target utterances refugees/(im)migrants have VERBed and refugees/(im)migrants are VERBing. The co-verbal images accompanying 474 utterances describing motion events are analyzed quantitatively and qualitatively. Among the results discussed are that refugees/migrants are depicted in large rather than small groups, that they are depicted in transit somewhere along the migratory journey rather than in countries of origin or destination countries, that they are depicted on land more than at sea, that they are depicted in security contexts, and that they are erased represented instead through abstract forms such as maps. Differences in the visual representation of people designated as ‘refugee’ versus ‘migrant’ are also observed and discussed.

‘If Madonna can … ’: emergence of the female indipop star in the 1990s MTV cultural economy

ABSTRACT As cinema moved from exhibition theatres to cable television in the late-1980s pre-liberal License Raj 1 economy, the competitive music label market in India pushed its executives to look beyond the established modes of music making and selling. This development also collided with the growing cultural demand for new and ‘fresh’ female voices, and innovative audio-visual modes inspired by the MTV consumerism model of the West. Additionally, music videos of international popstars such as Madonna, Michael Jackson, etc. started making waves in India through direct and indirect channels. As a result, the influence of western music video culture led the music industry officials to rope in ‘westernized in style, desi at heart’ female performers to represent the new wave of alternative non-film popular music – collectively popularised by music labels and the media as Hindi Pop or Indipop. Moreover, the cultural impact of American popstar Madonna’s stardom, performativity, and politics also played a critical role in the way the emerging Indipop ‘divas’ projected themselves, both in and outside the diegesis of their music videos, during the 90s MTV boom in India. Engaging with the textual and extratextual world of indipop music videos that belong to the ‘new women’ of the neoliberal-90s – Jasmine Bharucha, Alisha Chinai, Shweta Shetty, etc.—this paper attempts to track and locate the emergence of these ‘non-film’ 2 female voices in the traditional ‘song and dance’ aesthetic substructure of Bombay cinema.

The Role of Audiovisuals in Counter-Framing Dynamics: A Longitudinal Experiment on Causal Responsibility Attribution

ABSTRACT Opposing the interpretation of a societal problem with a countering framing is a common characteristic of the democratic discourse. While studies on the dynamics of such counter-framing situations point largely toward recency effects, it remains mostly unclear which characteristics of the initial frame influence these recency effects. Taking current developments in news reporting into account—the trend toward audio and audiovisual news—this study suggests that news framed and presented in textual and audio and audiovisual modalities can attenuate recency effects in counter-framing environments. Building upon the long-standing research strand of responsibility framing and theories of multimodal learning, the findings of this longitudinal probability-sample experiment indicate that the recency effects of frames largely endure even if the initial frame is presented in a multi-modality mode.

Fish behavior recognition based on an audio-visual multimodal interactive fusion network

In light of the challenges imposed by fish behavior recognition, which arise from environmental noise and dim lighting in aquaculture environments and adversely affect the effectiveness of unimodal recognition methods based on either sound or visual cues, this paper proposes a fish behavior recognition model, Mul-SEResNet50, based on the fusion of audio and visual information. To address issues such as image blurring and indistinct sounds in aquaculture environments, which hinder the effectiveness of multimodal fusion and complementary modalities, a multimodal interaction fusion (MIF) module is introduced. This module integrates audio-visual modalities at multiple stages to achieve a more comprehensive joint feature representation. To enhance complementarity during the fusion process, we designed a U-shaped bilinear fusion structure to fully utilize multimodal information, capture cross-modal associations, and extract high-level features. Furthermore, to address the potential loss of key features, a temporal aggregation and pooling (TAP) layer is introduced to preserve more fine-grained features by extracting both the maximum and average values within pooling regions. To validate the effectiveness of the proposed model, both ablation experiments and comparative experiments are conducted. The results demonstrate that Mul-SEResNet50 achieves a 5.04 % accuracy improvement over SEResNet50 without sacrificing detection speed. Compared to the state-of-the-art U-FusionNet-ResNet50 +SENet model, Mul-SEResNet50 achieves accuracy and F1 score improvements of 0.47 % and 1.32 %, respectively. These findings confirm the efficacy of the proposed model in terms of accurately recognizing fish behavior, facilitating the precise monitoring of fish behavior.

Multimodal Spatio-Temporal Framework for Real-World Affect Recognition

Deep learning models show great potential in applications involving video-based affect recognition, including human-computer interaction, robotic interfaces, stress and depression assessment, and Alzheimer's disease detection. The low complex Multimodal Diverse Spatio-Temporal Network (MDSTN) has been analysed to effectively capture spatio-temporal information from audio-visual modalities for affect recognition using the Acted Facial Expressions in the Wild (AFEW) dataset. The scarcity of data is handled by data augmented parallel feature extraction for visual network. Visual features extracted by carefully reviewing and customizing Convolutional 3D architecture over different ranges are combined to train a neural network for classification. Multi-resolution Cochleagram (MRCG) features from speech, along with spectral and prosodic audio features, are processed by a supervised classifier. The late fusion technique is explored to integrate audio and video modalities, considering their processing over different temporal spans. The MDSTN approach significantly boosts the accuracy of basic emotion recognition to 71.54% on the AFEW dataset. It demonstrates exceptional proficiency in identifying emotions such as disgust and surprise, thus exceeding current benchmarks in real-world affect recognition.

Relationships Between Hearing Status, Cognitive Abilities, and Reliance on Visual and Contextual Cues

Objectives: Visual and contextual cues facilitate speech recognition in suboptimal listening conditions (e.g., background noise, hearing loss, hearing aid signal processing). Moreover, successful speech recognition in challenging listening conditions is linked to cognitive abilities such as working memory and fluid intelligence. However, it is unclear which cognitive abilities facilitate the use of visual and contextual cues in individuals with normal hearing and hearing aid users. The first aim was to investigate whether individuals with hearing aid users rely on visual and contextual cues to a higher degree than individuals with normal hearing in a speech-in-noise recognition task. The second aim was to investigate whether working memory and fluid intelligence are associated with the use of visual and contextual cues in these groups. Design: Groups of participants with normal hearing and hearing aid users with bilateral, symmetrical mild to severe sensorineural hearing loss were included (n = 169 per group). The Samuelsson and Rönnberg task was administered to measure speech recognition in speech-shaped noise. The task consists of an equal number of sentences administered in the auditory and audiovisual modalities, as well as without and with contextual cues (visually presented word preceding the sentence, e.g.,: “Restaurant”). The signal to noise ratio was individually set to 1 dB below the level obtained for 50% correct speech recognition in the hearing-in-noise test administered in the auditory modality. The Reading Span test was used to measure working memory capacity and the Raven test was used to measure fluid intelligence. The data were analyzed using linear mixed-effects modeling. Results: Both groups exhibited significantly higher speech recognition performance when visual and contextual cues were available. Although the hearing aid users performed significantly worse compared to those with normal hearing in the auditory modality, both groups reached similar performance levels in the audiovisual modality. In addition, a significant positive relationship was found between the Raven test score and speech recognition performance only for the hearing aid users in the audiovisual modality. There was no significant relationship between Reading Span test score and performance. Conclusions: Both participants with normal hearing and hearing aid users benefitted from contextual cues, regardless of cognitive abilities. The hearing aid users relied on visual cues to compensate for the perceptual difficulties, reaching a similar performance level as the participants with normal hearing when visual cues were available, despite worse performance in the auditory modality. It is important to note that the hearing aid users who had higher fluid intelligence were able to capitalize on visual cues more successfully than those with poorer fluid intelligence, resulting in better speech-in-noise recognition performance.

Multimodality-guided Visual-Caption Semantic Enhancement

Video captions generated with single modality, e.g. video clips, often suffer from insufficient event discovery and inadequate scene description. Therefore, this paper aims to improve the quality of captions by addressing these issues through the integration of multi-modal information. Specifically, We first construct a multi-modal dataset and introduce the triplet annotations of video, audio and text, fostering a comprehensive exploration about the associations between different modalities. Build upon this, We propose to explore the collaborative perception of audio and visual concepts to mitigate inaccuracies and incompleteness in captions in vision-based benchmarks by incorporating audio-visual perception priors. To achieve this, we extract effective semantic features from visual and auditory modalities, bridge the semantic gap between audio-visual modalities and text, and form a more precise knowledge graph multimodal coherence checking and information pruning mechanism. Exhaustive experiments demonstrate that the proposed approach surpasses existing methods and generalizes well with the assistance of ChatGPT.

Informing Immersed Citizens: The Impact of Interactivity on Comprehending News in Immersive Journalism

Immersive journalism has emerged as an innovative journalistic approach promising enhanced attention and understanding through interactive, virtual environments. Previously, this was mostly tested through factual knowledge. However, evaluating immersive audio-visual modalities solely along factual knowledge falls short of considering, firstly, what about and how an immersive experience is remembered and, secondly, the importance of considering the perceptions surrounding information acquisition. Therefore, this study examines how interactivity in immersive journalism affects traditional as well as novel ways of comprehending news, such as episodic memory. In addition, we consider perceptions related to knowledge. We draw on a laboratory experiment (<em>N</em> = 149) testing the effect of three levels of interactivity provided (no interactivity vs. looking around vs. looking around + control over pace and order of storyline). Results indicate that a higher range of interactivity does not influence factual memory, but leads to an increase in perceived knowledge, thus indicating an illusion of knowledge. Moreover, there is a tendency to formulate more subjective takeaways in the high-interactivity condition, while interactivity did not influence the credibility evaluations. This provides partial empirical support for the credibility paradox of immersive journalism.

Importance of Visual Support Through Lipreading in the Identification of Words in Spanish Language.

We sought to examine the contribution of visual cues, such as lipreading, in the identification of familiar (words) and unfamiliar (phonemes) words in terms of percent accuracy. For that purpose, in this retrospective study, we presented lists of words and phonemes (adult female healthy voice) in auditory (A) and audiovisual (AV) modalities to 65 Spanish normal-hearing male and female listeners classified in four age groups. Our results showed a remarkable benefit of AV information in word and phoneme recognition. Regarding gender, women exhibited better performance than men in both A and AV modalities, although we only found significant differences for words but not for phonemes. Concerning age, significant differences were detected in word recognition in the A modality between the youngest (18-29 years old) and oldest (⩾50 years old) groups only. We conclude visual information enhances word and phoneme recognition and women are more influenced by visual signals than men in AV speech perception. On the contrary, it seems that, overall, age is not a limiting factor for word recognition, with no significant differences observed in the AV modality.

RainMind: Investigating Dynamic Natural Soundscape of Physiological Data to Promote Self-Reflection for Stress Management

Metaphorical auditory displays are increasingly recognized for data presentation and self-reflection. This article presents the design and evaluation of RainMind, a web-based soundscape application that represents physiological data with dynamic natural sounds for daily stress reflection. Based on different combinations of auditory display with visualization, we identified three modes: the Visual-Aided Mode (VAM), the Audio-Aided Mode (AAM), and the Audio-Visual Mode (AVM). Through a within-subject study involving 30 participants, we conducted a mixed-methods evaluation to assess the task load, engagement, and user experience among the three modes. The findings indicated that the combination of dynamic natural soundscapes with visualization (AVM) contributes to a lower task load compared to the other two modes. Moreover, dynamic natural soundscapes as metaphors for stress data significantly enhanced engagement and user experience of self-reflection compared to static natural sounds. Based on our study, we discuss the potential of leveraging dynamic natural soundscapes as a new way of data-driven self-reflection.

Discourse Markers Audiovisual Rendition into Modern Standard Arabic and Colloquial Egyptian Arabic

Discourse markers are omnipresent in language and are gaining more attention, especially, in audiovisual translation studies. This study aimed at examining the strategies used by translators in rendering six commonly used English discourse markers into Arabic. Moreover, it shed light on the discourse markers found in seven children’s animated movies and seeks to investigate the differences between three renditions of these movies: Arabic subtitles, Modern Standard Arabic dubbed version and colloquial Egyptian Arabic dubbed version. It also focused on the effect that the two audiovisual modes, subtitling and dubbing, have on the translation of these devices and the choices of translation strategies. The data consisted of 944 instances of the six discourse markers extracted from the movies’ original scripts and their equivalents taken from the Arabic subtitled and dubbed versions of the movies. The findings reveal that the avoidance of SL discourse marker strategy was used by translators in both modes of audiovisual translation. Moreover, the equivalents found in the colloquial Egyptian Arabic dubbed version were more diverse compared to the other two translations (the subtitled version and MSA dubbed version).

The Effect of Visual Articulatory Cues on the Identification of Mandarin Tones by Children With Cochlear Implants.

This study explored the facilitatory effect of visual articulatory cues on the identification of Mandarin lexical tones by children with cochlear implants (CIs) in both quiet and noisy environments. It also explored whether early implantation is associated with better use of visual cues in tonal identification. Participants included 106 children with CIs and 100 normal-hearing (NH) controls. A tonal identification task was employed using a two-alternative forced-choice picture-pointing paradigm. Participants' tonal identification accuracies were compared between audio-only (AO) and audiovisual (AV) modalities. Correlations between implantation ages and visual benefits (accuracy differences between AO and AV modalities) were also examined. Children with CIs demonstrated an improved identification accuracy from AO to AV modalities in the noisy environment. Additionally, earlier implantation was significantly correlated with a greater visual benefit in noise. These findings indicated that children with CIs benefited from visual cues on tonal identification in noise, and early implantation enhanced the visual benefit. These results thus have practical implications on tonal perception interventions for Mandarin-speaking children with CIs.

A cross-disciplinary evaluation of digitally recorded feedback in higher education

Research demonstrates that assessment feedback created using audio, video, and screencast recordings can offer advantages over text-based feedback. However, the majority of research and experience in this domain has largely been limited to a single disciplinary or cohort context. This project aimed to empirically investigate if recorded feedback (i.e. audio, video, and screencast) could be effectively implemented across different contexts, including disciplines. As part of this, teaching staff from five discrete subjects provided digitally recorded feedback to students on at least one assessment task. Assessment types included various forms of written assignments, completed by individuals or groups of students. This paper reports on survey data collected from 351 students who received recorded feedback or text-based feedback. Survey respondents were enrolled in five subjects across four disciplines (Education, Pharmacy, Engineering, and Management). To triangulate the survey findings, interview data from nine students are also included. Overall, the findings indicate that students in all disciplines found digitally recorded feedback to be more satisfying, more useful, and more engaging than text-based feedback alone. However, these outcomes differed across contexts; results tended to be elevated in subjects with smaller cohorts, and when richer audiovisual modalities were used. In two of the cases students, while still being overall positive, indicated that the feedback was less clear, usable and satisfying than indicated in other cases. These differences are explored and issues of teacher experience, cohort size, group assessment, and disciplinary cognate traditions are considered.

Options for subtitling English movie lyrics into Arabic

Subtitling lyrics poses a problem regarding conveying the content and the form, which include meaning, rhyme, rhythm, and other musical features. This requires having well-trained professional translators to master the rendering of those features in the target language, bearing in mind the constraints imposed by subtitling as an audio-visual mode. This study examines the Arabic subtitled version of English movie lyrics and explores the translator’s options in rendering their content and form. To this end, sixteen lyrics were extracted from five movies that have been subtitled. The compiled lyrics were classified into different categories based on the translation options proposed by Franzon (2008). The option of “translating lyrics without taking the music into consideration” was used 11 times, while the option of “aesthetic translation” was opted for four times. “Not translating the lyric” was used only once, while the option of “creating new lyrics for the music” was not adopted in any of the five movies. The study recommends two research areas: exploring dubbed Arabic versions of English movie lyrics and analyzing subtitled lyrics in other language pairs.

HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition

Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-aware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. Firstly, it incorporates hierarchical skip connections between the encoder and decoder to encourage intermediate layers to learn more meaningful representations and bolster masked audio-visual reconstruction. Secondly, hierarchical cross-modal contrastive learning is also exerted on intermediate representations to narrow the audio-visual modality gap progressively and facilitate subsequent cross-modal fusion. Finally, during downstream fine-tuning, HiCMAE employs hierarchical feature fusion to comprehensively integrate multi-level features from different layers. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models are publicly available at https://github.com/sunlicai/HiCMAE.

Effectiveness of understanding the post-operative instructions with respect to extraction of teeth in rural population through verbal, written and audio-visual aids.

Aim: To compare the efficacy between verbal, written & audio-visual modes of communication in relation to post-operative instructions after extraction of teeth. Materials & Methods: Ninety patients were equally divided into three groups. Group 1 were given verbal instructions, Group 2 in written format and Group 3- Audio visual mode. Their responses were recorded using a standard questionnaire after 24 hours of extraction and subjected to statistical analysis. Results: Significant differences (p<0.05) was found when the responses were compared between Group 1 and Group 3 and Group 2 and Group 3. Conclusion: Patients who received post-operative instructions through audio-visual mode (Group 3) had more understanding when compared to other groups. Incorporation of technology in communication has to be encouraged as it helps for better understanding. Bangladesh Journal of Medical Science Vol.23 (Special Issue) 2024 p.S48-S52

Student evaluation of an instructional video on how to use a national medicines information resource.

The objective of this project was to obtain students' perception of an educational companion video designed to increase the understanding of a national medicines information resource. An instructional video was developed to guide students in utilizing the online version of the Australian Medicines Handbook (AMH). All students enrolled in the Bachelor of Pharmacy program during 2021 were given access to this video. A cross-sectional electronic survey was administered to evaluate the video's effectiveness and its impact on students' confidence. The survey results were analysed using both descriptive and inferential statistics, in addition to qualitative analysis to identify common themes. Ethics approval was obtained prior to conducting the study. Most students (78%; n = 72/92) reported that watching the video increased their understanding. The most growth in confidence was seen by students in their first or second year of study. Fifty-four percent (n = 48/89) of students was very likely or extremely likely to recommend the video to others, and 37% (n = 33/89) of students were somewhat likely. Students found the instructional video to be useful and expressed a desire for similar content to be integrated into other facets of teaching. The audio-visual mode of delivery was regarded as effective for this context. Constructive feedback included suggestions of incorporating more advanced educational features such as how to interpret comparative medication charts and interaction checkers. The 'How to use the AMH' video is a good introductory resource for undergraduate Australian healthcare students. Our results indicate that this video would be best suited to complement the teaching of students early on in their studies.

Non-spatial inhibition of return attenuates audiovisual integration owing to modality disparities.

Although previous studies have investigated the relationship between inhibition of return (IOR) and multisensory integration, the influence of non-spatial has not been explored. The present study aimed to investigate the influence of non-spatial IOR on audiovisual integration by using a "prime-neutral cue-target" paradigm. In Experiment 1, which manipulated prime validity and target modality, the targets were positioned centrally, revealing significant non-spatial IOR effects in the visual, auditory, and audiovisual modalities. Analysis of relative multisensory response enhancement (rMRE) indicated substantial audiovisual integration enhancement in both valid and invalid target conditions. Furthermore, the enhancement was weaker for valid targets than for invalid targets. In Experiment 2, the targets were positioned above and below to rule out repetition blindness (RB); this experiment successfully replicated the results observed in Experiment 1. Notably, Experiments 1 and 2 consistently found that the correlation between modality differences and rMRE for valid targets indicated that differences in signal strength between visual and auditory modalities contributed to a reduction in audiovisual integration. However, the absence of correlation with the invalid target suggests that attention, as a key factor, may play a significant role in this process. The present study highlights how non-spatial IOR reduces audiovisual integration and sheds light on the complex interaction between attention and multisensory integration.