Audiovisual Modalities Research Articles

‘If Madonna can … ’: emergence of the female indipop star in the 1990s MTV cultural economy

ABSTRACT As cinema moved from exhibition theatres to cable television in the late-1980s pre-liberal License Raj 1 economy, the competitive music label market in India pushed its executives to look beyond the established modes of music making and selling. This development also collided with the growing cultural demand for new and ‘fresh’ female voices, and innovative audio-visual modes inspired by the MTV consumerism model of the West. Additionally, music videos of international popstars such as Madonna, Michael Jackson, etc. started making waves in India through direct and indirect channels. As a result, the influence of western music video culture led the music industry officials to rope in ‘westernized in style, desi at heart’ female performers to represent the new wave of alternative non-film popular music – collectively popularised by music labels and the media as Hindi Pop or Indipop. Moreover, the cultural impact of American popstar Madonna’s stardom, performativity, and politics also played a critical role in the way the emerging Indipop ‘divas’ projected themselves, both in and outside the diegesis of their music videos, during the 90s MTV boom in India. Engaging with the textual and extratextual world of indipop music videos that belong to the ‘new women’ of the neoliberal-90s – Jasmine Bharucha, Alisha Chinai, Shweta Shetty, etc.—this paper attempts to track and locate the emergence of these ‘non-film’ 2 female voices in the traditional ‘song and dance’ aesthetic substructure of Bombay cinema.

The Role of Audiovisuals in Counter-Framing Dynamics: A Longitudinal Experiment on Causal Responsibility Attribution

ABSTRACT Opposing the interpretation of a societal problem with a countering framing is a common characteristic of the democratic discourse. While studies on the dynamics of such counter-framing situations point largely toward recency effects, it remains mostly unclear which characteristics of the initial frame influence these recency effects. Taking current developments in news reporting into account—the trend toward audio and audiovisual news—this study suggests that news framed and presented in textual and audio and audiovisual modalities can attenuate recency effects in counter-framing environments. Building upon the long-standing research strand of responsibility framing and theories of multimodal learning, the findings of this longitudinal probability-sample experiment indicate that the recency effects of frames largely endure even if the initial frame is presented in a multi-modality mode.

Fish behavior recognition based on an audio-visual multimodal interactive fusion network

In light of the challenges imposed by fish behavior recognition, which arise from environmental noise and dim lighting in aquaculture environments and adversely affect the effectiveness of unimodal recognition methods based on either sound or visual cues, this paper proposes a fish behavior recognition model, Mul-SEResNet50, based on the fusion of audio and visual information. To address issues such as image blurring and indistinct sounds in aquaculture environments, which hinder the effectiveness of multimodal fusion and complementary modalities, a multimodal interaction fusion (MIF) module is introduced. This module integrates audio-visual modalities at multiple stages to achieve a more comprehensive joint feature representation. To enhance complementarity during the fusion process, we designed a U-shaped bilinear fusion structure to fully utilize multimodal information, capture cross-modal associations, and extract high-level features. Furthermore, to address the potential loss of key features, a temporal aggregation and pooling (TAP) layer is introduced to preserve more fine-grained features by extracting both the maximum and average values within pooling regions. To validate the effectiveness of the proposed model, both ablation experiments and comparative experiments are conducted. The results demonstrate that Mul-SEResNet50 achieves a 5.04 % accuracy improvement over SEResNet50 without sacrificing detection speed. Compared to the state-of-the-art U-FusionNet-ResNet50 +SENet model, Mul-SEResNet50 achieves accuracy and F1 score improvements of 0.47 % and 1.32 %, respectively. These findings confirm the efficacy of the proposed model in terms of accurately recognizing fish behavior, facilitating the precise monitoring of fish behavior.

Multimodal Spatio-Temporal Framework for Real-World Affect Recognition

Deep learning models show great potential in applications involving video-based affect recognition, including human-computer interaction, robotic interfaces, stress and depression assessment, and Alzheimer's disease detection. The low complex Multimodal Diverse Spatio-Temporal Network (MDSTN) has been analysed to effectively capture spatio-temporal information from audio-visual modalities for affect recognition using the Acted Facial Expressions in the Wild (AFEW) dataset. The scarcity of data is handled by data augmented parallel feature extraction for visual network. Visual features extracted by carefully reviewing and customizing Convolutional 3D architecture over different ranges are combined to train a neural network for classification. Multi-resolution Cochleagram (MRCG) features from speech, along with spectral and prosodic audio features, are processed by a supervised classifier. The late fusion technique is explored to integrate audio and video modalities, considering their processing over different temporal spans. The MDSTN approach significantly boosts the accuracy of basic emotion recognition to 71.54% on the AFEW dataset. It demonstrates exceptional proficiency in identifying emotions such as disgust and surprise, thus exceeding current benchmarks in real-world affect recognition.

Relationships Between Hearing Status, Cognitive Abilities, and Reliance on Visual and Contextual Cues

Objectives: Visual and contextual cues facilitate speech recognition in suboptimal listening conditions (e.g., background noise, hearing loss, hearing aid signal processing). Moreover, successful speech recognition in challenging listening conditions is linked to cognitive abilities such as working memory and fluid intelligence. However, it is unclear which cognitive abilities facilitate the use of visual and contextual cues in individuals with normal hearing and hearing aid users. The first aim was to investigate whether individuals with hearing aid users rely on visual and contextual cues to a higher degree than individuals with normal hearing in a speech-in-noise recognition task. The second aim was to investigate whether working memory and fluid intelligence are associated with the use of visual and contextual cues in these groups. Design: Groups of participants with normal hearing and hearing aid users with bilateral, symmetrical mild to severe sensorineural hearing loss were included (n = 169 per group). The Samuelsson and Rönnberg task was administered to measure speech recognition in speech-shaped noise. The task consists of an equal number of sentences administered in the auditory and audiovisual modalities, as well as without and with contextual cues (visually presented word preceding the sentence, e.g.,: “Restaurant”). The signal to noise ratio was individually set to 1 dB below the level obtained for 50% correct speech recognition in the hearing-in-noise test administered in the auditory modality. The Reading Span test was used to measure working memory capacity and the Raven test was used to measure fluid intelligence. The data were analyzed using linear mixed-effects modeling. Results: Both groups exhibited significantly higher speech recognition performance when visual and contextual cues were available. Although the hearing aid users performed significantly worse compared to those with normal hearing in the auditory modality, both groups reached similar performance levels in the audiovisual modality. In addition, a significant positive relationship was found between the Raven test score and speech recognition performance only for the hearing aid users in the audiovisual modality. There was no significant relationship between Reading Span test score and performance. Conclusions: Both participants with normal hearing and hearing aid users benefitted from contextual cues, regardless of cognitive abilities. The hearing aid users relied on visual cues to compensate for the perceptual difficulties, reaching a similar performance level as the participants with normal hearing when visual cues were available, despite worse performance in the auditory modality. It is important to note that the hearing aid users who had higher fluid intelligence were able to capitalize on visual cues more successfully than those with poorer fluid intelligence, resulting in better speech-in-noise recognition performance.

Multimodality-guided Visual-Caption Semantic Enhancement

Video captions generated with single modality, e.g. video clips, often suffer from insufficient event discovery and inadequate scene description. Therefore, this paper aims to improve the quality of captions by addressing these issues through the integration of multi-modal information. Specifically, We first construct a multi-modal dataset and introduce the triplet annotations of video, audio and text, fostering a comprehensive exploration about the associations between different modalities. Build upon this, We propose to explore the collaborative perception of audio and visual concepts to mitigate inaccuracies and incompleteness in captions in vision-based benchmarks by incorporating audio-visual perception priors. To achieve this, we extract effective semantic features from visual and auditory modalities, bridge the semantic gap between audio-visual modalities and text, and form a more precise knowledge graph multimodal coherence checking and information pruning mechanism. Exhaustive experiments demonstrate that the proposed approach surpasses existing methods and generalizes well with the assistance of ChatGPT.

Informing Immersed Citizens: The Impact of Interactivity on Comprehending News in Immersive Journalism

Immersive journalism has emerged as an innovative journalistic approach promising enhanced attention and understanding through interactive, virtual environments. Previously, this was mostly tested through factual knowledge. However, evaluating immersive audio-visual modalities solely along factual knowledge falls short of considering, firstly, what about and how an immersive experience is remembered and, secondly, the importance of considering the perceptions surrounding information acquisition. Therefore, this study examines how interactivity in immersive journalism affects traditional as well as novel ways of comprehending news, such as episodic memory. In addition, we consider perceptions related to knowledge. We draw on a laboratory experiment (<em>N</em> = 149) testing the effect of three levels of interactivity provided (no interactivity vs. looking around vs. looking around + control over pace and order of storyline). Results indicate that a higher range of interactivity does not influence factual memory, but leads to an increase in perceived knowledge, thus indicating an illusion of knowledge. Moreover, there is a tendency to formulate more subjective takeaways in the high-interactivity condition, while interactivity did not influence the credibility evaluations. This provides partial empirical support for the credibility paradox of immersive journalism.

Importance of Visual Support Through Lipreading in the Identification of Words in Spanish Language.

We sought to examine the contribution of visual cues, such as lipreading, in the identification of familiar (words) and unfamiliar (phonemes) words in terms of percent accuracy. For that purpose, in this retrospective study, we presented lists of words and phonemes (adult female healthy voice) in auditory (A) and audiovisual (AV) modalities to 65 Spanish normal-hearing male and female listeners classified in four age groups. Our results showed a remarkable benefit of AV information in word and phoneme recognition. Regarding gender, women exhibited better performance than men in both A and AV modalities, although we only found significant differences for words but not for phonemes. Concerning age, significant differences were detected in word recognition in the A modality between the youngest (18-29 years old) and oldest (⩾50 years old) groups only. We conclude visual information enhances word and phoneme recognition and women are more influenced by visual signals than men in AV speech perception. On the contrary, it seems that, overall, age is not a limiting factor for word recognition, with no significant differences observed in the AV modality.

RainMind: Investigating Dynamic Natural Soundscape of Physiological Data to Promote Self-Reflection for Stress Management

Metaphorical auditory displays are increasingly recognized for data presentation and self-reflection. This article presents the design and evaluation of RainMind, a web-based soundscape application that represents physiological data with dynamic natural sounds for daily stress reflection. Based on different combinations of auditory display with visualization, we identified three modes: the Visual-Aided Mode (VAM), the Audio-Aided Mode (AAM), and the Audio-Visual Mode (AVM). Through a within-subject study involving 30 participants, we conducted a mixed-methods evaluation to assess the task load, engagement, and user experience among the three modes. The findings indicated that the combination of dynamic natural soundscapes with visualization (AVM) contributes to a lower task load compared to the other two modes. Moreover, dynamic natural soundscapes as metaphors for stress data significantly enhanced engagement and user experience of self-reflection compared to static natural sounds. Based on our study, we discuss the potential of leveraging dynamic natural soundscapes as a new way of data-driven self-reflection.

Discourse Markers Audiovisual Rendition into Modern Standard Arabic and Colloquial Egyptian Arabic

Discourse markers are omnipresent in language and are gaining more attention, especially, in audiovisual translation studies. This study aimed at examining the strategies used by translators in rendering six commonly used English discourse markers into Arabic. Moreover, it shed light on the discourse markers found in seven children’s animated movies and seeks to investigate the differences between three renditions of these movies: Arabic subtitles, Modern Standard Arabic dubbed version and colloquial Egyptian Arabic dubbed version. It also focused on the effect that the two audiovisual modes, subtitling and dubbing, have on the translation of these devices and the choices of translation strategies. The data consisted of 944 instances of the six discourse markers extracted from the movies’ original scripts and their equivalents taken from the Arabic subtitled and dubbed versions of the movies. The findings reveal that the avoidance of SL discourse marker strategy was used by translators in both modes of audiovisual translation. Moreover, the equivalents found in the colloquial Egyptian Arabic dubbed version were more diverse compared to the other two translations (the subtitled version and MSA dubbed version).

The Effect of Visual Articulatory Cues on the Identification of Mandarin Tones by Children With Cochlear Implants.

This study explored the facilitatory effect of visual articulatory cues on the identification of Mandarin lexical tones by children with cochlear implants (CIs) in both quiet and noisy environments. It also explored whether early implantation is associated with better use of visual cues in tonal identification. Participants included 106 children with CIs and 100 normal-hearing (NH) controls. A tonal identification task was employed using a two-alternative forced-choice picture-pointing paradigm. Participants' tonal identification accuracies were compared between audio-only (AO) and audiovisual (AV) modalities. Correlations between implantation ages and visual benefits (accuracy differences between AO and AV modalities) were also examined. Children with CIs demonstrated an improved identification accuracy from AO to AV modalities in the noisy environment. Additionally, earlier implantation was significantly correlated with a greater visual benefit in noise. These findings indicated that children with CIs benefited from visual cues on tonal identification in noise, and early implantation enhanced the visual benefit. These results thus have practical implications on tonal perception interventions for Mandarin-speaking children with CIs.

A cross-disciplinary evaluation of digitally recorded feedback in higher education

Research demonstrates that assessment feedback created using audio, video, and screencast recordings can offer advantages over text-based feedback. However, the majority of research and experience in this domain has largely been limited to a single disciplinary or cohort context. This project aimed to empirically investigate if recorded feedback (i.e. audio, video, and screencast) could be effectively implemented across different contexts, including disciplines. As part of this, teaching staff from five discrete subjects provided digitally recorded feedback to students on at least one assessment task. Assessment types included various forms of written assignments, completed by individuals or groups of students. This paper reports on survey data collected from 351 students who received recorded feedback or text-based feedback. Survey respondents were enrolled in five subjects across four disciplines (Education, Pharmacy, Engineering, and Management). To triangulate the survey findings, interview data from nine students are also included. Overall, the findings indicate that students in all disciplines found digitally recorded feedback to be more satisfying, more useful, and more engaging than text-based feedback alone. However, these outcomes differed across contexts; results tended to be elevated in subjects with smaller cohorts, and when richer audiovisual modalities were used. In two of the cases students, while still being overall positive, indicated that the feedback was less clear, usable and satisfying than indicated in other cases. These differences are explored and issues of teacher experience, cohort size, group assessment, and disciplinary cognate traditions are considered.

Options for subtitling English movie lyrics into Arabic

Subtitling lyrics poses a problem regarding conveying the content and the form, which include meaning, rhyme, rhythm, and other musical features. This requires having well-trained professional translators to master the rendering of those features in the target language, bearing in mind the constraints imposed by subtitling as an audio-visual mode. This study examines the Arabic subtitled version of English movie lyrics and explores the translator’s options in rendering their content and form. To this end, sixteen lyrics were extracted from five movies that have been subtitled. The compiled lyrics were classified into different categories based on the translation options proposed by Franzon (2008). The option of “translating lyrics without taking the music into consideration” was used 11 times, while the option of “aesthetic translation” was opted for four times. “Not translating the lyric” was used only once, while the option of “creating new lyrics for the music” was not adopted in any of the five movies. The study recommends two research areas: exploring dubbed Arabic versions of English movie lyrics and analyzing subtitled lyrics in other language pairs.

HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition

Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-aware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. Firstly, it incorporates hierarchical skip connections between the encoder and decoder to encourage intermediate layers to learn more meaningful representations and bolster masked audio-visual reconstruction. Secondly, hierarchical cross-modal contrastive learning is also exerted on intermediate representations to narrow the audio-visual modality gap progressively and facilitate subsequent cross-modal fusion. Finally, during downstream fine-tuning, HiCMAE employs hierarchical feature fusion to comprehensively integrate multi-level features from different layers. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models are publicly available at https://github.com/sunlicai/HiCMAE.

Effectiveness of understanding the post-operative instructions with respect to extraction of teeth in rural population through verbal, written and audio-visual aids.

Aim: To compare the efficacy between verbal, written & audio-visual modes of communication in relation to post-operative instructions after extraction of teeth. Materials & Methods: Ninety patients were equally divided into three groups. Group 1 were given verbal instructions, Group 2 in written format and Group 3- Audio visual mode. Their responses were recorded using a standard questionnaire after 24 hours of extraction and subjected to statistical analysis. Results: Significant differences (p<0.05) was found when the responses were compared between Group 1 and Group 3 and Group 2 and Group 3. Conclusion: Patients who received post-operative instructions through audio-visual mode (Group 3) had more understanding when compared to other groups. Incorporation of technology in communication has to be encouraged as it helps for better understanding. Bangladesh Journal of Medical Science Vol.23 (Special Issue) 2024 p.S48-S52

Student evaluation of an instructional video on how to use a national medicines information resource.

The objective of this project was to obtain students' perception of an educational companion video designed to increase the understanding of a national medicines information resource. An instructional video was developed to guide students in utilizing the online version of the Australian Medicines Handbook (AMH). All students enrolled in the Bachelor of Pharmacy program during 2021 were given access to this video. A cross-sectional electronic survey was administered to evaluate the video's effectiveness and its impact on students' confidence. The survey results were analysed using both descriptive and inferential statistics, in addition to qualitative analysis to identify common themes. Ethics approval was obtained prior to conducting the study. Most students (78%; n = 72/92) reported that watching the video increased their understanding. The most growth in confidence was seen by students in their first or second year of study. Fifty-four percent (n = 48/89) of students was very likely or extremely likely to recommend the video to others, and 37% (n = 33/89) of students were somewhat likely. Students found the instructional video to be useful and expressed a desire for similar content to be integrated into other facets of teaching. The audio-visual mode of delivery was regarded as effective for this context. Constructive feedback included suggestions of incorporating more advanced educational features such as how to interpret comparative medication charts and interaction checkers. The 'How to use the AMH' video is a good introductory resource for undergraduate Australian healthcare students. Our results indicate that this video would be best suited to complement the teaching of students early on in their studies.

Non-spatial inhibition of return attenuates audiovisual integration owing to modality disparities.

Although previous studies have investigated the relationship between inhibition of return (IOR) and multisensory integration, the influence of non-spatial has not been explored. The present study aimed to investigate the influence of non-spatial IOR on audiovisual integration by using a "prime-neutral cue-target" paradigm. In Experiment 1, which manipulated prime validity and target modality, the targets were positioned centrally, revealing significant non-spatial IOR effects in the visual, auditory, and audiovisual modalities. Analysis of relative multisensory response enhancement (rMRE) indicated substantial audiovisual integration enhancement in both valid and invalid target conditions. Furthermore, the enhancement was weaker for valid targets than for invalid targets. In Experiment 2, the targets were positioned above and below to rule out repetition blindness (RB); this experiment successfully replicated the results observed in Experiment 1. Notably, Experiments 1 and 2 consistently found that the correlation between modality differences and rMRE for valid targets indicated that differences in signal strength between visual and auditory modalities contributed to a reduction in audiovisual integration. However, the absence of correlation with the invalid target suggests that attention, as a key factor, may play a significant role in this process. The present study highlights how non-spatial IOR reduces audiovisual integration and sheds light on the complex interaction between attention and multisensory integration.

The Visual Advantage Effect in Comparing Uni-Modal and Cross-Modal Probabilistic Category Learning.

People rely on multiple learning systems to complete weather prediction (WP) tasks with visual cues. However, how people perform in audio and audiovisual modalities remains elusive. The present research investigated how the cue modality influences performance in probabilistic category learning and conscious awareness about the category knowledge acquired. A modified weather prediction task was adopted, in which the cues included two dimensions from visual, auditory, or audiovisual modalities. The results of all three experiments revealed better performances in the visual modality relative to the audio and audiovisual modalities. Moreover, participants primarily acquired unconscious knowledge in the audio and audiovisual modalities, while conscious knowledge was acquired in the visual modality. Interestingly, factors such as the amount of training, the complexity of visual stimuli, and the number of objects to which the two cues belonged influenced the amount of conscious knowledge acquired but did not change the visual advantage effect. These findings suggest that individuals can learn probabilistic cues and category associations across different modalities, but a robust visual advantage persists. Specifically, visual associations can be learned more effectively, and are more likely to become conscious. The possible causes and implications of these effects are discussed.

Audiovisual integration in children with cochlear implants revealed through EEG and fNIRS

Sensory deprivation can offset the balance of audio versus visual information in multimodal processing. Such a phenomenon could persist for children born deaf, even after they receive cochlear implants (CIs), and could potentially explain why one modality is given priority over the other. Here, we recorded cortical responses to a single speaker uttering two syllables, presented in audio-only (A), visual-only (V), and audio-visual (AV) modes. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were successively recorded in seventy-five school-aged children. Twenty-five were children with normal hearing (NH) and fifty wore CIs, among whom 26 had relatively high language abilities (HL) comparable to those of NH children, while 24 others had low language abilities (LL). In EEG data, visual-evoked potentials were captured in occipital regions, in response to V and AV stimuli, and they were accentuated in the HL group compared to the LL group (the NH group being intermediate). Close to the vertex, auditory-evoked potentials were captured in response to A and AV stimuli and reflected a differential treatment of the two syllables but only in the NH group. None of the EEG metrics revealed any interaction between group and modality. In fNIRS data, each modality induced a corresponding activity in visual or auditory regions, but no group difference was observed in A, V, or AV stimulation. The present study did not reveal any sign of abnormal AV integration in children with CI. An efficient multimodal integrative network (at least for rudimentary speech materials) is clearly not a sufficient condition to exhibit good language and literacy.

Revisiting raters’ accent familiarity in speaking tests: Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings

This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings (χ² = 88.005, df = 3, p < .0001), with presentation mode having a stronger effect in the interaction than accent familiarity (χ² = 59.328, df = 1, p < .0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio-only or audio-visual modes of presentation should be evaluated through research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-COVID-19.