Audiovisual Segments Research Articles

Thinking diegetically

In the process of thinking diegetically, the videographic practitioner is guided by the diegetic (story world) logic of the films or media works under scrutiny. As opposed to videographic approaches that extract audiovisual segments from a narrative and spatiotemporal logic, this form of videographic work engages with the constraints of the source materials’ diegetic tethers to (re)construct a story world in meaningful and productive ways. This essay seeks to explore the ramifications of such diegetic argumentation through an analysis of several videographic works: the author’s “Imagining Orphée | Orphée imaginé” (Oyallon-Koloski 2023), Catherine Grant’s “Fated to be Mated: An Architectural Promenade” (2018), Dayna McLeod’s “Speculative Queer Autoethnography: Desert Hearts” (2023), and Liz Greene’s “Spencer Bell, Nobody Knows My Name” (2022). These examples embrace the intrinsic form of their source material’s diegesis, prioritize the rhetorical impact of spatiotemporal construction, and deliberately balance the pulls between original and videographic diegetic logic through the application of precise videographic techniques. Video essayists use the formal, performative, and nonverbal options afforded by centering diegetic principles in powerful ways to shape their rhetoric.

Audio–Visual Segmentation based on robust principal component analysis

Audio–Visual Segmentation (AVS) aims to extract the sounding objects from a video. The current learning-based AVS methods are often supervised, which rely on specific task data annotations and expensive model training. Recognizing that the video background captured by a static camera is represented as a low-rank matrix, we introduce the non-convex robust principal component analysis into AVS task in this paper. This approach is unsupervised and only relies on input data patterns. Specifically, the proposed method decomposes each modality into the sum of two parts: the low-rank part that represents the background audio and visual information, and the sparse part that represents the foreground information. Furthermore, CUR decomposition is employed at each iteration to reduce the computational complexity in optimization. The experimental results also show that the proposed AVS outperforms the supervised methods on AVS-Bench Single-Source datasets.

Enhance audio-visual segmentation with hierarchical encoder and audio guidance

As one of the pivotal technologies leading towards embodied intelligence, audio-visual segmentation is geared towards achieving precise segmentation of sounding objects, offering vast application prospects in scenarios such as emergency rescue and natural exploration. Nevertheless, the performance of audio-visual segmentation technology encounters limitations stemming from challenges related to the adaptation and fusion of cross-modal information encoding, as well as the decoding and generation of masks. To address these issues, this paper explores the adaptation of multi-modal information based on a shared encoder by employing a neural architecture search method to design a hierarchical encoder cooperation module for enhanced information interaction. An intermediate loss is leveraged to help the encoder to keep spatial knowledge reserved. Furthermore, an audio-guided class-aware decoder is devised to guide the generation of masks. Our approach has yielded competitive experimental results across multiple datasets, thus substantiating its effectiveness.

AVSegFormer: Audio-Visual Segmentation with Transformer

Audio-visual segmentation (AVS) aims to locate and segment the sounding objects in a given video, which demands audio-driven pixel-level scene understanding. The existing methods cannot fully process the fine-grained correlations between audio and visual cues across various situations dynamically. They also face challenges in adapting to complex scenarios, such as evolving audio, the coexistence of multiple objects, and more. In this paper, we propose AVSegFormer, a novel framework for AVS that leverages the transformer architecture. Specifically, It comprises a dense audio-visual mixer, which can dynamically adjust interested visual features, and a sparse audio-visual decoder, which implicitly separates audio sources and automatically matches optimal visual features. Combining both components provides a more robust bidirectional conditional multi-modal representation, improving the segmentation performance in different scenarios. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is available at https://github.com/vvvb-github/AVSegFormer.

Improving Audio-Visual Segmentation with Bidirectional Generation

The aim of audio-visual segmentation (AVS) is to precisely differentiate audible objects within videos down to the pixel level. Traditional approaches often tackle this challenge by combining information from various modalities, where the contribution of each modality is implicitly or explicitly modeled. Nevertheless, the interconnections between different modalities tend to be overlooked in audio-visual modeling. In this paper, inspired by the human ability to mentally simulate the sound of an object and its visual appearance, we introduce a bidirectional generation framework. This framework establishes robust correlations between an object's visual characteristics and its associated sound, thereby enhancing the performance of AVS. To achieve this, we employ a visual-to-audio projection component that reconstructs audio features from object segmentation masks and minimizes reconstruction errors. Moreover, recognizing that many sounds are linked to object movements, we introduce an implicit volumetric motion estimation module to handle temporal dynamics that may be challenging to capture using conventional optical flow methods. To showcase the effectiveness of our approach, we conduct comprehensive experiments and analyses on the widely recognized AVSBench benchmark. As a result, we establish a new state-of-the-art performance level in the AVS benchmark, particularly excelling in the challenging MS3 subset which involves segmenting multiple sound sources. Code is released in: https://github.com/OpenNLPLab/AVS-bidirectional.

Each Performs Its Functions: Task Decomposition and Feature Assignment for Audio-Visual Segmentation

Each Performs Its Functions: Task Decomposition and Feature Assignment for Audio-Visual Segmentation

Desarrollo y aplicación de contenidos educativos digitales desde un teléfono inteligente para un tema de Estadística en un curso universitario

El propósito del artículo es describir el desarrollo de contenidos educativos digitales para un tema estadístico y su consulta desde un teléfono inteligente. La aplicación de los contenidos se realizó en el contexto de un curso formal universitario y permitió recuperar información del porcentaje de posesión de teléfonos inteligentes del grupo participante. De la misma forma, se obtuvieron datos relacionados con los aspectos tecnológicos y de diseño, del contexto de la consulta, del agrado y desagrado de los propios desarrollos. Se expusieron tres temas estadísticos basados en el diseño instruccional ADDIE (Análisis, Diseño, Desarrollo, Implementación y Evaluación). La selección de la muestra intencional consideró 40 estudiantes universitarios, a quienes se les aplicó un cuestionario con el propósito de identificar las bondades y limitaciones de este tipo de desarrollo. Los resultados mostraron que todo el estudiantado dispone de un teléfono inteligente con conexión a Internet, por lo que fue posible acceder de forma ubicua y anticipada a los contenidos. Asimismo, se señalaron aspectos favorables al diseño, como la organización, funcionalidad y utilidad de los contenidos educativos consultados. Las principales limitaciones fueron las explicaciones concretas y la corta duración de los segmentos audiovisuales. Los distractores identificados hacen mención del ruido ambiental, las notificaciones de redes sociales y mensajes de texto. Los hallazgos sugieren que el teléfono inteligente puede funcionar como una herramienta pedagógica con una adecuada orientación en su uso.

Analysis of state aid policy to stimulate innovation and internationalization in Lithuanian animation industry

Research background: In order to achieve systematic development and growth of creative industries, instead of its being a random process, effective cultural policy tools need to be implemented. This task is greatly burdened by the fact that creative industries are a heterogeneous concept. Evidence suggests that indirect state aid is one of the most effective ways to stimulate growth and development of audiovisual segment. Lithuanian indirect state aid case suggests that there is still a great deal of confusion and misconception with the implementation and exploitation of this tool.
 Purpose of the article: To analyze how effectively indirect state aid is constructed to stimulate growth and development of the Lithuanian animation industry.
 Methods: This paper uses two methods to gather data. The first one is a quantitative questionnaire design to evaluate the overall situation of Lithuanian animation industry. The second method involves qualitative semi-structured interviews to collect specific data about Lithuanian film tax incentive from representatives of government institutions that administrate local fiscal incentive scheme and representatives of audiovisual industries.
 Findings & Value added: Results show that although Lithuanian film tax incentive can be considered as a successful cultural policy tool that brought a noticeable inward investment to the local film industry and economy in general, it had no positive effect on the local animation industry. The gathered evidence suggests that indirect aid tool should be constructed with more of surgical precision for each industry, rather than a fit-for-all cultural policy tool.

Watching a Singapore Drag Comedian: A Semiotic Analysis of Kumar in a YouTube Video

This visual study of Kumar, a drag comedian from Singapore, examines aspects of political semiotics on aging in Singapore. As a digital record of performing arts, the YouTube-video (Nur Fasihah 2012, http://buttonsinthebread.com/2012/06/29/happy-ever-laughter/) shows footage of Kumar poking fun at Toh Yi Drive residents’ initial reactions to a government-proposed nursing home construction in their estate. The thunderous laughter from the floor suggests that Kumar manages successfully his comedic connection with a primarily English-speaking audience. Based on the audio-visual segment, this discussion identifies the discursive variables underpinning the incisive twists in Kumar’s onstage comical repertoire.

Predicting episodic memory formation for movie events.

Episodic memories are long lasting and full of detail, yet imperfect and malleable. We quantitatively evaluated recollection of short audiovisual segments from movies as a proxy to real-life memory formation in 161 subjects at 15 minutes up to a year after encoding. Memories were reproducible within and across individuals, showed the typical decay with time elapsed between encoding and testing, were fallible yet accurate, and were insensitive to low-level stimulus manipulations but sensitive to high-level stimulus properties. Remarkably, memorability was also high for single movie frames, even one year post-encoding. To evaluate what determines the efficacy of long-term memory formation, we developed an extensive set of content annotations that included actions, emotional valence, visual cues and auditory cues. These annotations enabled us to document the content properties that showed a stronger correlation with recognition memory and to build a machine-learning computational model that accounted for episodic memory formation in single events for group averages and individual subjects with an accuracy of up to 80%. These results provide initial steps towards the development of a quantitative computational theory capable of explaining the subjective filtering steps that lead to how humans learn and consolidate memories.

MyStoryPlayer: experiencing multiple audiovisual content for education and training

There are several education and training cases where multi-camera view is a traditional way to work: performing arts and news, medical surgical actions, sport actions, instruments playing, speech training, etc. In most cases, users need to interact with multi camera and multi audiovisual to create among audiovisual segments their own relations and annotations with the purpose of: comparing actions, gesture and posture; explaining actions; providing alternatives, etc. Most of the present solutions are based on custom players and/or specific applications which force to create custom streams from server side, thus leading to restrictions on the user activity as to establishing dynamically additional relations. Web based solutions would be more appreciated and are complex to be realized for the problems related to the video desynchronization. In this paper, MyStoryPlayer/ECLAP solution is presented. The major contributions to the state of the art are related to: (i) the semantic model to formalize the relationships and play among audiovisual determining synchronizations, (ii) the model and modality to save and share user experiences in navigating among lessons including several related and connected audiovisual, (iii) the design and development of algorithm to shorten the production of relationships among media, (iv) the design and development of the whole system including its user interaction model, and (v) the solution and algorithm to keep the desynchronizations limited among media in the event of low network bandwidth. The proposed solution has been developed for and it is in use within ECLAP (European Collected Library of Performing Arts) for accessing and commenting performing arts training content. The paper also reports validation results about performance assessment and tuning, and about the usage of tools on ECLAP services. In ECLAP, users may navigate in the audiovisual relationships, thus creating and sharing experience paths. The resulting solution includes a uniform semantic model, a corresponding semantic database for the knowledge, a distribution server for semantic knowledge and media, and the MyStoryPlayer Client for web applications.

Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.

De palavras e imagens: estigmas sociais em discursos audiovisuais

O tema deste artigo localiza-se na articulação de dois campos aparentemente distantes: os discursos midiáticos, em suas formas audiovisuais, e o estudo da questão dos estigmas sociais. A tematização desses estigmas é um processo específico presente nas mídias, tomado aqui como objeto de estudo. Entre os objetivos do texto está apresentar como se opera, nas mídias, a construção de estigmas sociais e suas configurações predominantes, a partir de leitura e análise de discursos audiovisuais. Na primeira parte do artigo, discutimos teoricamente a questão dos estigmas sociais e algumas de suas relações com as mídias; na segunda parte, observamos as mídias para refletir sobre os estigmas sociais a partir de um conjunto de filmes, destacando a minissérie televisiva Antônia e o longa-metragem homônimo exibido nos cinemas. Ao longo do texto, a referência a jornais e revistas, o acompanhamento de telejornais e programas televisivos, bem como de produções cinematográficas, contribuirão para a demonstração das questões tratadas.

Neural correlates of bimodal speech and gesture comprehension

The present study examined the neural correlates of speech and hand gesture comprehension in a naturalistic context. Fifteen participants watched audiovisual segments of speech and gesture while event-related potentials (ERPs) were recorded to the speech. Gesture influenced the ERPs to the speech. Specifically, there was a right-lateralized N400 effect—reflecting semantic integration—when gestures mismatched versus matched the speech. In addition, early sensory components in bilateral occipital and frontal sites differentiated speech accompanied by matching versus non-matching gestures. These results suggest that hand gestures may be integrated with speech at early and late stages of language processing.