Audiovisual Analysis Research Articles

La metáfora del discurso cinematográfico

A través de su historia, las teorías y metodologías del análisis del audiovisual han recurrido a explicaciones metafóricas tomadas de disciplinas como la lingüística, la semiología y la narratología literaria, haciendo extensivo el uso de las nociones de éstas. El análisis del discurso audiovisual o cinematográfico se ha convertido en un tópico habitual. Con la finalidad de explorar los límites disciplinarios, metodológicos y conceptuales que permiten este uso, presento un recorrido que explicita los puntos de contacto de teorías específicas del discurso audiovisual y las metodologías de análisis que las sustentan. Finalmente, me remito a los aportes de Charles Sanders Peirce y Umberto Eco para pensar el audiovisual como un discurso retórico basado en la metáfora.

Reflexões sobre as diásporas queer a partir do documentário “Les travestis pleurent aussi”, de Sebastiano d’Ayala Valva

Mia and Romina are transvestites, prostitutes and unregulated immigrants living in Paris. Their lives at the margins of society are the focus of the documentary panorama of Sebastiano d’Ayala Valva in “Les travestis pleurent aussi”. From this media production, the objective is to problematize what can be called the “queer diasporas”. With theoretical contributions from the field of communication, Cultural Studies and Queer Theory, we seek to reflect on the culture, representations, levels of precariousness and agency included in the transnational experiences of Mia and Romina. Articulating as a methodological procedure a theorizing about the above mentioned points with the audiovisual analysis of this documentary, the analytical approach is focused on the sociabilities of these immigrants and their relations with the communication processes in the context of transnational migrations.Keywords: Queer diasporas. LGBT immigration. Media. Documentary.

Communication through simulation: Developing a curriculum to teach interpersonal skills

BackgroundNontechnical issues are linked to up to 60% of adverse events in surgery and have become one of the core competencies of surgical education. Simulation-based training allows development of these skills. We hypothesized that we could both assess and improve the communication skills of interdisciplinary teams through a simulated crisis situation. MethodsWe designed a simulated crisis involving the intraoperative development of a tension pneumothorax. Audiovisual analysis was used to evaluate team interaction, and a combination of proctored assessment and self-assessment was used to evaluate individual learners. Time to communication and event intervention was measured. Data were analyzed using paired t-tests and 1-way analysis of variance in SPSS. ResultsTime from development of hypotension to communication of deterioration was 57 ± 60 seconds; time to intervention was 93 ± 30 seconds. Nontechnical skills in surgery II scores were greatest for surgery senior residents and least for surgery junior residents. Insight was least in surgery junior residents. Correctability was not different between groups but tended to be greater in surgery junior residents. ConclusionsPoor communication delayed timing of intervention, indicating a need for enhanced communication in the operating room. Junior surgery residents scored lowest both on nontechnical skills and insight and should be taught nontechnical skills through simulation. Our curriculum is a suitable and reproducible model to educate surgery residents in communication skills.

РОЛЬ ВНУТРІШНЬОКАДРОВОЇ МУЗИКИ В КІНОФІЛЬМІ

The purpose of the article is to analyze the role of source music in a motion picture in the process of organizing its sound design; to reveal the features of its application in films of different genres. The research methodology consisted in the application of audiovisual analysis of foreign and Ukrainian movies, which allowed for understanding the role of source music in a motion picture. The scientific novelty of the work lies in the fact that the basic principles of using source music in a film were clearly formed and its impact on the sound design of the movie was shown. Conclusions. Source music is a factor that is closely related to the plot of the film and is its driving force. Most often, source music has been represented by a song that can let the singer express their feelings, emphasize heroism and courage, and build up the action of the movie. Dance is also frequently used as source music in various motion pictures, primarily those related to everyday life and national characteristics. Dance music combines very different episodes, reflects and characterizes changes in the time frame, acts as an example of the coexistence and mutual influence of different musical cultures. Instrumental source music can also be in the form of a large concerted item and lead a pictorial installation of the stage. Besides, it emphasizes tragic moments of the film and is dramatically important as background music; it provides the scene with an ironic subtext, plays the role of an espionage cipher or password; it may act as the background opposite to the content of the frame, and create an explicit contrast with this content.

Multimodal communication in academic oral presentations by L2 Spanish students

ABSTRACTPresenting in a second language (L2) is perhaps the greatest challenge for international students in a foreign academic context. In this study, we examine the multimodal (verbal and non-verbal) oral performances of two L2 and two L1 groups of Spanish students enrolled in a university Applied Linguistics course. To determine the specific demands and challenges that each group faces, we conducted a questionnaire with 16 participants and carried out an audiovisual discourse analysis of the presentations based on a multimodal framework applied to the use of academic Spanish. It was discovered in the analysis of the data that multimodal competence, the ability to make use of and combine verbal and non-verbal modes of communication to construct and communicate meaning, can compensate for L2 students’ linguistic deficiencies. The article concludes with guidelines to support the development of multimodal competence of L2 students.

РОЛЬ ПІСНІ В УКРАЇНСЬКОМУ ФІЛЬМІ

Мета роботи. Проаналізувати роль пісні в українському фільмі при організації звукового рішення фільму, виявити особливості застосування пісні в фільмах різних жанрів. Методологія дослідження передбачає застосування аудіовізуального аналізу українських фільмів, що дозволить прийти до розуміння основних значення пісні в українському фільмі. Наукова новизна дослідження полягає в тому, що вперше сформовані основні принципи застосування пісні в фільмі та показано її вплив на звукове вирішення кінофільму. Висновки. Пісня може бути лейтмотивом усього фільму, звукозоровим контрапунктом, коментарем до фільму. Об’єднати воєдино і рухати вперед дію та підсилювати драматизм ситуації, чітко охарактеризувати час дії в фільмі. У кінофільмі пісня звернена до зображуваних на екрані «слухачам», вона впливає на їх почуття, переживання, поведінку, тобто бере участь у розвитку сюжету. Пісня у фільмі, насамперед, сольна, представляє не тільки себе, але одночасно й певний тип виконання. Іноді пісня функціонує на прикладі симфонічної коди в фіналі фільму.

ДОСЛІДЖЕННЯ ФУНКЦІЇ ЦИТАТИ В КІНОМУЗИЦІ

Мета роботи. Здійснити аналіз функцій цитати в кіномузиці в системі організації звукового рішення фільму, виявити особливості її застосування в різних фільмах та визначити способи її реалізації. Методологія дослідження передбачає застосування аудіовізуального аналізу кінофільмів, що дозволить перейти до розуміння основних функції цитати в кіномузиці. Наукова новизна дослідження в тому, що вперше сформовані основні принципи застосування музичних цитат в кінофільмі та показано їх вплив на звукове вирішення кінофільму. Висновки. Вдаючись до цитування, композитор і режисер повинні подбати про те, щоб глядач їх упізнав. Цитата привносить емоційні й смислові асоціації. В німому фільмі музична ілюстрація складена виключно з музичних цитат. Музичні цитати можуть виконувати різноманітні функції: підкреслювати трагізм сцени, використовуватись як іронічний контраст. Також цитата може бути застосована як драматичний елемент і музичний символ, може створювати пародію, характеризувати певний персонаж. Таким чином, аналіз матеріалу про цитату в фільмі дозволив автору статті сформувати систему понять про цитату в кіномузиці з обговоренням її застосування, що має суттєве значення для практики звукового вирішення кінофільму.

Video activism in the Brazilian protests: Genres, narratives and political participation

Abstract This article examines communication practices, specific genres and audio-visual narratives developed by Brazilian video activists in the context of the protests against the FIFA World Cup competition in 2014. Analysing 170 videos produced between June and July 2014, using digital methods, audio-visual analysis and the Lilleker and Vedel scheme crossed with the Carpentier model to analyse political participation, we try to reveal what level of participation these videos represent and to discover their political nature, classifying them by level of participation. In addition, this article will attempt to highlight that the videos worked as a source of information, debate and deliberation, promoting participation, which could generate more empowerment and participation if the activists improved contextualization and the quality of the narrative.

Diagnostic Value of Isolated Mentalis Versus Mentalis Plus Upper Limb Electromyography in Idiopathic REM Sleep Behavior Disorder Patients Eventually Developing a Neurodegenerative Syndrome.

To compare two electromyographic (EMG) montages, isolated mentalis muscle versus mentalis in combination with upper limb muscles in the baseline diagnostic video-polysomnography (V-PSG) of patients with idiopathic REM sleep behaviors disorder (IRBD) who eventually were diagnosed with a clinically defined neurodegenerative syndrome. Forty-nine patients were included. At baseline, diagnosis of RBD was based on a typical history of dream enactment behaviors plus V-PSG showing REM sleep with qualitative increased EMG activity and/or abnormal behaviors. Quantification of EMG activity (tonic, phasic and "any") in the mentalis and upper limb muscles (biceps brachii-BB, n = 36 or flexor digitorum superficialis-FDS, n = 13) was performed manually and compared with published cut-offs. Nine (18.4%) patients had either tonic or phasic EMG below the cut-offs for the isolated mentalis and four of them (11.1 %) also had values below the cut-off for the mentalis combined with BB. All 13 patients recorded with the FDS were above the mentalis combined with FDS cut-off. For the diagnosis of IRBD, sensitivity of isolated mentalis was 81.6% and of the combination of mentalis plus upper limb muscles was 91.8% (p = .03). Audiovisual analysis showed abnormal REM sleep behaviors in all nine patients with values below the cut-offs. Quantification of EMG activity in the upper limbs combined with the mentalis increases the ability to diagnose IRBD when compared with the isolated measurement of the mentalis. Detection of typical abnormal behaviors during REM sleep with audiovisual analysis is essential for the diagnosis of IRBD in patients with EMG values below the published cut-offs.

Mirrored Triangulation: A Methodology for the Audiovisual Analysis of Surgical Team Performance

Mirrored Triangulation: A Methodology for the Audiovisual Analysis of Surgical Team Performance

Methodology for the Audiovisual Analysis of Surgical Team Performance in Orthopedic Ankle Surgery

Category: Basic Sciences/Biologics Introduction/Purpose: Defining and quantitatively measuring surgeon and operating room team performance remains a scientific and clinical challenge. Evidence-based, objective methods to assess surgical skill have been identified; however, a quantifiable and valid methodology to measure intraoperative performance is lacking. We aimed to determine the optimal approach to audiovisually assess a surgeon and a surgical team. We aimed to develop a high-fidelity method to analyze and improve surgeon and team performance. Methods: Funding of this study was provided through a grant awarded by the American Orthopaedic Foot and Ankle Society (#2015-24-P). Sixteen predetermined, bilateral ankle surgeries (frame placement, frame removal, and total ankle replacement) were performed between July and December 2015 in both live and simulated operating room settings. Still photography and web- based three-dimensional modeling software were used to determine optimum camera position for the first eight procedures. Three compact, high-resolution camcorders (GoPro® Hero, San Mateo, CA) were then employed in different locations with a variety of mounting strategies to record eight additional, matched ankle procedures in a simulated environment. A multiview box and wireless routers were used for live viewing of surgeries on a high-definition television screen. Video-based editing software was used to replay and analyze audiovisual output. Results: There are several visual angles that optimally capture the dynamics of a surgical team and are specific to surgery type. These angles are measurable (in degrees) and vary according to the position of the surgeon, team members, operating room table, and C-arm, as well as laterality of the procedure itself. Triangulated placement of multiple cameras, including a head mounted device, are required to assess subtle elements of operating room team performance, including individual strengths and weaknesses, nontechnical skills, and team dynamics. High-fidelity recording equipment and video-based editing software facilitate the assessment of characteristics unique to reconstructive ankle surgery. Conclusion: The effectiveness of a surgical team can be assessed with high-fidelity recording and editing equipment in defined configurations. Our approach, which to our knowledge has not been previously described, facilitates the evaluation of communication styles, technical skills, and teamwork and provides a foundation for intraoperative leadership. We propose that our audiovisual methodology can be used to coach surgeons and their teams, in order to make collaborative improvements in a non-punitive manner. Enhancing the performance of each member of the surgical team may reduce risk and ultimately improve patient outcomes, and is likely to be reproducible across disciplines.

Design of Music Learning Assistant Based on Audio Music and Music Score Recognition

Mastering a musical instrument for an unskilled beginning learner is not an easy task. It requires playing every note correctly and maintaining the tempo accurately. Any music comes in two forms, a music score and it rendition into an audio music. The proposed method of assisting beginning music players in both aspects employs two popular pattern recognition methods for audio-visual analysis; they are support vector machine (SVM) for music score recognition and hidden Markov model (HMM) for audio music performance tracking. With proper synchronization of the two results, the proposed music learning assistant system can give useful feedback to self-training beginners.

Ao Vivo em Vai Que Cola

A primeira temporada de Vai Que Cola (MULTISHOW, 2013-2015) podeter produzido efeito singular no espectador televisivo. A simulação da transmissãoao vivo, junto da quebra da quarta parede, a exposição dos bastidores, a reação daplateia e o palco giratório, pode ter proporcionado ao telespectador a sensaçãode sair do sofá, sentar na plateia do teatro, subir no palco de Vai Que Cola e aindaatravessar as cortinas da sitcom. Com base no conceito da técnica do real sobre atransmissão direta (ao vivo) de Mario Carlón e do conceito de ao vivo de ArlindoMachado, pretende-se identificar por meio da análise audiovisual a contribuiçãodo teatro para a simulação do ao vivo, além de levantar um possível efeito doproduto sobre seu público.

‘It’s a-me, Mario!’ Exploring dynamic changes and similarities in the composition of early Nintendo video game music

As with films, a thoughtfully composed video game soundtrack has the ability to dramatically enhance and elevate the experience for the audience or player. This article explores the potential issues and difficulties of composing for video game systems by studying the sound-producing hardware and music for two popular systems from one manufacturer. By comparing two of Nintendo’s Super Mario titles, which appeared on both 8-bit and 16-bit systems, through an analysis of the technology, audio, visual (audiovisual), music, and gameplay elements, it is shown that the musical composition was affected by the limitations of processing power. The discussion shows how the composer, Koji Kondo, overcame the issues of limited computing power by using layers of repetition while applying various functions of music for film to enhance player immersion. Kondo composed theme music that has become engrained in popular culture and is synonymous with one of Nintendo’s flagship franchises (Greening, 2014). By attempting to understand the method or approach behind the composition for earlier systems, it is possible to investigate and discuss the evolution of video game music while acknowledging and contributing to the study of music for games. A musical analysis of the Castle and Underwater themes on each system allows for a direct comparison of the compositional approach, while an audiovisual analysis reveals the presence of existing cinematic tropes and identifies potential influences on the creation of effective musical soundtracks for video games. Applying audiovisual theory to games will require the use of existing literature from Lissa (1965), Gorbman (1987), Chion (1994) and Tagg (2004), along with the work of Collins (2005; 2007a; 2007b; 2008a; 2008b), which adapts and applies audiovisual analysis to video games.

Audio Visual Integration with Competing Sources in the Framework of Audio Visual Speech Scene Analysis.

We introduce "Audio-Visual Speech Scene Analysis" (AVSSA) as an extension of the two-stage Auditory Scene Analysis model towards audiovisual scenes made of mixtures of speakers. AVSSA assumes that a coherence index between the auditory and the visual input is computed prior to audiovisual fusion, enabling to determine whether the sensory inputs should be bound together. Previous experiments on the modulation of the McGurk effect by audiovisual coherent vs. incoherent contexts presented before the McGurk target have provided experimental evidence supporting AVSSA. Indeed, incoherent contexts appear to decrease the McGurk effect, suggesting that they produce lower audiovisual coherence hence less audiovisual fusion. The present experiments extend the AVSSA paradigm by creating contexts made of competing audiovisual sources and measuring their effect on McGurk targets. The competing audiovisual sources have respectively a high and a low audiovisual coherence (that is, large vs. small audiovisual comodulations in time). The first experiment involves contexts made of two auditory sources and one video source associated to either the first or the second audio source. It appears that the McGurk effect is smaller after the context made of the visual source associated to the auditory source with less audiovisual coherence. In the second experiment with the same stimuli, the participants are asked to attend to either one or the other source. The data show that the modulation of fusion depends on the attentional focus. Altogether, these two experiments shed light on audiovisual binding, the AVSSA process and the role of attention.

Lost Voices of Kurdish Cinema

The belated emergence and the visibility of the cinema by Kurdish filmmakers is widely interpreted as a reason to categorize it as part of transnational cinemas characterized by a lack of standardized language and national domestic industry. This contributes to a negation of the promise of a cinema that specifically caters to a Kurdish public in terms of enunciation and reception. Considering Kurdish cinema outside transnational conditions, this paper examines enunciation in cinema of (national) subject through an audiovisual analysis of three feature-length films equipped with acoustic means in Kurdish: Voice of My Father (Dengê Bavê Min, Orhan Eskiköy and Zeynel Doğan, 2012), Song of My Mother (Klama Dayîka Min, Erol Mintaş, 2014), and My Sweet Pepper Land (Hiner Saleem, 2013). I employ Mladen Dolar’s concept of voice in understanding enunciation of subject through body and language and Michel Chion’s concept of acousmatic voice in understanding the constitutive division of the subject by means of suture. Through this analysis, lost memories, absence of the father (read as nation-state), and fetishization of mother(land) emerge as icons of the past haunting the present on behalf of recognition. In this regard, I address Kurdish cinema as a force of subjectification that transcends and transforms the experience of trauma through an impure production of meaning. Accordingly, the paper concludes that a primary characteristic of Kurdish cinema is its potential as a self-reflexive means for recognition and identification.

PyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis.

Audio information plays a rather important role in the increasing digital content that is available today, resulting in a need for methodologies that automatically analyze such content: audio event recognition for home automations and surveillance systems, speech recognition, music information retrieval, multimodal analysis (e.g. audio-visual analysis of online videos for content-based recommendation), etc. This paper presents pyAudioAnalysis, an open-source Python library that provides a wide range of audio analysis procedures including: feature extraction, classification of audio signals, supervised and unsupervised segmentation and content visualization. pyAudioAnalysis is licensed under the Apache License and is available at GitHub (https://github.com/tyiannak/pyAudioAnalysis/). Here we present the theoretical background behind the wide range of the implemented methodologies, along with evaluation metrics for some of the methods. pyAudioAnalysis has been already used in several audio analysis research applications: smart-home functionalities through audio event detection, speech emotion recognition, depression classification based on audio-visual features, music segmentation, multimodal content-based movie recommendation and health applications (e.g. monitoring eating habits). The feedback provided from all these particular audio applications has led to practical enhancement of the library.

Naturalizing Male Authority and the Power of the Producer

Hip-hop videos have for some time inspired a substantial amount of academic work investigating how ethnicity and gender is represented. Much has been written about the problematic representations of African-American women in hip-hop videos and how they are reduced to props for the male star (Pough et al., Home Girls Make Some Noise: Hip Hop Feminism Anthology, 2007; Rose, The Hip Hop Wars, 2008). Recently several scholars have also examined the representations of black masculinity, and the limited negative stereotypes African-American male artists are offered to embody in these videos (Forman and Neal, That’s the Joint, 2012; Perry, Prophets of the Hood, Politics and Poetics in Hip Hop, 2004). A currently under-studied area is the power structures present in the relationship between a male producer and a female artist. In this article I will explore how gender and ethnicity is represented in the relationship between the male producer and the female artist in folk/rock singer Nelly Furtado’s video Promiscuous. In this video the famous Miami producer Timbaland shows up alongside Nelly Furtado. After her second album flopped, Furtado retained Timbaland to produce her third album, Loose, which became her bestselling effort; the single “Promiscuous” became her first number-one US single as well. In the video for the song, Timbaland and Furtado rap and sing a flirtatious dialogue together, and in this article I will compare their professional histories with the video’s representation of Timbaland’s masculinity. Are Furtado and Timbaland presented as equal partners here? How is their potential partnership referred to through the music, lyrics, and images, respectively? As the female star of the video, how is Furtado contrasted with her producer/co-performer? Most importantly, how is the relationship between Furtado and Timbaland presented in this video? I will supply a brief historical background for the relationship between female performers and male producers to shed light on the interaction depicted in Promiscuous. Using audiovisual analysis as my approach, I will identify several strategies in Promiscuous that appear to naturalize male authority in this particular context.

Empowered or Objectified? Personal Narrative and Audiovisual Aesthetics in Beyoncé’s Partition

This article examines the feminist argument that Beyoncé presents through her personal narrative in relation to the audiovisual aesthetics of the music video Partition (2013), which arguably objectifies the female in accordance with normalized perceptions of gender and sexuality. The video is characterized by striptease aesthetics and hyperembodied display, firmly grounded within the tradition of contemporary pop videos that stylize the body both visually and sonically through gloss and excess. Situated within the field of popular musicology, I approach my study through a model of audiovisual analysis. With a primary focus on the strategies of production that fetishize the body through manipulation, I provide a close reading of this video by elucidating gender politics within a mainstream pop context. I conclude that the performative power of Beyoncé’s erotic display is contingent upon both the dissemination of her persona in terms of agency and the fetishizing aspects of audiovisual production.

Opinion Question Answering by Sentiment Clip Localization

This article considers multimedia question answering beyond factoid and how-to questions. We are interested in searching videos for answering opinion-oriented questions that are controversial and hotly debated. Examples of questions include “Should Edward Snowden be pardoned?” and “Obamacare—unconstitutional or not?”. These questions often invoke emotional response, either positively or negatively, hence are likely to be better answered by videos than texts, due to the vivid display of emotional signals visible through facial expression and speaking tone. Nevertheless, a potential answer of duration 60s may be embedded in a video of 10min, resulting in degraded user experience compared to reading the answer in text only. Furthermore, a text-based opinion question may be short and vague, while the video answers could be verbal, less structured grammatically, and noisy because of errors in speech transcription. Direct matching of words or syntactic analysis of sentence structure, such as adopted by factoid and how-to question-answering, is unlikely to find video answers. The first problem, the answer localization, is addressed by audiovisual analysis of the emotional signals in videos for locating video segments likely expressing opinions. The second problem, questions and answers matching, is tackled by a deep architecture that nonlinearly matches text words in questions and speeches in videos. Experiments are conducted on eight controversial topics based on questions crawled from Yahoo! Answers and Internet videos from YouTube.