Audiovisual Context Research Articles

English Subtitling of Culture-Loaded Words in Chinese Yu Opera: Strategies and Quality Assessment

Subtitling culture-loaded words poses significant challenges for subtitlers to maintain cultural nuances and conveying intended meaning in audio-visual context. This qualitative study aimed to investigate a professional translator’s subtitling strategies for culture-loaded words in the Chinese Yu Opera Cheng Ying Rescues the Orphan and assess the quality of the translation of these words. The current study is underpinned by Pedersen’s (2011) typology of subtitling strategies and his (2017) quality assessment model. Key findings revealed that most of strategies proposed by Pedersen were employed. The study also found that the strategy of translating formal language into informal was used by the subtitler. The quality evaluation revealed that most subtitles were of high quality, with only a few minor errors. The study contributes to the growing body of knowledge in opera translation, enriching our understanding of professional translator’s subtitling strategies and the translation quality to enhance cross-cultural communication through the medium of traditional Chinese opera.

Glassware Influences the Perception of Orange Juice in Simulated Naturalistic versus Urban Conditions.

The latest research demonstrates that people's perception of orange juice can be influenced by the shape/type of receptacle in which it happens to be served. Two studies are reported that were designed to investigate the impact, if any, that the shape/type of glass might exert over the perception of the contents, the emotions induced on tasting the juice and the consumer's intention to purchase orange juice. The same quantity of orange juice (100ml) was presented and evaluated in three different glasses: a straight-sided, a curved and a tapered glass. Questionnaires were used to assess taste (aroma, flavour intensity, sweetness, freshness and fruitiness), pleasantness and intention to buy orange juice. Study 2 assessed the impact of the same three glasses in two digitally rendered atmospheric conditions (nature vs urban). In Study 1, the perceived sweetness and pleasantness of the orange juice was significantly influenced by the shape/type of the glass in which it was presented. Study 2 reported significant interactions between condition (nature vs urban) and glass shape (tapered, straight-sided and curved). Perceived aroma, flavour intensity and pleasantness were all significantly affected by the simulated audiovisual context or atmosphere. Compared to the urban condition, perceived aroma, freshness, fruitiness and pleasantness were rated significantly higher in the nature condition. On the other hand, flavour intensity and sweetness were rated significantly higher in the urban condition than in the natural condition. These results are likely to be relevant for those interested in providing food services, or company managers offering beverages to their customers.

Is Front associated with Above and Back with Below? Association between Allocentric Representations of Spatial Dimensions.

Previous research has revealed congruency effects between different spatial dimensions such as right and up. In the audiovisual context, high-pitched sounds are associated with the spatial dimensions of up/above and front, while low-pitched sounds are associated with the spatial dimensions of down/below and back. This opens the question of whether there could also be a spatial association between above and front and/or below and back. Participants were presented with a high- or low-pitch stimulus at the time of the onset of the visual stimulus. In one block, participants responded according to the above/below location of the visual target stimulus if the target appeared in front of the reference object, and in the other block, they performed these above/below responses if the target appeared at the back of the reference. In general, reaction times revealed an advantage in processing the target location in the front-above and back-below locations. The front-above/back-below effect was more robust concerning the back-below component of the effect, and significantly larger in reaction times that were slower rather than faster than the median value of a participant. However, the pitch did not robustly influence responding to front/back or above/below locations. We propose that this effect might be based on the conceptual association between different spatial dimensions.

A Perceptual Model-Based Approach to Plausible Authoring of Vibration for the Haptic Metaverse.

Haptic virtual reality is often misunderstood as being solely a physically identical copy of real environments. Thus, a perfect recording and reproduction of vibration that is indistinguishable in an A:B comparison is often the aim. However, in most virtual reality applications the real environment is not available for direct comparison. Instead, when judging the plausibility of a presented scene, the user compares the vibration to his expectations shaped by the audiovisual context. Therefore, it should be sufficient to find any vibration that the user expects to potentially occur in the given context. Such a vibration needs to elicit a perceptual profile with a minimal distance to an expected profile in the sensory tactile perceptual space. Building onto this formalization, this article demonstrates a novel generative model-based approach to authoring vibrations. First, users quantify expectations as tactile profiles consisting of ratings of six sensory tactile attributes without the presence of vibrations. Subsequently, the model predicts vibration parameters from such profiles. This ensures the fulfillment of user expectations and thus high plausibility. Furthermore, it eliminates the necessity of recordings, infeasible for scenes with no real counterpart and opens the door to crowdsourcing the authoring process with laypersons for the haptic metaverse.

CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments

Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.

Fatal flaws? Investigating the effects of machine translation errors on audience reception in the audiovisual context

ABSTRACT This study reports on an experiment where machine translation errors in subtitling are evaluated from the perspective of nine viewers who did not know the source language and seven viewers who were studying the source language. Screen recordings, think-aloud protocols, comprehension tests, and interviews were employed to explore participants’ responses and reactions to erroneous subtitles and to investigate how specific errors impacted comprehension and immersion in the viewing experience. The analysis identifies which errors were most noticed and to what extent those errors affected viewers’ trust in the subtitles. Errors causing significant misunderstanding and distrust are initially considered ‘fatal’, as they may halt viewer immersion and prompt disengagement from the audiovisual product. However, the findings highlight a remarkable tolerance of the uncertainty that results from errors, as viewers filter out misinformation or draw on other sources of information to construe and rectify their interpretations. This tolerance is explained in terms of a general trade-off with the enjoyment of the viewing experience, which varies in accordance with the viewer’s knowledge of the source language.

Subtitling Arabic humour into English

This article examines how humour in Arabic stand-up comedies is translated into English in an audio-visual context. The study uses a case study of Arabic stand-up comedies streamed on Netflix, including Live from Beirut by Adel Karam and Comedians of the World/ Middle East. The shows which are subtitled into English involve a variety of Arab comedians speaking different dialects, including the Levant dialect (Lebanese, Jordanian, and Palestinian) and Gulf dialects, particularly the Saudi dialect. While several studies were conducted to examine the translation of English humour into Arabic, very few explore the translation of Arabic humour into English, especially in the realm of audio-visual translation. Arabic and English are two different languages reflecting different norms and cultures and, therefore, many linguistic and cultural challenges are expected to arise in the process of translation between them. The study draws on Pederson’s (2005) strategies for translating cultural references and Díaz-Pérez’s (2013) strategies for translating wordplay and puns. The study identifies two types of humour used in the Arabic stand-up comedies, namely language-restricted jokes (wordplay, puns, language variation, and taboo language) and culture-restricted jokes which require knowledge about the concept or character being referred to. Several translation strategies were used by Netflix subtitlers to render these types of jokes into English, including paraphrasing, generalizing, specification, substitution, and omission.

Neural and Behavioral Differences in Speech Perception for Children With Autism Spectrum Disorders Within an Audiovisual Context.

Reduced use of visible articulatory information on a speaker's face has been implicated as a possible contributor to language deficits in autism spectrum disorders (ASD). We employ an audiovisual (AV) phonemic restoration paradigm to measure behavioral performance (button press) and event-related potentials (ERPs) of visual speech perception in children with ASD and their neurotypical peers to assess potential neural substrates that contribute to group differences. Two sets of speech stimuli, /ba/-"/a/" ("/a/" was created from the /ba/ token by a reducing the initial consonant) and /ba/-/pa/, were presented within an auditory oddball paradigm to children aged 6-13 years with ASD (n = 17) and typical development (TD; n = 33) within two conditions. The AV condition contained a fully visible speaking face; the pixelated (PX) condition included a face, but the mouth and jaw were PX, removing all articulatory information. When articulatory features were present for the /ba/-"/a/" contrast, it was expected that the influence of the visual articulators would facilitate a phonemic restoration effect in which "/a/" would be perceived as /ba/. ERPs were recorded during the experiment while children were required to press a button for the deviant sound for both sets of speech contrasts within both conditions. Button press data revealed that TD children were more accurate in discriminating between /ba/-"/a/" and /ba/-/pa/ contrasts in the PX condition relative to the ASD group. ERPs in response to the /ba/-/pa/ contrast within both AV and PX conditions differed between children with ASD and TD children (earlier P300 responses for children with ASD). Children with ASD differ in the underlying neural mechanisms responsible for speech processing compared with TD peers within an AV context.

Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (ACF) blocks is proposed to effectively conduct audio-visual interaction. The ACF block equipped with spherical positional encoding enables the fusion in the 3D context to capture the spatial correspondence between pixels and sound sources from the equirectangular frames and ambisonic audios. Experimental results verify the effectiveness of our proposed components and demonstrate that our method achieves state-of-the-art performance on the ASOD60K dataset.

Audio–Visual Predictive Processing in the Perception of Humans and Robots

Recent work in cognitive science suggests that our expectations affect visual perception. With the rise of artificial agents in human life in the last few decades, one important question is whether our expectations about non-human agents such as humanoid robots affect how we perceive them. In the present study, we addressed this question in an audio–visual context. Participants reported whether a voice embedded in a noise belonged to a human or a robot. Prior to this judgment, they were presented with a human or a robot image that served as a cue and allowed them to form an expectation about the category of the voice that would follow. This cue was either congruent or incongruent with the category of the voice. Our results show that participants were faster and more accurate when the auditory target was preceded by a congruent cue than an incongruent cue. This was true regardless of the human-likeness of the robot. Overall, these results suggest that our expectations affect how we perceive non-human agents and shed light on future work in robot design.

Using Multimedia Learning Objects in Special Needs Classroom

The research project highlights the use and creation of multimedia learning objects to improve the learning of special needs students. The researcher investigated how the audiovisual context had progressed into a new approach to learning that sharpens the analytical reasoning and rational social skills of special needs students. This research proposed a two-group pre-test – post-test qualitative research to measure the impact of multimedia learning objects in improving the learning abilities of students with special needs in schools. The experiment aimed to see that a multimedia education paradigm may benefit two groups of students with different learning impairments. It was aimed toward students with Autism or Down Syndrome under the age of 15. Many of these students are taught in ordinary special needs classrooms without special intervention tools. The findings suggest that multimedia technology in learning can assist students with learning disabilities. The well-thought-out system benefited autistic students, demonstrating the educational potential of a multimedia system. A plan targeted to one particular disability group may be more effective than a comprehensive system that tries to fulfill the needs of many different disability groups.

The multimodal facilitation effect in human communication

During face-to-face communication, recipients need to rapidly integrate a plethora of auditory and visual signals. This integration of signals from many different bodily articulators, all offset in time, with the information in the speech stream may either tax the cognitive system, thus slowing down language processing, or may result in multimodal facilitation. Using the classical shadowing paradigm, participants shadowed speech from face-to-face, naturalistic dyadic conversations in an audiovisual context, an audiovisual context without visual speech (e.g., lips), and an audio-only context. Our results provide evidence of a multimodal facilitation effect in human communication: participants were faster in shadowing words when seeing multimodal messages compared with when hearing only audio. Also, the more visual context was present, the fewer shadowing errors were made, and the earlier in time participants shadowed predicted lexical items. We propose that the multimodal facilitation effect may contribute to the ease of fast face-to-face conversational interaction.

A three-layered typology for the subtitling of taboo

Abstract The translation of taboo words has attracted scholars’ interest in studying it in the audiovisual context over the last decades. The surge of research on this predominant form of translation in everyday life has brought to light the communicative, pragmatic, and semiotic aspects as well as the technical constraints for subtitling taboo words. Previous research has primarily taken a quantitative method, discussed issues that justify their results, and suggested possible outcomes from a potential receiver’s point of view. While contributing to existing related literature, this paper argues that there is a need for a thorough, detailed examination of translation options in subtitling taboo words. The paper presents a three-layered typology of methods, strategies, and techniques, which provides a comprehensive description of audiovisual translators’ options. Following a bottom-up/top-down approach, the proposed typography is then put to the test in a corpus-based case study comprising six movies and their professional subtitles broadcast on Portuguese televisions.

Contextualising facial expressions: The effect of temporal context and individual differences on classification.

The influence of context on facial expression classification is most often investigated using simple cues in static faces portraying basic expressions with a fixed emotional intensity. We examined (1) whether a perceptually rich, dynamic audiovisual context, presented in the form of movie clips (to achieve closer resemblance to real life), affected the subsequent classification of dynamic basic (happy) and non-basic (sarcastic) facial expressions and (2) whether people's susceptibility to contextual cues was related to their ability to classify facial expressions viewed in isolation. Participants classified facial expressions-gradually progressing from neutral to happy/sarcastic in increasing intensity-that followed movie clips. Classification was relatively more accurate and faster when the preceding context predicted the upcoming expression, compared with when the context did not. Speeded classifications suggested that predictive contexts reduced the emotional intensity required to be accurately classified. More importantly, we show for the first time that participants' accuracy in classifying expressions without an informative context correlated with the magnitude of the contextual effects experienced by them-poor classifiers of isolated expressions were more susceptible to a predictive context. Our findings support the emerging view that contextual cues and individual differences must be considered when explaining mechanisms underlying facial expression classification.

Estimating How Sounds Modulate Orientation Representation in the Primary Visual Cortex Using Shallow Neural Networks.

Audiovisual perception results from the interaction between visual and auditory processing. Hence, presenting auditory and visual inputs simultaneously usually improves the accuracy of the unimodal percepts, but can also lead to audiovisual illusions. Cross-talks between visual and auditory inputs during sensory processing were recently shown to occur as early as in the primary visual cortex (V1). In a previous study, we demonstrated that sounds improve the representation of the orientation of visual stimuli in the naïve mouse V1 by promoting the recruitment of neurons better tuned to the orientation and direction of the visual stimulus. However, we did not test if this type of modulation was still present when the auditory and visual stimuli were both behaviorally relevant. To determine the effect of sounds on active visual processing, we performed calcium imaging in V1 while mice were performing an audiovisual task. We then compared the representations of the task stimuli orientations in the unimodal visual and audiovisual context using shallow neural networks (SNNs). SNNs were chosen because of the biological plausibility of their computational structure and the possibility of identifying post hoc the biological neurons having the strongest influence on the classification decision. We first showed that SNNs can categorize the activity of V1 neurons evoked by drifting gratings of 12 different orientations. Then, we demonstrated using the connection weight approach that SNN training assigns the largest computational weight to the V1 neurons having the best orientation and direction selectivity. Finally, we showed that it is possible to use SNNs to determine how V1 neurons represent the orientations of stimuli that do not belong to the set of orientations used for SNN training. Once the SNN approach was established, we replicated the previous finding that sounds improve orientation representation in the V1 of naïve mice. Then, we showed that, in mice performing an audiovisual detection task, task tones improve the representation of the visual cues associated with the reward while deteriorating the representation of non-rewarded cues. Altogether, our results suggest that the direction of sound modulation in V1 depends on the behavioral relevance of the visual cue.

Multisensory synchrony of contextual boundaries affects temporal order memory, but not encoding or recognition

We memorize our daily life experiences, which are often multisensory in nature, by segmenting them into distinct event models, in accordance with perceived contextual or situational changes. However, very little is known about how multisensory boundaries affect segmentation, as most studies have focused on unisensory (visual or audio) segmentation. In three experiments, we investigated the effect of multisensory boundaries on segmentation in memory and perception. In Experiment 1, participants encoded lists of pictures while audio and visual contexts changed synchronously or asynchronously. After each list, we tested recognition and temporal associative memory for pictures that were encoded in the same audio-visual context or that crossed a synchronous or an asynchronous multisensory change. We found no effect of multisensory synchrony for recognition memory: synchronous and asynchronous changes similarly impaired recognition for pictures encoded at those changes, compared to pictures encoded further away from those changes. Multisensory synchrony did affect temporal associative memory, which was worse for pictures encoded at synchronous than at asynchronous changes. Follow up experiments showed that this effect was not due to the higher dimensionality of multisensory over unisensory contexts (Experiment 2), nor that it was due to the temporal unpredictability of contextual changes inherent to Experiment 1 (Experiment 3). We argue that participants formed situational expectations through multisensory synchronicity, such that synchronous multisensory changes deviated more strongly from those expectations than asynchronous changes. We discuss our findings in light of supportive and conflicting findings of uni- and multi-sensory segmentation.

Distilling the value of public service media: Towards a tenable conceptualisation in the European framework

This paper problematises the difficulty of defining the public value of public service media. Through a double qualitative methodology, a series of components are identified and assessed in order to propose a tenable conceptualisation in the European framework. First, a qualitative analysis of the main legislations in force ( n = 44) and the grey literature public service media organisations display on their websites ( n = 14) was developed to extract, on an exploratory basis, the common constituent components of public value. Based on this, we developed a guided focus group of experts ( n = 15) to help us build a conception of public service media's value that can retain its validity for a long period of time in the multiplatform era. The results highlight the problems in proposing a single definition given the socio-political diversity of the spaces in which public service media operate and the uncertainty of a constantly changing audiovisual context.

Distinct multisensory perceptual processes guide enhanced auditory recognition memory in older cochlear implant users

In naturalistic situations, sounds are often perceived in conjunction with matching visual impressions. For example, we see and hear the neighbor’s dog barking in the garden. Still, there is a good chance that we recognize the neighbor’s dog even when we only hear it barking, but do not see it behind the fence. Previous studies with normal-hearing (NH) listeners have shown that the audio-visual presentation of a perceptual object (like an animal) increases the probability to recognize this object later on, even if the repeated presentation of this object occurs in a purely auditory condition. In patients with a cochlear implant (CI), however, the electrical hearing of sounds is impoverished, and the ability to recognize perceptual objects in auditory conditions is significantly limited. It is currently not well understood whether CI users – as NH listeners – show a multisensory facilitation for auditory recognition. The present study used event-related potentials (ERPs) and a continuous recognition paradigm with auditory and audio-visual stimuli to test the prediction that CI users show a benefit from audio-visual perception. Indeed, the congruent audio-visual context resulted in an improved recognition ability of objects in an auditory-only condition, both in the NH listeners and the CI users. The ERPs revealed a group-specific pattern of voltage topographies and correlations between these ERP maps and the auditory recognition ability, indicating a different processing of congruent audio-visual stimuli in CI users when compared to NH listeners. Taken together, our results point to distinct cortical processing of naturalistic audio-visual objects in CI users and NH listeners, which however allows both groups to improve the recognition ability of these objects in a purely auditory context. Our findings are of relevance for future clinical research since audio-visual perception might also improve the auditory rehabilitation after cochlear implantation.

Technological processes of creating soundtrack in an audiovisual work

The purpose of the article is to study the main components of the technological process in the formation of the soundtrack of the audiovisual space. The research methodology is based on the use of methods of source search, systematization, and comparative analysis. The scientific novelty lies in the representation of the attempt to create a theoretical concept concerning the importance of technological processes in creating the sound design of works. Conclusions. The article presents the concept of technological processes for creating sound in the audiovisual context, as a scientific and practical phenomenon, in the form of a panoramic figure. The research will serve as an incentive to search for updated options for combining existing elements of the background composition, as well as to create previously unused sound effects.
 Keywords: sound design skills, background composition, sound field unit, sound recording techniques, sound design.

Musical drama in audiovisual space

The purpose of the article is to study musical drama, as an essential element of the audiovisual space and life, to reveal its role and influence on the quality and nature of the audio field as a whole. The research methodology is based on the use of analysis, and synthesis. The scientific novelty lies in the representation of the attempt to create a theoretical concept concerning the significance, role, and implementation in the reality of musical drama as the most important component of the audiovisual space. Conclusions. As a result of studying the above problem, there is a theoretical concept, which consists in determining the nature of the interaction, as well as the role and mutual influence of musical drama and audiovisual space. The results obtained make it possible to realize the importance of musical drama for the audiovisual context, its influence on its qualitative characteristics, and perception by the audience. This opens up prospects for such spheres as cinematography, media field, synthetic art compositions, combining audio and video elements. Keywords: musical art, architectonics, audiovisual field, audiovisual context, the theoretical concept of the phenomenon.