Audiovisual Interactions Research Articles

White Spaces Unveiled: Investigating the Restorative Potential of Environmentally Perceived Characteristics in Urban Parks during Winter

Creating attractive urban green spaces in severely cold and harsh climates is significant for promoting peoples’ health and perceived restoration. However, there is little evidence regarding the urban green spaces in wintery and cold climates and its restorative benefits. This study utilized a pixel grid approach to quantify winter landscape characteristics and a self-reporting method to assess the restorative benefits of audiovisual interactions. The results show the following: (1) Different types of roads in urban parks have significant differences in their level of restorativeness, and the restorativeness benefits of the primary path in winter parks are the strongest. (2) The presence of snowy elements in winter landscapes can enhance park users’ potential to experience restorative characteristics in relation to “being away”. Moreover, there exists a noteworthy positive correlation between deciduous trees and their restoration benefits. (3) People’s perceptions of the tranquility of the soundscape and the duration of environmental exposure are critical mediators in the impact of the restorative path effect. (4) Compared with women, men have a higher restorative level in both the landscape and soundscape. This elucidates the restorative role of white space landscapes and soundscapes in public psychological perception when proposing appropriate forest-based healthcare strategies. It also provides theoretical guidance and optimization schemes for the overall planning, health planning, and design of white spaces shaped by cold urban green spaces.

“Sorry, one more time”: The Role of Resolution Strategies in a Virtual Exchange partnership

With new technologies rapidly developing and the growing relevance of communicative competence in language education, Virtual Exchanges (VEs) are receiving increased attention in research within the framework of the Interaction Hypothesis. One of the intrinsic elements of interaction is Negotiation of Meaning (NoM), a process in which students attempt to solve communicative issues. Nevertheless, few studies have scrutinised how students solve these breakdowns in VE interaction. The purpose of this paper is to identify the most employed resolution strategies in three online audio-visual interactions. The participants are university students from Japan and Spain who carried out one-hour Zoom interactions. Methodology-wise, Clavel-Arroitia’s categorisation was adapted to the purposes of this study in order to identify the strategies in the corpus. The results illustrate how certain factors (language proficiency, cultural background, and communicative dynamics, among others) condition the strategies employed, emphasising the complexity of foreign language learners’ interaction.

Psycho-physiological restoration with audio-visual interactions through virtual reality simulations of soundscape and landscape experiences in urban, waterfront, and green environments

The literature lacks a model of the specific environmental element that induces restoration. Thus, this study contributes to the development of sustainable and healthy environments by examining the effect of the soundscape and landscape experience on psycho-physiological restoration from the audio-visual interaction perspective. It examined various urban and natural environments to propose a restoration model for environmental design application. Sixty participants evaluated three each of urban, waterfront, and green environments using virtual reality technology. The audio-visual characteristics of environments were quantified, and environmental perception was assessed per 14 audio-visual element identifications, six perceived affective qualities, and six overall environmental qualities. The restoration responses were classified as psychological and physiological. Accordingly, 10 restoration models that can interpret design factors of the urban environment with a maximum correlation of 0.80 and an explanatory power of 22 % emerged from urban audio-visual design elements and psycho-physiological responses. Consequently, to improve restoration response, visually attractive and spatial-natural landscapes and acoustically supportive and tranquil soundscapes should be created. Moreover, urban designers can use the models to clearly identify physical characteristics that should be prioritized when promoting the health improvement of urbanites. Furthermore, the study findings can serve as basic data for the development of healthy environments.

A study of noise exposure of traffic-type pocket parks in old urban districts based on audio-visual interaction

The small-scale green spaces next to major roads in old urban districts have long been subjected to traffic noise, posing health risks to the active people within these areas. In accordance with China's “Healthy China” national strategy and the construction of “pocket parks” in cities, it is of significant practical and research value to eliminate noise pollution and propose techniques for optimizing the noise environment of such traffic-type pocket parks. After conducting preliminary investigations and on-site monitoring, it was discovered that the level of noise pollution within the pocket park was relatively high. Given the limited noise reduction capabilities of green belts, it is worth exploring the potential of audio-visual interactions to improve and control noise perception in more detail. This study conducted panoramic image shooting and panorama-element-visibility calculations, generated a self-assessment report on noise perception using a questionnaire, and combined the report with structural equation model calculations. The findings revealed that the visual features of the pocket park environment have an impact on the noise perception of people from different age groups. This paper proposes an optimal range of indicators for the visibility of green space landscape elements related to noise-environment subjective perception improvement and establishes technological methods for noise-pollution control. It also offers insights and examples for promoting the development of healthy green spaces.

Road traffic noise annoyance mitigation by green window view: Optimizing green quantity and quality

There is convincing real-life evidence that seeing outdoor vegetation through the windows of one’s dwelling is able to mitigate negative health effects due to exposure to environmental noise, in particular for noise annoyance due to road traffic. However, design guidelines with respect to green quantity and quality to maximally benefit from this audio-visual interaction are currently lacking, but are mandatory when this idea is to be used in urban sound (and green) planning. Therefore, two virtual reality (VR) experiments were conducted, where participants were positioned near the window of a living room overlooking a city ring road, where the central reservation was used to design various greening scenarios. Participants were exposed to an A-weighted equivalent sound pressure level of 67 dB at eardrum (window partly opened). In the first experiment (79 participants), containing trees of two visually similar tree species, the optimal green quantity (using RGB greenness) was found to be near 30%. This effect, however, was not very pronounced and only amounted to 0.5 units on an 11-point noise annoyance scale. Only the very dense vegetation belt (50%) lead to a higher self-reported noise annoyance at the 5% statistical significance level. In the second VR experiment (62 other participants), vegetation quantity was fixed near this optimum, while green quality varied on the dimensions species richness, colorfulness, and maintenance degree. Green infrastructure containing most colors, or those containing most species, lead to a minimum in self-reported noise annoyance (0.7 units difference on the 11-point annoyance scale). Further analysis suggested that aesthetic value of the green infrastructure is the driving factor for the positive audio-visual interactions observed, consistent with the presumed mechanisms why green window view is able to reduce noise annoyance at home.

Electrophysiological evidence of crossmodal correspondence between auditory pitch and visual elevation affecting inhibition of return

Inhibition of return (IOR) has proved to be weakened by audiovisual integration because of the increased perceptual salience of targets. Although other audiovisual interactions, such as crossmodal correspondence, have also been shown to facilitate attentional processes, to the best of our knowledge, no study has investigated the interaction between crossmodal correspondence and IOR. The present study employed Posner’s spatial cueing paradigm and manipulated the cue validity, crossmodal correspondence congruency and time interval of auditory and visual stimuli (AV interval) to explore the effect of crossmodal correspondence on the IOR effect. The behavioral results showed a reduced IOR effect under the correspondence congruency condition in contrast to the correspondence incongruency condition at the AV interval of 200 ms, whereas at an AV interval of 80 ms, the decreased IOR effect under crossmodal correspondence congruency was eliminated. The electrophysiological results showed a reduced amplitude difference in P2 between valid and invalid cue conditions when the crossmodal correspondence effect decreased the IOR effect. The present study provided the first evidence of the weakened effect of the crossmodal correspondence effect on the IOR effect, which could be eliminated by audiovisual integration.

Research on the Healing Potential of Urban Parks from the Perspective of Audio-Visual Integration: A Case Study of Five Urban Parks in Chengdu

In recent decades, rapid urbanization has been linked to negative impacts on people’s mental health. However, the healing potential of urban parks as central features of cityscapes has not been fully explored from the perspective of audio-visual integration. This gap limits designers’ ability to leverage parks’ healing systems to their full potential. To address this gap, this study used the Mindwave monitoring and recovery component scale to evaluate the healing function of urban parks in Chengdu, specifically focusing on audio-visual integration. Based on audio-visual scene combination samples collected through shooting and recording, we found that: (1) The visual and audio qualities of birdsong (the sky above the blue space and the green space) can significantly differ. (2) Birdsong and conversation seem to provide healing effects from seemingly contradictory dimensions of “quiet” and “social”, with gender differences as the primary influencing factor. (3) Visible children laughing at low levels (30% and 50%) has a more negative impact than invisible traffic noise at the same sound pressure level. (4) Audio-visual interaction does not always have a positive effect, with the visibility of the sound source as the primary influencing factor. (5) An increase in audio proportion did not necessarily correspond to a linear increase in the healing effect. Through exploring the influence of audio-visual combination scenes on healing effects in urban parks, this study provides an empirical basis for park design and planning that considers audio-visual healing effects. It offers insights into designing healing systems for parks and urban areas as well.

Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (ACF) blocks is proposed to effectively conduct audio-visual interaction. The ACF block equipped with spherical positional encoding enables the fusion in the 3D context to capture the spatial correspondence between pixels and sound sources from the equirectangular frames and ambisonic audios. Experimental results verify the effectiveness of our proposed components and demonstrate that our method achieves state-of-the-art performance on the ASOD60K dataset.

An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound

Abstract This article presents a custom system combining hardware and software that senses physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.

Audio–Visual Cross-Modal Correspondences of Perceived Urgency: Examination through a Speeded Discrimination Task

Abstract When presenting information in vehicle cockpits, it is essential to convey an appropriate urgency to the drivers. Perceived urgency has been investigated over the years for each modality, particularly audition and vision. However, the interaction between the modalities of perceived urgency has rarely been examined. To expand the insight into the design application of information presentation, we investigated the audio–visual interaction of perceived urgency using a priming task that involved speeded visual-target discrimination. A total of 60 auditory stimuli were created using a synthesizer. In addition, 13 color squares were used as visual stimuli. Three auditory stimuli with high, medium, and low perceived urgency, and two visual stimuli with high and low perceived urgency were selected based on a subjective evaluation test using a seven-point scale. A priming task was conducted to examine the cross-modal interaction of perceived urgency. Auditory stimuli were presented as prime stimuli, and the participants were asked to discriminate the visual target as quickly as possible. The results revealed that auditory stimuli with high and low perceived urgency facilitated responses to each visual stimulus with similar perceived urgency relative to each visual stimulus with different perceived urgency. The auditory stimulus with medium perceived urgency also facilitated responses to the visual stimulus with high perceived urgency relative to low. The present study shows that cross-modal correspondences can be observed when the stimuli are selected based on their subjective perceived urgency.

The integration of continuous audio and visual speech in a cocktail-party environment depends on attention

In noisy environments, our ability to understand speech benefits greatly from seeing the speaker's face. This is attributed to the brain's ability to integrate audio and visual information, a process known as multisensory integration. In addition, selective attention plays an enormous role in what we understand, the so-called cocktail-party phenomenon. But how attention and multisensory integration interact remains incompletely understood, particularly in the case of natural, continuous speech. Here, we addressed this issue by analyzing EEG data recorded from participants who undertook a multisensory cocktail-party task using natural speech. To assess multisensory integration, we modeled the EEG responses to the speech in two ways. The first assumed that audiovisual speech processing is simply a linear combination of audio speech processing and visual speech processing (i.e., an A + V model), while the second allows for the possibility of audiovisual interactions (i.e., an AV model). Applying these models to the data revealed that EEG responses to attended audiovisual speech were better explained by an AV model, providing evidence for multisensory integration. In contrast, unattended audiovisual speech responses were best captured using an A + V model, suggesting that multisensory integration is suppressed for unattended speech. Follow up analyses revealed some limited evidence for early multisensory integration of unattended AV speech, with no integration occurring at later levels of processing. We take these findings as evidence that the integration of natural audio and visual speech occurs at multiple levels of processing in the brain, each of which can be differentially affected by attention.

Effect of Audio–Visual Factors in the Evaluation of Crowd Noise

A crowd can be both a sound source and an absorber. The sound of human voices significantly impacts evaluations of acoustic indicators in urban public spaces. This study aimed to investigate whether human sound impacts evaluations of the overall environment from both visual and auditory aspects. Primary sound sources and scenes in Harbin, China urban public space served as the research object. Four sets of sound sources and six sets of images were collected in situ in urban public spaces. A subjective evaluation of both visual and auditory aspects was then performed in the laboratory. The results showed that when different types of sounds in urban public spaces are superimposed with human sound, the volume of human sound (45.6 dBA, 55.6 dBA, and 65.6 dBA) significantly affects the acoustic evaluation of the environment. When the superimposed sounds were birds and music, the evaluation of the environment decreased with the human voice increasing in volume. Crowd density and the surrounding visual environment also influence evaluations of the overall sound environment. In this study, the sound preference and acoustic comfort of birdsong and music decreased as the human sound volume increased. The effect of human sound combined with traffic sounds significantly decreased the scores for sound preference and acoustic comfort at higher volumes. The results of the experiments on audio–visual interactions in which people evaluated visual scenes showed that the influence of the visual density of a crowd on assessments of the sound environment is negatively related to the magnitude of the sound of the crowd. When human voices are at 45.6 dBA and 65.6dBA, there is a significant effect on the evaluation of visual scenes for high-density people. When the sound pressure level of human voices is the same, changes in the visual environment are more likely to affect people’s evaluation of the overall sound environment.

Acoustical impacts of blindness on speech production strategies linked to Lombard speech and audiovisual interaction

This paper investigates the acoustical correlates of Lombard speech and audiovisual interaction, in blind and sighted adults. Ten blind and nine sighted adults were recorded while producing repetitions of the French vowels /i/, /u/, /a/ in a “pVp” context. Sentences were produced under four conditions, varying in terms of interaction type (whether the speaker could be heard and seen—audiovisual interaction—or only heard— auditory interaction—by the interlocutor) and presence/absence of noise. Inter-vocalic and intra-vocalic formant dispersion as well as fundamental frequency (F0), intensity and vowel duration values were measured. Results of linear mixed effects models showed that blind speakers increased their inter-vocalic distances in noise, but decreased it when they were told they were seen by the interlocutor (audiovisual interaction). Blind speakers decreased F0 in the audiovisual conditions while sighted speakers increased F0 in noise. Finally, sighted speakers produced louder vowels in the noisy conditions and in the auditory interactive conditions. This pattern was only found for unrounded vowels in blind speakers. These results highlight the role of vision on speech production and show that sighted speakers likely use active strategies to enhance visual cues in audiovisual interactions, compared to auditory only interactions.

A preliminary field experiment to predict soundscape perception based on the biophilia hypothesis paradigm

In a time when biophilic design is investigated mainly with the incorporation of greenery in indoor environments, this study seeks to investigate the relationship between biophilic quality and soundscape perception in oncology polyclinics with and without a garden in Ankara, Turkey. The study, which evolves in an actual environment, claims that the Biophilia Hypothesis paradigm can predict the soundscape perception. Therefore, it aims to examine the audio-visual interaction manifested in polyclinics and its effect on both soundscape perception and biophilic quality using the Biophilic Interior Design Index (BIDI) and soundscape analysis ISO/TS-12913-3 method A. For the objective characterization of the environment, both binaural audio recordings and photographs have been taken, whereas the BIDI has been back-translated by two bilingual experts. In line with the existing literature, the study’s preliminary result supports the tendency for polyclinics with a garden to score higher in biophilic quality, therefore soundscape perceptual affective quality is placed on the positive dimension of the circumplex model, whereas polyclinics without a garden score the opposite.

Effects of audiovisual interactions on working memory: Use of the combined N-back + Go/NoGo paradigm.

Approximately 94% of sensory information acquired by humans originates from the visual and auditory channels. Such information can be temporarily stored and processed in working memory, but this system has limited capacity. Working memory plays an important role in higher cognitive functions and is controlled by central executive function. Therefore, elucidating the influence of the central executive function on information processing in working memory, such as in audiovisual integration, is of great scientific and practical importance. This study used a paradigm that combined N-back and Go/NoGo tasks, using simple Arabic numerals as stimuli, to investigate the effects of cognitive load (modulated by varying the magnitude of N) and audiovisual integration on the central executive function of working memory as well as their interaction. Sixty college students aged 17-21 years were enrolled and performed both unimodal and bimodal tasks to evaluate the central executive function of working memory. The order of the three cognitive tasks was pseudorandomized, and a Latin square design was used to account for order effects. Finally, working memory performance, i.e., reaction time and accuracy, was compared between unimodal and bimodal tasks with repeated-measures analysis of variance (ANOVA). As cognitive load increased, the presence of auditory stimuli interfered with visual working memory by a moderate to large extent; similarly, as cognitive load increased, the presence of visual stimuli interfered with auditory working memory by a moderate to large effect size. Our study supports the theory of competing resources, i.e., that visual and auditory information interfere with each other and that the magnitude of this interference is primarily related to cognitive load.

A comparative study on the audio-visual evaluation of the grand Song of the Dong soundscape

Soundscape heritage records the cultural connotations of different ethnic groups, the destruction, and the disappearance of certain heritage makes its preservation urgent. This work aims to explore the influence of audio-visual different sensory stimuli on the subjective cognition, perceptual dimension and value evaluation of the Grand Song of the Dong soundscape heritage. We have conducted an audio-visual interaction experiment, using 2D video and high-fidelity audio systems to show typical visual elements and cultural scenes of the Grand Song of the Dong, and combined them with questionnaires to collect subjective data. The results show that the visual stimuli have no effect on the subjective evaluation of the Grand Song of the Dong soundscape, and that the different social characteristics of the participants are the main reason for the variations. The influence of audio-visual interaction on the perception of the of the Grand Song of the Dong soundscape is greater, with an enhanced perceptual dimension than the single auditory sensory, i.e., the scale dimension, indicating that visual stimuli can enhance the sensing of spatial limitation. The introduction of the visual element results in higher scores for each heritage value than the auditory element alone, indicating a positive impact on the value evaluation. The results of these studies provide a theoretical basis for the living transmission and holistic conservation of the Grand Song of the Dong and their scenes.

A Qualitative Approach to Explore Audio-Visual Interaction in a Hospital Environment

This study presents the findings of audio and visual evaluations of a hospital environment. The research focused on the waiting area of an oncology polyclinic. The study aimed to get subjective responses of the oncology patients via Grounded Theory and create a conceptual framework with the patients' answers. The polyclinic's acoustic and visual environments were identified, and LAeq levels were measured. Grounded Theory revealed patients' perception of the acoustic and visual environments of the polyclinic. Semi-structured interviews were conducted with 20 voluntary patients in three of the polyclinic's most crowded areas (e.g., reception, courtyard, and corridor). The conceptual framework showed that patients were affected by both audio and visual characteristics of the polyclinic. Patients explained the polyclinic with its existing condition and preferred condition. The results revealed that patients want to hear additional relaxing sound sources such as music and nature sound rather than silence or noise, and they want to see natural elements such as water and greenery rather than plain walls.

Mental Effort and Expressive Interaction in Expert and Student String Quartet Performance

Resilience to changes in performance environments is a hallmark of expertise in music performance. Research has shown that skilled non-expert ensemble musicians maintain synchronization when visual interaction between them is disrupted, but that the quality of their expressive body motion changes. Our study extended these findings by testing how an expert string quartet responds to playing conditions that disrupt visual contact. We tested for potential effects on the musicians’ expressive head motion, sound quality, and mental effort. The Danish String Quartet (DSQ), a world-class classical ensemble, performed an excerpt from a Haydn piece five times without an audience, then once for an audience of about 20 people. During the performances without audience, their seating configuration was manipulated to disrupt their audiovisual interaction. Audio, head motion, eye-tracking, and pupillometry data were collected. The DSQ's data were compared to data from a student quartet who completed the same experiment. Our results show that the DSQ maintained the quality of their musical sound and interactive body motion across disruptive and non-disruptive conditions, but mental effort (indexed by pupil size) was greater in non-disruptive conditions. In contrast, the student quartet moved less overall than the DSQ, moved less when they could not see each other, and did not show differences in pupil size across conditions. The quartets spent a similar percentage of performance time watching their co-performers. These findings suggest that the quality of audio and visual components of the DSQ's performance do not require visual interaction to maintain; however, these musicians do interact visually when given the opportunity. This visual interaction stimulates greater mental effort, perhaps reflecting increased social engagement.

Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones.

Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely "cold/warm," "soft/hard," "transparent/turbid," "far/near," "weak/strong," pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.

Eye responses reflect spatial congruency and perceptual bias in audiovisual interaction

Recent studies have reported that increased pupil dilation and reduced microsaccades are associated with perceptual task enhancement in the bisensory over unisensory conditions. However, little is known about the impacts of bimodal stimulus congruency on those eye responses. We adopted the spatial ventriloquist paradigm to investigate whether spatial congruency of audiovisual stimuli and differences in perceptual experience influence eye responses accompanied by behavioral responses. A simple beep sound and a single disk were used as auditory and visual stimuli respectively. We manipulated spatial congruency of audiovisual stimuli by presenting a visual stimulus on the left or right side of a monitor with an auditory stimulus from the left, right, or center of the monitor. 15 participants performed a sound localization task by clicking their perceived location of the auditory stimulus on the monitor. Participants maintained fixation on a small dot at the center of the monitor with their heads fixed on a chin rest, while their eye responses were recorded using an eye tracker (Eyelink-1000, SR Research). To examine differential perceptual experience in the audiovisual congruence conditions, we classified all responses of audiovisual trials into two groups based on whether the perceived location of the auditory stimulus biased toward the visual stimulus. We found greater pupil dilation accompanied by a greater ventriloquist effect (VE) in the spatially incongruent condition compared to the congruent condition. In addition, a decrease in the number of microsaccades was associated with the auditory perceptual bias towards visual stimuli in the spatially incongruent condition. Our results showed that both spatial congruency and perceptual bias modulate the VE. Results from eye responses suggest that pupillometry is a reliable marker for spatial congruency whereas microsaccades reflect the perceptual bias.