Audiovisual Perception Research Articles

Exploring perception and immersive experience of visually impaired people in 360° video and 3D ambisonic audio

For a long time, immersive experiences have been designed with primarily sighted users in mind. However, recent advancements open exciting possibilities for expanding this realm to include individuals with visual impairments such as Cataracts, Glaucoma, and Severe low visual. One promising avenue lies in utilizing 360-degree videos with 3D Ambisonic audio recordings to create immersive experiences specifically tailored to their unique perceptual abilities. In this context, this preliminary study investigates audio-visual perception and presence of visually impaired people (VIP) in immersive reality (IR) environments, by applying 360-degree video and 3D Ambisonic audio recordings in three different environments of a public park "Parking area, Green space, and Playground". The methodological approach of this study is based on a comparative study between sighted (SIG) and visually impaired people (VIP), by conducting questionnaires such as the presence questionnaire and the emotional salience questionnaire. The findings show that both categories can experience the same presence within the IR environment with a difference in the evaluation of the acoustic environment, where the visually impaired show a greater influence on mechanical sounds. By designing inclusive experiences for all, IR has the potential to be equally engaging and immersive for everyone, regardless of sight.

Multimodality-guided Visual-Caption Semantic Enhancement

Video captions generated with single modality, e.g. video clips, often suffer from insufficient event discovery and inadequate scene description. Therefore, this paper aims to improve the quality of captions by addressing these issues through the integration of multi-modal information. Specifically, We first construct a multi-modal dataset and introduce the triplet annotations of video, audio and text, fostering a comprehensive exploration about the associations between different modalities. Build upon this, We propose to explore the collaborative perception of audio and visual concepts to mitigate inaccuracies and incompleteness in captions in vision-based benchmarks by incorporating audio-visual perception priors. To achieve this, we extract effective semantic features from visual and auditory modalities, bridge the semantic gap between audio-visual modalities and text, and form a more precise knowledge graph multimodal coherence checking and information pruning mechanism. Exhaustive experiments demonstrate that the proposed approach surpasses existing methods and generalizes well with the assistance of ChatGPT.

Exploring the Impact of Visual and Aural Elements in Urban Parks on Human Behavior and Emotional Responses

As cities progress into high-quality developments, the demand for urban parks that enhance residents’ well-being and sustainability is increasing. Traditional visual-centric design methods no longer suffice. Given that vision and hearing are the primary sensory pathways through which people perceive their environment, exploring their relationship with landscape experiences offers a novel perspective for optimizing the audiovisual perception quality of urban parks. This study explores the relationship between visual and auditory elements and landscape experiences to optimize urban parks’ sensory quality. Using visual perception, soundscape perception, sound source perception, and behavioral vitality, this study evaluates the audiovisual perception quality of a representative wetland park in Chengdu’s ring ecological zone. By quantifying relationships between audiovisual characteristics, behavioral vitality, and emotional feedback, several emotional assessment models were constructed. The results show that lawns, pavements, and sound pressure levels significantly impact vitality. A sound pressure level of 77 dB has been identified as a critical threshold in emotional perception models. Consequently, distinct emotional prediction models can be employed to enhance landscape design across various sound pressure level zones. This research provides scientific evidence and flexible strategies for designing urban open spaces that improve landscape experiences based on multisensory perception.

Exploring the impact of university green spaces on Students' perceived restoration and emotional states through audio-visual perception

A growing body of evidence suggests that university green space (UGS) can substantially enhance students' perceived restoration. This study employed virtual reality technology to assess the characteristics of audio-visual elements in three types of UGS, then utilized Peason correlation analysis and structural equation modeling (SEM) to explored environmental audio-visual perception, perceived restoration, and their associations with emotional state. Findings indicate that: (1) Within the same UGS type, the dominance of different audio-visual elements was relatively consistent across various locations. However, there were differences among different UGS types: artificial sounds were dominant in Commuter Green Spaces, while natural sounds were more prominent in Natural Green Spaces, and the perception trends for visual elements were similar to those of the sound sources. (2) Soundscape pleasantness was significantly positively correlated with three dimensions of perceived restoration and positive emotion, negatively correlated with negative emotion, and had similar findings to soundscape pleasantness in visual perception and soundscape eventfulness. (3) The SEM results showed that soundscape pleasantness and eventfulness could directly influence visual perception and positive emotion. In addition, our study also found a cascading mediating effect with visual perception and perceived restoration as core elements. These findings deepen our understanding of how audio-visual perceptions influence emotional restoration and well-being, potentially enhancing both student experiences and the quality of UGS. Furthermore, the findings underscore the complexities involved in perception assessment, influenced by variations in social backgrounds, cultural factors, and individual characteristics. This emphasizes the need for future research to consider these variables comprehensively, aiming to provide a deeper understanding of the specific physical attributes of restorative environments.

Traffic sounds in office spaces: PLS-SEM study on audio-visual perception with open or closed windows and biophilic design

Herein, we investigated the audio–visual perception of indoor greenery and intruding traffic sounds through open or closed windows in a shared office context that accommodates two people. A set of 32 audio–visual stimuli was used, featuring combinations of greenery dosages (0 %, 7 %, 10 % and 13 %) and traffic sound levels (50 dBA, 55 dBA, 60 dBA and 65 dBA), for both open and closed windows. These stimuli were presented to 42 participants in an indoor office environment using two-dimensional screens and over-ear headphones. We employed a partial least squares structural equation model to integrate greenery dosages, traffic sound levels and perceptual attributes such as pleasantness, eventfulness, appropriateness, preference and visual aesthetics. The findings demonstrate the impact of traffic sound levels on visual perception, highlighting a bidirectional interaction between auditory and visual perceptions. Indoor greenery indirectly enhances the pleasantness of the auditory environment, supporting the implementation of biophilic design in office spaces that are visually appealing and acoustically pleasant. Furthermore, the study contributes to discussions on adaptive acoustic comfort in naturally ventilated spaces, providing evidence of how visual perception impacts the acoustic environment. While the research results highlight the positive impact of open windows on the appropriateness ratings of traffic sounds, it is important to consider the limitation of assessing outdoor-recorded traffic sounds at high sound pressure levels (50–65 dBA) without the filtering effect of the facade. This limitation reduces the realism and ecological validity of the experiment, which should be considered when interpreting these results.

Audiovisual perception of interrupted speech by nonnative listeners.

The purpose of the present study was to examine the influence of visual cues in audiovisual perception of interrupted speech by nonnative English listeners and to identify the role of working memory, long-term memory retrieval, and vocabulary knowledge in audiovisual perception by nonnative listeners. The participants included 31 Mandarin-speaking English learners between 19 and 41 years of age. The perceptual stimuli were noise-filled periodically interrupted AzBio and QuickSIN sentences with or without visual cues that showed a male speaker uttering the sentences. In addition to sentence recognition, the listeners completed a semantic fluency task, verbal (operation span) and visuospatial (symmetry span) working memory tasks, and two vocabulary knowledge tests (Vocabulary Level Test and Lexical Test for Advanced Learners of English). The results revealed significantly better speech recognition in the audio-visual condition than the audio-only condition, but the magnitude of visual benefit was substantially attenuated for sentences that had limited semantic context. The listeners' vocabulary size in English played a key role in the restoration of missing speech information and audiovisual integration in the perception of interrupted speech. Meanwhile, the listeners' verbal working memory capacity played an important role in audiovisual integration especially for the difficult stimuli with limited semantic context.

Audiovisual Perception of Lexical Stress: Beat Gestures and Articulatory Cues.

Human communication is inherently multimodal. Auditory speech, but also visual cues can be used to understand another talker. Most studies of audiovisual speech perception have focused on the perception of speech segments (i.e., speech sounds). However, less is known about the influence of visual information on the perception of suprasegmental aspects of speech like lexical stress. In two experiments, we investigated the influence of different visual cues (e.g., facial articulatory cues and beat gestures) on the audiovisual perception of lexical stress. We presented auditory lexical stress continua of disyllabic Dutch stress pairs together with videos of a speaker producing stress on the first or second syllable (e.g., articulating VOORnaam or voorNAAM). Moreover, we combined and fully crossed the face of the speaker producing lexical stress on either syllable with a gesturing body producing a beat gesture on either the first or second syllable. Results showed that people successfully used visual articulatory cues to stress in muted videos. However, in audiovisual conditions, we were not able to find an effect of visual articulatory cues. In contrast, we found that the temporal alignment of beat gestures with speech robustly influenced participants' perception of lexical stress. These results highlight the importance of considering suprasegmental aspects of language in multimodal contexts.

Comprehensive insights into the impact of bacterial indole-3-acetic acid on sensory preferences in Drosophila melanogaster

Several bacteria of environmental and clinical origins, including some human-associated strains secrete a cross-kingdom signaling molecule indole-3-acetic acid (IAA). IAA is a tryptophan (trp) derivative mainly known for regulating plant growth and development as a hormone. However, the nutritional sources that boost IAA secretion in bacteria and the impact of secreted IAA on non-plant eukaryotic hosts remained less explored. Here, we demonstrate significant trp-dependent IAA production in Pseudomonas juntendi NEEL19 when provided with ethanol as a carbon source in liquid cultures. IAA was further characterized to modulate the odor discrimination, motility and survivability in Drosophila melanogaster. A detailed analysis of IAA-fed fly brain proteome using high-resolution mass spectrometry showed significant (fold change, ± 2; p ≤ 0.05) alteration in the proteins governing neuromuscular features, audio-visual perception and energy metabolism as compared to IAA-unfed controls. Sex-wise variations in differentially regulated proteins were witnessed despite having similar visible changes in chemo perception and psychomotor responses in IAA-fed flies. This study not only revealed ethanol-specific enhancement in trp-dependent IAA production in P. juntendi, but also showed marked behavioral alterations in flies for which variations in an array of proteins governing odor discrimination, psychomotor responses, and energy metabolism are held responsible. Our study provided novel insights into disruptive attributes of bacterial IAA that can potentially influence the eukaryotic gut-brain axis having broad environmental and clinical implications.

Speech perception: Auditory and visual cue integration in children with and without phonological disorder in voiceless fricatives

ABSTRACT The literature reports contradictory results regarding the influence of visual cues on speech perception tasks in children with phonological disorder (PD). This study aimed to compare the performance of children with (n = 15) and without PD (n = 15) in audiovisual perception task in voiceless fricatives. Assuming that PD could be associated with an inability to integrate phonological information from two sensory sources, we presumed that children with PD would present difficulties in integrating auditory and visual cues compared to typical children. A syllable identification task was conducted. The stimuli were presented according to four conditions: auditory-only (AO); visual-only (VO); audiovisual congruent (AV+); and audiovisual incongruent (AV-). The percentages of correct answers and the respective reaction times in the AO, VO, and AV+ conditions were considered for the analysis. The correct percentage of auditory stimuli was considered for the AV- condition, as well as the percentage of perceptual preference: auditory, visual, and/or illusion (McGurk effect), with the respective reaction time. In comparing the four conditions, children with PD presented a lower number of correct answers and longer reaction time than children with typical development, mainly for the VO. Both groups showed a preference for auditory stimuli for the AV- condition. However, children with PD showed higher percentages for visual perceptual preference and the McGurk effect than typical children. The superiority of typical children over PD children in auditory-visual speech perception depends on type of stimuli and condition of presentation.

Tailoring Full STEAM Ahead Empowerment: Creative Coding for 21st Century Girls

INTRODUCTION: This paper provides a comprehensive examination of the Creative Coding project’s impact on female empowerment.OBJECTIVES: The project addresses the gender disparity in STEAM fields by integrating coding and art for girls aged 10-15, using the Strudel tool that combines real-time live coding and music composition.METHODS: Through observation and interviews with the girls who participated and performed at the end of the Creative Coding project, but also with those who dropped out, we cast a deep look into the positive and negative parameters to tailor concrete measures for every girl empowerment - not only super achievers. We collaborated with teachers and female mentors from the Computer Science and Engineering Department as relatable role models, to support feelings of belonging and spark curiosity. Using the new user-friendly Strudel tool that supports the integration of art, music, and audio-visual elements, we aimed at an engaging learning experience. To tailor new actions and increase attrition we used a mess map for analysis of the wicked problem of dropouts. Finally, collaboration among the girls and peer learning were expected as by-product effects.RESULTS: Positive outcomes include increased interest, self-confidence, creativity, and aspirations in STEAM subjects among the girls. However, a more comprehensive and supportive environment is recognized as necessary to engage a broader group of girls in STEAM activities. The findings highlight the importance of cultivating curiosity, providing mentorship, promoting inclusivity and collaboration, and integrating creativity in education to inspire and empower girls in coding and STEAM disciplines.CONCLUSION: The paper concludes by emphasizing the potential of Creative Coding practices and the audio-visual perception of music to enhance girls’ empowerment, creativity, and interest in coding and STEAM fields.

Concordance of movements and songs enhances receiver responses to multimodal display in the starling

Many animals produce signals that consist of vocalizations and movements to attract mates or deter rivals. We usually consider them as components of a single multimodal signal because they are temporally coordinated. Sometimes, however, this relationship takes on a more complex spatiotemporal character, resembling choreographed music. Timing is important for audio-visual integration, but choreographic concordance requires even more skill and competence from the signaller. Concordance should therefore have a strong impact on receivers; however, little is known about its role in audio-visual perception during natural interactions. We studied the effects of movement and song type concordance in audio-visual displays of the starling, Sturnus vulgaris. Starlings produce two types of movements that naturally appear in specific phrases of songs with a similar temporal structure and amplitude. In an experiment with a taxidermic robotic model, males responded more to concordant audio-visual displays, which are also naturally preferred, than to discordant displays. In contrast, the effect of concordance was independent of the specific combination of movement and song types in a display. Our results indicate that the concordance of movements and songs was critical to the efficacy of the display and suggest that the information that birds gained from concordance could not be obtained by adding information from movements and songs.

Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception.

The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.

Well-being and multisensory urban parks at different ages: The role of interoception and audiovisual perception

The aim of the present work was to verify whether the perception of visual and acoustic characteristics of urban parks is influenced by the age of the participants. Fifty young (average age 25 years) and fifty elderly (average age 66 years) people were shown with combinations of different types of urban parks (i.e., green, colorful parks and squares) and different types of sound (i.e., bird song, water, chattering and traffic). For each audio-video scenario, participants indicated how calm, happy and stimulated it made them feel and how pleasant, exciting and attractive each scenario was. In addition, participants performed a cross-modal attention task and filled out an interoceptive awareness questionnaire. The results showed that both young and elderly participants preferred green parks with natural sounds. In addition, elderly participants positively evaluated colorful parks over squares. Interestingly, young participants gave more importance to the evaluation of visual-only elements than auditory-only elements, whereas elderly participants gave more importance to the acoustic characteristics of scenarios than the visual ones. Finally, in the elderly group, both attentional abilities and interoceptive awareness contributed to the positive evaluation of urban parks. Theoretically, the results suggest that both signals from the body and from the external world contribute to a holistic representation of the environment. Cross-modal attention may play a crucial role in this process. On an applied level, the data suggest that the design of colorful parks and soundscapes that evoke nature should be encouraged, especially for older populations.

Grounded theory approach on audio-visual perception of users in the restaurant environment

This study aims to investigate the occupants' auditory and visual perception based on varying spectral characteristics of the soundscape in restaurant environment at Bilkent University, Türkiye. The study hypothesized that the sounds generated from background activity evoke a negative perception, whereas human-generated sounds positively affect users. It also hypothesized that there is a linear correlation between sound's spectrum and the negative soundscape and that users sitting adjacent or facing the buffet counter experienced negative soundscape compared to those sitting adjacent or facing the windows with an outside view. To analyze the users' perception of soundscape and visual environment, measurements were taken from other locations based on their auditory and visual differences due to the ceiling heights, the distance between the tables, and their distance from the buffet service counter. Volunteers were interviewed using semi-structured questions, and their responses were evaluated through a grounded theory method via ATLAS. Additionally, the volunteers were requested to complete a questionnaire based on Method A for each location and throughout the experiment. The results showed that the auditory perception of the users does not only depend on sound levels since the responses towards sound sources and sound levels change depending on the audio-visual characteristics of the space.

Visual cues in sound change: A cross-modal perceptual account for the typological rarity of labial palatalization

Auditory perceptibility is widely recognized as a major contributing factor to listener-based sound change. Although speech perception is also known to be influenced by vision, the role of visual cues in facilitating or inhibiting sound change is not well understood. Palatalization of /tʲ/ and /kʲ/ to /tʃ/ is a typologically common change that is frequently attributed to misperception, while full palatalization of labial consonants is exceedingly rare. We test whether visual cues contribute to this asymmetry by way of the visual distinctiveness of labial closure. An audiovisual perception experiment was conducted in which 40 native Korean speakers were presented with synthetically altered tokens along 5-step continua from /pʰ/, /tʰ/, or /kʰ/ to /t?ɕ/ in audio-only, audiovisual-congruent, and audiovisual-incongruent conditions. Acoustic cues to place of articulation, F2 onset, and spectral frequency during aspiration, were manipulated. Results show that presence of the visual lip closure cue significantly inhibited perception of labials as palatals: the /pʰ/-/tɕʰ/ boundary was shifted toward /tɕʰ/ in the AV-congruent group, and labials were more likely to be perceived as palatals in the absence of lip closure. This finding suggests that visual speech cues may contribute to the typological asymmetry and that sound change can be modulated by visual cues.

Multi-dimensional fusion: transformer and GANs-based multimodal audiovisual perception robot for musical performance art.

In the context of evolving societal preferences for deeper emotional connections in art, this paper explores the emergence of multimodal robot music performance art. It investigates the fusion of music and motion in robot performances to enhance expressiveness and emotional impact. The study employs Transformer models to combine audio and video signals, enabling robots to better understand music's rhythm, melody, and emotional content. Generative Adversarial Networks (GANs) are utilized to create lifelike visual performances synchronized with music, bridging auditory and visual perception. Multimodal reinforcement learning is employed to achieve harmonious alignment between sound and motion. The study leverages Transformer models to process audio and video signals in robot performances. Generative Adversarial Networks are employed to generate visually appealing performances that align with the musical input. Multimodal reinforcement learning is used to synchronize robot actions with music. Diverse music styles and emotions are considered in the experiments. Performance evaluation metrics include accuracy, recall rate, and F1 score. The proposed approach yields promising results across various music styles and emotional contexts. Performance smoothness scores exceed 94 points, demonstrating the fluidity of robot actions. An accuracy rate of 95% highlights the precision of the system in aligning robot actions with music. Notably, there is a substantial 33% enhancement in performance recall rate compared to baseline modules. The collective improvement in F1 score emphasizes the advantages of the proposed approach in the realm of robot music performance art. The study's findings demonstrate the potential of multimodal robot music performance art in achieving heightened emotional impact. By combining audio and visual cues, robots can better interpret and respond to music, resulting in smoother and more precise performances. The substantial improvement in recall rate suggests that the proposed approach enhances the robots' ability to accurately mirror the emotional nuances of the music. These results signify the potential of this approach to transform the landscape of artistic expression through robotics, opening new avenues for emotionally resonant performances.

Metacognition in the audiovisual McGurk illusion: perceptual and causal confidence.

Almost all decisions in everyday life rely on multiple sensory inputs that can come from common or independent causes. These situations invoke perceptual uncertainty about environmental properties and the signals' causal structure. Using the audiovisual McGurk illusion, this study investigated how observers formed perceptual and causal confidence judgements in information integration tasks under causal uncertainty. Observers were presented with spoken syllables, their corresponding articulatory lip movements or their congruent and McGurk combinations (e.g. auditory B/P with visual G/K). Observers reported their perceived auditory syllable, the causal structure and confidence for each judgement. Observers were more accurate and confident on congruent than unisensory trials. Their perceptual and causal confidence were tightly related over trials as predicted by the interactive nature of perceptual and causal inference. Further, observers assigned comparable perceptual and causal confidence to veridical 'G/K' percepts on audiovisual congruent trials and their causal and perceptual metamers onMcGurk trials (i.e. illusory 'G/K' percepts). Thus, observers metacognitively evaluate the integrated audiovisual percept with limited access to the conflicting unisensory stimulus components on McGurk trials. Collectively, our results suggest that observers form meaningful perceptual and causal confidence judgements about multisensory scenes that are qualitatively consistent with principles of Bayesian causal inference. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

An imbalance of excitation and inhibition in the multisensory cortex impairs the temporal acuity of audiovisual processing and perception.

The neural integration of closely timed auditory and visual stimuli can offer several behavioral advantages; however, an overly broad window of temporal integration-a phenomenon observed in various neurodevelopmental disorders-could have far-reaching perceptual consequences. Non-invasive studies in humans have suggested that the level of GABAergic inhibition in the multisensory cortex influences the temporal window over which auditory and visual stimuli are bound into a unified percept. Although this suggestion aligns with the theory that an imbalance of cortical excitation and inhibition alters multisensory processing, no prior studies have performed experimental manipulations to determine the causal effects of a reduction of GABAergic inhibition on audiovisual temporal perception. To that end, we used a combination of in vivo electrophysiology, neuropharmacology, and translational behavioral testing in rats to provide the first mechanistic evidence that a reduction of GABAergic inhibition in the audiovisual cortex is sufficient to disrupt unisensory and multisensory processing across the cortical layers, and ultimately impair the temporal acuity of audiovisual perception and its rapid adaptation to recent sensory experience. Looking forward, our findings provide support for using rat models to further investigate the neural mechanisms underlying the audiovisual perceptual alterations observed in neurodevelopmental disorders, such as autism, schizophrenia, and dyslexia.

The Effect of Cued-Speech (CS) Perception on Auditory Processing in Typically Hearing (TH) Individuals Who Are Either Naïve or Experienced CS Producers.

Cued Speech (CS) is a communication system that uses manual gestures to facilitate lipreading. In this study, we investigated how CS information interacts with natural speech using Event-Related Potential (ERP) analyses in French-speaking, typically hearing adults (TH) who were either naïve or experienced CS producers. The audiovisual (AV) presentation of lipreading information elicited an amplitude attenuation of the entire N1 and P2 complex in both groups, accompanied by N1 latency facilitation in the group of CS producers. Adding CS gestures to lipread information increased the magnitude of effects observed at the N1 time window, but did not enhance P2 amplitude attenuation. Interestingly, presenting CS gestures without lipreading information yielded distinct response patterns depending on participants' experience with the system. In the group of CS producers, AV perception of CS gestures facilitated the early stage of speech processing, while in the group of naïve participants, it elicited a latency delay at the P2 time window. These results suggest that, for experienced CS users, the perception of gestures facilitates early stages of speech processing, but when people are not familiar with the system, the perception of gestures impacts the efficiency of phonological decoding.

Dominance of face over voice in human attractiveness judgments: ERP evidence.

The attractiveness of a person, a complex, and socially relevant type of information, is transmitted in many ways, not least through face and voice. However, it is unclear how the stimulus domains carrying attractiveness information interact. The present study explored the audiovisual perception of attractiveness in a Stroop-like paradigm using event-related potentials (ERPs). Participants were presented with face-voice pairs carrying congruent or incongruent attractiveness information and, in turn, judged the attractiveness level of each domain while ignoring the other. Voice attractiveness judgments were influenced by unattended face attractiveness, in terms of both, early perceptual encoding (N170, P200) as well as later evaluative stages (N400, LPC). In contrast, effects of unattended voice attractiveness on face attractiveness judgments were confined to early perceptual encoding (N170). These results demonstrate not only the interaction of multiple domains in human attractiveness perception at different processing stages but also a relative dominance of face over voice attractiveness.