Audiovisual Stimuli Research Articles

Gaze-based detection of mind wandering during audio-guided panorama viewing.

Unlike classic audio guides, intelligent audio guides can detect users' level of attention and help them regain focus. In this paper, we investigate the detection of mind wandering (MW) from eye movements in a use case with a long focus distance. We present a novel MW annotation method for combined audio-visual stimuli and collect annotated MW data for the use case of audio-guided city panorama viewing. In two studies, MW classifiers are trained and validated, which are able to successfully detect MW in a 1-s time window. In study 1 (n = 27), MW classifiers from gaze features with and without eye vergence are trained (area under the curve of at least 0.80). We then re-validate the classifier with unseen data (study 2, n = 31) that are annotated using a memory task and find a positive correlation (repeated measure correlation = 0.49, p < 0.001) between incorrect quiz answering and the percentage of time users spent mind wandering. Overall, this paper contributes significant new knowledge on the detection of MW from gaze for use cases with audio-visual stimuli.

The effect of distance on audiovisual temporal integration in an outdoor virtual environment

In this study, we explore the influence of stimulus distance on human tolerance for (physical) asynchronies in virtual reality (VR). A repeated audiovisual (AV) stimulus with sound and light bursts was presented to the participants in an outdoor virtual environment (VE) using a head-mounted display (HMD). The investigation focused on quantifying the point of subjective simultaneity (PSS) for both visual and auditory stimuli. A synchrony judgment method (SJ-3) was used for 11 stimulus onset asynchronies (SOA) and five egocentric distances from 10 m up to 50 m with 10 m increments. The data analysis showed negative PSS values that decreased with distance, resulting in a negative slope (-3 ms/m) of the regression line between PSS values and simulated distances. In contrast to the recent study conducted in the indoor VE, we conclude that the presented study in the outdoor VE does not incorporate a distance compensation mechanism and refutes the hypothesis of an ‘implicit estimation’ of sound-arrival time. The reasons behind the observed contrast are discussed in this paper. Moreover, the negative slope of the regression line (−3 ms/m) is similar to previous research, concluding that there is a temporal integration of auditory-visual information within human neural processes without distance compensation.

Postoperative Auditory Progress in Cochlear-Implanted Children With Auditory Neuropathy.

This study aims to evaluate the effect of auditory neuropathy spectrum disorder (ANSD) on postoperative auditory perception and listening difficulties in pediatric cochlear implant (CI) recipients. The Children's Auditory Perception Test (CAPT) assesses auditory perception skills, and the Children's Home Inventory of Listening Difficulties (CHILD) Scale evaluates daily listening difficulties. The study involved pediatric CI recipients (n = 40) aged between 5 and 7 years, with and without diagnosis of ANSD. The research ensured homogeneity across various factors, including chronological age, age at diagnosis, age at initial implantation, bilateral simultaneous surgery, etiologies of hearing loss, and family education level. The findings have demonstrated that children without ANSD exhibited better performance in integrating visual-auditory stimuli and overall listening performance, distant sound source scores, and noisy environment scores (respectively p = .047, p = .001, p = .028, and p = .010). Additionally, children with better speech perception also have a better ability to integrate audiovisual stimuli (p = .005, r = .438). There are significant differences in postoperative listening skills and auditory perceptions between children with and without an ANSD who have CIs. Accordingly, children without an ANSD perform better.

Impaired emotional multimodal integration in inhibition of return in children with attention-deficit/hyperactivity disorder.

Attention-deficit/hyperactivity disorder (ADHD) is a common neurodevelopmental disorder in children. Previous studies have shown that children with ADHD have impaired processing of emotional stimuli, but it is unclear whether their ability to integrate multimodal emotional stimuli is impaired and at which processing pathway this impairment exists. The present study investigated the ability of children with ADHD to integrate emotional audiovisual stimuli under different emotional conditions, and the effect of audiovisual integration on IOR to reveal the impaired processing pathway of their emotional audiovisual integration. Fifty-eight school-age children (29 with ADHD and 29 matched typically developing (TD) children) performed an emotional valence discrimination task with a cue-target paradigm. The results showed that children with ADHD did not exhibit audiovisual integration of emotional stimuli in all experimental conditions. In addition, the IOR effect was significantly smaller for audiovisual targets than for visual targets under the negative but not the neutral emotion condition in children with ADHD, whereas this effect was present in all emotion conditions in TD children. These results indicate that the ability to integrate emotional audiovisual information is impaired in children with ADHD and this impairment exists in both bottom-up and top-down pathways. Additionally, although presenting emotional auditory stimuli at the same time as emotional faces reduced IOR both in children with ADHD and TD, the manner of reduction differed. These findings provide new evidence of emotional processing deficits and multimodal integration deficits in children with ADHD, and help provide support for children in educational settings.

Functional architecture of cerebral cortex during naturalistic movie watching.

Characterizing the functional organization of cerebral cortex is a fundamental step in understanding how different kinds of information are processed in the brain. However, it is still unclear how these areas are organized during naturalistic visual and auditory stimulation. Here, we used high-resolution functional MRI data from 176 human subjects to map the macro-architecture of the entire cerebral cortex based on responses to a 60-min audiovisual movie stimulus. A data-driven clustering approach revealed a map of 24 functional areas/networks, each explicitly linked to a specific aspect of sensory or cognitive processing. Novel features of this map included an extended scene-selective network in the lateral prefrontal cortex, separate clusters responsive to human-object and human-human interaction, and a push-pull interaction between three executive control (domain-general) networks and domain-specific regions of the visual, auditory, and language cortex. Our cortical parcellation provides a comprehensive and unified map of functionally defined areas in the human cerebral cortex.

Multisensory temporal processing in schizophrenia and bipolar disorder: implications for psychosis

Structuring sensory events in time is essential for interacting with the environment and producing adaptive behaviors. Over the past years, the microstructure of temporality received increasing attention, recognized as a fundamental factor influencing cognitive, affective, and social abilities, whose alteration can underlie the etiopathogeneses of some clinical symptoms in psychiatric disorders. The present research investigated multisensory temporal processing in individuals with schizophrenia (N = 21), bipolar disorder (N = 20) and healthy controls (N = 21) in order to explore a plausible link between multisensory alterations in the temporal order of events and the psychopathological dimensions underlying psychosis. We asked participants to temporally order audio-tactile, visual-tactile, and audio-visual stimuli, and we administered different psychopathological scales to explore depressive, manic and psychotic symptoms. Results demonstrated that both subjects with schizophrenia and bipolar disorder are less precise in temporal order judgment independently of the sensory modalities involved. Interestingly, reduced precision in temporal processing of patients is positively associated with the presence and severity of positive symptoms. Our findings support the hypothesis that low-level sensory alterations in temporal structure may contribute to the emergence of clinical symptoms such as delusions, hallucinations, and disorganized behaviors.

Resonance and beat perception of ballroom dancers: An EEG study.

The ability to synchronize the perceptual and motor systems is important for full motor coordination and the core determinant of motor skill performance. Dance-related training has been found to effectively improve sensorimotor synchronization, however, the underlying characteristics behind these improvements still warrant further exploration. This study was conducted to investigate the behavioral and neuroactivity characteristics of ballroom dancers relative to those of non-dancers. Thirty-two dancers (19.8 ± 1.8 years old) and 31 non-dancers (22.6 ± 3.1 years old) were recruited to perform a finger-tapping task in synchrony with audiovisual beat stimuli at two intervals: 400 and 800 ms, while simultaneously recording EEG data. Behavioral and neural activity data were recorded during the task. The dancers employed a predictive strategy when synchronizing with the beat. EEG recordings revealed stronger brain resonance with external rhythmic stimuli, indicating heightened neural resonance compared to non-dancers (p < 0.05). The task was more challenging with an 800-ms beat interval, as observed through both behavioral metrics and corresponding neural signatures in the EEG data, leading to poorer synchronization performance and necessitating a greater allocation of attentional resources (ps < 0.05). When performing the finger-tapping task involving audiovisual beats, the beat interval was the primary factor influencing movement synchronization, neural activity and attentional resource allocation. Although no significant behavioral differences were observed between dancers and non-dancers, dancers have enhanced neural resonance in response to rhythmic stimuli. Further research using more ecologically valid tasks and stimuli may better capture the full extent of dancers' synchronization abilities.

Audio visual stimuli based out of body illusion

Previous studies have reported that bodily self-consciousness could be altered so that one’s body was perceived in extra-personal space. However, whether this could be induced without tactile stimuli has not been investigated. We investigated whether out-of-body illusion could be induced via synchronized audio-visual stimuli, in which auditory stimuli were used instead of tactile stimuli. We conducted an experiment in which a sounding bell was moved in front of the participant, and synchronously, a non-sounding bell was moved in front of a camera that captured the image and projected on a head-mounted device. We expected the participants to experience that the sound came from the non-sounding bell in the video and they were in the camera’s position. Results from the questionnaires conducted after the experiment revealed that items related to out-of-body illusion were significantly enhanced in the synchronized conditions. Furthermore, participants reported a similarly strong out-of-body illusion for both the synchronized audio-visual and tactile-visual stimuli. This study demonstrated that out-of-body illusion could also be induced by synchronized audio-visual stimuli, which was a novel finding.

Intrinsic dynamic shapes responses to external stimulation in the human brain.

Sensory stimulation of the brain reverberates in its recurrent neuronal networks. However, current computational models of brain activity do not separate immediate sensory responses from intrinsic recurrent dynamics. We apply a vector-autoregressive model with external input (VARX), combining the concepts of "functional connectivity" and "encoding models", to intracranial recordings in humans. We find that the recurrent connectivity during rest is largely unaltered during movie watching. The intrinsic recurrent dynamic enhances and prolongs the neural responses to scene cuts, eye movements, and sounds. Failing to account for these exogenous inputs, leads to spurious connections in the intrinsic "connectivity". The model shows that an external stimulus can reduce intrinsic noise. It also shows that sensory areas have mostly outward, whereas higher-order brain areas mostly incoming connections. We conclude that the response to an external audiovisual stimulus can largely be attributed to the intrinsic dynamic of the brain, already observed during rest.

Assessing the Effectiveness of Audio-Visual vs. Visual Neurofeedback for Attention Enhancement: A Pilot Study with Neurological, Behavioural, and Neuropsychological Measures.

Electroencephalogram (EEG) based Neurofeedback training has gained traction as a practical method for enhancing executive functions, particularly attention, among healthy individuals. The neurofeedback protocols based on EEG channel locations, frequency bands, or EEG features has been tested. However, the improvement in attention was not measured by comparing different feedback stimulus types. We believe that multisensory nature feedback even with few training sessions may induce strong effect. Therefore, this study compares the effect of audio-visual and visual feedback stimuli for attention enhancement utilizing neurophysiological, behavioural and neuropsychological measures. Total 21 subjects were recruited, undergoing six alternate days of neurofeedback training sessions to upregulate EEG beta power of frontocentral (FC5). Dwell time, fractional occupancy and transition probability were also estimated from the EEG beta power. Audiovisual group (G1) as compared to visual group (G2) demonstrate a significant increase of global EEG beta activity alongside improved dwell time (t = 2.76, p = 0.003), fractional occupancy (t = 1.73, p = 0.042) and transition probability (t = 2.46, p = 0.008) over the course of six neurofeedback training sessions. Similarly, the group (G1) shows higher scores (t = 2.13, p = 0.032) and faster reaction times (t = 2.22, p = 0.028) in Stroop task, along with increased score in Mindfulness Attention Awareness Scale (MAAS-15) questionnaire (t = 2.306, p = 0.012). Audiovisual neurofeedback may enhance training effectiveness, potentially achieving the same outcomes in fewer sessions compared to visual-only feedback. However, sufficient training days are essential for effect consolidation. This highlights the feasibility of completing neurofeedback training, a significant challenge in practice.

Aesthetic chills modulate reward learning in anhedonic depression

ObjectiveThis study aimed to examine the potential of experiencing aesthetic chills to enhance reward learning in individuals with elevated depressive symptoms, specifically anhedonia, by investigating the effect of chills on participants' ability to modulate behavior as a function of rewards. MethodsA total of 103 participants with elevated depressive symptoms took part in the experiment. Among them, 59 participants had depressive symptoms (BDI ≥ 20), with 26 classified as “High Anhedonic” (HA) and 33 as “Low Anhedonic” (LA). Additionally, 39 participants without elevated depressive symptoms (BDI < 20 and SHAPs <32) were included as the control group. We utilized ChillsDB, an open-source database of validated audiovisual stimuli known to elicit chills in the US population. ResultsAnhedonic participants who experienced chills demonstrated a significant increase in response bias (p = .004) towards rewards compared to those who did not experience chills. Highlighting specificity, no significant difference in reward bias was observed among LA participants. ConclusionsThese findings suggest that the experience of chills has the potential to impact reward learning in anhedonic individuals, aligning with the known neurobiology of this phenomenon. These results highlight the potential of aesthetic chills as a novel approach to elicit and enhance positive affect in depressed populations.

Enhanced multisensory gain in older adults may be a by-product of inverse effectiveness: Evidence from a speeded response-time task.

Older adults experience a greater benefit from multisensory integration than their younger counterparts, but it is unclear why. One hypothesis is that age-related sensory decline weakens unisensory stimulus effectiveness, causing a boost in multisensory gain through inverse effectiveness. Many previous studies present stimuli at the same intensity for both younger and older adults (i.e., stimulus-matched), as opposed to accounting for each participant's unique perceptual ability (i.e., perception-matched). This makes it difficult to discern the source of age-related differences in multisensory gain. As such, we used two experiments to examine whether sensory decline is contributing to age-related differences in multisensory gain. In the first, we presented auditory (pure tones in noise), visual (Gabor patches in noise), and audiovisual stimuli and recorded response times from 31 younger (18-25) and 30 older (55-80) adults. Importantly, all participants were given identical stimuli, with the expectation that older adults would show worse unisensory performance, inducing inverse effectiveness. The second task was identical (younger N = 31, older N = 34), except stimuli were presented at each participant's 50% detection threshold, identified with an adaptive psychophysical staircase, controlling for any influence of inverse effectiveness. Older adults were found to exhibit greater multisensory gain (as measured by race model violations) on stimulus- but not perception-matched tasks, thus aligning with the principle of inverse effectiveness. That is, when accounting for potential age-related differences in perceptual abilities, older adults no longer experienced greater benefit from multisensory integration. These two experiments together suggest that the age-related increases in multisensory integration previously reported may be in part due to age-related declines in vision and audition. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Exploring cross-modal perception: audio, visual, and thermal responses to changes in audio-visual Stimuli within virtual urban parks

Human perception is an essential factor in soundscape. Humans are multisensory beings, and thus, their perceptions are influenced by various multisensorial experiences. Our perception of a soundscape is not solely influenced by the sound in itself but also by various non-auditory factors. This research examines how visual factors, other than auditory, influence people's perceptions in public open spaces. An experiment was conducted using audio and visual stimuli. Three setups were compared: audio-only, visual-only, and combined audio-visual stimuli to observe differences in the respondent's perception. Furthermore, this research examines not only auditory perception but also visual and thermal perceptions, all of which are known to be important in public open spaces. The statistical test ANOVA showed that the Audio-Visual and visual-only stimuli produced a significant difference (p&lt;0.05) in the thermal perception. On the other hand, the Audio-Visual and audio-only stimuli did not result in any significant differences. This study contributes to understanding the interaction between different sensory modalities and how it affects our perception. The findings underscore the importance of considering multisensory experiences, including auditory, visual, and thermal aspects when designing or evaluating environments in urban settings.

Perceptual assessment of urban park audio-visual stimuli a comparison between Italian and Chinese groups

Previous research demonstrated a preference for natural auditory stimuli over human-made sounds in urban park environments. However, little attention has been given to the perceived dimensions of these sounds and their consistency across diverse cultural groups. The present study aimed at revealing possible culture differences among visual (Green, Square, Color) and auditory (Nature, Music, Anthropic) elements in urban parks. Eighty participants from both Chinese and Italian groups were administered a questionnaire to emotionally assess the stimuli. The differences in emotional salience were found across both sensory inputs. For the visual inputs, Green parks were perceived as more pleasant than Colorful parks, and the latter were more pleasant than Square parks, indicating the more natural the setting, the more restorative the effects. An important finding refers to the cross-culture consistency of the preferences. For the auditory inputs in urban parks, although both components of emotional salience were cross-culture consistent, some specific differences emerged.

EEG-based responses of patients with disorders of consciousness and healthy controls to familiar and non-familiar emotional videos

ObjectiveTo investigate the differences in the brain responses of healthy controls (HC) and patients with disorders of consciousness (DOC) to familiar and non-familiar audiovisual stimuli and their consistency with the clinical progress. MethodsEEG responses of 19 HC and 19 patients with DOC were recorded while watching emotionally-valenced familiar and non-familiar videos. Differential entropy of the EEG recordings was used to train machine learning models aimed to distinguish brain responses to stimuli type. The consistency of brain responses with the clinical progress of the patients was also evaluated. ResultsModels trained using data from HC outperformed those for patients. However, the performance of the models for patients was not influenced by their clinical condition. The models were successfully trained for over 75% of participants, regardless of their clinical condition. More than 75% of patients whose CRS-R scores increased post-study displayed distinguishable brain responses to both stimuli. ConclusionsResponses to emotionally-valenced stimuli enabled modelling classifiers that were sensitive to the familiarity of the stimuli, regardless of the clinical condition of the participants and were consistent with their clinical progress in most cases. SignificanceEEG responses are sensitive to familiarity of emotionally-valenced stimuli in HC and patients with DOC.

Interference inhibition of multimodal information in digital interfaces and its rule of cognitive processing

AbstractIn the digital interface of multimodal audio–visual presentation, the appearance of irrelevant information often brings cognitive interference or even confusion, leading to decision‐making errors when users focus on or manipulate the interface target. However, few studies have explored the brain's inhibition effect and cognitive law evoked by audio–visual interference from the perspective of interface information design. On the basis of Stroop's classic interference task, an experimental paradigm of multimodal audio–visual stimuli to induce event‐related potential (ERP) components was designed for digital interfaces in this study. Combining behavioral measurement and ERP technology, this study discussed the differences in the induced inhibition effects between the two carriers under various audio–visual interferences. The findings demonstrated that all five interference stimuli, based on functional icons and Chinese characters, elicited significant N250 and N400, with a similar time course. Compared with the Chinese character group, the functional icon group elicited more negative activity in the frontal and some parietal‐occipital regions, indicating that the functional icon required more cognitive inhibitory resources to resist interference stimuli. Moreover, the inhibition effect induced by audio–visual interference with the same semantics was significantly lower than that of opposite semantics and even lower than that of single‐sensory interference. The findings offered physiological evidence for the inhibition effect induced by audio–visual semantic interference in digital interfaces and proposed design principles for the interface information of human–machine systems.

The influence of spatial frequency information on temporal synchrony perception on audiovisual stimuli.

Previous studies have shown that the spatial frequency (SF) of visual stimuli alters the perceived timing of subjective simultaneity. However, these studies have been limited to the effects of a single SF component. In this study, I measured and compared the points of subjective simultaneity (PSS) for audiovisual stimuli among low, high, and composited SF components. This experiment comprised a dual-presentation timing task and a ternary response format to eliminate response bias. The results indicated that the PSS value of the composition-SF stimuli was more toward visual-lead timing than the low-SF stimuli and did not differ significantly from that of the high-SF stimuli. The correlation coefficients showed that the PSS in composition-SF stimuli marginally approximated that of high-SF stimuli higher than that of low-SF stimuli. Future studies are needed to confirm these findings using visual stimuli with a wider range of SF components and with a modulated contrast.

Predicting the impact of integrated audio-visual environments on perceived restorative benefits across different park types: A field study based on seven parks in Hangzhou, China

The human perception of environments is a complex mechanism, influenced not only by visual and auditory stimuli and their perceptual outcomes, but also by personal characteristics, site types, and other factors that affect people's restorative experiences. However, there is little research that treats it as a complete systemic phenomenon to reveal its integrated impact. We surveyed 1161 visitors in seven urban parks in Hangzhou, China, and collected information on audio-visual stimuli. Using structural equation modeling and multi-group analysis, we initially examined the comprehensive impact of acoustic factors, non-acoustic factors, and individual characteristic variables on perceived restorativeness. Subsequently, we studied the effects of different park types on the constructed multivariate comprehensive model. The results confirm that the presence of different types of sound sources within landscape features can impact the auditory affective quality dimension, which in turn affects visitors' perceived restorative experiences. In addition, the reported mediating pathways revealed that visual elements related to nature, by providing high-quality visual landscapes, enhance the pleasantness of soundscapes and yield greater restorative benefits in park environments. A new finding of the study was that, although the eventfulness of the soundscape negatively affects perceived restorativeness, the presence of high-quality visual landscapes in the environment can compensate for this deficiency. Further multi-group analysis results show that landscape features and visual perceptual quality indirectly influence perceived restorativeness through the auditory affective quality dimension, resulting in varied correlations across two types of park. Moreover, the pleasantness of soundscapes consistently has a significant effect on perceived restorativeness in both types of parks; however, in parks oriented towards activity, the eventfulness dimension of soundscapes does not significantly affect perceived restorativeness. Lastly, the influence of landscape features on perceived restorativeness, mediated by types of sound sources, is both direct and indirect, leading to diverse correlations across different park environments. These detailed findings and guidelines for audio-visual interaction design will aid in the planning and development of urban soundscapes.

A deep learning method for contactless emotion recognition from ballistocardiogram

Emotion recognition is a major research point in the field of affective computing. Existing research on the application of physiological signals to emotion recognition mainly focuses on the processing of contact signals. However, there are issues with contact signal acquisition equipment, such as limited portability and poor user compliance, which make it difficult to promote its use. To explore a new method for emotion recognition based on contactless ballistocardiogram (BCG), we proposed a SE-CNN model with a multi-class focal loss function. To construct the dataset, we used audio-visual stimuli to evoke the subjects’ emotions and collected data on the subjects’ three discrete emotions, positive, neutral, and negative, through our established BCG signal acquisition system based on a piezoelectric ceramics sensor. Root mean square filter and thresholding were used to detect and eliminate motion artifacts of BCG signals. We did two kinds of preprocessing on BCG signals: wavelet transform and bandpass filtering, to explore the effect of different components of BCG on emotion recognition. Subsequently, we verified the model’s performance and cross-time working ability through traditional K-Fold and our proposed K-Session cross-validation methods. The results showed that the band-pass filtering method was more beneficial to the current classification task. Under K-Fold cross-validation, the model’s accuracy, precision, and recall were 97.21%, 97.00%, and 97.11%. Under K-Session cross-validation, the model’s accuracy, precision, and recall were 94.66%, 93.92%, and 94.86%, respectively, all of which were better than the classification effect of synchronous ECG. The reliability of BCG in contactless emotion recognition was proved.

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts.

Communication in the real world is inherently multimodal. When having a conversation, typically sighted and hearing people use both auditory and visual cues to understand one another. For example, objects may make sounds as they move in space, or we may use the movement of a person's mouth to better understand what they are saying in a noisy environment. Still, many neuroscience experiments rely on unimodal stimuli to understand encoding of sensory features in the brain. The extent to which visual information may influence encoding of auditory information and vice versa in natural environments is thus unclear. Here, we addressed this question by recording scalp electroencephalography (EEG) in 11 subjects as they listened to and watched movie trailers in audiovisual (AV), visual (V) only, and audio (A) only conditions. We then fit linear encoding models that described the relationship between the brain responses and the acoustic, phonetic, and visual information in the stimuli. We also compared whether auditory and visual feature tuning was the same when stimuli were presented in the original AV format versus when visual or auditory information was removed. In these stimuli, visual and auditory information was relatively uncorrelated, and included spoken narration over a scene as well as animated or live-action characters talking with and without their face visible. For this stimulus, we found that auditory feature tuning was similar in the AV and A-only conditions, and similarly, tuning for visual information was similar when stimuli were presented with the audio present (AV) and when the audio was removed (V only). In a cross prediction analysis, we investigated whether models trained on AV data predicted responses to A or V only test data similarly to models trained on unimodal data. Overall, prediction performance using AV training and V test sets was similar to using V training and V test sets, suggesting that the auditory information has a relatively smaller effect on EEG. In contrast, prediction performance using AV training and A only test set was slightly worse than using matching A only training and A only test sets. This suggests the visual information has a stronger influence on EEG, though this makes no qualitative difference in the derived feature tuning. In effect, our results show that researchers may benefit from the richness of multimodal datasets, which can then be used to answer more than one research question.