Audiovisual Information Processing Research Articles

Differential audiovisual information processing in emotion recognition: An eye-tracking study.

Recent research has suggested that dynamic emotion recognition involves strong audiovisual association; that is, facial or vocal information alone automatically induces perceptual processes in the other modality. We hypothesized that different emotions may differ in the automaticity of audiovisual association, resulting in differential audiovisual information processing. Participants judged the emotion of a talking-head video under audiovisual, video-only (with no sound), and audio-only (with a static neutral face) conditions. Among the six basic emotions, disgust had the largest audiovisual advantage over the unimodal conditions in recognition accuracy. In addition, in the recognition of all the emotions except for disgust, participants' eye-movement patterns did not change significantly across the three conditions, suggesting mandatory audiovisual information processing. In contrast, in disgust recognition, participants' eye movements in the audiovisual condition were less eyes-focused than the video-only condition and more eyes-focused than the audio-only condition, suggesting that audio information in the audiovisual condition interfered with eye-movement planning for important features (eyes) for disgust. In addition, those whose eye-movement pattern was affected less by concurrent disgusted voice information benefited more in recognition accuracy. Disgust recognition is learned later in life and thus may involve a reduced amount of audiovisual associative learning. Consequently, audiovisual association in disgust recognition is less automatic and demands more attentional resources than other emotions. Thus, audiovisual information processing in emotion recognition depends on the automaticity of audiovisual association of the emotion resulting from associative learning. This finding has important implications for real-life emotion recognition and multimodal learning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Diagnosis of audiovisual information processing in local television archives. Case study in Cuba

The processing of audiovisual information on local television stations is an area that has been little addressed since scientific production. In the methodological order, there is a shortage of tools and instruments that allow diagnosing the processing of audiovisual information on local television. There are specific techniques and procedures that allow the processing of information, but which are limited to the traditional technical documentary processes: classification, cataloging, indexing and abstracting. Due to its characteristics, local television stations requires specific media and models that allow diagnosing the processing of audiovisual information from the materials they produce.

Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones.

Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely "cold/warm," "soft/hard," "transparent/turbid," "far/near," "weak/strong," pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.

Audio-Visual Low Power System for Endangered Waterbirds Monitoring

Recently, more attention is being payed to the ecological environment due to the rapid decline of animal species, especially birds. Wireless Multimedia Sensor Networks (WMSN) can be leveraged to monitor, protect and assess changes in bird populations by periodically capturing and sending images to a collect station. In this paper, we propose a three-tier architecture following the Cloud-Fog-Edge model with an adequate placement of the surveillance system tasks. To address the sensor network limited resources, we propose to reduce the amount of visual data to be reported by sending compressed region of interest (ROI) of only target species images. Endangered birds are identified based on their calls before triggering the camera. However, audio recognition is likely to fail due to the ambient noise that can be encountered in a natural environment. As a result, we augmented our automatic bird song recognition system with an appropriate noise reduction technique. Thanks to the efficient cooperation of its three tiers, the proposed system, based on low-complexity audiovisual information processing, results in a better use of network resources while maximizing the amount of pertinent information.

Investigation of functional brain network reconfiguration during exposure to naturalistic stimuli using graph-theoretical analysis

Objective. One of the most significant features of the human brain is that it can dynamically reconfigure itself to adapt to a changing environment. However, dynamic interaction characteristics of the brain networks in naturalistic scenes remain unclear. Approach. We used open-source functional magnetic resonance imaging (fMRI) data from 15 participants who underwent fMRI scans while watching an audio–visual movie ‘Forrest Gump’. The community detection algorithm based on inter-subject functional correlation was used to study the time-varying functional networks only induced by the movie stimuli. The whole brain reconfiguration patterns were quantified by the temporal co-occurrence matrix that describes the probability of two brain regions engage in the same community (or putative functional module) across time and the time-varying brain modularity. Four graph metrics of integration, recruitment, spatio-temporal diversity and within-community normalised centrality were further calculated to summarise the brain network dynamic roles and hub features in their spatio-temporal topology. Main results. Our results suggest that the networks that were involved in attention and audio–visual information processing, such as the visual network, auditory network, and dorsal attention network, were considered to play a role of ‘stable loners’. By contrast, ‘unstable loner’ networks such as the default mode network (DMN) and fronto-parietal network tended to interact more flexibly with the other networks. In addition, global brain network showed significant fluctuations in modularity. The ‘stable loner’ networks always maintained high functional connectivity (FC) strength while ‘unstable loner’ networks, especially the DMN, exhibited high intra- and inter-network FC only during a low modularity period. Finally, changes in brain modularity were significantly associated with variations in emotions induced by the movie. Significance. Our findings provide new insight for understanding the dynamic interaction characteristics of functional brain networks during naturalistic stimuli.

Changes of Effective Connectivity in the Alpha Band Characterize Differential Processing of Audiovisual Information in Cross-Modal Selective Attention.

Cross-modal selective attention enhances the processing of sensory inputs that are most relevant to the task at hand. Such differential processing could be mediated by a swift network reconfiguration on the macroscopic level, but this remains a poorly understood process. To tackle this issue, we used a behavioral paradigm to introduce a shift of selective attention between the visual and auditory domains, and recorded scalp electroencephalographic signals from eight healthy participants. The changes in effective connectivity caused by the cross-modal attentional shift were delineated by analyzing spectral Granger Causality (GC), a metric of frequency-specific effective connectivity. Using data-driven methods of pattern-classification and feature-analysis, we found that a change in the α band (12Hz-15 Hz) of GC is a stable feature across different individuals that can be used to decode the attentional shift. Specifically, auditory attention induces more pronounced information flow in the α band, especially from the parietal-occipital areas to the temporal-parietal areas, compared to the case of visual attention, reflecting a reconfiguration of interaction in the macroscopic brain network accompanying different processing. Our results support the role of α oscillation in organizing the information flow across spatially-separated brain areas and, thereby, mediating cross-modal selective attention.

Concepción, etapas y modelo del ciclo de vida de la información audiovisual

El audiovisual como documento tiene características que lo diferencian del resto de las fuentes. Las actividades correspondientes al tratamiento de la información audiovisual se corresponden con una serie de etapas que pueden ser descritas a partir de un ciclo de vida. La ceración de un ciclo constituye un modelo de trabajo en centros donde se produzcan o custodien colecciones audiovisuales. Se planteó como objetivo del estudio: establecer el Ciclo de vida de la información audiovisual. El presente estudio clasifica como descriptivo con aporte teórico y práctico. Para la obtención de resultados se emplearon los métodos analítico-sintético, histórico-lógico, inductivo-deductivo, sistémico-estructural y el análisis documental clásico. Se establece el Ciclo de vida de la información audiovisual a partir del establecimiento de seis etapas: 1) origen, 2) selección, 3) creación, 4) difusión, 5) procesamiento y 6) almacenamiento. El ciclo Integra las acciones relativas a los procesos documentales y las relativas a la concepción y realización del audiovisual.

Frequency-specific alteration of functional connectivity density in antipsychotic-naive adolescents with early-onset schizophrenia

Early-onset schizophrenia (EOS) is a severe mental illness associated with dysconnectivity that widespread in the brain. However, the functional dysconnectivity in EOS are still mixed. Recently, studies have identified that functional connectivity (FC) arises from a band-limited slow rhythmic mechanism and suggested that the dysconnectivity at specific frequency bands may provide more robust biomarkers for schizophrenia. The frequency-specific changes of FC pattern in EOS remain unclear. To address this issue, resting-state functional magnetic resonance imaging data scans from 39 EOS patients (drug-naive) and 31 healthy controls (HCs) were used to assess the FC density (FCD) across slow-4 (0.027–0.073 Hz) and slow-5 (0.01–0.027 Hz). Results revealed that a remarkable difference between the FCD of the two bands existed mainly in the default mode network (DMN) and subcortical areas. Compared with the HCs, EOS patients showed significantly altered FCD involved in audiovisual information processing, sensorimotor system, and social cognition. Importantly, a significant frequency-by-group interaction was observed in the left precuneus with significantly lower FCD in the slow-4 frequency band, but no significant effect in the slow-5 frequency band. In addition, decreased FC was found between the precuneus and other DMN regions in the slow-4 band. Furthermore, the change in FCD in precuneus was inversely proportional to the clinical symptom in slow-4 band, indicating the key role of precuneus in schizophrenia progress. Our findings demonstrated that the dysconnectivity pattern in EOS could be frequency-dependent.

Attention to affective audio-visual information: Comparison between musicians and non-musicians

Individuals with more musical training repeatedly demonstrate enhanced auditory perception abilities. The current study examined how these enhanced auditory skills interact with attention to affective audio-visual stimuli. A total of 16 participants with more than 5 years of musical training (musician group) and 16 participants with less than 2 years of musical training (non-musician group) took part in a version of the audio-visual emotional Stroop test, using happy, neutral, and sad emotions. Participants were presented with congruent and incongruent combinations of face and voice stimuli while judging the emotion of either the face or the voice. As predicted, musicians were less susceptible to interference from visual information on auditory emotion judgments than non-musicians, as evidenced by musicians being more accurate when judging auditory emotions when presented with congruent and incongruent visual information. Musicians were also more accurate than non-musicians at identifying visual emotions when presented with concurrent auditory information. Thus, musicians were less influenced by congruent/incongruent information in a non-target modality compared to non-musicians. The results suggest that musical training influences audio-visual information processing.

Instrument-Specific Effects of Musical Expertise on Audiovisual Processing (Clarinet vs. Violin)

A large body of evidence has shown that musicians’ brains differ in many ways from nonmusicians’ brains due to the particularly intense and prolonged sensorimotor training involved. Not much is known about the effects of the specific musical instrument played on brain processing of audiovisual information. In this study the effect of musical expertise was investigated in professional clarinetists and violinists. One hundred and eighty videos showing fragments of musical performances played on a violin or a clarinet were presented to musicians of G. Verdi Milan Conservatory and age-matched controls. Half of the musicians were violinists, the other half were clarinetists; event-related potentials (ERPs) were recorded from 128 scalp sites and analyzed. Participants judged how many notes were played in each clip. The task was extremely easy for all participants. Over prefrontal areas an anterior negativity response was found to be much larger in controls than in musicians, and in musicians for the unfamiliar over the familiar musical instrument. Furthermore, a later central negativity response showed a lack of note numerosity effect in the brains of musicians for their own instrument, but not for unfamiliar instrument. The data indicate that music training is instrument-specific and that it profoundly affects prefrontal encoding of music-related information and auditory processing.

Gender Differences in Cerebral Processing of Audio-Visual Information

Background: In daily life, visual information plays an important role to improve accuracy of auditory perception, and such phenomenon might be explained by audio-visual integration. However, audio-visual integration differs between genders, i.e., female tends to perceive voice with mouth movement more accurately. In this study, we examined gender differences of cerebral processing of audio-visual information. Methods: Cerebral processing was examined in male and female participants under different audio-visual conditions: 1) Auditory (A) condition: presentation of white-noise (noise) or a vowel (/a/ or /i/) pronounced by a male; 2) Visual (V) condition: presentation of a face image pronouncing a vowel ([a] or [i]), 3) Audio-Visual (AV) condition: simultaneous presentation of auditory (noise or vowel) and visual stimuli. Participants were asked to press a button when the vowel /a/ or noise was presented in the A or AV condition. Subtracted auditory ERPs in the AV condition (congruent: /a/ with [a]-[a] and incongruent: /a/ with [i]-[i]) were compared to ERPs in the A condition (auditory /a/). Results: In the AV condition, amplitude of auditory P2 in the male participants was smaller than that in the females, whether the stimulus set was congruent or incongruent. The lower P2 amplitude in males may have been caused by overlapping of negative components, which may be related to selective attention. Subtracted visual ERPs in the AV condition (congruent: [a] with /a/-/a/ and incongruent: [a] with /i/-/i/) were compared to ERPs in the V condition (visual [a]). Amplitude of visual N170 to congruent stimuli was smaller than that for alone visual stimuli only for the females. Conclusion: Lower amplitude of the N170 may reflect enhanced facial processing in females. Gender differences in audio-visual information processing were discussed. Keywords: Audio-visual integration, event-related potential, gender difference, N170.

Impaired timing of audiovisual events in the elderly.

Perceptual binding of multisensory events occurs within a limited time span known as the temporal binding window. Failure to correctly identify whether multisensory events occur simultaneously, what their temporal order is, or whether they should be causally bound can lead to inaccurate representations of the physical world, poor decision-making, and dangerous behavior. It has been shown that the ability to discriminate simultaneity, temporal order, and causal relationships among stimuli can become increasingly difficult as we age. In the present study, we assessed the relationship between these three attributes of temporally processing multisensory information in both younger and older adults. Performance on three tasks (temporal order judgment: TOJ, simultaneity judgment: SJ, and stream/bounce illusion) was compared using a large sample within-subjects design consisting of younger and older adults to determine aging effects as well as relationships between the three tasks. Older adults had more difficulty (larger temporal binding window) discriminating temporal order and perceived collision than younger adults. Simultaneity judgments in younger and older adults were indistinguishable. Positive correlations between TOJ and SJ as well as SJ and stream/bounce tasks were found in younger adults, which identify common (SJ) and distinct (TOJ, stream/bounce) neural mechanisms that sub-serve temporal processing of audiovisual information that is lost in older adults. We conclude that older adults have an extended temporal binding window for TOJ and stream/bounce tasks, but the temporal binding window in SJ is preserved, suggesting that age-related changes in multisensory integration are task specific and not a general trait of aging.

A New Quadtree Histogram-based Spatial Modeling based on Cloud Model

Spatial modeling is one of the most important parts in the area of spatial relations, and it is necessary to develop a new model with high practicality. Firstly this paper analyzes the characteristics and defects of present models; secondly introduces in detail the construction method of quadtree histogram; thirdly cloud model are used to judge spatial directional relation. The experimental results show that the model is feasible. Computer vision is concerned with automatic extraction of useful information from images. The description for human space perception is an important task of the computer vision research. The description of spatial relation is a basic ability of human cognition, it is an important basic task for computer audio-visual information processing to build automatically the description of spatial relation between objects in the image. Spatial relation is mainly constituted of directional relation, topological relation and distance relation. The fundamental theories of topology relation and distance relation are comparatively perfect, but the fundamental theories of directional relation have not any unified standard. There are three main approaches for assessing the basic directional relation. The first one is based on cone model (1) which assimilates an object to the centroid, but it doesn't take into account shape, size and distance information. The second one is based on projection model such as minimum bounding rectangles (2), and directional relation matrix (3), etc., but these models assimilate an object to the minimum bounding rectangle, so shape, size and distance information also can't be taken into account. The third one is based on angle-histogram model (4). Given an object of target A and a reference object B, the angle-histogram is computed from the angles between any two points in both objects and normalized by the maximum frequency. This histogram represents the directional relations of the object A with respect to the reference object B. This model has taken into account shape, size and distance information. But this model has high computational complexity, so it is not practical. Numerous studies are based on this notion of angle-histogram, e.g., (5-8). In (5-7), R-histogram, R*-histogram and F-histogram are proposed, and they can reduce the computational complexity. In (8), a quadtree histogram model of spatial directional relation is proposed, and it can get high accuracy and small computational complexity together.

Processing of audiovisual stimuli in aphasic and non-brain-damaged listeners

Background: During speech perception not only auditory but also visual information (seen speech) is processed. This was shown by, for example, McGurk and MacDonald (1976). In their study participants watched videos showing a speaker articulating the syllable /ka/, while the sound of /pa/ was dubbed onto the video. The most commonly reported perception was neither of the two syllables, but a fusion of both (/ta/). This effect of audiovisual integration has been called the “McGurk effect”. Klitsch (2008) showed that the McGurk effect was also found for participants with aphasia and its strength was comparable to an age-matched control group. However her task was an offline measure and therefore provided limited information about the processing itself. Aims: In this study reaction time data will be analysed in addition to answer types. By this we aim to find out whether there are qualitative differences between healthy and aphasic processing, shown by different patterns in the reaction times. Methods & Procedures: Three aphasic participants and a group of fourteen non-brain-damaged control participants took part in a nonword identification task in which they were presented with a speaker pronouncing a syllable. Afterwards they had to choose the matching syllable from three written options. This task was carried out in four conditions: “auditory only”, “visual only” (articulatory movements), “audiovisual”, and “McGurk” (incongruent auditory and audiovisual information). Along with the answer types, the reaction times were also recorded for all participants. Outcomes & Results: The aphasic participants made more errors and were slower than the non-brain-damaged control participants across conditions. Within the McGurk condition we found different patterns for the non-brain-damaged and the aphasic participants: reaction times were increased for the control group whenever a McGurk-type answer was chosen (compared with answers representing the auditory or visual part of the stimulus), while there were no influences of the chosen answer type on the reaction times for the aphasic participants. Conclusions: Processing of audiovisual information was not only slower in three aphasic participants but also differed qualitatively from the non-brain-damaged control group. For the control participants reaction times differed depending on what answer type was chosen in the McGurk condition. This was not the case for either of the aphasic participants. We argue that the slow-down in undisturbed processing is caused by conscious access to the unimodal information, which might be lacking for the aphasic participants.

OS7 交通安全における人間・機械系の視点

Traffic safety is a very important society problem. In order to solve this problem, we can investigate the traffic accident from three sides of the car, the traffic environment and human. This paper addresses a new study approach of traffic safety from the viewpoint of man-machine system. The investigation of the human mechanism on visual and auditory interactive information processing is very important for safe uses of multimedia machines and driver. In this study, (1) human characteristics of the audio-visual information processing are investigated by functional magnetic resonance imaging (fMRI), (2) a virtual driver system and (3) a portable eyesight test system are proposed.

Output, error, equivocation, and recalled information in auditory, visual, and audiovisual information processing with constraint and noise.

Past controversies over the relative effectiveness of auditory, visual, and audiovisual channels are seen as the results of nondifferentiation of error and equivocation. Seventh grade students were used as Ss in an experiment consisting of noise and no noise conditions with constrained and nonconstrained communication in A, V, and AV treatments. Data were examined in terms of output, error, equivocation, and recalled (shared) information based upon information theory. The superiority of the AV treatment was substantiated in all respects of information processing, making less error and equivocation, but recalling more information correctly as compared with the A or V channel. Comparisons between A and V unequivocally established the fact that V made less error but more equivocation, whereas A processed more output information and also made more error; however, there was no significant difference between them in recalled information. The dependence of A upon constraint was found to be far greater than that of AV whose dependence in turn was greater than that of V. Noise was found to affect A most, V the next most, and AV the least. With between-channel redundancy AV seemed to be capable of reducing the effect of noise.

Comments on Hsia's Auditory, Visual, and Audiovisual Information Processing

Comments on Hsia's Auditory, Visual, and Audiovisual Information Processing

Effects of noise and difficulty level of input information in auditory, visual, and audiovisual information processing.

To examine the effects of noise in auditory (A), visual (V), and audiovisual (AV) channels with varying difficulty levels of input information, 192 seventh grade Ss were tested under six conditions: A, V, and AV with and without noise. It was determined that noise had no detrimental effects upon information processing when input was not difficult, only A was significantly deteriorated by the presence of noise, whereas noise seemed to alert Ss in the AV treatment, AV was generally found to be better than A, which in turn was generally better than V, channel difference vanished when input was very difficult.