Audiovisual Objects Research Articles

Enhanced attention to speaking faces versus other event types emerges gradually across infancy.

The development of attention to dynamic faces versus objects providing synchronous audiovisual versus silent visual stimulation was assessed in a large sample of infants. Maintaining attention to the faces and voices of people speaking is critical for perceptual, cognitive, social, and language development. However, no studies have systematically assessed when, if, or how attention to speaking faces emerges and changes across infancy. Two measures of attention maintenance, habituation time (HT) and look-away rate (LAR), were derived from cross-sectional data of 2- to 8-month-old infants (N = 801). Results indicated that attention to audiovisual faces and voices was maintained across age, whereas attention to each of the other event types (audiovisual objects, silent dynamic faces, silent dynamic objects) declined across age. This reveals a gradually emerging advantage in attention maintenance (longer HTs, lower LARs) for audiovisual speaking faces compared with the other 3 event types. At 2 months, infants showed no attentional advantage for faces (with greater attention to audiovisual than to visual events); at 3 months, they attended more to dynamic faces than objects (in the presence or absence of voices), and by 4 to 5 and 6 to 8 months, significantly greater attention emerged to temporally coordinated faces and voices of people speaking compared with all other event types. Our results indicate that selective attention to coordinated faces and voices over other event types emerges gradually across infancy, likely as a function of experience with multimodal, redundant stimulation from person and object events. (PsycINFO Database Record

Are the literacy difficulties that characterize developmental dyslexia associated with a failure to integrate letters and speech sounds?

The ‘automatic letter‐sound integration hypothesis’ (Blomert, 2011) proposes that dyslexia results from a failure to fully integrate letters and speech sounds into automated audio‐visual objects. We tested this hypothesis in a sample of English‐speaking children with dyslexic difficulties (N = 13) and samples of chronological‐age‐matched (CA; N = 17) and reading‐age‐matched controls (RA; N = 17) aged 7–13 years. Each child took part in two priming experiments in which speech sounds were preceded by congruent visual letters (congruent condition) or Greek letters (baseline). In a behavioural experiment, responses to speech sounds in the two conditions were compared using reaction times. These data revealed faster reaction times in the congruent condition in all three groups. In a second electrophysiological experiment, responses to speech sounds in the two conditions were compared using event‐related potentials (ERPs). These data revealed a significant effect of congruency on (1) the P1 ERP over left frontal electrodes in the CA group and over fronto‐central electrodes in the dyslexic group and (2) the P2 ERP in the dyslexic and RA control groups. These findings suggest that our sample of English‐speaking children with dyslexic difficulties demonstrate a degree of letter‐sound integration that is appropriate for their reading level, which challenges the letter‐sound integration hypothesis.

Selective Audiovisual Semantic Integration Enabled by Feature-Selective Attention.

An audiovisual object may contain multiple semantic features, such as the gender and emotional features of the speaker. Feature-selective attention and audiovisual semantic integration are two brain functions involved in the recognition of audiovisual objects. Humans often selectively attend to one or several features while ignoring the other features of an audiovisual object. Meanwhile, the human brain integrates semantic information from the visual and auditory modalities. However, how these two brain functions correlate with each other remains to be elucidated. In this functional magnetic resonance imaging (fMRI) study, we explored the neural mechanism by which feature-selective attention modulates audiovisual semantic integration. During the fMRI experiment, the subjects were presented with visual-only, auditory-only, or audiovisual dynamical facial stimuli and performed several feature-selective attention tasks. Our results revealed that a distribution of areas, including heteromodal areas and brain areas encoding attended features, may be involved in audiovisual semantic integration. Through feature-selective attention, the human brain may selectively integrate audiovisual semantic information from attended features by enhancing functional connectivity and thus regulating information flows from heteromodal areas to brain areas encoding the attended features.

Bayesian-based integration of multisensory naturalistic perithreshold stimuli

Most studies exploring multisensory integration have used clearly perceivable stimuli. According to the principle of inverse effectiveness, the added neural and behavioral benefit of integrating clear stimuli is reduced in comparison to stimuli with degraded and less salient unisensory information. Traditionally, speed and accuracy measures have been analyzed separately with few studies merging these to gain an understanding of speed-accuracy trade-offs in multisensory integration.In two separate experiments, we assessed multisensory integration of naturalistic audio-visual objects consisting of individually-tailored perithreshold dynamic visual and auditory stimuli, presented within a multiple-choice task, using a Bayesian Hierarchical Drift Diffusion Model that combines response time and accuracy. For both experiments, unisensory stimuli were degraded to reach a 75% identification accuracy level for all individuals and stimuli to promote multisensory binding. In Experiment 1, we subsequently presented uni- and their respective bimodal stimuli followed by a 5-alternative-forced-choice task. In Experiment 2, we controlled for low-level integration and attentional differences.Both experiments demonstrated significant superadditive multisensory integration of bimodal perithreshold dynamic information. We present evidence that the use of degraded sensory stimuli may provide a link between previous findings of inverse effectiveness on a single neuron level and overt behavior. We further suggest that a combined measure of accuracy and reaction time may be a more valid and holistic approach of studying multisensory integration and propose the application of drift diffusion models for studying behavioral correlates as well as brain-behavior relationships of multisensory integration.

Interacting parallel pathways associate sounds with visual identity in auditory cortices

Spatial and non-spatial information of sound events is presumably processed in parallel auditory cortex (AC) “what” and “where” streams, which are modulated by inputs from the respective visual-cortex subsystems. How these parallel processes are integrated to perceptual objects that remain stable across time and the source agent's movements is unknown. We recorded magneto- and electroencephalography (MEG/EEG) data while subjects viewed animated video clips featuring two audiovisual objects, a black cat and a gray cat. Adaptor-probe events were either linked to the same object (the black cat meowed twice in a row in the same location) or included a visually conveyed identity change (the black and then the gray cat meowed with identical voices in the same location). In addition to effects in visual (including fusiform, middle temporal or MT areas) and frontoparietal association areas, the visually conveyed object-identity change was associated with a release from adaptation of early (50–150ms) activity in posterior ACs, spreading to left anterior ACs at 250–450ms in our combined MEG/EEG source estimates. Repetition of events belonging to the same object resulted in increased theta-band (4–8Hz) synchronization within the “what” and “where” pathways (e.g., between anterior AC and fusiform areas). In contrast, the visually conveyed identity changes resulted in distributed synchronization at higher frequencies (alpha and beta bands, 8–32Hz) across different auditory, visual, and association areas. The results suggest that sound events become initially linked to perceptual objects in posterior AC, followed by modulations of representations in anterior AC. Hierarchical what and where pathways seem to operate in parallel after repeating audiovisual associations, whereas the resetting of such associations engages a distributed network across auditory, visual, and multisensory areas.

Audio-visual object search is changed by bilingual experience.

The current study examined the impact of language experience on the ability to efficiently search for objects in the face of distractions. Monolingual and bilingual participants completed an ecologically-valid, object-finding task that contained conflicting, consistent, or neutral auditory cues. Bilinguals were faster than monolinguals at locating the target item, and eye movements revealed that this speed advantage was driven by bilinguals' ability to overcome interference from visual distractors and focus their attention on the relevant object. Bilinguals fixated the target object more often than did their monolingual peers, who, in contrast, attended more to a distracting image. Moreover, bilinguals', but not monolinguals', object-finding ability was positively associated with their executive control ability. We conclude that bilinguals' executive control advantages extend to real-world visual processing and object finding within a multi-modal environment.

Detecting Functional Connectivity During Audiovisual Integration with MEG: A Comparison of Connectivity Metrics.

In typical magnetoencephalography and/or electroencephalography functional connectivity analysis, researchers select one of several methods that measure a relationship between regions to determine connectivity, such as coherence, power correlations, and others. However, it is largely unknown if some are more suited than others for various types of investigations. In this study, the authors investigate seven connectivity metrics to evaluate which, if any, are sensitive to audiovisual integration by contrasting connectivity when tracking an audiovisual object versus connectivity when tracking a visual object uncorrelated with the auditory stimulus. The authors are able to assess the metrics' performances at detecting audiovisual integration by investigating connectivity between auditory and visual areas. Critically, the authors perform their investigation on a whole-cortex all-to-all mapping, avoiding confounds introduced in seed selection. The authors find that amplitude-based connectivity measures in the beta band detect strong connections between visual and auditory areas during audiovisual integration, specifically between V4/V5 and auditory cortices in the right hemisphere. Conversely, phase-based connectivity measures in the beta band as well as phase and power measures in alpha, gamma, and theta do not show connectivity between audiovisual areas. The authors postulate that while beta power correlations detect audiovisual integration in the current experimental context, it may not always be the best measure to detect connectivity. Instead, it is likely that the brain utilizes a variety of mechanisms in neuronal communication that may produce differential types of temporal relationships.

Developmental trends in the facilitation of multisensory objects with distractors.

Sensory integration and the ability to discriminate target objects from distractors are critical to survival, yet the developmental trajectories of these abilities are unknown. This study investigated developmental changes in 9- (n = 18) and 11-year-old (n = 20) children, adolescents (n = 19) and adults (n = 22) using an audiovisual object discrimination task with uni- and multisensory distractors. Reaction times (RTs) were slower with visual/audiovisual distractors, and although all groups demonstrated facilitation of multisensory RTs in these conditions, children's and adolescents' responses corresponded to fewer race model violations than adults', suggesting protracted maturation of multisensory processes. Multisensory facilitation could not be explained by changes in RT variability, suggesting that tests of race model violations may still have theoretical value at least for familiar multisensory stimuli.

Task-Specific, Age Related Effects in the Cross-Modal Identification and Localisation of Objects.

We investigated age-related effects in cross-modal interactions using tasks assessing spatial perception and object perception. Specifically, an audio-visual object identification task and an audio-visual object localisation task were used to assess putatively distinct perceptual functions in four age groups: children (8-11 years), adolescents (12-14 years), young and older adults. Participants were required to either identify or locate target objects. Targets were specified as unisensory (visual/auditory) or multisensory (audio-visual congruent/audio-visual incongruent) stimuli. We found age-related effects in performance across both tasks. Both children and older adults were less accurate at locating objects than adolescents or young adults. Children were also less accurate at identifying objects relative to young adults, but the performance between young adults, adolescents and older adults did not differ. A greater cost in accuracy for audio-visual incongruent relative to audio-visual congruent targets was found for older adults, children and adolescents relative to young adults. However, we failed to find a benefit in performance for any age group in either the identification or localisation task for audio-visual congruent targets relative to visual-only targets. Our findings suggest that visual information dominated when identifying or localising audio-visual stimuli. Furthermore, on the basis of our results, object identification and object localisation abilities seem to mature late in development and that spatial abilities may be more prone to decline as we age relative to object identification abilities. In addition, the results suggest that multisensory facilitation may require more sensitive measures to reveal differences in cross-modal interactions across higher-level perceptual tasks.

Vision-guided robot hearing

Natural human–robot interaction (HRI) in complex and unpredictable environments is important with many potential applications. While vision-based HRI has been thoroughly investigated, robot hearing and audio-based HRI are emerging research topics in robotics. In typical real-world scenarios, humans are at some distance from the robot and, hence, the sensory (microphone) data are strongly impaired by background noise, reverberations and competing auditory sources. In this context, the detection and localization of speakers plays a key role that enables several tasks, such as improving the signal-to-noise ratio for speech recognition, speaker recognition, speaker tracking, etc. In this paper we address the problem of how to detect and localize people that are both seen and heard. We introduce a hybrid deterministic/probabilistic model. The deterministic component allows us to map 3D visual data onto a 1D auditory space. The probabilistic component of the model enables the visual features to guide the grouping of the auditory features in order to form audiovisual (AV) objects. The proposed model and the associated algorithms are implemented in real-time (17 FPS) using a stereoscopic camera pair and two microphones embedded into the head of the humanoid robot NAO. We perform experiments with (i) synthetic data, (ii) publicly available data gathered with an audiovisual robotic head, and (iii) data acquired using the NAO robot. The results validate the approach and are an encouragement to investigate how vision and hearing could be further combined for robust HRI.

141. Audiovisual integration effect in lateral occipital cortex during an object recognition task: A TMS study

Perceptual events occur in multiple sensory modalities. Recent neuroimaging data challenge the traditional models arguing for sensory integration to occur in higher-order association cortices and support the hypothesis of a multisensory interplay at low-level sensory-specific cortex. We used an on-line interference approach by rTMS to investigate the role of the left lateral occipital cortex (LOC) in audio-visual (AV) object recognition process. Fifteen healthy volunteers performed a visual identification task of degraded pictures (living and non-living) presented alone (V) or simultaneously to coherent (c-AV) or non-coherent (nc-AV) sounds. Focal 10-Hz rTMS at an intensity of 100% resting motor threshold was delivered simultaneously to the picture (500 ms). Two blocks of 60 pictures were randomly displayed in two different experimental conditions: rTMS of the left LOC and over Cz (cortical control area). Accuracy and reaction time of identification were analyzed. rTMS of the left LOC produced a worsening of the accuracy compared to rTMS over Cz specifically in the c-AV condition. Interestingly, the effect was significant for stimuli belonging to the living semantic category. These data support the view that audiovisual interaction effect may occur at early stage of recognition processing.

A one-shot domain-independent robust multimedia clustering methodology based on hybrid multimodal fusion

The existence of multiple modalities poses a challenge to the design of multimedia data clustering systems, as the unsupervised nature of the problem makes it very difficult to determine a priori whether a single modality should dominate the clustering process, or if modalities should be combined somehow. In order to fight against these indeterminacies--which come on top of those referring to the selection of the optimal clustering algorithm and data representation for the problem at hand---, this work introduces robust multimedia clustering, a one-shot methodology for domain independent multimedia data clustering based on hybrid multimodal fusion. By means of experimentation, we firstly justify the motivation of the proposed methodology by proving the relevance of multimedia clustering indeterminacies. Subsequently, a specific multimedia clustering system based on the requirements of the methodology is implemented and evaluated on three multimedia clustering applications--music genres, photographic topics and audio-visual objects classification--as a proof of concept, analyzing the quality of the obtained partitions and the time complexity of the proposal. The experimental results reveal that the implemented system, which includes a self-refining consensus clustering procedure for attaining high levels of robustness, allows to obtain, in a fully unsupervised manner, better quality partitions than 93 % of the clusterers available in our experiments, being even able to improve the quality of the best ones and outperforming state-of-the-art alternatives.

Cueing multimedia search with audiovisual blur

Situated in the context of multimedia browsing, this study concerns perceptual processes involved in searching for an audiovisual object displayed among several distractors. The aim of the study is to increase the perceptual saliency of the target in order to enhance the search process. As blurring distractors and maintaining the target sharp has proved to be a great facilitator of visual search, we propose combining visual blur with an audio blur analogue to improve multimodal search. Three perceptual experiments were performed in which participants had to retrieve an audiovisual object from a set of six competing stimuli. The first two experiments explored the effect of blur level on unimodal search tasks. A third experiment investigated the influence of an audio and visual modality combination with both modalities cued on an audiovisual search task. Results showed that both visual and audio blurs render stimuli distractors less prominent and thus helped users focus on a sharp target more easily. Performances were also faster and more accurate in the bimodal condition than in either unimodal search task, auditory or visual. Our work suggests that audio and audiovisual interfaces dedicated to multimedia search could benefit from different uses of blur on presentation strategies.

Audiovisual Temporal Recalibration: The Combined and Separate Effect of Adapted Space and Object

Previous researches have revealed that temporal recalibration arises from perceptual change rather than cognitive shift. And such a PSS(the point of subjective simultaneity) shift might coincide with changes in perceptual latency of a certain modality. Given the rich and variable environments we live in, multisensory events might not always appear in the central field as considerable experiments have done. There is no doubt that they could simultaneously or successively appear in the lateral space. If the participants were repeatedly exposed to asynchrony audiovisual pairs presented in lateral space (left or right), will the aftereffects only be expected in the adapted space? A recent study has shown that temporal recalibration was spatial-specific (Hanson, Roach, Heron, McGraw, 2008), but can certain audio-visual(abbreviated to AV) object play some roles in temporal recalibration? Now we found a new way to test the combined effect of adapted space and object, and to separate the two individual effects. To the end, two experiments were designed. Each experiment was comprised of an exposed and a test phase. In the exposed phase of the first experiment, certain asynchrony audiovisual stimulus pairs were repeatedly displayed in the left or right space with a fixation in the screen center on which participants were required to freeze their eyes throughout the whole experiment. While in the test phase, we presented the adapted AV pairs in the adapted space but novel AV pairs in the opposite space. In the second experiment, the procedure was the same with the first one except that we put the adapted AV pairs in the opposite space and the novel ones in the adapted space. Through these two experiments, we first aim to replicate the spatial-specific findings when adapted AV pairs (object) recurred in the adapted space, then to explore whether the separate effect of adapted object and adapted space would induce temporal recalibration as well. In Experiment 1, after repeated exposure to 240 trials with constant temporal lag in a block, either with auditory stimulus leading 128ms or visual stimulus leading 128ms, the participants shifted their PSS towards the leading modality, only in the adapted space. In Experiment 2, if the temporal recalibration were restricted to the combined effect of adapted space and object, there would be no observed PSS shifts. But violating such inference, the participants still showed significant PSS shifts in the adapted AV pairs and approaching-significant PSS shifts in the adapted space, although the magnitude was no so great as in Experiment 1. Comparing the results of the two experiments, we concluded that when adapted object kept its position across the exposed and test phase, audiovisual temporal recalibration showed as specific, in other words, aftereffects only exists in the adapted space or adapted object; but when separating the effect of space and object, temporal recalibration was neither spatial- nor object-specific. That is to say if the adapted stimulus pairs changes its location after exposure, wherever it stops, aftereffects would be subsequently induced in that new space, while in the adapted space aftereffects stills remains although novel object replaced the adapted one. The outcome of temporal recalibration then ought to be characterized as the separate effect of adapted space and adapted object, or the combined effect of both.

The Museum of All: Institutional Communication Practices in a Participatory Networked World

ABSTRACTIn this paper we will maintain that cultural institutions are increasingly facing the inevitability of a profound revision in their traditional parameters of unidirectional communication. Given the increasing availability of tools for audiovisual production as well as the diversity of networked communication contexts, the roles of the user and the audience have come to assume a participatory potential in the content they consume and attend; this will dictate their repositioning in relation to the universe of institutions' work.In order to communicate new messages, media narratives, views or perspectives on the same reality are constantly constructed, deconstructed and reconstructed. In this age of ‘ConsumerGenerated Media’, identities and representations are constantly changing. The idea and the concept of a brand now extends far beyond its own limits, taking on important emotional values, and people enjoy being a part of that brand's mythology.The Serralves Foundation, along with its Museum of Contemporary Art, in Porto, Portugal, was the subject of a first study of an empirical nature: a series of audio-visual objects were developed, in order to generate material for analysis and proposition.In this subsequent stage, we aim at the identification of new procedures and practices that may be effectively implemented within the institutional universe. We intend to establish more efficient communication contexts, including the maximizing of a set of relationships between institutions and audiences regarding dimensions that are traditionally outside the institutional radar: identity, narrative and affection.By referring to key examples, we will posit that in order to achieve more success in their missions, cultural institutions should focus on participatory education and communication processes regarding contemporary issues. By doing this, they will strengthen their engagement with their audiences and bring them into the institutions' projects.This project is currently undertaking a wide inventory and taxonomy of the various institutional paradigms in regards to communication practices. We aim at producing a map of vocations and positions within the institutional world in regards to the aforementioned issues that require participatory communication.

Redundancy gains in audio–visual search

This study concerns stimuli-driven perceptual processes involved in target search among concurrent distractors with a focus on comparing auditory, visual, and audio–visual search tasks. Previous works, concerning unimodal search tasks, highlighted different preattentive features that can enhance target saliency, making it ‘pop-out’, e.g., a visually sharp target among blurred distractors. A cue from another modality can also help direct attention towards the target. Our study investigates a new kind of search task, where stimuli consist of audio–visual objects presented using both audio and visual modalities simultaneously. Redundancy effects are evaluated, first from the combination of audio and visual modalities, second from the combination of each unimodal cue in such a bimodal search task. A perceptual experiment was performed where the task was to identify an audio–visual object from a set of six competing stimuli. We employed static visual blur and developed an auditory blur analogue to cue the search. Results show that both visual and auditory blurs render distractors less prominent and automatically attracts attention toward a sharp target. The combination of both unimodal blurs, i.e., audio–visual blur, also proved to be an efficient cue to facilitate bimodal search task. Results also showed that search tasks were performed faster in redundant bimodal conditions than in unimodal ones. That gain was due to redundant target effect only without any redundancy gain due to the cue combination, as solely cueing the visual component was sufficient, with no improvement found by the addition of the redundant audio cue in bimodal search tasks.

Mapping Symbols to Sounds: Electrophysiological Correlates of the Impaired Reading Process in Dyslexia

Dyslexic and control first-grade school children were compared in a Symbol-to-Sound matching test based on a non-linguistic audiovisual training which is known to have a remediating effect on dyslexia. Visual symbol patterns had to be matched with predicted sound patterns. Sounds incongruent with the corresponding visual symbol (thus not matching the prediction) elicited the N2b and P3a event-related potential (ERP) components relative to congruent sounds in control children. Their ERPs resembled the ERP effects previously reported for healthy adults with this paradigm. In dyslexic children, N2b onset latency was delayed and its amplitude significantly reduced over left hemisphere whereas P3a was absent. Moreover, N2b amplitudes significantly correlated with the reading skills. ERPs to sound changes in a control condition were unaffected. In addition, correctly predicted sounds, that is, sounds that are congruent with the visual symbol, elicited an early induced auditory gamma band response (GBR) reflecting synchronization of brain activity in normal-reading children as previously observed in healthy adults. However, dyslexic children showed no GBR. This indicates that visual symbolic and auditory sensory information are not integrated into a unitary audiovisual object representation in them. Finally, incongruent sounds were followed by a later desynchronization of brain activity in the gamma band in both groups. This desynchronization was significantly larger in dyslexic children. Although both groups accomplished the task successfully remarkable group differences in brain responses suggest that normal-reading children and dyslexic children recruit (partly) different brain mechanisms when solving the task. We propose that abnormal ERPs and GBRs in dyslexic readers indicate a deficit resulting in a widespread impairment in processing and integrating auditory and visual information and contributing to the reading impairment in dyslexia.

Developmental processes in audiovisual object recognition and object location

This study investigated whether performance in recognising and locating target objects benefited from the simultaneous presentation of a crossmodal cue. Furthermore, we examined whether these ‘what’ and ‘where’ tasks were affected by developmental processes by testing across different age groups. Using the same set of stimuli, participants conducted either an object recognition task, or object location task. For the recognition task, participants were required to respond to two of four target objects (animals) and withhold response to the remaining two objects. For the location task, participants responded when an object occupied either of two target locations and withheld response if the object occupied a different location. Target stimuli were presented either by vision alone, audition alone, or bimodally. In both tasks cross-modal cues were either congruent or incongruent. The results revealed that response time performance in both the object recognition task and in the object location task benefited from the presence of a congruent cross-modal cue, relative to incongruent or unisensory conditions. In the younger adult group, the effect was strongest for response times although the same pattern was found for accuracy in the object location task but not for the recognition task. Following recent studies on multisensory integration in children (e.g., Brandwein, 2010; Gori, 2008), we then tested performance in children (i.e., 8–14 year olds) using the same task. Although overall performance was affected by age, our findings suggest interesting parallels in the benefit of congruent, cross-modal cues between children and adults, for both object recognition and location tasks.

Are the senses enough for sense? Early high-level feedback shapes our comprehension of multisensory objects.

A key question in cognitive neuroscience is how the brain combines low-level features processed in remote sensory cortices to represent meaningful multisensory objects in our everyday environment. Models of visual object processing typically assume a feedforward cascade through the hierarchically organized ventral stream. We contrasted this feedforward view with an alternate hypothesis in which object processing is viewed as an interactive, feedforward and feedback process. We found that higher-order regions in anterior temporal (AT) and inferior prefrontal cortex (IPC) performed audio-visual (AV) integration 100 ms earlier than a sensory-driven region in the posterior occipital (pO) cortex, and were modulated by semantic variables (congruency), from as early as 50–100 ms. We propose that the brain represents familiar and complex multisensory objects through early interactivity between higher-order and sensory-driven regions. This interactivity may underpin the enhanced behavioral performance reported for semantically congruent AV objects.

The spatiotemporal characteristics of elementary audiovisual speech and music processing in musically untrained subjects

Previously, the EEG technique has been used to investigate the spatiotemporal properties of audiovisual (AV) processing by taking advantage of the violation of the “additive model”, which is considered to be a very conservative approach. In the present work, we used a less conservative and novel approach than the criterion of superadditivity for estimating AV interactions. Hence, we estimated AV interaction patterns by comparing the responses to AV stimuli with the averaged responses to the unimodal visual and auditory stimuli in musically untrained subjects and by presenting syllables and piano tones coupled with flashlights. Our results suggest that the two AV objects elicited consistent interaction patterns within the time course of unisensory processing in the time range between 80 and 250ms post stimulus onset. The scalp topographies, as well as the source estimation approach we adopted, indicate that the first interaction pattern at around 100ms was partially driven by auditory-related cortical regions. Additionally, we found evidence for a second interaction pattern at around 200ms that was mainly associated with the responsiveness of extra-sensory brain regions. During this later processing stage, only the music condition was associated with putative responses that originated from auditory-related cortical fields. This study provides a novel approach to investigate the basic principles underlying elementary AV speech and music processing in subjects without formal musical education.