Audiovisual Interactions Research Articles

Neural correlates of audiovisual integration of semantic category information

Previous studies have found a late frontal–central audiovisual interaction during the time period about 150–220ms post-stimulus. However, it is unclear to which process is this audiovisual interaction related: to processing of acoustical features or to classification of stimuli? To investigate this question, event-related potentials were recorded during a words-categorization task with stimuli presented in the auditory–visual modality. In the experiment, congruency of the visual and auditory stimuli was manipulated. Results showed that within the window of about 180–210ms post-stimulus more positive values were elicited by category-congruent audiovisual stimuli than category-incongruent audiovisual stimuli. This indicates that the late frontal–central audiovisual interaction is related to audiovisual integration of semantic category information.

Steady-state responses in MEG demonstrate information integration within but not across the auditory and visual senses

To form a unified percept of our environment, the human brain integrates information within and across the senses. This MEG study investigated interactions within and between sensory modalities using a frequency analysis of steady-state responses that are elicited time-locked to periodically modulated stimuli. Critically, in the frequency domain, interactions between sensory signals are indexed by crossmodulation terms (i.e. the sums and differences of the fundamental frequencies). The 3 × 2 factorial design, manipulated (1) modality: auditory, visual or audiovisual (2) steady-state modulation: the auditory and visual signals were modulated only in one sensory feature (e.g. visual gratings modulated in luminance at 6 Hz) or in two features (e.g. tones modulated in frequency at 40 Hz & amplitude at 0.2 Hz). This design enabled us to investigate crossmodulation frequencies that are elicited when two stimulus features are modulated concurrently (i) in one sensory modality or (ii) in auditory and visual modalities. In support of within-modality integration, we reliably identified crossmodulation frequencies when two stimulus features in one sensory modality were modulated at different frequencies. In contrast, no crossmodulation frequencies were identified when information needed to be combined from auditory and visual modalities. The absence of audiovisual crossmodulation frequencies suggests that the previously reported audiovisual interactions in primary sensory areas may mediate low level spatiotemporal coincidence detection that is prominent for stimulus transients but less relevant for sustained SSR responses. In conclusion, our results indicate that information in SSRs is integrated over multiple time scales within but not across sensory modalities at the primary cortical level.

Auditory effects to complement the absence of visual information

The majority of research on audio–visual interaction focused on spatio-temporal factors and synesthesia-like phenomena. Especially, research on synesthesia-like phenomena has been advanced by Marks et al., and they found synesthesia-like correlation between brightness and size of visual stimuli and pitch of auditory stimuli (Marks, 1987). It seems that main interest of research on synesthesia-like phenomena is what perceptual similarity/difference between synesthetes and non-synesthetes is. We guessed that cross-modal phenomena of non-synesthetes on perceptual level emerge as a function to complement the absence or ambiguity of a certain stimulus. To verify the hypothesis, we investigated audio–visual interaction using movement (speed) of an object as visual stimuli and sine-waves as auditory stimuli. In this experiment objects (circles) moved at a fixed speed in one trial and the objects were masked in arbitrary positions, and auditory stimuli (high, middle, low pitch) were given simultaneously with the disappearance of objects. Subject reported the expected position of the objects when auditory stimuli stopped. Result showed that correlation between the position, i.e., the movement speed, of the object and pitch of sound was found. We conjecture that cross-modal phenomena on non-synesthetes tend to occur when one of sensory stimuli are absent/ambiguous.

The Stream/Bounce Effect Occurs for Luminance- and Disparity-Defined Motion Targets

We generalised the stream/bounce effect to dynamic random element displays containing luminance- or disparity-defined targets. Previous studies investigating audio-visual interactions in this context have exclusively employed motion sequences with luminance-defined disks or squares and have focused on properties of the accompanying brief stimuli rather than the visual properties of the motion targets. We found that the presence of a brief sound temporally close to coincidence, or a visual flash at coincidence significantly promote bounce perception for motion targets defined by either luminance contrast or disparity contrast. A brief tone significantly promoted bouncing of luminance-defined targets above a no-sound baseline when it was presented at least 250 ms before coincidence and 100 ms after coincidence. A similar pattern was observed for disparity-defined targets, though the tone promoted bouncing above the no-sound baseline when presented at least 350 ms before and 300 ms after coincidence. We further explored the temporal properties of audio-visual interactions for these two display types and found that bounce perception saturated at similar durations after coincidence. The stream/bounce illusion manifests itself in dynamic random-element displays and is similar for luminance- and disparity-defined motion targets.

Electrophysiological correlates of predictive coding of auditory location in the perception of natural audiovisual events

In many natural audiovisual events (e.g., a clap of the two hands), the visual signal precedes the sound and thus allows observers to predict when, where, and which sound will occur. Previous studies have reported that there are distinct neural correlates of temporal (when) versus phonetic/semantic (which) content on audiovisual integration. Here we examined the effect of visual prediction of auditory location (where) in audiovisual biological motion stimuli by varying the spatial congruency between the auditory and visual parts. Visual stimuli were presented centrally, whereas auditory stimuli were presented either centrally or at 90° azimuth. Typical sub-additive amplitude reductions (AV − V < A) were found for the auditory N1 and P2 for spatially congruent and incongruent conditions. The new finding is that this N1 suppression was greater for the spatially congruent stimuli. A very early audiovisual interaction was also found at 40–60 ms (P50) in the spatially congruent condition, while no effect of congruency was found on the suppression of the P2. This indicates that visual prediction of auditory location can be coded very early in auditory processing.

Investigating the In-Between: Multisensory Integration of Auditory and Visual Motion Streams

We investigated audiovisual interactions in motion perception by behavioral experiments testing both, the influence of visual stimuli on auditory apparent motion and the influence of auditory stimuli on visual apparent motion perception. A set of loudspeakers with an LED mounted in the middle of each speaker cone was arranged in a semicircle. Apparent motion streams were presented for each modality alone in the unimodal conditions. In the bimodal conditions, stimuli of the second modality were added to fill the temporal and spatial gaps of the sampled trajectory of the reference stream. The participants' task was to observe the quasi-naturalistic stimulus sequences and to perform a standard classification. The addition of stimuli of the second modality indeed facilitated apparent motion perception. Bimodal presentation increased the upper temporal interval up to which the stimuli could be separated in time while still being perceived as continuous motion. We interpret these results as evidence for an ecologically advantageous audiovisual motion integration mechanism which operates beyond the constraints of strict spatiotemporal coincidence. Functional considerations suggest that this mechanism may represent an amodal stage suited for the processing of both unimodal and bimodal signals.

Temporal disparity effects on audiovisual integration in low vision individuals

Our recent findings have shown that sounds improve visual detection in low vision individuals when the audiovisual pairs are presented simultaneously. The present study purports to investigate possible temporal aspects of the audiovisual enhancement effect that we have previously reported. Low vision participants were asked to detect the presence of a visual stimulus (yes/no task) either presented in isolation or together with an auditory stimulus at different SOAs. In the first experiment, when the sound was always leading the visual stimuli, there was a significant visual detection enhancement even when the visual stimulus was temporally delayed by 400 ms. However, the visual detection improvement was reduced in the second experiment when the sound could randomly lead or lag the visual stimulus. A significant enhancement was found only when the audiovisual stimuli were synchronized. Taken together, the results of the present study seem to suggest that high-level associations between modalities might modulate audiovisual interactions in low vision individuals.

Different classes of audiovisual correspondences are processed at distinct levels of the cortical hierarchy

The brain should integrate sensory inputs only when they emanate from a common source and segregate those from different sources. Sensory correspondences are important cues informing the brain whether two sensory inputs are generated by a common event and should hence be integrated. Most prominently, sensory inputs should co-occur in time and space. More complex audiovisual stimuli may also be congruent in terms of semantics (e.g., objects and source sounds) or phonology (e.g., spoken and written words; linked via common linguistic labels). Surprisingly, metaphoric relations (e.g., pitch and height) have also been shown to influence audiovisual integration. The neural mechanisms that mediate these metaphoric congruency effects are only poorly understood. They may be mediated via (i) natural multisensory binding, (ii) common linguistic labels or (iii) semantics. In this talk, we will present a series of studies that investigate whether these different types of audiovisual correspondences are processed by distinct neural systems. Further, we investigate how those systems are employed by metaphoric audiovisual correspondences. Our results demonstrate that different classes of audiovisual correspondences influence multisensory integration at distinct levels of the cortical hierarchy. Spatiotemporal incongruency is detected already at the primary cortical level. Natural (e.g., motion direction) and phonological incongruency influences MSI in areas involved in motion or phonological processing. Critically, metaphoric interactions emerge in neural systems that are shared with natural and semantic incongruency. This activation pattern may reflect the ambivalent nature of metaphoric audiovisual interactions relying on both natural and semantic correspondences.

Audiovisual interactions depend on context of congruency

In this study, we addressed how the particular context of stimulus congruency influences audiovisual interactions. We combined an audiovisual congruency task with a proportion-of-congruency manipulation. In Experiment 1, we demonstrated that the perceived duration of a visual stimulus is modulated by the actual duration of a synchronously presented auditory stimulus. In the following experiments, we demonstrated that this crossmodal congruency effect is modulated by the proportion of congruent trials between (Exp. 2) and within (Exp. 4) blocks. In particular, the crossmodal congruency effect was reduced in the context with a high proportion of incongruent trials. This effect was attributed to changes in participants' control set as a function of the congruency context, with greater control applied in the context where the majority of the trials were incongruent. These data contribute to the ongoing debate concerning crossmodal interactions and attentional processes. In sum, context can provide a powerful cue for selective attention to modulate the interaction between stimuli from different sensory modalities.

Audiovisual speech integration in autism spectrum disorders: ERP evidence for atypicalities in lexical-semantic processing.

In typically developing (TD) individuals, behavioral and event-related potential (ERP) studies suggest that audiovisual (AV) integration enables faster and more efficient processing of speech. However, little is known about AV speech processing in individuals with autism spectrum disorders (ASD). This study examined ERP responses to spoken words to elucidate the effects of visual speech (the lip movements accompanying a spoken word) on the range of auditory speech processing stages from sound onset detection to semantic integration. The study also included an AV condition, which paired spoken words with a dynamic scrambled face in order to highlight AV effects specific to visual speech. Fourteen adolescent boys with ASD (15-17 years old) and 14 age- and verbal IQ-matched TD boys participated. The ERP of the TD group showed a pattern and topography of AV interaction effects consistent with activity within the superior temporal plane, with two dissociable effects over frontocentral and centroparietal regions. The posterior effect (200-300 ms interval) was specifically sensitive to lip movements in TD boys, and no AV modulation was observed in this region for the ASD group. Moreover, the magnitude of the posterior AV effect to visual speech correlated inversely with ASD symptomatology. In addition, the ASD boys showed an unexpected effect (P2 time window) over the frontocentral region (pooled electrodes F3, Fz, F4, FC1, FC2, FC3, FC4), which was sensitive to scrambled face stimuli. These results suggest that the neural networks facilitating processing of spoken words by visual speech are altered in individuals with ASD.

Understanding the deafened brain: Implications for cochlear implant rehabilitation

The cochlear implant (CI), by enabling oral communication in severely to profoundly deaf subjects, is one of the major medical advances over the last fifty years. Despite the globally very satisfactory results, individual outcomes vary considerably. The objective of this review is to describe the various factors influencing the results of CI rehabilitation with particular emphasis on the better understanding of neurocognitive mechanisms provided by functional brain imaging. The following aspects will be discussed: 1. Peripheral predictors such as the degree of preservation of nerve structures and the positioning of the electrode array. 2. The duration of auditory deprivation whose influence on brain reorganization is now becoming more clearly understood. 3. The age of initiation of hearing rehabilitation in subjects with pre-lingual deafness influencing the possibility of physiological maturation of nerve structures. 4. The concepts of sensitive period, decoupling and cross-modality. 5. In post-lingually deaf adults, brain plasticity can allow adaptation to the disability induced by deafness, subsequently potentiating CI rehabilitation, particularly as a result of audiovisual interactions. 6. Several studies provide concordant evidence that implanted patients present different phonological analysis and primary linguistic capacities. The results of CI rehabilitation are dependent on factors situated between the cochlea and cortical associative areas. The importance of higher cognitive influences on the functional results of cochlear implantation justify adaptation of coding strategies, as well as global cognitive management of deaf patients by utilising brain plasticity capacities.

Effects of denotative congruency on audio-visual impressions

Abstract: We examined how the denotative congruency in audio-visual stimuliaffects the pleasant and restful impressions of the stimuli. In Experiment 1, in whichwe combined a single sound and a single motion picture, we found that the impres-sions of the audio-visual stimuli would be determined in terms of averaging theimpressions obtained from each of the audio and visual stimuli. In Experiment 2, inwhich we combined two sounds and a single motion picture, we found that thepleasant and restful impressions positively shifted when one of the sounds wascombined with the denotatively congruent motion picture, even if the unpleasant andunrestful sound was combined with an unpleasant and unrestful motion picture.These results suggested that the denotative congruency between the audio andvisual stimuli would improve the pleasant and restful impressions. We are proposingthat the stimulus complexity, which would be reduced by the denotative congruencybetween the audio and visual stimuli, underlies the positive shift of the pleasant andrestful impressions in observing the stimuli.Key words: impression formation, integration process, relaxing music, noise, motionpicture.Some previous studies concerned with audio-visual interaction in perception and cognitionhave demonstrated that audio processing isaffected by visual stimuli. For instance, theMcGurk effect shows that audio processing ismodulated by visual stimulus in speech per-ception (McGurk & MacDonald, 1976). Theeffects, known as “visual capture” (Hay, Pick,& Ikeda, 1965; Jackson, 1953), or the “ventrilo-quism effect” (Bertelson & Radeau, 1981;Howard & Templeton, 1966; Jack & Thurlow,1973) imply that the perceived location of thesound source and events tends to be deter-mined by the location of the visual stimulus.Other previous studies have demonstratedthat visual processing is affected by audiostimuli. For instance, the studies pertaining tothe streaming-bouncing phenomena show thatvisual perception is influenced by a transientaudio stimulus (Sekuler, Sekuler, & Lau, 1997;Shimojo & Shams, 2001). The perceivedfrequency of the visual flash would varywith the frequency of the accompaniedaudio stimuli (Shams, Kamitani, & Shimojo,2000; Shimojo & Shams, 2001). The apparentdistance (Ichikawa & Masakura, 2006) andacceleration (Wada, Kitagawa, & Noguchi,2002) of visual motion are modulated by audiostimuli. These studies indicate that the interac-tion between audio processing and visual pro-cessing is very strong in perception andcognition.

Context and Crossmodal Interactions: An ERP Study

In a previous behavioural study, we reported that a particular context of stimulus congruency influenced audiovisual interactions. In particular, audiovisual interaction, measured in terms of congruency effect, was reduced when a high proportion of incongruent trials was presented. We argued that this modulation was due to changes in participants' control set as a function of the context of congruency, with greater control applied when most of the trials were incongruent. Since behavioural data do not allow to specify the level at which control was affecting audiovisual interaction, we conducted an event-related potentials (ERPs) study to further investigate each context of audiovisual congruency. Participants performed an audiovisual congruency task, where the stimulus onset could be present on two different contexts mixed at random: a high proportion congruent context and a low proportion congruent context. The context manipulation was found to modulate brain ERPs related to perceptual and response sele...

The spatiotemporal characteristics of elementary audiovisual speech and music processing in musically untrained subjects

Previously, the EEG technique has been used to investigate the spatiotemporal properties of audiovisual (AV) processing by taking advantage of the violation of the “additive model”, which is considered to be a very conservative approach. In the present work, we used a less conservative and novel approach than the criterion of superadditivity for estimating AV interactions. Hence, we estimated AV interaction patterns by comparing the responses to AV stimuli with the averaged responses to the unimodal visual and auditory stimuli in musically untrained subjects and by presenting syllables and piano tones coupled with flashlights. Our results suggest that the two AV objects elicited consistent interaction patterns within the time course of unisensory processing in the time range between 80 and 250ms post stimulus onset. The scalp topographies, as well as the source estimation approach we adopted, indicate that the first interaction pattern at around 100ms was partially driven by auditory-related cortical regions. Additionally, we found evidence for a second interaction pattern at around 200ms that was mainly associated with the responsiveness of extra-sensory brain regions. During this later processing stage, only the music condition was associated with putative responses that originated from auditory-related cortical fields. This study provides a novel approach to investigate the basic principles underlying elementary AV speech and music processing in subjects without formal musical education.

Neural mechanisms of audio tactile integration in the flutter range

Unveiling the mechanisms that integrate sensory information from different modalities has become a main aim of psychological research in the last decade. In this project we are interested in the specific case of auditory-tactile interactions in the perception of vibrotactile events, an aspect that has not received as much attention as audio-visual or visuo-tactile interactions. The case of audio-tactile interactions deserves special attention. As both modalities respond to the same physical property (mechanical pressure), it’s tempting to predict that such coincidence would allow integrative interactions at a variety of levels of processing, including the earliest ones. In fact, there is ample evidence that tactile events can influence activity in the earliest stages of auditory processing and there are several behavioral demonstrations of auditory influences on tactile perception. Particularly, vibrations in the flutter range (~10-50 Hz) constitute an interesting case for study given that vibratory patterns within this range are represented in the firing rates of the neurons in the sensory cortices for both somatosensory and auditory stimuli. Moreover in this range we have well detailed information given by the neurophysiological experiments. In particular studies describing the neural correlates of perceptual detection of vibrotactile events on the flutter range (e.g. see [2]) show that neurons in S1 respond with a very high degree of fidelity to the temporal profile of the stimulus. Similar results have been achieved with audio stimuli (e.g. see [3]) always on the flutter range. Our intention is to compare human behavioral data with an appropriate computational model in order to offer an appropriate hypothesis on the processing stage or stages where this audio-tactile interactions take place. Thus, we want to test if integration between somatosensory and auditory information does indeed take place on the flutter range as it does for high frequencies [4]. To this aim, we implement the same experiment of [4] for the flutter range; the experiment consists on comparing the detection threshold for audio, tactile and the two senses at the same time. As we want to ascertain whether stimuli integration takes place before or after perceptual detection, we back the behavioral data with a computational analysis of the processes involved by means of a network model composed of integrate-and-fire neurons. The neural network consists of multiple interacting layers, with the hypothesis that the integration happens at a high level processing areas in accordance with recent neurophysiological results [5]. The model can be applied both to flutter and no-flutter ranges. Our preliminary results (see figure ​figure1)1) are in accordance with the behavioral results of [4] (our experimental results are not conclusive). Figure 1 Model's detection probability for different stimulus amplitude and frequency (8,20,32,50 Hz). Dashed line is when is present only one stimulus (audio or tactile), solid line when the two senses are combined. Inset: Differences of detection probability ...

The effect of lip-reading on primary stream segregation

Lip-reading has been shown to improve the intelligibility of speech in multitalker situations, where auditory stream segregation naturally takes place. This study investigated whether the benefit of lip-reading is a result of a primary audiovisual interaction that enhances the obligatory streaming mechanism. Two behavioral experiments were conducted involving sequences of French vowels that alternated in fundamental frequency. In Experiment 1, subjects attempted to identify the order of items in a sequence. In Experiment 2, subjects attempted to detect a disruption to temporal isochrony across alternate items. Both tasks are disrupted by streaming, thus providing a measure of primary or obligatory streaming. Visual lip gestures articulating alternate vowels were synchronized with the auditory sequence. Overall, the results were consistent with the hypothesis that visual lip gestures enhance segregation by affecting primary auditory streaming. Moreover, increases in the naturalness of visual lip gestures and auditory vowels, and corresponding increases in audiovisual congruence may potentially lead to increases in the effect of visual lip gestures on streaming.

Person identification through faces and voices: An ERP study

Different models have been proposed to explain how identity is extracted from faces and voices and how these two sensory systems interact. The neural loci of audio–visual interactions have been studied using neuroimaging techniques; however, the time course of these interactions is not well established. Here, we use event related potentials (ERPs) to study the temporal dynamics of the interaction of face and voice processing modules. We presented to the subjects either faces alone (F), voices alone (V) or faces and voices together (FV) in a familiarity detection task. Responses obtained for FV were compared with the sum of the responses obtained for F plus responses for V, for familiar and unfamiliar stimuli. This comparison shows differences in amplitude for different latencies, indicating cross-modal interactions. For unfamiliar stimuli, this interaction began very early (around 200 ms) and was restricted to the time window corresponding to the face N170 component. For familiar stimuli, the interaction was longer, began earlier and remained until after the N170 component. These results indicate that the interaction between faces and voices occurs from the initial stages of processing and continues as the person identification process goes on. This study is the first electrophysiological evidence of cross-modal interaction of faces and voices during the recognition of acquaintances' identities. It suggests that the assessment of person familiarity can result in direct information sharing between voice and face sensory modules from the early processing stages, before access to the person identity nodes.

Cross-modal Interactions during Perception of Audiovisual Speech and Nonspeech Signals: An fMRI Study

During speech communication, visual information may interact with the auditory system at various processing stages. Most noteworthy, recent magnetoencephalography (MEG) data provided first evidence for early and preattentive phonetic/phonological encoding of the visual data stream--prior to its fusion with auditory phonological features [Hertrich, I., Mathiak, K., Lutzenberger, W., & Ackermann, H. Time course of early audiovisual interactions during speech and non-speech central-auditory processing: An MEG study. Journal of Cognitive Neuroscience, 21, 259-274, 2009]. Using functional magnetic resonance imaging, the present follow-up study aims to further elucidate the topographic distribution of visual-phonological operations and audiovisual (AV) interactions during speech perception. Ambiguous acoustic syllables--disambiguated to /pa/ or /ta/ by the visual channel (speaking face)--served as test materials, concomitant with various control conditions (nonspeech AV signals, visual-only and acoustic-only speech, and nonspeech stimuli). (i) Visual speech yielded an AV-subadditive activation of primary auditory cortex and the anterior superior temporal gyrus (STG), whereas the posterior STG responded both to speech and nonspeech motion. (ii) The inferior frontal and the fusiform gyrus of the right hemisphere showed a strong phonetic/phonological impact (differential effects of visual /pa/ vs. /ta/) upon hemodynamic activation during presentation of speaking faces. Taken together with the previous MEG data, these results point at a dual-pathway model of visual speech information processing: On the one hand, access to the auditory system via the anterior supratemporal “what" path may give rise to direct activation of "auditory objects." On the other hand, visual speech information seems to be represented in a right-hemisphere visual working memory, providing a potential basis for later interactions with auditory information such as the McGurk effect.

Visual modulation of auditory cortical activity correlates with behaviour

Event Abstract Back to Event Visual modulation of auditory cortical activity correlates with behaviour Jeremy D Thorne1*, Maarten De Vos1, Filipa C. Viola1 and Stefan Debener1 1 University of Oldenburg, Germany Behavioural responses to auditory stimuli are facilitated by visual input even when visual stimuli are task irrelevant. In a recent study, we showed that a significant proportion of the variance in such behavioural response times was accounted for by the degree of visually-induced phase reorganisation of oscillatory activity in auditory cortex (Thorne et al., 2011). These results may in part have related to the widespread entrainment of cortical responses, evident in the EEG, to the rhythmic stimulus presentation used. Here we investigated electrophysiological correlates of audiovisual (AV) interactions using simple, isolated AV stimuli. Twenty healthy adults performed a simple auditory discrimination task using both auditory and AV stimuli. Visual-auditory stimulus onset asynchrony (SOA) was varied between 0 and 100 ms. Analysis of behavioural responses showed that with short SOAs (< 50 ms), responses to AV stimuli were slower than to auditory stimuli alone, while with long SOAs, responses to AV stimuli were faster. A comparison of auditory evoked potentials with the equivalent waveforms derived from AV data revealed that the amplitude of the N1 peak was increased by visual input with short SOAs (< 50 ms) but was attenuated by visual input with longer SOAs. Although the role of cross-modal phase resetting in the present data is as yet unclear, the electrophysiological results were thus remarkably consistent with the behavioural results. We conclude that these data provide further evidence that the efficiency of auditory processing is improved when auditory onset can be predicted by visual input. Keywords: Perception, SOA Conference: XI International Conference on Cognitive Neuroscience (ICON XI), Palma, Mallorca, Spain, 25 Sep - 29 Sep, 2011. Presentation Type: Poster Presentation Topic: Poster Sessions: Neurophysiology of Sensation and Perception Citation: D Thorne J, De Vos M, Viola FC and Debener S (2011). Visual modulation of auditory cortical activity correlates with behaviour. Conference Abstract: XI International Conference on Cognitive Neuroscience (ICON XI). doi: 10.3389/conf.fnhum.2011.207.00357 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 23 Nov 2011; Published Online: 28 Nov 2011. * Correspondence: Dr. Jeremy D Thorne, University of Oldenburg, Oldenburg, Germany, jeremy.thorne@uni-oldenburg.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Jeremy D Thorne Maarten De Vos Filipa C Viola Stefan Debener Google Jeremy D Thorne Maarten De Vos Filipa C Viola Stefan Debener Google Scholar Jeremy D Thorne Maarten De Vos Filipa C Viola Stefan Debener PubMed Jeremy D Thorne Maarten De Vos Filipa C Viola Stefan Debener Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Spatial Shifts of Audio-Visual Interactions by Perceptual Learning are Specific to the Trained Orientation and Eye

A large proportion of the human cortex is devoted to visual processing. Contrary to the traditional belief that multimodal integration takes place in multimodal processing areas separate from visual cortex, several studies have found that sounds may directly alter processing in visual brain areas. Furthermore, recent findings show that perceptual learning can change the perceptual mechanisms that relate auditory and visual senses. However, there is still a debate about the systems involved in cross-modal learning. Here, we investigated the specificity of audio-visual perceptual learning. Audio-visual cuing effects were tested on a Gabor orientation task and an object discrimination task in the presence of lateralised sound cues before and after eight-days of cross-modal task-irrelevant perceptual learning. During training, the sound cues were paired with visual stimuli that were misaligned at a proximal (trained) visual field location relative to the sound. Training was performed with one eye patched and with only one Gabor orientation. Consistent with previous findings we found that cross-modal perceptual training shifted the audio-visual cueing effect towards the trained retinotopic location. However, this shift in audio-visual tuning was only observed for the trained stimulus (Gabors), at the trained orientation, and in the trained eye. This specificity suggests that multimodal interactions resulting from cross-modal (audio-visual) task-irrelevant perceptual learning involves so-called unisensory visual processing areas in humans. Our findings provide further support for recent anatomical and physiological findings that suggest relatively early interactions in cross-modal processing.