Unisensory Stimuli Research Articles

Enhanced multisensory gain in older adults may be a by-product of inverse effectiveness: Evidence from a speeded response-time task.

Older adults experience a greater benefit from multisensory integration than their younger counterparts, but it is unclear why. One hypothesis is that age-related sensory decline weakens unisensory stimulus effectiveness, causing a boost in multisensory gain through inverse effectiveness. Many previous studies present stimuli at the same intensity for both younger and older adults (i.e., stimulus-matched), as opposed to accounting for each participant's unique perceptual ability (i.e., perception-matched). This makes it difficult to discern the source of age-related differences in multisensory gain. As such, we used two experiments to examine whether sensory decline is contributing to age-related differences in multisensory gain. In the first, we presented auditory (pure tones in noise), visual (Gabor patches in noise), and audiovisual stimuli and recorded response times from 31 younger (18-25) and 30 older (55-80) adults. Importantly, all participants were given identical stimuli, with the expectation that older adults would show worse unisensory performance, inducing inverse effectiveness. The second task was identical (younger N = 31, older N = 34), except stimuli were presented at each participant's 50% detection threshold, identified with an adaptive psychophysical staircase, controlling for any influence of inverse effectiveness. Older adults were found to exhibit greater multisensory gain (as measured by race model violations) on stimulus- but not perception-matched tasks, thus aligning with the principle of inverse effectiveness. That is, when accounting for potential age-related differences in perceptual abilities, older adults no longer experienced greater benefit from multisensory integration. These two experiments together suggest that the age-related increases in multisensory integration previously reported may be in part due to age-related declines in vision and audition. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Addressing the Association Between Action Video Game Playing Experience and Visual Search in Naturalistic Multisensory Scenes.

Prior studies investigating the effects of routine action video game play have demonstrated improvements in a variety of cognitive processes, including improvements in attentional tasks. However, there is little evidence indicating that the cognitive benefits of playing action video games generalize from simplified unisensory stimuli to multisensory scenes- a fundamental characteristic of natural, everyday life environments. The present study addressed if video game experience has an impact on crossmodal congruency effects when searching through such multisensory scenes. We compared the performance of action video game players (AVGPs) and non-video game players (NVGPs) on a visual search task for objects embedded in video clips of realistic scenes. We conducted two identical online experiments with gender-balanced samples, for a total of N = 130. Overall, the data replicated previous findings reporting search benefits when visual targets were accompanied by semantically congruent auditory events, compared to neutral or incongruent ones. However, according to the results, AVGPs did not consistently outperform NVGPs in the overall search task, nor did they use multisensory cues more efficiently than NVGPs. Exploratory analyses with self-reported gender as a variable revealed a potential difference in response strategy between experienced male and female AVGPs when dealing with crossmodal cues. These findings suggest that the generalization of the advantage of AVG experience to realistic, crossmodal situations should be made with caution and considering gender-related issues.

Linear integration of multisensory signals in the pupil.

The pupil of the eye responds to various salient signals from different modalities, but there is no consensus on how these pupillary responses are integrated when multiple signals appear simultaneously. Both linear and nonlinear integration have been found previously. The current study aimed to reexamine the nature of pupillary integration, and specifically focused on the early, transient pupillary responses due to its close relationship with orienting. To separate the early pupillary responses out of the pupil time series, we adopted a pupil oscillation paradigm in which sensory stimuli were periodically presented. The simulation analysis confirmed that the amplitude of the pupil oscillation, induced by stimuli repeatedly presented at relatively high rates, can precisely reflect the early, transient pupillary responses without involving the late and sustained pupillary responses. The experimental results then showed that the amplitude of pupil oscillation induced by a series of simultaneous audiovisual stimuli equaled to a linear summation of the oscillatory amplitudes when unisensory stimuli were presented alone. Moreover, the tonic arousal levels, indicated by the baseline pupil size, cannot shift the summation from linear to nonlinear. These findings together support the additive nature of multisensory pupillary integration for the early, orienting-related pupillary responses. The additive nature of pupillary integration further implies that multiple pupillary responses may be independent of each other, irrespective of their potential cognitive and neural drivers.

Metacognition in the audiovisual McGurk illusion: perceptual and causal confidence.

Almost all decisions in everyday life rely on multiple sensory inputs that can come from common or independent causes. These situations invoke perceptual uncertainty about environmental properties and the signals' causal structure. Using the audiovisual McGurk illusion, this study investigated how observers formed perceptual and causal confidence judgements in information integration tasks under causal uncertainty. Observers were presented with spoken syllables, their corresponding articulatory lip movements or their congruent and McGurk combinations (e.g. auditory B/P with visual G/K). Observers reported their perceived auditory syllable, the causal structure and confidence for each judgement. Observers were more accurate and confident on congruent than unisensory trials. Their perceptual and causal confidence were tightly related over trials as predicted by the interactive nature of perceptual and causal inference. Further, observers assigned comparable perceptual and causal confidence to veridical 'G/K' percepts on audiovisual congruent trials and their causal and perceptual metamers onMcGurk trials (i.e. illusory 'G/K' percepts). Thus, observers metacognitively evaluate the integrated audiovisual percept with limited access to the conflicting unisensory stimulus components on McGurk trials. Collectively, our results suggest that observers form meaningful perceptual and causal confidence judgements about multisensory scenes that are qualitatively consistent with principles of Bayesian causal inference. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy

In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.

Self-prioritization with unisensory and multisensory stimuli in a matching task

A shape-label matching task is commonly used to examine the self-advantage in motor reaction-time responses (the Self-Prioritization Effect; SPE). In the present study, auditory labels were introduced, and, for the first time, responses to unisensory auditory, unisensory visual, and multisensory object-label stimuli were compared across block-type (i.e., trials blocked by sensory modality type, and intermixed trials of unisensory and multisensory stimuli). Auditory stimulus intensity was presented at either 50 dB (Group 1) or 70 dB (Group 2). The participants in Group 2 also completed a multisensory detection task, making simple speeded motor responses to the shape and sound stimuli and their multisensory combinations. In the matching task, the SPE was diminished in intermixed trials, and in responses to the unisensory auditory stimuli as compared with the multisensory (visual shape+auditory label) stimuli. In contrast, the SPE did not differ in responses to the unisensory visual and multisensory (auditory object+visual label) stimuli. The matching task was associated with multisensory ‘costs’ rather than gains, but response times to self- versus stranger-associated stimuli were differentially affected by the type of multisensory stimulus (auditory object+visual label or visual shape+auditory label). The SPE was thus modulated both by block-type and the combination of object and label stimulus modalities. There was no SPE in the detection task. Taken together, these findings suggest that the SPE with unisensory and multisensory stimuli is modulated by both stimulus- and task-related parameters within the matching task. The SPE does not transfer to a significant motor speed gain when the self-associations are not task-relevant.

A systematic review and meta-analysis on the use of tactile stimulation in vection research.

Vection is classically defined as the illusory perception of self-motion induced via visual stimuli. The utility of vection research lies in its potential to enhance simulation fidelity, as measured through presence, and reduce the probability that motion sickness symptoms occur. Studies have shown a multimodal interaction of various sensory systems in facilitating vection, and the utility of co-stimulating some of these sensory systems along with the presentation of visual stimuli have been reviewed. However, a review on the use of tactile stimulation in vection research appears to be missing from literature. The purpose of this review was to evaluate the current methodologies, and outcomes, of tactile stimulation in vection research. We searched for articles through EBSCOHost, Scopus and Web of Science. Studies were included only if they detailed an experiment on the effect of tactile stimulation on vection. Twenty-four studies were obtained and distilled in tabular form. Eighteen studies contained sufficient information to be included in a meta-analysis. We identified that tactile stimulation has mostly been applied in the form of vibrational stimulation to the feet. Furthermore, tactile stimulation is most effective when it is presented in a temporally congruent manner to other sensory cues, whereas tactile stimulation as a unisensory stimulus does not appear to be effective in eliciting vection. We discuss the need for more qualitative research to reduce methodological inhomogeneities and recommend future research in tactile-mediated vection to investigate stimulation to the torso and investigate the use of forces as a tactile stimulus.

Multisensory Processing Can Compensate for Top-Down Attention Deficits in Schizophrenia.

Studies on schizophrenia (SCZ) and aberrant multisensory integration (MSI) show conflicting results, which are potentially confounded by attention deficits in SCZ. To test this, we examined the interplay between MSI and intersensory attention (IA) in healthy controls (HCs) (N = 27) and in SCZ (N = 27). Evoked brain potentials to unisensory-visual (V), unisensory-tactile (T), or spatiotemporally aligned bisensory VT stimuli were measured with high-density electroencephalography, while participants attended blockwise to either visual or tactile inputs. Behaviorally, IA effects in SCZ, relative to HC, were diminished for unisensory stimuli, but not for bisensory stimuli. At the neural level, we observed reduced IA effects for bisensory stimuli over mediofrontal scalp regions (230-320 ms) in SCZ. The analysis of MSI, using the additive approach, revealed multiple phases of integration over occipital and frontal scalp regions (240-364 ms), which did not differ between HC and SCZ. Furthermore, IA and MSI effects were both positively related to the behavioral performance in SCZ, indicating that IA and MSI mutually facilitate bisensory stimulus processing. Multisensory processing could facilitate stimulus processing and compensate for top-down attention deficits in SCZ. Differences in attentional demands, which may be differentially compensated by multisensory processing, could account for previous conflicting findings on MSI in SCZ.

Top\u2013down task-specific determinants of multisensory motor reaction time enhancements and sensory switch costs

This study was designed to investigate the complex interplay between multisensory processing, top–down processes related to the task relevance of sensory signals, and sensory switching. Thirty-five adults completed either a speeded detection or a discrimination task using the same auditory and visual stimuli and experimental setup. The stimuli consisted of unisensory and multisensory presentations of the letters ‘b’ and ‘d’. The multisensory stimuli were either congruent (e.g., the grapheme ‘b’ with the phoneme /b/) or incongruent (e.g., the grapheme ‘b’ with the phoneme /d/). In the detection task, the participants had to respond to all of the stimuli as rapidly as possible while, in the discrimination task, they only responded on those trials where one prespecified letter (either ‘b’ or ‘d’) was present. Incongruent multisensory stimuli resulted in faster responses as compared to unisensory stimuli in the detection task. In the discrimination task, only the dual-target congruent stimuli resulted in faster RTs, while the incongruent multisensory stimuli led to slower RTs than to unisensory stimuli; RTs were the slowest when the visual (rather than the auditory) signal was irrelevant, thus suggesting visual dominance. Switch costs were also observed when switching between unisensory target stimuli, while dual-target multisensory stimuli were less likely to be affected by sensory switching. Taken together, these findings suggest that multisensory motor enhancements and sensory switch costs are influenced by top–down modulations determined by task instructions, which can override the influence of prior learnt associations.

Loss of high- or low-frequency audibility can partially explain effects of hearing loss on emotional responses to non-speech sounds

Hearing loss can disrupt emotional responses to sound. However, the impact of stimulus modality (multisensory versus unisensory) on this disruption, and the underlying mechanisms responsible, are unclear. The purposes of this project were to evaluate the effects of stimulus modality and filtering on emotional responses to non-speech stimuli. It was hypothesized that low- and high-pass filtering would result in less extreme ratings, but only for unisensory stimuli. Twenty-four adults (22- 34 years old; 12 male) with normal hearing participated. Participants made ratings of valence and arousal in response to pleasant, neutral, and unpleasant non-speech sounds and/or pictures. Each participant completed ratings of five stimulus modalities: auditory-only, visual-only, auditory-visual, filtered auditory-only, and filtered auditory-visual. Half of the participants rated low-pass filtered stimuli (800 Hz cutoff), and half of the participants rated high-pass filtered stimuli (2000 Hz cutoff). Combining auditory and visual modalities resulted in more extreme (more pleasant and more unpleasant) ratings of valence in response to pleasant and unpleasant stimuli. In addition, low- and high-pass filtering of sounds resulted in less extreme ratings of valence (less pleasant and less unpleasant) and arousal (less exciting) in response to both auditory-only and auditory-visual stimuli. These results suggest that changes in audible spectral information are partially responsible for the noted changes in emotional responses to sound that accompany hearing loss. The findings also suggest the effects of hearing loss will generalize to multisensory stimuli if the stimuli include sound, although further work is warranted to confirm this in listeners with hearing loss.

The Neurophysiological Basis of the Trial-Wise and Cumulative Ventriloquism Aftereffects.

Our senses often receive conflicting multisensory information, which our brain reconciles by adaptive recalibration. A classic example is the ventriloquism aftereffect, which emerges following both cumulative (long-term) and trial-wise exposure to spatially discrepant multisensory stimuli. Despite the importance of such adaptive mechanisms for interacting with environments that change over multiple timescales, it remains debated whether the ventriloquism aftereffects observed following trial-wise and cumulative exposure arise from the same neurophysiological substrate. We address this question by probing electroencephalography recordings from healthy humans (both sexes) for processes predictive of the aftereffect biases following the exposure to spatially offset audiovisual stimuli. Our results support the hypothesis that discrepant multisensory evidence shapes aftereffects on distinct timescales via common neurophysiological processes reflecting sensory inference and memory in parietal-occipital regions, while the cumulative exposure to consistent discrepancies additionally recruits prefrontal processes. During the subsequent unisensory trial, both trial-wise and cumulative exposure bias the encoding of the acoustic information, but do so distinctly. Our results posit a central role of parietal regions in shaping multisensory spatial recalibration, suggest that frontal regions consolidate the behavioral bias for persistent multisensory discrepancies, but also show that the trial-wise and cumulative exposure bias sound position encoding via distinct neurophysiological processes.SIGNIFICANCE STATEMENT Our brain easily reconciles conflicting multisensory information, such as seeing an actress on screen while hearing her voice over headphones. These adaptive mechanisms exert a persistent influence on the perception of subsequent unisensory stimuli, known as the ventriloquism aftereffect. While this aftereffect emerges following trial-wise or cumulative exposure to multisensory discrepancies, it remained unclear whether both arise from a common neural substrate. We here rephrase this hypothesis using human electroencephalography recordings. Our data suggest that parietal regions involved in multisensory and spatial memory mediate the aftereffect following both trial-wise and cumulative adaptation, but also show that additional and distinct processes are involved in consolidating and implementing the aftereffect following prolonged exposure.

Purkinje Cell Activity Determines the Timing of Sensory-Evoked Motor Initiation

SummaryCerebellar neurons can signal sensory and motor events, but their role in active sensorimotor processing remains unclear. We record and manipulate Purkinje cell activity during a task that requires mice to rapidly discriminate between multisensory and unisensory stimuli before motor initiation. Neuropixels recordings show that both sensory stimuli and motor initiation are represented by short-latency simple spikes. Optogenetic manipulation of short-latency simple spikes abolishes or delays motor initiation in a rate-dependent manner, indicating a role in motor initiation and its timing. Two-photon calcium imaging reveals task-related coherence of complex spikes organized into conserved alternating parasagittal stripes. The coherence of sensory-evoked complex spikes increases with learning and correlates with enhanced temporal precision of motor initiation. These results suggest that both simple spikes and complex spikes govern sensory-driven motor initiation: simple spikes modulate its latency, and complex spikes refine its temporal precision, providing specific cellular substrates for cerebellar sensorimotor control.

An Experimenter's Influence on Motor Enhancements: The Effects of Letter Congruency and Sensory Switch-Costs on Multisensory Integration.

Multisensory integration can alter information processing, and previous research has shown that such processes are modulated by sensory switch costs and prior experience (e.g., semantic or letter congruence). Here we report an incidental finding demonstrating, for the first time, the interplay between these processes and experimental factors, specifically the presence (vs. absence) of the experimenter in the testing room. Experiment 1 demonstrates that multisensory motor facilitation in response to audiovisual stimuli (circle and tone with no prior learnt associations) is higher in those trials in which the sensory modality switches than when it repeats. Those participants who completed the study while alone exhibited increased RT variability. Experiment 2 replicated these findings using the letters “b” and “d” presented as unisensory stimuli or congruent and incongruent multisensory stimuli (i.e., grapheme-phoneme pairs). Multisensory enhancements were inflated following a sensory switch; that is, congruent and incongruent multisensory stimuli resulted in significant gains following a sensory switch in the monitored condition. However, when the participants were left alone, multisensory enhancements were only observed for repeating incongruent multisensory stimuli. These incidental findings therefore suggest that the effects of letter congruence and sensory switching on multisensory integration are partly modulated by the presence of an experimenter.

Multisensory Facilitation of Working Memory Training.

Research suggests that memorization of multisensory stimuli benefits performance compared to memorization of unisensory stimuli; however, little is known about multisensory facilitation in the context of working memory (WM) training and transfer. To investigate this, 240 adults were randomly assigned to an N-back training task that consisted of visual-only stimuli, alternating visual and auditory blocks, or audio-visual (multisensory) stimuli, or to a passive control group. Participants in the active groups completed 13 sessions of N-back training (6.7 hours in total) and all groups completed a battery of WM tasks: untrained N-back tasks, Corsi Blocks, Sequencing, and Symmetry Span. The Multisensory group showed similar training N-level gain compared to the Visual Only group, and both of these groups outperformed the Alternating group on the training task. As expected, all three active groups significantly improved on untrained visual N-back tasks compared to the Control group. In contrast, the Multisensory group showed significantly greater gains on the Symmetry Span task and to a certain extent on the Sequencing task compared to other groups. These results tentatively suggest that incorporating multisensory objects in a WM training protocol can benefit performance on the training task and potentially facilitate transfer to complex WM span tasks.

Cross-sensory inhibition or unisensory facilitation: A potential neural architecture of modality switch effects

In a simple reaction time task in which auditory and visual stimuli are presented in random sequence alone (A or V) or together (AV), there is a so-called reaction time (RT) cost on trials in which sensory modality switches (A→V) compared to when it repeats (A→A). This is always true for unisensory trials, whereas RTs to AV stimuli preceded by unisensory stimuli are statistically comparable with the Repeat condition (AV→AV). Neural facilitation for Repeat trials or neural inhibition for Switch trials could both account for these effects. Here we used a neural network model (Multisensory Integration with Crossed Inhibitory Dynamics (MICID) model) to test the ability of these two distinct mechanisms, inhibition and facilitation, to produce the specific patterns of behavior that we see experimentally, modeling switch and repeat trials as well as the influence of the interval between the present and the previous trial. The model results are consistent with an inhibitory account in which there is competition between the different sensory modalities, instead of a facilitation account in which the preceding stimulus sensitizes the neural system to its particular sensory modality. Moreover, the model shows that multisensory integration can explain the results in case of multisensory stimuli, where the preceding stimulus has little effect. This is due to faster dynamics for multisensory facilitation compared to cross-sensory inhibition. These findings link the cognitive framework delineated by the empirical results to a plausible neural implementation.

The neural mechanisms of audiotactile binding depend on asynchrony.

Asynchrony is a critical cue informing the brain whether sensory signals are caused by a common source and should be integrated or segregated. This psychophysics-electroencephalography (EEG) study investigated the influence of asynchrony on how the brain binds audiotactile (AT) signals to enable faster responses in a redundant target paradigm. Human participants actively responded (psychophysics) or passively attended (EEG) to noise bursts, "taps-to-the-face" and their AT combinations at seven AT asynchronies: 0, ±20, ±70 and ±500ms. Behaviourally, observers were faster at detecting AT than unisensory stimuli within a temporal integration window: the redundant target effect was maximal for synchronous stimuli and declined within a ≤70ms AT asynchrony. EEG revealed a cascade of AT interactions that relied on different neural mechanisms depending on AT asynchrony. At small (≤20ms) asynchronies, AT interactions arose for evoked response potentials (ERPs) at 110ms and ~400ms post-stimulus. Selectively at ±70ms asynchronies, AT interactions were observed for the P200 ERP, theta-band inter-trial coherence (ITC) and power at ~200ms post-stimulus. In conclusion, AT binding was mediated by distinct neural mechanisms depending on the asynchrony of the AT signals. Early AT interactions in ERPs and theta-band ITC and power were critical for the behavioural response facilitation within a ≤±70ms temporal integration window.

Microsaccadic Eye Movements but not Pupillary Dilation Response Characterizes the Crossmodal Freezing Effect.

In typical spatial orienting tasks, the perception of crossmodal (e.g., audiovisual) stimuli evokes greater pupil dilation and microsaccade inhibition than unisensory stimuli (e.g., visual). The characteristic pupil dilation and microsaccade inhibition has been observed in response to “salient” events/stimuli. Although the “saliency” account is appealing in the spatial domain, whether this occurs in the temporal context remains largely unknown. Here, in a brief temporal scale (within 1 s) and with the working mechanism of involuntary temporal attention, we investigated how eye metric characteristics reflect the temporal dynamics of perceptual organization, with and without multisensory integration. We adopted the crossmodal freezing paradigm using the classical Ternus apparent motion. Results showed that synchronous beeps biased the perceptual report for group motion and triggered the prolonged sound-induced oculomotor inhibition (OMI), whereas the sound-induced OMI was not obvious in a crossmodal task-free scenario (visual localization without audiovisual integration). A general pupil dilation response was observed in the presence of sounds in both visual Ternus motion categorization and visual localization tasks. This study provides the first empirical account of crossmodal integration by capturing microsaccades within a brief temporal scale; OMI but not pupillary dilation response characterizes task-specific audiovisual integration (shown by the crossmodal freezing effect).

Selective attention to sound features mediates cross-modal activation of visual cortices

Contemporary schemas of brain organization now include multisensory processes both in low-level cortices as well as at early stages of stimulus processing. Evidence has also accumulated showing that unisensory stimulus processing can result in cross-modal effects. For example, task-irrelevant and lateralised sounds can activate visual cortices; a phenomenon referred to as the auditory-evoked contralateral occipital positivity (ACOP). Some claim this is an example of automatic attentional capture in visual cortices. Other results, however, indicate that context may play a determinant role. Here, we investigated whether selective attention to spatial features of sounds is a determining factor in eliciting the ACOP. We recorded high-density auditory evoked potentials (AEPs) while participants selectively attended and discriminated sounds according to four possible stimulus attributes: location, pitch, speaker identity or syllable. Sound acoustics were held constant, and their location was always equiprobable (50% left, 50% right). The only manipulation was to which sound dimension participants attended. We analysed the AEP data from healthy participants within an electrical neuroimaging framework. The presence of sound-elicited activations of visual cortices depended on the to-be-discriminated, goal-based dimension. The ACOP was elicited only when participants were required to discriminate sound location, but not when they attended to any of the non-spatial features. These results provide a further indication that the ACOP is not automatic. Moreover, our findings showcase the interplay between task-relevance and spatial (un)predictability in determining the presence of the cross-modal activation of visual cortices.

Age-related hearing loss influences functional connectivity of auditory cortex for the McGurk illusion

Age-related hearing loss affects hearing at high frequencies and is associated with difficulties in understanding speech. Increased audio-visual integration has recently been found in age-related hearing impairment, the brain mechanisms that contribute to this effect are however unclear. We used functional magnetic resonance imaging in elderly subjects with normal hearing and mild to moderate uncompensated hearing loss. Audio-visual integration was studied using the McGurk task. In this task, an illusionary fused percept can occur if incongruent auditory and visual syllables are presented. The paradigm included unisensory stimuli (auditory only, visual only), congruent audio-visual and incongruent (McGurk) audio-visual stimuli. An illusionary precept was reported in over 60% of incongruent trials. These McGurk illusion rates were equal in both groups of elderly subjects and correlated positively with speech-in-noise perception and daily listening effort. Normal-hearing participants showed an increased neural response in left pre- and postcentral gyri and right middle frontal gyrus for incongruent stimuli (McGurk) compared to congruent audio-visual stimuli. Activation patterns were however not different between groups. Task-modulated functional connectivity differed between groups showing increased connectivity from auditory cortex to visual, parietal and frontal areas in hard of hearing participants as compared to normal-hearing participants when comparing incongruent stimuli (McGurk) with congruent audio-visual stimuli. These results suggest that changes in functional connectivity of auditory cortex rather than activation strength during processing of audio-visual McGurk stimuli accompany age-related hearing loss.

Differential Rapid Plasticity in Auditory and Visual Responses in the Primarily Multisensory Orbitofrontal Cortex.

Given the connectivity of orbitofrontal cortex (OFC) with the sensory areas and areas involved in goal execution, it is likely that OFC, along with its function in reward processing, also has a role to play in perception-based multisensory decision-making. To understand mechanisms involved in multisensory decision-making, it is important to first know the encoding of different sensory stimuli in single neurons of the mouse OFC. Ruling out effects of behavioral state, memory, and others, we studied the anesthetized mouse OFC responses to auditory, visual, and audiovisual/multisensory stimuli, multisensory associations and sensory-driven input organization to the OFC. Almost all, OFC single neurons were found to be multisensory in nature, with sublinear to supralinear integration of the component unisensory stimuli. With a novel multisensory oddball stimulus set, we show that the OFC receives both unisensory as well as multisensory inputs, further corroborated by retrograde tracers showing labeling in secondary auditory and visual cortices, which we find to also have similar multisensory integration and responses. With long audiovisual pairing/association, we show rapid plasticity in OFC single neurons, with a strong visual bias, leading to a strong depression of auditory responses and effective enhancement of visual responses. Such rapid multisensory association driven plasticity is absent in the auditory and visual cortices, suggesting its emergence in the OFC. Based on the above results, we propose a hypothetical local circuit model in the OFC that integrates auditory and visual information which participates in computing stimulus value in dynamic multisensory environments.