Audiovisual Simultaneity Judgment Task Research Articles

Individual Differences in Multisensory Processing Are Related to Broad Differences in the Balance of Local versus Distributed Information.

The brain's ability to extract information from multiple sensory channels is crucial to perception and effective engagement with the environment, but the individual differences observed in multisensory processing lack mechanistic explanation. We hypothesized that, from the perspective of information theory, individuals with more effective multisensory processing will exhibit a higher degree of shared information among distributed neural populations while engaged in a multisensory task, representing more effective coordination of information among regions. To investigate this, healthy young adults completed an audiovisual simultaneity judgment task to measure their temporal binding window (TBW), which quantifies the ability to distinguish fine discrepancies in timing between auditory and visual stimuli. EEG was then recorded during a second run of the simultaneity judgment task, and partial least squares was used to relate individual differences in the TBW width to source-localized EEG measures of local entropy and mutual information, indexing local and distributed processing of information, respectively. The narrowness of the TBW, reflecting more effective multisensory processing, was related to a broad pattern of higher mutual information and lower local entropy at multiple timescales. Furthermore, a small group of temporal and frontal cortical regions, including those previously implicated in multisensory integration and response selection, respectively, played a prominent role in this pattern. Overall, these findings suggest that individual differences in multisensory processing are related to widespread individual differences in the balance of distributed versus local information processing among a large subset of brain regions, with more distributed information being associated with more effective multisensory processing. The balance of distributed versus local information processing may therefore be a useful measure for exploring individual differences in multisensory processing, its relationship to higher cognitive traits, and its disruption in neurodevelopmental disorders and clinical conditions.

Lower multisensory temporal acuity in individuals with high schizotypal traits: a web-based study

Natural events are often multisensory, requiring the brain to combine information from the same spatial location and timing, across different senses. The importance of temporal coincidence has led to the introduction of the temporal binding window (TBW) construct, defined as the time range within which multisensory inputs are highly likely to be perceptually bound into a single entity. Anomalies in TBWs have been linked to confused perceptual experiences and inaccurate filtering of sensory inputs coming from different environmental sources. Indeed, larger TBWs have been associated with disorders such as schizophrenia and autism and are also correlated to a higher level of subclinical traits of these conditions in the general population. Here, we tested the feasibility of using a web-based version of a classic audio-visual simultaneity judgment (SJ) task with simple flash-beep stimuli in order to measure multisensory temporal acuity and its relationship with schizotypal traits as measured in the general population. Results show that: (i) the response distribution obtained in the web-based SJ task was strongly similar to those reported by studies carried out in controlled laboratory settings, and (ii) lower multisensory temporal acuity was associated with higher schizotypal traits in the “cognitive-perceptual” domains. Our findings reveal the possibility of adequately using a web-based audio-visual SJ task outside a controlled laboratory setting, available to a more diverse and representative pool of participants. These results provide additional evidence for a close relationship between lower multisensory acuity and the expression of schizotypal traits in the general population.

Brief period of monocular deprivation drives changes in audiovisual temporal perception

The human brain retains a striking degree of plasticity into adulthood. Recent studies have demonstrated that a short period of altered visual experience (via monocular deprivation) can change the dynamics of binocular rivalry in favor of the deprived eye, a compensatory action thought to be mediated by an upregulation of cortical gain control mechanisms. Here, we sought to better understand the impact of monocular deprivation on multisensory abilities, specifically examining audiovisual temporal perception. Using an audiovisual simultaneity judgment task, we discovered that 90 minutes of monocular deprivation produced opposing effects on the temporal binding window depending on the eye used in the task. Thus, in those who performed the task with their deprived eye there was a narrowing of the temporal binding window, whereas in those performing the task with their nondeprived eye there was a widening of the temporal binding window. The effect was short lived, being observed only in the first 10 minutes of postdeprivation testing. These findings indicate that changes in visual experience in the adult can rapidly impact multisensory perceptual processes, a finding that has important clinical implications for those patients with adult-onset visual deprivation and for therapies founded on monocular deprivation.

Improving audio-visual temporal perception through training enhances beta-band activity

Multisensory integration strongly depends on the temporal proximity between two inputs. In the audio-visual domain, stimulus pairs with delays up to a few hundred milliseconds can be perceived as simultaneous and integrated into a unified percept. Previous research has shown that the size of this temporal window of integration can be narrowed by feedback-guided training on an audio-visual simultaneity judgment task. Yet, it has remained uncertain how the neural network that processes audio-visual asynchronies is affected by the training. In the present study, participants were trained on a 2-interval forced choice audio-visual simultaneity judgment task. We recorded their neural activity with magnetoencephalography in response to three different stimulus onset asynchronies (0 ms, each participant’s individual binding window, 300 ms) before, and one day following training. The Individual Window stimulus onset asynchrony condition was derived by assessing each participant’s point of subjective simultaneity. Training improved performance in both asynchronous stimulus onset conditions (300 ms, Individual Window). Furthermore, beta-band amplitude (12–30 Hz) increased from pre-compared to post-training sessions. This increase moved across central, parietal, and temporal sensors during the time window of 80–410 ms post-stimulus onset. Considering the putative role of beta oscillations in carrying feedback from higher to lower cortical areas, these findings suggest that enhanced top-down modulation of sensory processing is responsible for the improved temporal acuity after training. As beta oscillations can be assumed to also preferentially support neural communication over longer conduction delays, the widespread topography of our effect could indicate that training modulates not only processing within primary sensory cortex, but rather the communication within a large-scale network.

Temporal Metrics of Multisensory Processing Change in the Elderly.

Older adults exhibit greater multisensory response time (RT) facilitation by violating the race model more than young adults; this is commonly interpreted as an enhancement in perception. Older adults typically exhibit wider temporal binding windows (TBWs) and points of subjective simultaneity (PSS) that typically lie farther from true simultaneity as compared to young adults when simultaneity judgment (SJ) and temporal-order judgment (TOJ) tasks are utilized; this is commonly interpreted as an impairment in perception. Here we explore the relation between the three tasks in order to better assess audiovisual multisensory temporal processing in both young and older adults. Our results confirm previous reports showing that audiovisual RT, TBWs and PSSs change with age; however, we show for the first time a significant positive relation between the magnitude of race model violation in young adults as a function of the PSS obtained from the audiovisual TOJ task (r: 0.49, p: 0.007), that is absent in older adults (r: 0.13, p: 0.58). Furthermore, we find no evidence for the relation between race model violation as a function of the PSS obtained from the audiovisual SJ task in both young (r: -0.01, p: 0.94) and older adults (r: 0.1, p: 0.66). Our results confirm previous reports that (i) audiovisual temporal processing changes with age; (ii) distinct processes are likely involved in simultaneity and temporal-order perception; and (iii) common processing between race model violation and temporal-order judgment is impaired in the elderly.

Atypical audiovisual temporal function in autism and schizophrenia: similar phenotype, different cause.

Binding across sensory modalities yields substantial perceptual benefits, including enhanced speech intelligibility. The coincidence of sensory inputs across time is a fundamental cue for this integration process. Recent work has suggested that individuals with diagnoses of schizophrenia (SZ) and autism spectrum disorder (ASD) will characterize auditory and visual events as synchronous over larger temporal disparities than their neurotypical counterparts. Namely, these clinical populations possess an enlarged temporal binding window (TBW). Although patients with SZ and ASD share aspects of their symptomatology, phenotypic similarities may result from distinct etiologies. To examine similarities and variances in audiovisual temporal function in these two populations, individuals diagnosed with ASD (n=46; controls n=40) and SZ (n=16, controls=16) completed an audiovisual simultaneity judgment task. In addition to standard psychometric analyses, synchrony judgments were assessed using Bayesian causal inference modeling. This approach permits distinguishing between distinct causes of an enlarged TBW: an a priori bias to bind sensory information and poor fidelity in the sensory representation. Findings indicate that both ASD and SZ populations show deficits in multisensory temporal acuity. Importantly, results suggest that while the wider TBWs in ASD most prominently results from atypical priors, the wider TBWs in SZ results from a trend toward changes in prior and weaknesses in the sensory representations. Results are discussed in light of current ASD and SZ theories and highlight that different perceptual training paradigms focused on improving multisensory integration may be most effective in these two clinical populations and emphasize that similar phenotypes may emanate from distinct mechanistic causes.

Being First Matters: Topographical Representational Similarity Analysis of ERP Signals Reveals Separate Networks for Audiovisual Temporal Binding Depending on the Leading Sense

In multisensory integration, processing in one sensory modality is enhanced by complementary information from other modalities. Intersensory timing is crucial in this process because only inputs reaching the brain within a restricted temporal window are perceptually bound. Previous research in the audiovisual field has investigated various features of the temporal binding window, revealing asymmetries in its size and plasticity depending on the leading input: auditory–visual (AV) or visual–auditory (VA). Here, we tested whether separate neuronal mechanisms underlie this AV–VA dichotomy in humans. We recorded high-density EEG while participants performed an audiovisual simultaneity judgment task including various AV–VA asynchronies and unisensory control conditions (visual-only, auditory-only) and tested whether AV and VA processing generate different patterns of brain activity. After isolating the multisensory components of AV–VA event-related potentials (ERPs) from the sum of their unisensory constituents, we ran a time-resolved topographical representational similarity analysis (tRSA) comparing the AV and VA ERP maps. Spatial cross-correlation matrices were built from real data to index the similarity between the AV and VA maps at each time point (500 ms window after stimulus) and then correlated with two alternative similarity model matrices: AVmaps = VAmaps versus AVmaps ≠ VAmaps. The tRSA results favored the AVmaps ≠ VAmaps model across all time points, suggesting that audiovisual temporal binding (indexed by synchrony perception) engages different neural pathways depending on the leading sense. The existence of such dual route supports recent theoretical accounts proposing that multiple binding mechanisms are implemented in the brain to accommodate different information parsing strategies in auditory and visual sensory systems.SIGNIFICANCE STATEMENT Intersensory timing is a crucial aspect of multisensory integration, determining whether and how inputs in one modality enhance stimulus processing in another modality. Our research demonstrates that evaluating synchrony of auditory-leading (AV) versus visual-leading (VA) audiovisual stimulus pairs is characterized by two distinct patterns of brain activity. This suggests that audiovisual integration is not a unitary process and that different binding mechanisms are recruited in the brain based on the leading sense. These mechanisms may be relevant for supporting different classes of multisensory operations, for example, auditory enhancement of visual attention (AV) and visual enhancement of auditory speech (VA).

Grouping and Segregation of Sensory Events by Actions in Temporal Audio-Visual Recalibration.

Perception in multi-sensory environments involves both grouping and segregation of events across sensory modalities. Temporal coincidence between events is considered a strong cue to resolve multisensory perception. However, differences in physical transmission and neural processing times amongst modalities complicate this picture. This is illustrated by cross-modal recalibration, whereby adaptation to audio-visual asynchrony produces shifts in perceived simultaneity. Here, we examined whether voluntary actions might serve as a temporal anchor to cross-modal recalibration in time. Participants were tested on an audio-visual simultaneity judgment task after an adaptation phase where they had to synchronize voluntary actions with audio-visual pairs presented at a fixed asynchrony (vision leading or vision lagging). Our analysis focused on the magnitude of cross-modal recalibration to the adapted audio-visual asynchrony as a function of the nature of the actions during adaptation, putatively fostering cross-modal grouping or, segregation. We found larger temporal adjustments when actions promoted grouping than segregation of sensory events. However, a control experiment suggested that additional factors, such as attention to planning/execution of actions, could have an impact on recalibration effects. Contrary to the view that cross-modal temporal organization is mainly driven by external factors related to the stimulus or environment, our findings add supporting evidence for the idea that perceptual adjustments strongly depend on the observer's inner states induced by motor and cognitive demands.

The Impact of Feedback on the Different Time Courses of Multisensory Temporal Recalibration.

The capacity to rapidly adjust perceptual representations confers a fundamental advantage when confronted with a constantly changing world. Unexplored is how feedback regarding sensory judgments (top-down factors) interacts with sensory statistics (bottom-up factors) to drive long- and short-term recalibration of multisensory perceptual representations. Here, we examined the time course of both cumulative and rapid temporal perceptual recalibration for individuals completing an audiovisual simultaneity judgment task in which they were provided with varying degrees of feedback. We find that in the presence of feedback (as opposed to simple sensory exposure) temporal recalibration is more robust. Additionally, differential time courses are seen for cumulative and rapid recalibration dependent upon the nature of the feedback provided. Whereas cumulative recalibration effects relied more heavily on feedback that informs (i.e., negative feedback) rather than confirms (i.e., positive feedback) the judgment, rapid recalibration shows the opposite tendency. Furthermore, differential effects on rapid and cumulative recalibration were seen when the reliability of feedback was altered. Collectively, our findings illustrate that feedback signals promote and sustain audiovisual recalibration over the course of cumulative learning and enhance rapid trial-to-trial learning. Furthermore, given the differential effects seen for cumulative and rapid recalibration, these processes may function via distinct mechanisms.

Developmental changes in the multisensory temporal binding window persist into adolescence

We live in a world rich in sensory information, and consequently the brain is challenged with deciphering which cues from the various sensory modalities belong together. Determinations regarding the relatedness of sensory information appear to be based, at least in part, on the spatial and temporal relationships between the stimuli. Stimuli that are presented in close spatial and temporal correspondence are more likely to be associated with one another and thus 'bound' into a single perceptual entity. While there is a robust literature delineating behavioral changes in perception induced by multisensory stimuli, maturational changes in multisensory processing, particularly in the temporal realm, are poorly understood. The current study examines the developmental progression of multisensory temporal function by analyzing responses on an audiovisual simultaneity judgment task in 6- to 23-year-old participants. The overarching hypothesis for the study was that multisensory temporal function will mature with increasing age, with the developmental trajectory for this change being the primary point of inquiry. Results indeed reveal an age-dependent decrease in the size of the 'multisensory temporal binding window', the temporal interval within which multisensory stimuli are likely to be perceptually bound, with changes occurring over a surprisingly protracted time course that extends into adolescence.

Audiovisual Temporal Recalibration for Speech in Synchrony Perception and Speech Identification

We investigated whether audiovisual synchrony perception for speech could change after observation of the audiovisual temporal mismatch. Previous studies have revealed that audiovisual synchrony perception is re-calibrated after exposure to a constant timing difference between auditory and visual signals in non-speech. In the present study, we examined whether this audiovisual temporal recalibration occurs at the perceptual level even for speech (monosyllables). In Experiment 1, participants performed an audiovisual simultaneity judgment task (i.e., a direct measurement of the audiovisual synchrony perception) in terms of the speech signal after observation of the speech stimuli which had a constant audiovisual lag. The results showed that the “simultaneous” responses (i.e., proportion of responses for which participants judged the auditory and visual stimuli to be synchronous) at least partly depended on exposure lag. In Experiment 2, we adopted the McGurk identification task (i.e., an indirect measurement of the audiovisual synchrony perception) to exclude the possibility that this modulation of synchrony perception was solely attributable to the response strategy using stimuli identical to those of Experiment 1. The characteristics of the McGurk effect reported by participants depended on exposure lag. Thus, it was shown that audiovisual synchrony perception for speech could be modulated following exposure to constant lag both in direct and indirect measurement. Our results suggest that temporal recalibration occurs not only in non-speech signals but also in monosyllabic speech at the perceptual level.

Decoding subjective simultaneity from neuromagnetic signals

Event Abstract Back to Event Decoding subjective simultaneity from neuromagnetic signals Kohske Takahashi1*, Shohei Hidaka2 and Katsumi Watanabe3, 4, 5 1 The University of Tokyo, Japan 2 Indiana University, Department of Psychological and Brain Sciences, United States 3 National Institute of Advanced Industrial Science and Technology, Japan 4 Japan Science and Technology Agency, Japan 5 National Center for Child Health and Development, Japan The present study examined neural correlate of subjective simultaneity by using whole-head magnetoencephalography. Nine participants performed an audiovisual simultaneity judgment task in the 160-channel MEG scanner (YOKOGAWA Corp.). The auditory and visual stimuli were presented with 5 stimulus onset asynchronies (SOA; 0, ±90, ±180 ms), and the participants were asked to judge whether the auditory and visual stimuli occurred simultaneously. The behavioral data showed that in the case of –90-ms-SOA (visual stimulus before auditory stimulus: VA) and +90-ms-SOA (visual stimulus after auditory stimulus: AV), subjective judgment varied (“simultaneous” response of VA and AV conditions were 59% and 65 %, respectively). In the MEG analysis, we chose 6 channels showing visual evoked activity for –90-ms-SOA, and also 6 channels showing auditory evoked activity for +90-ms-SOA. Then, we applied a wavelet analysis to the neuromagnetic signal recorded at each channel. In the classification test, we used a limited range of time–frequency component of the wavelet analysis (0 - 150 ms after the onset of the first stimulus; 8 Hz – 64 Hz), and compressed the components to 12 dimensions using principal component analysis. Subjective simultaneity was successfully classified, irrespective of which sensory modality was stimulated first. That is, classification trained with the VA data was able to decode the AV data, vice versa, at the rate better than chance. These results suggested that (1) simultaneity can be decoded by using MEG signals related to the sensory area; (2) the neural activity for the first stimulus, rather than the second stimulus, may predict subjective simultaneity; and (3) the neural activity reflecting subjective simultaneity is common among different sensory modalities. Conference: Biomag 2010 - 17th International Conference on Biomagnetism , Dubrovnik, Croatia, 28 Mar - 1 Apr, 2010. Presentation Type: Poster Presentation Topic: Sensory Processing and Functional Connectivity Citation: Takahashi K, Hidaka S and Watanabe K (2010). Decoding subjective simultaneity from neuromagnetic signals. Front. Neurosci. Conference Abstract: Biomag 2010 - 17th International Conference on Biomagnetism . doi: 10.3389/conf.fnins.2010.06.00232 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 31 Mar 2010; Published Online: 31 Mar 2010. * Correspondence: Kohske Takahashi, The University of Tokyo, Tokyo, Japan, takahashi.kohske@gmail.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Kohske Takahashi Shohei Hidaka Katsumi Watanabe Google Kohske Takahashi Shohei Hidaka Katsumi Watanabe Google Scholar Kohske Takahashi Shohei Hidaka Katsumi Watanabe PubMed Kohske Takahashi Shohei Hidaka Katsumi Watanabe Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Perceptual Training Narrows the Temporal Window of Multisensory Binding

The brain's ability to bind incoming auditory and visual stimuli depends critically on the temporal structure of this information. Specifically, there exists a temporal window of audiovisual integration within which stimuli are highly likely to be bound together and perceived as part of the same environmental event. Several studies have described the temporal bounds of this window, but few have investigated its malleability. Here, the plasticity in the size of this temporal window was investigated using a perceptual learning paradigm in which participants were given feedback during a two-alternative forced choice (2-AFC) audiovisual simultaneity judgment task. Training resulted in a marked (i.e., approximately 40%) narrowing in the size of the window. To rule out the possibility that this narrowing was the result of changes in cognitive biases, a second experiment using a two-interval forced choice (2-IFC) paradigm was undertaken during which participants were instructed to identify a simultaneously presented audiovisual pair presented within one of two intervals. The 2-IFC paradigm resulted in a narrowing that was similar in both degree and dynamics to that using the 2-AFC approach. Together, these results illustrate that different methods of multisensory perceptual training can result in substantial alterations in the circuits underlying the perception of audiovisual simultaneity. These findings suggest a high degree of flexibility in multisensory temporal processing and have important implications for interventional strategies that may be used to ameliorate clinical conditions (e.g., autism, dyslexia) in which multisensory temporal function may be impaired.

Audiovisual prior entry

It is almost one hundred years since Titchener [E.B. Titchener, Lectures on the Elementary Psychology of Feeling and Attention, Macmillan, New York, 1908] published his influential claim that attending to a particular sensory modality (or location) can speed up the relative time of arrival of stimuli presented in that modality (or location). However, the evidence supporting the existence of prior entry is, to date, mixed. In the present study, we used an audiovisual simultaneity judgment task in an attempt to circumvent the potential methodological confounds inherent in previous research in this area. Participants made simultaneous versus successive judgment responses regarding pairs of auditory and visual stimuli at varying stimulus onset asynchronies (SOAs) using the method of constant stimuli. In different blocks of trials, the participants were instructed to attend either to the auditory or to the visual modality, or else to divide their attention equally between the two modalities. The probability of trials containing intramodal stimulus pairs (e.g., vision–vision or audition–audition) was increased in the focused attention blocks to encourage participants to follow the instructions. The perception of simultaneity was modulated by this attentional manipulation: Visual stimuli had to lead auditory stimuli by a significantly smaller interval for simultaneity to be perceived when attention was directed to vision than when it was directed to audition. These results provide the first unequivocal evidence for the existence of audiovisual prior entry.