Multisensory Temporal Function Research Articles

Temporal binding window between three different sensory modalities: Auditory, visual, and tactile

Multisensory integration is crucial to perceptual understanding, but cues from multiple modalities do not necessarily arrive at the brain simultaneously. The window of time in which sensory inputs can be interpreted as simultaneous is termed the “temporal binding window” (TBW). Evidence suggests unimodal temporal discrimination is altered in a series of neurodevelopmental disorders but its relationship to perceptual timing of multisensory events (auditory, visual, and tactile) is not known, even in healthy populations. The purpose of this study was to characterize temporal integration of sensory modalities within a young healthy population. Twenty-five adults participated in TBW measurements with three different modalities. 100-ms stimuli (auditory: beep; visual: flash; tactile: vibration) were presented with various onset asynchrony (-500 to 500 ms) between two stimuli (AV: auditory-visual; AT: auditory-tactile; VT: visual-tactile). Participants were instructed to select “simultaneous” or “asynchronous” responses. Results show that the TBW for the AV condition ranged from 222 to 689 ms (mean ± SD = 418 ± 103), the AT condition ranged from 164 to 666 ms (376 ± 130), and the VT condition ranged from 70 to 751 ms (279 ± 131). These findings suggest that deficits in multisensory temporal function may factor into perceptual and cognitive weaknesses that characterize clinical disorders, raising the possibility that it may be a contributing pathologic factor in these conditions.

Brief Sensory Training Narrows the Temporal Binding Window and Enhances Long-Term Multimodal Speech Perception.

Our ability to integrate multiple sensory-based representations of our surrounding supplies us with a more holistic view of our world. There are many complex algorithms our nervous system uses to construct a coherent perception. An indicator to solve this ‘binding problem’ are the temporal characteristics with the specificity that environmental information has different propagation speeds (e.g., sound and electromagnetic waves) and sensory processing time and thus the temporal relationship of a stimulus pair derived from the same event must be flexibly adjusted by our brain. This tolerance can be conceptualized in the form of the cross-modal temporal binding window (TBW). Several studies showed the plasticity of the TBW and its importance concerning audio-visual illusions, synesthesia, as well as psychiatric disturbances. Using three audio-visual paradigms, we investigated the importance of length (short vs. long) as well as modality (uni- vs. multimodal) of a perceptual training aiming at reducing the TBW in a healthy population. We also investigated the influence of the TBW on speech intelligibility, where participants had to integrate auditory and visual speech information from a videotaped speaker. We showed that simple sensory trainings can change the TBW and are capable of optimizing speech perception at a very naturalistic level. While the training-length had no different effect on the malleability of the TBW, the multisensory trainings induced a significantly stronger narrowing of the TBW than their unisensory counterparts. Furthermore, a narrowing of the TBW was associated with a better performance in speech perception, meaning that participants showed a greater capacity for integrating informations from different sensory modalities in situations with one modality impaired. All effects persisted at least seven days. Our findings show the significance of multisensory temporal processing regarding ecologically valid measures and have important clinical implications for interventions that may be used to alleviate debilitating conditions (e.g., autism, schizophrenia), in which multisensory temporal function is shown to be impaired.

Multisensory temporal function and EEG complexity in patients with epilepsy and psychogenic nonepileptic events.

Cognitive and perceptual comorbidities frequently accompany epilepsy and psychogenic nonepileptic events (PNEE). However, and despite the fact that perceptual function is built upon a multisensory foundation, little knowledge exists concerning multisensory function in these populations. Here, we characterized facets of multisensory processing abilities in patients with epilepsy and PNEE, and probed the relationship between individual resting-state EEG complexity and these psychophysical measures in each patient. We prospectively studied a cohort of patients with epilepsy (N=18) and PNEE (N=20) patients who were admitted to Vanderbilt's Epilepsy Monitoring Unit (EMU) and weaned off of anticonvulsant drugs. Unaffected age-matched persons staying with the patients in the EMU (N=15) were also recruited as controls. All participants performed two tests of multisensory function: an audio-visual simultaneity judgment and an audio-visual redundant target task. Further, in the cohort of patients with epilepsy and PNEE we quantified resting state EEG gamma power and complexity. Compared with both patients with epilepsy and control subjects, patients with PNEE exhibited significantly poorer acuity in audiovisual temporal function as evidenced in significantly larger temporal binding windows (i.e., they perceived larger stimulus asynchronies as being presented simultaneously). These differences appeared to be specific for temporal function, as there was no difference among the three groups in a non-temporally based measure of multisensory function - the redundant target task. Further, patients with PNEE exhibited more complex resting state EEG patterns as compared to their patients with epilepsy, and EEG complexity correlated with multisensory temporal performance on a subject-by-subject manner. Taken together, findings seem to indicate that patients with PNEE bind information from audition and vision over larger temporal intervals when compared with control subjects as well as patients with epilepsy. This difference in multisensory function appears to be specific to the temporal domain, and may be a contributing factor to the behavioral and perceptual alterations seen in this population.

Atypical rapid audio-visual temporal recalibration in autism spectrum disorders.

Changes in sensory and multisensory function are increasingly recognized as a common phenotypic characteristic of Autism Spectrum Disorders (ASD). Furthermore, much recent evidence suggests that sensory disturbances likely play an important role in contributing to social communication weaknesses-one of the core diagnostic features of ASD. An established sensory disturbance observed in ASD is reduced audiovisual temporal acuity. In the current study, we substantially extend these explorations of multisensory temporal function within the framework that an inability to rapidly recalibrate to changes in audiovisual temporal relations may play an important and under-recognized role in ASD. In the paradigm, we present ASD and typically developing (TD) children and adolescents with asynchronous audiovisual stimuli of varying levels of complexity and ask them to perform a simultaneity judgment (SJ). In the critical analysis, we test audiovisual temporal processing on trial t as a condition of trial t - 1. The results demonstrate that individuals with ASD fail to rapidly recalibrate to audiovisual asynchronies in an equivalent manner to their TD counterparts for simple and non-linguistic stimuli (i.e., flashes and beeps, hand-held tools), but exhibit comparable rapid recalibration for speech stimuli. These results are discussed in terms of prior work showing a speech-specific deficit in audiovisual temporal function in ASD, and in light of current theories of autism focusing on sensory noise and stability of perceptual representations. Autism Res 2017, 10: 121-129. © 2016 International Society for Autism Research, Wiley Periodicals, Inc.

Keeping time in the brain: Autism spectrum disorder and audiovisual temporal processing.

A growing area of interest and relevance in the study of autism spectrum disorder (ASD) focuses on the relationship between multisensory temporal function and the behavioral, perceptual, and cognitive impairments observed in ASD. Atypical sensory processing is becoming increasingly recognized as a core component of autism, with evidence of atypical processing across a number of sensory modalities. These deviations from typical processing underscore the value of interpreting ASD within a multisensory framework. Furthermore, converging evidence illustrates that these differences in audiovisual processing may be specifically related to temporal processing. This review seeks to bridge the connection between temporal processing and audiovisual perception, and to elaborate on emerging data showing differences in audiovisual temporal function in autism. We also discuss the consequence of such changes, the specific impact on the processing of different classes of audiovisual stimuli (e.g. speech vs. nonspeech, etc.), and the presumptive brain processes and networks underlying audiovisual temporal integration. Finally, possible downstream behavioral implications, and possible remediation strategies are outlined. Autism Res 2016, 9: 720-738. © 2015 International Society for Autism Research, Wiley Periodicals, Inc.

Multisensory temporal integration in autism spectrum disorders.

The new DSM-5 diagnostic criteria for autism spectrum disorders (ASDs) include sensory disturbances in addition to the well-established language, communication, and social deficits. One sensory disturbance seen in ASD is an impaired ability to integrate multisensory information into a unified percept. This may arise from an underlying impairment in which individuals with ASD have difficulty perceiving the temporal relationship between cross-modal inputs, an important cue for multisensory integration. Such impairments in multisensory processing may cascade into higher-level deficits, impairing day-to-day functioning on tasks, such as speech perception. To investigate multisensory temporal processing deficits in ASD and their links to speech processing, the current study mapped performance on a number of multisensory temporal tasks (with both simple and complex stimuli) onto the ability of individuals with ASD to perceptually bind audiovisual speech signals. High-functioning children with ASD were compared with a group of typically developing children. Performance on the multisensory temporal tasks varied with stimulus complexity for both groups; less precise temporal processing was observed with increasing stimulus complexity. Notably, individuals with ASD showed a speech-specific deficit in multisensory temporal processing. Most importantly, the strength of perceptual binding of audiovisual speech observed in individuals with ASD was strongly related to their low-level multisensory temporal processing abilities. Collectively, the results represent the first to illustrate links between multisensory temporal function and speech processing in ASD, strongly suggesting that deficits in low-level sensory processing may cascade into higher-order domains, such as language and communication.

Developmental changes in the multisensory temporal binding window persist into adolescence

We live in a world rich in sensory information, and consequently the brain is challenged with deciphering which cues from the various sensory modalities belong together. Determinations regarding the relatedness of sensory information appear to be based, at least in part, on the spatial and temporal relationships between the stimuli. Stimuli that are presented in close spatial and temporal correspondence are more likely to be associated with one another and thus 'bound' into a single perceptual entity. While there is a robust literature delineating behavioral changes in perception induced by multisensory stimuli, maturational changes in multisensory processing, particularly in the temporal realm, are poorly understood. The current study examines the developmental progression of multisensory temporal function by analyzing responses on an audiovisual simultaneity judgment task in 6- to 23-year-old participants. The overarching hypothesis for the study was that multisensory temporal function will mature with increasing age, with the developmental trajectory for this change being the primary point of inquiry. Results indeed reveal an age-dependent decrease in the size of the 'multisensory temporal binding window', the temporal interval within which multisensory stimuli are likely to be perceptually bound, with changes occurring over a surprisingly protracted time course that extends into adolescence.

Altered auditory and multisensory temporal processing in autism spectrum disorders.

Autism spectrum disorders (ASD) are characterized by deficits in social reciprocity and communication, as well as by repetitive behaviors and restricted interests. Unusual responses to sensory input and disruptions in the processing of both unisensory and multisensory stimuli also have been reported frequently. However, the specific aspects of sensory processing that are disrupted in ASD have yet to be fully elucidated. Recent published work has shown that children with ASD can integrate low-level audiovisual stimuli, but do so over an extended range of time when compared with typically developing (TD) children. However, the possible contributions of altered unisensory temporal processes to the demonstrated changes in multisensory function are yet unknown. In the current study, unisensory temporal acuity was measured by determining individual thresholds on visual and auditory temporal order judgment (TOJ) tasks, and multisensory temporal function was assessed through a cross-modal version of the TOJ task. Whereas no differences in thresholds for the visual TOJ task were seen between children with ASD and TD, thresholds were higher in ASD on the auditory TOJ task, providing preliminary evidence for impairment in auditory temporal processing. On the multisensory TOJ task, children with ASD showed performance improvements over a wider range of temporal intervals than TD children, reinforcing prior work showing an extended temporal window of multisensory integration in ASD. These findings contribute to a better understanding of basic sensory processing differences, which may be critical for understanding more complex social and cognitive deficits in ASD, and ultimately may contribute to more effective diagnostic and interventional strategies.

Binding of sights and sounds: Age-related changes in multisensory temporal processing

We live in a multisensory world and one of the challenges the brain is faced with is deciding what information belongs together. Our ability to make assumptions about the relatedness of multisensory stimuli is partly based on their temporal and spatial relationships. Stimuli that are proximal in time and space are likely to be bound together by the brain and ascribed to a common external event. Using this framework we can describe multisensory processes in the context of spatial and temporal filters or windows that compute the probability of the relatedness of stimuli. Whereas numerous studies have examined the characteristics of these multisensory filters in adults and discrepancies in window size have been reported between infants and adults, virtually nothing is known about multisensory temporal processing in childhood. To examine this, we compared the ability of 10 and 11 year olds and adults to detect audiovisual temporal asynchrony. Findings revealed striking and asymmetric age-related differences. Whereas children were able to identify asynchrony as readily as adults when visual stimuli preceded auditory cues, significant group differences were identified at moderately long stimulus onset asynchronies (150–350ms) where the auditory stimulus was first. Results suggest that changes in audiovisual temporal perception extend beyond the first decade of life. In addition to furthering our understanding of basic multisensory developmental processes, these findings have implications on disorders (e.g., autism, dyslexia) in which emerging evidence suggests alterations in multisensory temporal function.

An extended multisensory temporal binding window in autism spectrum disorders.

Autism spectrum disorders (ASD) form a continuum of neurodevelopmental disorders, characterized by deficits in communication and reciprocal social interaction, as well as by repetitive behaviors and restricted interests. Sensory disturbances are also frequently reported in clinical and autobiographical accounts. However, surprisingly few empirical studies have characterized the fundamental features of sensory and multisensory processing in ASD. The current study is structured to test for potential differences in multisensory temporal function in ASD by making use of a temporally dependent, low-level multisensory illusion. In this illusion, the presentation of a single flash of light accompanied by multiple sounds often results in the illusory perception of multiple flashes. By systematically varying the temporal structure of the audiovisual stimuli, a “temporal window” within which these stimuli are likely to be bound into a single perceptual entity can be defined. The results of this study revealed that children with ASD report the flash-beep illusion over an extended range of stimulus onset asynchronies relative to children with typical development, suggesting that children with ASD have altered multisensory temporal function. These findings provide valuable new insights into our understanding of sensory processing in ASD and may hold promise for the development of more sensitive diagnostic measures and improved remediation strategies.

Perceptual Training Narrows the Temporal Window of Multisensory Binding

The brain's ability to bind incoming auditory and visual stimuli depends critically on the temporal structure of this information. Specifically, there exists a temporal window of audiovisual integration within which stimuli are highly likely to be bound together and perceived as part of the same environmental event. Several studies have described the temporal bounds of this window, but few have investigated its malleability. Here, the plasticity in the size of this temporal window was investigated using a perceptual learning paradigm in which participants were given feedback during a two-alternative forced choice (2-AFC) audiovisual simultaneity judgment task. Training resulted in a marked (i.e., approximately 40%) narrowing in the size of the window. To rule out the possibility that this narrowing was the result of changes in cognitive biases, a second experiment using a two-interval forced choice (2-IFC) paradigm was undertaken during which participants were instructed to identify a simultaneously presented audiovisual pair presented within one of two intervals. The 2-IFC paradigm resulted in a narrowing that was similar in both degree and dynamics to that using the 2-AFC approach. Together, these results illustrate that different methods of multisensory perceptual training can result in substantial alterations in the circuits underlying the perception of audiovisual simultaneity. These findings suggest a high degree of flexibility in multisensory temporal processing and have important implications for interventional strategies that may be used to ameliorate clinical conditions (e.g., autism, dyslexia) in which multisensory temporal function may be impaired.