Temporal Window Of Integration Research Articles

Audiovisual speech asynchrony asymmetrically modulates neural binding

Previous psychophysical and neurophysiological studies in young healthy adults have provided evidence that audiovisual speech integration occurs with a large degree of temporal tolerance around true simultaneity. To further determine whether audiovisual speech asynchrony modulates auditory cortical processing and neural binding in young healthy adults, N1/P2 auditory evoked responses were compared using an additive model during a syllable categorization task, without or with an audiovisual asynchrony ranging from 240 ms visual lead to 240 ms auditory lead. Consistent with previous psychophysical findings, the observed results converge in favor of an asymmetric temporal integration window. Three main findings were observed: 1) predictive temporal and phonetic cues from pre-phonatory visual movements before the acoustic onset appeared essential for neural binding to occur, 2) audiovisual synchrony, with visual pre-phonatory movements predictive of the onset of the acoustic signal, was a prerequisite for N1 latency facilitation, and 3) P2 amplitude suppression and latency facilitation occurred even when visual pre-phonatory movements were not predictive of the acoustic onset but the syllable to come. Taken together, these findings help further clarify how audiovisual speech integration partly operates through two stages of visually-based temporal and phonetic predictions.

EXPRESS: When 2 become 1: autistic simultaneity judgements about asynchronous audiovisual speech.

It has been proposed that autistic people experience a temporal distortion whereby the temporal binding window of multisensory integration is extended. Research to date has focused on autistic children so whether these differences persist into adulthood remains unknown. Additionally, the possibility that the previous observations have arisen from between group differences in response bias, rather than perceptual differences, has not been addressed.Participants completed simultaneity judgements of audiovisual speech stimuli at a range of stimulus onset asynchronies. Response times and accuracy data were fitted to a drift diffusion model so that the drift rate (a measure of processing efficiency) and starting point (response bias) could be estimated. In Experiment 1, we tested a sample of non-autistic adults who completed the Autism Quotient questionnaire. Autism Quotient score was not correlated with either drift rate or response bias, nor were there between group differences when splitting based on the 1st and 3rd quantiles of scores. In Experiment 2, we compared the performance of autistic with a group of non-autistic adults. There were no between group differences in either drift rate or starting point.The results of the present study do not support the previous suggestion that autistic people have an extended temporal binding window for audiovisual speech. Additionally, exploratory analysis revealed that operationalising the temporal binding window in different ways influenced whether a group difference was observed, which is an important consideration for future work .

Fast binaural processing but sluggish masker representation reconfiguration.

Perceptual organization of complex acoustic scenes requires fast binaural processing for accurate localization or lateralization based on short single-source-dominated glimpses. This sensitivity also manifests in the ability to detect rapid oscillating interaural time and phase differences as well as interaural correlation. However, binaural processing has also been termed "sluggish" based on experiments that require binaural detection in a masker with an additional binaural cue change in temporal proximity. The present study shows that the temporal integration windows obtained from data on binaural sluggishness cannot account for the detection of rapid binaural oscillations. A model with fast IPD encoding but a slower process of updating the internal representation of the masker IPD statistics accounted for both experiments of the "fast" and the "sluggish" categories.

Achieving nanoscale precision using neuromorphic localization microscopy.

Neuromorphic cameras are a new class of dynamic-vision-inspired sensors that encode the rate of change of intensity as events. They can asynchronously record intensity changes as spikes, independent of the other pixels in the receptive field, resulting in sparse measurements. This recording of such sparse events makes them ideal for imaging dynamic processes, such as the stochastic emission of isolated single molecules. Here we show the application of neuromorphic detection to localize nanoscale fluorescent objects below the diffraction limit, with a precision below 20 nm. We demonstrate a combination of neuromorphic detection with segmentation and deep learning approaches to localize and track fluorescent particles below 50 nm with millisecond temporal resolution. Furthermore, we show that combining information from events resulting from the rate of change of intensities improves the classical limit of centroid estimation of single fluorescent objects by nearly a factor of two. Additionally, we validate that using post-processed data from the neuromorphic detector at defined windows of temporal integration allows a better evaluation of the fractalized diffusion of single particle trajectories. Our observations and analysis is useful for event sensing by nonlinear neuromorphic devices to ameliorate real-time particle localization approaches at the nanoscale.

Differences in temporal processing speeds between the right and left auditory cortex reflect the strength of recurrent synaptic connectivity.

Brain asymmetry in the sensitivity to spectrotemporal modulation is an established functional feature that underlies the perception of speech and music. The left auditory cortex (ACx) is believed to specialize in processing fast temporal components of speech sounds, and the right ACx slower components. However, the circuit features and neural computations behind these lateralized spectrotemporal processes are poorly understood. To answer these mechanistic questions we use mice, an animal model that captures some relevant features of human communication systems. In this study, we screened for circuit features that could subserve temporal integration differences between the left and right ACx. We mapped excitatory input to principal neurons in all cortical layers and found significantly stronger recurrent connections in the superficial layers of the right ACx compared to the left. We hypothesized that the underlying recurrent neural dynamics would exhibit differential characteristic timescales corresponding to their hemispheric specialization. To investigate, we recorded spike trains from awake mice and estimated the network time constants using a statistical method to combine evidence from multiple weak signal-to-noise ratio neurons. We found longer temporal integration windows in the superficial layers of the right ACx compared to the left as predicted by stronger recurrent excitation. Our study shows substantial evidence linking stronger recurrent synaptic connections to longer network timescales. These findings support speech processing theories that purport asymmetry in temporal integration is a crucial feature of lateralization in auditory processing.

Musical training refines audiovisual integration but does not influence temporal recalibration

When the brain is exposed to a temporal asynchrony between the senses, it will shift its perception of simultaneity towards the previously experienced asynchrony (temporal recalibration). It is unknown whether recalibration depends on how accurately an individual integrates multisensory cues or on experiences they have had over their lifespan. Hence, we assessed whether musical training modulated audiovisual temporal recalibration. Musicians (n = 20) and non-musicians (n = 18) made simultaneity judgements to flash-tone stimuli before and after adaptation to asynchronous (± 200 ms) flash-tone stimuli. We analysed these judgements via an observer model that described the left and right boundaries of the temporal integration window (decisional criteria) and the amount of sensory noise that affected these judgements. Musicians’ boundaries were narrower (closer to true simultaneity) than non-musicians’, indicating stricter criteria for temporal integration, and they also exhibited enhanced sensory precision. However, while both musicians and non-musicians experienced cumulative and rapid recalibration, these recalibration effects did not differ between the groups. Unexpectedly, cumulative recalibration was caused by auditory-leading but not visual-leading adaptation. Overall, these findings suggest that the precision with which observers perceptually integrate audiovisual temporal cues does not predict their susceptibility to recalibration.

Perceptual training narrows the temporal binding window of audiovisual integration in both younger and older adults

There is a growing body of evidence to suggest that multisensory processing changes with advancing age - usually in the form of an enlarged temporal binding window - with some studies linking these multisensory changes to negative clinical outcomes. Perceptual training regimes represent a promising means for enhancing the precision of multisensory integration in ageing; however, to date, the vast majority of studies examining the efficacy of multisensory perceptual learning have focused solely on healthy young adults. Here, we measured the temporal binding windows of younger and older participants before and after training on an audiovisual temporal discrimination task to assess (i) how perceptual training affected the shape of the temporal binding window and (ii) whether training effects were similar in both age groups. Our results replicated previous findings of an enlarged temporal binding window in older adults, as well as providing further evidence that both younger and older participants can improve the precision of their audiovisual timing estimation via perceptual training. We also show that this training protocol led to a narrowing of the temporal binding window associated with the sound-induced flash illusion in both age groups indicating a general refinement of audiovisual integration. However, while younger adults also displayed a general reduction in crossmodal interactions following training, this effect was not observed in the older adult group. Together, our results suggest that perceptual training narrows the temporal binding window of audiovisual integration in both younger and older adults but has less of an impact on prior expectations regarding the source of audiovisual signals in older adults.

Multiple timescales of sensory-evidence accumulation across the dorsal cortex.

Cortical areas seem to form a hierarchy of intrinsic timescales, but the relevance of this organization for cognitive behavior remains unknown. In particular, decisions requiring the gradual accrual of sensory evidence over time recruit widespread areas across this hierarchy. Here, we tested the hypothesis that this recruitment is related to the intrinsic integration timescales of these widespread areas. We trained mice to accumulate evidence over seconds while navigating in virtual reality and optogenetically silenced the activity of many cortical areas during different brief trial epochs. We found that the inactivation of all tested areas affected the evidence-accumulation computation. Specifically, we observed distinct changes in the weighting of sensory evidence occurring during and before silencing, such that frontal inactivations led to stronger deficits on long timescales than posterior cortical ones. Inactivation of a subset of frontal areas also led to moderate effects on behavioral processes beyond evidence accumulation. Moreover, large-scale cortical Ca2+ activity during task performance displayed different temporal integration windows. Our findings suggest that the intrinsic timescale hierarchy of distributed cortical areas is an important component of evidence-accumulation mechanisms.

Task demands modulate pre-stimulus alpha frequency and sensory template during bistable apparent motion perception.

Despite ambiguous environmental inputs, top-down attention biases our subjective perception toward the preferred percepts, via modulating prestimulus neural activity or inducing prestimulus sensory templates that carry concrete internal sensory representations of the preferred percepts. In contrast to frequent changes of behavioral goals in the typical cue-target paradigm, human beings are often engaged in a prolonged task state with only 1 specific behavioral goal. It remains unclear how prestimulus neural signals and sensory templates are modulated in the latter case. To answer this question in the present electroencephalogram study on human subjects, we manipulated sustained task demands toward one of the 2 possible percepts in the bistable Ternus display, emphasizing either temporal integration or segregation. First, the prestimulus peak alpha frequency, which gated the temporal window of temporal integration, was effectively modulated by task demands. Furthermore, time-resolved decoding analyses showed that task demands biased neural representations toward the preferred percepts after the full presentation of bottom-up stimuli. More importantly, sensory templates resembling the preferred percepts emerged even before the bottom-up sensory evidence were sufficient enough to induce explicit percepts. Taken together, task demands modulate both prestimulus alpha frequency and sensory templates, to eventually bias subjective perception toward the preferred percepts.

Effect of the Temporal Window of Integration of Speech Sound on Mismatch Negativity.

Speech-sound stimuli have a complex structure, and it is unclear how the brain processes them. An event-related potential (ERP), known as mismatch negativity (MMN), is elicited when an individual's brain detects a rare sound. In this study, MMNs were measured in response to an omitted segment of a complex sound consisting of a Japanese vowel. The results indicated that the latency from onset in the right hemisphere was significantly shorter than that in the frontal midline and left hemispheres during left ear stimulation. Additionally, the results of latency from omission showed that the latency of stimuli omitted in the latter part of the temporal window of integration (TWI) was longer than that of stimuli omitted in the first part of the TWI. The mean peak amplitude was found to be higher in the right hemisphere than in the frontal midline and left hemispheres in response to left ear stimulation. In conclusion, the results of this study suggest that would be incorrect to believe that the stimuli have strictly the characteristics of speech-sound. However. the results of the interaction effect in the latencies from omission were insignificant. These results suggest that the detection time for deviance may not be related to the stimulus ear. However, the type of deviant stimuli on latencies was found to be significant. This is because the detection of the deviants was delayed when a deviation occurred in the latter part of the TWI, regardless of the stimulation of the ear.

Towards auditory stream segregation in Tursiops truncatus

In humans, understanding a voice amidst competing sounds depends on parsing the sound mixture into “streams” representing each source's content. Streaming can be influenced by top-down attentional focus, while acoustic features can affect streaming percepts through bottom-up, automatic processing of pitch, timbre, and location. Dolphins regularly navigate a cacophony of echoes generated during echolocation, conspecific echolocation clicks, and other, passively received sounds (eg. communication signals, environmental noise), and thus, may rely on streaming. Using auditory evoked potentials (AEPs), we asked whether dolphins exhibit evidence of bottom-up, frequency-based stream segregation. Initial results using a classic A-B-A sequence of repeated tone triplets (either low-high-low or high-low-high tones) suggest that the triplets break apart perceptually into low and high streams; specifically, the AEP magnitude evoked by the middle tone increases with increasing frequency separation. Differences in dolphin hearing sensitivity across the frequencies tested seem to account for some of the frequency manipulation effect. Additionally, earlier studies suggest that the dolphin auditory temporal integration window decreases as frequency increases, which may interact with streaming processes. This lays a foundation for future tests of top-down effects on dolphin auditory stream processing such as attention and expectation.

Variation in the temporal window of integration with changing the sound pressure level of a sound stimulus

Variation in the temporal window of integration with changing the sound pressure level of a sound stimulus

Adaptive integration of self-motion and goals in posterior parietal cortex.

SUMMARYRats readily switch between foraging and more complex navigational behaviors such as pursuit of other rats or prey. These tasks require vastly different tracking of multiple behaviorally significant variables including self-motion state. To explore whether navigational context modulates self-motion tracking, we examined self-motion tuning in posterior parietal cortex neurons during foraging versus visual target pursuit. Animals performing the pursuit task demonstrate predictive processing of target trajectories by anticipating and intercepting them. Relative to foraging, pursuit yields multiplicative gain modulation of self-motion tuning and enhances self-motion state decoding. Self-motion sensitivity in parietal cortex neurons is, on average, history dependent regardless of behavioral context, but the temporal window of self-motion integration extends during target pursuit. Finally, many self-motion-sensitive neurons conjunctively track the visual target position relative to the animal. Thus, posterior parietal cortex functions to integrate the location of navigationally relevant target stimuli into an ongoing representation of past, present, and future locomotor trajectories.

Collision Narrows the Temporal Binding Window of Multisensory Integration

Collision Narrows the Temporal Binding Window of Multisensory Integration

Neural readout of a latency code in the active electrosensory system.

In briefIn many systems, information is conveyed by the precise latency of spikes relative to a sensory stimulus. Perks and Sawtell use intracellular recordings and modeling to reveal how such a latency code is read out by neurons in the active electrosensory system of fish.

Spatio-temporal Integration of Speech Reflections in Hearing-Impaired Listeners.

Speech recognition in rooms requires the temporal integration of reflections which arrive with a certain delay after the direct sound. It is commonly assumed that there is a certain temporal window of about 50-100 ms, during which reflections can be integrated with the direct sound, while later reflections are detrimental to speech intelligibility. This concept was challenged in a recent study by employing binaural room impulse responses (RIRs) with systematically varied interaural phase differences (IPDs) and amplitude of the direct sound and a variable number of reflections delayed by up to 200 ms. When amplitude or IPD favored late RIR components, normal-hearing (NH) listeners appeared to be capable of focusing on these components rather than on the precedent direct sound, which contrasted with the common concept of considering early RIR components as useful and late components as detrimental. The present study investigated speech intelligibility in the same conditions in hearing-impaired (HI) listeners. The data indicate that HI listeners were generally less able to "ignore" the direct sound than NH listeners, when the most useful information was confined to late RIR components. Some HI listeners showed a remarkable inability to integrate across multiple reflections and to optimally "shift" their temporal integration window, which was quite dissimilar to NH listeners. This effect was most pronounced in conditions requiring spatial and temporal integration and could provide new challenges for individual prediction models of binaural speech intelligibility.

Reinforcer pathology's alternative reinforcer hypothesis: A preliminary examination.

The novel theory of Reinforcer Pathology states that the temporal window of integration, measured by delay discounting (DD), determines the value of reinforcers. Based on the Reinforcer Pathology theory, the valuation of alternative reinforcers is positively associated with the length of the temporal window (negatively with rates of discounting). The objective of this article is to test the Reinforcer Pathology theory by reanalyzing data from a prior report (Carr & Epstein, 2018) in order to examine the association between the reinforcing value of alternative reinforcement and the length of the measured temporal window. Participants (N = 250) who completed an adjusting delay discounting task and the Pleasant Events Schedule (PES) were included in the study. PES provides a measurement of a reinforcement score (a cross-product of PES measures of frequency and pleasantness) for 45 noneating alternative activities. Analysis of Variance and Spearman product-moment correlations were completed. Delay discounting was significantly positively correlated with sedentary reinforcement and significantly negatively correlated with cognitively enriching activities. A significant interaction was observed in the preference for between food and cognitive-enriching context as a function ln(k) in predicting the cognitive-enriching activity class. Nonsignificant relationships were also observed with the correlation of delay discounting to the reinforcement of social and physical activities. These findings provide initial support for Reinforcer Pathology theory and indicate that the temporal window over which reinforcers are integrated may be a factor contributing to a preference for some healthy or unhealthy alternative reinforcers. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Temporal integration window of the double-drift illusion: an immediate effect of the internal motion

Temporal integration window of the double-drift illusion: an immediate effect of the internal motion

The temporal integration windows for visual mirror symmetry

Symmetry perception in dot patterns is tolerant to temporal delays of up to 60 ms within and between element pairs. However, it is not known how factors effecting symmetry discrimination in static patterns might affect temporal integration in dynamic patterns. One such feature is luminance polarity. Using dynamic stimuli with increasing temporal delay (SOA) between the onset of the first and second element in a symmetric pair, we investigated how four different luminance-polarity conditions affected the temporal integration of symmetric patterns. All four luminance polarity conditions showed similar upper temporal limits of approximately 60 ms. However psychophysical performance over all delay durations showed significantly higher symmetry thresholds for unmatched-polarity patterns at short delays, but also significantly less sensitivity to increasing temporal delay relative to matched-polarity patterns. These varying temporal windows are consistent with the involvement of a fast, sensitive first-order mechanism for matched-polarity patterns, and a slower, more robust second-order mechanism for unmatched-polarity patterns. Temporal integration windows for unmatched-polarity patterns were not consistent with performance expected from attentional mechanisms alone, and instead supports the involvement of second-order mechanisms that combines information from ON and OFF channels.

Surmising synchrony of sound and sight: Factors explaining variance of audiovisual integration in hurdling, tap dancing and drumming.

Auditory and visual percepts are integrated even when they are not perfectly temporally aligned with each other, especially when the visual signal precedes the auditory signal. This window of temporal integration for asynchronous audiovisual stimuli is relatively well examined in the case of speech, while other natural action-induced sounds have been widely neglected. Here, we studied the detection of audiovisual asynchrony in three different whole-body actions with natural action-induced sounds-hurdling, tap dancing and drumming. In Study 1, we examined whether audiovisual asynchrony detection, assessed by a simultaneity judgment task, differs as a function of sound production intentionality. Based on previous findings, we expected that auditory and visual signals should be integrated over a wider temporal window for actions creating sounds intentionally (tap dancing), compared to actions creating sounds incidentally (hurdling). While percentages of perceived synchrony differed in the expected way, we identified two further factors, namely high event density and low rhythmicity, to induce higher synchrony ratings as well. Therefore, we systematically varied event density and rhythmicity in Study 2, this time using drumming stimuli to exert full control over these variables, and the same simultaneity judgment tasks. Results suggest that high event density leads to a bias to integrate rather than segregate auditory and visual signals, even at relatively large asynchronies. Rhythmicity had a similar, albeit weaker effect, when event density was low. Our findings demonstrate that shorter asynchronies and visual-first asynchronies lead to higher synchrony ratings of whole-body action, pointing to clear parallels with audiovisual integration in speech perception. Overconfidence in the naturally expected, that is, synchrony of sound and sight, was stronger for intentional (vs. incidental) sound production and for movements with high (vs. low) rhythmicity, presumably because both encourage predictive processes. In contrast, high event density appears to increase synchronicity judgments simply because it makes the detection of audiovisual asynchrony more difficult. More studies using real-life audiovisual stimuli with varying event densities and rhythmicities are needed to fully uncover the general mechanisms of audiovisual integration.