Temporal Response Function Research Articles

Machine Learning Approaches to Analyze Speech-Evoked Neurophysiological Responses.

Purpose Speech-evoked neurophysiological responses are often collected to answer clinically and theoretically driven questions concerning speech and language processing. Here, we highlight the practical application of machine learning (ML)-based approaches to analyzing speech-evoked neurophysiological responses. Method Two categories of ML-based approaches are introduced: decoding models, which generate a speech stimulus output using the features from the neurophysiological responses, and encoding models, which use speech stimulus features to predict neurophysiological responses. In this review, we focus on (a) a decoding model classification approach, wherein speech-evoked neurophysiological responses are classified as belonging to 1 of a finite set of possible speech events (e.g., phonological categories), and (b) an encoding model temporal response function approach, which quantifies the transformation of a speech stimulus feature to continuous neural activity. Results We illustrate the utility of the classification approach to analyze early electroencephalographic (EEG) responses to Mandarin lexical tone categories from a traditional experimental design, and to classify EEG responses to English phonemes evoked by natural continuous speech (i.e., an audiobook) into phonological categories (plosive, fricative, nasal, and vowel). We also demonstrate the utility of temporal response function to predict EEG responses to natural continuous speech from acoustic features. Neural metrics from the 3 examples all exhibit statistically significant effects at the individual level. Conclusion We propose that ML-based approaches can complement traditional analysis approaches to analyze neurophysiological responses to speech signals and provide a deeper understanding of natural speech and language processing using ecologically valid paradigms in both typical and clinical populations.

Decoding listener’s attention :Can it be improved with behavioural measure of selective attention?

During the past decade, there has been growing interest in the neural correlates of selective attention to speech. In these studies, listeners were instructed to focus their attention on one of two concurrent speech streams. However, in everyday-life situations, a listener's attention can switch rapidly between different voices. Thus, we have developed a behavioural protocol to infer the dynamics of auditory attention over time. After listening to two simultaneous stories—a target and an interferer—the participants have to find, among a set of words, those present in the target story. The participant’s responses are then used to estimate, retrospectively, when their attention was directed toward the target, or toward the interferer. Neural data, recorded with EEG, and behavioural measures are combined to extract the brain's temporal response function in response to these stimuli. Moreover, to promote attention switches between the two voices, the interferers were uttered by the same talker as the target stories, but the voice parameters were manipulated to parametrically control the similarity of the two voices. We will discuss the results in terms of attentional selection and voice confusion, and suggest possible applications of this dynamic behavioral test of selective auditory attention.

Proactive neural processing of native and non-native speech

The impact of attention and language-experience in neural speech processing is typically assessed using sentences, words, and syllables that, when presented in isolation, may not engage the cortical speech network as well as more realistic continuous speech. Here, we explore the neuromodulatory effects of attention and language experience using continuous speech. We recorded electroencephalographic responses from native speakers of English and late Chinese-English bilinguals while listening to a story recorded in English. The story was mixed with a tone sequence, and listeners were instructed to focus either on the speech (attended speech condition) or the tone sequence (ignored speech condition). We used the multivariate temporal response function and the accuracy of a machine-learning based brain-to-speech decoder to quantify differences in cortical entrainment and speech-sound category processing. Our analyses revealed a more robust, context-independent neural encoding of speech-sound categories when they were attended and native. Interestingly, while cortical entrainment to speech was also enhanced by attention, the enhancement was stronger among non-native speakers. Our results suggest that while listeners can manage attention to improve the neural parsing of continuous speech via cortical entrainment, the benefits of attention for speech-sound category processing can be attenuated when the sounds are not native.The impact of attention and language-experience in neural speech processing is typically assessed using sentences, words, and syllables that, when presented in isolation, may not engage the cortical speech network as well as more realistic continuous speech. Here, we explore the neuromodulatory effects of attention and language experience using continuous speech. We recorded electroencephalographic responses from native speakers of English and late Chinese-English bilinguals while listening to a story recorded in English. The story was mixed with a tone sequence, and listeners were instructed to focus either on the speech (attended speech condition) or the tone sequence (ignored speech condition). We used the multivariate temporal response function and the accuracy of a machine-learning based brain-to-speech decoder to quantify differences in cortical entrainment and speech-sound category processing. Our analyses revealed a more robust, context-independent neural encoding of speech-sound categories when t...

Time- and angle-resolved photoemission spectroscopy of solids in the extreme ultraviolet at 500 kHz repetition rate.

Time- and angle-resolved photoemission spectroscopy (trARPES) employing a 500 kHz extreme-ultraviolet light source operating at 21.7 eV probe photon energy is reported. Based on a high-power ytterbium laser, optical parametric chirped pulse amplification, and ultraviolet-driven high-harmonic generation, the light source produces an isolated high-harmonic with 110 meV bandwidth and a flux of more than 1011 photons/s on the sample. Combined with a state-of-the-art ARPES chamber, this table-top experiment allows high-repetition rate pump-probe experiments of electron dynamics in occupied and normally unoccupied (excited) states in the entire Brillouin zone and with a temporal system response function below 40 fs.

Temporal imaging based on first-order field correlation

Different from second-order temporal ghost imaging usually realized by means of second-order correlation measurement, in this paper, we investigate theoretically temporal imaging with temporally thermal light via first-order field correlation based on a Mach-Zehnder interferometer. The paraxial wave equation describing the diffraction of light and the differential equation characterizing the dispersion of light pulse are given. Based on the similarity between these equations, the duality between the paraxial diffraction of the light in the spatial domain and the dispersion of the temporal narrow-band pulse in the dispersive medium (i.e. the space-time duality) is obtained, and the impulse response functions in the time domain for several optical systems are also presented. Then in terms of the space-time duality, we design the scheme for temporal imaging via first-order thermal field correlation based on a Mach-Zehnder interferometer and obtain the intensity expression for first-order temporal imaging according to the temporal impulse response functions, and discuss the influences of the source pulse width and coherence time on the image visibility and resolution. The result shows that the temporal signal can be reconstructed through temporal first-order temporal imaging. Furthermore, when the source’s coherence time is fixed, the image visibility decreases as the pulse width increases. However, the image resolution increases. When the source’s pulse width is fixed, the image visibility increases as the coherence time increases. And yet the image resolution decreases. Specially, when the source’s pulse width is 100 ps and the coherence time is 0.5 ps, the image quality (taking both the visibility and resolution into account) of a temporally rectangular object is satisfactory. In the simulation, the distance and width of the temporal rectangular object are 20 ps and 8 ps, respectively. It is shown that there is a dilemma between the visibility and resolution of first-order temporal imaging which is similar to the result of second-order ghost imaging. Our result discussed herein could be valuable in the reconstruction and detection of temporal signal via first-order temporal ghost imaging with temporally thermal light.

Fast-backward replay of sequentially memorized items in humans.

Storing temporal sequences of events (i.e., sequence memory) is fundamental to many cognitive functions. However, it is unknown how the sequence order information is maintained and represented in working memory and its behavioral significance, particularly in human subjects. We recorded electroencephalography (EEG) in combination with a temporal response function (TRF) method to dissociate item-specific neuronal reactivations. We demonstrate that serially remembered items are successively reactivated during memory retention. The sequential replay displays two interesting properties compared to the actual sequence. First, the item-by-item reactivation is compressed within a 200 - 400 ms window, suggesting that external events are associated within a plasticity-relevant window to facilitate memory consolidation. Second, the replay is in a temporally reversed order and is strongly related to the recency effect in behavior. This fast-backward replay, previously revealed in rat hippocampus and demonstrated here in human cortical activities, might constitute a general neural mechanism for sequence memory and learning.

Distortion-tolerated high-speed FBG demodulation method using temporal response of high-gain photodetector

Abstract A distortion-tolerated fiber Bragg grating (FBG) demodulation method for high-gain or high-speed sensing is proposed. The temporal response of photodetector (PD) on FBG reflection spectrum for different electric gains and wavelength scanning frequencies is investigated theoretically and experimentally. Simulation results show that high electric gains and high wavelength scanning frequencies will induce a tailed distortion on the right side of the FBG reflection peak output by the PD. A temporal response function of PD is presented to fit the peak with distortion and determine the peak position on FBG reflection spectrum. Experiments with varying degrees of FBG acquisition distortion are carried out. Compared with the traditional Gaussian fitting method, the measurement error of the temporal-response based method is limited within ±1.777 pm and the standard deviation is less than 0.940 pm, which are respectively improved by 6.87 times and 4.38 times at least.

Neural processing of task-irrelevant but salient object: an MEG study

In a cluttered visual environment, in addition to focusing on task-relevant objects, it is also important for attention to be occasionally captured to task-irrelevant but salient stimuli. Our previous EEG study has revealed that attention dynamically coordinates neural representations of multiple objects and flexibly modulates the sampling profile to manage different task demands (Jia et al., 2017). Meanwhile, it remains unclear how the brain deals with task-irrelevant but salient objects and further registers them into the task framework. To address the issue, we recorded magnetoencephalography (MEG) signals from 20 human subjects while they were presented with a multi-object visual display and performed a central fixation task. Notably, one of the items was made salient by being distinct from the others (e.g., red among green, etc.) but was completely task-irrelevant (i.e., demanding central fixation task). Next, we employed a temporal response function (TRF) approach (Liu et al., in press) to dissociate the time-resolved neuronal response that specifically tracks the salient (S-TRF) and non-salient (NS-TRF) items from the same MEG signals. We then performed source analysis on the S-TRF and NS-TRF responses to examine the fine spatiotemporal neuronal profiles. Our data demonstrates that salient object, compared to the non-salient one, first initiated response in right temporoparietal area (TPJ) at about 120 ms, followed by activation in left intraparietal sulcus (IPS) and bilateral frontal area, and finally reached early visual area (EVA). Interestingly, after the information was conveyed back to EVA, salient object showed higher alpha-band response compared to the non-salient one, suggesting a registration process during which the salient but task-irrelevant item was further inhibited. Our results support the crucial role of dorsal attentional network in saliency representation and provide a temporally precise and whole-brain description of how the salient information is processed, transferred, and integrated into the current task context. Meeting abstract presented at VSS 2018

Broadband time-resolved multi-channel functional near-infrared spectroscopy system to monitor in vivo physiological changes of human brain activity.

We have developed a broadband time-resolved multi-channel near-infrared spectroscopy system that can monitor the physiological responses of the adult human brain. This system is composed of a supercontinuum laser for the source part and of an intensified charge-coupled device camera coupled with an imaging spectrometer for the detection part. It allows the detection of the spectral, from 600 to 900nm, and spatial dimensions as well as the arrival time of photon information simultaneously. We describe the setup and its characterization in terms of temporal instrument response function, wavelength sensitivity, and stability. The ability of the system to detect the hemodynamic response is then demonstrated. First, an in vivo experiment on an adult volunteer was performed to monitor the response in the arm during a cuff occlusion. Second, the response in the brain during a cognitive task was monitored on a group of five healthy volunteers. Moreover, looking at the response at different time windows, we could monitor the hemodynamic response in depth, enhancing the detection of the cortical activation. Those first results demonstrate the ability of our system to discriminate between the responses of superficial and deep tissues, addressing an important issue in functional near-infrared spectroscopy.

Response of a human visual system to continuous color variation: An EEG-based approach

Abstract Color is a powerful descriptor that considerably expands our ability to characterize and distinguish objects and facilitates interactions with dynamic environment. Although our understanding of neuronal mechanisms involved in color processing has significantly advanced over the past decades, a number of questions related to visual pathways engaged in color perception still remain unanswered. We propose a novel method of an EEG-based study of human color perception utilizing multivariate Temporal Response Functions (mTRFs) that quantify spatiotemporal brain responses to sensory stimuli. Such functions can be derived from the Electroencephalogram (EEG) recorded from a subject exposed to the screen of a color varying over time. The results of a pilot study that included nineteen subjects with normal color vision indicate that mTRFs may be instrumental in characterizing the dorsal stream of the visual pathway. We conclude that pronounced local extrema observed in mTRFs should be attributed to the colors perceived by the participant rather than to the post-stimulus latencies. No interhemispheric asymmetry in mTRFs may be an evidence of a similar processing of color information by both cerebral hemispheres. On the other hand, observed dorsal dependence of mTRFs may suggest that different areas of the same hemisphere show stronger response to specific colors (hues) of the stimuli. The latter may indicate, for instance, differences in the functional “color specialization” between the occipital and frontal regions.

Extrinsic cone-mediated post-receptoral noise inhibits the rod temporal impulse response function.

We determined how extrinsic white noise correlating with cone inputs to the three primary visual pathways affects both rod-pathway temporal contrast sensitivity and the impulse response function. A four-primary photostimulator provided independent control of rod and cone photoreceptor excitations under mesopic illumination (20 photopic Td). We show that rod-pathway temporal contrast sensitivity uniformly decreases across all temporal frequencies in the presence of cone noise correlating with the inferred magnocellular, parvocellular, or koniocellular pathways. The rod-pathway temporal impulse response functions derived using the Stork-Falk procedure (with a minimum phase assumption) had lower amplitudes in the pathway-specific cone noise. Therefore, cone noise impairs rod-pathway temporal contrast sensitivity without delaying rod-pathway signal transmission.

Neural encoding of phonetic features relates to phonological processing in PPA

Neural encoding of phonetic features relates to phonological processing in PPA

Perceptual integration rapidly activates dorsal visual pathway to guide local processing in early visual areas.

Rapidly grouping local elements into an organized object (i.e., perceptual integration) is a fundamental yet challenging task, especially in noisy contexts. Previous studies demonstrate that ventral visual pathway, which is widely known to mediate object recognition, engages in the process by conveying object-level information processed in high-level areas to modulate low-level sensory areas. Meanwhile, recent evidence suggests that the dorsal visual pathway, which is not typically attributable to object recognition, is also involved in the process. However, the underlying whole-brain fine spatiotemporal neuronal dynamics remains unknown. Here we used magnetoencephalography (MEG) recordings in combination with a temporal response function (TRF) approach to dissociate the time-resolved neuronal response that specifically tracks the perceptual grouping course. We demonstrate that perceptual integration initiates robust and rapid responses along the dorsal visual pathway in a reversed hierarchical manner, faster than the ventral pathway. Specifically, the anterior intraparietal sulcus (IPS) responds first (i.e., within 100 ms), followed by activities backpropagating along the dorsal pathway to early visual areas (EVAs). The IPS activity causally modulates the EVA response, even when the global form information is task-irrelevant. The IPS-to-EVA response profile fails to appear when the global form could not be perceived. Our results support the crucial function of the dorsal visual pathway in perceptual integration, by quickly extracting a coarse global template (i.e., an initial object representation) within first 100 ms to guide subsequent local sensory processing so that the ambiguities in the visual inputs can be efficiently resolved.

Multiple visual objects are sampled sequentially.

When acting in a complex visual environment, it is essential to be able to flexibly allocate attention to parts of the visual scene that may contain goal-relevant information. The paper by Jia et al. provides novel evidence that our brains sequentially sample different objects in a visual scene. The results were obtained using “temporal response functions,” in which unique electroencephalographic (EEG) signals corresponding to the processing of 2 continuously presented objects were isolated in an object-specific way. These response functions were dominated by 10-Hz alpha-band activity. Crucially, the different objects were sequentially sampled at a rate of about 2 Hz. These findings provide important neurophysiological insights into how our visual system operates in complex environments.

Sequential sampling of visual objects during sustained attention.

In a crowded visual scene, attention must be distributed efficiently and flexibly over time and space to accommodate different contexts. It is well established that selective attention enhances the corresponding neural responses, presumably implying that attention would persistently dwell on the task-relevant item. Meanwhile, recent studies, mostly in divided attentional contexts, suggest that attention does not remain stationary but samples objects alternately over time, suggesting a rhythmic view of attention. However, it remains unknown whether the dynamic mechanism essentially mediates attentional processes at a general level. Importantly, there is also a complete lack of direct neural evidence reflecting whether and how the brain rhythmically samples multiple visual objects during stimulus processing. To address these issues, in this study, we employed electroencephalography (EEG) and a temporal response function (TRF) approach, which can dissociate responses that exclusively represent a single object from the overall neuronal activity, to examine the spatiotemporal characteristics of attention in various attentional contexts. First, attention, which is characterized by inhibitory alpha-band (approximately 10 Hz) activity in TRFs, switches between attended and unattended objects every approximately 200 ms, suggesting a sequential sampling even when attention is required to mostly stay on the attended object. Second, the attentional spatiotemporal pattern is modulated by the task context, such that alpha-mediated switching becomes increasingly prominent as the task requires a more uniform distribution of attention. Finally, the switching pattern correlates with attentional behavioral performance. Our work provides direct neural evidence supporting a generally central role of temporal organization mechanism in attention, such that multiple objects are sequentially sorted according to their priority in attentional contexts. The results suggest that selective attention, in addition to the classically posited attentional “focus,” involves a dynamic mechanism for monitoring all objects outside of the focus. Our findings also suggest that attention implements a space (object)-to-time transformation by acting as a series of concatenating attentional chunks that operate on 1 object at a time.

Homeostatic Plasticity Shapes Cell-Type-Specific Wiring in the Retina

Convergent input from different presynaptic partners shapes the responses of postsynaptic neurons. Whether developing postsynaptic neurons establish connections with each presynaptic partner independently or balance inputs to attain specific responses is unclear. Retinal ganglion cells (RGCs) receive convergent input from bipolar cell types with different contrast responses and temporal tuning. Here, using optogenetic activation and pharmacogenetic silencing, we found that type 6 bipolar (B6) cells dominate excitatory input to ONα-RGCs. We generated mice in which B6 cells were selectively removed from developing circuits (B6-DTA). In B6-DTA mice, ONα-RGCs adjusted connectivity with other bipolar cells in a cell-type-specific manner. They recruited new partners, increased synapses with some existing partners, and maintained constant input from others. Patch-clamp recordings revealed that anatomical rewiring precisely preserved contrast and temporal frequency response functions of ONα-RGCs, indicating that homeostatic plasticity shapes cell-type-specific wiring in the developing retina to stabilize visual information sent to the brain.

Relationship between the Rotating Snakes illusion and the temporal impulse response function in the visual system(Summary of Awarded Presentation at the 26th Annual Meeting)

Relationship between the Rotating Snakes illusion and the temporal impulse response function in the visual system(Summary of Awarded Presentation at the 26th Annual Meeting)

The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli.

Understanding how brains process sensory signals in natural environments is one of the key goals of twenty-first century neuroscience. While brain imaging and invasive electrophysiology will play key roles in this endeavor, there is also an important role to be played by noninvasive, macroscopic techniques with high temporal resolution such as electro- and magnetoencephalography. But challenges exist in determining how best to analyze such complex, time-varying neural responses to complex, time-varying and multivariate natural sensory stimuli. There has been a long history of applying system identification techniques to relate the firing activity of neurons to complex sensory stimuli and such techniques are now seeing increased application to EEG and MEG data. One particular example involves fitting a filter—often referred to as a temporal response function—that describes a mapping between some feature(s) of a sensory stimulus and the neural response. Here, we first briefly review the history of these system identification approaches and describe a specific technique for deriving temporal response functions known as regularized linear regression. We then introduce a new open-source toolbox for performing this analysis. We describe how it can be used to derive (multivariate) temporal response functions describing a mapping between stimulus and response in both directions. We also explain the importance of regularizing the analysis and how this regularization can be optimized for a particular dataset. We then outline specifically how the toolbox implements these analyses and provide several examples of the types of results that the toolbox can produce. Finally, we consider some of the limitations of the toolbox and opportunities for future development and application.

Dynamic Estimation of the Auditory Temporal Response Function From MEG in Competing-Speaker Environments.

A central problem in computational neuroscience is to characterize brain function using neural activity recorded from the brain in response to sensory inputs with statistical confidence. Most of existing estimation techniques, such as those based on reverse correlation, exhibit two main limitations: first, they are unable to produce dynamic estimates of the neural activity at a resolution comparable with that of the recorded data, and second, they often require heavy averaging across time as well as multiple trials in order to construct statistical confidence intervals for a precise interpretation of data. In this paper, we address the above-mentioned issues for estimating auditory temporal response function (TRF) as a parametric computational model for selective auditory attention in competing-speaker environments. The TRF is a sparse kernel which regresses auditory MEG data with respect to the envelopes of the speech streams. We develop an efficient estimation technique by exploiting the sparsity of the TRF and adopting an ℓ1-regularized least squares estimator which is capable of producing dynamic TRF estimates as well as confidence intervals at sampling resolution from single-trial MEG data. We evaluate the performance of our proposed estimator using evoked MEG responses from the human brain in an auditory attention experiment with two competing speakers. The TRFs are estimated dynamically over time using the proposed technique with multisecond resolution, which is a significant improvement over previous results with a temporal resolution of the order of a minute. Application of our method to MEG data reveals a precise characterization of the modulation of M50 and M100 evoked responses with respect to the attentional state of the subject at multisecond resolution. Our proposed estimation technique provides a high resolution real-time attention decoding framework in multispeaker environments with potential application in smart hearing aid technology.

Temporal impulse response function of the visual system estimated from ocular following responses in humans

Early visual processing functions as a set of spatiotemporal image filters. Our ability to sense changes in retinal images is determined by these filters along the temporal axis. In this study, we developed a paradigm to identify the kernel of the temporal filters based on ocular following responses (OFRs) to two-frame apparent motion stimuli. We first conducted two experiments to acquire fundamental data. In the first experiment, in which a quarter wavelength step of a sinusoidal grating was presented with various inter-stimulus intervals (ISIs), we found that OFRs were reversed by the ISI, which is consistent with previous findings. In the second experiment, a quarter wavelength step of a sinusoidal grating was applied with various durations of the initial image frame (motion onset delays; MODs); we found that longer exposure to the initial image reduced OFRs. Parameters of motion energy model involving temporal filters were optimized so that the model could reproduce the dependence of OFRs on ISIs and MODs. We were then able to successfully obtain quantitative estimates of the biphasic temporal filters with optimal frequencies in 6–8Hz. This method is completely objective and will thus be applicable to a wide range of human subjects and model animals.