Spatiotemporal Receptive Fields Research Articles

Few-shot bioacoustic event detection utilizing spectro-temporal receptive field

Bioacoustic event detection is a task of identifying specific animal sounds within a biological audio recordings. Specialists only can label temporal information of animal sounds since knowledge for biological sound is demanded for labeling and they can give only few annotations due to high labor intensity caused by large duration of biological recordings. Therefore, the task is set to few-shot learning scheme and method called prototypical network effectively deal with the few-shot learning scheme by learning representation space which represent each class for given few examples. In this work, we utilized spectro-temporal receptive field, which is inspired auditory cortex which responses actively to certain spectro-temporal modulation, as a convolutional layer kernel of prototypical network. Bioacoustic events retain plentiful spectro-temporal modulation that STRF is expected to capture animal sounds effectively. Also, STRF kernels are constructed to fixed shape rather than trained that a model utilizing STRF kernels would learn representation space with less parameter. We built a model called Two Branch STRFNet (TB-STRFNet), in which STRF branch captures spectro-temporal modulation by STRF kernels and the other branch captures detailed time-frequency information which can be smashed in STRF branch. TB-STRFNet outperformed other models showing that effectiveness of auditory system inspired method.

Correction to: A statistical significance test for spatio-temporal receptive field estimates obtained using spike-triggered averaging of binary pseudo-random sequences

Correction to: A statistical significance test for spatio-temporal receptive field estimates obtained using spike-triggered averaging of binary pseudo-random sequences

Receptive-field nonlinearities in primary auditory cortex: a comparative perspective.

Cortical processing of auditory information can be affected by interspecies differences as well as brain states. Here we compare multifeature spectro-temporal receptive fields (STRFs) and associated input/output functions or nonlinearities (NLs) of neurons in primary auditory cortex (AC) of four mammalian species. Single-unit recordings were performed in awake animals (female squirrel monkeys, female, and male mice) and anesthetized animals (female squirrel monkeys, rats, and cats). Neuronal responses were modeled as consisting of two STRFs and their associated NLs. The NLs for the STRF with the highest information content show a broad distribution between linear and quadratic forms. In awake animals, we find a higher percentage of quadratic-like NLs as opposed to more linear NLs in anesthetized animals. Moderate sex differences of the shape of NLs were observed between male and female unanesthetized mice. This indicates that the core AC possesses a rich variety of potential computations, particularly in awake animals, suggesting that multiple computational algorithms are at play to enable the auditory system's robust recognition of auditory events.

STAM-LSGRU: a spatiotemporal radar echo extrapolation algorithm with edge computing for short-term forecasting

With the advent of Mobile Edge Computing (MEC), shifting data processing from cloud centers to the network edge presents an advanced computational paradigm for addressing latency-sensitive applications. Specifically, in radar systems, the real-time processing and prediction of radar echo data pose significant challenges in dynamic and resource-constrained environments. MEC, by processing data near its source, not only significantly reduces communication latency and enhances bandwidth utilization but also diminishes the necessity of transmitting large volumes of data to the cloud, which is crucial for improving the timeliness and efficiency of radar data processing. To meet this demand, this paper proposes a model that integrates a spatiotemporal Attention Module (STAM) with a Long Short-Term Memory Gated Recurrent Unit (ST-ConvLSGRU) to enhance the accuracy of radar echo prediction while leveraging the advantages of MEC. STAM, by extending the spatiotemporal receptive field of the prediction units, effectively captures key inter-frame motion information, while optimizations to the convolutional structure and loss function further boost the model’s predictive performance. Experimental results demonstrate that our approach significantly improves the accuracy of short-term weather forecasting in a mobile edge computing environment, showcasing an efficient and practical solution for processing radar echo data under dynamic, resource-limited conditions.

A sparse code for natural sound context in auditory cortex

Accurate sound perception can require integrating information over hundreds of milliseconds or even seconds. Spectro-temporal models of sound coding by single neurons in auditory cortex indicate that the majority of sound-evoked activity can be attributed to stimuli with a few tens of milliseconds. It remains uncertain how the auditory system integrates information about sensory context on a longer timescale. Here we characterized long-lasting contextual effects in auditory cortex (AC) using a diverse set of natural sound stimuli. We measured context effects as the difference in a neuron's response to a single probe sound following two different context sounds. Many AC neurons showed context effects lasting longer than the temporal window of a traditional spectro-temporal receptive field. The duration and magnitude of context effects varied substantially across neurons and stimuli. This diversity of context effects formed a sparse code across the neural population that encoded a wider range of contexts than any constituent neuron. Encoding model analysis indicates that context effects can be explained by activity in the local neural population, suggesting that recurrent local circuits support a long-lasting representation of sensory context in auditory cortex.

Direct comparison reveals algorithmic similarities in fly and mouse visual motion detection

Evolution has equipped vertebrates and invertebrates with neural circuits that selectively encode visual motion. While similarities in the computations performed by these circuits in mouse and fruit fly have been noted, direct experimental comparisons have been lacking. Because molecular mechanisms and neuronal morphology in the two species are distinct, we directly compared motion encoding in these two species at the algorithmic level, using matched stimuli and focusing on a pair of analogous neurons, the mouse ON starburst amacrine cell (ON SAC) and Drosophila T4 neurons. We find that the cells share similar spatiotemporal receptive field structures, sensitivity to spatiotemporal correlations, and tuning to sinusoidal drifting gratings, but differ in their responses to apparent motion stimuli. Both neuron types showed a response to summed sinusoids that deviates from models for motion processing in these cells, underscoring the similarities in their processing and identifying response features that remain to be explained.

A statistical significance test for spatio-temporal receptive field estimates obtained using spike-triggered averaging of binary pseudo-random sequences

A statistical significance test for spatio-temporal receptive field estimates obtained using spike-triggered averaging of binary pseudo-random sequences

Event-driven spectrotemporal feature extraction and classification using a silicon cochlea model.

This paper presents a reconfigurable digital implementation of an event-based binaural cochlear system on a Field Programmable Gate Array (FPGA). It consists of a pair of the Cascade of Asymmetric Resonators with Fast Acting Compression (CAR-FAC) cochlea models and leaky integrate-and-fire (LIF) neurons. Additionally, we propose an event-driven SpectroTemporal Receptive Field (STRF) Feature Extraction using Adaptive Selection Thresholds (FEAST). It is tested on the TIDIGTIS benchmark and compared with current event-based auditory signal processing approaches and neural networks.

Cones and cone pathways remain functional in advanced retinal degeneration

Most defects causing retinal degeneration in retinitis pigmentosa (RP) are rod-specific mutations, but the subsequent degeneration of cones, which produces loss of daylight vision and high-acuity perception, is the most debilitating feature of the disease. To understand better why cones degenerate and how cone vision might be restored, we have made the first single-cell recordings of light responses from degenerating conesand retinal interneurons after most rods have died and cones have lost their outer-segment disk membranes and synaptic pedicles. We show that degenerating cones have functional cyclic-nucleotide-gated channels and can continue to give light responses, apparently produced by opsin localized either to small areas of organized membrane near the ciliary axoneme or distributed throughout the inner segment. Light responses of second-order horizontal and bipolar cells are less sensitive but otherwise resemble those of normal retina. Furthermore, retinal output as reflected in responses of ganglion cells is less sensitive but maintains spatiotemporal receptive fields at cone-mediated light levels. Together, these findings show that cones and their retinal pathways can remain functional even as degeneration is progressing, an encouraging result for future research aimed at enhancing the light sensitivity of residual cones to restore vision in patients with genetically inherited retinal degeneration.

PhysFormer++: Facial Video-Based Physiological Measurement with SlowFast Temporal Difference Transformer

Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.

A simulation model for analyzing the spatiotemporal receptive field of retinal ganglion cells in the presence of fixational eye movements

A simulation model for analyzing the spatiotemporal receptive field of retinal ganglion cells in the presence of fixational eye movements

A time-causal and time-recursive scale-covariant scale-space representation of temporal signals and past time

This article presents an overview of a theory for performing temporal smoothing on temporal signals in such a way that: (i) temporally smoothed signals at coarser temporal scales are guaranteed to constitute simplifications of corresponding temporally smoothed signals at any finer temporal scale (including the original signal) and (ii) the temporal smoothing process is both time-causal and time-recursive, in the sense that it does not require access to future information and can be performed with no other temporal memory buffer of the past than the resulting smoothed temporal scale-space representations themselves. For specific subsets of parameter settings for the classes of linear and shift-invariant temporal smoothing operators that obey this property, it is shown how temporal scale covariance can be additionally obtained, guaranteeing that if the temporal input signal is rescaled by a uniform temporal scaling factor, then also the resulting temporal scale-space representations of the rescaled temporal signal will constitute mere rescalings of the temporal scale-space representations of the original input signal, complemented by a shift along the temporal scale dimension. The resulting time-causal limit kernel that obeys this property constitutes a canonical temporal kernel for processing temporal signals in real-time scenarios when the regular Gaussian kernel cannot be used, because of its non-causal access to information from the future, and we cannot additionally require the temporal smoothing process to comprise a complementary memory of the past beyond the information contained in the temporal smoothing process itself, which in this way also serves as a multi-scale temporal memory of the past. We describe how the time-causal limit kernel relates to previously used temporal models, such as Koenderink’s scale-time kernels and the ex-Gaussian kernel. We do also give an overview of how the time-causal limit kernel can be used for modelling the temporal processing in models for spatio-temporal and spectro-temporal receptive fields, and how it more generally has a high potential for modelling neural temporal response functions in a purely time-causal and time-recursive way, that can also handle phenomena at multiple temporal scales in a theoretically well-founded manner. We detail how this theory can be efficiently implemented for discrete data, in terms of a set of recursive filters coupled in cascade. Hence, the theory is generally applicable for both: (i) modelling continuous temporal phenomena over multiple temporal scales and (ii) digital processing of measured temporal signals in real time. We conclude by stating implications of the theory for modelling temporal phenomena in biological, perceptual, neural and memory processes by mathematical models, as well as implications regarding the philosophy of time and perceptual agents. Specifically, we propose that for A-type theories of time, as well as for perceptual agents, the notion of a non-infinitesimal inner temporal scale of the temporal receptive fields has to be included in representations of the present, where the inherent nonzero temporal delay of such time-causal receptive fields implies a need for incorporating predictions from the actual time-delayed present in the layers of a perceptual hierarchy, to make it possible for a representation of the perceptual present to constitute a representation of the environment with timing properties closer to the actual present.

STSC-SNN: Spatio-Temporal Synaptic Connection with temporal convolution and attention for spiking neural networks.

Spiking neural networks (SNNs), as one of the algorithmic models in neuromorphic computing, have gained a great deal of research attention owing to temporal information processing capability, low power consumption, and high biological plausibility. The potential to efficiently extract spatio-temporal features makes it suitable for processing event streams. However, existing synaptic structures in SNNs are almost full-connections or spatial 2D convolution, neither of which can extract temporal dependencies adequately. In this work, we take inspiration from biological synapses and propose a Spatio-Temporal Synaptic Connection SNN (STSC-SNN) model to enhance the spatio-temporal receptive fields of synaptic connections, thereby establishing temporal dependencies across layers. Specifically, we incorporate temporal convolution and attention mechanisms to implement synaptic filtering and gating functions. We show that endowing synaptic models with temporal dependencies can improve the performance of SNNs on classification tasks. In addition, we investigate the impact of performance via varied spatial-temporal receptive fields and reevaluate the temporal modules in SNNs. Our approach is tested on neuromorphic datasets, including DVS128 Gesture (gesture recognition), N-MNIST, CIFAR10-DVS (image classification), and SHD (speech digit recognition). The results show that the proposed model outperforms the state-of-the-art accuracy on nearly all datasets.

Deep neural networks effectively model neural adaptation to changing background noise and suggest nonlinear noise filtering methods in auditory cortex

The human auditory system displays a robust capacity to adapt to sudden changes in background noise, allowing for continuous speech comprehension despite changes in background environments. However, despite comprehensive studies characterizing this ability, the computations that underly this process are not well understood. The first step towards understanding a complex system is to propose a suitable model, but the classical and easily interpreted model for the auditory system, the spectro-temporal receptive field (STRF), cannot match the nonlinear neural dynamics involved in noise adaptation. Here, we utilize a deep neural network (DNN) to model neural adaptation to noise, illustrating its effectiveness at reproducing the complex dynamics at the levels of both individual electrodes and the cortical population. By closely inspecting the model's STRF-like computations over time, we find that the model alters both the gain and shape of its receptive field when adapting to a sudden noise change. We show that the DNN model's gain changes allow it to perform adaptive gain control, while the spectro-temporal change creates noise filtering by altering the inhibitory region of the model's receptive field. Further, we find that models of electrodes in nonprimary auditory cortex also exhibit noise filtering changes in their excitatory regions, suggesting differences in noise filtering mechanisms along the cortical hierarchy. These findings demonstrate the capability of deep neural networks to model complex neural adaptation and offer new hypotheses about the computations the auditory cortex performs to enable noise-robust speech perception in real-world, dynamic environments.

Role of superior temporal gyrus and planum temporale in talker segregation

Recent studies suggest the brain tracks both attended and unattended speech streams. Here, we describe the cortical mechanisms that support active talker segregation by vocal gender. Thirty-three participants with normal or near-normal hearing performed a competing speech task during fMRI scanning. The target (competing) talker was female (male). Spectrotemporal modulation filtering was applied to stochastically modulate female and male vocal pitch across trials. Using the modulation-filter patterns as predictors, spectrotemporal receptive fields (STRFs) were obtained at each voxel using coordinate descent. STRF weights associated with female- (∼6 cyc/kHz) and male-talker (∼12 cyc/kHz) pitch were analyzed across subjects to identify pitch-sensitive voxels (logical OR, corrected p < 0.01), which were then characterized by preference for female vs. male. Anterior regions in Heschl’s gyrus and the superior temporal gyrus (STG) responded best to the female talker, while posterior regions in STG and planum temporale (PT) responded best to the male talker. In a control task where the talkers did not compete, the same pattern was observed but the posterior network shifted from STG to PT and responded to the acoustic boundary between talkers (∼9 cyc/kHz), suggesting that acoustically coded pitch in PT becomes voice-coded in STG during active segregation.

Joint coding of visual input and eye/head position in V1 of freely moving mice

Visual input during natural behavior is highly dependent on movements of the eyes and head, but how information about eye and head position is integrated with visual processing during free movement is unknown, as visual physiology is generally performed under head fixation. To address this, we performed single-unit electrophysiology in V1 of freely moving mice while simultaneously measuring the mouse's eye position, head orientation, and the visual scene from the mouse's perspective. From these measures, we mapped spatiotemporal receptive fields during free movement based on the gaze-corrected visual input. Furthermore, we found a significant fraction of neurons tuned for eye and head position, and these signals were integrated with visual responses through a multiplicative mechanism in the majority of modulated neurons. These results provide new insight into coding in the mouse V1 and, more generally, provide a paradigm for investigating visual physiology under natural conditions, including active sensing and ethological behavior.

Robust cone-mediated signaling persists late into rod photoreceptor degeneration.

Rod photoreceptor degeneration causes deterioration in the morphology and physiology of cone photoreceptors along with changes in retinal circuits. These changes could diminish visual signaling at cone-mediated light levels, thereby limiting the efficacy of treatments such as gene therapy for rescuing normal, cone-mediated vision. However, the impact of progressive rod death on cone-mediated signaling remains unclear. To investigate the fidelity of retinal ganglion cell (RGC) signaling throughout disease progression, we used a mouse model of rod degeneration (Cngb1neo/neo). Despite clear deterioration of cone morphology with rod death, cone-mediated signaling among RGCs remained surprisingly robust: spatiotemporal receptive fields changed little and the mutual information between stimuli and spiking responses was relatively constant. This relative stability held until nearly all rods had died and cones had completely lost well-formed outer segments. Interestingly, RGC information rates were higher and more stable for natural movies than checkerboard noise as degeneration progressed. The main change in RGC responses with photoreceptor degeneration was a decrease in response gain. These results suggest that gene therapies for rod degenerative diseases are likely to prolong cone-mediated vision even if there are changes to cone morphology and density.

The prefrontal cortex of the Mexican free-tailed bat is more selective to communication calls than primary auditory cortex.

In this study, we examined the auditory responses of a prefrontal area, the frontal auditory field (FAF), of an echolocating bat (Tadarida brasiliensis) and presented a comparative analysis of the neuronal response properties between the FAF and the primary auditory cortex (A1). We compared single-unit responses from the A1 and the FAF elicited by pure tones, downward frequency-modulated sweeps (dFMs), and species-specific vocalizations. Unlike the A1, FAFs were not frequency tuned. However, progressive increases in dFM sweep rate elicited a systematic increase of response precision, a phenomenon that does not take place in the A1. Call selectivity was higher in the FAF versus A1. We calculated the neuronal spectrotemporal receptive fields (STRFs) and spike-triggered averages (STAs) to predict responses to the communication calls and provide an explanation for the differences in call selectivity between the FAF and A1. In the A1, we found a high correlation between predicted and evoked responses. However, we did not generate reasonable STRFs in the FAF, and the prediction based on the STAs showed lower correlation coefficient than that of the A1. This suggests nonlinear response properties in the FAF that are stronger than the linear response properties in the A1. Stimulating with a call sequence increased call selectivity in the A1, but it remained unchanged in the FAF. These data are consistent with a role for the FAF in assessing distinctive acoustic features downstream of A1, similar to the role proposed for primate ventrolateral prefrontal cortex.NEW & NOTEWORTHY In this study, we examined the neuronal responses of a frontal cortical area in an echolocating bat to behaviorally relevant acoustic stimuli and compared them with those in the primary auditory cortex (A1). In contrast to the A1, neurons in the bat frontal auditory field are not frequency tuned but showed a higher selectivity for social signals such as communication calls. The results presented here indicate that the frontal auditory field may represent an additional processing center for behaviorally relevant sounds.

DIANA, a Process-Oriented Model of Human Auditory Word Recognition.

This article presents DIANA, a new, process-oriented model of human auditory word recognition, which takes as its input the acoustic signal and can produce as its output word identifications and lexicality decisions, as well as reaction times. This makes it possible to compare its output with human listeners’ behavior in psycholinguistic experiments. DIANA differs from existing models in that it takes more available neuro-physiological evidence on speech processing into account. For instance, DIANA accounts for the effect of ambiguity in the acoustic signal on reaction times following the Hick–Hyman law and it interprets the acoustic signal in the form of spectro-temporal receptive fields, which are attested in the human superior temporal gyrus, instead of in the form of abstract phonological units. The model consists of three components: activation, decision and execution. The activation and decision components are described in detail, both at the conceptual level (in the running text) and at the computational level (in the Appendices). While the activation component is independent of the listener’s task, the functioning of the decision component depends on this task. The article also describes how DIANA could be improved in the future in order to even better resemble the behavior of human listeners.

High-Frequency Hearing Is Required to Compute a Topographic Map of Auditory Space in the Mouse Superior Colliculus.

A topographic map of auditory space is a feature found in the superior colliculus (SC) of many species, including CBA/CaJ mice. In this genetic background, high-frequency monaural spectral cues and interaural level differences (ILDs) are used to compute spatial receptive fields (RFs) that form a topographic map along the azimuth. Unfortunately, C57BL/6 mice, a strain widely used for transgenic manipulation, display age-related hearing loss (AHL) because of an inbred mutation in the Cadherin 23 gene (Cdh23) that affects hair cell mechanotransduction. To overcome this problem, researchers have used young C57BL/6 mice in their studies, as they have been shown to have normal hearing thresholds. However, important details of the auditory response characteristics of the SC such as spectral responses and spatial localization, have not been characterized in young C57BL/6 mice. Here, we show that two- to four-month C57BL/6 mice lack neurons with frontal auditory RFs and therefore lack a topographic representation of auditory space in the SC. Analysis of the spectrotemporal RFs (STRFs) of the SC auditory neurons shows that C57BL/6 mouse SC neurons lack the ability to detect the high-frequency (>40 kHz) spectral cues that are needed to compute frontal RFs. We also show that crossing C57BL/6 mice with CBA/CaJ mice or introducing one copy of the wild-type Cdh23 to C57BL/6 mice rescues the high-frequency hearing deficit and improves the topographic map of auditory space. Taken together, these results demonstrate the importance of high-frequency hearing in computing a topographic map of auditory space.