Receptive Field Model Research Articles

Assessing Goodness-of-Fit in Marked Point Process Models of Neural Population Coding via Time and Rate Rescaling.

Marked point process models have recently been used to capture the coding properties of neural populations from multiunit electrophysiological recordings without spike sorting. These clusterless models have been shown in some instances to better describe the firing properties of neural populations than collections of receptive field models for sorted neurons and to lead to better decoding results. To assess their quality, we previously proposed a goodness-of-fit technique for marked point process models based on time rescaling, which for a correct model produces a set of uniform samples over a random region of space. However, assessing uniformity over such a region can be challenging, especially in high dimensions. Here, we propose a set of new transformations in both time and the space of spike waveform features, which generate events that are uniformly distributed in the new mark and time spaces. These transformations are scalable to multidimensional mark spaces and provide uniformly distributed samples in hypercubes, which are well suited for uniformity tests. We discuss the properties of these transformations and demonstrate aspects of model fit captured by each transformation. We also compare multiple uniformity tests to determine their power to identify lack-of-fit in the rescaled data. We demonstrate an application of these transformations and uniformity tests in a simulation study. Proofs for each transformation are provided in the appendix.

Investigating the Reliability of Population Receptive Field Size Estimates Using fMRI.

In functional MRI (fMRI), population receptive field (pRF) models allow a quantitative description of the response as a function of the features of the stimuli that are relevant for each voxel. The most popular pRF model used in fMRI assumes a Gaussian shape in the features space (e.g., the visual field) reducing the description of the voxel’s pRF to the Gaussian mean (the pRF preferred feature) and standard deviation (the pRF size). The estimation of the pRF mean has been proven to be highly reliable. However, the estimate of the pRF size has been shown not to be consistent within and between subjects. While this issue has been noted experimentally, here we use an optimization theory perspective to describe how the inconsistency in estimating the pRF size is linked to an inherent property of the Gaussian pRF model. When fitting such models, the goodness of fit is less sensitive to variations in the pRF size than to variations in the pRF mean. We also show how the same issue can be considered from a bias-variance perspective. We compare different estimation procedures in terms of the reliability of their estimates using simulated and real fMRI data in the visual (using the Human Connectome Project database) and auditory domain. We show that, the reliability of the estimate of the pRF size can be improved considering a linear combination of those pRF models with similar goodness of fit or a permutation based approach. This increase in reliability of the pRF size estimate does not affect the reliability of the estimate of the pRF mean and the prediction accuracy.

Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”

Neural encoding, a crucial aspect to understand the human brain information processing system, aims to establish a quantitative relationship between the stimuli and the evoked brain activities. In the field of visual neuroscience, with the ability to explain how neurons in the primary visual cortex work, population receptive field (pRF) models have enjoyed high popularity and made reliable progress in recent years. However, existing models rely on either the inflexible prior assumptions about pRF or the clumsy parameter estimation methods, severely limiting the expressiveness and interpretability. In this article, we propose a novel neural encoding framework by learning “what” and “where” with deep neural networks. It involves two separate aspects: 1) the spatial characteristic (“where”) and 2) feature selection (“what”) of neuron populations in the visual cortex. Specifically, our approach first encodes visual stimuli into hierarchically intermediate features through a pretrained deep neural network (DNN), then converts DNN features into refined features with the channel attention and spatial receptive field (RF) to learn “where”, and finally regresses refined features simultaneously onto voxel activities to learn “what”. The sparsity regularization and smoothness regularization are adopted in our modeling approach so that the crucial RF can be estimated automatically without prior assumptions about shapes. Furthermore, an attempt is made to extend the voxel-wise modeling approach to multi-voxel joint encoding models, and we show that it is conducive to rescuing voxels with poor signal-to-noise characteristics. Extensive empirical results demonstrate that the method developed herein provides an effective strategy to establish neural encoding for the human visual cortex, with the weaker prior constraints but the higher encoding performance.

Blind S3D image quality prediction using classical and non-classical receptive field models

We develop a novel no-reference image quality assessment model for stereoscopic 3D (S3D) images that is inspired by functional receptive field models of perceptual mechanisms in primary visual cortex (V1). The approach is called the Blind S3D Integrated Quality Evaluator (BSIQE). BSIQE simulates monocular and binocular responses to stereo views using channel separation and weighted multi-channel combination models. Binocular responses are modeled as the fusion of the two channels using a weighted multi-channel combination. The responses to stereoscopic image content of both classical and non-classical anisotropic receptive fields are then modeled based on a determination of the relative importance of the receptive field responses. In the last stage of feature extraction, we deploy a simple and efficient way of decorrelating the picture data. We extract local binary pattern (LBP) statistical features from the computed receptive field responses, and use them to train a regressor to predict the perceptual quality of stereoscopic images. We carefully evaluate BSIQE on four public-domain 3D image quality databases, and find that it is statistically superior to all compared 2D and 3D IQA algorithms. BSIQE exhibits good performance across the datasets suggesting that it is general, and it has relatively low complexity.

A robust contour detection operator with combined push-pull inhibition and surround suppression

Contour detection is a salient operation in many computer vision applications as it extracts features that are important for distinguishing objects in scenes. It is believed to be a primary role of simple cells in visual cortex of the mammalian brain. Many of such cells receive push-pull inhibition or surround suppression. We propose a computational model that exhibits a combination of these two phenomena. It is based on two existing models, which have been proven to be very effective for contour detection. In particular, we introduce a brain-inspired contour operator that combines push-pull and surround inhibition. It turns out that this combination results in a more effective contour detector, which suppresses texture while keeping the strongest responses to lines and edges, when compared to existing models. The proposed model consists of a Combination of Receptive Field (or CORF) model with push-pull inhibition, extended with surround suppression. We demonstrate the effectiveness of the proposed approach on the RuG and Berkeley benchmark data sets of 40 and 500 images, respectively. The proposed push-pull CORF operator with surround suppression outperforms the one without suppression with high statistical significance.

Resolving the Spatial Profile of Figure Enhancement in Human V1 through Population Receptive Field Modeling.

The detection and segmentation of meaningful figures from their background is one of the primary functions of vision. While work in nonhuman primates has implicated early visual mechanisms in this figure-ground modulation, neuroimaging in humans has instead largely ascribed the processing of figures and objects to higher stages of the visual hierarchy. Here, we used high-field fMRI at 7 Tesla to measure BOLD responses to task-irrelevant orientation-defined figures in human early visual cortex (N = 6, four females). We used a novel population receptive field mapping-based approach to resolve the spatial profiles of two constituent mechanisms of figure-ground modulation: a local boundary response, and a further enhancement spanning the full extent of the figure region that is driven by global differences in features. Reconstructing the distinct spatial profiles of these effects reveals that figure enhancement modulates responses in human early visual cortex in a manner consistent with a mechanism of automatic, contextually driven feedback from higher visual areas.SIGNIFICANCE STATEMENT A core function of the visual system is to parse complex 2D input into meaningful figures. We do so constantly and seamlessly, both by processing information about visible edges and by analyzing large-scale differences between figure and background. While influential neurophysiology work has characterized an intriguing mechanism that enhances V1 responses to perceptual figures, we have a poor understanding of how the early visual system contributes to figure-ground processing in humans. Here, we use advanced computational analysis methods and high-field human fMRI data to resolve the distinct spatial profiles of local edge and global figure enhancement in the early visual system (V1 and LGN); the latter is distinct and consistent with a mechanism of automatic, stimulus-driven feedback from higher-level visual areas.

Auditory cortical representation of music favours the perceived beat.

Previous research has shown that musical beat perception is a surprisingly complex phenomenon involving widespread neural coordination across higher-order sensory, motor and cognitive areas. However, the question of how low-level auditory processing must necessarily shape these dynamics, and therefore perception, is not well understood. Here, we present evidence that the auditory cortical representation of music, even in the absence of motor or top-down activations, already favours the beat that will be perceived. Extracellular firing rates in the rat auditory cortex were recorded in response to 20 musical excerpts diverse in tempo and genre, for which musical beat perception had been characterized by the tapping behaviour of 40 human listeners. We found that firing rates in the rat auditory cortex were on average higher on the beat than off the beat. This ‘neural emphasis’ distinguished the beat that was perceived from other possible interpretations of the beat, was predictive of the degree of tapping consensus across human listeners, and was accounted for by a spectrotemporal receptive field model. These findings strongly suggest that the ‘bottom-up’ processing of music performed by the auditory system predisposes the timing and clarity of the perceived musical beat.

Mapping sequences can bias population receptive field estimates

Population receptive field (pRF) modelling is a common technique for estimating the stimulus-selectivity of populations of neurons using neuroimaging. Here, we aimed to address if pRF properties estimated with this method depend on the spatio-temporal structure and the predictability of the mapping stimulus. We mapped the polar angle preference and tuning width of voxels in visual cortex (V1–V4) of healthy, adult volunteers. We compared sequences sweeping orderly through the visual field or jumping from location to location employing stimuli of different width (45° vs 6°) and cycles of variable duration (8s vs 60s). While we did not observe any systematic influence of stimulus predictability, the temporal structure of the sequences significantly affected tuning width estimates. Ordered designs with large wedges and short cycles produced systematically smaller estimates than random sequences. Interestingly, when we used small wedges and long cycles, we obtained larger tuning width estimates for ordered than random sequences. We suggest that ordered and random mapping protocols show different susceptibility to other design choices such as stimulus type and duration of the mapping cycle and can produce significantly different pRF results.

Robust Rate-Place Coding of Resolved Components in Harmonic and Inharmonic Complex Tones in Auditory Midbrain.

Harmonic complex tones (HCTs) commonly occurring in speech and music evoke a strong pitch at their fundamental frequency (F0), especially when they contain harmonics individually resolved by the cochlea. When all frequency components of an HCT are shifted by the same amount, the pitch of the resulting inharmonic tone (IHCT) can also shift, although the envelope repetition rate is unchanged. A rate-place code, whereby resolved harmonics are represented by local maxima in firing rates along the tonotopic axis, has been characterized in the auditory nerve and primary auditory cortex, but little is known about intermediate processing stages. We recorded single-neuron responses to HCT and IHCT with varying F0 and sound level in the inferior colliculus (IC) of unanesthetized rabbits of both sexes. Many neurons showed peaks in firing rate when a low-numbered harmonic aligned with the neuron's characteristic frequency, demonstrating "rate-place" coding. The IC rate-place code was most prevalent for F0 > 800 Hz, was only moderately dependent on sound level over a 40 dB range, and was not sensitive to stimulus harmonicity. A spectral receptive-field model incorporating broadband inhibition better predicted the neural responses than a purely excitatory model, suggesting an enhancement of the rate-place representation by inhibition. Some IC neurons showed facilitation in response to HCT relative to pure tones, similar to cortical "harmonic template neurons" (Feng and Wang, 2017), but to a lesser degree. Our findings shed light on the transformation of rate-place coding of resolved harmonics along the auditory pathway.SIGNIFICANCE STATEMENT Harmonic complex tones are ubiquitous in speech and music and produce strong pitch percepts when they contain frequency components that are individually resolved by the cochlea. Here, we characterize a "rate-place" code for resolved harmonics in the auditory midbrain that is more robust across sound levels than the peripheral rate-place code and insensitive to the harmonic relationships among frequency components. We use a computational model to show that inhibition may play an important role in shaping the rate-place code. Our study fills a major gap in understanding the transformations in neural representations of resolved harmonics along the auditory pathway.

Bayesian population receptive field modeling in human somatosensory cortex

Somatosensation is fundamental to our ability to sense our body and interact with the world. Our body is continuously sampling the environment using a variety of receptors tuned to different features, and this information is routed up to primary somatosensory cortex. Strikingly, the spatial organization of the peripheral receptors in the body are well maintained, with the resulting representation of the body in the brain being referred to as the somatosensory homunculus. Recent years have seen considerable advancements in the field of high-resolution fMRI, which have enabled an increasingly detailed examination of the organization and properties of this homunculus. Here we combined advanced imaging techniques at ultra-high field (7T) with a recently developed Bayesian population receptive field (pRF) modeling framework to examine pRF properties in primary somatosensory cortex. In each subject, vibrotactile stimulation of the fingertips (i.e., the peripheral mechanoreceptors) modulated the fMRI response along the post-central gyrus and these signals were used to estimate pRFs. We found the pRF center location estimates to be in accord with previous work as well as evidence of other properties in line with the underlying neurobiology. Specifically, as expected from the known properties of cortical magnification, we find a larger representation of the index finger compared to the other stimulated digits (middle, index, little). We also show evidence that the little finger is marked by the largest pRF sizes, and that pRF size increases from anterior to posterior regions of S1. The ability to estimate somatosensory pRFs in humans provides an unprecedented opportunity to examine the neural mechanisms underlying somatosensation and is critical for studying how the brain, body, and environment interact to inform perception and action.

Provably Scale-Covariant Continuous Hierarchical Networks Based on Scale-Normalized Differential Expressions Coupled in Cascade

This article presents a theory for constructing hierarchical networks in such a way that the networks are guaranteed to be provably scale covariant. We first present a general sufficiency argument for obtaining scale covariance, which holds for a wide class of networks defined from linear and nonlinear differential expressions expressed in terms of scale-normalized scale-space derivatives. Then, we present a more detailed development of one example of such a network constructed from a combination of mathematically derived models of receptive fields and biologically inspired computations. Based on a functional model of complex cells in terms of an oriented quasi quadrature combination of first- and second-order directional Gaussian derivatives, we couple such primitive computations in cascade over combinatorial expansions over image orientations. Scale-space properties of the computational primitives are analysed, and we give explicit proofs of how the resulting representation allows for scale and rotation covariance. A prototype application to texture analysis is developed, and it is demonstrated that a simplified mean-reduced representation of the resulting QuasiQuadNet leads to promising experimental results on three texture datasets.

Spectral tuning of adaptation supports coding of sensory context in auditory cortex.

Perception of vocalizations and other behaviorally relevant sounds requires integrating acoustic information over hundreds of milliseconds. Sound-evoked activity in auditory cortex typically has much shorter latency, but the acoustic context, i.e., sound history, can modulate sound evoked activity over longer periods. Contextual effects are attributed to modulatory phenomena, such as stimulus-specific adaption and contrast gain control. However, an encoding model that links context to natural sound processing has yet to be established. We tested whether a model in which spectrally tuned inputs undergo adaptation mimicking short-term synaptic plasticity (STP) can account for contextual effects during natural sound processing. Single-unit activity was recorded from primary auditory cortex of awake ferrets during presentation of noise with natural temporal dynamics and fully natural sounds. Encoding properties were characterized by a standard linear-nonlinear spectro-temporal receptive field (LN) model and variants that incorporated STP-like adaptation. In the adapting models, STP was applied either globally across all input spectral channels or locally to subsets of channels. For most neurons, models incorporating local STP predicted neural activity as well or better than LN and global STP models. The strength of nonlinear adaptation varied across neurons. Within neurons, adaptation was generally stronger for spectral channels with excitatory than inhibitory gain. Neurons showing improved STP model performance also tended to undergo stimulus-specific adaptation, suggesting a common mechanism for these phenomena. When STP models were compared between passive and active behavior conditions, response gain often changed, but average STP parameters were stable. Thus, spectrally and temporally heterogeneous adaptation, subserved by a mechanism with STP-like dynamics, may support representation of the complex spectro-temporal patterns that comprise natural sounds across wide-ranging sensory contexts.

Receptive Field Size vs. Model Depth for Single Image Super-resolution.

The performance of single image super-resolution (SISR) has been largely improved by innovative designs of deep architectures. An important claim raised by these designs is that the deep models have large receptive field size and strong nonlinearity. However, we are concerned about the question that which factor, receptive field size or model depth, is more critical for SISR. Towards revealing the answers, in this paper, we propose a strategy based on dilated convolution to investigate how the two factors affect the performance of SISR. Our findings from exhaustive investigations suggest that SISR is more sensitive to the changes of receptive field size than to the model depth variations, and that the model depth must be congruent with the receptive field size to produce improved performance. These findings inspire us to design a shallower architecture which can save computational and memory cost while preserving comparable effectiveness with respect to a much deeper architecture.

Comparing population receptive fields in human and macaque visual cortex

The population receptive field (pRF) modelling approach has been applied widely in human fMRI to inform the organization of both early and late stages of the cortical visual hierarchy. Recently, a pRF implementation was developed that allows for oriented and elliptical pRFs and demonstrated that dorsal and ventral divisions of early visual cortex (EVC) differentially sample visual space. Thus, differences between the larger dorsal and ventral visual processing streams may be set early on in the visual hierarchy. However, despite its common use, we do not yet understand how pRFs relate to the receptive fields of single neurons. Therefore, cross-species validation by means of comparative fMRI would go a long way to bridging the gap between single-unit recordings in monkeys and fMRI in humans. Here, we compare pRFs from dorsal and ventral divisions of EVC in both humans (n=12) and macaques (Macaca mulatta; n=2). Participants (both humans and macaques) fixated centrally while a bar aperture traversed gradually across the visual field revealing brief scene fragments. Data from both study populations were analyzed in AFNI using both standard (circular) and elliptical pRF models. Cross-validation analyses revealed that, similar to the human, the elliptical pRF model captured more variance in the time-course of macaque data than a circular pRF implementation. Further, pRF eccentricity was positively correlated with both pRF aspect ratio and pRF area, with the strength of correlation increasing generally as a function of position in the visual hierarchy. Also consistent with our human data, the average pRF angles of dorsal and ventral divisions of EVC are oriented differentially. Finally, we found that on average pRF size estimates in macaques tended to be larger than in humans, despite the overall similarity. Overall, these results reveal a very similar pattern of pRF properties between macaque and human visual cortex

Saliency and the population receptive field model to identify images from brain activity

Saliency and the population receptive field model to identify images from brain activity

Reconciling Color Vision Models With Midget Ganglion Cell Receptive Fields.

Midget retinal ganglion cells (RGCs) make up the majority of foveal RGCs in the primate retina. The receptive fields of midget RGCs exhibit both spectral and spatial opponency and are implicated in both color and achromatic form vision, yet the exact mechanisms linking their responses to visual perception remain unclear. Efforts to develop color vision models that accurately predict all the features of human color and form vision based on midget RGCs provide a case study connecting experimental and theoretical neuroscience, drawing on diverse research areas such as anatomy, physiology, psychophysics, and computer vision. Recent technological advances have allowed researchers to test some predictions of color vision models in new and precise ways, producing results that challenge traditional views. Here, we review the progress in developing models of color-coding receptive fields that are consistent with human psychophysics, the biology of the primate visual system and the response properties of midget RGCs.

Unusual Physiological Properties of Smooth Monostratified Ganglion Cell Types in Primate Retina

The functions of the diverse retinal ganglion cell types in primates and the parallel visual pathways they initiate remain poorly understood. Here, unusual physiological and computational properties of the ON and OFF smooth monostratified ganglion cells are explored. Large-scale multi-electrode recordings from 48 macaque retinas revealed that these cells exhibit irregular receptive field structure composed of spatially segregated hotspots, quite different from the classic center-surround model of retinal receptive fields. Surprisingly, visual stimulation of different hotspots in the same cell produced spikes with subtly different spatiotemporal voltage signatures, consistent with a dendritic contribution to hotspot structure. Targeted visual stimulation and computational inference demonstrated strong nonlinear subunit properties associated with each hotspot, supporting a model in which the hotspots apply nonlinearities at a larger spatial scale than bipolar cells. These findings reveal a previously unreported nonlinear mechanism in the output of the primate retina that contributes to signaling spatial information.

Distinguishing Hemodynamics from Function in the Human LGN Using a Temporal Response Model

We developed a temporal population receptive field model to differentiate the neural and hemodynamic response functions (HRF) in the human lateral geniculate nucleus (LGN). The HRF in the human LGN is dominated by the richly vascularized hilum, a structure that serves as a point of entry for blood vessels entering the LGN and supplying the substrates of central vision. The location of the hilum along the ventral surface of the LGN and the resulting gradient in the amplitude of the HRF across the extent of the LGN have made it difficult to segment the human LGN into its more interesting magnocellular and parvocellular regions that represent two distinct visual processing streams. Here, we show that an intrinsic clustering of the LGN responses to a variety of visual inputs reveals the hilum, and further, that this clustering is dominated by the amplitude of the HRF. We introduced a temporal population receptive field model that includes separate sustained and transient temporal impulse response functions that vary on a much short timescale than the HRF. When we account for the HRF amplitude, we demonstrate that this temporal response model is able to functionally segregate the residual responses according to their temporal properties.

A dynamic network model of temporal receptive fields in primary auditory cortex.

Auditory neurons encode stimulus history, which is often modelled using a span of time-delays in a spectro-temporal receptive field (STRF). We propose an alternative model for the encoding of stimulus history, which we apply to extracellular recordings of neurons in the primary auditory cortex of anaesthetized ferrets. For a linear-non-linear STRF model (LN model) to achieve a high level of performance in predicting single unit neural responses to natural sounds in the primary auditory cortex, we found that it is necessary to include time delays going back at least 200 ms in the past. This is an unrealistic time span for biological delay lines. We therefore asked how much of this dependence on stimulus history can instead be explained by dynamical aspects of neurons. We constructed a neural-network model whose output is the weighted sum of units whose responses are determined by a dynamic firing-rate equation. The dynamic aspect performs low-pass filtering on each unit’s response, providing an exponentially decaying memory whose time constant is individual to each unit. We find that this dynamic network (DNet) model, when fitted to the neural data using STRFs of only 25 ms duration, can achieve prediction performance on a held-out dataset comparable to the best performing LN model with STRFs of 200 ms duration. These findings suggest that integration due to the membrane time constants or other exponentially-decaying memory processes may underlie linear temporal receptive fields of neurons beyond 25 ms.

Larger Auditory Cortical Area and Broader Frequency Tuning Underlie Absolute Pitch

Absolute pitch (AP), the ability of some musicians to precisely identify and name musical tones in isolation, is associated with a number of gross morphological changes in the brain, but the fundamental neural mechanisms underlying this ability have not been clear. We presented a series of logarithmic frequency sweeps to age- and sex-matched groups of musicians with or without AP and controls without musical training. We used fMRI and population receptive field (pRF) modeling to measure the responses in the auditory cortex in 61 human subjects. The tuning response of each fMRI voxel was characterized as Gaussian, with independent center frequency and bandwidth parameters. We identified three distinct tonotopic maps, corresponding to primary (A1), rostral (R), and rostral-temporal (RT) regions of auditory cortex. We initially hypothesized that AP abilities might manifest in sharper tuning in the auditory cortex. However, we observed that AP subjects had larger cortical area, with the increased area primarily devoted to broader frequency tuning. We observed anatomically that A1, R and RT were significantly larger in AP musicians than in non-AP musicians or control subjects, which did not differ significantly from each other. The increased cortical area in AP in areas A1 and R were primarily low frequency and broadly tuned, whereas the distribution of responses in area RT did not differ significantly. We conclude that AP abilities are associated with increased early auditory cortical area devoted to broad-frequency tuning and likely exploit increased ensemble encoding.SIGNIFICANCE STATEMENT Absolute pitch (AP), the ability of some musicians to precisely identify and name musical tones in isolation, is associated with a number of gross morphological changes in the brain, but the fundamental neural mechanisms have not been clear. Our study shows that AP musicians have significantly larger volume in early auditory cortex than non-AP musicians and non-musician controls and that this increased volume is primarily devoted to broad-frequency tuning. We conclude that AP musicians are likely able to exploit increased ensemble representations to encode and identify frequency.