Category selectivity as a window into behavioral relevance

Ritchie and colleagues propose that the functional organization of higher visual cortex is best understood through the lens of behavioral relevance, advocating for a shift away from theories that center around category selectivity. Building on this, I suggest the statistical structure of visual inputs acts as an additional critical constraint on visual cortex, and that a complete understanding of visual system organization must account for input statistics and how they interact with behavioral relevance. I discuss this using cortical food selectivity as a case study, and additionally describe how deep neural networks can provide new avenues for testing these theories.

A comprehensive understanding of the neural mechanisms of cognitive function requires an understanding of how neural representations are transformed across different scales of neural organization: from within local microcircuits to across different brain areas. However, the neural transformations within the local microcircuits are poorly understood. Particularly, the role that two main cell classes of neurons in cortical microcircuits (i.e. pyramidal neurons and interneurons) have in auditory behaviour and cognition remains unknown. In this study, we tested the hypothesis that pyramidal cells and interneurons in the auditory cortex play a differential role in auditory categorization. To test this hypothesis, we recorded single-unit activity from the auditory cortex of rhesus monkeys while they categorized speech sounds. Based on the spike-waveform shape, a neuron was classified as either a narrow-spiking putative interneuron or a broad-spiking putative pyramidal neuron. We found that putative interneurons and pyramidal neurons in the auditory cortex differentially coded category information: interneurons were more selective for auditory categories than pyramidal neurons. These differences between cell classes may be an essential property of the neural computations underlying auditory categorization within the microcircuitry of the auditory cortex.

Are age-related changes in cortical motor representations linked with facilitation/inhibition in the primary motor cortex?

Decades of work demonstrate that the ventral temporal cortex (VTC) comprises category selective regions. Ritchie et al. urge a shift in perspective: new research should be grounded in behavioral relevance, not category selectivity. Here, we outline how leveraging, not shifting away from category selectivity, expands our understanding of brain function, complex cognition, and development. Further, while we agree that naturalistic paradigms will accelerate progress in this field, given category selectivity is central to VTC’s information processing, we suggest future work to examine information transfer from VTC initial object recognition computation to other cortices for facilitating complex human behavior.

Neural “Ignition”: Enhanced Activation Linked to Perceptual Awareness in Human Ventral Stream Visual Cortex

In the typically developing (TD) brain, neural representations for visual stimulus categories (e.g., faces, objects, and words) emerge in bilateral occipitotemporal cortex (OTC), albeit with weighted asymmetry; in parallel, recognition behavior continues to be refined. A fundamental question is whether two hemispheres are necessary or redundant for the emergence of neural representations and recognition behavior typically distributed across both hemispheres. The rare population of patients undergoing unilateral OTC resection in childhood offers a unique opportunity to evaluate whether neural computations for visual stimulus individuation suffice for recognition with only a single developing OTC. Here, using functional magnetic resonance imaging, we mapped category selectivity (CS) and neural representations for individual stimulus exemplars using repetition suppression (RS) in the non-resected hemisphere of pediatric OTC resection patients (n = 9) and control patients with resection outside of OTC (n = 12), as well as in both hemispheres of TD controls (n = 21). There were no univariate group differences in the magnitude of CS or RS or any multivariate differences (per representational similarity analysis) in neural activation to faces, objects, or words across groups. Notwithstanding their comparable neural profiles, accuracy of OTC resection patients on face and object recognition, but not word recognition, was statistically inferior to that of controls. The comparable neural signature of the OTC resection patients’ preserved hemisphere and the other two groups highlights the resilience of the system following damage to the contralateral homologue. Critically, however, a single OTC does not suffice for normal behavior, and, thereby, implicates the necessity for two hemispheres.

Does the nature of representation in the category-selective regions in the occipitotemporal cortex reflect visual or conceptual properties? Previous research showed that natural variability in visual features across categories, quantified by image gist statistics, is highly correlated with the different neural responses observed in the occipitotemporal cortex. Using fMRI, we examined whether category selectivity for animals and tools would remain, when image gist statistics were comparable across categories. Critically, we investigated how category, shape, and spatial frequency may contribute to the category selectivity in the animal- and tool-selective regions. Female and male human observers viewed low- or high-passed images of round or elongated animals and tools that shared comparable gist statistics in the main experiment, and animal and tool images of naturally varied gist statistics in a separate localizer. Univariate analysis revealed robust category-selective responses for images with comparable gist statistics across categories. Successful classification for category (animals/tools), shape (round/elongated), and spatial frequency (low/high) was also observed, with highest classification accuracy for category. Representational similarity analyses further revealed that the activation patterns in the animal-selective regions were most correlated with a model that represents only animal information, whereas the activation patterns in the tool-selective regions were most correlated with a model that represents only tool information, suggesting that these regions selectively represent information of only animals or tools. Together, in addition to visual features, the distinction between animal and tool representations in the occipitotemporal cortex is likely shaped by higher-level conceptual influences such as categorization or interpretation of visual inputs.SIGNIFICANCE STATEMENT Since different categories often vary systematically in both visual and conceptual features, it remains unclear what kinds of information determine category-selective responses in the occipitotemporal cortex. To minimize the influences of low- and mid-level visual features, here we used a diverse image set of animals and tools that shared comparable gist statistics. We manipulated category (animals/tools), shape (round/elongated), and spatial frequency (low/high), and found that the representational content of the animal- and tool-selective regions is primarily determined by their preferred categories only, regardless of shape or spatial frequency. Our results show that category-selective responses in the occipitotemporal cortex are influenced by higher-level processing such as categorization or interpretation of visual inputs, and highlight the specificity in these category-selective regions.

Object recognition is a crucial brain function that involves a complex interplay between various brain regions. However, the behavioral relevance of functional interactions between these regions remains largely unexplored. In this study, we examined the functional interactions between different brain regions during object recognition using intracranial electrocorticography (ECoG) recordings in subjects diagnosed with pharmacologically intractable epilepsy. We computed the phase locking value (PLV) between different brain areas and its category selectivity, and assessed its behavioral relevance by comparing correctly and incorrectly performed trials. Our results revealed that phase locking between brain regions varies across different object categories and that this variability significantly influences the perceptual behavior of subjects. Importantly, we found that the behavioral relevance of these interactions is spatially organized, with the high behaviorally relevant connections being longer for the frontal lobe and shorter for the occipital lobe. These findings underscore the unique roles of different brain areas in object recognition and pave the way for more nuanced explorations of the interplay between brain regions in object recognition and other cognitive functions.

A wealth of studies report evidence that occipitotemporal cortex tessellates into ‘category-selective’ brain regions that are apparently specialized for representing ecologically important visual stimuli like faces, bodies, scenes, and tools. Here, we argue that while valuable insights have been gained through the lens of category-selectivity, a more complete view of visual function in occipitotemporal cortex requires centering the behavioral relevance of visual properties in real-world environments rather than stimulus category. Focusing on behavioral relevance challenges a simple mapping between stimulus and visual function in occipitotemporal cortex because the environmental properties relevant to a behavior are visually diverse and how a given property is represented is modulated by our goals. Grounding our thinking in behavioral relevance rather than category-selectivity raises a host of theoretical and empirical issues that we discuss while providing proposals for how existing tools can be harnessed in this light to better understand visual function in occipitotemporal cortex.

Allocating attentional resources to currently relevant information in a dynamically changing environment is critical to goal-directed behavior. Previous studies in nonhuman primates (NHPs) have demonstrated modulation of neural representations of stimuli, in particular visual categorizations, by behavioral significance in the lateral prefrontal cortex. In the human brain, a network of frontal and parietal regions, the "multiple demand" (MD) system, is involved in cognitive and attentional control. To test for the effect of behavioral significance on categorical discrimination in the MD system in humans, we adapted a previously used task in the NHP and used multivoxel pattern analysis for fMRI data. In a cued-detection categorization task, participants detected whether an image from one of two target visual categories was present in a display. Our results revealed that categorical discrimination is modulated by behavioral relevance, as measured by the distributed pattern of response across the MD network. Distinctions between categories with different behavioral status (e.g., a target and a nontarget) were significantly discriminated. Category distinctions that were not behaviorally relevant (e.g., between two targets) were not discriminated. Other aspects of the task that were orthogonal to the behavioral decision did not modulate categorical discrimination. In a high visual region, the lateral occipital complex, modulation by behavioral relevance was evident in its posterior subregion but not in the anterior subregion. The results are consistent with the view of the MD system as involved in top-down attentional and cognitive control by selective coding of task-relevant discriminations. Significance statement: Control of cognitive demands fundamentally involves flexible allocation of attentional resources depending on a current behavioral context. Essential to such a mechanism is the ability to select currently relevant information and at the same time filter out information that is irrelevant. In an fMRI study, we measured distributed patterns of activity for objects from different visual categories while manipulating the behavioral relevance of the categorical distinctions. In a network of frontal and parietal cortical regions, the multiple-demand (MD) network, patterns reflected category distinctions that were relevant to behavior. Patterns could not be used to make task-irrelevant category distinctions. These findings demonstrate the ability of the MD network to implement complex goal-directed behavior by focused attention.

The human brain contains cortical areas specialized in representing object categories. Visual experience is known to change the responses in these category-selective areas of the brain. However, little is known about how category training specifically affects cortical category selectivity. Here, we investigated the experience-dependent formation of object categories using an fMRI adaptation paradigm. Outside the scanner, subjects were trained to categorize artificial bird types into arbitrary categories (jungle birds and desert birds). After training, neuronal populations in the occipito-temporal cortex, such as the fusiform and the lateral occipital gyrus, were highly sensitive to perceptual stimulus differences. This sensitivity was not present for novel birds, indicating experience-related changes in neuronal representations. Neurons in STS showed category selectivity. A release from adaptation in STS was only observed when two birds in a pair crossed the category boundary. This dissociation could not be explained by perceptual similarities because the physical difference between birds from the same side of the category boundary and between birds from opposite sides of the category boundary was equal. Together, the occipito-temporal cortex and the STS have the properties suitable for a system that can both generalize across stimuli and discriminate between them.

Neuronal encoding of meaning: Establishing category-selective response patterns in the avian ‘prefrontal cortex’

Building computational models to account for the cortical representation of language plays an important role in understanding the human linguistic system. Recent progress in distributed semantic models (DSMs), especially transformer-based methods, has driven advances in many language understanding tasks, making DSM a promising methodology to probe brain language processing. DSMs have been shown to reliably explain cortical responses to word stimuli. However, characterizing the brain activities for sentence processing is much less exhaustively explored with DSMs, especially the deep neural network-based methods. What is the relationship between cortical sentence representations against DSMs? What linguistic features that a DSM catches better explain its correlation with the brain activities aroused by sentence stimuli? Could distributed sentence representations help to reveal the semantic selectivity of different brain areas? We address these questions through the lens of neural encoding and decoding, fueled by the latest developments in natural language representation learning. We begin by evaluating the ability of a wide range of 12 DSMs to predict and decipher the functional magnetic resonance imaging (fMRI) images from humans reading sentences. Most models deliver high accuracy in the left middle temporal gyrus (LMTG) and left occipital complex (LOC). Notably, encoders trained with transformer-based DSMs consistently outperform other unsupervised structured models and all the unstructured baselines. With probing and ablation tasks, we further find that differences in the performance of the DSMs in modeling brain activities can be at least partially explained by the granularity of their semantic representations. We also illustrate the DSM's selectivity for concept categories and show that the topics are represented by spatially overlapping and distributed cortical patterns. Our results corroborate and extend previous findings in understanding the relation between DSMs and neural activation patterns and contribute to building solid brain-machine interfaces with deep neural network representations.

In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotics for deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and classes of Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in several contexts of practical interest, e.g. Bayesian inference under Gaussian SP priors, kernel regression for infinite-wide deep NNs trained via gradient descent, and information propagation within infinite-wide NNs. Motivated by empirical analysis, showing the potential of replacing Gaussian distributions with Stable distributions for the NN's weights, in this paper we investigate large-width asymptotics for (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stable-distributed weights. First, we show that as the width goes to infinity jointly over the NN's layers, a suitable rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN's layers. Because of the non-triangular NN's structure, this is a non-standard asymptotic problem, to which we propose a novel and self-contained inductive approach, which may be of independent interest. Then, we establish sup-norm convergence rates of a deep Stable NN to a Stable SP, quantifying the critical difference between the settings of ``joint growth and ``sequential growth of the width over the NN's layers. Our work extends recent results on infinite-wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates for infinite-wide deep NNs.

Deep learning based on structured deep neural networks has provided powerful applications in various fields. The structures imposed on the deep neural networks are crucial, which makes deep learning essentially different from classical schemes based on fully connected neural networks. One of the commonly used deep neural network structures is generated by convolutions. The produced deep learning algorithms form the family of deep convolutional neural networks. Despite of their power in some practical domains, little is known about the mathematical foundation of deep convolutional neural networks such as universality of approximation. In this paper, we propose a family of new structured deep neural networks: deep distributed convolutional neural networks. We show that these deep neural networks have the same order of computational complexity as the deep convolutional neural networks, and we prove their universality of approximation. Some ideas of our analysis are from ridge approximation, wavelets, and learning theory.