Representational Geometry Research Articles

Predictive learning shapes the representational geometry of the human brain

Predictive coding theories propose that the brain constantly updates internal models to minimize prediction errors and optimize sensory processing. However, the neural mechanisms that link prediction error encoding and optimization of sensory representations remain unclear. Here, we provide evidence how predictive learning shapes the representational geometry of the human brain. We recorded magnetoencephalography (MEG) in humans listening to acoustic sequences with different levels of regularity. We found that the brain aligns its representational geometry to match the statistical structure of the sensory inputs, by clustering temporally contiguous and predictable stimuli. Crucially, the magnitude of this representational shift correlates with the synergistic encoding of prediction errors in a network of high-level and sensory areas. Our findings suggest that, in response to the statistical regularities of the environment, large-scale neural interactions engaged in predictive processing modulate the representational content of sensory areas to enhance sensory processing.

Two-dimensional neural geometry underpins hierarchical organization of sequence in human working memory.

Working memory (WM) is constructive in nature. Instead of passively retaining information, WM reorganizes complex sequences into hierarchically embedded chunks to overcome capacity limits and facilitate flexible behaviour. Here, to investigate the neural mechanisms underlying hierarchical reorganization in WM, we performed two electroencephalography and one magnetoencephalography experiments, wherein humans retain in WM a temporal sequence of items, that is, syllables, which are organized into chunks, that is, multisyllabic words. We demonstrate that the one-dimensional sequence is represented by two-dimensional neural representational geometry in WM arising from left prefrontal and temporoparietal regions, with separate dimensions encoding item position within a chunk and chunk position in the sequence. Critically, this two-dimensional geometry is observed consistently in different experimental settings, even during tasks not encouraging hierarchical reorganization in WM and correlates with WM behaviour. Overall, these findings strongly support that complex sequences are reorganized into factorized multidimensional neural representational geometry in WM, which also speaks to general structure-based organizational principles given WM's involvement in many cognitive functions.

The representational geometry for naturalistic textures in macaque V1 and V2.

Our understanding of visual cortical processing has relied primarily on studying the selectivity of individual neurons in different areas. A complementary approach is to study how the representational geometry of neuronal populations differs across areas. Though the geometry is derived from individual neuronal selectivity, it can reveal encoding strategies difficult to infer from single neuron responses. In addition, recent theoretical work has begun to relate distinct functional objectives to different representational geometries. To understand how the representational geometry changes across stages of processing, we measured neuronal population responses in primary visual cortex (V1) and area V2 of macaque monkeys to an ensemble of synthetic, naturalistic textures. Responses were lower dimensional in V2 than V1, and there was a better alignment of V2 population responses to different textures. The representational geometry in V2 afforded better discriminability between out-of-sample textures. We performed complementary analyses of standard convolutional network models, which did not replicate the representational geometry of cortex. We conclude that there is a shift in the representational geometry between V1 and V2, with the V2 representation exhibiting features of a low-dimensional, systematic encoding of different textures and of different instantiations of each texture. Our results suggest that comparisons of representational geometry can reveal important transformations that occur across successive stages of visual processing.

Using fMRI to examine nonlinear mixed selectivity tuning to task and category in the human brain

Abstract Recent experimental and theoretical work has shown that nonlinear mixed selectivity, where neurons exhibit interaction effects in their tuning to multiple variables (e.g., stimulus and task) plays a key role in enabling the primate brain to form representations that can adapt to changing task contexts. Thus far all such studies have relied on invasive neural recording techniques. In this study, we demonstrate the feasibility of measuring nonlinear mixed selectivity tuning in the human brain noninvasively using fMRI pattern decoding. To do so, we examined the joint representation of object category and task information across human early, ventral stream, and dorsal stream areas while participants performed either an oddball detection task or a one-back repetition detection task on the same stimuli. These tasks were chosen to equate spatial, object-based, and feature-based attention, in order to test whether task modulations of visual representations still occur when the inputs to visual processing are kept constant between the two tasks, with only the subsequent cognitive operations varying. We found moderate but significant evidence for nonlinear mixed selectivity tuning to object category and task in fMRI response patterns in both human ventral and dorsal areas, suggesting that neurons exhibiting nonlinear mixed selectivity for category and task not only exist in these regions, but cluster at a scale visible to fMRI. Importantly, while such coding in ventral areas corresponds to a rotation or shift in the object representational geometry without changing the representational content (i.e., with the relative similarity among the categories preserved), nonlinear mixed selectivity coding in dorsal areas corresponds to a reshaping of representational geometry, indicative of a change in representational content.

The topology and geometry of neural representations

A central question for neuroscience is how to characterize brain representations of perceptual and cognitive content. An ideal characterization should distinguish different functional regions with robustness to noise and idiosyncrasies of individual brains that do not correspond to computational differences. Previous studies have characterized brain representations by their representational geometry, which is defined by the representational dissimilarity matrix (RDM), a summary statistic that abstracts from the roles of individual neurons (or responses channels) and characterizes the discriminability of stimuli. Here, we explore a further step of abstraction: from the geometry to the topology of brain representations. We propose topological representational similarity analysis, an extension of representational similarity analysis that uses a family of geotopological summary statistics that generalizes the RDM to characterize the topology while de-emphasizing the geometry. We evaluate this family of statistics in terms of the sensitivity and specificity for model selection using both simulations and functional MRI (fMRI) data. In the simulations, the ground truth is a data-generating layer representation in a neural network model and the models are the same and other layers in different model instances (trained from different random seeds). In fMRI, the ground truth is a visual area and the models are the same and other areas measured in different subjects. Results show that topology-sensitive characterizations of population codes are robust to noise and interindividual variability and maintain excellent sensitivity to the unique representational signatures of different neural network layers and brain regions.

Practice Reshapes the Geometry and Dynamics of Task-tailored Representations.

Extensive practice makes task performance more efficient and precise, leading to automaticity. However, theories of automaticity differ on which levels of task representations (e.g., low-level features, stimulus-response mappings, or high-level conjunctive memories of individual events) change with practice, despite predicting the same pattern of improvement (e.g., power law of practice). To resolve this controversy, we built on recent theoretical advances in understanding computations through neural population dynamics. Specifically, we hypothesized that practice optimizes the neural representational geometry of task representations to minimally separate the highest-level task contingencies needed for successful performance. This involves efficiently reaching conjunctive neural states that integrate task-critical features nonlinearly while abstracting over non-critical dimensions. To test this hypothesis, human participants (n = 40) engaged in extensive practice of a simple, context-dependent action selection task over 3 days while recording EEG. During initial rapid improvement in task performance, representations of the highest-level, context-specific conjunctions of task-features were enhanced as a function of the number of successful episodes. Crucially, only enhancement of these conjunctive representations, and not lower-order representations, predicted the power-law improvement in performance. Simultaneously, over sessions, these conjunctive neural states became more stable earlier in time and more aligned, abstracting over redundant task features, which correlated with offline performance gain in reducing switch costs. Thus, practice optimizes the dynamic representational geometry as task-tailored neural states that minimally tesselate the task space, taming their high-dimensionality.

Encoding modelling for working memory research: Pattern similarity, representational geometry, and model comparison

Encoding modelling for working memory research: Pattern similarity, representational geometry, and model comparison

Erratum: Cheng et al., "Representational Geometries Reveal Differential Effects of Response Correlations on Population Codes in Neurophysiology and Functional Magnetic Resonance Imaging".

Erratum: Cheng et al., "Representational Geometries Reveal Differential Effects of Response Correlations on Population Codes in Neurophysiology and Functional Magnetic Resonance Imaging".

Abstract representations emerge in human hippocampal neurons during inference

Humans have the remarkable cognitive capacity to rapidly adapt to changing environments. Central to this capacity is the ability to form high-level, abstract representations that take advantage of regularities in the world to support generalization1. However, little is known about how these representations are encoded in populations of neurons, how they emerge through learning and how they relate to behaviour2,3. Here we characterized the representational geometry of populations of neurons (single units) recorded in the hippocampus, amygdala, medial frontal cortex and ventral temporal cortex of neurosurgical patients performing an inferential reasoning task. We found that only the neural representations formed in the hippocampus simultaneously encode several task variables in an abstract, or disentangled, format. This representational geometry is uniquely observed after patients learn to perform inference, and consists of disentangled directly observable and discovered latent task variables. Learning to perform inference by trial and error or through verbal instructions led to the formation of hippocampal representations with similar geometric properties. The observed relation between representational format and inference behaviour suggests that abstract and disentangled representational geometries are important for complex cognition.

Neural representational geometries reflect behavioral differences in monkeys and recurrent neural networks

Animals likely use a variety of strategies to solve laboratory tasks. Traditionally, combined analysis of behavioral and neural recording data across subjects employing different strategies may obscure important signals and give confusing results. Hence, it is essential to develop techniques that can infer strategy at the single-subject level. We analyzed an experiment in which two male monkeys performed a visually cued rule-based task. The analysis of their performance shows no indication that they used a different strategy. However, when we examined the geometry of stimulus representations in the state space of the neural activities recorded in dorsolateral prefrontal cortex, we found striking differences between the two monkeys. Our purely neural results induced us to reanalyze the behavior. The new analysis showed that the differences in representational geometry are associated with differences in the reaction times, revealing behavioral differences we were unaware of. All these analyses suggest that the monkeys are using different strategies. Finally, using recurrent neural network models trained to perform the same task, we show that these strategies correlate with the amount of training, suggesting a possible explanation for the observed neural and behavioral differences.

Irrational choices via a curvilinear representational geometry for value

We make decisions by comparing values, but it is not yet clear how value is represented in the brain. Many models assume, if only implicitly, that the representational geometry of value is linear. However, in part due to a historical focus on noisy single neurons, rather than neuronal populations, this hypothesis has not been rigorously tested. Here, we examine the representational geometry of value in the ventromedial prefrontal cortex (vmPFC), a part of the brain linked to economic decision-making, in two male rhesus macaques. We find that values are encoded along a curved manifold in vmPFC. This curvilinear geometry predicts a specific pattern of irrational decision-making: that decision-makers will make worse choices when an irrelevant, decoy option is worse in value, compared to when it is better. We observe this type of irrational choices in behavior. Together, these results not only suggest that the representational geometry of value is nonlinear, but that this nonlinearity could impose bounds on rational decision-making.

Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning.

Hierarchical models have been proposed to explain how the brain encodes actions, whereby different areas represent different features, such as gesture kinematics, target object, action goal, and meaning. The visual processing of action-related information is distributed over a well-known network of brain regions spanning separate anatomical areas, attuned to specific stimulus properties, and referred to as action observation network (AON). To determine the brain organization of these features, we measured representational geometries during the observation of a large set of transitive and intransitive gestures in two independent functional magnetic resonance imaging experiments. We provided evidence for a partial dissociation between kinematics, object characteristics, and action meaning in the occipito-parietal, ventro-temporal, and lateral occipito-temporal cortex, respectively. Importantly, most of the AON showed low specificity to all the explored features, and representational spaces sharing similar information content were spread across the cortex without being anatomically adjacent. Overall, our results support the notion that the AON relies on overlapping and distributed coding and may act as a unique representational space instead of mapping features in a modular and segregated manner.

A cortical surface template for human neuroscience

Neuroimaging data analysis relies on normalization to standard anatomical templates to resolve macroanatomical differences across brains. Existing human cortical surface templates sample locations unevenly because of distortions introduced by inflation of the folded cortex into a standard shape. Here we present the onavg template, which affords uniform sampling of the cortex. We created the onavg template based on openly available high-quality structural scans of 1,031 brains—25 times more than existing cortical templates. We optimized the vertex locations based on cortical anatomy, achieving an even distribution. We observed consistently higher multivariate pattern classification accuracies and representational geometry inter-participant correlations based on onavg than on other templates, and onavg only needs three-quarters as much data to achieve the same performance compared with other templates. The optimized sampling also reduces CPU time across algorithms by 1.3–22.4% due to less variation in the number of vertices in each searchlight.

Task structure tailors the geometry of neural representations in human lateral prefrontal cortex.

How do human brains represent tasks of varying structure? The lateral prefrontal cortex (lPFC) flexibly represents task information. However, principles that shape lPFC representational geometry remain unsettled. We use fMRI and pattern analyses to reveal the structure of lPFC representational geometries as humans perform two distinct categorization tasks- one with flat, conjunctive categories and another with hierarchical, context-dependent categories. We show that lPFC encodes task-relevant information with task-tailored geometries of intermediate dimensionality. These geometries preferentially enhance the separability of task-relevant variables while encoding a subset in abstract form. Specifically, in the flat task, a global axis encodes response-relevant categories abstractly, while category-specific local geometries are high-dimensional. In the hierarchy task, a global axis abstractly encodes the higher-level context, while low-dimensional, context-specific local geometries compress irrelevant information and abstractly encode the relevant information. Comparing these task geometries exposes generalizable principles by which lPFC tailors representations to different tasks.

Representations and generalization in artificial and brain neural networks

Humans and animals excel at generalizing from limited data, a capability yet to be fully replicated in artificial intelligence. This perspective investigates generalization in biological and artificial deep neural networks (DNNs), in both in-distribution and out-of-distribution contexts. We introduce two hypotheses: First, the geometric properties of the neural manifolds associated with discrete cognitive entities, such as objects, words, and concepts, are powerful order parameters. They link the neural substrate to the generalization capabilities and provide a unified methodology bridging gaps between neuroscience, machine learning, and cognitive science. We overview recent progress in studying the geometry of neural manifolds, particularly in visual object recognition, and discuss theories connecting manifold dimension and radius to generalization capacity. Second, we suggest that the theory of learning in wide DNNs, especially in the thermodynamic limit, provides mechanistic insights into the learning processes generating desired neural representational geometries and generalization. This includes the role of weight norm regularization, network architecture, and hyper-parameters. We will explore recent advances in this theory and ongoing challenges. We also discuss the dynamics of learning and its relevance to the issue of representational drift in the brain.

A population of neurons selective for human voice in the monkey brain

Many animals can extract useful information from the vocalizations of other species. Neuroimaging studies have evidenced areas sensitive to conspecific vocalizations in the cerebral cortex of primates, but how these areas process heterospecific vocalizations remains unclear. Using fMRI-guided electrophysiology, we recorded the spiking activity of individual neurons in the anterior temporal voice patches of two macaques while they listened to complex sounds including vocalizations from several species. In addition to cells selective for conspecific macaque vocalizations, we identified an unsuspected subpopulation of neurons with strong selectivity for human voice, not merely explained by spectral or temporal structure of the sounds. The auditory representational geometry implemented by these neurons was strongly related to that measured in the human voice areas with neuroimaging and only weakly to low-level acoustical structure. These findings provide new insights into the neural mechanisms involved in auditory expertise and the evolution of communication systems in primates.

Erratum: Cheng et al., "Representational Geometries Reveal Differential Effects of Response Correlations on Population Codes in Neurophysiology and Functional Magnetic Resonance Imaging".

Erratum: Cheng et al., "Representational Geometries Reveal Differential Effects of Response Correlations on Population Codes in Neurophysiology and Functional Magnetic Resonance Imaging".

Spontaneous eye movements reflect the representational geometries of conceptual spaces

Functional neuroimaging studies indicate that the human brain can represent concepts and their relational structure in memory using coding schemes typical of spatial navigation. However, whether we can read out the internal representational geometries of conceptual spaces solely from human behavior remains unclear. Here, we report that the relational structure between concepts in memory might be reflected in spontaneous eye movements during verbal fluency tasks: When we asked participants to randomly generate numbers, their eye movements correlated with distances along the left-to-right one-dimensional geometry of the number space (mental number line), while they scaled with distance along the ring-like two-dimensional geometry of the color space (color wheel) when they randomly generated color names. Moreover, when participants randomly produced animal names, eye movements correlated with low-dimensional similarity in word frequencies. These results suggest that the representational geometries used to internally organize conceptual spaces might be read out from gaze behavior.

Geometry of visuospatial working memory information in miniature gaze patterns.

Stimulus-dependent eye movements have been recognized as a potential confound in decoding visual working memory information from neural signals. Here we combined eye-tracking with representational geometry analyses to uncover the information in miniature gaze patterns while participants (n = 41) were cued to maintain visual object orientations. Although participants were discouraged from breaking fixation by means of real-time feedback, small gaze shifts (<1°) robustly encoded the to-be-maintained stimulus orientation, with evidence for encoding two sequentially presented orientations at the same time. The orientation encoding on stimulus presentation was object-specific, but it changed to a more object-independent format during cued maintenance, particularly when attention had been temporarily withdrawn from the memorandum. Finally, categorical reporting biases increased after unattended storage, with indications of biased gaze geometries already emerging during the maintenance periods before behavioural reporting. These findings disclose a wealth of information in gaze patterns during visuospatial working memory and indicate systematic changes in representational format when memory contents have been unattended.

The neuronal implementation of representational geometry in primate prefrontal cortex

Modern neuroscience has seen the rise of a population-doctrine that represents cognitive variables using geometrical structures in activity space. Representational geometry does not, however, account for how individual neurons implement these representations. Leveraging the principle of sparse coding, we present a framework to dissect representational geometry into biologically interpretable components that retain links to single neurons. Applied to extracellular recordings from the primate prefrontal cortex in a working memory task with interference, the identified components revealed disentangled and sequential memory representations including the recovery of memory content after distraction, signals hidden to conventional analyses. Each component was contributed by small subpopulations of neurons with distinct spiking properties and response dynamics. Modeling showed that such sparse implementations are supported by recurrently connected circuits as in prefrontal cortex. The perspective of neuronal implementation links representational geometries to their cellular constituents, providing mechanistic insights into how neural systems encode and process information.