Representational Geometry Research Articles

Functional and representational differences between bilateral inferior temporal numeral areas

The processing of numerals as visual objects is supported by an “Inferior Temporal Numeral Area” (ITNA) in the bilateral inferior temporal gyri (ITG). Extant findings suggest some degree of hemispheric asymmetry in how the bilateral ITNAs process numerals. Pollack and Price (2019) reported such a hemispheric asymmetry by which a region in the left ITG was sensitive to digits during a visual search for a digit among letters, and a homologous region in the right ITG that showed greater digit sensitivity in individuals with higher calculation skills. However, the ITG regions were localized with separate analyses without directly contrasting their digit sensitivities and relation to calculation skills. So, the extent of and reasons for these functional asymmetries remain unclear. Here we probe whether the functional and representational properties of the ITNAs are asymmetric by applying both univariate and multivariate region-of-interest analyses to Pollack and Price’s (2019) data. Contrary to the implications of the original findings, digit sensitivity did not differ between ITNAs, and digit sensitivity in both left and right ITNAs was associated with calculation skills. Representational similarity analyses revealed that the overall representational geometries of digits in the ITNAs were also correlated, albeit weakly, but the representational contents of the ITNAs were largely inconclusive. Nonetheless, we found a right lateralization in engagement in alphanumeric categorization, and that the right ITNA showed greater discriminability between digits and letters. Greater right lateralization of digit sensitivity and digit discriminability in the left ITNA were also related to higher calculation skills. Our findings thus suggest that the ITNAs may not be functionally identical and should be directly contrasted in future work. Our study also highlights the importance of within-individual comparisons for understanding hemispheric asymmetries, and analyses of individual differences and multivariate features to uncover effects that would otherwise be obscured by averages.

Modeling naturalistic face processing in humans with deep convolutional neural networks.

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.

Parietal-driven visual working memory representation in occipito-temporal cortex

Human fMRI studies have documented extensively that the content of visual working memory (VWM) can be reliably decoded from fMRI voxel response patterns during the delay period in both the occipito-temporal cortex (OTC), including early visual areas (EVC), and the posterior parietal cortex (PPC).1,2,3,4 Further work has revealed that VWM signal in OTC is largely sustained by feedback from associative areas such as prefrontal cortex (PFC) and PPC.4,5,6,7,8,9 It is unclear, however, if feedback during VWM simply restores sensory representations initially formed in OTC or if it can reshape the representational content of OTC during VWM delay. Taking advantage of a recent finding showing that object representational geometry differs between OTC and PPC in perception,10 here we find that, during VWM delay, the object representational geometry in OTC becomes more aligned with that of PPC during perception than with itself during perception. This finding supports the role of feedback in shaping the content of VWM in OTC, with the VWM content of OTC more determined by information retained in PPC than by the sensory information initially encoded in OTC.

Dimensional emotions are represented by distinct topographical brain networks

The ability to recognize others’ facial emotions has become increasingly important after the COVID-19 pandemic, which causes stressful situations in emotion regulation. Considering the importance of emotion in maintaining a social life, emotion knowledge to perceive and label emotions of oneself and others requires an understanding of affective dimensions, such as emotional valence and emotional arousal. However, limited information is available about whether the behavioral representation of affective dimensions is similar to their neural representation. To explore the relationship between the brain and behavior in the representational geometries of affective dimensions, we constructed a behavioral paradigm in which emotional faces were categorized into geometric spaces along the valence, arousal, and valence and arousal dimensions. Moreover, we compared such representations to neural representations of the faces acquired by functional magnetic resonance imaging. We found that affective dimensions were similarly represented in the behavior and brain. Specifically, behavioral and neural representations of valence were less similar to those of arousal. We also found that valence was represented in the dorsolateral prefrontal cortex, frontal eye fields, precuneus, and early visual cortex, whereas arousal was represented in the cingulate gyrus, middle frontal gyrus, orbitofrontal cortex, fusiform gyrus, and early visual cortex. In conclusion, the current study suggests that dimensional emotions are similarly represented in the behavior and brain and are presented with differential topographical organizations in the brain.

Statistical inference on representational geometries.

Neuroscience has recently made much progress, expanding the complexity of both neural activity measurements and brain-computational models. However, we lack robust methods for connecting theory and experiment by evaluating our new big models with our new big data. Here, we introduce new inference methods enabling researchers to evaluate and compare models based on the accuracy of their predictions of representational geometries: A good model should accurately predict the distances among the neural population representations (e.g. of a set of stimuli). Our inference methods combine novel 2-factor extensions of crossvalidation (to prevent overfitting to either subjects or conditions from inflating our estimates of model accuracy) and bootstrapping (to enable inferential model comparison with simultaneous generalization to both new subjects and new conditions). We validate the inference methods on data where the ground-truth model is known, by simulating data with deep neural networks and by resampling of calcium-imaging and functional MRI data. Results demonstrate that the methods are valid and conclusions generalize correctly. These data analysis methods are available in an open-source Python toolbox (rsatoolbox.readthedocs.io).

Statistical inference on representational geometries

Statistical inference on representational geometries

Unveiling the benefits of multitasking in disentangled representation formation.

Johnston and Fusi recently investigated the emergence of disentangled representations when a neural network was trained to perform multiple simultaneous tasks. Such experiments explore the benefits of flexible representations and add to a growing field of research investigating the representational geometry of artificial and biological neural networks.

Abstraction of Reward Context Facilitates Relative Reward Coding in Neural Populations of the Macaque Anterior Cingulate Cortex.

The anterior cingulate cortex (ACC) is believed to be involved in many cognitive processes, including linking goals to actions and tracking decision-relevant contextual information. ACC neurons robustly encode expected outcomes, but how this relates to putative functions of ACC remains unknown. Here, we approach this question from the perspective of population codes by analyzing neural spiking data in the ventral and dorsal banks of the ACC in two male monkeys trained to perform a stimulus-motor mapping task to earn rewards or avoid losses. We found that neural populations favor a low dimensional representational geometry that emphasizes the valence of potential outcomes while also facilitating the independent, abstract representation of multiple task-relevant variables. Valence encoding persisted throughout the trial, and realized outcomes were primarily encoded in a relative sense, such that cue valence acted as a context for outcome encoding. This suggests that the population coding we observe could be a mechanism that allows feedback to be interpreted in a context-dependent manner. Together, our results point to a prominent role for ACC in context setting and relative interpretation of outcomes, facilitated by abstract, or untangled, representations of task variables.SIGNIFICANCE STATEMENT The ability to interpret events in light of the current context is a critical facet of higher-order cognition. The ACC is suggested to be important for tracking contextual information, whereas alternate views hold that its function is more related to the motor system and linking goals to appropriate actions. We evaluated these possibilities by analyzing geometric properties of neural population activity in monkey ACC when contexts were determined by the valence of potential outcomes and found that this information was represented as a dominant, abstract concept. Ensuing outcomes were then coded relative to these contexts, suggesting an important role for these representations in context-dependent evaluation. Such mechanisms may be critical for the abstract reasoning and generalization characteristic of biological intelligence.

Representational geometry of incomplete faces in macaque face patches

The neural code of faces has been intensively studied in the macaque face patch system. Although the majority of previous studies used complete faces as stimuli, faces are often seen partially in daily life. Here, we investigated how face-selective cells represent two types of incomplete faces: face fragments and occluded faces, with the location of the fragment/occluder and the facial features systematically varied. Contrary to popular belief, we found that the preferred face regions identified with two stimulus types are dissociated in many face cells. This dissociation can be explained by the nonlinear integration of information from different face parts and is closely related to a curved representation of face completeness in the state space, which allows a clear discrimination between different stimulus types. Furthermore, identity-related facial features are represented in a subspace orthogonal to the nonlinear dimension of face completeness, supporting a condition-general code of facial identity.

Decoding and geometry of ten finger movements in human posterior parietal cortex and motor cortex

Objective. Enable neural control of individual prosthetic fingers for participants with upper-limb paralysis. Approach. Two tetraplegic participants were each implanted with a 96-channel array in the left posterior parietal cortex (PPC). One of the participants was additionally implanted with a 96-channel array near the hand knob of the left motor cortex (MC). Across tens of sessions, we recorded neural activity while the participants attempted to move individual fingers of the right hand. Offline, we classified attempted finger movements from neural firing rates using linear discriminant analysis with cross-validation. The participants then used the neural classifier online to control individual fingers of a brain–machine interface (BMI). Finally, we characterized the neural representational geometry during individual finger movements of both hands. Main Results. The two participants achieved 86% and 92% online accuracy during BMI control of the contralateral fingers (chance = 17%). Offline, a linear decoder achieved ten-finger decoding accuracies of 70% and 66% using respective PPC recordings and 75% using MC recordings (chance = 10%). In MC and in one PPC array, a factorized code linked corresponding finger movements of the contralateral and ipsilateral hands. Significance. This is the first study to decode both contralateral and ipsilateral finger movements from PPC. Online BMI control of contralateral fingers exceeded that of previous finger BMIs. PPC and MC signals can be used to control individual prosthetic fingers, which may contribute to a hand restoration strategy for people with tetraplegia.

Representational Geometries Reveal Differential Effects of Response Correlations on Population Codes in Neurophysiology and Functional Magnetic Resonance Imaging.

Two sensory neurons usually display trial-by-trial spike-count correlations given the repeated representations of a stimulus. The effects of such response correlations on population-level sensory coding have been the focal contention in computational neuroscience over the past few years. In the meantime, multivariate pattern analysis (MVPA) has become the leading analysis approach in functional magnetic resonance imaging (fMRI), but the effects of response correlations among voxel populations remain underexplored. Here, instead of conventional MVPA analysis, we calculate linear Fisher information of population responses in human visual cortex (five males, one female) and hypothetically remove response correlations between voxels. We found that voxelwise response correlations generally enhance stimulus information, a result standing in stark contrast to the detrimental effects of response correlations reported in empirical neurophysiological studies. By voxel-encoding modeling, we further show that these two seemingly opposite effects actually can coexist within the primate visual system. Furthermore, we use principal component analysis to decompose stimulus information in population responses onto different principal dimensions in a high-dimensional representational space. Interestingly, response correlations simultaneously reduce and enhance information on higher- and lower-variance principal dimensions, respectively. The relative strength of the two antagonistic effects within the same computational framework produces the apparent discrepancy in the effects of response correlations in neuronal and voxel populations. Our results suggest that multivariate fMRI data contain rich statistical structures that are directly related to sensory information representation, and the general computational framework to analyze neuronal and voxel population responses can be applied in many types of neural measurements.SIGNIFICANCE STATEMENT Despite the vast research interest in the effect of spike-count noise correlations on population codes in neurophysiology, it remains unclear how the response correlations between voxels influence MVPA in human imaging. We used an information-theoretic approach and showed that unlike the detrimental effects of response correlations reported in neurophysiology, voxelwise response correlations generally improve sensory coding. We conducted a series of in-depth analyses and demonstrated that neuronal and voxel response correlations can coexist within the visual system and share some common computational mechanisms. These results shed new light on how the population codes of sensory information can be evaluated via different neural measurements.

Comparing the Dominance of Color and Form Information across the Human Ventral Visual Pathway and Convolutional Neural Networks.

Color and form information can be decoded in every region of the human ventral visual hierarchy, and at every layer of many convolutional neural networks (CNNs) trained to recognize objects, but how does the coding strength of these features vary over processing? Here, we characterize for these features both their absolute coding strength-how strongly each feature is represented independent of the other feature-and their relative coding strength-how strongly each feature is encoded relative to the other, which could constrain how well a feature can be read out by downstream regions across variation in the other feature. To quantify relative coding strength, we define a measure called the form dominance index that compares the relative influence of color and form on the representational geometry at each processing stage. We analyze brain and CNN responses to stimuli varying based on color and either a simple form feature, orientation, or a more complex form feature, curvature. We find that while the brain and CNNs largely differ in how the absolute coding strength of color and form vary over processing, comparing them in terms of their relative emphasis of these features reveals a striking similarity: For both the brain and for CNNs trained for object recognition (but not for untrained CNNs), orientation information is increasingly de-emphasized, and curvature information is increasingly emphasized, relative to color information over processing, with corresponding processing stages showing largely similar values of the form dominance index.

Feature-based attention warps the perception of visual features

Selective attention improves sensory processing of relevant information but can also impact the quality of perception. For example, attention increases visual discrimination performance and at the same time boosts apparent stimulus contrast of attended relative to unattended stimuli. Can attention also lead to perceptual distortions of visual representations? Optimal tuning accounts of attention suggest that processing is biased towards “off-tuned” features to maximize the signal-to-noise ratio in favor of the target, especially when targets and distractors are confusable. Here, we tested whether such tuning gives rise to phenomenological changes of visual features. We instructed participants to select a color among other colors in a visual search display and subsequently asked them to judge the appearance of the target color in a 2-alternative forced choice task. Participants consistently judged the target color to appear more dissimilar from the distractor color in feature space. Critically, the magnitude of these perceptual biases varied systematically with the similarity between target and distractor colors during search, indicating that attentional tuning quickly adapts to current task demands. In control experiments we rule out possible non-attentional explanations such as color contrast or memory effects. Overall, our results demonstrate that selective attention warps the representational geometry of color space, resulting in profound perceptual changes across large swaths of feature space. Broadly, these results indicate that efficient attentional selection can come at a perceptual cost by distorting our sensory experience.

Deep learning models to study sentence comprehension in the human brain

ABSTRACT Recent artificial neural networks that process natural language achieve unprecedented performance in tasks requiring sentence-level understanding. As such, they could be interesting models of the integration of linguistic information in the human brain. We review works that compare these artificial language models with human brain activity and we assess the extent to which this approach has improved our understanding of the neural processes involved in natural language comprehension. Two main results emerge. First, the neural representation of word meaning aligns with the context-dependent, dense word vectors used by the artificial neural networks. Second, the processing hierarchy that emerges within artificial neural networks broadly matches the brain, but is surprisingly inconsistent across studies. We discuss current challenges in establishing artificial neural networks as process models of natural language comprehension. We suggest exploiting the highly structured representational geometry of artificial neural networks when mapping representations to brain data.

Geometric transformation of cognitive maps for generalization across hippocampal-prefrontal circuits

The ability to abstract information to guide decisions during navigation across changing environments is essential for adaptation and requires the integrity of the hippocampal-prefrontal circuitry. The hippocampus encodes navigational information in a cognitive map, but it remains unclear how cognitive maps are transformed across hippocampal-prefrontal circuits to support abstraction and generalization. Here, we simultaneously record hippocampal-prefrontal ensembles as rats generalize navigational rules across distinct environments. We find that, whereas hippocampal representational maps maintain specificity of separate environments, prefrontal maps generalize across environments. Furthermore, while both maps are structured within a neural manifold of population activity, they have distinct representational geometries. Prefrontal geometry enables abstraction of rule-informative variables, a representational format that generalizes to novel conditions of existing variable classes. Hippocampal geometry lacks such abstraction. Together, these findings elucidate how cognitive maps are structured into distinct geometric representations to support abstraction and generalization while maintaining memory specificity.

Abstract representations emerge naturally in neural networks trained to perform multiple tasks

Humans and other animals demonstrate a remarkable ability to generalize knowledge across distinct contexts and objects during natural behavior. We posit that this ability to generalize arises from a specific representational geometry, that we call abstract and that is referred to as disentangled in machine learning. These abstract representations have been observed in recent neurophysiological studies. However, it is unknown how they emerge. Here, using feedforward neural networks, we demonstrate that the learning of multiple tasks causes abstract representations to emerge, using both supervised and reinforcement learning. We show that these abstract representations enable few-sample learning and reliable generalization on novel tasks. We conclude that abstract representations of sensory and cognitive variables may emerge from the multiple behaviors that animals exhibit in the natural world, and, as a consequence, could be pervasive in high-level brain regions. We also make several specific predictions about which variables will be represented abstractly.

Morphology, Connectivity, and Encoding Features of Tactile and Motor Representations of the Fingers in the Human Precentral and Postcentral Gyrus.

Despite the tight coupling between sensory and motor processing for fine manipulation in humans, it is not yet totally clear which specific properties of the fingers are mapped in the precentral and postcentral gyrus. We used fMRI to compare the morphology, connectivity, and encoding of the motor and tactile finger representations (FRs) in the precentral and postcentral gyrus of 25 5-fingered participants (8 females). Multivoxel pattern and structural and functional connectivity analyses demonstrated the existence of distinct motor and tactile FRs within both the precentral and postcentral gyrus, integrating finger-specific motor and tactile information. Using representational similarity analysis, we found that the motor and tactile FRs in the sensorimotor cortex were described by the perceived structure of the hand better than by the actual hand anatomy or other functional models (finger kinematics, muscles synergies). We then studied a polydactyly individual (i.e., with a congenital 6-fingered hand) showing superior manipulation abilities and divergent anatomic-functional hand properties. The perceived hand model was still the best model for tactile representations in the precentral and postcentral gyrus, while finger kinematics better described motor representations in the precentral gyrus. We suggest that, under normal conditions (i.e., in subjects with a standard hand anatomy), the sensorimotor representations of the 5 fingers in humans converge toward a model of perceived hand anatomy, deviating from the real hand structure, as the best synthesis between functional and structural features of the hand.SIGNIFICANCE STATEMENT Distinct motor and tactile finger representations exist in both the precentral and postcentral gyrus, supported by a finger-specific pattern of anatomic and functional connectivity across modalities. At the representational level, finger representations reflect the perceived structure of the hand, which might result from an adapting process harmonizing (i.e., uniformizing) the encoding of hand function and structure in the precentral and postcentral gyrus. The same analyses performed in an extremely rare polydactyly subject showed that the emergence of such representational geometry is also found in neuromechanical variants with different hand anatomy and function. However, the harmonization process across the precentral and postcentral gyrus might not be possible because of divergent functional-structural properties of the hand and associated superior manipulation abilities.

Multi-head second-order pooling for graph transformer networks

Graph transformer networks (GTNs) have great potential in graph-related tasks, particularly graph classification. GTNs use self-attention mechanism to extract both semantic and structural information, after which a class token is used as the global representation for graph classification.However, the class token completely abandons all information held by nodes, which leads to a huge loss of information and neglects the dependencies among different feature dimensions. Therefore, we propose a Multi-head Global Second-Order Pooling (MGSOP) to utilize the rich second-order statistics of all nodes. To this end, a second-order pooling module incorporated with multi-head structure generates multiple covariance representations for GTNs. Then, the first-order representation and concatenation of all covariance matrices are combined by an adaptive fusion module for better performance on different types of data. Furthermore, we introduce an efficient dropout-based normalization to utilize the geometry of covariance representation. Experiments conducted on six real-world graph benchmarks show our MGSOP outperforms its counterparts and achieves competitive results against state-of-the-arts.

Harmonic Oscillator Coherent States from the Standpoint of Orbit Theory

We study the known coherent states of a quantum harmonic oscillator from the standpoint of the originally developed noncommutative integration method for linear partial differential equations. The application of the method is based on the symmetry properties of the Schrödinger equation and on the orbit geometry of the coadjoint representation of Lie groups. We have shown that analogs of coherent states constructed by the noncommutative integration can be expressed in terms of the solution to a system of differential equations on the Lie group of the oscillatory Lie algebra. The solutions constructed are directly related to irreducible representation of the Lie algebra on the Hilbert space functions on the Lagrangian submanifold to the orbit of the coadjoint representation.

The geometry of cortical representations of touch in rodents.

Neurons often encode highly heterogeneous non-linear functions of multiple task variables, a signature of a high-dimensional geometry. We studied the representational geometry in the somatosensory cortex of mice trained to report the curvature of objects touched by their whiskers. High-speed videos of the whiskers revealed that the task can be solved by linearly integrating multiple whisker contacts over time. However, the neural activity in somatosensory cortex reflects non-linear integration of spatio-temporal features of the sensory inputs. Although the responses at first appeared disorganized, we identified an interesting structure in the representational geometry: different whisker contacts are disentangled variables represented in approximately, but not fully, orthogonal subspaces of the neural activity space. This geometry allows linear readouts to perform a broad class of tasks of different complexities without compromising the ability to generalize to novel situations.