Representational Geometry Research Articles

The role of stimulus-based cues and conceptual information in processing facial expressions of emotion

Face shape and surface textures are two important cues that aid in the perception of facial expressions of emotion. Additionally, this perception is also influenced by high-level emotion concepts. Across two studies, we use representational similarity analysis to investigate the relative roles of shape, surface, and conceptual information in the perception, categorisation, and neural representation of facial expressions. In Study 1, 50 participants completed a perceptual task designed to measure the perceptual similarity of expression pairs, and a categorical task designed to measure the confusability between expression pairs when assigning emotion labels to a face. We used representational similarity analysis and constructed three models of the similarities between emotions using distinct information. Two models were based on stimulus-based cues (face shapes and surface textures) and one model was based on emotion concepts. Using multiple linear regression, we found that behaviour during both tasks was related with the similarity of emotion concepts. The model based on face shapes was more related with behaviour in the perceptual task than in the categorical, and the model based on surface textures was more related with behaviour in the categorical than the perceptual task. In Study 2, 30 participants viewed facial expressions while undergoing fMRI, allowing for the measurement of brain representational geometries of facial expressions of emotion in three core face-responsive regions (the Fusiform Face Area, Occipital Face Area, and Superior Temporal Sulcus), and a region involved in theory of mind (Medial Prefrontal Cortex). Across all four regions, the representational distances between facial expression pairs were related to the similarities of emotion concepts, but not to either of the stimulus-based cues. Together, these results highlight the important top-down influence of high-level emotion concepts both in behavioural tasks and in the neural representation of facial expressions.

Neural tuning and representational geometry.

A central goal of neuroscience is to understand the representations formed by brain activity patterns and their connection to behaviour. The classic approach is to investigate how individual neurons encode stimuli and how their tuning determines the fidelity of the neural representation. Tuning analyses often use the Fisher information to characterize the sensitivity of neural responses to small changes of the stimulus. In recent decades, measurements of large populations of neurons have motivated a complementary approach, which focuses on the information available to linear decoders. The decodable information is captured by the geometry of the representational patterns in the multivariate response space. Here we review neural tuning and representational geometry with the goal of clarifying the relationship between them. The tuning induces the geometry, but different sets of tuned neurons can induce the same geometry. The geometry determines the Fisher information, the mutual information and the behavioural performance of an ideal observer in a range of psychophysical tasks. We argue that future studies can benefit from considering both tuning and geometry to understand neural codes and reveal the connections between stimuli, brain activity and behaviour.

Which deep learning model can best explain object representations of within-category exemplars?

Deep neural network (DNN) models realize human-equivalent performance in tasks such as object recognition. Recent developments in the field have enabled testing the hierarchical similarity of object representation between the human brain and DNNs. However, the representational geometry of object exemplars within a single category using DNNs is unclear. In this study, we investigate which DNN model has the greatest ability to explain invariant within-category object representations by computing the similarity between representational geometries of visual features extracted at the high-level layers of different DNN models. We also test for the invariability of within-category object representations of these models by identifying object exemplars. Our results show that transfer learning models based on ResNet50 best explained both within-category object representation and object identification. These results suggest that the invariability of object representations in deep learning depends not on deepening the neural network but on building a better transfer learning model.

Functionally homologous representation of vocalizations in the auditory cortex of humans and macaques

SummaryHow the evolution of speech has transformed the human auditory cortex compared to other primates remains largely unknown. While primary auditory cortex is organized largely similarly in humans and macaques,1 the picture is much less clear at higher levels of the anterior auditory pathway,2 particularly regarding the processing of conspecific vocalizations (CVs). A “voice region” similar to the human voice-selective areas3,4 has been identified in the macaque right anterior temporal lobe with functional MRI;5 however, its anatomical localization, seemingly inconsistent with that of the human temporal voice areas (TVAs), has suggested a “repositioning of the voice area” in recent human evolution.6 Here we report a functional homology in the cerebral processing of vocalizations by macaques and humans, using comparative fMRI and a condition-rich auditory stimulation paradigm. We find that the anterior temporal lobe of both species possesses cortical voice areas that are bilateral and not only prefer conspecific vocalizations but also implement a representational geometry categorizing them apart from all other sounds in a species-specific but homologous manner. These results reveal a more similar functional organization of higher-level auditory cortex in macaques and humans than currently known.

Comparing representational geometries using whitened unbiased-distance-matrix similarity

Representational similarity analysis (RSA) tests models of brain computation by investigating how neural activity patterns reflect experimental conditions. Instead of predicting activity patterns directly, the models predict the geometry of the representation, as defined by the representational dissimilarity matrix (RDM), which captures how similar or dissimilar different activity patterns associated by different experimental conditions are. RSA therefore first quantifies the representational geometry by calculating a dissimilarity measure for each pair of conditions, and then compares the estimated representational dissimilarities to those predicted by each model. Here we address two central challenges of RSA: First, dissimilarity measures such as the Euclidean, Mahalanobis, and correlation distance, are biased by measurement noise, which can lead to incorrect inferences. Unbiased dissimilarity estimates can be obtained by crossvalidation, at the price of increased variance. Second, the pairwise dissimilarity estimates are not statistically independent, and ignoring this dependency makes model comparison statistically suboptimal. We present an analytical expression for the mean and (co)variance of both biased and unbiased estimators of the squared Euclidean and Mahalanobis distance, allowing us to quantify the bias-variance trade-off. We also use the analytical expression of the covariance of the dissimilarity estimates to whiten the RDM estimation errors. This results in a new criterion for RDM similarity, the whitened unbiased RDM cosine similarity (WUC), which allows for near-optimal model selection combined with robustness to correlated measurement noise.

Between-subject prediction reveals a shared representational geometry in the rodent hippocampus

The rodent hippocampus constructs statistically independent representations across environments ("global remapping") and assigns individual neuron firing fields to locations within an environment in an apparently random fashion, processes thought to contribute to the role of the hippocampus in episodic memory. This random mapping implies that it should be challenging to predict hippocampal encoding of a given experience in one subject based on the encoding of that same experience in another subject. Contrary to this prediction, we find that by constructing a common representational space across rats in which neural activity is aligned using geometric operations (rotation, reflection, and translation; "hyperalignment"), we can predict data of "right" trials (R) on a T-maze in a target rat based on (1) the "left" trials (L) of the target rat and (2) the relationship between L and R trials from a different source rat. These cross-subject predictions relied on ensemble activity patterns, including both firing rate and field location, and outperformed a number of control mappings, such as those based on permuted data that broke the relationship between L and R activity for individual neurons and those based solely on within-subject prediction. This work constitutes proof of principle for successful cross-subject prediction of ensemble activity patterns in the hippocampus and provides new insights in understanding how different experiences are structured, enabling further work identifying what aspects of experience encoding are shared versus unique to an individual.

Diverse Deep Neural Networks All Predict Human Inferior Temporal Cortex Well, After Training and Fitting

Deep neural networks (DNNs) trained on object recognition provide the best current models of high-level visual cortex. What remains unclear is how strongly experimental choices, such as network architecture, training, and fitting to brain data, contribute to the observed similarities. Here, we compare a diverse set of nine DNN architectures on their ability to explain the representational geometry of 62 object images in human inferior temporal cortex (hIT), as measured with fMRI. We compare untrained networks to their task-trained counterparts and assess the effect of cross-validated fitting to hIT, by taking a weighted combination of the principal components of features within each layer and, subsequently, a weighted combination of layers. For each combination of training and fitting, we test all models for their correlation with the hIT representational dissimilarity matrix, using independent images and subjects. Trained models outperform untrained models (accounting for 57% more of the explainable variance), suggesting that structured visual features are important for explaining hIT. Model fitting further improves the alignment of DNN and hIT representations (by 124%), suggesting that the relative prevalence of different features in hIT does not readily emerge from the Imagenet object-recognition task used to train the networks. The same models can also explain the disparate representations in primary visual cortex (V1), where stronger weights are given to earlier layers. In each region, all architectures achieved equivalently high performance once trained and fitted. The models' shared properties-deep feedforward hierarchies of spatially restricted nonlinear filters-seem more important than their differences, when modeling human visual representations.

Learning and Representation of Hierarchical Concepts in Hippocampus and Prefrontal Cortex.

A key aspect of conceptual knowledge is that it can be flexibly applied at different levels of abstraction, implying a hierarchical organization. It is yet unclear how this hierarchical structure is acquired and represented in the brain. Here we investigate the computations underlying the acquisition and representation of the hierarchical structure of conceptual knowledge in the hippocampal-prefrontal system of 32 human participants (22 females). We assessed the hierarchical nature of learning during a novel tree-like categorization task via computational model comparisons. The winning model allowed to extract and quantify estimates for accumulation and updating of hierarchical compared with single-feature-based concepts from behavior. We find that mPFC tracks accumulation of hierarchical conceptual knowledge over time, and mPFC and hippocampus both support trial-to-trial updating. As a function of those learning parameters, mPFC and hippocampus further show connectivity changes to rostro-lateral PFC, which ultimately represented the hierarchical structure of the concept in the final stages of learning. Our results suggest that mPFC and hippocampus support the integration of accumulated evidence and instantaneous updates into hierarchical concept representations in rostro-lateral PFC.SIGNIFICANCE STATEMENT A hallmark of human cognition is the flexible use of conceptual knowledge at different levels of abstraction, ranging from a coarse category level to a fine-grained subcategory level. While previous work probed the representational geometry of long-term category knowledge, it is unclear how this hierarchical structure inherent to conceptual knowledge is acquired and represented. By combining a novel hierarchical concept learning task with computational modeling of categorization behavior and concurrent fMRI, we differentiate the roles of key concept learning regions in hippocampus and PFC in learning computations and the representation of a hierarchical category structure.

Representation learning of resting state fMRI with variational autoencoder

Resting state functional magnetic resonance imaging (rsfMRI) data exhibits complex but structured patterns. However, the underlying origins are unclear and entangled in rsfMRI data. Here we establish a variational auto-encoder, as a generative model trainable with unsupervised learning, to disentangle the unknown sources of rsfMRI activity. After being trained with large data from the Human Connectome Project, the model has learned to represent and generate patterns of cortical activity and connectivity using latent variables. The latent representation and its trajectory represent the spatiotemporal characteristics of rsfMRI activity. The latent variables reflect the principal gradients of the latent trajectory and drive activity changes in cortical networks. Representational geometry captured as covariance or correlation between latent variables, rather than cortical connectivity, can be used as a more reliable feature to accurately identify subjects from a large group, even if only a short period of data is available in each subject. Our results demonstrate that VAE is a valuable addition to existing tools, particularly suited for unsupervised representation learning of resting state fMRI activity.

The Geometry of Domain-General Performance Monitoring in the Human Medial Frontal Cortex

Flexibly controlling behavior to achieve a desired goal depends on the ability to monitor one’s own performance. A key open question is how performance monitoring can be both flexible to support multiple tasks and specialized to support specific tasks. We characterized performance monitoring representations by recording single neurons in the human medial frontal cortex (MFC). Subjects performed two tasks that involve three types of cognitive conflict. Neurons encoding error, conflict and predicted control demand in one or both tasks coexisted in the same population. Collectively, they gave rise to a representational geometry that simultaneously allowed task specialization and generalization. Representations of conflicts were compositional. Neurons encoding conflict retrospectively served to update estimates of control demand as predicted by a Bayesian model. These findings reveal how the MFC representation of evaluative signals are both abstract and specific, suggesting a mechanism for computing and maintaining control demand estimates across trials and tasks.

Representational geometry of perceptual decisions in the monkey parietal cortex

Lateral intraparietal (LIP) neurons represent formation of perceptual decisions involving eye movements. In circuit models for these decisions, neural ensembles that encode actions compete to form decisions. Consequently, representation and readout of the decision variables (DVs) are implemented similarly for decisions with identical competing actions, irrespective of input and task context differences. Further, DVs are encoded as partially potentiated action plans through balance of activity of action-selective ensembles. Here, we test those core principles. We show that in a novel face-discrimination task, LIP firing rates decrease with supporting evidence, contrary to conventional motion-discrimination tasks. These opposite response patterns arise from similar mechanisms in which decisions form along curved population-response manifolds misaligned with action representations. These manifolds rotate in state space based on context, indicating distinct optimal readouts for different tasks. We show similar manifolds in lateral and medial prefrontal cortices, suggesting similar representational geometry across decision-making circuits.

Ferro-nanofluid squeeze film lubrication with heat transfer

A theoretical study of a squeezing ferro-nanofluid flow including thermal effects is carried out with application to bearings and articular cartilages. A representational geometry of the thin layer of a ferro-nanofluid squeezed between a flat rigid disk and a thin porous bed is considered. The flow behaviours and heat transfer in the fluid and porous regions are investigated. The mathematical problem is formulated based on the Neuringer–Rosensweig model for ferro-nanofluids in the fluid region including an external magnetic field, Darcy law for the porous region and Beavers–Joseph slip condition at the fluid–porous interface. The expressions for velocity, fluid film thickness, contact time, fluid flux, streamlines, pathlines, mean temperature and heat transfer rate in the fluid and porous regions are obtained by using a perturbation method. An asymptotic solution for the fluid layer thickness is also presented. The problem is also solved by a numerical method and the results by asymptotic analysis, perturbation and numerical methods are obtained assuming a constant force squeezing state and are compared. It is shown that the results obtained by all the methods agree well with each other. The effects of various parameters such as Darcy number, Beavers–Joseph constant and magnetization parameter on the flow behaviours, contact time, mean temperature and heat transfer rate are investigated. The novel results showing the impact of using ferro-nanofluids in the two applications under consideration are presented. The results under special cases are further compared with the existing results in the literature and are found to agree well.

Species-Specific, But Functionally Homologous Representation of Vocalizations in the Auditory Cortex of Humans and Macaques

How the evolution of speech has transformed the human auditory cortex compared to other primates remains unclear. Here we report a functional homology in the cerebral processing of vocalizations by macaques and humans, using comparative fMRI and a condition-rich auditory stimulation paradigm. We find that the anterior temporal lobe of both species possess cortical voice areas that not only prefer conspecific vocalizations but also implement a representational geometry categorizing them apart from all other sounds in a species-specific but homologous manner. These results reveal a more similar functional organization of higher-level auditory cortex in macaques and humans than currently known.

Scene context shapes category representational geometry during processing of tools

Tools are ubiquitous in human environments and to think about them we use concepts. Increasingly, conceptual representation is thought to be dynamic and sensitive to the goals of the observer. Indeed, observer goals can reshape representational geometry within cortical networks supporting concepts. In the present study, we investigated the novel hypothesis that task-irrelevant scene context may implicitly alter the representational geometry of regions within the tool network. Participants performed conceptual judgments on images of tools embedded in scenes that either suggested their use (i.e., a kitchen timer sitting on a kitchen counter with vegetables in a frying pan) or that they would simply be moved (i.e., a kitchen timer sitting in an open drawer with other miscellaneous kitchen items around). We investigated whether representations in the tool network reflect category, grip, and shape information using a representational similarity analysis (RSA). We show that a) a number of regions of the tool network reflect category information about tools and b) category information predicts patterns in supramarginal gyrus more strongly in use contexts than in move contexts. Together, these results show that information about tool category is distributed across different regions of the tool network and that scene context helps shape the representational geometry of the tool network.

The representational dynamics of perceived voice emotions evolve from categories to dimensions.

Long-standing affective science theories conceive the perception of emotional stimuli either as discrete categories (e.g., an angry voice) or continuous dimensional attributes (e.g., an intense and negative vocal emotion). Which position provides a better account is still widely debated. Here, we contrast them to account for acoustics-independent perceptual and cerebral representational geometry of perceived voice emotions. We combined multimodal imaging of the cerebral response to heard vocal stimuli (functional magnetic resonance imaging – fMRI – and magneto-encephalography – MEG) with post-scanning behavioral assessment of voice emotion perception. By using representational similarity analysis (RSA), we find that categories prevail in perceptual and early (< 200ms) fronto-temporal cerebral representational geometries and that dimensions impinge predominantly on a later limbic-temporal network (240ms and > 500ms). These results reconcile the two long-opposing views by reframing the perception of emotions as the interplay of cerebral networks with different representational dynamics that emphasize either categories or dimensions.

Relation between palm and finger cortical representations in primary somatosensory cortex: A 7T fMRI study.

Many studies focused on the cortical representations of fingers, while the palm is relatively neglected despite its importance for hand function. Here, we investigated palm representation (PR) and its relationship with finger representations (FRs) in primary somatosensory cortex (S1). Few studies in humans suggested that PR is located medially with respect to FRs in S1, yet to date, no study directly quantified the somatotopic organization of PR and the five FRs. Importantly, the link between the somatotopic organization of PR and FRs and their activation properties remains largely unexplored. Using 7T fMRI, we mapped PR and the five FRs at the single subject level. First, we analyzed the cortical distance between PR and FRs to determine their somatotopic organization. Results show that PR was located medially with respect to D5. Second, we tested whether the observed cortical distances would predict the relationship between PR and FRs activations. Using three complementary measures (cross‐activations, pattern similarity and resting‐state connectivity), we show that the relationship between PR and FRs activations were not determined by their somatotopic organization, that is, there was no gradient moving from D5 to D1, except for resting‐state connectivity, which was predicted by the somatotopy. Instead, we show that the representational geometry of PR and FRs activations reflected the physical structure of the hand. Collectively, our findings suggest that the spatial proximity between topographically organized neuronal populations do not necessarily predicts their functional properties, rather the structure of the sensory space (e.g., the hand shape) better describes the observed results.

Within-category representational stability through the lens of manipulable objects

Our ability to recognize an object amongst many exemplars is one of our most important features, and one that putatively distinguishes humans from non-human animals and potentially from (current) computational and artificial intelligence models. We can recognize objects consistently regardless of when we see them suggesting that we have stable representations across time and different contexts. Importantly, little is known about how humans can replicate within-category object representations across time. Here, we investigate neural stability of within-category object representations by computing the similarity between representational geometries of activity patterns for 80 images of tools obtained on different functional magnetic resonance imaging (fMRI) scanning days. We show that within-category representational stability is observable in regions that span lateral and ventral temporal cortex, inferior and superior parietal cortex, and premotor cortex – regions typically associated with tool processing and visuospatial processing. We then focus on what kinds of representations best explain the representational geometries within these regions. We test the similarity of these geometries with those coming from the different layers of a convolutional neural network, and those coming from perceived and veridical visual similarity models. We find that regions supporting within-category representational stability show stronger relationship with higher-level visual/semantic features, suggesting that neural replicability is derived from perceived and higher-level visual information. Within category representational stability may thus originate from long-range cross talk between category-specific regions (and in this case strongly within ventral and lateral temporal cortex) over more abstract, rather than veridical/lower-level, visual (sensorial) representations, and perhaps in the service of object-centered representations.

Representations of common event structure in medial temporal lobe and frontoparietal cortex support efficient inference

Prior work has shown that the brain represents memories within a cognitive map that supports inference about connections between individual related events. Real-world adaptive behavior is also supported by recognizing common structure among numerous distinct contexts; for example, based on prior experience with restaurants, when visiting a new restaurant one can expect to first get a table, then order, eat, and finally pay the bill. We used a neurocomputational approach to examine how the brain extracts and uses abstract representations of common structure to support novel decisions. Participants learned image pairs (AB, BC) drawn from distinct triads (ABC) that shared the same internal structure and were then tested on their ability to infer indirect (AC) associations. We found that hippocampal and frontoparietal regions formed abstract representations that coded cross-triad relationships with a common geometric structure. Critically, such common representational geometries were formed despite the lack of explicit reinforcement to do so. Furthermore, we found that representations in parahippocampal cortex are hierarchical, reflecting both cross-triad relationships and distinctions between triads. We propose that representations with common geometric structure provide a vector space that codes inferred item relationships with a direction vector that is consistent across triads, thus supporting faster inference. Using computational modeling of response time data, we found evidence for dissociable vector-based retrieval and pattern-completion processes that contribute to successful inference. Moreover, we found evidence that these processes are mediated by distinct regions, with pattern completion supported by hippocampus and vector-based retrieval supported by parahippocampal cortex and lateral parietal cortex.

Color Space Geometry Uncovered with Magnetoencephalography

The geometry that describes the relationship among colors, and the neural mechanisms that support color vision, are unsettled. Here, we use multivariate analyses of measurements of brain activity obtained with magnetoencephalography to reverse-engineer a geometry of the neural representation of color space. The analyses depend upon determining similarity relationships among the spatial patterns of neural responses to different colors and assessing how these relationships change in time. We evaluate the approach by relating the results to universal patterns in color naming. Two prominent patterns of color naming could be accounted for by the decoding results: the greater precision in naming warm colors compared to cool colors evident by an interaction of hue and lightness, and the preeminence among colors of reddish hues. Additional experiments showed that classifiers trained on responses to color words could decode color from data obtained using colored stimuli, but only at relatively long delays after stimulus onset. These results provide evidence that perceptual representations can give rise to semantic representations, but not the reverse. Taken together, the results uncover a dynamic geometry that provides neural correlates for color appearance and generates new hypotheses about the structure of color space.

A unifying framework for understanding neural tuning and representational geometry

A unifying framework for understanding neural tuning and representational geometry