Representational Similarity Analysis Research Articles

Multidimensional neural representations of social features during movie viewing.

The social world is dynamic and contextually embedded. Yet, most studies utilize simple stimuli that do not capture the complexity of everyday social episodes. To address this, we implemented a movie viewing paradigm and investigated how the everyday social episodes are processed in the brain. Participants watched one of two movies during an MRI scan. Neural patterns from brain regions involved in social perception, mentalization, action observation, and sensory processing were extracted. Representational similarity analysis results revealed that several labeled social features (including social interaction, mentalization, the actions of others, characters talking about themselves, talking about others, and talking about objects) were represented in superior temporal gyrus (STG) and middle temporal gyrus (MTG). The mentalization feature was also represented throughout the theory of mind network, and characters talking about others engaged the temporoparietal junction (TPJ), suggesting that listeners may spontaneously infer the mental state of those being talked about. In contrast, we did not observe the action representations in frontoparietal regions of the action observation network. The current findings indicate that STG and MTG serve as key regions for social processing, and that listening to characters talk about others elicits spontaneous mental state inference in TPJ during natural movie viewing.

An inclusive multivariate approach to neural localization of language components.

To determine how language is implemented in the brain, it is important to know which brain areas are primarily engaged in language processing and which are not. Existing protocols for localizing language are typically univariate, treating each small unit of brain volume as independent. One prominent example that focuses on the overall language network in functional magnetic resonance imaging (fMRI) uses a contrast between neural responses to sentences and sets of pseudowords (pronounceable nonwords). This contrast reliably activates peri-sylvian language areas but is less sensitive to extra-sylvian areas that are also known to support aspects of language such as word meanings (semantics). In this study, we assess areas where a multivariate, pattern-based approach shows high reproducibility across multiple measurements and participants, identifying these areas as multivariate regions of interest (mROI). We then perform a representational similarity analysis (RSA) of an fMRI dataset where participants made familiarity judgments on written words. We also compare those results to univariate regions of interest (uROI) taken from previous sentences > pseudowords contrasts. RSA with word stimuli defined in terms of their semantic distance showed greater correspondence with neural patterns in mROI than uROI. This was confirmed in two independent datasets, one involving single-word recognition, and the other focused on the meaning of noun-noun phrases by contrasting meaningful phrases > pseudowords. In all cases, areas of spatial overlap between mROI and uROI showed the greatest neural association. This suggests that ROIs defined in terms of multivariate reproducibility can help localize components of language such as semantics. The multivariate approach can also be extended to focus on other aspects of language such as phonology, and can be used along with the univariate approach for inclusively mapping language cortex.

Comparing expert systems and their explainability through similarity

In our work, we propose the use of Representational Similarity Analysis (RSA) for explainable AI (XAI) approaches to enhance the reliability of XAI-based decision support systems. To demonstrate how similarity analysis of explanations can assess the output stability of post-hoc explainers, we conducted a computational evaluative study. This study addresses how our approach can be leveraged to analyze the stability of explanations amidst various changes in the ML pipeline. Our results show that modifications such as altered preprocessing or different ML models lead to changes in the explanations and illustrate the extent to which stability can suffer. Explanation similarity analysis enables practitioners to compare different explanation outcomes, thus monitoring stability in explanations. Alongside discussing the results and practical applications in operationalized ML, including both benefits and limitations, we also delve into insights from computational neuroscience and neural information processing.

Forming cognitive maps for abstract spaces: the roles of the human hippocampus and orbitofrontal cortex.

How does the human brain construct cognitive maps for decision-making and inference? Here, we conduct an fMRI study on a navigation task in multidimensional abstract spaces. Using a deep neural network model, we assess learning levels and categorized paths into exploration and exploitation stages. Univariate analyses show higher activation in the bilateral hippocampus and lateral prefrontal cortex during exploration, positively associated with learning level and response accuracy. Conversely, the bilateral orbitofrontal cortex (OFC) and retrosplenial cortex show higher activation during exploitation, negatively associated with learning level and response accuracy. Representational similarity analysis show that the hippocampus, entorhinal cortex, and OFC more accurately represent destinations in exploitation than exploration stages. These findings highlight the collaboration between the medial temporal lobe and prefrontal cortex in learning abstract space structures. The hippocampus may be involved in spatial memory formation and representation, while the OFC integrates sensory information for decision-making in multidimensional abstract spaces.

Neural substrates of affective temperaments: An intersubject representational similarity analysis to resting-state functional magnetic resonance imaging in nonclinical subjects.

Previous research has suggested that certain types of the affective temperament, including depressive, cyclothymic, hyperthymic, irritable, and anxious, are subclinical manifestations and precursors of mental disorders. However, the neural mechanisms that underlie these temperaments are not fully understood. The aim of this study was to identify the brain regions associated with different affective temperaments. We collected the resting-state functional magnetic resonance imaging (fMRI) data from 211 healthy adults and evaluated their affective temperaments using the Temperament Evaluation of Memphis, Pisa, Paris and San Diego Autoquestionnaire. We used intersubject representational similarity analysis to identify brain regions associated with each affective temperament. Brain regions associated with each affective temperament were detected. These regions included the prefrontal cortex, anterior cingulate cortex (ACC), precuneus, amygdala, thalami, hippocampus, and visual areas. The ACC, lingual gyri, and precuneus showed similar activity across several affective temperaments. The similarity in related brain regions was high among the cyclothymic, irritable, and anxious temperaments, and low between hyperthymic and the other affective temperaments. These findings may advance our understanding of the neural mechanisms underlying affective temperaments and their potential relationship to mental disorders and may have potential implications for personalized treatment strategies for mood disorders.

Hearing what is being said: the distributed neural substrate for early speech interpretation

ABSTRACT Speech comprehension is remarkable for the immediacy with which the listener hears what is being said. Here, we focus on the neural underpinnings of this process in isolated spoken words. We analysed source-localised MEG data for nouns using Representational Similarity Analysis to probe the spatiotemporal coordinates of phonology, lexical form, and the semantics of emerging word candidates. Phonological model fit was detectable within 40–50 ms, engaging a bilateral network including superior and middle temporal cortex and extending into anterior temporal and inferior parietal regions. Lexical form emerged within 60–70 ms, and model fit to semantics from 100-110 ms. Strikingly, the majority of vertices in a central core showed model fit to all three dimensions, consistent with a distributed neural substrate for early speech analysis. The early interpretation of speech seems to be conducted in a unified integrative representational space, in conflict with conventional views of a linguistically stratified representational hierarchy.

Prediction error processing and sharpening of expected information across the face-processing hierarchy

The perception and neural processing of sensory information are strongly influenced by prior expectations. The integration of prior and sensory information can manifest through distinct underlying mechanisms: focusing on unexpected input, denoted as prediction error (PE) processing, or amplifying anticipated information via sharpened representation. In this study, we employed computational modeling using deep neural networks combined with representational similarity analyses of fMRI data to investigate these two processes during face perception. Participants were cued to see face images, some generated by morphing two faces, leading to ambiguity in face identity. We show that expected faces were identified faster and perception of ambiguous faces was shifted towards priors. Multivariate analyses uncovered evidence for PE processing across and beyond the face-processing hierarchy from the occipital face area (OFA), via the fusiform face area, to the anterior temporal lobe, and suggest sharpened representations in the OFA. Our findings support the proposition that the brain represents faces grounded in prior expectations.

Use of functional magnetic resonance imaging to identify cortical loci for lower limb movements and their efficacy for individuals after stroke.

Identification of cortical loci for lower limb movements for stroke rehabilitation is crucial for better rehabilitation outcomes via noninvasive brain stimulation by targeting the fine-grained cortical loci of the movements. However, identification of the cortical loci for lower limb movements using functional MRI (fMRI) is challenging due to head motion and difficulty in isolating different types of movement. Therefore, we developed a custom-made MR-compatible footplate and leg cushion to identify the cortical loci for lower limb movements and conducted multivariate analysis on the fMRI data. We evaluated the validity of the identified loci using both fMRI and behavioral data, obtained from healthy participants as well as individuals after stroke. We recruited 33 healthy participants who performed four different lower limb movements (ankle dorsiflexion, ankle rotation, knee extension, and toe flexion) using our custom-built equipment while fMRI data were acquired. A subgroup of these participants (Dataset 1; n = 21) was used to identify the cortical loci associated with each lower limb movement in the paracentral lobule (PCL) using multivoxel pattern analysis and representational similarity analysis. The identified cortical loci were then evaluated using the remaining healthy participants (Dataset 2; n = 11), for whom the laterality index (LI) was calculated for each lower limb movement using the cortical loci identified for the left and right lower limbs. In addition, we acquired a dataset from 15 individuals with chronic stroke for regression analysis using the LI and the Fugl-Meyer Assessment (FMA) scale. The cortical loci associated with the lower limb movements were hierarchically organized in the medial wall of the PCL following the cortical homunculus. The LI was clearer using the identified cortical loci than using the PCL. The healthy participants (mean ± standard deviation: 0.12 ± 0.30; range: -0.63 to 0.91) exhibited a higher contralateral LI than the individuals after stroke (0.07 ± 0.47; -0.83 to 0.97). The corresponding LI scores for individuals after stroke showed a significant positive correlation with the FMA scale for paretic side movement in ankle dorsiflexion (R2 = 0.33, p = 0.025) and toe flexion (R2 = 0.37, p = 0.016). The cortical loci associated with lower limb movements in the PCL identified in healthy participants were validated using independent groups of healthy participants and individuals after stroke. Our findings suggest that these cortical loci may be beneficial for the neurorehabilitation of lower limb movement in individuals after stroke, such as in developing effective rehabilitation interventions guided by the LI scores obtained for neuronal activations calculated from the identified cortical loci across the paretic and non-paretic sides of the brain.

Early life adversity is associated with greater similarity in neural representations of ambiguous and threatening stimuli.

Exposure to early life adversity (ELA) is hypothesized to sensitize threat-responsive neural circuitry. This may lead individuals to overestimate threat in the face of ambiguity, a cognitive-behavioral phenotype linked to poor mental health. The tendency to process ambiguity as threatening may stem from difficulty distinguishing between ambiguous and threatening stimuli. However, it is unknown how exposure to ELA relates to neural representations of ambiguous and threatening stimuli, or how processing of ambiguity following ELA relates to psychosocial functioning. The current fMRI study examined multivariate representations of threatening and ambiguous social cues in 41 emerging adults (aged 18 to 19 years). Using representational similarity analysis, we assessed neural representations of ambiguous and threatening images within affective neural circuitry and tested whether similarity in these representations varied by ELA exposure. Greater exposure to ELA was associated with greater similarity in neural representations of ambiguous and threatening images. Moreover, individual differences in processing ambiguity related to global functioning, an association that varied as a function of ELA. By evidencing reduced neural differentiation between ambiguous and threatening cues in ELA-exposed emerging adults and linking behavioral responses to ambiguity to psychosocial wellbeing, these findings have important implications for future intervention work in at-risk, ELA-exposed populations.

Decoding visual fatigue in a visual search task selectively manipulated via myopia-correcting lenses.

Visual fatigue resulting from sustained, high-workload visual activities can significantly impact task performance and general wellbeing. So far, however, little is known about the underlying brain networks of visual fatigue. This study aimed to identify such potential networks using a unique paradigm involving myopia-correcting lenses known to directly modulate subjectively-perceived fatigue levels. A sample of N = 31 myopia participants [right eye-SE: -3.77D (SD: 2.46); left eye-SE: -3.75D (SD: 2.45)] performed a demanding visual search task with varying difficulty levels, both with and without the lenses, while undergoing fMRI scanning. There were a total of 20 trials, after each of which participants rated the perceived difficulty and their subjective visual fatigue level. We used representational similarity analysis to decode brain regions associated with fatigue and difficulty, analyzing their individual and joint decoding pattern. Behavioral results showed correlations between fatigue and difficulty ratings and above all a significant reduction in fatigue levels when wearing the lenses. Imaging results implicated the cuneus, lingual gyrus, middle occipital gyrus (MOG), and declive for joint fatigue and difficulty decoding. Parts of the lingual gyrus were able to selectively decode perceived difficulty. Importantly, a broader network of visual and higher-level association areas showed exclusive decodability of fatigue (culmen, middle temporal gyrus (MTG), parahippocampal gyrus, precentral gyrus, and precuneus). Our findings enhance our understanding of processing within the context of visual search, attention, and mental workload and for the first time demonstrate that it is possible to decode subjectively-perceived visual fatigue during a challenging task from imaging data. Furthermore, the study underscores the potential of myopia-correcting lenses in investigating and modulating fatigue.

Finding structure during incremental speech comprehension.

A core aspect of human speech comprehension is the ability to incrementally integrate consecutive words into a structured and coherent interpretation, aligning with the speaker's intended meaning. This rapid process is subject to multidimensional probabilistic constraints, including both linguistic knowledge and non-linguistic information within specific contexts, and it is their interpretative coherence that drives successful comprehension. To study the neural substrates of this process, we extract word-by-word measures of sentential structure from BERT, a deep language model, which effectively approximates the coherent outcomes of the dynamic interplay among various types of constraints. Using representational similarity analysis, we tested BERT parse depths and relevant corpus-based measures against the spatiotemporally resolved brain activity recorded by electro-/magnetoencephalography when participants were listening to the same sentences. Our results provide a detailed picture of the neurobiological processes involved in the incremental construction of structured interpretations. These findings show when and where coherent interpretations emerge through the evaluation and integration of multifaceted constraints in the brain, which engages bilateral brain regions extending beyond the classical fronto-temporal language system. Furthermore, this study provides empirical evidence supporting the use of artificial neural networks as computational models for revealing the neural dynamics underpinning complex cognitive processes in the brain.

Finding structure during incremental speech comprehension

A core aspect of human speech comprehension is the ability to incrementally integrate consecutive words into a structured and coherent interpretation, aligning with the speaker’s intended meaning. This rapid process is subject to multidimensional probabilistic constraints, including both linguistic knowledge and non-linguistic information within specific contexts, and it is their interpretative coherence that drives successful comprehension. To study the neural substrates of this process, we extract word-by-word measures of sentential structure from BERT, a deep language model, which effectively approximates the coherent outcomes of the dynamic interplay among various types of constraints. Using representational similarity analysis, we tested BERT parse depths and relevant corpus-based measures against the spatiotemporally resolved brain activity recorded by electro-/magnetoencephalography when participants were listening to the same sentences. Our results provide a detailed picture of the neurobiological processes involved in the incremental construction of structured interpretations. These findings show when and where coherent interpretations emerge through the evaluation and integration of multifaceted constraints in the brain, which engages bilateral brain regions extending beyond the classical fronto-temporal language system. Furthermore, this study provides empirical evidence supporting the use of artificial neural networks as computational models for revealing the neural dynamics underpinning complex cognitive processes in the brain.

Visual recognition memory of scenes is driven by categorical, not sensory, visual representations.

When we perceive a scene, our brain processes various types of visual information simultaneously, ranging from sensory features, such as line orientations and colors, to categorical features, such as objects and their arrangements. Whereas the role of sensory and categorical visual representations in predicting subsequent memory has been studied using isolated objects, their impact on memory for complex scenes remains largely unknown. To address this gap, we conducted an fMRI study in which female and male participants encoded pictures of familiar scenes (e.g., an airport picture) and later recalled them, while rating the vividness of their visual recall. Outside the scanner, participants had to distinguish each seen scene (e.g., an airport picture) from three similar lures (e.g., 3 airport pictures). We modeled the sensory and categorical visual features of multiple scenes using both early and late layers of a deep convolutional neural network. Then, we applied representational similarity analysis to determine which brain regions represented stimuli in accordance with the sensory and categorical models. We found that categorical, but not sensory, representations predicted subsequent memory. In line with the previous result, only for the categorical model, the average recognition performance of each scene exhibited a positive correlation with the average visual dissimilarity between the item in question and its respective lures. These results strongly suggest that even in memory tests that ostensibly rely solely on visual cues (such as force-choice visual recognition with similar distractors), memory decisions for scenes may be primarily influenced by categorical rather than sensory representations.Significance Statement Our memory for real-world scenes often comprises a tableau of complex visual features, but recent findings challenge the view that our memories of such stimuli rely on purely visual information. Instead, it appears that our memory for scenes is heavily influenced by higher-level categorical information. Analyzing cortical representations in regions responsive to both categorical and sensory features, we discovered that only the former can reliably predict memory outcomes. Moreover, the distinctiveness of scenes in terms of their categoric features among similar examples is positively associated with our ability to accurately recognize previously encountered scenes. In essence, this study sheds light on how our brains rely on categorical information to recognize natural scenes.

Vicarious punishment of moral violations in naturalistic drama narratives predicts cortical synchronization

Punishment of moral norm violators is instrumental for human cooperation. Yet, social and affective neuroscience research has primarily focused on second- and third-party norm enforcement, neglecting the neural architecture underlying observed (vicarious) punishment of moral wrongdoers. We used naturalistic television drama as a sampling space for observing outcomes of morally-relevant behaviors to assess how individuals cognitively process dynamically evolving moral actions and their consequences. Drawing on Affective Disposition Theory, we derived hypotheses linking character morality with viewers’ neural processing of characters’ rewards and punishments. We used functional magnetic resonance imaging (fMRI) to examine neural responses of 28 female participants while free-viewing 15 short story summary video clips of episodes from a popular US television soap opera. Each summary included a complete narrative structure, fully crossing main character behaviors (moral/immoral) and the consequences (reward/punishment) characters faced for their actions. Narrative engagement was examined via intersubject correlation and representational similarity analysis. Highest cortical synchronization in 9 specifically selected regions previously implicated in processing moral information was observed when characters who act immorally are punished for their actions with participants’ empathy as an important moderator. The results advance our understanding of the moral brain and the role of normative considerations and character outcomes in viewers’ engagement with popular narratives.

Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response.

Behavioral and neuroscience studies in humans and primates have shown that memorability is an intrinsic property of an image that predicts its strength of encoding into and retrieval from memory. While previous work has independently probed when or where this memorability effect may occur in the human brain, a description of its spatiotemporal dynamics is missing. Here, we used representational similarity analysis (RSA) to combine functional magnetic resonance imaging (fMRI) with source-estimated magnetoencephalography (MEG) to simultaneously measure when and where the human cortex is sensitive to differences in image memorability. Results reveal that visual perception of High Memorable images, compared to Low Memorable images, recruits a set of regions of interest (ROIs) distributed throughout the ventral visual cortex: a late memorability response (from around 300 ms) in early visual cortex (EVC), inferior temporal cortex, lateral occipital cortex, fusiform gyrus, and banks of the superior temporal sulcus. Image memorability magnitude results are represented after high-level feature processing in visual regions and reflected in classical memory regions in the medial temporal lobe (MTL). Our results present, to our knowledge, the first unified spatiotemporal account of visual memorability effect across the human cortex, further supporting the levels-of-processing theory of perception and memory.

Intersubject representational similarity analysis uncovers the impact of state anxiety on brain activation patterns in the human extrastriate cortex.

The current study used functional magnetic resonance imaging (fMRI) and showed that state anxiety modulated extrastriate cortex activity in response to emotionally-charged visual images. State anxiety and neuroimaging data from 53 individuals were subjected to an intersubject representational similarity analysis (ISRSA), wherein the geometries between neural and behavioral data were compared. This analysis identified the extrastriate cortex (fusiform gyrus and area MT) to be the sole regions whose activity patterns covaried with state anxiety. Importantly, we show that this brain-behavior association is revealed when treating state anxiety data as a multidimensional response pattern, rather than a single composite score. This suggests that ISRSA using multivariate distances may be more sensitive in identifying the shared geometries between self-report questionnaires and brain imaging data. Overall, our findings demonstrate that a transient state of anxiety may influence how visual information - especially those relevant to the valence dimension - is processed in the extrastriate cortex.

Adaptive-to-maladaptive gradient of emotion regulation tendencies are embedded in the functional-structural hybrid connectome.

Emotion regulation tendencies are well-known transdiagnostic markers of psychopathology, but their neurobiological foundations have mostly been examined within the theoretical framework of cortical-subcortical interactions. We explored the connectome-wide neural correlates of emotion regulation tendencies using functional and diffusion magnetic resonance images of healthy young adults (N = 99; age 20-30; 28 females). We first tested the importance of considering both the functional and structural connectome through intersubject representational similarity analyses. Then, we employed a canonical correlation analysis between the functional-structural hybrid connectome and 23 emotion regulation strategies. Lastly, we sought to externally validate the results on a transdiagnostic adolescent sample (N = 93; age 11-19; 34 females). First, interindividual similarity of emotion regulation profiles was significantly correlated with interindividual similarity of the functional-structural hybrid connectome, more so than either the functional or structural connectome. Canonical correlation analysis revealed that an adaptive-to-maladaptive gradient of emotion regulation tendencies mapped onto a specific configuration of covariance within the functional-structural hybrid connectome, which primarily involved functional connections in the motor network and the visual networks as well as structural connections in the default mode network and the subcortical-cerebellar network. In the transdiagnostic adolescent dataset, stronger functional signatures of the found network were associated with higher general positive affect through more frequent use of adaptive coping strategies. Taken together, our study illustrates a gradient of emotion regulation tendencies that is best captured when simultaneously considering the functional and structural connections across the whole brain.

High performers demonstrate greater neural synchrony than low performers across behavioral domains

Abstract Heterogeneity in brain activity can give rise to heterogeneity in behavior, which in turn comprises our distinctive characteristics as individuals. Studying the path from brain to behavior, however, often requires making assumptions about how similarity in behavior scales with similarity in brain activity. Here, we expand upon recent work (Finn et al. 2020) which proposes a theoretical framework for testing the validity of such assumptions. Using intersubject representational similarity analysis in two independent movie-watching fMRI datasets, we probe how brain-behavior relationships vary as a function of behavioral domain and participant sample. We find evidence that, in some cases, the neural similarity of two individuals is not correlated with behavioral similarity. Rather, individuals with higher behavioral scores are more similar to other high scorers whereas individuals with lower behavioral scores are dissimilar from everyone else. Ultimately, our findings motivate a more extensive investigation of both the structure of brain-behavior relationships and the tacit assumption that people who behave similarly will demonstrate shared patterns of brain activity.

Spatial representation of multidimensional information in emotional faces revealed by fMRI

Face perception is a complex process that involves highly specialized procedures and mechanisms. Investigating into face perception can help us better understand how the brain processes fine-grained, multidimensional information. This research aimed to delve deeply into how different dimensions of facial information are represented in specific brain regions or through inter-regional connections via an implicit face recognition task. To capture the representation of various facial information in the brain, we employed support vector machine decoding, functional connectivity, and model-based representational similarity analysis on fMRI data, resulting in the identification of three crucial findings. Firstly, despite the implicit nature of the task, emotions were still represented in the brain, contrasting with all other facial information. Secondly, the connection between the medial amygdala and the parahippocampal gyrus was found to be essential for the representation of facial emotion in implicit tasks. Thirdly, in implicit tasks, arousal representation occurred in the parahippocampal gyrus, while valence depended on the connection between the primary visual cortex and the parahippocampal gyrus. In conclusion, these findings dissociate the neural mechanisms of emotional valence and arousal, revealing the precise spatial patterns of multidimensional information processing in faces.

Altered neural representation of olfactory food reward in the nucleus accumbens after acute stress

Acute stress impairs reward processing. The nucleus accumbens (NAcc) plays an important role in the processing of primary rewards such as food. The present study investigates how acute stress affects the olfactory food reward processing in the NAcc using the representational similarity analysis. Forty-eight participants underwent an olfactory fMRI session following either an acute psychosocial stress (N = 24; stress group) or a control (N = 24; control group). Brain activation was recorded during the anticipatory and the perceptual phases of high-calorie food, low-calorie food, and non-food odor stimuli. Compared to the control group, the stress group rated the high-calorie food odor as significantly more pleasant (p = 0.005). In the NAcc, acute stress significantly reduced the dissimilarity of food and non-food odors in the perceptual phase (p = 0.027) and marginally reduced the dissimilarity of high- and low-calorie foods in the anticipatory phase (p = 0.095). Significant negative correlations were observed between the level of NAcc representational differentiation for high- and low-calorie food odors during perception and the difference in pleasantness ratings between high- and low-calorie food odors (r = -0.40, p = 0.005). These findings suggest that acute stress may impair participants’ ability to discriminate between olfactory food rewards, leading individuals to seek out more palatable foods in stressful situations in order to maintain positive emotions.