Inter-subject Correlation Research Articles

Decoding Cortical Chronotopy - Comparing the Influence of Different Cortical Organizational Schemes

The brain's diverse intrinsic timescales enable us to perceive stimuli with varying temporal persistency. This study aimed to uncover the cortical organizational schemes underlying these variations, revealing the neural architecture for processing a wide range of sensory experiences. We collected resting-state fMRI, task-fMRI, and diffusion-weighted imaging data from 47 individuals. Based on this data, we extracted six organizational schemes: (1) the structural Rich Club (RC) architecture, shown to synchronize the connectome; (2) the structural Diverse Club architecture, as an alternative to the RC based on the network's module structure; (3) the functional uni-to-multimodal gradient, reflected in a wide range of structural and functional features; and (4) the spatial posterior/lateral-to-anterior/medial gradient, established for hierarchical levels of cognitive control. Also, we explored the effects of (5) structural graph theoretical measures of centrality and (6) cytoarchitectural differences. Using Bayesian model comparison, we contrasted the impact of these organizational schemes on (1) intrinsic resting-state timescales and (2) inter-subject correlation (ISC) from a task involving hierarchically nested digit sequences. As expected, resting-state timescales were slower in structural network hubs, hierarchically higher areas defined by the functional and spatial gradients, and thicker cortical regions. ISC analysis demonstrated hints for the engagement of higher cortical areas with more temporally persistent stimuli. Finally, the model comparison identified the uni-to-multimodal gradient as the best organizational scheme for explaining the chronotopy in both task and rest. Future research should explore the microarchitectural features that shape this gradient, elucidating how our brain adapts and evolves across different modes of processing.

Contrastive learning of shared spatiotemporal EEG representations across individuals for naturalistic neuroscience

Neural representations induced by naturalistic stimuli offer insights into how humans respond to stimuli in daily life. Understanding neural mechanisms underlying naturalistic stimuli processing hinges on the precise identification and extraction of the shared neural patterns that are consistently present across individuals. Targeting the Electroencephalogram (EEG) technique, known for its rich spatial and temporal information, this study presents a framework for Contrastive Learning of Shared SpatioTemporal EEG Representations across individuals (CL-SSTER). CL-SSTER utilizes contrastive learning to maximize the similarity of EEG representations across individuals for identical stimuli, contrasting with those for varied stimuli. The network employs spatial and temporal convolutions to simultaneously learn the spatial and temporal patterns inherent in EEG. The versatility of CL-SSTER was demonstrated on three EEG datasets, including a synthetic dataset, a natural speech comprehension EEG dataset, and an emotional video watching EEG dataset. CL-SSTER attained the highest inter-subject correlation (ISC) values compared to the state-of-the-art ISC methods. The latent representations generated by CL-SSTER exhibited reliable spatiotemporal EEG patterns, which can be explained by properties of the naturalistic stimuli. CL-SSTER serves as an interpretable and scalable framework for the identification of inter-subject shared neural representations in naturalistic neuroscience.

Idiosyncratic pupil regulation in autistic children.

Recent neuroimaging and eye-tracking studies have suggested that children with autism exhibit more variable and idiosyncratic brain responses and eye movements than typically developing (TD) children. Here, we extended this research to pupillometry recordings. We successfully acquired pupillometry recordings from 111 children (74 with autism), 4.5-years-old on average, who viewed three 90 s movies, twice. We extracted their pupillary time-course for each movie, capturing their stimulus evoked pupillary responses. We then computed the correlation between the time-course of each child and those of all others in their group as well as between each autistic child and all children in the TD group. This yielded an average inter-subject correlation value per child, representing how similar their pupillary responses were to all others in their group or the comparison group. Children with autism exhibited significantly weaker inter-subject correlations than TD children in all comparisons. These differences were independent of previously reported differences in gaze inter-subject correlations and were largest in responses to a naturalistic movie containing footage of a social interaction between two TD children. The results demonstrate the utility of measuring the idiosyncrasy of pupil regulation, which can be performed with passive viewing of movies even by young children with co-occurring intellectual disability. These findings reveal that a considerable number of children with autism have significantly less stable, idiosyncratic pupil regulation than TD children, indicative of more variable, weakly regulated, underlying neural activity.

Assessing the consistency and sensitivity of the neural correlates of narrative stimuli using functional near-infrared spectroscopy

Abstract Investigating how the brain responds to rich and complex narratives, such as engaging movies, has helped researchers study higher-order cognition in 'real-world' scenarios. These neural correlates are particularly useful in populations where behavioural evidence of cognition alone is inadequate, such as children and certain patient populations. While this research has been primarily conducted in fMRI and EEG, whether functional near-infrared spectroscopy (fNIRS) can reliably detect these neural correlates at an individual level, which is required for effective use in these populations, has yet to be established. This study replicated widespread inter-subject correlations (ISCs) in the frontal, parietal and temporal cortices in fNIRS in healthy participants when they watched part of the TV episode Bang! You're Dead and listened to an audio clip from the movie Taken. Conversely, these ISCs were primarily restricted to temporal cortices when participants viewed scrambled versions of those clips. To assess whether these results were reliable at the single-participant level, two follow-up analyses were conducted. First, the consistency analysis compared each participant's ISCs against group results that excluded that individual. This approach found that 24/26 participants in Bang! You're Dead and 20/26 participants in Taken were statistically similar to the group. Second, the sensitivity analysis measured whether machine learning algorithms could decode between intact conditions and their scrambled counterparts. This approach yielded balanced accuracy scores of 81% in Bang! You're Dead and 79% in Taken. Overall, the neural correlates of narrative stimuli, as assessed by fNIRS, are reproducible across participants, supporting its broad application to clinical and developmental populations.

The magic, memory, and curiosity fMRI dataset of people viewing magic tricks

Videos of magic tricks offer lots of opportunities to study the human mind. They violate the expectations of the viewer, causing prediction errors, misdirect attention, and elicit epistemic emotions. Herein we describe and share the Magic, Memory, and Curiosity (MMC) Dataset where 50 participants watched 36 magic tricks filmed and edited specifically for functional magnetic imaging (fMRI) experiments. The MMC Dataset includes a contextual incentive manipulation, curiosity ratings for the magic tricks, and incidental memory performance tested a week later. We additionally measured individual differences in working memory and constructs relevant to motivated learning. fMRI data were acquired before, during, and after learning. We show that both behavioural and fMRI data are of high quality, as indicated by basic validation analysis, i.e., variance decomposition as well as intersubject correlation and seed-based functional connectivity, respectively. The richness and complexity of the MMC Dataset will allow researchers to explore dynamic cognitive and motivational processes from various angles during task and rest.

Preserved Spontaneous Mentalizing amid Reduced Intersubject Variability in Autism during a Movie Narrative

While individuals with autism often face challenges in everyday social interactions, they may demonstrate proficiency in structured Theory of Mind (ToM) tasks that assess their ability to infer others' mental states. Using functional MRI and pupillometry, we investigated whether these discrepancies stem from diminished spontaneous mentalizing or broader difficulties in unstructured contexts. Fifty-two adults diagnosed with autism and 52 neurotypical controls viewed 'Partly Cloudy', a nonverbal animated film with a dynamic social narrative known to engage the ToM brain network during specific scenes. Analysis focused on comparing brain and pupil responses to these ToM events. Additionally, dynamic intersubject correlations explored the variability of these responses throughout the film. Both groups showed similar brain and pupil responses to ToM events and provided comparable descriptions of the characters' mental states. However, participants with autism exhibited significantly stronger correlations in their responses across the film's social narrative, indicating reduced inter-individual variability. This distinct pattern emerged well before any ToM events and involved brain regions beyond the ToM network. Our findings provide functional evidence of spontaneous mentalizing in autism, demonstrating this capacity in a context affording but not requiring mentalizing. Rather than responses to ToM events, a novel neurocognitive signature - inter-individual variability in brain and pupil responses to evolving social narratives - differentiated neurotypical individuals from those with autism. These results suggest that idiosyncratic narrative processing in unstructured settings, a common element of everyday social interactions, may offer a more sensitive scenario for understanding the autistic mind.

Inter-subject gamma oscillation synchronization as a biomarker of abnormal processing of social interaction in ADHD

Children with attention-deficit hyperactivity disorder (ADHD) have difficulties in social interactions. Studying brain activity during social interactions is difficult with conventional artificial stimuli. This pioneering study examined the neural correlates of social perception in children with ADHD and matched controls using naturalistic stimuli. We presented 20 children with ADHD and 20 age-and-sex-matched controls with tailored movies featuring high- or low-level social interactions while recording electroencephalographic signals. Both groups exhibited synchronized gamma-band oscillations, but controls demonstrated greater inter-subject correlations. Additionally, the difference in inter-subject correlations between high- and low-interaction movies was significantly larger in controls compared to ADHD patients. Between 55 and 75 Hz comparing viewing high interaction movies with low interaction moves, controls had a significantly larger weighting in the right parietal lobe, while ADHD patients had a significantly smaller weighting in the left occipital lobe. These findings reveal distinct spatiotemporal neural signatures in social interaction processing among children with ADHD and controls using naturalistic stimuli. These neural markers offer potential for group differentiation and assessing intervention efficacy, advancing our understanding ADHD-related social interaction mechanisms.

Heroes and villains: opposing narrative roles engage neural synchronization in the inferior frontal gyrus.

Neuroscientific studies have highlighted the role of the default mode network (DMN) in processing narrative information. Here, we examined whether the neural synchronization of the DMN tracked the appearances of protagonists and antagonists when viewing highly engaging, socially rich audiovisual narratives. Using inter-subject correlation analysis on two independent, publicly available movie-watching fMRI datasets, we computed whole-brain neural synchronization during the appearance of the protagonists and antagonists. Results showed that the inferior frontal gyrus (IFG) had higher ISC values during the appearance of the protagonists than the antagonists. Importantly, these findings were generalized in both datasets. We discuss the results in the context of information integration and emotional empathy, which are relevant to functions of the IFG. Our study presents generalizable evidence that the IFG show distinctive synchronization patterns due to differences in narrative roles.

Multimodal prediction of the need of clozapine in treatment resistant schizophrenia; a pilot study in first-episode psychosis

As many as one third of the patients diagnosed with schizophrenia do not respond to first-line antipsychotic medication. This group may benefit from the atypical antipsychotic medication clozapine, but initiation of treatment is often delayed, which may worsen prognosis. Predicting which patients do not respond to traditional antipsychotic medication at the onset of symptoms would provide fast-tracked treatment for this group of patients. We collected data from patient records of 38 first-episode psychosis patients, of whom seven did not respond to traditional antipsychotic medications. We used clinical data including medical records, voxel-based morphometry MRI data and inter-subject correlation fMRI data, obtained during movie viewing, to predict future treatment resistance. Using a neural network model, we correctly predicted future treatment resistance in six of the seven treatment resistance patients and 25 of 31 patients who did not require clozapine treatment. Prediction improved significantly when using imaging data in tandem with clinical data. The accuracy of the neural network model was significantly higher than the accuracy of a support vector machine algorithm. These results support the notion that treatment resistant schizophrenia could represent a separate entity of psychotic disorders, characterized by morphological and functional changes in the brain which could represent biomarkers detectable at early onset of symptoms.

Familiarity enhances functional connectivity between visual and nonvisual regions of the brain during natural viewing.

We explored the neural correlates of familiarity with people and places using a naturalistic viewing paradigm. Neural responses were measured using functional magnetic resonance imaging, while participants viewed a movie taken from Game of Thrones. We compared inter-subject correlations and functional connectivity in participants who were either familiar or unfamiliar with the TV series. Higher inter-subject correlations were found between familiar participants in regions, beyond the visual brain, that are typically associated with the processing of semantic, episodic, and affective information. However, familiarity also increased functional connectivity between face and scene regions in the visual brain and the nonvisual regions of the familiarity network. To determine whether these regions play an important role in face recognition, we measured responses in participants with developmental prosopagnosia (DP). Consistent with a deficit in face recognition, the effect of familiarity was significantly attenuated across the familiarity network in DP. The effect of familiarity on functional connectivity between face regions and the familiarity network was also attenuated in DP. These results show that the neural response to familiarity involves an extended network of brain regions and that functional connectivity between visual and nonvisual regions of the brain plays an important role in the recognition of people and places during natural viewing.

Sisterhood predicts similar neural processing of a film

Relationships between humans are essential for how we see the world. Using fMRI, we explored the neural basis of homophily, a sociological concept that describes the tendency to bond with similar others. Our comparison of brain activity between sisters, friends and acquaintances while they watched a movie, indicate that sisters’ brain activity is more similar than that of friends and friends’ activity is more similar than that of acquaintances. The increased similarity in brain activity measured as inter-subject correlation (ISC) was found both in higher-order brain areas including the default-mode network (DMN) and sensory areas. Increased ISC could not be explained by genetic relation between sisters neither by similarities in eye-movements, emotional experiences, and physiological activity. Our findings shed light on the neural basis of homophily by revealing that similarity in brain activity in the DMN and sensory areas is the stronger the closer is the relationship between the people.

Happy people are always similar: The evidence from brain morphological and functional inter-subject correlations

A fundamental question in the study of happiness is whether there is neural evidence to support a well-known hypothesis that happy people are always similar while unfortunate people have their own misfortunes. To investigate this, we employed several happiness-related questionnaires to identify potential components of happiness, and further investigated and confirmed their associations with personality, mood, aggressive behaviors, and amygdala reactivity to fearful faces within a substantial sample size of college students (n = 570). Additionally, we examined the functional and morphological similarities and differences among happy individuals using the inter-subject representational similarity analysis (IS-RSA). IS-RSA emphasizes the geometric properties in a high-dimensional space constructed by brain or behavioral patterns and focuses on individual subjects. Our behavioral findings unveiled two factors of happiness: individual and social, both of which mediated the effect of personality traits on individual aggression. Subsequently, mood mediated the impact of happiness on aggressive behaviors across two subgroup splits. Functional imaging data revealed that individuals with higher levels of happiness exhibited reduced amygdala reactivity to fearful faces, as evidenced by a conventional face-matching task (n = 104). Moreover, IS-RSA demonstrated that these participants manifested similar neural activation patterns when processing fearful faces within the visual pathway, but not within the emotional network (e.g., amygdala). Morphological observations (n = 425) indicated that individuals with similar high happiness levels exhibited comparable gray matter volume patterns within several networks, including the default mode network, fronto-parietal network, visual network, and attention network. Collectively, these findings offer early neural evidence supporting the proposition that happy individuals may share common neural characteristics.

Intersubject correlations in reward and mentalizing brain circuits separately predict persuasiveness of two types of ISIS video propaganda

The Islamist group ISIS has been particularly successful at recruiting Westerners as terrorists. A hypothesized explanation is their simultaneous use of two types of propaganda: Heroic narratives, emphasizing individual glory, alongside Social narratives, which emphasize oppression against Islamic communities. In the current study, functional MRI was used to measure brain responses to short ISIS propaganda videos distributed online. Participants were shown 4 Heroic and 4 Social videos categorized as such by another independent group of subjects. Persuasiveness was measured using post-scan predictions of recruitment effectiveness. Inter-subject correlation (ISC) was used to measure commonality of brain activity time courses across individuals. ISCs in ventral striatum predicted rated persuasiveness for Heroic videos, while ISCs in mentalizing and default networks, especially in dmPFC, predicted rated persuasiveness for Social videos. This work builds on past findings that engagement of the reward circuit and of mentalizing brain regions predicts preferences and persuasion. The observed dissociation as a function of stimulus type is novel, as is the finding that intersubject synchrony in ventral striatum predicts rated persuasiveness. These exploratory results identify possible neural mechanisms by which political extremists successfully recruit prospective members and specifically support the hypothesized distinction between Heroic and Social narratives for ISIS propaganda.

The Language Network Reliably "Tracks" Naturalistic Meaningful Nonverbal Stimuli.

The language network, comprised of brain regions in the left frontal and temporal cortex, responds robustly and reliably during language comprehension but shows little or no response during many nonlinguistic cognitive tasks (e.g., Fedorenko & Blank, 2020). However, one domain whose relationship with language remains debated is semantics-our conceptual knowledge of the world. Given that the language network responds strongly to meaningful linguistic stimuli, could some of this response be driven by the presence of rich conceptual representations encoded in linguistic inputs? In this study, we used a naturalistic cognition paradigm to test whether the cognitive and neural resources that are responsible for language processing are also recruited for processing semantically rich nonverbal stimuli. To do so, we measured BOLD responses to a set of ∼5-minute-long video and audio clips that consisted of meaningful event sequences but did not contain any linguistic content. We then used the intersubject correlation (ISC) approach (Hasson et al., 2004) to examine the extent to which the language network "tracks" these stimuli, that is, exhibits stimulus-related variation. Across all the regions of the language network, meaningful nonverbal stimuli elicited reliable ISCs. These ISCs were higher than the ISCs elicited by semantically impoverished nonverbal stimuli (e.g., a music clip), but substantially lower than the ISCs elicited by linguistic stimuli. Our results complement earlier findings from controlled experiments (e.g., Ivanova et al., 2021) in providing further evidence that the language network shows some sensitivity to semantic content in nonverbal stimuli.

Inter-subject correlation of electroencephalographic and behavioural responses reflects time-varying engagement with natural music.

Musical engagement can be conceptualized through various activities, modes of listening and listener states. Recent research has reported that a state of focused engagement can be indexed by the inter-subject correlation (ISC) of audience responses to a shared naturalistic stimulus. While statistically significant ISC has been reported during music listening, we lack insight into the temporal dynamics of engagement over the course of musical works-such as those composed in the Western classical style-which involve the formulation of expectations that are realized or derailed at subsequent points of arrival. Here, we use the ISC of electroencephalographic (EEG) and continuous behavioural (CB) responses to investigate the time-varying dynamics of engagement with functional tonal music. From a sample of adult musicians who listened to a complete cello concerto movement, we found that ISC varied throughout the excerpt for both measures. In particular, significant EEG ISC was observed during periods of musical tension that built to climactic highpoints, while significant CB ISC corresponded more to declarative entrances and points of arrival. Moreover, we found that a control stimulus retaining envelope characteristics of the intact music, but little other temporal structure, also elicited significantly correlated EEG and CB responses, though to lesser extents than the original version. In sum, these findings shed light on the temporal dynamics of engagement during music listening and clarify specific aspects of musical engagement that may be indexed by each measure.

Time-varying similarity of neural responses to musical tension is shaped by physical features and musical themes

The similarity of understanding is important for music experience and communication, but little is understood about the sources of this common knowledge. Although neural responses to the same piece of music are known to be similar across listeners, it remains unclear whether this neural response similarity is linked to musical understanding and the role of dynamic musical attributes in shaping it. Our study addresses this gap by investigating the relationship between neural response similarity, musical tension, and dynamic musical attributes. Using electroencephalography-based inter-subject correlation (EEG-ISC), we examined how the neural response similarity among listeners varies throughout the evaluation of musical tension in the first movement of Beethoven's Piano Sonata No. 8. Participants continuously rated the degree of alignment between musical events and their expectations, while neural activity was recorded using electroencephalography (EEG). The results showed that neural response similarity fluctuated in tandem with musical tension, with increased similarity observed during moments of heightened tension. This time-varying neural response similarity was influenced by two dynamic attributes contributing to musical tension: physical features and musical themes. Specifically, its fluctuation was driven by physical features, and the patterns of its variation were modulated by musical themes, with similar time-varying patterns observed across similar thematic materials. These findings offer valuable insight into the role of dynamic musical attributes in shaping neural response similarity, and reveal an important source and mechanism of shared musical understandings.

Reliability for music-induced heart rate synchronization

Common inputs synchronize various biological systems, including human physical and cognitive processes. This mechanism potentially explains collective human emotions in theater as unintentional behavioral synchronization. However, the inter-subject correlation of physiological signals among individuals is small. Based on findings on the common-input synchronization of nonlinear systems, we hypothesized that individual differences in perceptual and cognitive systems reduce the reliability of physiological responses to aesthetic stimuli and, thus, disturb synchronization. We tested this by comparing the inter- and intra-subject Pearson’s correlation coefficients and nonlinear phase synchronization, calculated using instantaneous heart rate data measured while appreciating music. The results demonstrated that inter-subject correlations were consistently lower than intra-subject correlations, regardless of participants’ music preferences and daily moods. Further, music-induced heart rate synchronization depends on the reliability of physiological responses to musical pieces rather than mood or motivation. This study lays the foundation for future empirical research on collective emotions in theater.

Multi-View and Multimodal Graph Convolutional Neural Network for Autism Spectrum Disorder Diagnosis

Autism Spectrum Disorder (ASD) presents significant diagnostic challenges due to its complex, heterogeneous nature. This study explores a novel approach to enhance the accuracy and reliability of ASD diagnosis by integrating resting-state functional magnetic resonance imaging with demographic data (age, gender, and IQ). This study is based on improving the spectral graph convolutional neural network (GCN). It introduces a multi-view attention fusion module to extract useful information from different views. The graph’s edges are informed by demographic data, wherein an edge-building network computes weights grounded in demographic information, thereby bolstering inter-subject correlation. To tackle the challenges of oversmoothing and neighborhood explosion inherent in deep GCNs, this study introduces DropEdge regularization and residual connections, thus augmenting feature diversity and model generalization. The proposed method is trained and evaluated on the ABIDE-I and ABIDE-II datasets. The experimental results underscore the potential of integrating multi-view and multimodal data to advance the diagnostic capabilities of GCNs for ASD.

Using Synchronized Eye Movements to Predict Attention in Online Video Learning

Concerns persist about attentional engagement in online learning. The inter-subject correlation of eye movements (ISC) has shown promise as an accessible and effective method for attention assessment in online learning. This study extends previous studies investigating ISC of eye movements in online learning by addressing two research questions. Firstly, can ISC predict students’ attentional states at a finer level beyond a simple dichotomy of attention states (e.g., attending and distracted states)? Secondly, whether learners’ learning styles affect ISC’s prediction rate of attention assessment in video learning? Previous studies have shown that learners of different learning styles have different eye movement patterns when viewing static materials. However, limited research has explored the impact of learning styles on viewing patterns in video learning. An eye tracking experiment with participants watching lecture videos demonstrated a connection between ISC and self-reported attention states at a finer level. We also demonstrated that learning styles did not significantly affect ISC’s prediction rate of attention assessment in video learning, suggesting that ISC of eye movements can be effectively used without considering learners’ learning styles. These findings contribute to the ongoing discourse on optimizing attention assessment in the evolving landscape of online education.

Consistent movement of viewers' facial keypoints while watching emotionally evocative videos.

Neuropsychological research aims to unravel how diverse individuals' brains exhibit similar functionality when exposed to the same stimuli. The evocation of consistent responses when different subjects watch the same emotionally evocative stimulus has been observed through modalities like fMRI, EEG, physiological signals and facial expressions. We refer to the quantification of these shared consistent signals across subjects at each time instant across the temporal dimension as Consistent Response Measurement (CRM). CRM is widely explored through fMRI, occasionally with EEG, physiological signals and facial expressions using metrics like Inter-Subject Correlation (ISC). However, fMRI tools are expensive and constrained, while EEG and physiological signals are prone to facial artifacts and environmental conditions (such as temperature, humidity, and health condition of subjects). In this research, facial expression videos are used as a cost-effective and flexible alternative for CRM, minimally affected by external conditions. By employing computer vision-based automated facial keypoint tracking, a new metric similar to ISC, called the Average t-statistic, is introduced. Unlike existing facial expression-based methodologies that measure CRM of secondary indicators like inferred emotions, keypoint, and ICA-based features, the Average t-statistic is closely associated with the direct measurement of consistent facial muscle movement using the Facial Action Coding System (FACS). This is evidenced in DISFA dataset where the time-series of Average t-statistic has a high correlation (R2 = 0.78) with a metric called AU consistency, which directly measures facial muscle movement through FACS coding of video frames. The simplicity of recording facial expressions with the automated Average t-statistic expands the applications of CRM such as measuring engagement in online learning, customer interactions, etc., and diagnosing outliers in healthcare conditions like stroke, autism, depression, etc. To promote further research, we have made the code repository publicly available.