Types Of Modalities Research Articles

Biometrics in extended reality: a review

AbstractIn the domain of Extended Reality (XR), particularly Virtual Reality (VR), extensive research has been devoted to harnessing this transformative technology in various real-world applications. However, a critical challenge that must be addressed before unleashing the full potential of XR in practical scenarios is to ensure robust security and safeguard user privacy. This paper presents a systematic survey of the utility of biometric characteristics applied in the XR environment. To this end, we present a comprehensive overview of the different types of biometric modalities used for authentication and representation of users in a virtual environment. We discuss different biometric vulnerability gateways in general XR systems for the first time in the literature along with taxonomy. A comprehensive discussion on generating and authenticating biometric-based photorealistic avatars in XR environments is presented with a stringent taxonomy. We also discuss the availability of different datasets that are widely employed in evaluating biometric authentication in XR environments together with performance evaluation metrics. Finally, we discuss the open challenges and potential future work that need to be addressed in the field of biometrics in XR.

Quantifying tumor specificity using Bayesian probabilistic modeling for drug and immunotherapeutic target discovery.

In diseases such as cancer, the design of new therapeutic strategies requires extensive, costly, and unfortunately sometimes deadly testing to reveal life threatening off-target effects. We hypothesized that the disease specificity of targets can be systematically learned for all genes by jointly evaluating complementary molecular measurements of healthy tissues using a hierarchical Bayesian modeling approach. Our method, BayesTS, integrates protein and gene expression evidence and includes tunable parameters to moderate tissue essentiality. Applied to all protein coding genes, BayesTS outperforms alternative strategies to define therapeutic targets and nominates previously unknown targets while allowing for incorporation of new types of modalities. To expand target repertoires, we show that extension of BayesTS to splicing antigens and combinatorial target pairs results in more specific targets for therapy. We expect that BayesTS will facilitate improved target prioritization for oncology drug development, ultimately leading to the discovery of more effective and safer treatments.

Hypercomplex Graph Neural Network: Towards Deep Intersection of Multi-modal Brain Networks.

The multi-modal neuroimage study has provided insights into understanding the heteromodal relationships between brain network organization and behavioral phenotypes. Integrating data from various modalities facilitates the characterization of the interplay among anatomical, functional, and physiological brain alterations or developments. Graph Neural Networks (GNNs) have recently become popular in analyzing and fusing multi-modal, graph-structured brain networks. However, effectively learning complementary representations from other modalities remains a significant challenge due to the sophisticated and heterogeneous inter-modal dependencies. Furthermore, most existing studies often focus on specific modalities (e.g., only fMRI and DTI), which limits their scalability to other types of brain networks. To overcome these limitations, we propose a HyperComplex Graph Neural Network (HC-GNN) that models multi-modal networks as hypercomplex tensor graphs. In our approach, HC-GNN is conceptualized as a dynamic spatial graph, where the attentively learned inter-modal associations are represented as the adjacency matrix. HC-GNN leverages hypercomplex operations for inter-modal intersections through cross-embedding and cross-aggregation, enriching the deep coupling of multi-modal representations. We conduct a statistical analysis on the saliency maps to associate disease biomarkers. Extensive experiments on three datasets demonstrate the superior classification performance of our method and its strong scalability to various types of modalities. Our work presents a powerful paradigm for the study of multi-modal brain networks.

Salient Object Detection From Arbitrary Modalities.

Toward desirable saliency prediction, the types and numbers of inputs for a salient object detection (SOD) algorithm may dynamically change in many real-life applications. However, existing SOD algorithms are mainly designed or trained for one particular type of inputs, failing to be generalized to other types of inputs. Consequentially, more types of SOD algorithms need to be prepared in advance for handling different types of inputs, raising huge hardware and research costs. Differently, in this paper, we propose a new type of SOD task, termed Arbitrary Modality SOD (AM SOD). The most prominent characteristics of AM SOD are that the modality types and modality numbers will be arbitrary or dynamically changed. The former means that the inputs to the AM SOD algorithm may be arbitrary modalities such as RGB, depths, or even any combination of them. While, the latter indicates that the inputs may have arbitrary modality numbers as the input type is changed, e.g. single-modality RGB image, dual-modality RGB-Depth (RGB-D) images or triple-modality RGB-Depth-Thermal (RGB-D-T) images. Accordingly, a preliminary solution to the above challenges, i.e. a modality switch network (MSN), is proposed in this paper. In particular, a modality switch feature extractor (MSFE) is first designed to extract discriminative features from each modality effectively by introducing some modality indicators, which will generate some weights for modality switching. Subsequently, a dynamic fusion module (DFM) is proposed to adaptively fuse features from a variable number of modalities based on a novel Transformer structure. Finally, a new dataset, named AM-XD, is constructed to facilitate research on AM SOD. Extensive experiments demonstrate that our AM SOD method can effectively cope with changes in the type and number of input modalities for robust salient object detection. Our code and AM-XD dataset will be released on https://github.com/nexiakele/AMSODFirst.

Non-linear communication via graded modal session types

Session types provide guarantees about concurrent behaviour and can be understood through their correspondence with linear logic, with propositions as sessions and proofs as processes. However, a strictly linear setting is somewhat limiting since there exist various useful patterns of communication that rely on non-linear behaviours. For example, shared channels provide a way to repeatedly spawn a process with binary communication along a fresh linear session-typed channel. Non-linearity can be introduced in a controlled way in programming through the general concept of graded modal types, which are a framework encompassing various kinds of coeffect typing (describing how computations make demands on their context). This paper shows how graded modal types can be leveraged alongside session types to enable various well-known non-linear concurrency behaviours to be re-introduced in a precise manner in a type system with a linear basis. The ideas here are demonstrated using Granule, a functional programming language with linear, indexed, and graded modal types. We then define a core calculus capturing the requisite type features and our new graded modal session-typed primitives. We give an operational model and establish key properties following from the typing.

Robust Multimodal Representation under Uncertain Missing Modalities

Multimodal representation learning has gained significant attention across various fields, yet it faces challenges when dealing with missing modalities in real-world applications. Existing solutions are confined to specific scenarios, such as single-modality missing or missing modalities in test cases, thereby restricting their applicability. To address a more general scenario of uncertain missing modalities in both training and testing phases, we propose Robust Multimodal Representation under Uncertain Missing Modalities (RMRU). This framework projects each modality's representation into a shared subspace, enabling the reconstruction of any missing modalities within a unified model. We propose an interaction refinement module that utilizes cross-modal attention to enhance these reconstructions, particularly beneficial in scenarios with limited complete modality data. Furthermore, we introduce an iterative training strategy that alternately trains different modules to effectively utilize both complete and incomplete modality data. Experimental results on four benchmark datasets demonstrate the superiority of RMRU over existing baselines, particularly in scenarios with a high rate of missing modalities. Remarkably, our proposed RMRU can be broadly applied to diverse scenarios, regardless of modality types and quantities.

17 Caregiver experiences with enteral feeding transitions

Abstract Background Children with medical complexity (CMC) are an expanding group within paediatrics due to advances in medical care, often including a dependence on medical technology. Issues such as oromotor swallowing dysfunction and feeding difficulties are common in this population. As a result, many CMC require the use of enteral tubes for feeding. While evidence does support improved nutritional status with the use of enteral feeding tubes, there is limited literature discussing how these interventions impact other patient centered outcomes, such as quality of life for the child, changes in parental caregiving, and logistical implications for the family. A comprehensive understanding of the positive and negative outcomes associated with selection and transition between different feeding tube options is needed. Objectives To describe and explore the caregiver and patient experience with different feeding tubes, with a focus on the experience of transitioning between different enteral feeding modalities. Design/Methods We conducted semi-structured interviews with 16 participants, including both caregivers and two patients. Reflexive thematic analysis was utilized to identify preliminary themes from interviews. Themes were further refined by the study team. Results The themes identified through interviews included: A) the pros and cons with each type of enteral feeding modality, B) the emotional impacts on patients and caregivers, C) logistical impacts on day-to-day life for families, and D) the experience of transitioning between feeding tubes. Globally, participants spoke positively about their transition from nasogastric feeding to either gastrostomy or gastrojejunal feeding tubes. Caregivers also felt that they were integral to the decision-making process, though some felt in hindsight the transition process was too slow. Conclusion This study highlights both the complexity and heterogeneity of the caregiver experience with different feeding tubes. It found that most families consistently viewed their transitions with enteral feeding tubes positively, as this was seen as a significant milestone in helping their child both nutritionally and with their quality of life. Finally, further research could be done to explore experiences with gastrojejunal and jejunal feeding, as this patient population is not well elucidated within the literature.

Modality deep-learning frameworks for fake news detection on social networks: a systematic literature review

Fake news on social networks is a challenging problem due to the rapid dissemination and volume of information, as well as the ease of creating and sharing content anonymously. Fake news stories are problematic not only for the credibility of online journalism, but also due to their detrimental real-world consequences. The primary research objective of this study is: What are the recent state-of-the-art modalities based on deep learning to detect fake news in social networks. This paper presents a systematic literature review of deep learning-based fake news detection models in social networks. The methodology followed a rigorous approach, including predefined criteria for study selection of deep learning modalities. This study focuses on the types of deep learning modalities; unimodal (refers to the use of a single model for analysis or modeling purposes) and multimodal models (refers to the integration of multiple models). The results of this review reveal the strengths and weaknesses of modalities approaches, as well as the limitations of low-resource languages datasets. Furthermore, it provides insights into future directions for deep learning models and different fact checking techniques. At the end of this study, we discuss the problem of fake news detection in the era of large language models in terms of advantages, drawbacks, and challenges.

Internally-consistent and fully-unbiased multimodal MRI brain template construction from UK Biobank: Oxford-MM

Abstract Anatomical MRI templates of the brain are essential to group-level analyses and image processing pipelines, as they provide a reference space for spatial normalisation. While it has become common for studies to acquire multimodal MRI data, many templates are still limited to one type of modality, usually either scalar or tensor-based. Aligning each modality in isolation does not take full advantage of the available complementary information, such as strong contrast between tissue types in structural images, or axonal organisation in the white matter in diffusion tensor images. Most existing strategies for multimodal template construction either do not use all modalities of interest to inform the template construction process, or do not use them in a unified framework. Here, we present multimodal, cross-sectional templates constructed from UK Biobank data: the OMM-1 template, and age-dependent templates for each year of life between 45 to 81. All templates are fully unbiased to represent the average shape of the populations they were constructed from, and internally consistent through jointly informing the template construction process with T1, T2-FLAIR and DTI data. The OMM-1 template was constructed with a multi-resolution, iterative approach using 240 individuals in the 50-55 year age range. The age-dependent templates were estimated using a Gaussian Process, which describes the change in average brain shape with age in 37,330 individuals. All templates show excellent contrast and alignment within and between modalities. The global brain shape and size is not preconditioned on existing templates, although maximal possible compatibility with MNI-152 space was maintained through rigid alignment. We showed benefits in registration accuracy across two datasets (UK Biobank and HCP), when using the OMM-1 as the template compared with FSL’s MNI-152 template, and found that the use of age-dependent templates further improved accuracy to a small but detectable extent. All templates are publicly available and can be used as a new reference space for uni- or multimodal spatial alignment.

Modality in Climate Change-Related Discourse by King Charles III

The article presents a corpus-based study that aims at establishing how modality is used in climate change-related discourse by King Charles III of the United Kingdom of Great Britain and Northern Ireland (the UK). The study involved a corpus of speeches on the topic of climate change delivered by King Charles III from 2005 to 2023. The corpus was analysed in the computer software AntConc (Anthony) in order to compute the frequency of the occurrence of the central modal verbs. Thereafter, they were analysed qualitatively to establish the types of modality associated with them. The analysis revealed that climate change-related discourse by King Charles III was marked by a high frequency of the occurrence of the modal verbs will and can. Their frequency, pragmatic roles, and association with the different types of modality are further discussed in the article.

Sensitivity by Parametricity

The work of Fuzz has pioneered the use of functional programming languages where types allow reasoning about the sensitivity of programs. Fuzz and subsequent work (e.g., DFuzz and Duet) use advanced technical devices like linear types, modal types, and partial evaluation. These features usually require the design of a new programming language from scratch—a significant task on its own! While these features are part of the classical toolbox of programming languages, they are often unfamiliar to non-experts in this field. Fortunately, recent studies (e.g., Solo) have shown that linear and complex types in general, are not strictly needed for the task of determining programs’ sensitivity since this can be achieved by annotating base types with static sensitivity information. In this work, we take a different approach. We propose to enrich base types with information about the metric relation between values, and we present the novel idea of applying parametricity to derive direct proofs for the sensitivity of functions. A direct consequence of our result is that calculating and proving the sensitivity of functions is reduced to simply type-checking in a programming language with support for polymorphism and type-level naturals. We formalize our main result in a calculus, prove its soundness, and implement a software library in the programming language Haskell–where we reason about the sensitivity of canonical examples. We show that the simplicity of our approach allows us to exploit the type inference of the host language to support a limited form of sensitivity inference. Furthermore, we extend the language with a privacy monad to showcase how our library can be used in practical scenarios such as the implementation of differentially private programs, where the privacy guarantees depend on the sensitivity of user-defined functions. Our library, called Spar, is implemented in less than 500 lines of code.

A Modal Type Theory of Expected Cost in Higher-Order Probabilistic Programs

The design of online learning algorithms typically aims to optimise the incurred loss or cost , e.g., the number of classification mistakes made by the algorithm. The goal of this paper is to build a type-theoretic framework to prove that a certain algorithm achieves its stated bound on the cost. Online learning algorithms often rely on randomness, their loss functions are often defined as expectations, precise bounds are often non-polynomial (e.g., logarithmic) and proofs of optimality often rely on potential-based arguments. Accordingly, we present pλ-amor, a type-theoretic graded modal framework for analysing (expected) costs of higher-order probabilistic programs with recursion. pλ-amor is an effect-based framework which uses graded modal types to represent potentials, cost and probability at the type level. It extends prior work (λ-amor) on cost analysis for deterministic programs. We prove pλ-amor sound relative to a Kripke step-indexed model which relates potentials with probabilistic coupling. We use pλ-amor to prove cost bounds of several examples from the online machine learning literature. Finally, we describe an extension of pλ-amor with a graded comonad and describe the relationship between the different modalities.

The influence of assistance in home-based exercise programmes for individuals with intellectual disabilities.

Regular physical activity (PA) decreases the risk of comorbidities associated with a sedentary lifestyle in individuals with intellectual disabilities (IDs). They also may experience additional barriers that may prevent PA, including access, proper instruction and support. At-home PA programming is a feasible alternative to long-term adherence. This study aimed to determine (1) how an at-home DVD programme affects PA adherence and (2) the extent to which caregiver support influences PA. Thirty-one adolescent or adult-aged participants with mild or moderate ID were randomised into a DVD or control group exercise intervention. Participants were given autonomy to choose what type of PA modality from the intervention they would like to participate in. Caregiver support during the PA was also measured. The two-way repeated-measures MANOVA demonstrated that the intervention DVD group was statistically significant, with a large effect size in PA minutes (P=0.014, ηP2=0.236) and rate of perceived exertion (P=0.002, ηP2=0.342) compared with the control group. The findings demonstrated that using an adapted DVD increased PA minutes and the rate of perceived exertion without high levels of caregiver support.

E-scooter-related imaging: a new emerging high-energy trauma patient.

To quantify the volume and nature of radiological imaging performed for e-scooter related trauma presentations in a tertiary referral centre and to identify common radiological pattern of injuries. Our study received institutional review board approval. We retrospectively reviewed all anonymised radiological imaging studies performed in our institution for the term 'scooter' or 'e-scooter' between July 2020 and July 2021. The patient demographics, number of imaging studies performed and modality type as well as the nature of injuries identified were recorded. Within the study period, 147 patients with e-scooter-related injuries were referred for radiological imaging. Forty-two (29%) of those patients had positive radiological findings. The vast majority of injuries on radiographs were upper limb injuries typical of a fall on outstretched hand (FOOSH) pattern. Thirty-two patients (22%) required advanced diagnostics with CT or MRI with 11 of these patients having positive findings. Four patients suffered significant head injuries including skull fractures and/or intracranial haemorrhage. Five patients suffered facial bone fractures, and 2 suffered spinal fractures. One patient suffered a handlebar abdominal trauma with a resulting large bowel injury ultimately requiring bowel resection. E-scooter-related injuries represent a new and emerging high-energy trauma patient. These patients demonstrate injury patterns similar to other high-energy trauma such as road traffic accidents. Although the most common injuries are musculoskeletal upper limb injuries typical of FOOSH, a large proportion of these require advanced diagnostic imaging (CT, MRI) which were commonly positive for significant injuries.

Head and Neck Cancer Treatment Delays in 2021: Estimating Distributional Effects by Site, Surgery, and p16-positivity

BackgroundFollowing decades of policies increasing access to high-quality cancer treatment, the COVID-19 pandemic upended the U.S. healthcare system. The pandemic’s disruption likely affected an often-overlooked dimension of quality cancer treatment: timely initiation. Timely treatment initiation is especially critical for head and neck cancer (HNC). We aimed to assess how the treatment interval (diagnosis to treatment initiation) changed in 2021 for different types of HNCs and treatment modalities. MethodologyWe analyzed Surveillance, Epidemiological, End Results (SEER) case data for years 2007-2021. Using ICD site codes, cancers were restricted to oropharynx, oral cavity, other pharynx, larynx. Oropharynx cancers were stratified by Human Papillomavirus (HPV) subtype based on p16-positivity codes. The outcome of interest was a variable measuring the number of days from diagnosis to treatment initiation. Cases were stratified by site and whether they received surgery as first course of treatment. To overcome validity threats from skewed treatment interval data and unobserved heterogeneity, we constructed an unconditional quantile regression model to estimate the effect of treatment in 2021 across the distribution of the treatment interval. Results155,273 patients in SEER initiated HNC treatment between 2007-2021. The median treatment interval was 29days (Interquartile Range = 2-48). Among patients not receiving surgery, 2021 was associated with delayed treatment for all sites except oral cavity. For patients receiving surgery, 2021 was only associated with delayed treatment for p16+ Oropharynx cancer. DiscussionHNC patients overall, but HPV+ Oropharynx cancer patients especially, experienced treatment delays in 2021. These delays, and their consequences, warrant policymaking attention.

Kidney Replacement Therapies in Advanced Heart Failure - Timing, Modalities, and Clinical Considerations.

Acute kidney dysfunction is commonly encountered in advanced heart failure and carries significant prognostic implications, often leading to poorer outcomes and increased mortality. It can alter the course of decision making for left ventricular assist device (LVAD) and cardiac transplantation candidacy. Kidney replacement therapies (KRT) offer a critical intervention in this context but require careful consideration of timing, various types of KRT modalities, individual patient preferences and circumstances. This review discusses the intricacies of KRT in advanced heart failure, examining how to optimize timing and choose among the various KRT modalities. It also provides a detailed discussion on the unique clinical scenarios that clinicians may face when treating this vulnerable patient group.

Identifying correlations between nutritional impact symptoms and risk factors for malnutrition in adult cancer patients with solid tumors: a cross-sectional study from a developing country.

Many cancer patients experience malnutrition, which can worsen their health and survival outcomes. However, limited research exists in our region on how common malnutrition is among hospitalized adults with solid tumors and what factors contribute to it. This study aimed to identify these factors and assess the effects of nutritional impact symptoms (NISs) caused by solid tumors on patients' nutritional status. Between July 2022 and February 2023, a cross-sectional study was carried out on the campuses of two major cancer treatment centers located in a national university hospital and a governmental hospital. Twelve NISs were adopted from the Patient-Generated Subjective Global Assessments (PG-SGA). The Nutrition Risk Screening 2002 (NRS-2002) instrument was used to thoroughly assess the risk of malnutrition. Multiple linear regressions were used to conduct a thorough study. A cohort of 294 participants was included. The prevalence of malnutrition risk (NRS score ≥ 3) was 26.9%. Parameters such as age, gender, marital status, educational level, monthly income, type of cancer and treatment modality, and the need for mealtime assistance exhibited statistically significant associations with malnutrition (p < 0.05). The results revealed a substantial inverse correlation between fluid intake and the NRS-2002 score (p < 0.001). Furthermore, symptoms related to solid tumors and their treatment, including chewing difficulties, fatigue, dry mouth, anorexia, constipation, nausea, dizziness, and a sensation of fullness, were also significantly associated with malnutrition (p < 0.05). Additional insights from the regression analysis underscored the independent correlation between the risk of malnutrition in solid malignant malignancies and factors such as anorexia (p < 0.001), colorectal cancer (p = 0.003), gender (p = 0.018), educational attainment (p = 0.049), and the need for mealtime assistance among patients (p < 0.001). Malnutrition is a major issue among adult cancer patients, particularly those with solid tumors. Anorexia, colorectal cancer, gender, educational attainment, and the need for mealtime assistance were identified as factors that led to malnutrition in our research. This study emphasizes the need for a multidisciplinary plan of care to diagnose and treat malnutrition, improve overall therapy, and reduce mortality and morbidity.

Expanding the scope: multimodal dimensions in aphasia discourse analysis-preliminary findings.

Aphasia, resulting from acquired brain injury, disrupts language processing and usage, significantly impacting individuals' social communication and life participation. Given the limitations of traditional assessments in capturing the nuanced challenges faced by individuals with aphasia, this study seeks to explore the potential benefits of integrating multimodal communication elements into discourse analysis to better capture narrative proficiency in this population. This study examined how incorporating multimodal communication elements (e.g., physical gestures, writing, drawing) into discourse analysis may affect the narrative outcomes of persons with aphasia compared to those observed using methods that exclude multimodal considerations. Participants included individuals with chronic aphasia and age-and education-matched healthy controls who completed a storytelling task-the Bear and the Fly story. Macrolinguistic scores were obtained using verbal-only and multimodal scoring approaches. Additionally, the frequency and type of multimodal communication use during storytelling were examined in relation to aphasia characteristics. Statistical analyses included both within-group and between-group comparisons as well as correlational analyses. Individuals with aphasia scored significantly higher in terms of their macrolinguistic abilities when multimodal scoring was considered compared to verbal-only scoring. Within the aphasia group, there were prominent differences noted in macrolinguistic scores for both fluent and nonfluent aphasia. Specifically, both groups scored higher on Main Concepts when multimodal scoring was considered, with the nonfluent group demonstrating significantly higher Main Concept and total macrolinguistic rubric scores in multimodal scoring compared to verbal scoring on the storytelling task. Additionally, aphasia severity showed moderate positive correlations with total macrolinguistic scores, indicating that individuals with less severe aphasia tended to produce higher quality narratives. Lastly, although persons with aphasia used different types of nonverbal modalities (i.e., drawing, writing), the use of meaning-laden gestures was most predominant during storytelling, emphasizing the importance of multimodal elements in communication for individuals with aphasia. Our preliminary study findings underscore the importance of considering multimodal communication in assessing discourse performance among individuals with aphasia. Tailoring assessment approaches based on aphasia subtypes can provide valuable insights into linguistic abilities and inform targeted intervention strategies for improving communication outcomes.

AN UPDATE ON DIAGNOSIS AND MANAGEMENT OF THROMBOEMBOLISM IN PREGNANCY

Thromboembolism during pregnancy poses a significant challenge in maternal healthcare, substantially contributing to maternal morbidity and mortality worldwide. Understanding its diagnosis and management is crucial for improving maternal and fetal outcomes. Objective: To evaluate the diagnosis and management of thromboembolism in pregnant women. Methods: A retrospective cohort study was conducted at Lady Reading Hospital, Peshawar, from July 2022 to July 2024. The study included 175 pregnant females diagnosed with venous thromboembolism (VTE) during their pregnancy. Data were collected on patient demographics, trimester of diagnosis, type of thromboembolism (deep vein thrombosis [DVT] or pulmonary embolism [PE]), treatment modalities, and outcomes. Patients were treated with either low molecular weight heparin (LMWH) or unfractionated heparin (UFH). Descriptive statistics were used to analyze the data, with results presented as percentages and means with standard deviations. Results: The mean age of the participants was 32.01 ± 2.35 years. Thromboembolism was most commonly diagnosed in the third trimester (65%), followed by the second trimester (25%) and the first trimester (10%). DVT was observed in 74% of patients, while PE occurred in 26%. Among the 160 patients treated with LMWH, 3% (5 patients) experienced a recurrence of VTE. Of the 15 patients who received UFH, 5% (8 patients) had significant bleeding complications. Overall maternal outcomes included a 3% recurrence rate of VTE and a 5% rate of significant bleeding complications. No maternal mortality was reported. Conclusion: Early diagnosis and timely management of thromboembolism in pregnancy, mainly through the use of low molecular weight heparin and a multidisciplinary care approach, significantly improve maternal and fetal outcomes. LMWH was associated with fewer complications compared to UFH, highlighting its effectiveness and safety in treating pregnant women with VTE.

Commonalities and variations in emotion representation across modalities and brain regions

Humans express emotions through various modalities such as facial expressions and natural language. However, the relationships between emotions expressed through different modalities and their correlations with neural activities remain uncertain. Here, we aimed to unveil some of these uncertainties by investigating the similarity of emotion representations across modalities and brain regions. First, we represented various emotion categories as multi-dimensional vectors derived from visual (face), linguistic, and visio-linguistic data, and used representational similarity analysis to compare these modalities. Second, we examined the linear transferability of emotion representation from other modalities to the visual modality. Third, we compared the representational structure derived in the first step with those from brain activities across 360 regions. Our findings revealed that emotion representations share commonalities across modalities with modality-type dependent variations, and they can be linearly mapped from other modalities to the visual modality. Additionally, emotion representations in uni-modalities showed relatively higher similarity with specific brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region. These findings suggest that emotional experiences are represented differently across various brain regions with varying degrees of similarity to different modality types, and that they may be multi-modally conveyable in visual and linguistic domains.