Multimodal Clustering Research Articles

Multi-modal data clustering using deep learning: A systematic review

Multi-modal clustering represents a formidable challenge in the domain of unsupervised learning. The objective of multi-modal clustering is to categorize data collected from diverse modalities, such as audio, visual, and textual sources, into distinct clusters. These clustering techniques operate by extracting shared features across modalities in an unsupervised manner, where the identified common features exhibit high correlations within real-world objects. Recognizing the importance of perceiving the correlated nature of these features is vital for enhancing clustering accuracy in multi-modal settings. This survey explores Deep Learning (DL) techniques applied to multi-modal clustering, encompassing methodologies such as Convolutional Neural Networks (CNN), Autoencoders (AE), Recurrent Neural Networks (RNN), and Graph Convolutional Networks (GCN). Notably, this survey represents the first attempt to investigate DL techniques specifically for multi-modal clustering. The survey presents a novel taxonomy for DL-based multi-modal clustering, conducts a comparative analysis of various multi-modal clustering approaches, and deliberates on the datasets employed in the evaluation process. Additionally, the survey identifies research gaps within the realm of multi-modal clustering, offering insights into potential future avenues for research.

M-ClustEHR: A multimodal clustering approach for electronic health records

Sepsis refers to a potentially life-threatening situation where the immune system of the human body has an extreme response to an infection. In the presence of underlying comorbidities, the situation can become even worse and result in death. Employing unsupervised machine learning techniques, such as clustering, can assist in providing a better understanding of patient phenotypes by unveiling subgroups characterized by distinct sepsis progression and treatment patterns. More concretely, this study introduces M-ClustEHR, a clustering approach that utilizes medical data of multiple modalities by employing a multimodal autoencoder for learning comprehensive sepsis patient representations. M-ClustEHR consistently outperforms traditional clustering approaches in terms of several internal clustering performance metrics, as well as cluster stability in identifying phenotypes in the sepsis cohort. The unveiled patterns, supported by existing medical literature and clinicians, highlight the importance of multimodal clustering for advancing personalized sepsis care.

Brain high-throughput multi-omics data reveal molecular heterogeneity in Alzheimer's disease.

Unbiased data-driven omic approaches are revealing the molecular heterogeneity of Alzheimer disease. Here, we used machine learning approaches to integrate high-throughput transcriptomic, proteomic, metabolomic, and lipidomic profiles with clinical and neuropathological data from multiple human AD cohorts. We discovered 4 unique multimodal molecular profiles, one of them showing signs of poor cognitive function, a faster pace of disease progression, shorter survival with the disease, severe neurodegeneration and astrogliosis, and reduced levels of metabolomic profiles. We found this molecular profile to be present in multiple affected cortical regions associated with higher Braak tau scores and significant dysregulation of synapse-related genes, endocytosis, phagosome, and mTOR signaling pathways altered in AD early and late stages. AD cross-omics data integration with transcriptomic data from an SNCA mouse model revealed an overlapping signature. Furthermore, we leveraged single-nuclei RNA-seq data to identify distinct cell-types that most likely mediate molecular profiles. Lastly, we identified that the multimodal clusters uncovered cerebrospinal fluid biomarkers poised to monitor AD progression and possibly cognition. Our cross-omics analyses provide novel critical molecular insights into AD.

Integrated Heterogeneous Graph Attention Network for Incomplete Multi-modal Clustering

Integrated Heterogeneous Graph Attention Network for Incomplete Multi-modal Clustering

Deep contrastive representation learning for multi-modal clustering

Benefiting from the informative expression capability of contrastive representation learning (CRL), recent multi-modal learning studies have achieved promising clustering performance. However, it should be pointed out that the existing multi-modal clustering methods based on CRL fail to simultaneously take the similarity information embedded in inter- and intra-modal levels. In this study, we mainly explore deep multi-modal contrastive representation learning, and present a multi-modal learning network, named trustworthy multi-modal contrastive clustering (TMCC), which incorporates contrastive learning and adaptively reliable sample selection with multi-modal clustering. Specifically, we are concerned with an adaptive filter to learn TMCC via progressing from ‘easy’ to ‘complex’ samples. Based on this, with the highly confident clustering labels, we present a new contrastive loss to learn modal-consensus representation, which takes into account not only the inter-modal similarity but also the intra-modal similarity. Experimental results show that these principles in TMCC consistently help promote clustering performance improvement.

Integrating imaging and genomic data for the discovery of distinct glioblastoma subtypes: a joint learning approach

Glioblastoma is a highly heterogeneous disease, with variations observed at both phenotypical and molecular levels. Personalized therapies would be facilitated by non-invasive in vivo approaches for characterizing this heterogeneity. In this study, we developed unsupervised joint machine learning between radiomic and genomic data, thereby identifying distinct glioblastoma subtypes. A retrospective cohort of 571 IDH-wildtype glioblastoma patients were included in the study, and pre-operative multi-parametric MRI scans and targeted next-generation sequencing (NGS) data were collected. L21-norm minimization was used to select a subset of 12 radiomic features from the MRI scans, and 13 key driver genes from the five main signal pathways most affected in glioblastoma were selected from the genomic data. Subtypes were identified using a joint learning approach called Anchor-based Partial Multi-modal Clustering on both radiomic and genomic modalities. Kaplan–Meier analysis identified three distinct glioblastoma subtypes: high-risk, medium-risk, and low-risk, based on overall survival outcome (p < 0.05, log-rank test; Hazard Ratio = 1.64, 95% CI 1.17–2.31, Cox proportional hazard model on high-risk and low-risk subtypes). The three subtypes displayed different phenotypical and molecular characteristics in terms of imaging histogram, co-occurrence of genes, and correlation between the two modalities. Our findings demonstrate the synergistic value of integrated radiomic signatures and molecular characteristics for glioblastoma subtyping. Joint learning on both modalities can aid in better understanding the molecular basis of phenotypical signatures of glioblastoma, and provide insights into the biological underpinnings of tumor formation and progression.

Effective multi-modal clustering method via skip aggregation network for parallel scRNA-seq and scATAC-seq data.

In recent years, there has been a growing trend in the realm of parallel clustering analysis for single-cell RNA-seq (scRNA) and single-cell Assay of Transposase Accessible Chromatin (scATAC) data. However, prevailing methods often treat these two data modalities as equals, neglecting the fact that the scRNA mode holds significantly richer information compared to the scATAC. This disregard hinders the model benefits from the insights derived from multiple modalities, compromising the overall clustering performance. To this end, we propose an effective multi-modal clustering model scEMC for parallel scRNA and Assay of Transposase Accessible Chromatin data. Concretely, we have devised a skip aggregation network to simultaneously learn global structural information among cells and integrate data from diverse modalities. To safeguard the quality of integrated cell representation against the influence stemming from sparse scATAC data, we connect the scRNA data with the aggregated representation via skip connection. Moreover, to effectively fit the real distribution of cells, we introduced a Zero Inflated Negative Binomial-based denoising autoencoder that accommodates corrupted data containing synthetic noise, concurrently integrating a joint optimization module that employs multiple losses. Extensive experiments serve to underscore the effectiveness of our model. This work contributes significantly to the ongoing exploration of cell subpopulations and tumor microenvironments, and the code of our work will be public at https://github.com/DayuHuu/scEMC.

Exploring the Reform of College English Teaching from the Perspective of New Liberal Arts

Abstract The construction of new liberal arts requires college English teaching to introduce new concepts, utilize new technologies, and promote students’ effective learning through the intersection of disciplines and curricula, to cultivate integrative talents with innovative and practical abilities. Under the perspective of new liberal arts, this paper integrates deep commonality and uniqueness knowledge mining multimodal clustering, DCUMC algorithmic model and multimodal teaching means to construct a multimodal analysis model framework and reform the English teaching model in colleges and universities. In addition, the information entropy is used to improve the ID3 algorithm to construct multidimensional evaluation indexes for college English teaching, and the improved ID3 algorithm is used for data mining of multidimensional evaluation so that teachers can adjust the teaching progress and difficulty according to real-time and accurate feedback information. Experiments are conducted on multimodal teaching mode and teaching evaluation, and the experimental results show that compared with the traditional teaching mode, the average scores of multimodal English teaching mode after five months are 96 and 85, which is an improvement of 11 points, 69.58%, 67.93%, 56.25%, and 53.79% higher in the ratio of excellent and good evaluations, respectively. Compared to the single teaching evaluation, the multidimensional evaluation points out that the traditional teaching mode needs more designer-student interaction as well as a reduction in the difficulty of the teaching content. In conclusion, the multimodal English teaching model can effectively improve teaching effectiveness and is implementable.

ConGMC: Consistency-Guided Multimodal Clustering via Mutual Information Maximin

ConGMC: Consistency-Guided Multimodal Clustering via Mutual Information Maximin

Towards a Multimodal Hermeneutic Model: The case of Uber-Blog-mediated advertising discourse order of ‘Saudization’

This study proposes a Multimodal Hermeneutic Model (for short, MHM) as a methodology that extends the analytic scope of Ricoeur’s (1973, 1976, 1981) classic hermeneutic theory of text interpretation towards examining multimodal texts. The model has been empirically tested by examining the advertising discourse order of ‘Saudization’ as channelled via the Saudi Uber Blog’s multimodal text. A twofold social semiotic praxis has been theoretically incorporated into the distanciation-appropriation dialectics underlying the interpretation of multimodal texts in potentia . First, a multimodal cluster transcription (Baldry Thibault 2006) has been utilized in enhancing the description of distanciated text sense as a holistic configuration of clusters across different communicative modalities with interacting semiotic modes ( intra-textually ). Second, an interpersonal-meaning analysis of multimodal participants (Kress Van Leeuwen 2006a, 2006b, Halliday Matthiessen 2004) has been employed in enriching the explanation of appropriated text reference ( extra-textually ). The empirical site used for validating the MHM is the Uber-Blog-mediated multimodal text designed by Saudi Arabia-based Uber Company. The multimodal transcription of textual clusters has demonstrated how the text sense thematically revolves around the macro topic of ‘Saudization’ across verbal, visual, and pictorial modalities with material and semiotic modes, viz. linguistic, graphological, anthropic, sartorial, spatial, natural, and technological. The multimodal participant analysis has explained how the referents of (i) a model Saudi Uber driver, (ii) the Saudi Public Transport Authority, (iii) Saudi driver-partners, and (iv) Uber app collectively contribute to the recontextualization of ‘Saudization’ from a governmental discourse to an advertising discourse order realized in the multimodal text under analysis.

A Bottleneck Network with Light Attention for Multimodal Clustering

Multimodal Clustering (MMC) aims to boost clustering performance by utilizing complementary information from multimodal data. However, the heterogeneous gap between various modalities and the interference of superfluous information in multimodal data makes it challenging to comprehend the complex dependencies between them. To address these issues, we present a Bottleneck Network with Light Attention for Multimodal Clustering (LABMC) to understand the dependencies between different modalities and explore complementary information while maximally eliminating superfluous information. Specifically, Light Attention (LA) is proposed as a concise multimodal interaction mechanism that introduces a convolutional high-level semantic layer to understand the common high-level semantics among different modalities and explores related complementary information to obtain a multimodal fusion representation. It can efficiently comprehend the latent dependencies between diverse modalities to bridge the heterogeneous gap. Subsequently, we construct a Superfluity Elimination (SE) function to erase superfluous information from the modalities. We divide the mutual information between the fusion representation and multimodal data into superfluous information and complementary information. Under the premise of preserving complementary information with an approximation, we utilize a variational optimization method to minimize the mutual information to safely remove superfluity. The combination of LA and SE naturally forms a bottleneck, which enables the extraction of complementary information while reducing superfluous information. Finally, we measure the quality of the modalities to dynamically assign the optimal weights to SE function, which significantly enhances the rationality of SE. Extensive experiment results on five real-world datasets show that LABMC outperforms the state-of-the-art MMC methods.

Multimodal representative answer extraction in community question answering

To solve the information overload problem of multimodal answers in community question answering (CQA), this paper proposes a multimodal representative answer extraction method. First, the method of similarity calculation between multimodal answers is constructed, and multimodal clustering is used to cluster answers. Then, a binary multi-objective optimization model with three objective functions including multimodal answer coverage, multimodal answer redundancy, and multimodal answer consistency is constructed to extract a representative subset of answers. The improved Beluga whale optimization algorithm (MTRL-BWO), based on tent mapping, reinforcement learning, and multiple swarm strategy, is designed to increase the diversity of the population while avoiding local optima to improve the search capability and solution accuracy of the algorithm. Experimental results show the feasibility and superior performance of the proposed method.

Decision level integration of unimodal and multimodal single cell data with scTriangulate

Decisively delineating cell identities from uni- and multimodal single-cell datasets is complicated by diverse modalities, clustering methods, and reference atlases. We describe scTriangulate, a computational framework to mix-and-match multiple clustering results, modalities, associated algorithms, and resolutions to achieve an optimal solution. Rather than ensemble approaches which select the “consensus”, scTriangulate picks the most stable solution through coalitional iteration. When evaluated on diverse multimodal technologies, scTriangulate outperforms alternative approaches to identify high-confidence cell-populations and modality-specific subtypes. Unlike existing integration strategies that rely on modality-specific joint embedding or geometric graphs, scTriangulate makes no assumption about the distributions of raw underlying values. As a result, this approach can solve unprecedented integration challenges, including the ability to automate reference cell-atlas construction, resolve clonal architecture within molecularly defined cell-populations and subdivide clusters to discover splicing-defined disease subtypes. scTriangulate is a flexible strategy for unified integration of single-cell or multimodal clustering solutions, from nearly unlimited sources.

Graph Embedding Contrastive Multi-Modal Representation Learning for Clustering.

Multi-modal clustering (MMC) aims to explore complementary information from diverse modalities for clustering performance facilitating. This article studies challenging problems in MMC methods based on deep neural networks. On one hand, most existing methods lack a unified objective to simultaneously learn the inter- and intra-modality consistency, resulting in a limited representation learning capacity. On the other hand, most existing processes are modeled for a finite sample set and cannot handle out-of-sample data. To handle the above two challenges, we propose a novel Graph Embedding Contrastive Multi-modal Clustering network (GECMC), which treats the representation learning and multi-modal clustering as two sides of one coin rather than two separate problems. In brief, we specifically design a contrastive loss by benefiting from pseudo-labels to explore consistency across modalities. Thus, GECMC shows an effective way to maximize the similarities of intra-cluster representations while minimizing the similarities of inter-cluster representations at both inter- and intra-modality levels. So, the clustering and representation learning interact and jointly evolve in a co-training framework. After that, we build a clustering layer parameterized with cluster centroids, showing that GECMC can learn the clustering labels with given samples and handle out-of-sample data. GECMC yields superior results than 14 competitive methods on four challenging datasets. Codes and datasets are available: https://github.com/xdweixia/GECMC.

Clustering of single-cell multi-omics data with a multimodal deep learning method

Single-cell multimodal sequencing technologies are developed to simultaneously profile different modalities of data in the same cell. It provides a unique opportunity to jointly analyze multimodal data at the single-cell level for the identification of distinct cell types. A correct clustering result is essential for the downstream complex biological functional studies. However, combining different data sources for clustering analysis of single-cell multimodal data remains a statistical and computational challenge. Here, we develop a novel multimodal deep learning method, scMDC, for single-cell multi-omics data clustering analysis. scMDC is an end-to-end deep model that explicitly characterizes different data sources and jointly learns latent features of deep embedding for clustering analysis. Extensive simulation and real-data experiments reveal that scMDC outperforms existing single-cell single-modal and multimodal clustering methods on different single-cell multimodal datasets. The linear scalability of running time makes scMDC a promising method for analyzing large multimodal datasets.

NIMG-37. JOINT LEARNING OF IMAGING AND GENOMIC DATA REVEALS DISTINCT GLIOBLASTOMA SUBTYPES

Abstract PURPOSE The significant heterogeneity of glioblastoma is typically displayed on both phenotypical and molecular levels. Non-invasive in vivo approaches to characterize this heterogeneity would potentially facilitate personalized therapies. Here we leverage advanced unsupervised machine learning to integrate radiomic imaging features and genomics to identify distinct subtypes of glioblastoma. METHODS A retrospective cohort of 571 IDH-wildtype glioblastoma patients were collected with pre-operative multi-parametric MRI (T1, T1CE, T2, T2-FLAIR, DSC, DTI) scans (available in 462/571 patients) and targeted next-generation sequencing (NGS) data (available in 355/571 patients). Radiomic features (n= 971) were extracted from these MRI scans and a subset of 12 features were selected by L21-norm minimization. A total of 14 key driver genes in the 5 main pathways that are most frequently altered in glioblastoma were chosen. Subtypes were identified by a joint learning approach called Anchor-based Partial Multi-modal Clustering (APMC) on both radiomic and genomic modalities. RESULTS Three distinct glioblastoma subtypes were discovered by APMC based on 14-dimension NGS data together with 12 selected radiomic features representing characteristics from histograms, shape, and volumetric measures for different tumor sub-regions. The identified subtypes were 1) high-risk; 2) medium-risk; and 3) low-risk, in terms of their overall survival outcome in Kaplan-Meier analysis (p= 5.52e-6, log-rank test; HR= 1.51, 95%CI:1.20-1.74, Cox proportional hazard model). The three subtypes also displayed different molecular characteristics: subtype 1 exhibited increased frequency of mutation in [EGFR, PIK3CA, PTEN, NF1], subtype 3 showed frequently mutated [PDGFRA, ATRX], while subtype 2 did not show significant differences for mutations in any of these genes. CONCLUSION Our results revealed the synergistic value of integrated radiomic signatures and molecular characteristics for glioblastoma subtyping. Joint learning on both modalities could help better understand the molecular basis of phenotypical signatures of glioblastoma and further provide insights into the biologic underpinnings of tumor formation and progression.

Dark matter in galaxy clusters: Parametric strong-lensing approach

We present a parametric strong-lensing analysis of three massive galaxy clusters for which Hubble Space Telescope imaging is available, as well as spectroscopy of multiply imaged systems and galaxy cluster members. Our aim is to probe the inner shape of dark matter haloes, in particular the existence of a core. We adopted the following working hypothesis: any group- or cluster-scale dark matter clump introduced in the modelling should be associated with a luminous counterpart. We also adopted some additional well-motivated priors in the analysis, even when this degraded the quality of the fit, quantified using the root mean square between the observed and model-generated images. In particular, in order to alleviate the degeneracy between the smooth underlying component and the galaxy-scale perturbers, we used the results from previous spectroscopic campaigns, which allowed us to fix the mass of the galaxy-scale component. In the unimodal galaxy cluster AS 1063, a core mass model is favoured over a non-core mass model, and this is also the case in the multimodal cluster MACS J0416. In the unimodal cluster MACS J1206, we fail to reproduce the strong-lensing constraints using a parametric approach within the adopted working hypothesis. We then successfully added a mild perturbation in the form of a superposition of B-spline potentials, which allowed us to obtain a decent fit (root mean square = 0.5″), and finally find that a core mass model is favoured. Overall, our analysis suggest evidence for core cluster-scale dark matter haloes in these three clusters. These findings may be useful for the interpretation within alternative dark matter scenario, such as self-interacting dark matter. We propose a working hypothesis for parametric strong-lensing modelling in which the quest for the best-fit model is balanced by the quest for presenting a physically motivated mass model, in particular by imposing priors.

Mitonuclear genotype remodels the metabolic and microenvironmental landscape of Hürthle cell carcinoma.

Hürthle cell carcinomas (HCCs) display two exceptional genotypes: near-homoplasmic mutation of mitochondrial DNA (mtDNA) and genome-wide loss of heterozygosity (gLOH). To understand the phenotypic consequences of these genetic alterations, we analyzed genomic, metabolomic, and immunophenotypic data of HCC and other thyroid cancers. Both mtDNA mutations and profound depletion of citrate pools are common in HCC and other thyroid malignancies, suggesting that thyroid cancers are broadly equipped to survive tricarboxylic acid cycle impairment, whereas metabolites in the reduced form of NADH-dependent lysine degradation pathway were elevated exclusively in HCC. The presence of gLOH was not associated with metabolic phenotypes but rather with reduced immune infiltration, indicating that gLOH confers a selective advantage partially through immunosuppression. Unsupervised multimodal clustering revealed four clusters of HCC with distinct clinical, metabolomic, and microenvironmental phenotypes but overlapping genotypes. These findings chart the metabolic and microenvironmental landscape of HCC and shed light on the interaction between genotype, metabolism, and the microenvironment in cancer.

Construction of Knowledge Graph of 3D Clothing Design Resources Based on Multimodal Clustering Network.

The construction of 3D design model is a hotspot of applied research in the fields of clothing functional design system teaching and display. The simple 3D clothing visualization postprocessing lacks interactive functions, which is a hot issue that needs to be solved urgently at present. Based on analyzing the existing clothing modeling technology, template technology, and fusion technology, and based on the multimodal clustering network theory, this paper proposes a 3D clothing design resource knowledge graph modeling method with multiple fusion of features and templates. The position of each joint point is converted into the coordinate system centered on the torso point in advance and normalized to avoid the problem that the relative position of the camera and the collector cannot be determined, and the shape of different collectors is different. The paper provides a multimodal clustering network intelligence method, illustrates the interoperability of users switching between different design networks in the seamless connection movement, and combines the hybrid intelligence algorithm with the fuzzy logic interpretation algorithm to solve the problems in the field of 3D clothing design service quality. During the simulation process, the research scheme builds a logical multimodal clustering network framework, which integrates compatibility access and global access partition fusion of style templates to achieve information extraction of clothing parts. The experimental results show that the realistic 3D clothing modeling can be achieved by layering the 3D clothing map, contour features, clothing size features, and color texture features with the modeling template. The developed ActiveX control is mounted on MSN, and the system is compatible. The performance and integration rate reached 77.1% and 89.7%, respectively, which effectively strengthened the practical role of the 3D clothing design system.

Meaning construal through multimodal clusters in the theatrical discourse

The paper presents the results of an empirical study of monomodal and multimodal data collected from the plays by Russian playwrights of the second half of the 20th century (A. Vampilov, A. Kazantsev, N. Sadur) and the theatrical performances based on these plays. The main aim is to find out what semiotic modalities are more frequently involved in the process of recurrent semiosis and, as such, in the multimodal meaning construal. Following the classical as well as contemporary works in the field explored, the research sets out to elaborate the cognitive-semiotic approach to the analysis of the theatre-specific means of multimodal meaning construal to get insights into dynamicity, embodiment and figurativity of artistic expression typical of theatrical performativity and inaccessible to other arts. The data are explored by means of a complex methodology that comprises quantitative and qualitative methods with the diagraph analysis as a key analytical tool which serves to recognize meaningful correlations within and across modalities resulting from mappings between dialogically juxtaposed independent utterances. This paper demonstrates that the involvement of nonverbal resources is mainly manifested in the use of hand and head gestures, facial movements and body repositioning. Next to the quantitative results, the research shows that co-occurring conceptualizations through verbal and nonverbal components, gestures in particular, can trigger off the amplification of meaning or bring about the emergence of new conceptual projections.