Multi-view Representation Learning Research Articles

Label distribution-driven multi-view representation learning

In multi-view representation learning (MVRL), the challenge of category uncertainty is significant. Existing methods excel at deriving shared representations across multiple views, but often neglect the uncertainty associated with cluster assignments from each view, thereby leading to increased ambiguity in the category determination. Additionally, methods like kernel-based or neural network-based approaches, while revealing nonlinear relationships, lack attention to category uncertainty. To address these limitations, this paper proposes a method leveraging the uncertainty of label distributions to enhance MVRL. Specifically, our approach combines uncertainty reduction based on label distribution with view representation learning to improve clustering accuracy and robustness. It initially computes the within-view representation of the sample and semantic labels. Then, we introduce a novel constraint based on either variance or information entropy to mitigate class uncertainty, thereby improving the discriminative power of the learned representations. Extensive experiments conducted on diverse multi-view datasets demonstrate that our method consistently outperforms existing approaches, producing more accurate and reliable class assignments. The experimental results highlight the effectiveness of our method in enhancing MVRL by reducing category uncertainty and improving overall classification performance. This method is not only very interpretable but also enhances the model’s ability to learn multi-view consistent information.

SCAE: Structural Contrastive Auto-Encoder for Incomplete Multi-View Representation Learning

Describing an object from multiple perspectives often leads to incomplete data representation. Consequently, learning consistent representations for missing data from multiple views has emerged as a key focus in the realm of Incomplete Multi-view Representation Learning (IMRL). In recent years, various strategies, such as subspace learning, matrix decomposition, and deep learning, have been harnessed to develop numerous IMRL methods. In this article, our primary research revolves around IMRL, with a particular emphasis on addressing two main challenges. Firstly, we delve into the effective integration of intra-view similarity and contextual structure into a unified framework. Secondly, we explore the effective facilitation of information exchange and fusion across multiple views. To tackle these issues, we propose a deep learning approach known as Structural Contrastive Auto-Encoder (SCAE) to solve the challenges of IMRL. SCAE comprises two major components: intra-view structural representation learning and inter-view contrastive representation learning. The former involves capturing intra-view similarity by minimizing the Dirichlet energy of the feature matrix, while also applying spatial dispersion regularization to capture intra-view contextual structure. The latter encourages maximizing the mutual information of inter-view representations, facilitating information exchange and fusion across views. Experimental results demonstrate the efficacy of our approach in significantly enhancing model accuracy and robustly addressing IMRL problems. The code is available at https://github.com/limengran98/SCAE .

Few-Shot Action Recognition via Multi-View Representation Learning

Few-Shot Action Recognition via Multi-View Representation Learning

Efficient Multiview Representation Learning With Correntropy and Anchor Graph

Efficient Multiview Representation Learning With Correntropy and Anchor Graph

Multiview representation learning for identification of novel cancer genes and their causative biological mechanisms.

Tumorigenesis arises from the dysfunction of cancer genes, leading to uncontrolled cell proliferation through various mechanisms. Establishing a complete cancer gene catalogue will make precision oncology possible. Although existing methods based on graph neural networks (GNN) are effective in identifying cancer genes, they fall short in effectively integrating data from multiple views and interpreting predictive outcomes. To address these shortcomings, an interpretable representation learning framework IMVRL-GCN is proposed to capture both shared and specific representations from multiview data, offering significant insights into the identification of cancer genes. Experimental results demonstrate that IMVRL-GCN outperforms state-of-the-art cancer gene identification methods and several baselines. Furthermore, IMVRL-GCN is employed to identify a total of 74 high-confidence novel cancer genes, and multiview data analysis highlights the pivotal roles of shared, mutation-specific, and structure-specific representations in discriminating distinctive cancer genes. Exploration of the mechanisms behind their discriminative capabilities suggests that shared representations are strongly associated with gene functions, while mutation-specific and structure-specific representations are linked to mutagenic propensity and functional synergy, respectively. Finally, our in-depth analyses of these candidates suggest potential insights for individualized treatments: afatinib could counteract many mutation-driven risks, and targeting interactions with cancer gene SRC is a reasonable strategy to mitigate interaction-induced risks for NR3C1, RXRA, HNF4A, and SP1.

Cyclic Refiner: Object-Aware Temporal Representation Learning for Multi-view 3D Detection and Tracking

AbstractWe propose a unified object-aware temporal learning framework for multi-view 3D detection and tracking tasks. Having observed that the efficacy of the temporal fusion strategy in recent multi-view perception methods may be weakened by distractors and background clutters in historical frames, we propose a cyclic learning mechanism to improve the robustness of multi-view representation learning. The essence is constructing a backward bridge to propagate information from model predictions (e.g., object locations and sizes) to image and BEV features, which forms a circle with regular inference. After backward refinement, the responses of target-irrelevant regions in historical frames would be suppressed, decreasing the risk of polluting future frames and improving the object awareness ability of temporal fusion. We further tailor an object-aware association strategy for tracking based on the cyclic learning model. The cyclic learning model not only provides refined features, but also delivers finer clues (e.g., scale level) for tracklet association. The proposed cycle learning method and association module together contribute a novel and unified multi-task framework. Experiments on nuScenes show that the proposed model achieves consistent performance gains over baselines of different designs (i.e., dense query-based BEVFormer, sparse query-based SparseBEV and LSS-based BEVDet4D) on both detection and tracking evaluation. Codes and models will be released.

Variational Distillation for Multi-View Learning.

Information Bottleneck (IB) provides an information-theoretic principle for multi-view learning by revealing the various components contained in each viewpoint. This highlights the necessity to capture their distinct roles to achieve view-invariance and predictive representations but remains under-explored due to the technical intractability of modeling and organizing innumerable mutual information (MI) terms. Recent studies show that sufficiency and consistency play such key roles in multi-view representation learning, and could be preserved via a variational distillation framework. But when it generalizes to arbitrary viewpoints, such strategy fails as the mutual information terms of consistency become complicated. This paper presents Multi-View Variational Distillation (MV 2D), tackling the above limitations for generalized multi-view learning. Uniquely, MV 2D can recognize useful consistent information and prioritize diverse components by their generalization ability. This guides an analytical and scalable solution to achieving both sufficiency and consistency. Additionally, by rigorously reformulating the IB objective, MV 2D tackles the difficulties in MI optimization and fully realizes the theoretical advantages of the information bottleneck principle. We extensively evaluate our model on diverse tasks to verify its effectiveness, where the considerable gains provide key insights into achieving generalized multi-view representations under a rigorous information-theoretic principle.

Uncover spatially informed variations for single-cell spatial transcriptomics with STew.

The recent spatial transcriptomics (ST) technologies have enabled characterization of gene expression patterns and spatial information, advancing our understanding of cell lineages within diseased tissues. Several analytical approaches have been proposed for ST data, but effectively utilizing spatial information to unveil the shared variation with gene expression remains a challenge. We introduce STew, a Spatial Transcriptomic multi-viEW representation learning method, to jointly analyze spatial information and gene expression in a scalable manner, followed by a data-driven statistical framework to measure the goodness of model fit. Through benchmarking using human dorsolateral prefrontal cortex and mouse main olfactory bulb data with true manual annotations, STew achieved superior performance in both clustering accuracy and continuity of identified spatial domains compared with other methods. STew is also robust to generate consistent results insensitive to model parameters, including sparsity constraints. We next applied STew to various ST data acquired from 10× Visium, Slide-seqV2, and 10× Xenium, encompassing single-cell and multi-cellular resolution ST technologies, which revealed spatially informed cell type clusters and biologically meaningful axes. In particular, we identified a proinflammatory fibroblast spatial niche using ST data from psoriatic skins. Moreover, STew scales almost linearly with the number of spatial locations, guaranteeing its applicability to datasets with thousands of spatial locations to capture disease-relevant niches in complex tissues. Source code and the R software tool STew are available from github.com/fanzhanglab/STew.

A multi-view representation learning framework for commonsense knowledge bases

Commonsense knowledge bases play an essential role in a wide range of natural language processing tasks. This paper studies the problem of representation learning for commonsense knowledge bases to effectively incorporate their knowledge into numerical models. Most existing knowledge base representation learning methods are difficult to apply to commonsense knowledge bases since they are much sparser than general knowledge bases. Hence, in this paper, we propose a novel method for commonsense knowledge base representation learning. Specifically, we first model the nodes from multiple views, including word/phase information, context information, and graph information. Then, we design a scoring function to measure whether the commonsense triplets are established through relation representation learning. We conduct extensive experiments on two tasks and the results show that our proposed model outperforms other knowledge base representation learning methods.

Comprehensive Multiview Representation Learning via Deep Autoencoder-Like Nonnegative Matrix Factorization.

Learning a comprehensive representation from multiview data is crucial in many real-world applications. Multiview representation learning (MRL) based on nonnegative matrix factorization (NMF) has been widely adopted by projecting high-dimensional space into a lower order dimensional space with great interpretability. However, most prior NMF-based MRL techniques are shallow models that ignore hierarchical information. Although deep matrix factorization (DMF)-based methods have been proposed recently, most of them only focus on the consistency of multiple views and have cumbersome clustering steps. To address the above issues, in this article, we propose a novel model termed deep autoencoder-like NMF for MRL (DANMF-MRL), which obtains the representation matrix through the deep encoding stage and decodes it back to the original data. In this way, through a DANMF-based framework, we can simultaneously consider the multiview consistency and complementarity, allowing for a more comprehensive representation. We further propose a one-step DANMF-MRL, which learns the latent representation and final clustering labels matrix in a unified framework. In this approach, the two steps can negotiate with each other to fully exploit the latent clustering structure, avoid previous tedious clustering steps, and achieve optimal clustering performance. Furthermore, two efficient iterative optimization algorithms are developed to solve the proposed models both with theoretical convergence analysis. Extensive experiments on five benchmark datasets demonstrate the superiority of our approaches against other state-of-the-art MRL methods.

A Clustering-Guided Contrastive Fusion for Multi-View Representation Learning

A Clustering-Guided Contrastive Fusion for Multi-View Representation Learning

Research of Event Reconstruct Based on Multi-View Contrastive Learning (Student Abstract)

The proliferation of social media exacerbates information fragmentation, posing challenges to understanding public events. We address the problem of event reconstruction with a novel Multi-view Contrast Event Reconstruction (MCER) model. MCER maximizes feature dissimilarity between different views of the same event using contrastive learning, while minimizing mutual information between distinct events. This aggregates fragmented views to reconstruct comprehensive event representations. MCER employs momentum and weight-sharing encoders in a three-tower architecture with supervised contrastive loss for multi-view representation learning. Due to the scarcity of multi-view public datasets, we construct a new Mul-view-data benchmark.Experiments demonstrate MCER’s superior performance on public data and our Mul-view-data, significantly outperforming selfsupervised methods by incorporating supervised contrastive techniques. MCER advances multi-view representation learning to counter information fragmentation and enable robust event understanding.

Incomplete Contrastive Multi-View Clustering with High-Confidence Guiding

Incomplete multi-view clustering becomes an important research problem, since multi-view data with missing values are ubiquitous in real-world applications. Although great efforts have been made for incomplete multi-view clustering, there are still some challenges: 1) most existing methods didn't make full use of multi-view information to deal with missing values; 2) most methods just employ the consistent information within multi-view data but ignore the complementary information; 3) For the existing incomplete multi-view clustering methods, incomplete multi-view representation learning and clustering are treated as independent processes, which leads to performance gap. In this work, we proposed a novel Incomplete Contrastive Multi-View Clustering method with high-confidence guiding (ICMVC). Firstly, we proposed a multi-view consistency relation transfer plus graph convolutional network to tackle missing values problem. Secondly, instance-level attention fusion and high-confidence guiding are proposed to exploit the complementary information while instance-level contrastive learning for latent representation is designed to employ the consistent information. Thirdly, an end-to-end framework is proposed to integrate multi-view missing values handling, multi-view representation learning and clustering assignment for joint optimization. Experiments compared with state-of-the-art approaches demonstrated the effectiveness and superiority of our method. Our code is publicly available at https://github.com/liunian-Jay/ICMVC. The version with supplementary material can be found at http://arxiv.org/abs/2312.08697.

MultiCBR: Multi-view Contrastive Learning for Bundle Recommendation

Bundle recommendation seeks to recommend a bundle of related items to users to improve both user experience and the profits of platform. Existing bundle recommendation models have progressed from capturing only user-bundle interactions to the modeling of multiple relations among users, bundles, and items. CrossCBR, in particular, incorporates cross-view contrastive learning into a two-view preference learning framework, significantly improving SOTA performance. It does, however, have two limitations: (1) the two-view formulation does not fully exploit all the heterogeneous relations among users, bundles, and items; and (2) the “early contrast and late fusion” framework is less effective in capturing user preference and difficult to generalize to multiple views. In this article, we present MultiCBR, a novel Multi -view C ontrastive learning framework for B undle R ecommendation. First, we devise a multi-view representation learning framework capable of capturing all the user-bundle, user-item, and bundle-item relations, especially better utilizing the bundle-item affiliations to enhance sparse bundles’ representations. Second, we innovatively adopt an “early fusion and late contrast” design that first fuses the multi-view representations before performing self-supervised contrastive learning. In comparison to existing approaches, our framework reverses the order of fusion and contrast, introducing the following advantages: (1) Our framework is capable of modeling both cross-view and ego-view preferences, allowing us to achieve enhanced user preference modeling; and (2) instead of requiring quadratic number of cross-view contrastive losses, we only require two self-supervised contrastive losses, resulting in minimal extra costs. Experimental results on three public datasets indicate that our method outperforms SOTA methods. The code and dataset can be found in the github repo https://github.com/HappyPointer/MultiCBR .

Graph neural news recommendation based on multi-view representation learning

Graph neural news recommendation based on multi-view representation learning

Decoupled representation for multi-view learning

Learning multi-view data is a central topic for advanced deep model applications. Existing efforts mainly focus on exploring shared information to maximize the consensus among all the views. However, after reasonably discarding superfluous task-irrelevant noise, the view-specific information is equally essential to downstream tasks. In this paper, we propose to decouple the multi-view representation learning into the shared and specific information extractions with parallel branches, and seamlessly adopt feature fusion in end-to-end models. The common feature is obtained based on the view-agnostic contrastive learning and view-discriminative training to minimize the discrepancy within the views. Simultaneously, the specific feature is learned with orthogonality constraints to minimize the view-level correlation. Besides, the semantic information in the features is reserved with supervised training. After disentangling the representations, we fuse the mutually complementary common and specific features for downstream tasks. Particularly, we provide a theoretical explanation for our method from an information bottleneck perspective. Compared with state-of-the-art multi-view models on benchmark datasets, we empirically demonstrate the advantage of our method in several downstream tasks, such as ordinary classification and few-shot learning. In addition, extensive experiments validate the robustness and transferability of our approach, when applying the learned representation on the source dataset to several target datasets.

Multi-view representation learning for tabular data integration using inter-feature relationships

Objective:An applied problem facing all areas of data science is harmonizing data sources. Joining data from multiple origins with unmapped and only partially overlapping features is a prerequisite to developing and testing robust, generalizable algorithms, especially in healthcare. This integrating is usually resolved using meta-data such as feature names, which may be unavailable or ambiguous. Our goal is to design methods that create a mapping between structured tabular datasets derived from electronic health records independent of meta-data. Methods:We evaluate methods in the challenging case of numeric features without reliable and distinctive univariate summaries, such as nearly Gaussian and binary features. We assume that a small set of features are a priori mapped between two datasets, which share unknown identical features and possibly many unrelated features. Inter-feature relationships are the main source of identification which we expect. We compare the performance of contrastive learning methods for feature representations, novel partial auto-encoders, mutual-information graph optimizers, and simple statistical baselines on simulated data, public datasets, the MIMIC-III medical-record changeover, and perioperative records from before and after a medical-record system change. Performance was evaluated using both mapping of identical features and reconstruction accuracy of examples in the format of the other dataset. Results:Contrastive learning-based methods overall performed the best, often substantially beating the literature baseline in matching and reconstruction, especially in the more challenging real data experiments. Partial auto-encoder methods showed on-par matching with contrastive methods in all synthetic and some real datasets, along with good reconstruction. However, the statistical method we created performed reasonably well in many cases, with much less dependence on hyperparameter tuning. When validating feature match output in the EHR dataset we found that some mistakes were actually a surrogate or related feature as reviewed by two subject matter experts. Conclusion:In simulation studies and real-world examples, we find that inter-feature relationships are effective at identifying matching or closely related features across tabular datasets when meta-data is not available. Decoder architectures are also reasonably effective at imputing features without an exact match.

Autoencoder in Autoencoder Networks.

Modeling complex correlations on multiview data is still challenging, especially for high-dimensional features with possible noise. To address this issue, we propose a novel unsupervised multiview representation learning (UMRL) algorithm, termed autoencoder in autoencoder networks (AE2-Nets). The proposed framework effectively encodes information from high-dimensional heterogeneous data into a compact and informative representation with the proposed bidirectional encoding strategy. Specifically, the proposed AE2-Nets conduct encoding in two directions: the inner-AE-networks extract view-specific intrinsic information (forward encoding), while the outer-AE-networks integrate this view-specific intrinsic information from different views into a latent representation (backward encoding). For the nested architecture, we further provide a probabilistic explanation and extension from hierarchical variational autoencoder. The forward-backward strategy flexibly addresses high-dimensional (noisy) features within each view and encodes complementarity across multiple views in a unified framework. Extensive results on benchmark datasets validate the advantages compared to the state-of-the-art algorithms.

Effective binning of metagenomic contigs using contrastive multi-view representation learning

Contig binning plays a crucial role in metagenomic data analysis by grouping contigs from the same or closely related genomes. However, existing binning methods face challenges in practical applications due to the diversity of data types and the difficulties in efficiently integrating heterogeneous information. Here, we introduce COMEBin, a binning method based on contrastive multi-view representation learning. COMEBin utilizes data augmentation to generate multiple fragments (views) of each contig and obtains high-quality embeddings of heterogeneous features (sequence coverage and k-mer distribution) through contrastive learning. Experimental results on multiple simulated and real datasets demonstrate that COMEBin outperforms state-of-the-art binning methods, particularly in recovering near-complete genomes from real environmental samples. COMEBin outperforms other binning methods remarkably when integrated into metagenomic analysis pipelines, including the recovery of potentially pathogenic antibiotic-resistant bacteria (PARB) and moderate or higher quality bins containing potential biosynthetic gene clusters (BGCs).

Multiview Representation Learning With One-to-Many Dynamic Relationships.

Integrating information from multiple views to obtain potential representations with stronger expressive ability has received significant attention in practical applications. Most existing algorithms usually focus on learning either the consistent or complementary representation of views and, subsequently, integrate one-to-one corresponding sample representations between views. Although these approaches yield effective results, they do not fully exploit the information available from multiple views, limiting the potential for further performance improvement. In this article, we propose an unsupervised multiview representation learning method based on sample relationships, which enables the one-to-many fusion of intraview and interview information. Due to the heterogeneity of views, we need mainly face the two following challenges: 1) the discrepancy in the dimensions of data across different views and 2) the characterization and utilization of sample relationships across these views. To address these two issues, we adopt two modules: the dimension consistency relationship enhancement module and the multiview graph learning module. Thereinto, the relationship enhancement module addresses the discrepancy in data dimensions across different views and dynamically selects data dimensions for each sample that bolsters intraview relationships. The multiview graph learning module devises a novel multiview adjacency matrix to capture both intraview and interview sample relationships. To achieve one-to-many fusion and obtain multiview representations, we employ the graph autoencoder structure. Furthermore, we extend the proposed architecture to the supervised case. We conduct extensive experiments on various real-world multiview datasets, focusing on clustering and multilabel classification tasks, to evaluate the effectiveness of our method. The results demonstrate that our approach significantly improves performance compared to existing methods, highlighting the potential of leveraging sample relationships for multiview representation learning. Our code is released at https://github.com/ lilidan-orm/one-to-many-multiview on GitHub.