Multi-view Representation Learning Research Articles

Dual Contrastive Prediction for Incomplete Multi-View Representation Learning.

In this article, we propose a unified framework to solve the following two challenging problems in incomplete multi-view representation learning: i) how to learn a consistent representation unifying different views, and ii) how to recover the missing views. To address the challenges, we provide an information theoretical framework under which the consistency learning and data recovery are treated as a whole. With the theoretical framework, we propose a novel objective function which jointly solves the aforementioned two problems and achieves a provable sufficient and minimal representation. In detail, the consistency learning is performed by maximizing the mutual information of different views through contrastive learning, and the missing views are recovered by minimizing the conditional entropy through dual prediction. To the best of our knowledge, this is one of the first works to theoretically unify the cross-view consistency learning and data recovery for representation learning. Extensive experimental results show that the proposed method remarkably outperforms 20 competitive multi-view learning methods on six datasets in terms of clustering, classification, and human action recognition. The code could be accessed from https://pengxi.me.

CoSleep: A Multi-View Representation Learning Framework for Self-Supervised Learning of Sleep Stage Classification

Sleep stage classification is critical for diagnosing sleep quality. While deep neural networks are becoming popular for automatic sleep stage classification with supervised learning, large-scale labeled datasets are still hard to acquire. Recently, self-supervised learning (SSL) has become one of the most prominent approaches to alleviate the burden of labeling works. However, existing SSL methods are mainly designed for non-temporally correlated data and only learning representations from an instance level. Hence, the objective of this paper is to learn robust and generalizable representations for physiological signals with self-supervised learning. Specifically, we make the following contributions: (1) we propose a novel co-training scheme by exploiting complementary information from multiple views (time view and frequency view) of physiological signals to mine more positive samples, which overcomes the drawback of popular InfoNCE and achieves semantic-level representation learning; (2) we extend our framework with a memory module, implemented by a queue and a moving-averaged encoder, to enlarge the pool of negative candidates and keep the up-to-date representation; (3) extensive experiments conducted on sleep stage classification demonstrate state-of-the-art performance compared with SSL baselines, achieving 71.6% and 57.9% accuracies on two sleep datasets, SleepEDF and ISRUC respectively. The code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/larryshaw0079/CoSleep</uri> .

Letting Logos Speak: Leveraging Multiview Representation Learning for Data-Driven Branding and Logo Design

The authors develop a decision support system for design and branding based on a multimodal variational autoencoder that merges image, text, and ratings data.

Co-embedding: a semi-supervised multi-view representation learning approach

Co-embedding: a semi-supervised multi-view representation learning approach

Local2Global: Unsupervised multi-view deep graph representation learning with Nearest Neighbor Constraint

Multi-view feature fusion is a vital phase for multi-view representation learning. Recently, most Graph Auto-Encoders (GAEs) and their variants focus on multi-view learning. However, most of them ignore deep representation fusion of features of each multi-view. Furthermore, there are scarcely unsupervised constraints guiding to enhance the graph representation capability in training process. In this paper, we propose a novel unsupervised Multi-view Deep Graph Representation Learning (MDGRL) framework on multi-view data which is based on the Graph Auto-Encoders (GAEs) for local feature leaning, a feature fusion module for producing global representation and a valid variant of Variational Graph Auto-Encoder (VGAE) for global deep graph representation learning. To fuse Nearest Neighbor Constraint (NNC) between the maximal degree nodes which represents the most close joining node and their adjacent nodes into VGAE, we propose a new Nearest Neighbor Constraint Variational Graph Auto-Encoder (NNC-VGAE) to enhance the global deep graph representation capability for multi-view data. In the training process of NNC-VGAE, NNC makes the adjacent nodes gradually close to the maximal degree node. Hence, the proposed MDGRL has excellent deep graph representation capability for multi-view data. Experiments on eight non-medical benchmark multi-view data sets and four medical data sets confirm the effectiveness of our MDGRL compared with other state-of-the-art methods for unsupervised clustering.

Learning unsupervised node representation from multi-view network

This paper studies the problem of learning node representations for networks with multiple views, which aims to infer robust node representations by simultaneously considering multiple views during the representations learning process. We propose an effective method for this task, named as Multi-View Representation Learning (MVRL). The new method extends the matrix factorization model for node representation learning of multi-view network in unsupervised representation learning scenario, which simultaneously learns a set of view weights to identify the quality of each view, and the network representations as matrix factorization of the weighted combination of multiple views. An efficient optimization method with linear complexity is proposed to solve the new model, and a simple yet efficient method is proposed for fast updating of the new node’s vector representation without updating the whole nodes’ representation vectors. We have evaluated the performance of our proposed approach on five real-world multi-view network datasets. Experimental results on the node classification task demonstrated the superior performance and efficiency of our proposed method.

Patent2Vec: Multi-view representation learning on patent-graphs for patent classification

Patent2Vec: Multi-view representation learning on patent-graphs for patent classification

Variational graph autoencoders for multiview canonical correlation analysis

We present a novel approach for multiview canonical correlation analysis based on a variational graph neural network model. We propose a nonlinear model which takes into account the available graph-based geometric constraints while being scalable to large-scale datasets with multiple views. This model combines the probabilistic interpretation of CCA with an autoencoder architecture based on graph convolutional neural network layers. Experiments with the proposed method are conducted on classification, clustering, and recommendation tasks on real datasets. The algorithm is competitive with state-of-the-art multiview representation learning techniques, in addition to being scalable and robust to instances with missing views.

Analysis of Twitter Users' Lifestyle Choices using Joint Embedding Model

Multiview representation learning of data can help construct coherent and contextualized users' representations on social media. This paper suggests a joint embedding model, incorporating users' social and textual information to learn contextualized user representations used for understanding their lifestyle choices. We apply our model to tweets related to two lifestyle activities, `Yoga' and `Keto diet' and use it to analyze users' activity type and motivation. We explain the data collection and annotation process in detail and provide an in-depth analysis of users from different classes based on their Twitter content. Our experiments show that our model results in performance improvements in both domains.

Multi-View Representation Learning with Manifold Smoothness

Multi-view representation learning attempts to learn a representation from multiple views and most existing methods are unsupervised. However, representation learned only from unlabeled data may not be discriminative enough for further applications (e.g., clustering and classification). For this reason, semi-supervised methods which could use unlabeled data along with the labeled data for multi-view representation learning need to be developed. Manifold information plays an important role in semi-supervised learning, but it has not been considered for multi-view representation learning. In this paper, we introduce the manifold smoothness into multi-view representation learning and propose MvDGAT which learns the representation and the intrinsic manifold simultaneously with graph attention network. Experiments conducted on real-world datasets reveal that our MvDGAT can achieve better performance than state-of-the-art methods.

Multi-View Information-Bottleneck Representation Learning

In real-world applications, clustering or classification can usually be improved by fusing information from different views. Therefore, unsupervised representation learning on multi-view data becomes a compelling topic in machine learning. In this paper, we propose a novel and flexible unsupervised multi-view representation learning model termed Collaborative Multi-View Information Bottleneck Networks (CMIB-Nets), which comprehensively explores the common latent structure and the view-specific intrinsic information, and discards the superfluous information in the data significantly improving the generalization capability of the model. Specifically, our proposed model relies on the information bottleneck principle to integrate the shared representation among different views and the view-specific representation of each view, prompting the multi-view complete representation and flexibly balancing the complementarity and consistency among multiple views. We conduct extensive experiments (including clustering analysis, robustness experiment, and ablation study) on real-world datasets, which empirically show promising generalization ability and robustness compared to state-of-the-arts.

A Survey on Multi-View Clustering.

Clustering is a machine learning paradigm of dividing sample subjects into a number of groups such that subjects in the same groups are more similar to those in other groups. With advances in information acquisition technologies, samples can frequently be viewed from different angles or in different modalities, generating multi-view data. Multi-view clustering, that clusters subjects into subgroups using multi-view data, has attracted more and more attentions. Although MVC methods have been developed rapidly, there has not been enough survey to summarize and analyze the current progress. Therefore, we propose a novel taxonomy of the MVC approaches. Similar to other machine learning methods, we categorize them into generative and discriminative classes. In discriminative class, based on the way of view integration, we split it further into five groups: Common Eigenvector Matrix, Common Coefficient Matrix, Common Indicator Matrix, Direct Combination and Combination After Projection. Furthermore, we relate MVC to other topics: multi-view representation, ensemble clustering, multi-task clustering, multi-view supervised and semi-supervised learning. Several representative real-world applications are elaborated for practitioners. Some benchmark multi-view datasets are introduced and representative MVC algorithms from each group are empirically evaluated to analyze how they perform on benchmark datasets. To promote future development of MVC approaches, we point out several open problems that may require further investigation and thorough examination.

Multiview Concept Learning Via Deep Matrix Factorization.

Multiview representation learning (MVRL) leverages information from multiple views to obtain a common representation summarizing the consistency and complementarity in multiview data. Most previous matrix factorization-based MVRL methods are shallow models that neglect the complex hierarchical information. The recently proposed deep multiview factorization models cannot explicitly capture consistency and complementarity in multiview data. We present the deep multiview concept learning (DMCL) method, which hierarchically factorizes the multiview data, and tries to explicitly model consistent and complementary information and capture semantic structures at the highest abstraction level. We explore two variants of the DMCL framework, DMCL-L and DMCL-N, with respectively linear/nonlinear transformations between adjacent layers. We propose two block coordinate descent-based optimization methods for DMCL-L and DMCL-N. We verify the effectiveness of DMCL on three real-world data sets for both clustering and classification tasks.

Multi-view representation learning and understanding

Multi-view representation learning and understanding

Multi-View Representation Learning via Dual Optimal Transportation

Recently, multi-view representation learning has gained rapid growth in various fields. Most of previous multi-view learning methods rely on strong notions of distances that often provide no useful gradients in deep network training, which greatly degrades the performance in merging complementary information of views. To address this challenge, a multi-view representation learning network with dual optimal transportation (MDOT-Net) is proposed to capture fusion representations embedded in a common manifold with the weak topology. In MDOT-Net, the multi-view representation learning is modelled as an optimal transportation (OT) problem in manifold fitting, which is further decomposed into the intra-view OT and the inter-view OT. The intra-view OT is implemented by a view-specific adversarial variational network, which accurately captures local manifold structures within views by leveraging fusion knowledge. The inter-view OT is implemented by a view-fusion adversarial inference network, which models fusion representations compatible with diversities of sub-manifolds by utilizing view-specific knowledge. The two OTs boost mutually by providing prior knowledge to each other in multi-view representation learning. Finally, numerous experiments are conducted on four benchmark datasets, and the results demonstrate MDOT-Net achieves the state-of-the-art performance.

Deep Spectral Representation Learning From Multi-View Data.

Multi-view representation learning (MvRL) aims to learn a consensus representation from diverse sources or domains to facilitate downstream tasks such as clustering, retrieval, and classification. Due to the limited representative capacity of the adopted shallow models, most existing MvRL methods may yield unsatisfactory results, especially when the labels of data are unavailable. To enjoy the representative capacity of deep learning, this paper proposes a novel multi-view unsupervised representation learning method, termed as Multi-view Laplacian Network (MvLNet), which could be the first deep version of the multi-view spectral representation learning method. Note that, such an attempt is nontrivial because simply combining Laplacian embedding (i.e., spectral representation) with neural networks will lead to trivial solutions. To solve this problem, MvLNet enforces an orthogonal constraint and reformulates it as a layer with the help of Cholesky decomposition. The orthogonal layer is stacked on the embedding network so that a common space could be learned for consensus representation. Compared with numerous recent-proposed approaches, extensive experiments on seven challenging datasets demonstrate the effectiveness of our method in three multi-view tasks including clustering, recognition, and retrieval. The source code could be found at www.pengxi.me.

Merging Omnichannel Behavioral Data: A Similarity-based Multi-view Representation Learning Approach

Merging Omnichannel Behavioral Data: A Similarity-based Multi-view Representation Learning Approach

Multi-view Low-rank Preserving Embedding: A novel method for multi-view representation

In recent years, we have witnessed a surge of interest in multi-view representation learning. When facing multiple views that are highly related but sightly different from each other, most existing multi-view methods might fail to fully explore multi-view information. Additionally, pairwise correlations among multiple views often vary drastically, which makes multi-view representation challenging. Therefore, how to learn appropriate representation from multi-view information is still an open but challenging problem. To handle this issue, this paper proposes a novel multi-view learning method, named Multi-view Low-rank Preserving Embedding (MvLPE). It integrates all views into a common latent space, termed as centroid view, by minimizing the disagreement between centroid view and each view, which encourages different views to mutually learn from each other. Unlike existing methods with explicit weight definition, the proposed method could automatically allocate an ideal weight for each view according to its contribution. Besides, MvLPE could maintain its low-rank reconstruction structure for each view while integrating all views into centroid view. Since there is no closed-form solution for MvLPE, an effective algorithm based on iterative alternating strategy is provided to obtain the solution. The experiments on six benchmark datasets validate the effectiveness of the proposed method, which achieves superior performance over its counterparts.

Fusion of heterogeneous attention mechanisms in multi-view convolutional neural network for text classification

The rapid proliferation of user generated content has given rise to large volumes of text corpora. Increasingly, scholars, researchers, and organizations employ text classification to mine novel insights for high-impact applications. Despite their prevalence, conventional text classification methods rely on labor-intensive feature engineering efforts that are task specific, omit long-term relationships, and are not suitable for the rapidly evolving domains. While an increasing body of deep learning and attention mechanism literature aim to address these issues, extant methods often represent text as a single view and omit multiple sets of features at varying levels of granularity. Recognizing that these issues often result in performance degradations, we propose a novel Spatial View Attention Convolutional Neural Network (SVA-CNN). SVA-CNN leverages an innovative and carefully designed set of multi-view representation learning, a combination of heterogeneous attention mechanisms and CNN-based operations to automatically extract and weight multiple granularities and fine-grained representations. Rigorously evaluating SVA-CNN against prevailing text classification methods on five large-scale benchmark datasets indicates its ability to outperform extant deep learning-based classification methods in both performance and training time for document classification, sentiment analysis, and thematic identification applications. To facilitate model reproducibility and extensions, SVA-CNN’s source code is also available via GitHub.

Multi-view self-attention for interpretable drug-target interaction prediction.

The drug discovery stage is a vital aspect of the drug development process and forms part of the initial stages of the development pipeline. In recent times, machine learning-based methods are actively being used to model drug-target interactions for rational drug discovery due to the successful application of these methods in other domains. In machine learning approaches, the numerical representation of molecules is critical to the performance of the model. While significant progress has been made in molecular representation engineering, this has resulted in several descriptors for both targets and compounds. Also, the interpretability of model predictions is a vital feature that could have several pharmacological applications. In this study, we propose a self-attention-based multi-view representation learning approach for modeling drug-target interactions. We evaluated our approach using three benchmark kinase datasets and compared the proposed method to some baseline models. Our experimental results demonstrate the ability of our method to achieve competitive prediction performance and offer biologically plausible drug-target interaction interpretations.