Multi-view Learning Framework Research Articles

Joint Learning of Latent Similarity and Local Embedding for Multi-View Clustering.

Spectral clustering has been an attractive topic in the field of computer vision due to the extensive growth of applications, such as image segmentation, clustering and representation. In this problem, the construction of the similarity matrix is a vital element affecting clustering performance. In this paper, we propose a multi-view joint learning (MVJL) framework to achieve both a reliable similarity matrix and a latent low-dimensional embedding. Specifically, the similarity matrix to be learned is represented as a convex hull of similarity matrices from different views, where the nuclear norm is imposed to capture the principal information of multiple views and improve robustness against noise/outliers. Moreover, an effective low-dimensional representation is obtained by applying local embedding on the similarity matrix, which preserves the local intrinsic structure of data through dimensionality reduction. With these techniques, we formulate the MVJL as a joint optimization problem and derive its mathematical solution with the alternating direction method of multipliers strategy and the proximal gradient descent method. The solution, which consists of a similarity matrix and a low-dimensional representation, is ultimately integrated with spectral clustering or K-means for multi-view clustering. Extensive experimental results on real-world datasets demonstrate that MVJL achieves superior clustering performance over other state-of-the-art methods.

MLDH-Fold: Protein fold recognition based on multi-view low-rank modeling

Protein fold recognition is critical for understanding the molecular functions of proteins and drug design. Computational predictors have been proposed to identify protein into one of the known folds based only on the protein sequence information. However, how to combine different features to improve predictive performance remains a challenging problem. In this study, two novel methods (MVLR and MLDH-Fold) were proposed for protein fold recognition. We proposed a novel multi-view learning framework to combine the different views of protein sequences. Each view represents the similarity scores between the target sequences and template sequences calculated by the threading method. The proposed method extracts the low-rank principal features to precisely represent the similarity scores of each view and constructs the latent subspace with the common information of different views to predict the target proteins. Furthermore, we proposed an ensemble method called MLDH-Fold to combine the MVLR with the template-based methods. Predictive results on the two widely used datasets (LE and YK) show that the proposed computational methods outperform other computational predictors, indicating that the MVLR and MLDH-Fold are useful tools for protein fold recognition.

Deep Multiview Learning to Identify Population Structure with Multimodal Imaging.

We present an effective deep multiview learning framework to identify population structure using multimodal imaging data. Our approach is based on canonical correlation analysis (CCA). We propose to use deep generalized CCA (DGCCA) to learn a shared latent representation of non-linearly mapped and maximally correlated components from multiple imaging modalities with reduced dimensionality. In our empirical study, this representation is shown to effectively capture more variance in original data than conventional generalized CCA (GCCA) which applies only linear transformation to the multi-view data. Furthermore, subsequent cluster analysis on the new feature set learned from DGCCA is able to identify a promising population structure in an Alzheimer's disease (AD) cohort. Genetic association analyses of the clustering results demonstrate that the shared representation learned from DGCCA yields a population structure with a stronger genetic basis than several competing feature learning methods.

Multi-view generalized support vector machine via mining the inherent relationship between views with applications to face and fire smoke recognition

Multiview Generalized Eigenvalue Proximal Support Vector Machines (MvGSVMs) is an effective multi-view classification algorithm, which effectively combines multi-view learning and classification. Then it was found that in the classification learning task, the classifier combined with multi-view learning has a better classification effect than considering only a single view. In order to utilize the multi-view learning framework more fully and accurately, we further research this. We explore the internal relationship between different views between samples to replace the method of connecting different views through distance combinations. We propose a new method named Multi-view Generalized Support Vector Machine via Mining the Inherent Relationship between Views (MRMvGSVM). At the same time, we use the L2,1-norm constraint relationship matrix as a multi-view regularization term to select the most relevant sample data from different views. It not only helps to improve the accuracy of classification but also reduces the influence of extraneous factors to a certain extent and improves the robustness of the algorithm. The effectiveness of the algorithm is proved by theory and experiments on UCI, and Face and Fire Smoke image datasets.

Robust kernelized graph-based learning

The studies of hidden complex structures in data have popularized the use of graph-based learning methods in semi-supervised and unsupervised learning tasks. Kernelized graph-based methods are proven to perform better, but these methods suffer from the issue of appropriate kernel selection. Instead of using multiple views, these methods generally use a single view. But multi-view methods need a proper weight assignment technique to each view in proportion to their contribution to the learning task. To solve this, a novel Self-weighted Multi-view Multiple Kernel Learning (SMVMKL) framework is proposed using multiple kernels on multiple views that automatically assigns appropriate weight to each kernel of each view without introducing an additional parameter. But the real-world data that is either noisy or corrupt with outliers which may effect the performance of the proposed SMVMKL method. To deal with this, a Robust Self-weighted Multi-view Multiple Kernel Learning (RSMVMKL) framework using the l2,1-norm has also been proposed that reduces the effect of outliers present in the data set. Both the proposed methods have been evaluated on multiple benchmark data sets and result in a performance comparable with the other state-of-the-art multi-view methods considered in this paper.

Deep Multi-view Representation Learning for Video Anomaly Detection Using Spatiotemporal Autoencoders

Visual perception is a transformative technology that can recognize patterns from environments through visual inputs. Automatic surveillance of human activities has gained significant importance in both public and private spaces. It is often difficult to understand the complex dynamics of events in real-time scenarios due to camera movements, cluttered backgrounds, and occlusion. Existing anomaly detection systems are not efficient because of high intra-class variations and inter-class similarities existing among activities. Hence, there is a demand to explore different kinds of information extracted from surveillance videos to improve overall performance. This can be achieved by learning features from multiple forms (views) of the given raw input data. We propose two novel methods based on the multi-view representation learning framework. The first approach is a hybrid multi-view representation learning that combines deep features extracted from 3D spatiotemporal autoencoder (3D-STAE) and robust handcrafted features based on spatiotemporal autocorrelation of gradients. The second approach is a deep multi-view representation learning that combines deep features extracted from two-stream STAEs to detect anomalies. Results on three standard benchmark datasets, namely Avenue, Live Videos, and BEHAVE, show that the proposed multi-view representations modeled with one-class SVM perform significantly better than most of the recent state-of-the-art methods.

A novel deep quality-aware CNN for image edge smoothening

Image-edge smoothing is a practical technique in many image-editing implementations such as Visual Aesthetic Enhancement, Action Movie Production, and Privacy Protection. In the literature, there have been many well-adapted smoothing techniques used for image edges such as Gaussian and Bilateral Blurred utilizing visual semantics perceived by the human visual system. However, it is observed that the previously employed image-edge smoothing techniques are not both content-aware and semantically aware. In other words, the image edges are not smoothsimple and efficient, its performance might be less satisfactory. We tend to smooth semantically non-important regions while smoothing semantically important regions slightly in practice. The motivation behind this is that we propose a deep architecture for semantically aware image-edge smoothing. Hence, provided massive image patches extracted from highly aesthetic images crawled from Flickr, we extract multimodal visual features to characterize each. Then, these multimodal features are optimally combined via a multi-view learning framework. Hence, we employ an active learning framework to extract the visually attractive image patches from each image. Thus, we train a deep CNN to find out the edge attributes of actively selected image patches. Utilizing the trained CNN feature of each image patch, we construct a Gaussian mixture model to conduct image-edge smoothing. To verify the proposed method, extensive experimental runs on images with different styles are employed to present the attractiveness of image edges processed by the proposed method. Besides, the efficiency of the proposed method is compared to the well-known edge smoothing algorithms.

Few-shot Food Recognition via Multi-view Representation Learning

This article considers the problem of few-shot learning for food recognition. Automatic food recognition can support various applications, e.g., dietary assessment and food journaling. Most existing works focus on food recognition with large numbers of labelled samples, and fail to recognize food categories with few samples. To address this problem, we propose a Multi-View Few-Shot Learning (MVFSL) framework to explore additional ingredient information for few-shot food recognition. Besides category-oriented deep visual features, we introduce ingredient-supervised deep network to extract ingredient-oriented features. As general and intermediate attributes of food, ingredient-oriented features are informative and complementary to category-oriented features, and thus they play an important role in improving food recognition. Particularly in few-shot food recognition, ingredient information can bridge the gap between disjoint training categories and test categories. To take advantage of ingredient information, we fuse these two kinds of features by first combining their feature maps from their respective deep networks and then convolving combined feature maps. Such convolution is further incorporated into a multi-view relation network, which is capable of comparing pairwise images to enable fine-grained feature learning. MVFSL is trained in an end-to-end fashion for joint optimization on two types of feature learning subnetworks and relation subnetworks. Extensive experiments on different food datasets have consistently demonstrated the advantage of MVFSL in multi-view feature fusion. Furthermore, we extend another two types of networks, namely, Siamese Network and Matching Network, by introducing ingredient information for few-shot food recognition. Experimental results have also shown that introducing ingredient information into these two networks can improve the performance of few-shot food recognition.

Hashtag our stories: Hashtag recommendation for micro-videos via harnessing multiple modalities

Due to the short attention span and instant gratification phenomenon, micro-videos are growing exponentially while gaining more and more concerns. Yet the sheer number of micro-videos leads to severe information overload issues, making it difficult for users to identify their desired micro-videos. The hashtag, mainly utilized in the domain of the microblog or the image, is the indicator or the core idea of the target content and can be applied to various information retrieval scenarios (e.g., search, browse, and categorization). So far, however, little attention has been paid to perform the hashtag recommendation for micro-videos via harnessing multiple modalities.In this article, we devise a neural network-based solution, LOGO (short for “muLti-mOdal-based hashtaG recOmmendation”), to recommend hashtags for micro-videos by utilizing multiple modalities. The proposed LOGO approach first represents each modality as the combination of sequential units in it, weighted by the attention mechanism. In this way, the sequential and attentive features are captured simultaneously. After that, the LOGO integrates the representations of all modalities via a multi-view representation learning framework, which projects the representations into a common space under the restriction of the modality similarity. Ultimately, the LOGO feed the projections of three modalities in the common space and the embeddings of hashtags into a customized neural collaborative filtering framework to perform the hashtag recommendation. Extensive experiments on the scope of both overall performance comparison and micro-level analyses have well-justified the effectiveness and rationality of our proposed approach.

Multiview learning with variational mixtures of Gaussian processes

Gaussian processes (GPs) are powerful Bayesian nonparametric tools widely used in probabilistic modeling, and the mixture of GPs (MGPs) were introduced afterwards to make data modeling more flexible. However, MGPs are not directly applicable to multiview learning. In order to improve the modeling ability of MGPs, in this paper, we propose a new framework of multiview learning for the MGPs and instantiate it for classification. We make the divergence between views as small as possible while ensuring that the posterior probability of each view is as large as possible. Specifically, we regularize the posterior distribution of latent variables with the consistency of posterior distributions of the latent functions between different views. Since it is intractable to solve the model analytically, the variational inference and optimization algorithms of the classification model are also presented in this paper. Experimental results on multiple real-world datasets have shown that the proposed method has outperformed the original MGP model and several state-of-the-art multiview learning methods, which indicate the effectiveness of the proposed multiview learning framework for MGPs.

Protein Fold Recognition Based on Auto-Weighted Multi-View Graph Embedding Learning Model.

Protein fold recognition is critical for studies of the protein structure prediction and drug design. Several methods have been proposed to obtain discriminative features from the protein sequences for fold recognition. However, the ensemble methods that combine the various features to improve predictive performance remain the challenge problems. In this study, we proposed two novel algorithms: AWMG and EMfold. AWMG used a novel predictor based on the multi-view learning framework for fold recognition. Each view was treated as the intermediate representation of the corresponding data source of proteins, including the evolutionary information and the retrieval information. AWMG calculated the auto-weight for each view respectively and constructed the latent subspace which contains the common information shared by different views. The marginalized constraint was employed to enlarge the margins between different folds, improving the predictive performance of AWMG. Furthermore, we proposed a novel ensemble method called EMfold, which combines two complementary methods AWMG and DeepSS. The later method was a template-based algorithm using the SPARKS-X and DeepFR programs. EMfold integrated the advantages of template-based assignment and machine learning classifier. Experimental results on the two widely datasets (LE and YK) showed that the proposed methods outperformed some state-of-the-art methods, indicating that AWMG and EMfold are useful tools for protein fold recognition.

Deep Embedded Complementary and Interactive Information for Multi-View Classification

Multi-view classification optimally integrates various features from different views to improve classification tasks. Though most of the existing works demonstrate promising performance in various computer vision applications, we observe that they can be further improved by sufficiently utilizing complementary view-specific information, deep interactive information between different views, and the strategy of fusing various views. In this work, we propose a novel multi-view learning framework that seamlessly embeds various view-specific information and deep interactive information and introduces a novel multi-view fusion strategy to make a joint decision during the optimization for classification. Specifically, we utilize different deep neural networks to learn multiple view-specific representations, and model deep interactive information through a shared interactive network using the cross-correlations between attributes of these representations. After that, we adaptively integrate multiple neural networks by flexibly tuning the power exponent of weight, which not only avoids the trivial solution of weight but also provides a new approach to fuse outputs from different deterministic neural networks. Extensive experiments on several public datasets demonstrate the rationality and effectiveness of our method.

The similarity-consensus regularized multi-view learning for dimension reduction

During the last decades, learning a low-dimensional space with discriminative information for dimension reduction (DR) has gained a surge of interest. However, it is not accessible for these DR methods to achieve satisfactory performance when dealing with the features from multiple views. For multi-view learning problems, one instance can be represented by multiple heterogeneous features, which are highly relevant but sometimes look different from each other. In addition, correlations between features from multiple views always vary greatly, which challenges the capability of multi-view learning methods. Consequently, constructing a multi-view learning framework with generalization and scalability, which could take advantage of multi-view information as much as possible, is extremely necessary but challenging. To implement the above target, this paper proposes a novel multi-view learning framework based on similarity consensus, which makes full use of correlations among multi-view features while considering the scalability and robustness of the framework. It aims to straightforwardly extend those existing DR methods into multi-view learning domain by preserving the similarity consensus between different views to capture the low-dimensional embedding. Two schemes based on pairwise-consensus and centroid-consensus are separately proposed to force multiple views to learn from each other, then an iterative alternating strategy is developed to obtain the optimal solution. The proposed method is evaluated on 5 benchmark datasets and comprehensive experiments show that our proposed multi-view framework can yield comparable and promising performance with some famous methods.

Hierarchical Multi-View Semi-Supervised Learning for Very High-Resolution Remote Sensing Image Classification

Traditional classification methods used for very high-resolution (VHR) remote sensing images require a large number of labeled samples to obtain higher classification accuracy. Labeled samples are difficult to obtain and costly. Therefore, semi-supervised learning becomes an effective paradigm that combines the labeled and unlabeled samples for classification. In semi-supervised learning, the key issue is to enlarge the training set by selecting highly-reliable unlabeled samples. Observing the samples from multiple views is helpful to improving the accuracy of label prediction for unlabeled samples. Hence, the reasonable view partition is very important for improving the classification performance. In this paper, a hierarchical multi-view semi-supervised learning framework with CNNs (HMVSSL) is proposed for VHR remote sensing image classification. Firstly, a superpixel-based sample enlargement method is proposed to increase the number of training samples in each view. Secondly, a view partition method is designed to partition the training set into two independent views, and the partitioned subsets are characterized by being inter-distinctive and intra-compact. Finally, a collaborative classification strategy is proposed for the final classification. Experiments are conducted on three VHR remote sensing images, and the results show that the proposed method performs better than several state-of-the-art methods.

DeepMV

Recently, significant efforts are made to explore device-free human activity recognition techniques that utilize the information collected by existing indoor wireless infrastructures without the need for the monitored subject to carry a dedicated device. Most of the existing work, however, focuses their attention on the analysis of the signal received by a single device. In practice, there are usually multiple devices "observing" the same subject. Each of these devices can be regarded as an information source and provides us an unique "view" of the observed subject. Intuitively, if we can combine the complementary information carried by the multiple views, we will be able to improve the activity recognition accuracy. Towards this end, we propose DeepMV, a unified multi-view deep learning framework, to learn informative representations of heterogeneous device-free data. DeepMV can combine different views' information weighted by the quality of their data and extract commonness shared across different environments to improve the recognition performance. To evaluate the proposed DeepMV model, we set up a testbed using commercialized WiFi and acoustic devices. Experiment results show that DeepMV can effectively recognize activities and outperform the state-of-the-art human activity recognition methods.

Deep Collaborative Multi-view Hashing for Large-scale Image Search.

Hashing could significantly accelerate large-scale image search by transforming the high-dimensional features into binary Hamming space, where efficient similarity search can be achieved with very fast Hamming distance computation and extremely low storage cost. As an important branch of hashing methods, multi-view hashing takes advantages of multiple features from different views for binary hash learning. However, existing multi-view hashing methods are either based on shallow models which fail to fully capture the intrinsic correlations of heterogeneous views, or unsupervised deep models which suffer from insufficient semantics and cannot effectively exploit the complementarity of view features. In this paper, we propose a novel Deep Collaborative Multi-view Hashing (DCMVH) method to deeply fuse multi-view features and learn multi-view hash codes collaboratively under a deep architecture. DCMVH is a new deep multi-view hash learning framework. It mainly consists of 1) multiple view-specific networks to extract hidden representations of different views, and 2) a fusion network to learn multi-view fused hash code. DCMVH associates different layers with instance-wise and pair-wise semantic labels respectively. In this way, the discriminative capability of representation layers can be progressively enhanced and meanwhile the complementarity of different view features can be exploited effectively. Finally, we develop a fast discrete hash optimization method based on augmented Lagrangian multiplier to efficiently solve the binary hash codes. Experiments on public multi-view image search datasets demonstrate our approach achieves substantial performance improvement over state-of-the-art methods.

Variable star classification using multiview metric learning

ABSTRACTComprehensive observations of variable stars can include time domain photometry in a multitude of filters, spectroscopy, estimates of colour (e.g. U-B), etc. When the objective is to classify variable stars, traditional machine learning techniques distill these various representations (or views) into a single feature vector and attempt to discriminate among desired categories. In this work, we propose an alternative approach that inherently leverages multiple views of the same variable star. Our multiview metric learning framework enables robust characterization of star categories by directly learning to discriminate in a multifaceted feature space, thus, eliminating the need to combine feature representations prior to fitting the machine learning model. We also demonstrate how to extend standard multiview learning, which employs multiple vectorized views, to the matrix-variate case which allows very novel variable star signature representations. The performance of our proposed methods is evaluated on the UCR Starlight and LINEAR data sets. Both the vector and matrix-variate versions of our multiview learning framework perform favourably – demonstrating the ability to discriminate variable star categories.

Multi-task and multi-view training for end-to-end relation extraction

Transfer learning has shown promising results for transferring knowledge ofsource tasks to target tasks in natural language processing (NLP). In this paper, we investigate a multi-task and multi-view learning (MTMVL) framework for end-to-end neural relation extraction, using large noisy data as one auxiliary task for improving manually labeled data, and a dependency parsing objective as a second auxiliary task for leveraging syntax. Two views are taken for relation extraction, consisting of a joint tagging view and a novel context relation view. To capture sentence multi-level information, we explore a weighted average approach to represent shared deep bi-directional recurrent neural networks and encourage auxiliary tasks to accommodate relation extraction task via a time-related scheduled sampling strategy. We evaluate MTMVL framework on manual-labeled ACE2005 dataset, and experimental results show that our model outperforms the state-of-the-art methods, which indicates the effectiveness of multi-auxiliary information for knowledge transferring.

Explainable Recommendation through Attentive Multi-View Learning

Recommender systems have been playing an increasingly important role in our daily life due to the explosive growth of information. Accuracy and explainability are two core aspects when we evaluate a recommendation model and have become one of the fundamental trade-offs in machine learning. In this paper, we propose to alleviate the trade-off between accuracy and explainability by developing an explainable deep model that combines the advantages of deep learning-based models and existing explainable methods. The basic idea is to build an initial network based on an explainable deep hierarchy (e.g., Microsoft Concept Graph) and improve the model accuracy by optimizing key variables in the hierarchy (e.g., node importance and relevance). To ensure accurate rating prediction, we propose an attentive multi-view learning framework. The framework enables us to handle sparse and noisy data by co-regularizing among different feature levels and combining predictions attentively. To mine readable explanations from the hierarchy, we formulate personalized explanation generation as a constrained tree node selection problem and propose a dynamic programming algorithm to solve it. Experimental results show that our model outperforms state-of-the-art methods in terms of both accuracy and explainability.

Letting Logos Speak: Leveraging Multiview Representation Learning for Data-Driven Logo Design

Logos serve a fundamental role as the visual figureheads of brands. Yet, due to the difficulty of using unstructured image data, prior research on logo design has largely been limited to non-quantitative studies. In this work, we explore the interplay between logo design and brand identity creation from a data-driven perspective. We develop both a novel logo feature extraction algorithm that uses modern image processing tools to decompose pixel-level image data into meaningful features, and a multiview representation learning framework that links these visual features to textual descriptions, consumer ratings of brand personality, and other high-level tags describing firms. We apply this framework to a unique dataset of brands, to understand which brands use which logo features, and how consumers evaluate these brands’ personalities. Moreover, we show that manipulating the model’s learned representations through what we term “brand arithmetic” yields new brand identities, and can help with ideation. Finally, through an application to fast food branding, we show how our model can be used as a decision support tool for suggesting typical logo features for a brand, and for predicting consumers’ reactions to new brands or rebranding efforts.