Multi-view Learning Algorithm Research Articles

Electroencephalography (EEG) based epilepsy diagnosis via multiple feature space fusion using shared hidden space-driven multi-view learning.

Epilepsy is a chronic, non-communicable disease caused by paroxysmal abnormal synchronized electrical activity of brain neurons, and is one of the most common neurological diseases worldwide. Electroencephalography (EEG) is currently a crucial tool for epilepsy diagnosis. With the development of artificial intelligence, multi-view learning-based EEG analysis has become an important method for automatic epilepsy recognition because EEG contains difficult types of features such as time-frequency features, frequency-domain features and time-domain features. However, current multi-view learning still faces some challenges, such as the difference between samples of the same class from different views is greater than the difference between samples of different classes from the same view. In view of this, in this study, we propose a shared hidden space-driven multi-view learning algorithm. The algorithm uses kernel density estimation to construct a shared hidden space and combines the shared hidden space with the original space to obtain an expanded space for multi-view learning. By constructing the expanded space and utilizing the information of both the shared hidden space and the original space for learning, the relevant information of samples within and across views can thereby be fully utilized. Experimental results on a dataset of epilepsy provided by the University of Bonn show that the proposed algorithm has promising performance, with an average classification accuracy value of 0.9787, which achieves at least 4% improvement compared to single-view methods.

Autoencoder in Autoencoder Networks.

Modeling complex correlations on multiview data is still challenging, especially for high-dimensional features with possible noise. To address this issue, we propose a novel unsupervised multiview representation learning (UMRL) algorithm, termed autoencoder in autoencoder networks (AE2-Nets). The proposed framework effectively encodes information from high-dimensional heterogeneous data into a compact and informative representation with the proposed bidirectional encoding strategy. Specifically, the proposed AE2-Nets conduct encoding in two directions: the inner-AE-networks extract view-specific intrinsic information (forward encoding), while the outer-AE-networks integrate this view-specific intrinsic information from different views into a latent representation (backward encoding). For the nested architecture, we further provide a probabilistic explanation and extension from hierarchical variational autoencoder. The forward-backward strategy flexibly addresses high-dimensional (noisy) features within each view and encodes complementarity across multiple views in a unified framework. Extensive results on benchmark datasets validate the advantages compared to the state-of-the-art algorithms.

Forecasting resource usage pattern changes in clouds via contrast graph-evolution learning

The increasing large-scale workload trace collected by datacenter monitors has provided great potential for workload prediction. However, resource usage pattern changes of workloads often exist, which are usually caused by user demand changes, workload re-locations or other reasons. If the patterns of resource usage change, the accuracy of prediction may degrade significantly. Hence, how to forecast such changes brings an open and challenging issue. In this paper, we present a novel method named CEGWP. CEGWP uses shapelets to explicitly identify resource usage patterns in a fine-grained level. Specifically. a two-level importance based shapelet extraction mechanism is designed to mine new usage pattern changes in temporal dimension, and a multi-view contrast graph-evolution learning algorithm is designed to fuse the interference among resource usage patterns of different workloads in spatial dimension. With both temporal extraction of shapelets from each single workload and spatial interference of different workloads, a spatio-temporal GNN-based TCN encoder is developed to forecast the resource usage pattern changes. CEGWP is jointly trained in an end-to-end manner and is evaluated using real trace data. Experimental results show that CEGWP achieves notable improvements by up to 62.15% against existing prediction methods and indicate the potential to be deployed into real-world cloud datacenters.

A multi-view subspace representation learning approach powered by subspace transformation relationship

Traditional multi-view subspace representation learning typically captures the pairwise similarity among data objects but neglects the underlying relationship among subspace representations of different views, resulting in the essential structural information of multi-view data not being fully exploited. In this study, a multi-view representation learning approach is proposed using subspace transformation relationships. First, a subspace transformation relationship is defined to extract the underlying subspace relationship information. The transformation relationship is constrained by a low-rank term to enhance the consistency of the relationships and is used to construct a multi-view subspace representation model. Second, an optimization method is proposed for the multi-view subspace representation model using the ALM-ADM strategy. Third, a novel multi-view subspace representation learning algorithm is presented using the model and optimization method. Consequently, experimental findings on benchmark datasets indicated that the algorithm outperformed the state-of-the-art algorithm, benefitting from extracting subspace transformation relationships to guarantee high-order correlation information from multi-view data.

Safe screening rules for multi-view support vector machines

Multi-view learning aims to make use of the advantages of different views to complement each other and fully mines the potential information in the data. However, the complexity of multi-view learning algorithm is much higher than that of single view learning algorithm. Based on the optimality conditions of two classical multi-view models: SVM-2K and multi-view twin support vector machine (MvTwSVM), this paper analyzes the corresponding relationship between dual variables and samples, and derives their safe screening rules for the first time, termed as SSR-SVM-2K and SSR-MvTwSVM. It can assign or delete four groups of different dual variables in advance before solving the optimization problem, so as to greatly reduce the scale of the optimization problem and improve the solution speed. More importantly, the safe screening criterion is “safe”, that is, the solution of the reduced optimization problem is the same as that of the original problem before screening. In addition, we further give a sequence screening rule to speed up the parameter optimization process, and analyze its properties, including the similarities and differences of safe screening rules between multi-view SVMs and single-view SVMs, the computational complexity, and the relationship between the parameter interval and screening rate. Numerical experiments verify the effectiveness of the proposed methods.

A novel sagittal craniosynostosis classification system based on multi-view learning algorithm

A novel sagittal craniosynostosis classification system based on multi-view learning algorithm

Drug repositioning based on multi-view learning with matrix completion.

Determining drug indications is a critical part of the drug development process. However, traditional drug discovery is expensive and time-consuming. Drug repositioning aims to find potential indications for existing drugs, which is considered as an important alternative to the traditional drug discovery. In this article, we propose a multi-view learning with matrix completion (MLMC) method to predict the potential associations between drugs and diseases. Specifically, MLMC first learns the comprehensive similarity matrices from five drug similarity matrices and two disease similarity matrices based on the multi-view learning (ML) with Laplacian graph regularization, and updates the drug-disease association matrix simultaneously. Then, we introduce matrix completion (MC) to add some positive entries in original association matrix based on low-rank structure, and re-execute the multi-view learning algorithm for association prediction. At last, the prediction results of the above two operations are integrated as the final output. Evaluated by 10-fold cross-validation and de novo tests, MLMC achieves higher prediction accuracy than the current state-of-the-art methods. Moreover, case studies confirm the ability of our method in novel drug-disease association discovery. The codes of MLMC are available at https://github.com/BioinformaticsCSU/MLMC. Contact: jxwang@mail.csu.edu.cn.

Multiview Embeddings for Soundscape Classification

Acoustic scenes (or soundscapes) can be composed of various background and foreground sound events. Classification of soundscapes have to deal with the overlap of these sound events. In this work, we propose to reduce this overlap by considering foreground sound events and background sound events as multiple views of the soundscape. Robust principal components analysis is used to decompose a soundscape into the background view and the foreground view. We can control the amount of information retained in this decomposition by using the subspace projection technique of nuisance attribute projection. We represent the audio samples as features from a convolutional neural network, and view-invariant representations are derived using a deep neural network based multi-view learning algorithm. Experimental results demonstrate the effectiveness of our proposed method on standard datasets for acoustic scene classification.

An Enhanced Predictive Cruise Control System Design With Data-Driven Traffic Prediction

The predictive cruise control (PCC) is a promising method to optimize energy consumption of vehicles, especially the heavy-duty vehicles (HDV). Due to the limited sensing range and computational capabilities available on-board, the conventional PCC system can only obtain a sub-optimal speed trajectory based on a shorter prediction horizon. The recently emerging information and communication technologies such as vehicular communication, cloud computing, and Internet of Things provide huge potentials to improve the traditional PCC system. In this paper, we propose a general framework for the enhanced cloud-based PCC system which integrates a data-driven traffic predictive model and the instantaneous control algorithms. Specifically, we introduce a novel multi-view CNN deep learning algorithm to predict traffic situation based on the historical and real-time traffic data collected from fields, and the time-varying adaptive model predictive control (MPC) to calculate the instantaneous optimal speed profile with the aim of minimizing energy consumption. We verified our approach via simulations in which the impact of various traffic condition on the PCC-enabled HDV has been fully evaluated.

基于Universum数据的多视角学习算法

基于Universum数据的多视角学习算法

Multi-view learning based on maximum margin of twin spheres support vector machine

Multi-view learning utilizes information from multiple representations to advance the performance of categorization. Most of the multi-view learning algorithms based on support vector machines seek the separating hyperplanes in different feature spaces, which may be unreasonable in practical application. Besides, most of them are designed to balanced data, which may lead to poor performance. In this work, a novel multi-view learning algorithm based on maximum margin of twin spheres support vector machine (MvMMTSSVM) is introduced. The proposed method follows both maximum margin principle and consensus principle. By following the maximum margin principle, it constructs two homocentric spheres and tries to maximize the margin between the two spheres for each view separately. To realize the consensus principle, the consistency constraints of two views are introduced in the constraint conditions. Therefore, it not only deals with multi-view class-imbalanced data effectively, but also has fast calculation efficiency. To verify the validity and rationlity of our MvMMTSSVM, we do the experiments on 24 binary datasets. Furthermore, we use Friedman test to verify the effectiveness of MvMMTSSVM.

A Joint Analysis of Multi-Paradigm fMRI Data With Its Application to Cognitive Study.

With the development of neuroimaging techniques, a growing amount of multi-modal brain imaging data are collected, facilitating comprehensive study of the brain. In this paper, we jointly analyzed functional magnetic resonance imaging (fMRI) collected under different paradigms in order to understand cognitive behaviors of an individual. To this end, we proposed a novel multi-view learning algorithm called structure-enforced collaborative regression (SCoRe) to extract co-expressed discriminative brain regions under the guidance of anatomical structure of the brain. An advantage of SCoRe over its predecessor collaborative regression (CoRe) lies in its incorporation of group structures in the brain imaging data, which makes the model biologically more meaningful. Results from real data analysis has confirmed that by incorporating prior knowledge of brain structure, SCoRe can deliver better prediction performance and is less sensitive to hyper-parameters than CoRe. After validation with simulation experiments, we applied SCoRe to fMRI data collected from the Philadelphia Neurodevelopmental Cohort and adopted the scores from the wide range achievement test (WRAT) to evaluate an individual's cognitive skills. We located 14 relevant brain regions that can efficiently predict WRAT scores and these brain regions were further confirmed by other independent studies.

Multi-view clustering toward aerial images by combining spectral analysis and local refinement

Aerial images’ clustering is of great significance in many domains of the last-generation computer systems, such as intelligent navigation, geological analysis, and disaster prediction. Conventional methods exploit only the global visual information and unimodal visual feature to characterize each aerial image, which is simple and self-descriptive in some cases. However, it cannot adequately simulate tiny and fine-grained visual patterns within each aerial image, such as automobiles of the “intersection” category and houses of the “residence” category. Moreover, the unimodal feature cannot effectively capture each aerial image. Other visual channels, such as the light spectrum, should also be encoded into the aerial image modeling system. We propose a novel aerial image clustering system, with a seamless fusion of spectrum and local features in this work. The multispectral visual clue is characterized by color plus texture channels from aerial images of each spectral channel. Simultaneously, locally distributed and very small objects combined with their spatial configurations are described by a set of graphlets. That is, each edge connects a pair of small objects, which are spatially neighboring. Subsequently, a multi-view learning algorithm is formulated to fuse the above channels to characterize each aerial image optimally. The importance of multiple spectral channels is flexibly tuned. A graph-based clustering algorithm is adopted to categorize the massive-scale aerial images into multiple types based on each aerial image’s fused feature description. Extensive comparative results on a million-level real-world aerial photograph set prove our method’s advantage over available state-of-the-art ones. Besides, visualization results imply that our clustering algorithm can optimally discriminate aerial images from different categories.

Intelligent human hand gesture recognition by local–global fusing quality-aware features

Each normal human has pairwise hands, which can make sophisticated hand gestures delivering different semantic meanings. In practice, disabled persons can communicate with each other by their hand gestures. However, for normal people without specific training process, it is challenging for them to understand the meaning of the various hand gestures. In this work, we propose a novel quality-aware human–computer interaction (HCI) framework for understanding the sophisticated human hand gestures, wherein the key technique is a multi-view feature learning algorithm that optimally fuses hand silhouettes, different figure positions, and hand motions. More specifically, given each human hand, we first extract the multimodal visual features, such as the hand silhouettes based on background removal, different figure position localization using active learning, and hand motions by leveraging optical flow. Afterward, we propose a multi-view learning-based feature fusing scheme that optimizes the multimodal features both locally and globally. Based on this, the optimal weights of different features channels can be calculated. By leveraging the optimally fused feature, we train a multi-class kernel machine to classify the various human hand gestures into each category with a particular meaning. Comprehensive experimental results on a large-scale human hand gesture data set have demonstrated that our method can achieve a highly competitive recognition accuracy by spending less time consumption. Besides, our method can robustly support arbitrary number of semantic category characterizing human hand gestures. Last but not least, our proposed hand gesture understanding technique can be conveniently incorporated into many state-of-the-art HCI system.

Multiple-feature-driven co-training method for crop mapping based on remote sensing time series imagery

ABSTRACT Remote sensing time series imagery (RSTSI) provides a useful tool for crop mapping, as it provides crucial spectral, temporal, and spatial (STS) features. However, its high dimensionality coupled with the limited number of training samples leads to an ill-posed classification problem and the Hughes phenomenon. To solve this problem, this study presents a multiple-feature-driven co-training method (MFDC) for accurately mapping crop types based on RSTSI with a limited number of training samples. In MFDC, four complementary pre-defined views, which represent STS features, are generated for the utilization of multiple features. Then, to enhance the classifier’s generalization ability, a novel labelled sample augmentation method that combines the Breaking Tiles algorithm and co-training is proposed. Third, to ensure the effectiveness of ensemble learning in co-training as well as to further speed up the learning process, a multi-view semi-supervised feature learning algorithm that expands the single view semi-supervised learning algorithm to multiple views is proposed and embedded in co-training. Finally, a weighted majority vote method is utilized to obtain the classification results. The experimental results for study areas in the United States indicate that the proposed method can accurately map crop types with a limited number of labelled training samples without a significant computational cost.

Multi-View Representation Learning With Deep Gaussian Processes.

Multi-view representation learning is a promising and challenging research topic, which aims to integrate multiple data information from different views to improve the learning performance. The recent deep Gaussian processes (DGPs) have the advantages of good uncertainty estimates, powerful non-linear mapping ability and great generalization capability, which can be used as an excellent data representation learning method. However, DGPs only focus on single view data and are rarely applied to the multi-view scenario. In this paper, we propose a multi-view representation learning algorithm with deep Gaussian processes (named MvDGPs), which inherits the advantages of deep Gaussian processes and multi-view representation learning, and can learn more effective representation of multi-view data. The MvDGPs consist of two stages. The first stage is multi-view data representation learning, which is mainly used to learn more comprehensive representations of multi-view data. The second stage is classifier design, which aims to select an appropriate classifier to better employ the representations obtained in the first stage. In contrast with DGPs, MvDGPs support asymmetrical modeling depths for different views of data, resulting in better characterizations of the discrepancies among different views. Experimental results on real-world multi-view data sets verify the effectiveness of the proposed algorithm, which indicates that MvDGPs can integrate the complementary information in multiple views to discover a good representation of the data.

Flower classification with modified multimodal convolutional neural networks

The new multi-view learning algorithm is proposed by modifying an existing method, the multimodal convolutional neural networks originally developed for image-text matching (modified m-CNN), to use not only images but also texts for classification. Firstly, pre-trained CNN and word embedding models are applied to extract visual features and represent each word in a text as a vector, respectively. Secondly, textual features are extracted by employing a CNN model for text data. Finally, pairs of features extracted through the text and image CNNs are concatenated and input to convolutional layer which can obtain a better learn of the important feature information in the integrated representation of image and text. Features extracted from the convolutional layer are input to a fully connected layer to perform classification. Experimental results demonstrate that the proposed algorithm can obtain superior performance compared with other data fusion methods for flower classification using data of images of flowers and their Korean descriptions. More specifically, the accuracy of the proposed algorithm is 10.1% and 14.5% higher than m-CNN and multimodal recurrent neural networks algorithms, respectively. The proposed method can significantly improve the performance of flower classification. The code and related data are publicly available via our GitHub repository.

Visual computing resources distribution and balancing by multimodal cat swarm optimization

According to the fast development of Internet technologies, nowadays more and more web applications require accessing and processing massive-scale visual data, such as content image search and automatic navigation. Practically, the computing power of a single personal PC is insufficient for handling such massive-scale visual elements. Owing to the advancement of cloud platforms, this problem can be well handled. In this work, we study the problem of how to optimally distribute the P visual processing tasks to Q computing resources. Specifically, we proposed an enhanced multimodal cat swarm optimization (MCSO) algorithm to fulfill this task. Given a huge number of images/videos, we first extract color, texture, and semantic channel to represent the visual content of each image/video. Afterward, we develop a multi-view feature learning algorithm to intelligently combine the multiple features into a descriptive one, wherein the weights of different feature channels are adjusted automatically. Subsequently, we use the CSO algorithm to assign each image/video to different remote servers. The CSO algorithm mimics the conventional cat hunting process, that is, a set of cats is divided into two groups, one for searching and the other for tracking. The two patterns are interacted by the mixture ratio, which indicates how many cats will conduct tracking in the next round. Based on the output of CSO, each image/video will be assigned to the most optimal remote sensor. Finally, based on the optimal assignment, the image/video processing can be conducted with the minimal time consumption. Comprehensive experimental comparisons have demonstrated the advantages of our method, that is, our MCSO can achieve a visual processing time three times faster than the second best performer.

Robust Visual Ship Tracking with an Ensemble Framework via Multi-View Learning and Wavelet Filter.

Maritime surveillance videos provide crucial on-spot kinematic traffic information (traffic volume, ship speeds, headings, etc.) for varied traffic participants (maritime regulation departments, ship crew, ship owners, etc.) which greatly benefits automated maritime situational awareness and maritime safety improvement. Conventional models heavily rely on visual ship features for the purpose of tracking ships from maritime image sequences which may contain arbitrary tracking oscillations. To address this issue, we propose an ensemble ship tracking framework with a multi-view learning algorithm and wavelet filter model. First, the proposed model samples ship candidates with a particle filter following the sequential importance sampling rule. Second, we propose a multi-view learning algorithm to obtain raw ship tracking results in two steps: extracting a group of distinct ship contour relevant features (i.e., Laplacian of Gaussian, local binary pattern, Gabor filter, histogram of oriented gradient, and canny descriptors) and learning high-level intrinsic ship features by jointly exploiting underlying relationships shared by each type of ship contour features. Third, with the help of the wavelet filter, we performed a data quality control procedure to identify abnormal oscillations in the ship positions which were further corrected to generate the final ship tracking results. We demonstrate the proposed ship tracker’s performance on typical maritime traffic scenarios through four maritime surveillance videos.

Multiple instance deep learning for weakly-supervised visual object tracking

Intelligently tracking objects with varied shapes, color, lighting conditions, and backgrounds is an extremely useful application in many HCI applications, such as human body motion capture, hand gesture recognition, and virtual reality (VR) games. However, accurately tracking different objects under uncontrolled environments is a tough challenge due to the possibly dynamic object parts, varied lighting conditions, and sophisticated backgrounds. In this work, we propose a novel semantically-aware object tracking framework, wherein the key is weakly-supervised learning paradigm that optimally transfers the video-level semantic tags into various regions. More specifically, give a set of training video clips, each of which is associated with multiple video-level semantic tags, we first propose a weakly-supervised learning algorithm to transfer the semantic tags into various video regions. The key is a MIL (Zhong et al., 2020) [1]-based manifold embedding algorithm that maps the entire video regions into a semantic space, wherein the video-level semantic tags are well encoded. Afterward, for each video region, we use the semantic feature combined with the appearance feature as its representation. We designed a multi-view learning algorithm to optimally fuse the above two types of features. Based on the fused feature, we learn a probabilistic Gaussian mixture model to predict the target probability of each candidate window, where the window with the maximal probability is output as the tracking result. Comprehensive comparative results on a challenging pedestrian tracking task as well as the human hand gesture recognition have demonstrated the effectiveness of our method. Moreover, visualized tracking results have shown that non-rigid objects with moderate occlusions can be well localized by our method.