Multiple Kernel Learning Method Research Articles

A novel graph-based multiple kernel learning framework for hyperspectral image classification

ABSTRACT Multiple kernel learning (MKL) is an efficient way to improve hyperspectral image classification with few training samples by integrating spectral and spatial features. Nonetheless, presenting a MKL algorithm to capture significant information and eliminate redundancy is indispensable. In this paper, a novel graph-based MKL method (GMKL) is introduced to enhance the performance of the MKL algorithm and achieve high discriminative features, resulting in maximum separability. To do so, an informative low-rank graph is determined via a new algorithm called block diagonal representation that takes into account both local and global data structures. Then, in lieu of a time-consuming search for finding the optimal combination of the basic kernels, unlike the prior works, an optimal projective direction is acquired using graph theory. Various basic kernels generated from superpixels and attribute profiles are projected to produce a discriminative combination of kernels. Eventually, the optimum kernel is given to the support vector machine classifier. Experiments have been conducted on two popular hyperspectral datasets (Indian Pines and Pavia University). For the datasets, the proposed algorithm gained 90.43% and 99.28% overall accuracy, respectively, using only 1% training samples, which is much better than the state-of-the-art MKL algorithm and deep learning-based approaches with lower computational time.

Neural Generalization of Multiple Kernel Learning

Multiple Kernel Learning (MKL) is a conventional way to learn the kernel function in kernel-based methods. MKL algorithms enhance the performance of kernel methods. However, these methods have a lower complexity compared to deep models and are inferior to them regarding recognition accuracy. Deep learning models can learn complex functions by applying nonlinear transformations to data through several layers. In this paper, we show that a typical MKL algorithm can be interpreted as a one-layer neural network with linear activation functions. By this interpretation, we propose a Neural Generalization of Multiple Kernel Learning (NGMKL), which extends the conventional MKL framework to a multi-layer neural network with nonlinear activation functions. Our experiments show that the proposed method, which has a higher complexity than traditional MKL methods, leads to higher recognition accuracy on several benchmarks.

Multi-Objective Prediction of the Sound Insulation Performance of a Vehicle Body System Using Multiple Kernel Learning–Support Vector Regression

The sound insulation performance of an electric vehicle’s body system serves as a critical metric for evaluating the noise, vibration, and harshness (NVH) quality of the vehicle. The accurate and efficient prediction of sound insulation performance is foundational for undertaking noise reduction design and optimization. Current engineering practices predominantly rely on Computer-Aided Engineering (CAE) methodologies to address this challenge. However, inherent shortcomings such as low modeling efficiency and difficulty in ensuring prediction accuracy often characterize these approaches. In an effort to overcome these limitations, we propose a decomposition framework for predicting the sound insulation performance of the electric vehicle body system. This framework is established based on a comprehensive analysis of the noise transmission paths within the system. Subsequently, the support vector regression (SVR) method is introduced to construct a machine learning model specifically designed for predicting the sound insulation performance of the body system. This approach aims to mitigate the inherent weaknesses associated with the conventional CAE processes using a ‘data-driven’ paradigm. Furthermore, the Multiple Kernel Learning (MKL) method is used to enhance the processing efficacy of the SVR model. The proposed method is validated using practical application and testing on a specific electric vehicle. The results demonstrate commendable performance in terms of prediction accuracy and robustness. This research contributes to advancing the field by presenting a more effective and reliable approach to predicting the sound insulation performance of electric vehicle body systems, offering valuable insights for noise reduction strategies and optimization efforts in the automotive industry.

PDATC-NCPMKL: Predicting drug's Anatomical Therapeutic Chemical (ATC) codes based on network consistency projection and multiple kernel learning

The development and discovery of new drugs is time-consuming and needs lots of human and material resources. Therefore, discovery of novel effects of existing drugs is an important alternative way, which can accelerate the process of designing “new” drugs. The anatomical Therapeutic Chemical (ATC) classification system recommended by World Health Organization (WHO) is a basic research area in this regard. A novel ATC code of an existing drug suggests its novel effects. Some computational models have been proposed, which can predict the drug-ATC code associations. However, their performance is not very high. There still exist spaces for improvement. In this study, a new recommendation system (named PDATC-NCPMKL), which incorporated network consistency projection and multi-kernel learning, was designed to identify drug-ATC code associations. For drugs or ATC codes, several kernels were constructed, which were fused by a multiple kernel learning method and an additional kernel integration scheme. To enhance the performance, the drug-ATC code association adjacency matrix was reformulated by a variant of weighted K nearest known neighbors (WKNKN). The reformulated adjacency matrix, drug and ATC code kernels were fed into network consistency projection to generate the association score matrix. The proposed recommendation system was tested on the ATC codes at the second, third and fourth levels in drug ATC classification system using ten-fold cross-validation. The results indicated that all AUROC and AUPR values were close to or exceeded 0.96. Such performance was higher than some existing computational models. Some additional tests were conducted to prove the utility of adjacency matrix reformulation and to analyze the importance of drug and ATC code kernels.

3PNMF-MKL: A non-negative matrix factorization-based multiple kernel learning method for multi-modal data integration and its application to gene signature detection.

In this current era, biomedical big data handling is a challenging task. Interestingly, the integration of multi-modal data, followed by significant feature mining (gene signature detection), becomes a daunting task. Remembering this, here, we proposed a novel framework, namely, three-factor penalized, non-negative matrix factorization-based multiple kernel learning with soft margin hinge loss (3PNMF-MKL) for multi-modal data integration, followed by gene signature detection. In brief, limma, employing the empirical Bayes statistics, was initially applied to each individual molecular profile, and the statistically significant features were extracted, which was followed by the three-factor penalized non-negative matrix factorization method used for data/matrix fusion using the reduced feature sets. Multiple kernel learning models with soft margin hinge loss had been deployed to estimate average accuracy scores and the area under the curve (AUC). Gene modules had been identified by the consecutive analysis of average linkage clustering and dynamic tree cut. The best module containing the highest correlation was considered the potential gene signature. We utilized an acute myeloid leukemia cancer dataset from The Cancer Genome Atlas (TCGA) repository containing five molecular profiles. Our algorithm generated a 50-gene signature that achieved a high classification AUC score (viz., 0.827). We explored the functions of signature genes using pathway and Gene Ontology (GO) databases. Our method outperformed the state-of-the-art methods in terms of computing AUC. Furthermore, we included some comparative studies with other related methods to enhance the acceptability of our method. Finally, it can be notified that our algorithm can be applied to any multi-modal dataset for data integration, followed by gene module discovery.

Cancer subtyping with heterogeneous multi-omics data via hierarchical multi-kernel learning.

Differentiating cancer subtypes is crucial to guide personalized treatment and improve the prognosis for patients. Integrating multi-omics data can offer a comprehensive landscape of cancer biological process and provide promising ways for cancer diagnosis and treatment. Taking the heterogeneity of different omics data types into account, we propose a hierarchical multi-kernel learning (hMKL) approach, a novel cancer molecular subtyping method to identify cancer subtypes by adopting a two-stage kernel learning strategy. In stage 1, we obtain a composite kernel borrowing the cancer integration via multi-kernel learning (CIMLR) idea by optimizing the kernel parameters for individual omics data type. In stage 2, we obtain a final fused kernel through a weighted linear combination of individual kernels learned from stage 1 using an unsupervised multiple kernel learning method. Based on the final fusion kernel, k-means clustering is applied to identify cancer subtypes. Simulation studies show that hMKL outperforms the one-stage CIMLR method when there is data heterogeneity. hMKL can estimate the number of clusters correctly, which is the key challenge in subtyping. Application to two real data sets shows that hMKL identified meaningful subtypes and key cancer-associated biomarkers. The proposed method provides a novel toolkit for heterogeneous multi-omics data integration and cancer subtypes identification.

EEG-based Emotion Recognition Using Multiple Kernel Learning

Emotion recognition based on electroencephalography (EEG) has a wide range of applications and has great potential value, so it has received increasing attention from academia and industry in recent years. Meanwhile, multiple kernel learning (MKL) has also been favored by researchers for its data-driven convenience and high accuracy. However, there is little research on MKL in EEG-based emotion recognition. Therefore, this paper is dedicated to exploring the application of MKL methods in the field of EEG emotion recognition and promoting the application of MKL methods in EEG emotion recognition. Thus, we proposed a support vector machine (SVM) classifier based on the MKL algorithm EasyMKL to investigate the feasibility of MKL algorithms in EEG-based emotion recognition problems. We designed two data partition methods, random division to verify the validity of the MKL method and sequential division to simulate practical applications. Then, tri-categorization experiments were performed for neutral, negative and positive emotions based on a commonly used dataset, the Shanghai Jiao Tong University emotional EEG dataset (SEED). The average classification accuracies for random division and sequential division were 92.25% and 74.37%, respectively, which shows better classification performance than the traditional single kernel SVM. The final results show that the MKL method is obviously effective, and the application of MKL in EEG emotion recognition is worthy of further study. Through the analysis of the experimental results, we discovered that the simple mathematical operations of the features on the symmetrical electrodes could not effectively integrate the spatial information of the EEG signals to obtain better performance. It is also confirmed that higher frequency band information is more correlated with emotional state and contributes more to emotion recognition. In summary, this paper explores research on MKL methods in the field of EEG emotion recognition and provides a new way of thinking for EEG-based emotion recognition research.

Spaks: Self-paced multiple kernel subspace clustering with feature smoothing regularization

Considerable attention to kernel subspace clustering has emerged in recent years due to its excellent performance in grouping data points from multiple manifolds. In particular, multiple kernel learning (MKL) is commonly used in kernel subspace clustering models to alleviate the difficulty of choosing a suitable kernel function, which brings out numerous MKL-based subspace clustering methods. Despite these successes, these MKL methods cannot guarantee that the feature representation produced by the learned kernel function is easy-to-divide, which is critical for the clustering task. Besides, the weighting strategies in these methods ignore the respective fitting ability of each base kernel, which may result in inappropriate weight assignment and inferior performance. To address these problems, this paper proposes a Self-Paced Smooth Multiple Kernel Subspace Clustering (Spaks) method. In Spaks, we first incorporate a feature smoothing regularization into the kernel subspace clustering model to facilitate the smoothness of feature representation on the affinity graph, thus ensuring that the data points between different subspaces are sufficiently separable. Then we implement the multiple kernel mechanism in the self-paced learning framework. In this way, the weights of base kernels depend on their corresponding single-kernel optimization objective function value, which indicates that a greater weight will be assigned to a kernel with a smaller objective. At last, we optimize the formulated model by an alternative iteration scheme, so that the optimal affinity matrix is obtained for clustering. Extensive experiments on seven benchmark data sets validate the superiority of the proposed Spaks method on the clustering task.

MSGWO-MKL-SVM: A Missing Link Prediction Method for UAV Swarm Network Based on Time Series

Missing link prediction technology (MLP) is always a hot research area in the field of complex networks, and it has been extensively utilized in UAV swarm network reconstruction recently. UAV swarm is an artificial network with strong randomness, in the face of which prediction methods based on network similarity often perform poorly. To solve those problems, this paper proposes a Multi Kernel Learning algorithm with a multi-strategy grey wolf optimizer based on time series (MSGWO-MKL-SVM). The Multiple Kernel Learning (MKL) method is adopted in this algorithm to extract the advanced features of time series, and the Support Vector Machine (SVM) algorithm is used to determine the hyperplane of threshold value in nonlinear high dimensional space. Besides that, we propose a new measurable indicator of Multiple Kernel Learning based on cluster, transforming a Multiple Kernel Learning problem into a multi-objective optimization problem. Some adaptive neighborhood strategies are used to enhance the global searching ability of grey wolf optimizer algorithm (GWO). Comparison experiments were conducted on the standard UCI datasets and the professional UAV swarm datasets. The classification accuracy of MSGWO-MKL-SVM on UCI datasets is improved by 6.2% on average, and the link prediction accuracy of MSGWO-MKL-SVM on professional UAV swarm datasets is improved by 25.9% on average.

A bi-level formulation for multiple kernel learning via self-paced training

Multiple kernel learning (MKL) is a crucial issue which has been widely researched over the last two decades. Although existing MKL algorithms have achieved satisfactory performance in a broad range of applications, these methods do not adequately consider the adverse effects of unreliable or less reliable instances. To handle this shortcoming, we formulate multiple kernel learning in a bi-level learning paradigm consisting of the kernel combination weight learning (KWL) stage and the self-paced learning (SPL) stage, which alternatively negotiate with each other. The KWL stage dynamically absorbs reliable instances into model learning to accurately capture neighborhood relationships and obtains kernel coefficients via maximizing both global and local kernel alignment in a common schema. The SPL stage automatically evaluates the reliability of training samples via self-paced training. The extensive experiments indicate the robustness and superiority of the presented approach in comparison with existing MKL methods.

Multi-similarity fusion-based label propagation for predicting microbes potentially associated with diseases

More and more scholars have confirmed through research that the microbes in the human body are closely related to human health. Therefore, more and more scholars are committed to researching new prediction methods to discover potential microbe-disease associations. Newly predicted microbes associated with diseases are obtained by a novel way, namely, multi-similarity fusion-based label propagation (MDA-MSFLP). Specifically, the method firstly obtains multiple symmetric similarity matrices about microbes and diseases through calculation. Then, more comprehensive prior information is obtained by the fusion of multiple similarities with the multiple kernel learning (MKL) method. Also, it is considered that the items represented by 0 in the association matrix may be potential associations, so the association matrix is preprocessed according to the fused similarity and the corresponding items represented by 0 will be replaced by the obtained association probability scores. Finally, the label propagation method is used to make predictions for microbes which are potentially related to diseases. To verify its performance, 5-fold cross validation and leave one out cross validation are applied to MDA-MSFLP. It can be found that the results of our method are excellent. Moreover, detailed case studies of three diseases (Chronic Obstructive Pulmonary Disease (COPD), Cystic fibrosis and Psoriasis) are performed. Among the top 15 microbes associated with these three diseases in the predicted results, 10, 11, and 13 have the corresponding literature evidence. It can be concluded that MDA-MSFLP can contribute to the acquisition of microbes associated with diseases.

A multi-layer multi-kernel neural network for determining associations between non-coding RNAs and diseases

Identification of associations between non-coding RNAs and diseases plays an important role in the study of pathogenesis, which has been a hot topic in recent research. However, traditional methods are time-consuming to detect the associations between non-coding RNAs and diseases. Recently, associations of non-coding RNAs and diseases can be regarded as bipartite network. In this paper, we propose a novel deep multiple kernel learning method, called the multi-layer multi-kernel deep neural network (MLMKDNN). First, many feature matrices are built by multiple features of non-coding RNAs and diseases. Then, these feature matrices are mapped into kernel space and fused by deep neural network. Finally, combine two fused output of MLMKDNN as the predicted values. Three types of non-coding RNAs (miRNA, circRNA and lncRNA) are used to test the performance of MLMKDNN. Compared with other existing methods, our proposed model has high Area Under Precision Recall (AUPR) value on three types of datasets. Experimental results confirm that our method is an effective predictive tool. It provides a framework that can also be applied to the link prediction of other bipartite networks.

Short-term Wind Power Prediction Based on Soft Margin Multiple Kernel Learning Method

For short-term wind power prediction, a soft margin multiple kernel learning (MKL) method is proposed. In order to improve the predictive effect of the MKL method for wind power, a kernel slack variable is introduced into each base kernel to solve the objective function. Two kinds of soft margin MKL methods based on hinge loss function and square hinge loss function can be obtained when hinge loss functions and square hinge loss functions are selected. The improved methods demonstrate good robustness and avoid the disadvantage of the hard margin MKL method which only selects a few base kernels and discards other useful kernels when solving the objective function, thereby achieving an effective yet sparse solution for the MKL method. In order to verify the effectiveness of the proposed method, the soft margin MKL method was applied to the second wind farm of Tianfeng from Xinjiang for short-term wind power single-step prediction, and the single-step and multi-step predictions of short-term wind power was also carried out using measured data provided by alberta electric system operator (AESO). Compared with the support vector machine (SVM), extreme learning machine (ELM), kernel based extreme learning machine (KELM) methods as well as the SimpleMKL method under the same conditions, the experimental results demonstrate that the soft margin MKL method with different loss functions can efficiently achieve higher prediction accuracy and good generalization performance for short-term wind power prediction, which confirms the effectiveness of the method.

Combined Spatial-Spectral Schroedinger Eigenmaps with Multiple Kernel Learning for Hyperspectral Image Classification Using a Low Number of Training Samples

The classification of hyperspectral images is one of the most popular fields in remote sensing applications. It should be noted that spectral and spatial features have critical roles in this research area. This paper proposes a method based on spatial-spectral Schroedinger eigenmaps (SSSE) and multiple kernel learning (MKL) to classify hyperspectral images more efficiently while using a low number of training samples. In the proposed method, first SSSE is applied to spectral domain in order to extract significant features and reduce dimension of the original image. Then MKL is utilized to enhance the feature learning process and obtain an optimum combination of some specified kernels. Finally, the classification is carried out by substituting the optimal kernel in support vector machine (SVM) algorithm. Experimental results show that the proposed method improves classification accuracy significantly and provides highly efficient results in the case of a small number of training samples. Furthermore, the computation time of the proposed method is much lower than the state-of-the-art MKL methods.

Incomplete-view oriented kernel learning method with generalization error bound

Multi-view learning (MVL) is a promising direction and most MVL methods work under the assumption that data are complete in all views. However, this assumption is often violated in practice due to the difficulties of high cost, equipment failure, and so on. Besides, the amount of data with missing views is particularly huge. Therefore, it is challenging yet valuable to fully leverage the large-scale incomplete-view data. Enlightened by the reduced support vector machine (RSVM) and multi-view privileged support vector machine (PSVM-2V), this paper proposes an efficient kernel method called reduced PSVM-2V (RPSVM-2V). It can not only provide a novel solution to process incomplete-view data, but also can be adjusted to address large-scale complete-view learning problem efficiently. In addition, the idea of replacing the full kernel with a smaller rectangular reduced kernel is extended to develop another two MVL methods, i.e., reduced SVM-2K (RSVM-2K) and reduced multiple kernel learning method (RMKL). Furthermore, we analyze the generalization error bounds of RPSVM-2V and two extensions by using Rademacher complexity. The comprehensive experiments demonstrate that our proposed models can achieve comparable performance with less time and memory cost. The spectral analysis has further verified the effectiveness of the reduced kernel used in our models.

Learning the Kernel Matrix with Low-Rank Multiplicative Shaping

Selecting the optimal kernel is an important and difficult challenge in applying kernel methods to pattern recognition. To address this challenge, multiple kernel learning (MKL) aims to learn a kernel from a combination of base kernel functions that perform optimally on the task. In this paper, we propose a novel MKL-themed approach to combine base kernels that are multiplicatively shaped with low-rank positive semidefinitve matrices. The proposed approach generalizes several popular MKL methods and thus provides more flexibility in modeling data. Computationally, we show how these low-rank matrices can be learned efficiently from data using convex quadratic programming. Empirical studies on several standard benchmark datasets for MKL show that the new approach often improves prediction accuracy statistically significantly over very competitive single kernel and other MKL methods.

MKEL: Multiple Kernel Ensemble Learning via Unified Ensemble Loss for Image Classification

In this article, a novel ensemble model, called Multiple Kernel Ensemble Learning (MKEL), is developed by introducing a unified ensemble loss. Different from the previous multiple kernel learning (MKL) methods, which attempt to seek a linear combination of basis kernels as a unified kernel, our MKEL model aims to find multiple solutions in corresponding Reproducing Kernel Hilbert Spaces (RKHSs) simultaneously. To achieve this goal, multiple individual kernel losses are integrated into a unified ensemble loss. Therefore, each model can co-optimize to learn its optimal parameters by minimizing a unified ensemble loss in multiple RKHSs. Furthermore, we apply our proposed ensemble loss into the deep network paradigm and take the sub-network as a kernel mapping from the original input space into a feature space, named Deep-MKEL (D-MKEL). Our D-MKEL model can utilize the diversified deep individual sub-networks into a whole unified network to improve the classification performance. With this unified loss design, our D-MKEL model can make our network much wider than other traditional deep kernel networks and more parameters are learned and optimized. Experimental results on several mediate UCI classification and computer vision datasets demonstrate that our MKEL model can achieve the best classification performance among comparative MKL methods, such as Simple MKL, GMKL, Spicy MKL, and Matrix-Regularized MKL. On the contrary, experimental results on large-scale CIFAR-10 and SVHN datasets concretely show the advantages and potentialities of the proposed D-MKEL approach compared to state-of-the-art deep kernel methods.

Predicting Conversion from MCI to AD Combining Multi-Modality Data and Based on Molecular Subtype.

Alzheimer’s disease (AD) is a neurodegenerative brain disease in the elderly. Identifying patients with mild cognitive impairment (MCI) who are more likely to progress to AD is a key step in AD prevention. Recent studies have shown that AD is a heterogeneous disease. In this study, we propose a subtyping-based prediction strategy to predict the conversion from MCI to AD in three years according to MCI patient subtypes. Structural magnetic resonance imaging (sMRI) data and multi-omics data, including genotype data and gene expression profiling derived from peripheral blood samples, from 125 MCI patients were used in the Alzheimer’s Disease Neuroimaging Initiative (ADNI)-1 dataset and from 98 MCI patients in the ADNI-GO/2 dataset. A variational Bayes approximation model based on the multiple kernel learning method was constructed to predict whether an MCI patient will progress to AD within three years. In internal fivefold cross-validation within ADNI-1, we achieved an overall AUC of 0.83 (79.20% accuracy, 81.25% sensitivity, 77.92% specificity) compared to the model without subtyping, which achieved an AUC of 0.78 (76.00% accuracy, 77.08% sensitivity, 75.32% specificity). In external validation using ADNI-1 as a training set and ADNI-GO/2 as an independent test set, we attained an AUC of 0.78 (74.49% accuracy, 74.19% sensitivity, 74.63% specificity). Identifying MCI patient subtypes with omics data would improve the accuracy of predicting the conversion from MCI to AD. In addition to evaluating statistics, obtaining the significant sMRI, single nucleotide polymorphism (SNP) and mRNA expression data from peripheral blood of MCI patients is noninvasive and cost-effective for predicting conversion from MCI to AD.

Simultaneous Global and Local Graph Structure Preserving for Multiple Kernel Clustering.

Multiple kernel learning (MKL) is generally recognized to perform better than single kernel learning (SKL) in handling nonlinear clustering problem, largely thanks to MKL avoids selecting and tuning predefined kernel. By integrating the self-expression learning framework, the graph-based MKL subspace clustering has recently attracted considerable attention. However, the graph structure of data in kernel space is largely ignored by previous MKL methods, which is a key concept of affinity graph construction for spectral clustering purposes. In order to address this problem, a novel MKL method is proposed in this article, namely, structure-preserving multiple kernel clustering (SPMKC). Specifically, SPMKC proposes a new kernel affine weight strategy to learn an optimal consensus kernel from a predefined kernel pool, which can assign a suitable weight for each base kernel automatically. Furthermore, SPMKC proposes a kernel group self-expressiveness term and a kernel adaptive local structure learning term to preserve the global and local structure of the input data in kernel space, respectively, rather than the original space. In addition, an efficient algorithm is proposed to solve the resulting unified objective function, which iteratively updates the consensus kernel and the affinity graph so that collaboratively promoting each of them to reach the optimum condition. Experiments on both image and text clustering demonstrate that SPMKC outperforms the state-of-the-art MKL clustering methods in terms of clustering performance and computational cost.

A spectral clustering with self-weighted multiple kernel learning method for single-cell RNA-seq data.

Single-cell RNA-sequencing (scRNA-seq) data widely exist in bioinformatics. It is crucial to devise a distance metric for scRNA-seq data. Almost all existing clustering methods based on spectral clustering algorithms work in three separate steps: similarity graph construction; continuous labels learning; discretization of the learned labels by k-means clustering. However, this common practice has potential flaws that may lead to severe information loss and degradation of performance. Furthermore, the performance of a kernel method is largely determined by the selected kernel; a self-weighted multiple kernel learning model can help choose the most suitable kernel for scRNA-seq data. To this end, we propose to automatically learn similarity information from data. We present a new clustering method in the form of a multiple kernel combination that can directly discover groupings in scRNA-seq data. The main proposition is that automatically learned similarity information from scRNA-seq data is used to transform the candidate solution into a new solution that better approximates the discrete one. The proposed model can be efficiently solved by the standard support vector machine (SVM) solvers. Experiments on benchmark scRNA-Seq data validate the superior performance of the proposed model. Spectral clustering with multiple kernels is implemented in Matlab, licensed under Massachusetts Institute of Technology (MIT) and freely available from the Github website, https://github.com/Cuteu/SMSC/.