Multi-kernel Learning Research Articles

Distributed and Quantized Online Multi-Kernel Learning

Kernel-based learning has well-documented merits in various machine learning tasks. Most of the kernel-based learning approaches rely on a pre-selected kernel, the choice of which presumes task-specific prior information. In addition, most existing frameworks assume that data are collected centrally at batch. Such a setting may not be feasible especially for large-scale data sets that are collected sequentially over a network. To cope with these challenges, the present work develops an online multi-kernel learning scheme to infer the intended nonlinear function ‘on the fly’ from data samples that are collected in distributed locations. To address communication efficiency among distributed nodes, we study the effects of quantization and develop a distributed and quantized online multiple kernel learning algorithm. We provide regret analysis that indicates our algorithm is capable of achieving sublinear regret. Numerical tests on real datasets show the effectiveness of our algorithm.

Missing Value Imputation for Multi-view Urban Statistical Data via Spatial Correlation Learning

As a developing trend of urbanization, massive amounts of urban statistical data with multiple views (e.g., views of Population and Economy) are increasingly collected and benefited to diverse domains, including transportation service, regional analysis, etc. Unfortunately, these statistical data that are divided into fine-grained regions usually suffer from missing value problem during the acquisition and storage processes. It is mianly caused by some inevitable circumstances, e.g., the document defacement, statistical difficulty in remote districts, and inaccurate information cleaning, etc. Those missing entries which make valuable information invisible may distort the further urban analysis. To improve the quality of missing data imputation, we propose an improved spatial multi-kernel learning method to guide the imputation process incorporating with the adaptive-weight non-negative matrix factorization strategy. Our model takes into account the regional latent similarities and the real geographical positions as well as the correlations among various views that are able to complete missing values precisely. We conduct intensive experiments to evaluate our method and compare with other state-of-the-art approaches on real-world datasets. All the empirical results show that the proposed model outperforms all the other state-of-the-art methods. Additionally, our model represents a strong generalization ability across multiple cities.

Multimodal Data Fusion Using Non-Sparse Multi-Kernel Learning With Regularized Label Softening

Due to the need of practical application, multiple sensors are often used for data acquisition, so as to realize the multimodal description of the same object. How to effectively fuse multimodal data has become a challenge problem in different scenarios including remote sensing. Nonsparse multi-Kernel learning has won many successful applications in multimodal data fusion due to the full utilization of multiple Kernels. Most existing models assume that the nonsparse combination of multiple Kernels is infinitely close to a strict binary label matrix during the training process. However, this assumption is very strict so that label fitting has very little freedom. To address this issue, in this article, we develop a novel nonsparse multi-Kernel model for multimodal data fusion. To be specific, we introduce a label softening strategy to soften the binary label matrix which provides more freedom for label fitting. Additionally, we introduce a regularized term based on manifold learning to anti over fitting problems caused by label softening. Experimental results on one synthetic dataset, several UCI multimodal datasets and one multimodal remoting sensor dataset demonstrate the promising performance of the proposed model.

Accuracy-diversity trade-off in recommender systems via graph convolutions

Graph convolutions, in both their linear and neural network forms, have reached state-of-the-art accuracy on recommender system (RecSys) benchmarks. However, recommendation accuracy is tied with diversity in a delicate trade-off and the potential of graph convolutions to improve the latter is unexplored. Here, we develop a model that learns joint convolutional representations from a nearest neighbor and a furthest neighbor graph to establish a novel accuracy-diversity trade-off for recommender systems. The nearest neighbor graph connects entities (users or items) based on their similarities and is responsible for improving accuracy, while the furthest neighbor graph connects entities based on their dissimilarities and is responsible for diversifying recommendations. The information between the two convolutional modules is balanced already in the training phase through a regularizer inspired by multi-kernel learning. We evaluate the joint convolutional model on three benchmark datasets with different degrees of sparsity. The proposed method can either trade accuracy to improve substantially the catalog coverage or the diversity within the list; or improve both by a lesser amount. Compared with accuracy-oriented graph convolutional approaches, the proposed model shows diversity gains up to seven times by trading as little as 1% in accuracy. Compared with alternative accuracy-diversity trade-off solutions, the joint graph convolutional model retains the highest accuracy while offering a handle to increase diversity. To our knowledge, this is the first work proposing an accuracy-diversity trade-off with graph convolutions and opens the doors to learning over graphs approaches for improving such trade-off.

Multi-Modal Biometrics based on Data Fusion

With the development of intelligent application, biometrics recognition technology has been widely concerned and applied in many fields of the real world, such as access control and payment. The traditional biometrics are usually based on single modality data of the subjects, but they are limited by the feature information capacity and the bottleneck in recognition accuracy. In this paper, a multi-modal biometric recognition framework is presented, which utilizes a multi-kernel learning algorithm to fuse heterogeneous information of different modal data. In order to extract complementary information from them, we combine the kernel matrix to form the mixed kernel matrix, and then give the final classification results. The experimental results on multiple biometric datasets show that our method can obtain higher recognition accuracy compared with the existing single mode and multi-mode fusion methods.

Classification Of TI-RADS Class-4 Thyroid Nodules Via Ultrasound-Based Radiomics And Multi-Kernel Learning

The diagnosis of thyroid nodules is mainly based on the thyroid imaging reporting and data system (TI-RADS). Among different classes, class-4 thyroid nodules are of great uncertainty with malignant risk ranging from 5% to 80%. Class-4 thyroid nodules have three sub-classes, namely 4a, 4b and 4c. Accurate classification of these three sub-classes is of clinical importance. Therefore, we aim to classify TI-RADS class-4 thyroid nodules based on radiomics features from ultrasound images and multi-kernel learning algorithm. The study cohort consisted of 59 patients with TI-RADS class-4 thyroid nodules. Patients were classified into TI-RADS class-4a, 4b and 4c according to ultrasound-guided fine-needle aspiration biopsy results. Two experienced doctors drew the region-of-interest for each nodule on the ultrasound images. Pyradiomics was used to extract shape features, texture features, Laplacian of Gaussian features, wavelet features, square features, square root features, logarithm features, exponential features, gradient features and local binary pattern features from original ultrasound images. The selected features were normalized to a range from -1 to 1. Sequential backward elimination approach was used to select features from different categories. A multi-kernel support vector machine (SVM) classifier was configured with 10 linear kernels to combine features from different categories for classification. Centered alignment method was used to calculate the coefficient of each kernel for kernel combination. Accuracy, sensitivity, specificity, area under the curve (AUC) were applied to evaluate the performance of the classifier for each sub-class, and permutation test was used to test whether the AUC acquired from the proposed multi-kernel classifier was greater than random guess. Pyradiomics extracted 1493 radiomic features from the ultrasound images of thyroid nodules. After feature selection, there remained 332 features. Classification results demonstrated that the configured multi-kernel SVM could accurately classify class-4 thyroid nodules with a total classification accuracy of 86.44%. The accuracy for each of the sub-class was 94.44%, 73.33% and 75%, respectively. The detailed result is shown in the Table below. The radiomic features in combination with fine-tuned machine learning models showed high accuracy in classification of TI-RADS class-4 thyroid nodules. The results demonstrated potential clinical application of radiomic features and multi-kernel learning in the diagnosis of thyroid nodules.Abstract 3978; TableTypeAccuracySensitivitySpecificityAUCp-valuenumber of featuresTI-RADS class-4a94.44%0.94440.82610.9263<0.001332TI-RADS class-4b73.33%0.63640.86670.7667<0.001332TI-RADS class-4c75%0.58820.87500.69120.039332 Open table in a new tab

EEG-Based Epilepsy Recognition via Multiple Kernel Learning.

In the field of brain-computer interfaces, it is very common to use EEG signals for disease diagnosis. In this study, a style regularized least squares support vector machine based on multikernel learning is proposed and applied to the recognition of epilepsy abnormal signals. The algorithm uses the style conversion matrix to represent the style information contained in the sample, regularizes it in the objective function, optimizes the objective function through the commonly used alternative optimization method, and simultaneously updates the style conversion matrix and classifier during the iteration process parameter. In order to use the learned style information in the prediction process, two new rules are added to the traditional prediction method, and the style conversion matrix is used to standardize the sample style before classification.

Ensemble of kernel extreme learning machine based random forest classifiers for automatic heartbeat classification

Automatic heartbeat classification technology based on the ECG plays an important role in assisting doctors with arrhythmia diagnosis. While many heartbeat classification studies can achieve good performance under the intra-patient paradigm, they still cannot offer acceptable classification results under the inter-patient paradigm. Additionally, the available ECG datasets are highly class imbalanced since normal heartbeats appear much more frequently than abnormal heartbeats, resulting in most methods having low sensitives and positive predictive values on minority class ectopic heartbeats. To solve the above problems, this study proposes an automatic ECG heartbeat classification method based on ensemble learning and multi-kernel learning. First, we use a linear combination of the radial basis function kernel and the polynomial kernel to produce a mixed-kernel-based extreme learning machine (MKELM). Then, a MKELM-based random forest binary classifier (MKELM-RF) is constructed. Finally, an ensemble multiclass classifier MKELM-RF-OVO is proposed based on one-vs.-one (OVO) reduction and MKELM-RF. We evaluated the proposed method on the public MIT-BIH-AR benchmarks database, under the inter-patient paradigm, classifying four types of heartbeats, namely, normal beats (N), supraventricular ectopic beats (S), ventricular ectopic beats (V) and the fusion of ventricular and normal (F). The obtained overall accuracy and the average positive predictive value are 98.1% and 93.9%, respectively, which are higher than the current studies by approximately 4% and 6%, respectively. The sensitivities for classes S and V are 1 and 94.4%, respectively, which outperforms most methods. The evaluation results show that our proposed method achieves a superior classification performance compared to the state-of-the-art methods.

Late Fusion Multi-View Clustering Based on Local Multi-Kernel Learning

Late Fusion Multi-View Clustering Based on Local Multi-Kernel Learning

An interpretable regression approach based on bi-sparse optimization

Given the increasing amounts of data and high feature dimensionalities in forecasting problems, it is challenging to build regression models that are both computationally efficient and highly accurate. Moreover, regression models commonly suffer from low interpretability when using a single kernel function or a composite of multi-kernel functions to address nonlinear fitting problems. In this paper, we propose a bi-sparse optimization-based regression (BSOR) model and corresponding algorithm with reconstructed row and column kernel matrices in the framework of support vector regression (SVR). The BSOR model can predict continuous output values for given input points while using the zero-norm regularization method to achieve sparse instance and feature sets. Experiments were run on 16 datasets to compare BSOR to SVR, linear programming SVR (LPSVR), least squares SVR (LSSVR), multi-kernel learning SVR (MKLSVR), least absolute shrinkage and selection operator regression (LASSOR), and relevance vector regression (RVR). BSOR significantly outperformed the other six regression models in predictive accuracy, identification of the fewest representative instances, selection of the fewest important features, and interpretability of results, apart from its slightly high runtime.

Lithofacies identification using support vector machine based on local deep multi-kernel learning

Lithofacies identification is a crucial work in reservoir characterization and modeling. The vast inter-well area can be supplemented by facies identification of seismic data. However, the relationship between lithofacies and seismic information that is affected by many factors is complicated. Machine learning has received extensive attention in recent years, among which support vector machine (SVM) is a potential method for lithofacies classification. Lithofacies classification involves identifying various types of lithofacies and is generally a nonlinear problem, which needs to be solved by means of the kernel function. Multi-kernel learning SVM is one of the main tools for solving the nonlinear problem about multi-classification. However, it is very difficult to determine the kernel function and the parameters, which is restricted by human factors. Besides, its computational efficiency is low. A lithofacies classification method based on local deep multi-kernel learning support vector machine (LDMKL-SVM) that can consider low-dimensional global features and high-dimensional local features is developed. The method can automatically learn parameters of kernel function and SVM to build a relationship between lithofacies and seismic elastic information. The calculation speed will be expedited at no cost with respect to discriminant accuracy for multi-class lithofacies identification. Both the model data test results and the field data application results certify advantages of the method. This contribution offers an effective method for lithofacies recognition and reservoir prediction by using SVM.

Tensorial graph learning for link prediction in generalized heterogeneous networks

Tensorial graph learning frameworks are proposed for link predictions in heterogeneous, homogeneous and generalized heterogeneous networks. In these frameworks, tensorial graphs are used to represent different networks by incorporating node and edge tensors into the graphs. A tensorial graph kernel method is developed for link predictions in these networks using four types of, i.e., structural, behavioral, content and node/edge characteristics, data. In this method, a n-strand iterated algorithm and a tensorial graph based random walk algorithm are proposed to measure node similarities in different networks within the generalized heterogeneous networks, and a tensorial graph multi-kernel learning method is developed to integrate the results. Experimental results on two real-world social media databases show that the tensorial graph kernel method has better performance using all types of data than using one type of data alone or combinations of some types of data. The tensorial graph kernel method also performs considerably better than existing competitive methods.

Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamics

Kernel-based methods exhibit well-documented performance in various nonlinear learning tasks. Most of them rely on a preselected kernel, whose prudent choice presumes task-specific prior informatio...

Multi-modal egocentric activity recognition using multi-kernel learning

Egocentric activity recognition in first-person videos has an increasing importance with a variety of applications such as lifelogging, summarization, assisted-living and activity tracking. Existing methods for this task are based on interpretation of various sensor information using pre-determined weights for each feature. In this work, we propose a new framework for egocentric activity recognition problem based on combining audio-visual features with multi-kernel learning (MKL) and multi-kernel boosting (MKBoost). For that purpose, firstly grid optical-flow, virtual-inertia feature, log-covariance, cuboid are extracted from the video. The audio signal is characterized using a "supervector", obtained based on Gaussian mixture modelling of frame-level features, followed by a maximum a-posteriori adaptation. Then, the extracted multi-modal features are adaptively fused by MKL classifiers in which both the feature and kernel selection/weighing and recognition tasks are performed together. The proposed framework was evaluated on a number of egocentric datasets. The results showed that using multi-modal features with MKL outperforms the existing methods.

Facies Identification Based on Multikernel Relevance Vector Machine

Facies identification is a powerful means to predict reservoirs. We achieve facies identification using a relevance vector machine (RVM) and develop a facies discriminant method based on a multikernel RVM (MKRVM). An RVM has the same functional form as a support vector machine (SVM) that is widely used in geophysics and shows a promising performance in disposing of small-samples, nonlinear and high-dimensional problems. The RVM inherits these superiorities, and its training is implemented under the Bayesian framework. Thus, it can provide probability information about the classified facies, which is critical to evaluate uncertainty of the result. Besides, the penalty parameter of the RVM does not depend on human experience. Compared with single-kernel learning, multikernel learning (MKL) is more flexible. After mapping the original data into a combined space by MKL, the features can be more accurately expressed in the new space, thereby improving the classification accuracy. Therefore, we introduce the RVM into facies classification and extend it to the MKRVM-based facies identification. The proposed method has advantageous properties such as strong generalization ability and high accuracy. First, we apply the approach to well log facies classification with different input features. Then, it is applied to seismic lithofacies identification with inverted elastic attributes to predict the target reservoirs. All the examples verify the effect and potential of the new method.

Enhancing the feature representation of multi-modal MRI data by combining multi-view information for MCI classification

The classification of mild cognitive impairment (MCI), which is a early stage of Alzheimer’s disease and is associated with brain structural and functional changes, is still a challenging task. Recent studies have shown great promise for improving the performance of MCI classification by combining multiple structural and functional features, such as grey matter volume and clustering coefficient. However, extracting which features and how to combine multiple features to improve the performance of MCI classification have always been challenging problems. To address these problems, in this study we propose a new method to enhance the feature representation of multi-modal MRI data by combining multi-view information to improve the performance of MCI classification. Firstly, we extract two structural features (including grey matter volume and cortical thickness) and two functional features (including clustering coefficient and shortest path length) of each cortical brain region based on automated anatomical labeling (AAL) atlas from both T1w MRI and rs-fMRI data of each subject. Then, in order to obtain features that are more helpful in distinguishing MCI subjects, an improved multi-task feature selection method, namely MTFS-gLASSO-TTR, is proposed. Finally, a multi-kernel learning algorithm is adopted to combine multiple features to perform the MCI classification task. Our proposed MCI classification method is evaluated on 315 subjects (including 105 LMCI subjects, 105 EMCI subjects and 105 NCs) with both T1w MRI and rs-fMRI data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Experimental results show that our proposed method achieves an accuracy of 88.5% and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.897 for LMCI/NC classification, an accuracy of 82.7% and an AUC of 0.832 for EMCI/NC classification, and an accuracy of 79.6% and an AUC of 0.803 for LMCI/EMCI classification, respectively. In addition, by comparison, the accuracy and AUC values of our proposed method are better than those of some existing state-of-the-art methods in MCI classification. Overall, our proposed MCI classification method is effective and promising for automatic diagnosis of MCI in clinical practice.

Single-Cell Clustering Based on Shared Nearest Neighbor and Graph Partitioning.

Clustering of single-cell RNA sequencing (scRNA-seq) data enables discovering cell subtypes, which is helpful for understanding and analyzing the processes of diseases. Determining the weight of edges is an essential component in graph-based clustering methods. While several graph-based clustering algorithms for scRNA-seq data have been proposed, they are generally based on k-nearest neighbor (KNN) and shared nearest neighbor (SNN) without considering the structure information of graph. Here, to improve the clustering accuracy, we present a novel method for single-cell clustering, called structural shared nearest neighbor-Louvain (SSNN-Louvain), which integrates the structure information of graph and module detection. In SSNN-Louvain, based on the distance between a node and its shared nearest neighbors, the weight of edge is defined by introducing the ratio of the number of the shared nearest neighbors to that of nearest neighbors, thus integrating structure information of the graph. Then, a modified Louvain community detection algorithm is proposed and applied to identify modules in the graph. Essentially, each community represents a subtype of cells. It is worth mentioning that our proposed method integrates the advantages of both SNN graph and community detection without the need for tuning any additional parameter other than the number of neighbors. To test the performance of SSNN-Louvain, we compare it to five existing methods on 16 real datasets, including nonnegative matrix factorization, single-cell interpretation via multi-kernel learning, SNN-Cliq, Seurat and PhenoGraph. The experimental results show that our approach achieves the best average performance in these datasets.

Hypernetwork Construction and Feature Fusion Analysis Based on Sparse Group Lasso Method on fMRI Dataset

Recent works have shown that the resting-state brain functional connectivity hypernetwork, where multiple nodes can be connected, are an effective technique for brain disease diagnosis and classification research. The lasso method was used to construct hypernetworks by solving sparse linear regression models in previous research. But, constructing a hypernetwork based on the lasso method simply selects a single variable, in that it lacks the ability to interpret the grouping effect. Considering the group structure problem, the previous study proposed to create a hypernetwork based on the elastic net and the group lasso methods, and the results showed that the former method had the best classification performance. However, the highly correlated variables selected by the elastic net method were not necessarily in the active set in the group. Therefore, we extended our research to address this issue. Herein, we propose a new method that introduces the sparse group lasso method to improve the construction of the hypernetwork by solving the group structure problem of the brain regions. We used the traditional lasso, group lasso method, and sparse group lasso method to construct a hypernetwork in patients with depression and normal subjects. Meanwhile, other clustering coefficients (clustering coefficients based on pairs of nodes) were also introduced to extract features with traditional clustering coefficients. Two types of features with significant differences obtained after feature selection were subjected to multi-kernel learning for feature fusion and classification using each method, respectively. The network topology results revealed differences among the three networks, where hypernetwork using the lasso method was the strictest; the group lasso, most lenient; and the sgLasso method, moderate. The network topology of the sparse group lasso method was similar to that of the group lasso method but different from the lasso method. The classification results show that the sparse group lasso method achieves the best classification accuracy by using multi-kernel learning, which indicates that better classification performance can be achieved when the group structure exists and is properly extended.

Multi-Kernel Online Reinforcement Learning for Path Tracking Control of Intelligent Vehicles

Path tracking control of intelligent vehicles has to deal with the difficulties of model uncertainties and nonlinearities. As a class of adaptive optimal control methods, reinforcement learning (RL) has received increasing attention in solving difficult control problems. However, feature representation and online learning ability are two major problems to be solved for learning control of uncertain dynamic systems. In this article, we propose a multi-kernel online RL approach for path tracking control of intelligent vehicles. In the proposed approach, a multiple kernel feature learning framework is designed for online learning control based on dual heuristic programming (DHP) and the new online learning control algorithm is called multi-kernel DHP (MKDHP). In MKDHP, instead of the expert knowledge for selecting and fine-tuning of a suitable kernel function, only a set of basic kernel functions is required to be predefined and the multi-kernel features can be learned for value function approximation in the critic. The simulation studies on path tracking control for intelligent vehicles have been conducted under <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$S$ </tex-math></inline-formula> -curve and urban road conditions. The results demonstrated that compared with other typical path tracking controllers for intelligent vehicles, such as the linear quadratic regulator (LQR), the pure pursuit controller and the ribbon-based controller, the proposed multi-kernel learning controller can achieve better performance in terms of tracking precision and smoothness.

Latent correlation embedded discriminative multi-modal data fusion

Ever-growing volume of multi-modal data has garnered considerable research enthusiasm in many fields, such as biomedical image analysis and biometrics. It remains a challenge how to effectively fuse multi-modal data for improving the learning performance. Many data fusion methods have been proposed, such as multi-kernel learning (MKL) and canonical component analysis (CCA). Most of the previous fusion methods focus on concatenating the multi-modal data, which fail to preserve the structure information across different modalities. In addition, they treat the samples equally in fusion and ignore the difference of the contributions of samples to the fusion model. In this paper, we proposed a discriminative multi-modal dimensionality reduction method, which can seamlessly fuse the multi-modal data and explore the latent correlation among different modalities for robust representation learning. In optimization, self-paced learning is adopted to dynamically estimate the contribution of each sample to the overall fusion model. Following the easy-to-hard learning sequence, our model can adaptively and sequentially enhance the robustness of the learning system in a knowledge discovery manner. Extensive experimental results on multi-modal brain disease diagnosis and multi-spectral palmprint classification tasks demonstrate the proposed method outperforms the previous multi-modal classification methods.