Kernel Learning Approach Research Articles

Self-Supervised Representation Learning on Electronic Health Records with Graph Kernel Infomax

Learning Electronic Health Records (EHRs) representation is a preeminent yet under-discovered research topic. It benefits various clinical decision support applications, e.g., medication outcome prediction or patient similarity search. Current approaches focus on task-specific label supervision on vectorized sequential EHR, which is not applicable to large-scale unsupervised scenarios. Recently, contrastive learning has shown great success in self-supervised representation learning problems. However, complex temporality often degrades the performance. We propose Graph Kernel Infomax, a self-supervised graph kernel learning approach on the graphical representation of EHR, to overcome the previous problems. Unlike the state-of-the-art, we do not change the graph structure to construct augmented views. Instead, we use Kernel Subspace Augmentation to embed nodes into two geometrically different manifold views. The entire framework is trained by contrasting nodes and graph representations on those two manifold views through the commonly used contrastive objectives. Empirically, using publicly available benchmark EHR datasets, our approach yields performance on clinical downstream tasks that exceeds the state-of-the-art. Theoretically, the variation in distance metrics naturally creates different views as data augmentation without changing graph structures. Practically, our method is non-ad hoc and confirms superior performance on commonly used graph benchmark datasets.

Multi-View Kernel Learning for Identification of Disease Genes.

Gene expression data sets and protein-protein interaction (PPI) networks are two heterogeneous data sources that have been extensively studied, due to their ability to capture the co-expression patterns among genes and their topological connections. Although they depict different traits of the data, both of them tend to group co-functional genes together. This phenomenon agrees with the basic assumption of multi-view kernel learning, according to which different views of the data contain a similar inherent cluster structure. Based on this inference, a new multi-view kernel learning based disease gene identification algorithm, termed as DiGId, is put forward. A novel multi-view kernel learning approach is proposed that aims to learn a consensus kernel, which efficiently captures the heterogeneous information of individual views as well as depicts the underlying inherent cluster structure. Some low-rank constraints are imposed on the learned multi-view kernel, so that it can effectively be partitioned into k or fewer clusters. The learned joint cluster structure is used to curate a set of potential disease genes. Moreover, a novel approach is put forward to quantify the importance of each view. In order to demonstrate the effectiveness of the proposed approach in capturing the relevant information depicted by individual views, an extensive analysis is performed on four different cancer-related gene expression data sets and PPI network, considering different similarity measures.

Data-adaptive kernel clustering with half-quadratic-based neighborhood relationship preservation

Kernel learning approaches have recently attracted much attention to improve the performance of clustering tasks. Although extensive studies have attempted to suppress the adverse influence of less confident samples, they cannot sufficiently take into account the different importance of samples in kernel learning. To alleviate this issue, our proposed algorithm utilizes a half-quadratic (HQ) paradigm to automatically determine the reliability of samples and to adaptively capture neighborhood relation information according to sample importance. We model an effective parameter-free kernel exploration by exploiting both global and HQ-based local data relationships. This paper designs an effective Alternative Direction Method of Multiplier (ADMM)-based algorithm to convert the fourth-order objective function to the second-order one, and offers an efficient closed-form solution for a simplex optimization problem based on the closest point theorem. Moreover, we prove the convergence of our proposed algorithm in theoretical and experimental fashions. Comprehensive experiments on benchmark and real-world datasets demonstrate that on average, the presented framework against competitors can improve clustering performance in terms of accuracy, NMI, and purity by about 5%–22%, verifying the efficacy and superiority of our proposed method.

A Closer Look at the Kernels Generated by the Decision and Regression Tree Ensembles

Tree ensembles can be interpreted as implicit kernel generators, where the ensuing proximity matrix represents the data-driven tree ensemble kernel. Focus of our work is the utility of tree based ensembles as kernel generators that (in conjunction with a regularized linear model) enable kernel learning. We elucidate the performance of the tree based random forest (RF) and gradient boosted tree (GBT) kernels in a comprehensive simulation study comprising of continuous and binary targets. We show that for continuous targets (regression), this kernel learning approach is competitive to the respective tree ensemble in higher dimensional scenarios, particularly in cases with larger number of noisy features. For the binary target (classification), the tree ensemble based kernels and their respective ensembles exhibit comparable performance. We provide the results from several real life datasets for regression and classification relevant for biopharmaceutical and biomedical applications, that are in line with the simulations to show how these insights may be leveraged in practice. We discuss general applicability and extensions of the tree ensemble based kernels for survival targets and interpretable landmarking in classification and regression. Finally, we outline future research for kernel learning due to feature space partitionings.

Multiple kernel fusion: A novel approach for lake water depth modeling

Multiple kernel fusion (MKF) refers to the task of combining multiple sources of information in the Hilbert space for improved performance. Very often the combined kernel is formed as a linear composition of multiple base kernels where the combination weights are learned from the data. As the first application of an MKF approach in hydrological modeling, lake water depth as one of the pivot factors in the reservoir analysis is simulated by considering different hydro-meteorological variables. The role of each individual input parameter is initially investigated by applying a kernel regression approach. We then illustrate the utility of an MKF formalism which learns kernel combination of weights to yield an optimal composition for kernel regression. A set of 40-year data collected from 27 groundwater and streamflow stations and 7 meteorological stations for precipitation and evaporation parameters in the vicinity of Lake Urmia are utilized for model development. Both visual and quantitative statistical performance criteria illustrate a superior performance for the MKF approach compared to kernel ridge regression (KRR), the support vector regression (SVR), back propagation neural network (BPNN) and auto regressive (AR) models. More specifically, while each individual input parameter fails to provide an accurate prediction for lake water depth modeling, an optimal combination of all input parameters incorporating the groundwater level, streamflow, precipitation and evaporation via a multiple kernel learning approach enhances the predictive performance of the model accuracy in the multiple scenarios. The promising results (RMSE = 0.098 m; R2 = 0.987; NSE = 0.986) may motivate the application of a MKF approach towards solving alternative and complex hydrological problems.

Unified representation of molecules and crystals for machine learning

Accurate simulations of atomistic systems from first principles are limited by computational cost. In high-throughput settings, machine learning can reduce these costs significantly by accurately interpolating between reference calculations. For this, kernel learning approaches crucially require a representation that accommodates arbitrary atomistic systems. We introduce a many-body tensor representation that is invariant to translations, rotations, and nuclear permutations of same elements, unique, differentiable, can represent molecules and crystals, and is fast to compute. Empirical evidence for competitive energy and force prediction errors is presented for changes in molecular structure, crystal chemistry, and molecular dynamics using kernel regression and symmetric gradient-domain machine learning as models. Applicability is demonstrated for phase diagrams of Pt-group/transition-metal binary systems.

Multiple kernel learning for label relation and class imbalance in multi-label learning

There are two common challenges in multi-label learning (MLL), complex label relation and imbalanced class. Few studies have focused on addressing both problems at the same time. In this paper, we propose a multiple kernel learning (MKL) approach to tackle two challenges, named as Multi-Kernel Multi-Label (MKML) method. MKML contains three kernel modules. The first kernel module adopts the traditional kernel function which contains the global information, and the other two kernel modules are designed for the two problems respectively. The second kernel module learns the inter-label relation by adding supervised information. The third kernel module adjusts the imbalanced decision boundary through multi-layer fusion strategy, which is proved to improve the representation ability of kernels in this paper. Finally, the proposed joint optimization method in this MKL framework achieves good generalization ability. We conduct several related experiments using real-world datasets to evaluate the effectiveness of our method. The results demonstrate that MKML outperforms other state-of-the art methods in MLL task.

Semi-supervised regression with manifold: A Bayesian deep kernel learning approach

Semi-supervised learning (SSL) aims at utilizing the vast unlabeled data to help the supervised training. While existing SSL methods have shown promising results on image classification tasks, most of them rely on the cluster assumption that does not apply to image regression tasks. In this paper, we address the under-studied semi-supervised image regression problem, of which the outputs are continuous values instead of categorical distributions. To tackle this challenging task, we propose an algorithm, called ManiDKL, with the idea that the prediction function should be smooth with respect to the intrinsic manifold of data distribution and behave similarly on both labeled and unlabeled data. In particular, we propose a framework that implements the Tikhonov regularization with generative manifold learning to ensure manifold smoothness of regression function and also reduces the problem to kernel learning. Then a semi-supervised non-parametric Bayesian based deep kernel learning algorithm is proposed, in which unlabeled data are incorporated through posterior regularization. We show the effectiveness of ManiDKL with extensive experiments. It shows that ManiDKL performs comparatively with state-of-the-art SSL image classification methods. Most importantly, we show the superiority of ManiDKL over all existing SSL regression methods on public image datasets.

Bayesian sparse factor analysis with kernelized observations

Multi-view problems can benefit from latent representations since they find low-dimensional projections that fairly capture the correlations among the multiple views that characterise the data. On the other hand, high-dimensionality and non-linear issues are traditionally handled by kernel methods, inducing a (non)-linear function between the latent projection and the data itself. However, they usually come with exposition to overfitting. Here, we combine Bayesian factor analysis with what we refer to as kernelized observations, in which the proposed model focuses on reconstructing not the data itself, but its relationship with other data points measured by a kernel function. In turn, we extend previous Bayesian FA formulations to be able to model non-linear data relationships by means of kernelized data representations and, at the same time, include additional facilities to obtain compact kernel representations by means of an automatic selection of Bayesian Relevance Vectors (RVs), feature relevance analysis and, even, obtain an automatic multiple kernel learning approach. Besides, this is flexibly included into a modular framework where we can easily adapt the model capabilities to the data needs and, even, combine them with previous FA functionalities such as heterogeneous data representations or semi-supervised learning. Using several public databases, we demonstrate the potential of the approach (and its extensions) w.r.t. common multi-view learning models such as kernel canonical correlation analysis, heterogeneous incomplete – variational autoenconder or manifold relevance determination, where the proposed model shows its ability to outperform the baselines while indistinctly combining model extensions.

Deeptime: a Python library for machine learning dynamical models from time series data

Generation and analysis of time-series data is relevant to many quantitative fields ranging from economics to fluid mechanics. In the physical sciences, structures such as metastable and coherent sets, slow relaxation processes, collective variables, dominant transition pathways or manifolds and channels of probability flow can be of great importance for understanding and characterizing the kinetic, thermodynamic and mechanistic properties of the system. Deeptime is a general purpose Python library offering various tools to estimate dynamical models based on time-series data including conventional linear learning methods, such as Markov state models (MSMs), Hidden Markov Models and Koopman models, as well as kernel and deep learning approaches such as VAMPnets and deep MSMs. The library is largely compatible with scikit-learn, having a range of Estimator classes for these different models, but in contrast to scikit-learn also provides deep Model classes, e.g. in the case of an MSM, which provide a multitude of analysis methods to compute interesting thermodynamic, kinetic and dynamical quantities, such as free energies, relaxation times and transition paths. The library is designed for ease of use but also easily maintainable and extensible code. In this paper we introduce the main features and structure of the deeptime software. Deeptime can be found under https://deeptime-ml.github.io/.

Brain oscillatory correlates of visuomotor adaptive learning

Sensorimotor adaptation involves the recalibration of the mapping between motor command and sensory feedback in response to movement errors. Although adaptation operates within individual movements on a trial-to-trial basis, it can also undergo learning when adaptive responses improve over the course of many trials. Brain oscillatory activities related to these “adaptation” and “learning” processes remain unclear. The main reason for this is that previous studies principally focused on the beta band, which confined the outcome message to trial-to-trial adaptation. To provide a wider understanding of adaptive learning, we decoded visuomotor tasks with constant, random or no perturbation from EEG recordings in different bandwidths and brain regions using a multiple kernel learning approach. These different experimental tasks were intended to separate trial-to-trial adaptation from the formation of the new visuomotor mapping across trials. We found changes in EEG power in the post-movement period during the course of the visuomotor-constant rotation task, in particular an increased (i) theta power in prefrontal region, (ii) beta power in supplementary motor area, and (iii) gamma power in motor regions. Classifying the visuomotor task with constant rotation versus those with random or no rotation, we were able to relate power changes in beta band mainly to trial-to-trial adaptation to error while changes in theta band would relate rather to the learning of the new mapping. Altogether, this suggested that there is a tight relationship between modulation of the synchronization of low (theta) and higher (essentially beta) frequency oscillations in prefrontal and sensorimotor regions, respectively, and adaptive learning.

Rule Ensemble Learning Using Hierarchical Kernels in Structured Output Spaces

The goal in Rule Ensemble Learning (REL) is simultaneous discovery of a small set of simple rules and their optimal weights that lead to good generalization. Rules are assumed to be conjunctions of basic propositions concerning the values taken by the input features. It has been shown that rule ensembles for classification can be learnt optimally and efficiently using hierarchical kernel learning approaches that explore the exponentially large space of conjunctions by exploiting its hierarchical structure. The regularizer employed penalizes large features and thereby selects a small set of short features. In this paper, we generalize the rule ensemble learning using hierarchical kernels (RELHKL) framework to multi class structured output spaces. We build on the StructSVM model for sequence prediction problems and employ a ρ-norm hierarchical regularizer for observation features and a conventional 2-norm regularizer for state transition features. The exponentially large feature space is searched using an active set algorithm and the exponentially large set of constraints are handled using a cutting plane algorithm. The approach can be easily extended to other structured output problems. We perform experiments on activity recognition datasets which are prone to noise, sparseness and skewness. We demonstrate that our approach outperforms other approaches.

MKL-GRNI: A parallel multiple kernel learning approach for supervised inference of large-scale gene regulatory networks.

High throughput multi-omics data generation coupled with heterogeneous genomic data fusion are defining new ways to build computational inference models. These models are scalable and can support very large genome sizes with the added advantage of exploiting additional biological knowledge from the integration framework. However, the limitation with such an arrangement is the huge computational cost involved when learning from very large datasets in a sequential execution environment. To overcome this issue, we present a multiple kernel learning (MKL) based gene regulatory network (GRN) inference approach wherein multiple heterogeneous datasets are fused using MKL paradigm. We formulate the GRN learning problem as a supervised classification problem, whereby genes regulated by a specific transcription factor are separated from other non-regulated genes. A parallel execution architecture is devised to learn a large scale GRN by decomposing the initial classification problem into a number of subproblems that run as multiple processes on a multi-processor machine. We evaluate the approach in terms of increased speedup and inference potential using genomic data from Escherichia coli, Saccharomyces cerevisiae and Homo sapiens. The results thus obtained demonstrate that the proposed method exhibits better classification accuracy and enhanced speedup compared to other state-of-the-art methods while learning large scale GRNs from multiple and heterogeneous datasets.

Multiple Kernel k-Means With Low-Rank Neighborhood Kernel

Multiple kernel clustering algorithms achieve promising performances by exploring the complementary information from kernel matrices corresponding to each data view. Most of the existing methods aim to construct a consensus kernel for the afterward clustering. However, they ignore that the desired kernel is supposed to reveal the cluster structure among samples and thus to be low rank. As a consequence, the corresponding clustering performance could decrease. To address this issue, we propose a low-rank kernel learning approach for multiple kernel clustering. Specifically, instead of regularizing the consensus kernel with low-rank constraints, we use a re-parameterize scheme for the kernel matrix. Meanwhile, the consensus kernel is located in the neighborhood area of the linear combination of base kernels. An alternate optimization strategy is designed to solve the resulting optimization problem. We evaluate the proposed method on 13 benchmark datasets with 9 state-of-the-art algorithms. As is demonstrated by experimental results, our proposed algorithm achieves superior clustering scores against the compared algorithms on the reported popular multiple kernel datasets.

Deep Kernel machines: a survey

The emergence of deep learning frameworks paves the way for achieving higher-level data abstractions and possess the potential in consolidating both supervised and unsupervised learning paradigms. Researchers have made many successful explorations in the field of deep learning, with applications in the fields of face recognition, text mining, language translation, image prediction, and action recognition. Kernel machines act as a bridge between the linearity and nonlinearity for many machine learning algorithms such as support vector machines, extreme learning machines, and core vector machines. These Kernel machines play a vital role in mapping the data in the input space to a Kernel-induced high-dimensional feature space to obtain a better distribution of the data. In this Kernel-induced high-dimensional feature space, the distribution of data points will be more amenable to the classification problem under consideration. The Kernel trick facilitates in transforming the machine learning algorithms that require only inner product computations between the data vectors into a Kernel-based approach by selecting an appropriate Kernel function. In Kernel-based approaches, the Kernel functions can thus be utilized for accomplishing the inner product computations between the transformed data vectors in an implicitly defined Kernel-induced feature space. Unlike neural networks, the Kernel machines guarantee structural risk minimization and global optimal solutions. Also, the Kernel machines exhibit capabilities such as theoretical tractability and excellent performance in practical applications. These attempts motivated the researchers towards utilizing the emerging trends of deep learning with Kernel methods for building deep Kernel machines. Researchers integrate Kernel methods and deep learning networks for maintaining their advantages and make up their limitations, then apply the deep Kernel learning approaches for improving the performance of the learning algorithm in different applications. Different ways of building deep Kernel machines by integrating the Kernel methods and deep learning architectures include utilizing Kernel machines as the final classifier of deep learning networks, Kernelization in deep neural networks for better feature enrichment, and building deep Kernel machines by utilizing deep or multiple Kernels in different tasks. This survey attempts to provide an overview of different approaches in building several deep Kernel learning architectures for enhancing the learning algorithm properties and their performance in practical applications.

Online kernel classification with adjustable bandwidth using control-based learning approach

In this paper, a novel control-based kernel learning approach is proposed for inferring online binary classification tasks. Following a carefully designed alternating optimization scheme, the learning problems are transformed into two optimal feedback control problems for a series of linear, controllable systems. Model parameters including weights and kernel bandwidth can be efficiently updated by solving the control problems. These consequently lead to our control-based adaptive online kernel classification algorithm (CAOKC). The bandwidth, although nonlinear in our model, can still be updated accurately after linearization. Thus, compared with the existing benchmark algorithms with fixed kernels, the CAOKC algorithm is able to achieve a more adaptive, robust classification performance with better prediction accuracy by regarding the bandwidth as an adjustable parameter. The results presented in this paper also demonstrate how optimal control can provide novel insights and be an effective approach for addressing various learning tasks. Numerical results on benchmark synthetic and realistic datasets are provided to illustrate our method.

Deep kernel learning in extreme learning machines

Emergence of extreme learning machine as a breakneck learning algorithm has marked its prominence in solitary hidden layer feed-forward networks. Kernel-based extreme learning machine (KELM) reflected its efficiency in diverse applications where feature mapping functions of hidden nodes are concealed from users. The conventional KELM algorithms involve only solitary layer of kernels, thereby emulating shallow learning architectures for its feature transformation. Trend in migrating shallow-based learning models into deep learning architectures opens up a new outlook for machine learning domains. This paper attempts to bestow deep kernel learning approach in a conventional shallow architecture. The emerging arc-cosine kernels possess the potential to mimic the prevailing deep layered frameworks to a greater extent. Unlike other kernels such as linear, polynomial and Gaussian, arc-cosine kernels have a recursive nature by itself and have the potential to express multilayer computation in learning models. This paper explores the possibility of building a new deep kernel machine with extreme learning machine and multilayer arc-cosine kernels. This framework outperforms conventional KELM and deep support vector machine in terms of training time and accuracy.

Integration of Drug Perturbed Dose Dependent Response Data in Predictive Computational Models of Chemotherapeutic Response Using Multiple‐Kernel‐Learning

Recently there has been a great deal of work to understand the dynamics of genome expression under small molecule perturbation. This has been demonstrated by the recent expansion of the Connectivity Map (CMAP) ( www.broadinstitute.org/connectivity‐map‐cmap) with roughly 5000 small molecule signatures. While this has proven to be invaluable resource it is still limited due to many drugs being constrained to a handful of cancer cell genomes, because of this, it is difficult to leverage these datasets to make an accurate and robust predictive model. Outside CMAP there is limited data that systematically characterize small molecule induced genomic changes, however, recently Monks et al. (Cancer Res 2018 DEC 15) has made available data in the NCI60 cell line panel for treated and untreated cell lines at 2, 6, and 24hr and two different dosing concentrations for 15 anti‐cancer agents. In our work we explore how each dataset can be optimally leveraged to build a predictive model of cell IC50 within each of the 15 drugs using a multiple kernel learning approach. With this method we can reach correlations as high as 0.88 (p ~1e−6) between predicted and measured IC50 with an average correlation of 0.57 on an independent test set. Additionally, perturbation with higher drug dose yields better individual models with average correlation around 0.451 compared to non‐treated data of 0.391 suggesting drug induced genomic changes are an important predictor of drug performance. Furthermore, we present a novel method that derives signatures present in perturbed data that can be captured in basal expression data. This approach to selecting features yields model performance that is equal or better than a signature derived basally alone. Our work demonstrates the importance of both basal and dynamic gene expression in modeling and understanding of drug response. We hope this work motivates further experiments to characterize drug induced gene dynamics for both modeling applications and understanding the underlying genomic influences in drug response.Support or Funding InformationShipley Chair in Comparative Oncology

Features spaces and a learning system for structural-temporal data, and their application on a use case of real-time communication network validation data.

The service quality and system dependability of real-time communication networks strongly depends on the analysis of monitored data, to identify concrete problems and their causes. Many of these can be described by either their structural or temporal properties, or a combination of both. As current research is short of approaches sufficiently addressing both properties simultaneously, we propose a new feature space specifically suited for this task, which we analyze for its theoretical properties and its practical relevance. We evaluate its classification performance when used on real-world data sets of structural-temporal mobile communication data, and compare it to the performance achieved of feature representations used in related work. For this purpose we propose a system which allows the automatic detection and prediction of classes of pre-defined sequence behavior, greatly reducing costs caused by the otherwise required manual analysis. With our proposed feature spaces this system achieves a precision of more than 93% at recall values of 100%, with an up to 6.7% higher effective recall than otherwise similarly performing alternatives, notably outperforming alternative deep learning, kernel learning and ensemble learning approaches of related work. Furthermore the supported system calibration allows separating reliable from unreliable predictions more effectively, which is highly relevant for any practical application.

P.305 Inflammation and neuroimaging differentiate bipolar and unipolar depression: a multiple kernel learning approach

P.305 Inflammation and neuroimaging differentiate bipolar and unipolar depression: a multiple kernel learning approach