Kernel-based Model Research Articles

Improved nonlinear process monitoring based on ensemble KPCA with local structure analysis

A new nonlinear process monitoring algorithm called ensemble local kernel principal component analysis (ELKPCA) is proposed. Conventionally, the performance of kernel-based model depends on the width parameter selected empirically in Gaussian kernel function, which means a single parameter corresponds to a single model. Once a poor width parameter is decided, a single kernel-based model may be only effective for part faults. As a typical kernel-based method, the local kernel principal component analysis (LKPCA), considering both global and local structure information of the original data, faces the problem of width parameter selection as well. Since a single model is one-sided, an available way is to combine different single models and take advantage of them. The ensemble kernel principal component analysis (EKPCA) uses single KPCA models as its sub-models and ensemble learning approach is used to combine them. Due to inherit drawbacks from KPCA models, EKPCA only preserves global structure information of data, but ignores important local structure information. In this paper, to solve the above issues, both LKPCA and EKPCA is unified in the proposed framework. First, single LKPCA models are chosen instead and combined by using ensemble learning strategy. Then two monitoring statistics are turned into fault probabilities through Bayesian inference approach and weighted combination strategy, which makes the monitoring behavior easier and more clear. The result shows that ELKPCA model can not only take advantage of sub-LKPCA models effectively, allowing it selecting width parameter more easily and stably, but also retain both global and local structure information from input data by introducing the local structure analysis in the EKPCA model. Case studies on synthetic example and Tennessee Eastman process demonstrate the proposed method outperforms LKPCA and EKPCA and enhances monitoring performance significantly.

Estimation of inefficiency in stochastic frontier models: a Bayesian kernel approach

We propose a kernel-based Bayesian framework for the analysis of stochastic frontiers and efficiency measurement. The primary feature of this framework is that the unknown distribution of inefficiency is approximated by a transformed Rosenblatt-Parzen kernel density estimator. To justify the kernel-based model, we conduct a Monte Carlo study and also apply the model to a panel of U.S. large banks. Simulation results show that the kernel-based model is capable of providing more precise estimation and prediction results than the commonly-used exponential stochastic frontier model. The Bayes factor also favors the kernel-based model over the exponential model in the empirical application.

Gaussian-kernel-based adaptive critic design using two-phase value iteration

Adaptive critic design is an efficient way to learn optimal action policies on-line, in which a critic network plays an important role to estimate value functions. Because of its good generalization and easy configuration, kernel-based method is prevalently introduced to the construction of critic network. Conventionally the hyper-parameters of kernel-based model need to be predetermined, but empirical selection of them may mislead kernel-based regression with an improper modeling hypothesis space. To tackle this problem, a two-phase iteration of value function approximation and hyper-parameters optimization for Gaussian-kernel based adaptive critic design (GK-ACD) is presented in this paper, which not only approximates the value functions, but also updates the hyper-parameters on-line. Since the two phases are strong coupling, the theoretical proof based on stochastic approximation derives the sufficient conditions guaranteeing the convergence, and points out that the algorithm’s performance mostly relies on the design of coordinated learning rates w.r.t. the two phases. Finally a series of numerical experiments are given to discuss the necessity of two-phase updates and the performance under the coordinated learning rates.

Software effort estimation using FAHP and weighted kernel LSSVM machine

In the life cycle of software product development, the software effort estimation (SEE) has always been a critical activity. The researchers have proposed numerous estimation methods since the inception of software engineering as a research area. The diversity of estimation approaches is very high and increasing, but it has been interpreted that no single technique performs consistently for each project and environment. Multi-criteria decision-making (MCDM) approach generates more credible estimates, which is subjected to expert’s experience. In this paper, a hybrid model has been developed to combine MCDM (for handling uncertainty) and machine learning algorithm (for handling imprecision) approach to predict the effort more accurately. Fuzzy analytic hierarchy process (FAHP) has been used effectively for feature ranking. Ranks generated from FAHP have been integrated into weighted kernel least square support vector machine for effort estimation. The model developed has been empirically validated on data repositories available for SEE. The combination of weights generated by FAHP and the radial basis function (RBF) kernel has resulted in more accurate effort estimates in comparison with bee colony optimisation and basic RBF kernel-based model.

Kafnets: Kernel-based non-parametric activation functions for neural networks

Neural networks are generally built by interleaving (adaptable) linear layers with (fixed) nonlinear activation functions. To increase their flexibility, several authors have proposed methods for adapting the activation functions themselves, endowing them with varying degrees of flexibility. None of these approaches, however, have gained wide acceptance in practice, and research in this topic remains open. In this paper, we introduce a novel family of flexible activation functions that are based on an inexpensive kernel expansion at every neuron. Leveraging several properties of kernel-based models, we propose multiple variations for designing and initializing these kernel activation functions (KAFs), including a multidimensional scheme allowing to nonlinearly combine information from different paths in the network. The resulting KAFs can approximate any mapping defined over a subset of the real line, either convex or non-convex. Furthermore, they are smooth over their entire domain, linear in their parameters, and they can be regularized using any known scheme, including the use of ℓ1 penalties to enforce sparseness. To the best of our knowledge, no other known model satisfies all these properties simultaneously. In addition, we provide an overview on alternative techniques for adapting the activation functions, which is currently lacking in the literature. A large set of experiments validates our proposal.

Exploration in Mapping Kernel-Based Home Range Models from Remote Sensing Imagery with Conditional Adversarial Networks

Kernel-based home range models are widely-used to estimate animal habitats and develop conservation strategies. They provide a probabilistic measure of animal space use instead of assuming the uniform utilization within an outside boundary. However, this type of models estimates the home ranges from animal relocations, and the inadequate locational data often prevents scientists from applying them in long-term and large-scale research. In this paper, we propose an end-to-end deep learning framework to simulate kernel home range models. We use the conditional adversarial network as a supervised model to learn the home range mapping from time-series remote sensing imagery. Our approach enables scientists to eliminate the persistent dependence on locational data in home range analysis. In experiments, we illustrate our approach by mapping the home ranges of Bar-headed Geese in Qinghai Lake area. The proposed framework outperforms all baselines in both qualitative and quantitative evaluations, achieving visually recognizable results and high mapping accuracy. The experiment also shows that learning the mapping between images is a more effective way to map such complex targets than traditional pixel-based schemes.

Spectral Reflectance Estimation Using Gaussian Processes and Combination Kernels.

This paper explores hyperspectral reflectance factor estimation using Gaussian process regression with multispectral- and trichromatic measurements. Estimations are performed in visible- (400-700 nm) or visible-near infrared (400-980 nm) wavelength ranges using the learning-based approach, where sensor and light spectral characteristics are not required. We first construct new estimation models via Gaussian processes, show connection to previous kernel-based models, and then evaluate new models by using marginal likelihood optimization within the probabilistic interpretation. By using standard spectral ensembles and several images in experiments, we evaluate new models with anisotropic radial- and combination kernels (process covariance), marginal likelihood optimization (parameter selection), as well as with input data transformations (pre-processing). Several new Gaussian process models provide spectral accuracy improvements for simulated and real data, when compared with the previous kernel-based models. Most versatile new model is using spectral subspace coordinate learning and combination kernels, and can be efficiently optimized via marginal likelihood. Preliminary results suggest that new models provide uncertainty estimates, which can be used for iterative training set augmentation.

Weighted functional linear regression models for gene-based association analysis.

Functional linear regression models are effectively used in gene-based association analysis of complex traits. These models combine information about individual genetic variants, taking into account their positions and reducing the influence of noise and/or observation errors. To increase the power of methods, where several differently informative components are combined, weights are introduced to give the advantage to more informative components. Allele-specific weights have been introduced to collapsing and kernel-based approaches to gene-based association analysis. Here we have for the first time introduced weights to functional linear regression models adapted for both independent and family samples. Using data simulated on the basis of GAW17 genotypes and weights defined by allele frequencies via the beta distribution, we demonstrated that type I errors correspond to declared values and that increasing the weights of causal variants allows the power of functional linear models to be increased. We applied the new method to real data on blood pressure from the ORCADES sample. Five of the six known genes with P < 0.1 in at least one analysis had lower P values with weighted models. Moreover, we found an association between diastolic blood pressure and the VMP1 gene (P = 8.18×10−6), when we used a weighted functional model. For this gene, the unweighted functional and weighted kernel-based models had P = 0.004 and 0.006, respectively. The new method has been implemented in the program package FREGAT, which is freely available at https://cran.r-project.org/web/packages/FREGAT/index.html.

Selective Ensemble Least Square Support Vector Machine with Its Application

Kernel-based modeling methods have been used widely to estimate some difficulty-to-measure quality or efficient indices at different industrial applications. Least square support vector machine (LSSVM) is one of the popular ones. However, its learning parameters, i.e., kernel parameter and regularization parameter, are sensitive to the training data and the model’s prediction performance. Ensemble modeling method can improve the generalization performance and reliability of the soft measuring model. Aim at these problems, a new adaptive selective ensemble (SEN) LSSVM (SEN-LSSVM) algorithm is proposed by using multiple learning parameters. Candidate regularization parameters and candidate kernel parameters are used to construct many of candidate sub-sub-models based on LSSVM. These sub-sub-models based on the same kernel parameter are selected and combined as candidate SEN-sub-models by using branch and bound-based SEN (BBSEN). By employing BBSEN at the second time, these SEN-sub-models based on different kernel parameters are used to obtain the final soft measuring model. Thus, multiple kernel and regularization parameters are adaptive selected for building SEN-LSSVM model. UCI benchmark datasets and mechanical frequency spectral data are used to validate the effectiveness of this method.

Predicting bull fertility using genomic data and biological information

The genomic prediction of unobserved genetic values or future phenotypes for complex traits has revolutionized agriculture and human medicine. Fertility traits are undoubtedly complex traits of great economic importance to the dairy industry. Although genomic prediction for improved cow fertility has received much attention, bull fertility largely has been ignored. The first aim of this study was to investigate the feasibility of genomic prediction of sire conception rate (SCR) in US Holstein dairy cattle. Standard genomic prediction often ignores any available information about functional features of the genome, although it is believed that such information can yield more accurate and more persistent predictions. Hence, the second objective was to incorporate prior biological information into predictive models and evaluate their performance. The analyses included the use of kernel-based models fitting either all single nucleotide polymorphisms (SNP; 55K) or only markers with presumed functional roles, such as SNP linked to Gene Ontology or Medical Subject Heading terms related to male fertility, or SNP significantly associated with SCR. Both single- and multikernel models were evaluated using linear and Gaussian kernels. Predictive ability was evaluated in 5-fold cross-validation. The entire set of SNP exhibited predictive correlations around 0.35. Neither Gene Ontology nor Medical Subject Heading gene sets achieved predictive abilities higher than their counterparts using random sets of SNP. Notably, kernel models fitting significant SNP achieved the best performance with increases in accuracy up to 5% compared with the standard whole-genome approach. Models fitting Gaussian kernels outperformed their counterparts fitting linear kernels irrespective of the set of SNP. Overall, our findings suggest that genomic prediction of bull fertility is feasible in dairy cattle. This provides potential for accurate genome-guided decisions, such as early culling of bull calves with low SCR predictions. In addition, exploiting nonlinear effects through the use of Gaussian kernels together with the incorporation of relevant markers seems to be a promising alternative to the standard approach. The inclusion of gene set results into prediction models deserves further research.

Kernel-based orthogonal quantile regression model

Kernel-based orthogonal quantile regression model

Computational-experimental approach to drug-target interaction mapping: A case study on kinase inhibitors.

Due to relatively high costs and labor required for experimental profiling of the full target space of chemical compounds, various machine learning models have been proposed as cost-effective means to advance this process in terms of predicting the most potent compound-target interactions for subsequent verification. However, most of the model predictions lack direct experimental validation in the laboratory, making their practical benefits for drug discovery or repurposing applications largely unknown. Here, we therefore introduce and carefully test a systematic computational-experimental framework for the prediction and pre-clinical verification of drug-target interactions using a well-established kernel-based regression algorithm as the prediction model. To evaluate its performance, we first predicted unmeasured binding affinities in a large-scale kinase inhibitor profiling study, and then experimentally tested 100 compound-kinase pairs. The relatively high correlation of 0.77 (p < 0.0001) between the predicted and measured bioactivities supports the potential of the model for filling the experimental gaps in existing compound-target interaction maps. Further, we subjected the model to a more challenging task of predicting target interactions for such a new candidate drug compound that lacks prior binding profile information. As a specific case study, we used tivozanib, an investigational VEGF receptor inhibitor with currently unknown off-target profile. Among 7 kinases with high predicted affinity, we experimentally validated 4 new off-targets of tivozanib, namely the Src-family kinases FRK and FYN A, the non-receptor tyrosine kinase ABL1, and the serine/threonine kinase SLK. Our sub-sequent experimental validation protocol effectively avoids any possible information leakage between the training and validation data, and therefore enables rigorous model validation for practical applications. These results demonstrate that the kernel-based modeling approach offers practical benefits for probing novel insights into the mode of action of investigational compounds, and for the identification of new target selectivities for drug repurposing applications.

Superpixel-Based Adaptive Kernel Selection for Angular Effect Normalization of Remote Sensing Images With Kernel Learning

Considering that satellites rarely acquire data from the exact nadir direction, angular effect normalization needs to be conducted as an important preprocessing step to correct reflectance observations from off-nadir directions into the nadir direction. Kernel-based bidirectional reflectance distribution function models have been employed for angular effect correction. The kernels used in the model are often predetermined and fixed for an entire image. However, the fixed kernels are unable to accommodate the various reflective characteristics of different ground cover types present in the imaged area. In this paper, we propose a kernel learning procedure that enables the flexible selection of kernels for different land cover types within a scene. The kernels are selected from kernel dictionaries that contain multiple candidate kernels. The selection is conducted on the superpixel level instead of the pixel level in order to reduce within-class variation and overcome the overfitting problem. Experiments are conducted on multiangular images acquired by the Sentinel-2A satellite over a rural area in southeastern Australia. Cross-validation results show that the proposed method is able to adaptively select appropriate kernels for different land cover types, leading to an improved performance for image normalization.

Robust kernel-based model reference adaptive control for unstable aircraft

In this article, a robust kernel-based model reference adaptive control is proposed for an unstable nonlinear aircraft. The heart of the proposed kernel-based model reference adaptive control scheme comprises an offline neural identifier and an online neural controller. In the offline neural identifier, the kernel-based unified extreme learning machine algorithm is used to identify the aircraft model with the available input–output data in a finite time interval. The finite time interval is selected to avoid the response of the unstable aircraft growing unbounded. In the kernel-based unified extreme learning machine, the hidden layer feature mapping is determined by the kernel matrix. However, the unified extreme learning machine is a batch learning algorithm and is not suitable for the online control learning. To solve the problem, a recursive version of the unified extreme learning machine is developed in this study. Based on a given reference model and the identified model, the recursive version of the unified extreme learning machine algorithm is applied to construct the online control law to compensate for the changes in the aircraft dynamics or characteristics. The performance of the proposed kernel-based model reference adaptive control scheme is validated through the simulation studies of a locally nonlinear longitudinal high-performance aircraft. Simulation studies are also compared with a model reference adaptive control based on the back-propagation algorithm and a model reference adaptive control based on the basic extreme learning machine algorithm in terms of the identification and tracking abilities. The results show that the proposed kernel-based model reference adaptive control can achieve better identification and tracking performance.

Gas Source Parameter Estimation Using Machine Learning in WSNs

This paper introduces an original clusterized framework for the detection and estimation of the parameters of multiple gas sources in wireless sensor networks. The proposed method consists of defining a kernel-based detector that can detect gas releases within the network's clusters using concentration measures collected regularly from the network. Then, we define two kernel-based models that accurately estimate the gas release parameters, such as the sources locations and their release rates, using the collected concentrations.

Biobjective Nonnegative Matrix Factorization: Linear Versus Kernel-Based Models

Nonnegative matrix factorization (NMF) is a powerful class of feature extraction techniques that has been successfully applied in many fields, namely in signal and image processing. Current NMF techniques have been limited to a single-objective problem in either its linear or nonlinear kernel-based formulation. In this paper, we propose to revisit the NMF as a multi-objective problem, in particular a bi-objective one, where the objective functions defined in both input and feature spaces are taken into account. By taking the advantage of the sum-weighted method from the literature of multi-objective optimization, the proposed bi-objective NMF determines a set of nondominated, Pareto optimal, solutions instead of a single optimal decomposition. Moreover, the corresponding Pareto front is studied and approximated. Experimental results on unmixing real hyperspectral images confirm the efficiency of the proposed bi-objective NMF compared with the state-of-the-art methods.

Kernel-based models for prediction of cement compressive strength

This paper employs three different kernel-based models--support vector regression (SVR), relevance vector machine (RVM) and Gaussian process regression (GPR)--for the prediction of cement compressive strength. The input variables for the model are taken as C3S (%), SO3 (%), Alkali (%) and Blaine (cm2/g), while the output is 28-day cement compressive strength (N/mm2) of the cement. The hyperparameters of the SVR are obtained using two different metaheuristic optimization algorithms--particle swarm optimization (PSO) and symbiotic organism search (SOS). Trial-and-error-based approach is used for arriving at the hyperparameters of RVM and GPR. The compressive strength predicted using different kernel-based models is also compared with that obtained from ANN and fuzzy logic models reported in the literature. The performance of the different kernel-based models is benchmarked using six different error indices and residual analysis. The performance of the kernel-based models is found to be at par with ANN. The better generalization capability and excellent empirical performance of the kernel-based models overcome the disadvantages associated with ANN and provide a good tool for the prediction of the cement compressive strength.

Exploiting syntactic and semantics information for chemical-disease relation extraction.

Identifying chemical–disease relations (CDR) from biomedical literature could improve chemical safety and toxicity studies. This article proposes a novel syntactic and semantic information exploitation method for CDR extraction. The proposed method consists of a feature-based model, a tree kernel-based model and a neural network model. The feature-based model exploits lexical features, the tree kernel-based model captures syntactic structure features, and the neural network model generates semantic representations. The motivation of our method is to fully utilize the nice properties of the three models to explore diverse information for CDR extraction. Experiments on the BioCreative V CDR dataset show that the three models are all effective for CDR extraction, and their combination could further improve extraction performance. Database URL: http://www.biocreative.org/resources/corpora/biocreative-v-cdr-corpus/.

Kernel-based calibration methods combined with multivariate feature selection to improve accuracy of near-infrared spectroscopic analysis

We present kernel-based calibration models combined with multivariate feature selection for complex quantitative near-infrared (NIR) spectroscopic analysis of three different types of sample. Because the spectra include hundreds of features (variables), an optimal selection of features that provide relevant information for target analysis improves the accuracy of spectroscopic analysis. For this purpose, we combined feature selection with kernel partial least squares regression and kernel support vector regression (K-SVR) by evaluating ranking of the features based on their variable importance in projection scores and weight vector coefficients, respectively. Then, the methods were applied to identify components in three datasets of NIR spectra. The kernel-based models without feature selection and the kernel-based models with other feature selection methods were also used for comparison. K-SVR combined with feature selection was effective when the spectral features of samples were complex and recognition of minute spectral variation was necessary for modeling. The combination of feature selection and kernel calibration model can improve the accuracy of spectral analysis by keeping optimal features.

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

Adaptive Dynamic Programming (ADP) with critic-actor architecture is an effective way to perform online learning control. To avoid the subjectivity in the design of a neural network that serves as a critic network, kernel-based adaptive critic design (ACD) was developed recently. There are two essential issues for a static kernel-based model: how to determine proper hyperparameters in advance and how to select right samples to describe the value function. They all rely on the assessment of sample values. Based on the theoretical analysis, this paper presents a two-phase simultaneous learning method for a Gaussian-kernel-based critic network. It is able to estimate the values of samples without infinitively revisiting them. And the hyperparameters of the kernel model are optimized simultaneously. Based on the estimated sample values, the sample set can be refined by adding alternatives or deleting redundances. Combining this critic design with actor network, we present a Gaussian-kernel-based Adaptive Dynamic Programming (GK-ADP) approach. Simulations are used to verify its feasibility, particularly the necessity of two-phase learning, the convergence characteristics, and the improvement of the system performance by using a varying sample set.