Kernel-based Learning Research Articles

Deterministic error bounds for kernel-based learning techniques under bounded noise

We consider the problem of reconstructing a function from a finite set of noise-corrupted samples. Two kernel algorithms are analyzed, namely kernel ridge regression and ɛ-support vector regression. By assuming the ground-truth function belongs to the reproducing kernel Hilbert space of the chosen kernel, and the measurement noise affecting the dataset is bounded, we adopt an approximation theory viewpoint to establish deterministic, finite-sample error bounds for the two models. Finally, we discuss their connection with Gaussian processes and two numerical examples are provided. In establishing our inequalities, we hope to help bring the fields of non-parametric kernel learning and system identification for robust control closer to each other.

A novel machine learning-based electricity price forecasting model based on optimal model selection strategy

Current electricity price forecasting models rely on only simple hybridizations of data preprocessing and optimization methods while ignoring the significance of adaptive data preprocessing and effective optimization and selection strategies to obtain optimal models that improve the forecasting performance. To solve these problems, this study develops an improved electricity price forecasting model that offers the advantages of adaptive data preprocessing, advanced optimization method, kernel-based model, and optimal model selection strategy. Specifically, the adaptive parameter-based variational mode decomposition technology is proposed to provide desirable data preprocessing results, and a leave-one-out optimization strategy based on the chaotic sine cosine algorithm is proposed and applied to develop optimal kernel-based extreme learning machine models. In addition, a newly proposed optimal model selection strategy is applied to determine the developed model that provides the most desirable forecasting result. Numerical results show that the developed model's performance metrics were best, and the average values of mean absolute error, root mean square error, mean absolute percentage error, index of agreement, and Theil's inequality coefficient in four datasets are 0.5121, 0.7607, 0.5722%, 0.9997 and 0.0041, respectively, which imply that the developed model is a promising, applicable and effective electricity price forecasting technique in the real electricity market.

Development of urination recognition technology based on Support Vector Machine using a smart band.

The purpose of this study was to explore the feasibility of a urination management system by developing a smart band-based algorithm that recognizes the urination interval of women. We designed a device that recognizes the time and interval of urination based on the patient’s specific posture and posture changes. The technology used for recognition applied the Radial Basis Function kernel-based Support Vector Machine, a teaching and learning method that facilitates multidimensional analysis by simultaneously judging the characteristics of complex learning data. In order to evaluate the performance of the proposed recognition technique, we compared actual urination and device-sensed urination. An experiment was performed to evaluate the performance of the recognition technology proposed in this study. The efficacy of smart band monitoring urination was evaluated in 10 female patients without urination problems. The entire experiment was performed over a total of 3 days. The average age of the participants was 28.73 years (26–34 years), and there were no signs of dysuria. The final accuracy of the algorithm was calculated based on clinical guidelines for urologists. The experiment showed a high average accuracy of 91.0%, proving the robustness of the proposed algorithm. This urination behavior recognition technique shows high accuracy and can be applied in clinical settings to characterize urination patterns in female patients. As wearable devices develop and become more common, algorithms that detect specific sequential body movement patterns that reflect specific physiological behaviors could become a new methodology to study human physiological behavior.

Cooperative classification method for multi-fault diagnosis of machinery based on parameterized wavelet kernel extreme learning and sparse representation

Owing to complicated manual design and feature extraction processes of mechanical multi-fault classification and recognition methods, single-class classifiers do not adapt well to the high dimensionality and high self-similarity among mechanical vibration signals with nonlinear characteristics, resulting in insufficient classification accuracy. Combining the adaptive learning and feature extraction capabilities of the wavelet kernel-based extreme learning machine (WKELM) with the sparse representation classifier (SRC), this paper proposes a hybrid artificial bee colony (ABC)-optimized Morlet wavelet kernel-based extreme learning machine (ABC-MWKELM) with SRC. The two-level classifier is capable of cooperatively classifying multiple mechanical faults. First, a Morlet wavelet kernel-based extreme learning machine (MWKELM) model that matches the kernel function construction condition is presented. An ABC optimization algorithm is adopted to optimize critical parameters of the MWKELM model, thereby improving the accuracy and generalization ability of the model. In addition, the ABC algorithm is used to optimize the MWKELM model as a pre-classifier for classifying test samples. According to the posterior probability of each training sample category output, the threshold decision method selects candidate categories with a high confidence coefficient. When the preset meets the threshold, the classification result of the ABC-optimized MWKELM is directly adopted. Otherwise, the candidate training samples are used to construct a dictionary and the SRC is used to classify the test samples further. The experimental results demonstrate that compared with the ELM, WKELM, (ELM-SRC), and radial basis function (RBF)-KELM and SRC (RBFKELM-SRC), the method presented in this paper has significant advantages in terms of accuracy of mechanical multi-fault classification. The validity and applicability of this method in mechanical multi-fault classification were verified.

Molecular dynamics simulation and machine learning of mechanical response in non-equiatomic FeCrNiCoMn high-entropy alloy

Abstract High-entropy alloys (HEAs) have attracted a wide range of academic interest for their promising properties as structural materials, among which the equiatomic FeCrNiCoMn HEAs have been reported to possess a series of superior properties. However, one may have to change the alloy composition from the equiatomic composition to improve a specific material property. In this study, molecular dynamics simulation combined with machine learning methods was used to study the mechanical properties of non-equiatomic FeCrNiCoMn HEAs. A database describing the relationship between materials composition and mechanical properties was established based on a tensile test of 300 HEA single-crystal samples by MD simulation. We investigated and compared three ML models for the learning task of yield stress, including support vector machine (SVM), kernel-based extreme learning machine (KELM), and deep neural network (DNN). It was found that the DNN model outperformed others for the binary classification of yield stress. The elemental composition strategy was used to guide the design of polycrystal FeCrNiCoMn samples, and the accuracy of the DNN model was further verified by the polycrystal samples. We show in this contribution that computational study combined with machine learning method can provide instructive guidance for the design of high-strength HEA and accelerate the development of new alloy materials.

Tourism demand forecasting: An ensemble deep learning approach

The availability of tourism-related big data increases the potential to improve the accuracy of tourism demand forecasting but presents significant challenges for forecasting, including curse of dimensionality and high model complexity. A novel bagging-based multivariate ensemble deep learning approach integrating stacked autoencoder and kernel-based extreme learning machine (B-SAKE) is proposed to address these challenges in this study. By using historical tourist arrival data, economic variable data, and search intensity index (SII) data, we forecast tourist arrivals in Beijing from four countries. The consistent results of multiple schemes suggest that our proposed B-SAKE approach outperforms the benchmark models in terms of level accuracy, directional accuracy, and even statistical significance. Both bagging and stacked autoencoder can effectively alleviate the challenges brought by tourism big data and improve the forecasting performance of the models. The ensemble deep learning model we propose contributes to tourism demand forecasting literature and benefits relevant government officials and tourism practitioners.

A fuzzy-weighted Gaussian kernel-based machine learning approach for body fat prediction

A fuzzy-weighted Gaussian kernel-based machine learning approach for body fat prediction

Prediction on Mechanical Properties of Non-Equiatomic High-Entropy Alloy by Atomistic Simulation and Machine Learning

High-entropy alloys (HEAs) with multiple constituent elements have been extensively studied in the past 20 years, due to their promising engineering application. Previous experimental and computational studies of HEAs focused mainly on equiatomic or near equiatomic HEAs. However, there is probably far more treasure in those non-equiatomic HEAs with carefully designed composition. In this study, the molecular dynamics (MD) simulation combined with machine learning (ML) methods was used to predict the mechanical properties of non-equiatomic CuFeNiCrCo HEAs. A database was established based on a tensile test of 900 HEA single-crystal samples by MD simulation. Eight ML models were investigated and compared for the binary classification learning tasks, ranging from shallow models to deep models. It was found that the kernel-based extreme learning machine (KELM) model outperformed others for the prediction of yield stress and Young’s modulus. The accuracy of the KELM model was further verified by the large-sized polycrystal HEA samples. The results show that computational simulation combined with ML methods is an efficient way to predict the mechanical performance of HEAs, which provides new ideas for accelerating the development of novel alloy materials for engineering applications.

ESCAPED: Efficient Secure and Private Dot Product Framework for Kernel-based Machine Learning Algorithms with Applications in Healthcare

Training sophisticated machine learning models usually requires many training samples. Especially in healthcare settings these samples can be very expensive, meaning that one institution alone usually does not have enough. Merging privacy-sensitive data from different sources is usually restricted by data security and data protection measures. This can lead to approaches that reduce data quality by putting noise onto the variables (e.g., in epsilon-differential privacy) or omitting certain values (e.g., for k-anonymity). Other measures based on cryptographic methods can lead to very time-consuming computations, which is especially problematic for larger multi-omics data. We address this problem by introducing ESCAPED, which stands for Efficient SeCure And PrivatE Dot product framework. ESCAPED enables the computation of the dot product of vectors from multiple sources on a third-party, which later trains kernel-based machine learning algorithms, while neither sacrificing privacy nor adding noise. We have evaluated our framework on drug resistance prediction for HIV-infected people and multi-omics dimensionality reduction and clustering problems in precision medicine. In terms of execution time, our framework significantly outperforms the best-fitting existing approaches without sacrificing the performance of the algorithm. Even though we only present the benefit for kernel-based algorithms, our framework can open up new research opportunities for further machine learning models that require the dot product of vectors from multiple sources.

Polyphase uncertainty analysis through virtual modelling technique

A virtual model aided non-deterministic static analysis (including linear and nonlinear analyses) with polyphase uncertainty is presented in this paper. Within an uncertain system, the polyphase uncertainty integrates both probabilistic and non-probabilistic uncertainties, which is more sophisticated than the conventional uncertainty modelling through a single type. To further improve the computational stableness and robustness of the virtual model, a kernel-based machine learning technique, namely Twin Extended Support Vector Regression (T-X-SVR), is newly developed. The feature of auto-learning is fulfilled through the Bayesian optimization. The proposed approach is capable of providing sufficient statistical information, including the membership functions of mean and standard deviation, fuzzy-valued probabilistic density function (PDF) and cumulative distribution function (CDF) for the upper and lower bounds of the concerned structural response. To demonstrate the effectiveness and computational efficiency of the proposed approach, a verification case, where analytical solutions are available, is tested first. Then, two practically stimulated engineering applications are fully investigated.

Pairwise learning for predicting pollination interactions based on traits and phylogeny

Mutualistic bee–plant interaction networks are a vital part of terrestrial ecosystems. They frequently arise through co-evolutionary processes, which match the traits of both partners, facilitating their interaction. Insights in these interaction mechanisms are vital to be able to manage changing ecosystems. This entails the need for models to predict species interaction networks in general and pollination networks in particular.We show how kernel-based pairwise learning can predict bee–plant interactions based on the traits and the phylogeny of the plant and bee species. The traits and the phylogeny of the plant and bee species proved to be highly predictive. Although the traits were slightly more informative compared to the phylogeny, the best results were obtained by combining both the traits and the phylogeny in the model Notably, the model performance varied greatly depending on whether the goal was to pinpoint missing interactions in the network or to predict interactions for new bee species, new plant species, or both. This issue highlights the importance of proper stratification when fitting biological network prediction models. Our model, however, showed the capacity to generalize beyond the original dataset provided by FlorAbeilles. The model was validated by predicting potentially interacting plant species for the invasive bee species Megachile sculpturalis. Four out of the five plant species identified by the model could be validated based on literature.Our results indicate that pairwise learning has potential as a general method for supervised species interaction prediction. Caution should be taken to validate such models correctly.

Classification with Quantum Measurements

This paper reports a novel method for supervised machine learning based on the mathematical formalism that supports quantum mechanics. The method uses projective quantum measurement as a way of building a prediction function. Specifically, the relationship between input and output variables is represented as the state of a bipartite quantum system. The state is estimated from training samples through an averaging process that produces a density matrix. Prediction of the label for a new sample is made by performing a projective measurement on the bipartite system with an operator, prepared from the new input sample, and applying a partial trace to obtain the state of the subsystem representing the output. The method can be seen as a generalization of Bayesian inference classification and as a type of kernel-based learning method. One remarkable characteristic of the method is that it does not require learning any parameters through optimization. We illustrate the method with different 2-D classification benchmark problems and different quantum information encodings.

Gradient local auto-correlation features for depth human action recognition

Human action classification is a dynamic research topic in computer vision and has applications in video surveillance, human–computer interaction, and sign-language recognition. This paper aims to present an approach for the categorization of depth video oriented human action. In the approach, the enhanced motion and static history images are computed and a set of 2D auto-correlation gradient feature vectors is obtained from them to describe an action. Kernel-based Extreme Learning Machine is used with the extracted features to distinguish the diverse action types promisingly. The proposed approach is thoroughly assessed for the action datasets namely MSRAction3D, DHA, and UTD-MHAD. The approach achieves an accuracy of 97.44% for MSRAction3D, 99.13% for DHA, and 88.37% for UTD-MHAD. The experimental results and analysis demonstrate that the classification performance of the proposed method is considerable and surpasses the state-of-the-art human action classification methods. Besides, from the complexity analysis of the approach, it is turn out that our method is consistent for the real-time operation with low computational complexity.

Progressive Ensemble Kernel-Based Broad Learning System for Noisy Data Classification.

The broad learning system (BLS) is an algorithm that facilitates feature representation learning and data classification. Although weights of BLS are obtained by analytical computation, which brings better generalization and higher efficiency, BLS suffers from two drawbacks: 1) the performance depends on the number of hidden nodes, which requires manual tuning, and 2) double random mappings bring about the uncertainty, which leads to poor resistance to noise data, as well as unpredictable effects on performance. To address these issues, a kernel-based BLS (KBLS) method is proposed by projecting feature nodes obtained from the first random mapping into kernel space. This manipulation reduces the uncertainty, which contributes to performance improvements with the fixed number of hidden nodes, and indicates that manually tuning is no longer needed. Moreover, to further improve the stability and noise resistance of KBLS, a progressive ensemble framework is proposed, in which the residual of the previous base classifiers is used to train the following base classifier. We conduct comparative experiments against the existing state-of-the-art hierarchical learning methods on multiple noisy real-world datasets. The experimental results indicate our approaches achieve the best or at least comparable performance in terms of accuracy.

Kernel Partial Least Square Regression with High Resistance to Multiple Outliers and Bad Leverage Points on Near-Infrared Spectral Data Analysis

Multivariate statistical analysis such as partial least square regression (PLSR) is the common data processing technique used to handle high-dimensional data space on near-infrared (NIR) spectral datasets. The PLSR is useful to tackle the multicollinearity and heteroscedasticity problem that can be commonly found in such data space. With the problem of the nonlinear structure in the original input space, the use of the classical PLSR model might not be appropriate. In addition, the contamination of multiple outliers and high leverage points (HLPs) in the dataset could further damage the model. Generally, HLPs contain both good leverage points (GLPs) and bad leverage points (BLPs); therefore, in this case, removing the BLPs seems relevant since it has a significant impact on the parameter estimates and can slow down the convergence process. On the other hand, the GLPs provide a good efficiency in the model calibration process; thus, they should not be eliminated. In this study, robust alternatives to the existing kernel partial least square (KPLS) regression, which are called the kernel partial robust GM6-estimator (KPRGM6) regression and the kernel partial robust modified GM6-estimator (KPRMGM6) regression are introduced. The nonlinear solution on PLSR was handled through kernel-based learning by nonlinearly projecting the original input data matrix into a high-dimensional feature mapping that corresponded to the reproducing kernel Hilbert spaces (RKHS). To increase the robustness, the improvements on GM6 estimators are presented with the nonlinear PLSR. Based on the investigation using several artificial dataset scenarios from Monte Carlo simulations and two sets from the near-infrared (NIR) spectral dataset, the proposed robust KPRMGM6 is found to be superior to the robust KPRGM6 and non-robust KPLS.

A novel hybrid method based on KELM with SAPSO for fault diagnosis of rolling bearing under variable operating conditions

It’s a great challenge to detect fault types from the measured vibration signals automatically and efficiently under variable running condition. A novel hybrid method based on simulated annealing particle swarm optimization (SAPSO) and improved kernel-based extreme learning machine (IKELM) is proposed for diagnosing rolling bearing faults under variable conditions in the paper. Multi-domain fault features from time domain, frequency domain and time–frequency domain are extracted and combined to reconstruct a high-dimensional feature collection by applying VMD, and isometric feature mapping is adopted to lower the dimension of the high-dimensional feature collection. The parameters of KELM are optimized to adapt for the fault feature identification by using SAPSO method. The proposed SAPSO-IKELM can be applied to diagnose the fault automatically and accurately from the two rolling bearing experiments. The results indicate SAPSO-IKELM could remove dependence on artificial feature selection and reach high classification accuracy under varying conditions.

Comparison between fuzzy kernel k-medoids using radial basis function kernel and polynomial kernel function in hepatitis classification

<span id="docs-internal-guid-10508d4e-7fff-5011-7a0e-441840e858c8"><span>This paper compares the fuzzy kernel k-medoids using radial basis function (RBF) and polynomial kernel function in hepatitis classification. These two kernel functions were chosen due to their popularity in any kernel-based machine learning method for solving the classification task. The hepatitis dataset then used to evaluate the performance of both methods that were expected to provide an accurate diagnosis in patients to obtain treatment at an early phase. The data were obtained from two hospitals in Indonesia, consisting of 89 hepatitis-B and 31 hepatitis-C samples. The data were analyzed using several cases of k-fold cross-validation, and the performances were compared according to their accuracy, sensitivity, precision, F1-Score, and running time. From the experiments, it was concluded that fuzzy kernel k-medoids using RBF kernel function is better compared to polynomial kernel function with the 6% increment of accuracy, 13% enhancement of sensitivity, and 5% improvement in F1-Score. On the other side, the precision of fuzzy kernel k-medoids using polynomial kernel function is 2% higher than using the RBF kernel function. According to the results, the use of RBF or polynomial kernel function in fuzzy kernel medoids can be considered according to the primary goal of the classification.</span></span>

Monitoring of Photovoltaic Systems Using Improved Kernel-Based Learning Schemes

Data-based procedures for monitoring the operating performance of a PV system are proposed in this article. The only information required to apply the procedures is the availability of system measurements, which are routinely on-line collected via sensors. Here, kernel-based machine learning methods, including support vector regression (SVR) and Gaussian process regression (GPR), are used to model multivariate data from the PV system for fault detection because of their flexibility and capability to nonlinear approximation. Essentially, the SVR and GPR models are adopted to obtain residuals to detect and identify occurred faults. Then, residuals are passed through an exponential smoothing filter to reduce noise and improve data quality. In this work, a monitoring scheme based on kernel density estimation is used to sense faults by examining the generated residuals. Several different scenarios of faults were considered in this study, including PV string fault, partial shading, PV modules short-circuited, module degradation, and line-line faults on the PV array. Using data from a 20 MWp grid-connected PV system, the considered faults were successfully traced using the developed procedures. Also, it has been demonstrated that GPR-based monitoring procedures achieve better detection performance over SVRs to monitor PV systems.

On kernel machine learning for propensity score estimation under complex confounding structures.

Post marketing data offer rich information and cost-effective resources for physicians and policy-makers to address some critical scientific questions in clinical practice. However, the complex confounding structures (e.g., nonlinear and nonadditive interactions) embedded in these observational data often pose major analytical challenges for proper analysis to draw valid conclusions. Furthermore, often made available as electronic health records (EHRs), these data are usually massive with hundreds of thousands observational records, which introduce additional computational challenges. In this paper, for comparative effectiveness analysis, we propose a statistically robust yet computationally efficient propensity score (PS) approach to adjust for the complex confounding structures. Specifically, we propose a kernel-based machine learning method for flexibly and robustly PS modeling to obtain valid PS estimation from observational data with complex confounding structures. The estimated propensity score is then used in the second stage analysis to obtain the consistent average treatment effect estimate. An empirical variance estimator based on the bootstrap is adopted. A split-and-merge algorithm is further developed to reduce the computational workload of the proposed method for big data, and to obtain a valid variance estimator of the average treatment effect estimate as a by-product. As shown by extensive numerical studies and an application to postoperative pain EHR data comparative effectiveness analysis, the proposed approach consistently outperforms other competing methods, demonstrating its practical utility.

Supervised learning strategy for classification and regression tasks applied to aeronautical structural health monitoring problems

This paper presents the use of a kernel-based machine learning strategy targeting classification and regression tasks in view of automatic flaw(s) detection, localization and characterization. The studied use-case is a structural health monitoring configuration with an array of piezoelectric sensors integrated on aluminium panels affected by flaws of various positions and dimensions. The measured guided wave signals are post processed with a guided wave imaging algorithm in order to obtain an image representing the health of each specimen. These images are then used as inputs to build classification and regression models. In this paper, an extensive numerical validation campaign is conducted to validate the process. Then the inversion is applied to an experimental campaign, which demonstrate the ability to use a numerically-built model to invert experimental data.