Kernel Partial Least Squares Research Articles

KF-PLS: Optimizing Kernel Partial Least-Squares (K-PLS) with Kernel Flows

Partial Least-Squares (PLS) regression is a widely used tool in chemometrics for performing multivariate regression. As PLS has a limited capacity of modelling non-linear relations between the predictor variables and the response, Kernel PLS (K-PLS) has been introduced for modelling non-linear predictor-response relations. Most available studies use fixed kernel parameters, reducing the performance potential of the method. Only a few studies have been conducted on optimizing the kernel parameters for K-PLS. In this article, we propose a methodology for the kernel function optimization based on Kernel Flows (KF), a technique developed for Gaussian Process Regression (GPR). The results are illustrated with four case studies. The case studies represent both numerical examples and real data used in classification and regression tasks. K-PLS optimized with KF, called KF-PLS in this study, is shown to yield good results in all illustrated scenarios, outperforming literature results and other non-linear regression methodologies. In the present study, KF-PLS has been compared to convolutional neural networks (CNN), random trees, ensemble methods, support vector machines (SVM), and GPR, and it has proved to perform very well.

Novel intrusion detection system based on a downsized kernel method for cybersecurity in smart agriculture

Smart farming aims to ameliorate the quantity and quality of farms products using modern information and communication technologies. Agriculture has always used cutting-edge technology to improve profitability, efficiency, and safety as well as creating the opportunity for more environmentally friendly operations. In smart agriculture network communication between smart devices is necessary to managing the agriculture process. Consequently, the possibility of information being exposed to hacking is very likely, which makes the information security system extremely important and cannot be neglected. Even though many solutions have been developed to overcome cyberattacks in smart networks, some limitations are observed, such as solving multi-class problems, high feature dimensions, and overhead computation. To address the aforementioned issues, an intelligent Intrusion Detection System (IDS) for identifying cyberattacks in the Internet of Agriculture Things (IoAT) is proposed. The developed model uses a proposed reduced kernel method, the Downsized Kernel Partial Least Square (DKPLS) to extract and reduce data feature dimension to improve detection performance. The DRKPLS approach is used to reduce the dimension of a kernel matrix generated using the Kernel Partial Least Square (KPLS) technique by selecting the most important features. In classification phase, the Kernel Extreme Learning Machine (KELM) is used to classify data for binary and multi-class classification. The proposed IDS is evaluated on a new Industrial IoT Dataset, X-IIoTID. The developed approach achieved higher performances for binary classification and for multi–class classification compared to others machine learning and deep learning approaches. The proposed approach achieved an accuracy rate of 99.92% for binary classification and 99.99% for multi–class classification. Furthermore, the higher percentage in multi-class classification is obtained for the two types of attacks Ransom Denial of Service (RDoS) and Command_Control respectively.

A new prediction method for sodium aluminate solution evaporation integrating process knowledge and data-driven spatial-temporal adaptive model

In alumina production, the evaporation as the key process uses recyclable resources and reduces environmental pollution. In fact, the quality of export product with offline and delayed, results in low precision of process control and high energy consumption. To ensure green and efficient production, in this paper, a new prediction method integrating process knowledge and data-driven spatial-temporal adaptive model is put forward. First, to preprocessed production data for ensuring modeling accuracy, data reconciliation technology is adopted. Then, based on material and heat transfer mechanism, for equipment and industrial process, the mechanism models are established. Furthermore, with time difference and moving window model, an error compensation method is utilized in terms of double locally weighted kernel PLS for estimation error in hypothesis-based mechanism modeling. Finally, the data-driven spatial-temporal adaptive model and the process knowledge-based mechanism model are integrated. To illustrate the model feasibility, an industrial sodium aluminate solution evaporation is used. It demonstrates that, for the developed model, the prediction accuracy can reach more than 90% within the ±2% error range, and effectively estimate the actual product quality and ensure the prediction effect.

Exploring the Path of Teacher Team Building for Innovation and Entrepreneurship in Colleges and Universities in the Information Age

Abstract With the in-depth promotion of education informatization, colleges and universities focus on developing innovation and entrepreneurship education and are committed to cultivating talents with innovative spirit and technology, which has become the key to the reform of higher education. This article profoundly analyzes the characteristics of the current innovation and entrepreneurship education teacher team, and proceeds to optimize and build a dual-creative teacher team with information literacy. Aiming at the construction effect of this path, the article constructs an evaluation index system of the construction effect of the dual-creation teacher team from multiple aspects. It combines the Kernel Partial Least Squares (KPLS) and Particle Swarm Optimization Algorithm (PSO) to empower the indexes to assess the effectiveness of the construction path. The research analysis shows that the proposed faculty building path effectively promotes the rejuvenation of the faculty team, and the percentage of teachers under 35 years old rises to 34%, significantly improving teachers’ acceptance of new things. Regarding teaching results, the weight of students’ participation in entrepreneurial practice reaches 0.146. The weight of filing innovation patents is 0.101. The results show that the construction path has a positive impact on improving the faculty team’s teaching quality and students’ innovation ability.

Kernel-Based KPI-Oriented Fault Detection Approach in the Presence of Outliers

Kernel partial least squares (KPLS) has poor robustness and cannot achieve effective monitoring for key performance indicators (KPI). This study investigates a new KPI-oriented robust KPLS approach to mitigate these drawbacks. In this methodology, the robust KPLS inspired by the gradient boosting principle incorporates a weighting matrix into the KPLS to mitigate the influence of outliers. Simultaneously, the associated coefficient matrix is derived in detail. Then, a decomposition approach is used to separate the process variable space into two orthogonal parts. Two strategies are discussed to obtain the unknown projection matrices based on the kernel principal component analysis and the elastic network frameworks. Finally, the performance of the proposed methods in terms of prediction, monitoring and robustness to outliers is evaluated by the Tennessee Eastman process and the three-phase flow facility. The results show that the proposed methods have good prediction accuracy and monitoring performance in the presence of outliers, demonstrating the effectiveness and advantages of the proposed approaches.

Analyzing the occurrence of foaming in batch fermentation processes using multiway partial least square approaches

AbstractThis article presents an application of multiway partial least squares (MPLS) methods to develop interpretative correlation models to monitor the foaming occurrence and improve batch fermentation. We choose the exhaust differential pressure as a quality variable to quantify the foaming occurrence and consider three‐dimensional datasets of different batches, process variables, and measurements. We integrate batch‐wise unfolding (BWU) and observation‐wise unfolding (OWU) of plant datasets with standard, dynamic, and kernel PLS methods. We find that dynamic PLS (DPLS) with OWU and time‐lagged quality variables to be the most efficient, accurate, and easy to implement. The BWU approach is useful for analyzing the differences between batches and identifying abnormalities and outliers, while the OWU quantifies the variation within a given batch. With OWU, the DPLS method with one unit of time lag in the quality variable is the most effective, accurate, and easy to implement. With both BWU and OWU, we identify the quantitative effects of process variables on the quality variable and providence guidance to improve fermentation performance.

Colorimetric Characterization of Color Imaging System Based on Kernel Partial Least Squares

Colorimetric characterization is the basis of color information management in color imaging systems. In this paper, we propose a colorimetric characterization method based on kernel partial least squares (KPLS) for color imaging systems. This method takes the kernel function expansion of the three-channel response values (RGB) in the device-dependent space of the imaging system as input feature vectors, and CIE-1931 XYZ as output vectors. We first establish a KPLS color-characterization model for color imaging systems. Then we determine the hyperparameters based on nested cross validation and grid search; a color space transformation model is realized. The proposed model is validated with experiments. The CIELAB, CIELUV and CIEDE2000 color differences are used as evaluation metrics. The results of the nested cross validation test for the ColorChecker SG chart show that the proposed model is superior to the weighted nonlinear regression model and the neural network model. The method proposed in this paper has good prediction accuracy.

Two sides of the same coin: Kernel partial least-squares (KPLS) for linear and non-linear multivariate calibration. A tutorial

A tutorial is presented on the operation and properties of the non-linear multivariate regression model kernel partial least-squares (KPLS). After the discussion of a simple non-linear univariate problem, solved by regressing a dependent variable on the projection of an independent variable onto a set of Gaussian functions, the principles of KPLS are introduced for processing non-linear multivariate data. The following aspects are covered: (1) the estimation of the model sensitivity as a function of analyte concentration from error propagation concepts, (2) the proposal of a parameter measuring the degree of non-linearity, to avoid a black-and-white description of data sets as either linear or non-linear, (3) the use of the model parameters for variable selection. The application of KPLS to both simulated and experimental data sets is discussed, in the latter case involving near infrared spectra employed for the determination of quality parameters in foodstuff samples and fluorescence spectroscopic data for the study of systems of biological relevance. Computer codes written in the popular MATLAB and R environments are also provided.

Practical Online Characterization of the Properties of Hydrocracking Bottom Oil via Near-Infrared Spectroscopy

Providing real-time information on the chemical properties of hydrocracking bottom oil (HBO) as the feedstock for ethylene cracker while minimizing processing time, is important to improve the real-time optimization of ethylene production. In this study, a novel approach for estimating the properties of HBO samples was developed on the basis of near-infrared (NIR) spectra. The main noise and extreme samples in the spectral data were removed by combining discrete wavelet transform with principal component analysis and Hotelling’s T2 test. Kernel partial least squares (KPLS) regression was utilized to account for the nonlinearities between NIR data and the chemical properties of HBO. Compared with the principal component regression, partial least squares regression, and artificial neural network, the KPLS model had a better performance of obtaining acceptable values of root mean square error of prediction (RMSEP) and mean absolute relative error (MARE). All RMSEP and MARE values of density, Bureau of Mines correlation index, paraffins, isoparaffins, and naphthenes were less than 1.0 and 3.0, respectively. The accuracy of the industrial NIR online measurement system during consecutive running periods in predicting the chemical properties of HBO was satisfactory. The yield of high value-added products increased by 0.26 percentage points and coil outlet temperature decreased by 0.25 °C, which promoted economic benefits of the ethylene cracking process and boosted industrial reform from automation to digitization and intelligence.

Spectral pre-processing and non-linear calibration with convolutional kernel partial least-squares. Teaching new tricks to an old dog

A recent trend in multivariate calibration of non-linear systems is to simplify data processing models, avoiding, if possible, some complex deep learning approaches. Contributing to this line of work, convolutional kernel partial least-squares (CKPLS) is introduced both for finding the best spectral pre-processing procedure for reducing the impact of radiation scattering and for handling non-linearities in the data. CKPLS is a combination of a previous convolutional step for pre-processing the spectra with the well-known kernel PLS regression model for coping with non-linear relationships between spectral signatures and analyte concentrations or target sample properties. The convolutional step is driven by particle swarm optimization (PSO), which estimates the coefficients of a moving window spectral pre-processing. This convolutional phase, previous to KPLS, is a viable alternative to the few available methods for finding the best mathematical pre-processing of the spectra, which is usually performed by trial and error. Analytical results concerning the calibration of selected analytes from partially selective spectra are employed to illustrate the performance of CKPLS. For this purpose, both simulated and experimental data sets have been employed, showing that automatic pre-processing of spectra is possible, with a success which is comparable to classical methods such as computing the spectral derivatives.

Kernel PLS with AdaBoost ensemble learning for particulate matters forecasting in subway environment

Excessive exposure to hazardous air pollutants remains the leading cause of serious cardio-respiratory disease among commuters. To monitor and improve the air quality in time, soft-sensing techniques in metro systems need to be introduced. Adaptive boosting (AdaBoost), as an ensemble learning method, has greater flexibility and adaptability. Compared with a single conventional model, it has a stronger capability to explain the complex features contained in air quality data. In this study, an improved scheme employs AdaBoost to predict key air quality variables, such as PM2.5, in subway environment. To eliminate the adverse effects of noise interference and the collinearity problem among data variables, kernel latent variables are embedded into the original AdaBoost. The proposed method combines kernel partial least squares with AdaBoost (KPLS-AdaBoost). Root mean square error (RMSE) was used to evaluate the prediction performance of KPLS-AdaBoost. The results demonstrate that KPLS-AdaBoost could achieve better modeling performance, providing a new way to realize real-time monitoring of hazardous air pollutants. More precisely, in terms of PM2.5 in the hall, the RMSE value was optimized by approximately 14.44–31.86% compared with that of AdaBoost, random forest, artificial neural network, PLS, KPLS, and PLS-AdaBoost. For PM2.5 at the platform, RMSE value can be decreased by 13.57–38.51%.

Fault Diagnosis of Check Valve Based on KPLS Optimal Feature Selection and Kernel Extreme Learning Machine

The check valve is the core part of high-pressure diaphragm pumps. It has complex operation conditions and has difficulty characterizing fault states completely with its single feature. Therefore, a fault signal diagnosis model based on the kernel extreme learning machine (KELM) was constructed to diagnose the check valve. The model adopts a multi-feature extraction method and reduces dimensionality through kernel partial least squares (KPLS). Firstly, we divided the check valve vibration signal into several non-overlapping samples. Then, we extracted 16 time-domain features, 13 frequency-domain features, 16 wavelet packet energy features, and energy entropy features from each sample to construct a multi-feature set characterizing the operation state of the check valve. Next, we used the KPLS method to optimize the 45 dimension multi-feature data and employed the processed feature set to establish a KELM fault diagnosis model. Experiments showed that the method based on KPLS optimal feature selection could fully characterize the operating state of the equipment with an accuracy rate of 96.88%. This result indicates the high accuracy and effectiveness of the multi-feature set constructed with the KELM fault diagnosis model.

Face Recognition Through Multi-Resolution Images

In this paper, we introduce a new method for face recognition in multi-resolution images. The proposed method is composed of two phases: an off-line phase and an inference phase. In the off-line phase, we built the Kernel Partial Least Squares (KPLS) regression model to map the LR facial features to HR ones. The KPLS predictor was then used in the inference phase to map HR features from LR features. We applied in both phases the Block-Based Discrete Cosine Transform (BBDCT) descriptor to enhance the facial feature description. Finally, the identity matching was carried out with the K-Nearest Neighbor (KNN) classifier. Experimental study was conducted on the AR and ORL databases and the obtained results proved the efficiency of the proposed method to deal with LR and VLR face recognition problem.

A Novel Three-Stage Quality Oriented Data-Driven Nonlinear Industrial Process Monitoring Strategy

It is well known that the partial least squares (PLS) and its nonlinear extension kernel PLS (KPLS) are efficient tools widely utilized in fault detection. Nonetheless, these two techniques project obliquely into the process variable space, limiting their ability to discriminate between the quality-related and quality-unrelated faults. Although various enhanced ways based on the basic PLS and KPLS, such as the total PLS and total KPLS methods, have been published to cope with this problem, these methods fail to reduce the false alarm rates of the quality-unrelated fault when the fault amplitude increases, and they need to monitor four statistics to determine the fault categories. To address this issue, this paper proposes a three-stage data-driven method for complicated nonlinear industrial quality-related process monitoring, consisting of three parts: data preprocessing, modeling, and decomposing. The kernel direct orthogonalization (KDO) approach is initially investigated in this strategy, which is based on the traditional direct orthogonalization methodology and the kernel theory. The unnecessary fluctuations in the process variable space are eliminated after the data is preprocessed using the KDO approach. A standard KPLS model is built for the filtered component. Then, a quality-related KPLS (QRKPLS) method is proposed as the decomposing part to further divide the filtered process mapping matrix into two orthogonal parts, namely, the quality-related part and the quality-unrelated part. Additionally, the corresponding test statistics are calculated for these two parts, and the diagnosis logic is also given in this paper. The effectiveness and superiority of the novel established KDO-QRKPLS method is investigated using a widely numerical simulation as well as an industrial simulation.

Developing new products with kernel partial least squares model inversion

In recent years, data-driven approaches (e.g., latent variable model) have excited the development of new products and the control of product quality. To derive an input space within which raw material properties or initial operating conditions can yield the required product quality, previous researchers developed a linear latent variable model and derived the solution of inputs through PLS model inversion. In this research, a novel method based on kernel partial least squares (KPLS) model inversion is proposed for product design of nonlinear processes and an input domain is derived. Constraints on the model inputs or outputs and on the KPLS model are discussed, and solutions are provided based on an optimization framework and the Monte Carlo sampling method. The effectiveness of the method is demonstrated by numerical simulation and a beer fermentation experiment, and the KPLS model inversion method outperforms the PLS model inversion method in terms of the precision.

Improved Dynamic Optimized Kernel Partial Least Squares for Nonlinear Process Fault Detection

We suggest in this article a dynamic reduced algorithm in order to enhance the monitoring abilities of nonlinear processes. Dynamic fault detection using data-driven methods is among the key technologies, which shows its ability to improve the performance of dynamic systems. Among the data-driven techniques, we find the kernel partial least squares (KPLS) which is presented as an interesting method for fault detection and monitoring in industrial systems. The dynamic reduced KPLS method is proposed for the fault detection procedure in order to use the advantages of the reduced KPLS models in online mode. Furthermore, the suggested method is developed to monitor the time-varying dynamic system and also update the model of reduced reference. The reduced model is used to minimize the computational cost and time and also to choose a reduced set of kernel functions. Indeed, the dynamic reduced KPLS allows adaptation of the reduced model, observation by observation, without the risk of losing or deleting important information. For each observation, the update of the model is available if and only if a further normal observation that contains new pertinent information is present. The general principle is to take only the normal and the important new observation in the feature space. Then the reduced set is built for the fault detection in the online phase based on a quadratic prediction error chart. Thereafter, the Tennessee Eastman process and air quality are used to precise the performances of the suggested methods. The simulation results of the dynamic reduced KPLS method are compared with the standard one.

Kernel Partial Least Square Regression with High Resistance to Multiple Outliers and Bad Leverage Points on Near-Infrared Spectral Data Analysis

Multivariate statistical analysis such as partial least square regression (PLSR) is the common data processing technique used to handle high-dimensional data space on near-infrared (NIR) spectral datasets. The PLSR is useful to tackle the multicollinearity and heteroscedasticity problem that can be commonly found in such data space. With the problem of the nonlinear structure in the original input space, the use of the classical PLSR model might not be appropriate. In addition, the contamination of multiple outliers and high leverage points (HLPs) in the dataset could further damage the model. Generally, HLPs contain both good leverage points (GLPs) and bad leverage points (BLPs); therefore, in this case, removing the BLPs seems relevant since it has a significant impact on the parameter estimates and can slow down the convergence process. On the other hand, the GLPs provide a good efficiency in the model calibration process; thus, they should not be eliminated. In this study, robust alternatives to the existing kernel partial least square (KPLS) regression, which are called the kernel partial robust GM6-estimator (KPRGM6) regression and the kernel partial robust modified GM6-estimator (KPRMGM6) regression are introduced. The nonlinear solution on PLSR was handled through kernel-based learning by nonlinearly projecting the original input data matrix into a high-dimensional feature mapping that corresponded to the reproducing kernel Hilbert spaces (RKHS). To increase the robustness, the improvements on GM6 estimators are presented with the nonlinear PLSR. Based on the investigation using several artificial dataset scenarios from Monte Carlo simulations and two sets from the near-infrared (NIR) spectral dataset, the proposed robust KPRMGM6 is found to be superior to the robust KPRGM6 and non-robust KPLS.

Efficient cross-validatory algorithm for identifying dynamic nonlinear process models

Kernel partial least squares (KPLS) is an effective nonlinear modeling technique for control engineering applications, including model predictive control, process monitoring or general system diagnosis. It can deal with small sample sizes and variable sets that are noisy and highly correlated. Kernel partial least squares maps the input (or cause) variables to a feature space and carries out the task of producing an optimal prediction model for the process output (or effect) variables using the standard linear partial least squares (PLS) approach. Resulting from the typically large size of the feature space, the kernel partial least squares procedure can be computational intensive. In particular, if the optimal model structure is estimated using cross-validation, KPLS is not efficient in handling large data sets. This paper first modifies the conventional kernel partial least squares procedure in order to embed it within a leave-one-out cross-validation (LOOCV) framework. The proposed efficient kernel partial least squares (EKPLS) is able to reduce the computational complexity by an order of magnitude compared to the conventional approach, which is proven both analytically and through modeling applications to three industrial data sets.

A nonlinear calibration transfer method based on joint kernel subspace

A nonlinear strategy is proposed to develop calibration transfer method with the available joint spectral data composed of the master and slave spectra of standard samples. Its core idea is to minimize the instrument-induced spectral variation in the reproducing kernel Hilbert space (RKHS) where the joint spectral data is implicitly mapped by proper kernel function. The nonlinear feature representation of master or slave spectra are reconstructed by singular value decomposition (SVD) in the RKHS. Then the transferred feature of slave spectra approaching the reconstructed feature of master spectra in the RKHS can be acquired by the procedure that (1) calculates the transfer matrix to match the maser and slave kernel features in the joint kernel subspace and (2) utilizes the transferred slave kernel feature to reconstruct the nonlinear feature representation of slave spectra. As a better feature representation suited for multivariate calibration, both the reconstructed feature of master calibration spectra and the transferred feature of slave test spectra are derived in the RKHS. A kernel partial least squares (KPLS) model built on the former is applied to the latter. The KPLS master calibration model is equivalent to a partial least squares (PLS) model built with the corresponding reconstructed master kernel feature in the joint kernel subspace. By exploiting the kernel eigenvector representations and kernel trick, a series of computationally less demanding formulas with linear operation is derived to realize the nonlinear calibration model, termed as joint kernel subspace based calibration transfer (JKSCT). JKSCT is compared with boxcar signal transfer (BST), piecewise direct standardization (PDS), multi-level simultaneous component analysis (MSCA), canonical correlation analysis based calibration transfer (CCACT), generalized least squares (GLS), slope and bias correction (SBC), spectral space transformation (SST), external parameter orthogonalization (EPO) and double competitive adaptive reweighted sampling (DCARS) on three spectral datasets. Experimental results show that JKSCT performs at least comparable with the DCARS or SST, and frequently better than the other methods.

Optimizing the quantitative analysis of solid biomass fuel properties using laser induced breakdown spectroscopy (LIBS) coupled with a kernel partial least squares (KPLS) model.

The rapid analysis of fuel properties is important for the utilization of solid biomass due to its great variation in feedstock. Laser-induced breakdown spectroscopy (LIBS) technology combined with quantitative analysis models can be used for this analysis. Most existing prediction models used in LIBS for fuel property analysis are linear methods, such as the partial least squares (PLS) model, which fail to reflect the non-linear relationships between the LIBS spectrum and the fuel property index being predicted. In the present work, LIBS data combined with a kernel partial least squares (KPLS) method are used to analyze the gross calorific value, and the volatile matter, ash and fixed carbon content of the solid biomass fuel. Quantitative analysis performance of the KPLS model was compared to that of the widely used PLS method, with the results showing some improvements. The KPLS model was further improved using three data normalization methods (i.e., C internal standardization, total intensity standardization and standard normal variate). The best quantitative analysis results of the volatile matter and ash content were obtained when the KPLS model was combined with C internal standardization, with root mean square errors of prediction (RMSEP) of 1.365% and 0.290%, and average standard deviations (ASD) of 0.277% and 0.080%, respectively. The best quantitative analysis results of the gross calorific value and fixed carbon content were obtained when using KPLS without normalization. The RMSEP and ASD of the gross calorific value and fixed carbon content were 0.198 MJ kg-1 and 0.746%, and 0.070 MJ kg-1 and 0.111% respectively.