Nonlinear Partial Least Squares Research Articles

Data driven latent variable adaptive control for nonlinear multivariable processes

This article aims at a new partial least squares (PLS) control design and analysis in a data-driven framework for nonlinear multivariable processes whose mechanistic models are completely unknown. First, a general nonlinear autoregressive moving average with the exogenous input (NARMAX) model is used as the dynamic nonlinear PLS model. Then, by introducing a dynamic linearisation approach in each latent variable (LV) space, the unknown NARMAX-based PLS model is transformed to a linear dynamic PLS data model (dPLSDM), which can be improved in real time by estimating its unknown parameter using the latent input and output (I/O) data. Next, a data-driven latent variable adaptive control (DDLVAC) is proposed in each LV loop. By virtue of the dPLSDM, the multivariable nonlinear process is decoupled into multiple single-loop systems and the high dimensions of the process data are reduced such that the corresponding DDLVAC is simplified. Further, the DDLVAC only depends on the I/O data without requiring any model information of the original process. Theoretical analysis confirms the validity of the DDLVAC. The simulation study demonstrates the advantages of the DDLVAC such as less storage space, smaller computation burden, less control cost, as well as more robustness against uncertainties.

A structured evaluation of regression models for predicting CO2 concentration from plasma emission spectra

This study is an extensive comparison of the predictive performance of a bagging neural network (BANN), partial least squares (PLS) regressor, and a kernel-based, nonlinear PLS (KPLS) regressor, given experimentally obtained CO2-N2-Ar plasma emission spectra. The spectra, 3,62,31 in total, were obtained from controlled gas mixtures with varying CO2 concentrations fed to a stripline split-ring resonator microplasma source and recorded with an UV-NIR, 2nm resolution spectrometer. The regression methods’ dependence on (i) number of observations used in training, (ii) preprocessing steps, and (iii) feature selection (in this case wavelength ranges) was evaluated by training and testing 60–66 models per method, each with a unique combination of the aforementioned configuration options, and each trained 4 times with random train-test splits. To compare the models a custom metric that compounds R2, Pearson correlation, and a weighted root mean squared relative error was used. The results show that the BANN models outperform both PLS and KPLS, reaching a peak score of 0.873, with the others getting 0.561 and 0.581, respectively, using the (−∞, 1] metric. The top performing BANN model was trained without any feature selection or preprocessing, these steps were, however, required for both the best PLS and KPLS models. In a wider perspective, the results show that BANNs are not only suitable as in-place replacements for PLS-based methods, but increase regression model prediction accuracy on low resolution spectra to such an extent that they offer modelling of previously unattainable, nonlinear information contained in emission spectra datasets.

Probabilistic Power Flow Analysis Based on Partial Least Square and Arbitrary Polynomial Chaos Expansion

This paper presents a new algorithm based on the partial least square (PLS) techniques and the arbitrary polynomial chaos expansion (aPCE) for the probabilistic power flow (PPF) analysis of a power system having many uncertain variables. The proposed method uses the nonlinear PLS to transform a set of random input variables to a smaller number of de-correlated random variables. Then, the aPCE technique is applied to generate the basis polynomial functions and build a surrogate model of the power system response. The algorithm has been implemented and tested with the modified IEEE 118-bus and European 1354-bus systems. The numerical results indicate that, similar to the sparse PCE method and the low-rank approximation technique, the proposed method can be applied to high-dimensional PPF problems. Nonetheless, the proposed approach uses smaller computation time, and estimates statistical characteristics of the response with higher accuracy.

A Novel Three-Stage Quality Oriented Data-Driven Nonlinear Industrial Process Monitoring Strategy

It is well known that the partial least squares (PLS) and its nonlinear extension kernel PLS (KPLS) are efficient tools widely utilized in fault detection. Nonetheless, these two techniques project obliquely into the process variable space, limiting their ability to discriminate between the quality-related and quality-unrelated faults. Although various enhanced ways based on the basic PLS and KPLS, such as the total PLS and total KPLS methods, have been published to cope with this problem, these methods fail to reduce the false alarm rates of the quality-unrelated fault when the fault amplitude increases, and they need to monitor four statistics to determine the fault categories. To address this issue, this paper proposes a three-stage data-driven method for complicated nonlinear industrial quality-related process monitoring, consisting of three parts: data preprocessing, modeling, and decomposing. The kernel direct orthogonalization (KDO) approach is initially investigated in this strategy, which is based on the traditional direct orthogonalization methodology and the kernel theory. The unnecessary fluctuations in the process variable space are eliminated after the data is preprocessed using the KDO approach. A standard KPLS model is built for the filtered component. Then, a quality-related KPLS (QRKPLS) method is proposed as the decomposing part to further divide the filtered process mapping matrix into two orthogonal parts, namely, the quality-related part and the quality-unrelated part. Additionally, the corresponding test statistics are calculated for these two parts, and the diagnosis logic is also given in this paper. The effectiveness and superiority of the novel established KDO-QRKPLS method is investigated using a widely numerical simulation as well as an industrial simulation.

Kernel Partial Least Square Regression with High Resistance to Multiple Outliers and Bad Leverage Points on Near-Infrared Spectral Data Analysis

Multivariate statistical analysis such as partial least square regression (PLSR) is the common data processing technique used to handle high-dimensional data space on near-infrared (NIR) spectral datasets. The PLSR is useful to tackle the multicollinearity and heteroscedasticity problem that can be commonly found in such data space. With the problem of the nonlinear structure in the original input space, the use of the classical PLSR model might not be appropriate. In addition, the contamination of multiple outliers and high leverage points (HLPs) in the dataset could further damage the model. Generally, HLPs contain both good leverage points (GLPs) and bad leverage points (BLPs); therefore, in this case, removing the BLPs seems relevant since it has a significant impact on the parameter estimates and can slow down the convergence process. On the other hand, the GLPs provide a good efficiency in the model calibration process; thus, they should not be eliminated. In this study, robust alternatives to the existing kernel partial least square (KPLS) regression, which are called the kernel partial robust GM6-estimator (KPRGM6) regression and the kernel partial robust modified GM6-estimator (KPRMGM6) regression are introduced. The nonlinear solution on PLSR was handled through kernel-based learning by nonlinearly projecting the original input data matrix into a high-dimensional feature mapping that corresponded to the reproducing kernel Hilbert spaces (RKHS). To increase the robustness, the improvements on GM6 estimators are presented with the nonlinear PLSR. Based on the investigation using several artificial dataset scenarios from Monte Carlo simulations and two sets from the near-infrared (NIR) spectral dataset, the proposed robust KPRMGM6 is found to be superior to the robust KPRGM6 and non-robust KPLS.

GMMPLS: A Novel Automatic State-Based Algorithm for Continuous Decoding in BMIs

In this paper, a novel fully-automated state-based decoding method has been proposed for continuous decoding problems in brain-machine interface (BMI) systems, called Gaussian mixture of model (GMM)-assisted PLS (GMMPLS). In contrast to other state-based and hierarchical decoders, the proposed method does not demand any prior information about the desired output structure. Instead, GMMPLS uses the GMM algorithm to divide the desired output into a specific number of states (clusters). Next, a logistic regression model is trained to predict the probability membership of each time sample for each state. Finally, using the concept of the partial least square (PLS) algorithm, GMMPLS constructs a model for decoding the desired output using the input data and the achieved membership probabilities. The performance of the GMMPLS has been evaluated and compared to PLS, the nonlinear quadratic PLS (QPLS), and the bayesian PLS (BPLS) methods through a simulated dataset and two different real-world BMI datasets. The achieved results demonstrated that the GMMPLS significantly outperformed PLS, QPLS, and BPLS overall datasets.

Nonlinear sparse partial least squares: an investigation of the effect of nonlinearity and sparsity on the decoding of intracranial data

Objective. Partial Least Squares (PLS) regression is a suitable linear decoder model for correlated and high dimensional neural data. This algorithm has been widely used in the application of brain–computer interface (BCI) for the decoding of motor parameters. PLS does not consider nonlinear relations between brain signal features. The nonlinear version of PLS that considers a nonlinear relationship between the latent variables has not been proposed for the decoding of intracranial data. This nonlinear model may cause overfitting in some cases due to a larger number of free parameters. In this paper, we develop a new version of nonlinear PLS, namely nonlinear sparse PLS (NLS PLS) and test it in BCI applications. Approach. In motor related BCI systems, improving the decoding accuracy of both kinetic and kinematic parameters of movement is crucial. To do this, two BCI datasets were chosen to decode the force amplitude and position of hand trajectory using the nonlinear and sparse versions of PLS algorithm. In our new NLS PLS method, we considered a polynomial relationship between the latent variables and used the lasso penalization in the latent space to avoid overfitting and to improve the decoding accuracy. Main results. Some linear and nonlinear based PLS models and our new proposed method, NLS PLS, were applied to the two datasets. According to our results, significant improvement from the NLS PLS method is confirmed over other methods. Our results show that nonlinear PLS outperforms generic PLS in the force decoding but it has lower accuracy in the hand trajectory decoding because of high dimensional feature space. By using lasso penalization, we presented a sparse nonlinear PLS-based model that outperforms generic PLS in both datasets and improves the coefficient of determination, 34% in the force decoding and 10% in the hand trajectory decoding. Significance. We constructed a simple PLS-based model that considers a nonlinear relationship between features and it is also robust to overfitting because of using the lasso penalty in the latent space. This model is suitable for a high dimensional and correlated datasets, like intracranial data and can improve the accuracy of estimation.

Knowledge based recursive non-linear partial least squares (RNPLS)

Soft sensors driven by data are very common in industrial plants to perform indirect measurements of difficult to measure critical variables by using other variables that are relatively easier to obtain. The use of soft sensors implies some challenges, such as the colinearity of the predictor variables, the time-varying and possible non-linear nature of the industrial process. To deal with the first challenge, the partial least square (PLS) regression has been employed in many applications to model the linear relations between process variables, with noisy and highly correlated data. However, the PLS model needs to deal with the other two issues: the non-linear and time-varying characteristics of the processes. In this work, a new knowledge-based methodology for a recursive non-linear PLS algorithm (RNPLS) is systematized to deal with these issues. Here, the non-linear PLS algorithm is set up by carrying out the PLS regression over the augmented input matrix, which includes knowledge based non-linear transformations of some of the variables. This transformation depends on the system’s nature, and takes into account the available knowledge about the process, which is provided by expert knowledge or emulated using software tools. Then, the recursive exponential weighted PLS is used to modify and adapt the model according to the process changes. This RNPLS algorithm has been tested using two case studies according to the available knowledge, a real industrial evaporation station of the sugar industry, where the expert knowledge about the process permits the formulation of the relationships, and a simulated wastewater treatment plant, where the necessary knowledge about the process is obtained by a software tool. The results show that the methodology involving knowledge regarding the process is able to adjust the process changes, providing highly accurate predictions.

Likelihood Maximization Inverse Regression: A novel non-linear multivariate model

Common multivariate regression models are calculated with the objective of directly predicting calibration y data from X observations. Our proposed methodology, presented in this paper, inverses the problem. Indeed, we propose a regression model which relies on predicting y by the likelihood maximization of expected errors in X. We named our parameter-free algorithm Likelihood Maximization Inverse Regression (LMIR). Using 4 different datasets, we compared LMIR performance with Partial Least Squares-1 (PLS1), a non-linear PLS variant and another inverse regression method: Sliced Inverse Regression (SIR). LMIR yielded better validation performances in almost all study cases. We also demonstrated that LMIR was able to consider any known and additional noise present in validation X observations without creating a new model, as required in PLS1 and SIR. A LMIR model built from one instrument could then be easily transferred to another.

Dynamic Nonlinear Partial Least Squares Modeling Using Gaussian Process Regression

A dynamic Gaussian process regression based partial least-squares (D-GPR-PLS) model is proposed to improve estimation ability and compared to the conventional nonlinear PLS. Considering the strong ability of GPR in nonlinear process modeling, this method is used to build a nonlinear regression between each pair of latent variables in the partial least-squares. In addition, augmented matrices are embedded into the D-GPR-PLS model to obtain better prediction accuracy in nonlinear dynamic processes. To evaluate the modeling performance of the proposed method, two simulated cases and a real industrial process based on wastewater treatment processes (WWTPs) are considered. The simulated cases use data from two high fidelity simulators: benchmark simulation model no. 1 and its long-term version. The second study uses data from a real biological wastewater treatment process. The results show the superiority of D-GPR-PLS in modeling performance for both data sets. More specifically, in terms of the prediction for effluent chemical oxygen demand of the real WWTP data, the value of the root-mean-square error is decreased by 31%, 16%, and 52%, respectively, in comparison with that for linear PLS, quadratic PLS, and least-squares support vector machine based PLS.

Nonlinear Multivariate Quality Prediction Based on OSC-SVM-PLS

Predicting data with noise and high correlation using latent structural projection or partial least-squares (PLS) is a commonly used linear regression method. In order to solve the nonlinear problem existing in the industrial processes, many methods for extending the PLS to the nonlinear space have been proposed. In this paper, the support vector machine (SVM) is integrated into the PLS to get a nonlinear PLS model. In order to reduce information in process variables that are independent of quality variables, a orthogonal signal correction (OSC) method has been introduced. Nonlinear data from industrial processes is used to build nonlinear PLS models and SVM methods are adopted to approximate nonlinear internal relationships between the input and the output. The model proposed in this paper can predict industrial process quality variables and improve prediction accuracy compared with traditional PLS.

A class of multidimensional NIPALS algorithms for quaternion and tensor partial least squares regression

Abstract Quaternion and tensor-based signal processing benefits from exploiting higher dimensional structure in data to outperform the corresponding approaches using multivariate real algebras. Along with the extended range of processing options, these methods produce opportunities for a physically-meaningful interpretation. In this paper, we propose a class of novel partial least squares (PLS) algorithms for tensor- and quaternion-valued data, the widely linear quaternion PLS (WL-QPLS), the higher order nonlinear iterative PLS (HONIPALS) and the generalised higher order PLS (GHOPLS). This enables a regularised regression solution along with a latent variable decomposition of the original data based on the mutual information in the input and output block. The performance of the proposed algorithms is verified through analysis, together with a detailed comparison between quaternions and tensors and their application for image classification.

A Novel Nonlinear Partial Least Square Integrated With Error-Based Extreme Learning Machine

This paper proposes a novel nonlinear partial least square (PLS) approach for dealing with the modeling problem of industrial processes with input variables in collinearity. The new method combines the external linear PLS framework with the internal extreme learning machine (ELM) function. First, PLS is used as the outer framework to extract the input and output latent variables as well as eliminating the collinearity of the original variables, and then ELM is employed to describe the nonlinear relation between pairs of latent variables. Besides, the weight updating strategy based on errors minimization is also involved to improve the prediction accuracy. Then, the pH-neutralization process is taken as a benchmark to verify the validity of the new model. Finally, this method is applied to model the NOx emission of a 1000-MW coal-fired boiler, and root-mean-square error (RMSE) values are 5.9541 for the training dataset and 6.8323 for the testing dataset. Compared with linear PLS and another two nonlinear PLS methods, smaller prediction errors is obtained. The results indicate new nonlinear PLS model can be a better choice for establishing the model of NOx emission for coal-fired boilers.

Indoor Positioning Algorithm Based on Nonlinear PLS Integrated With RVM

Indoor positioning based on received signal strength indicator of WLAN has received more and more attention because of low cost and easy implementation. However, traditional localization algorithms often fail to achieve better positioning results because of multi-path effect and shadow effect. In order to solve the problem of multi-collinearity and more noise in WLAN indoor location data, this paper presents a novel nonlinear partial least square (PLS) method to address the problem of low precision in WLAN location. The proposed method integrates an inner relevant vector machine (RVM) function with an external linear PLS framework. First, the localization area is divided into a number of small areas by K-means algorithm. Then, PLS is applied to extract the features of the fingerprint database to reduce the number of the variable dimensions and eliminate the correlations. The obtained score matrices are used as the input and output of RVM. Finally, the coordinates of test points are regressed and predicted by the RVM-PLS algorithm. Simulation and experiments in real scenario prove the effectiveness of the proposed method. Compared with SVM-PLS, RBF-PLS, SVM-PCA, EBQPLS, PLS, SVM, RBF, RVM, and WKNN algorithm, the experimental results show that the proposed algorithm has higher positioning accuracy.

Self-adaptive evolving forecast models with incremental PLS space updating for on-line prediction of micro-fluidic chip quality

An important predictive maintenance task in modern production systems is to predict the quality of products in order to be able to intervene at an early stage to avoid faults and waste. Here, we address the prediction of the most important quality criteria in micro-fluidics chips: the flatness and critical size of the chips (in the form of RMSE values) and several transmission characteristics. Due to semi-manual inspection, these quality criteria are typically measured only intermittently. This leads to a high-dimensional batch process modeling problem with the goal of predicting chip quality based on the trends in these process values (time series). We apply time-series based transformation for dimension reduction to the lagged time-series space using of partial least squares (PLS), and combine this with a generalized form of Takagi–Sugeno(TS) fuzzy systems to obtain a non-linear PLS forecast model (termed as PLS-fuzzy). The rule consequent functions are robustly estimated by a weighted regularization scheme based on the idea of the elastic net approach. To address particular system dynamics over time, we propose dynamic updating of the non-linear PLS-fuzzy models using new on-line time-series data, with the options 1.) adapt and evolve the rule base on the fly, 2.) smoothly down-weight older samples to increase flexibility of the fuzzy models, and 3.) update the PLS space by incrementally adapting the loading vectors, where processing is achieved in a single-pass stream mining manner. We call our method IPLS-GEFS (incremental PLS combined with generalized evolving fuzzy systems). We applied our predictive modeling approach to data from on-line microfluidic chip production over a time period of about 6 months (July to December 2016). The results show that there is significant non-linearity in the predictive modeling problem, as the non-linear PLS-fuzzy modeling approach significantly outperformed classical PLS for most of the targets (quality criteria). Furthermore, it is important to update the models on the fly with incremental updating of the PLS space and/or with down-weighting older samples, as this significantly decreased the accumulated error trends of the prediction models compared to conventional updating. Reliable predictions of flatness quality (with around 10% error) and of RMSE values and transmissions (with around 15% errors) can be achieved with prediction horizons of up to 4 to 5 h into the future.

Quality-Relevant Fault Monitoring Based on Locality-Preserving Partial Least-Squares Statistical Models

Nonlinear partial least-squares (NPLS) is widely used in quality-relevant process control and fault diagnosis for strongly nonlinear systems; however, the existing NPLS approaches suffer from various disadvantages. This study proposes a novel statistical model based on locality-preserving partial least-squares (LPPLS) to enhance the processing capacity for system nonlinearity. The main concept of the LPPLS model is to utilize the locality-preserving projection to extract the principal components and preserve nonlinearities within the partial least-squares (PLS) process. The intuitive presentations for three types of LPPLS models are established within the proposed framework for strongly nonlinear systems, in which the process variables can correlate nonlinearly with each other and with the quality variables simultaneously. A canonical algorithm, which is easily applied in actual processes and is similar to the traditional linear PLS, is deduced to extract the principal components. Then, a quality-related ...

Linear guided autoencoder: Representation learning with linearity

In materials science, good representations of materials are important for use with prediction models in order to ensure accurate prediction of the properties of the output. In this paper, in order to address this issue, we use a learning system, linear guided autoencoder (LGAE) we call, which consists of an autoencoder and a linear predictor. For the autoencoder, we adopt a variant of the denoising autoencoder. In the LGAE, the learning addresses the unsupervised and supervised tasks simultaneously. Thus, the LGAE can be regarded as a form of nonlinear partial least squares (PLS) regression. Previous studies have not found the optimal solution for the encoder for an objective that contains both tasks. Our main contributions are a first-order approximation of the optimal solution and determination of the condition for linear solution that applies to the LGAE after training, in order to acquire knowledge from the nonlinear model (i.e., the LGAE). The main drawback of nonlinear PLS regression is that it is difficult to interpret the latent representation. Therefore, we propose a technical method for interpreting the latent representation. Experiments on benchmark datasets are conducted in order to compare the LGAE with kernel PLS regression, which is a powerful nonlinear PLS regression method. We also applied the LGAE to a dataset of methane storage materials in order to interpret the methane uptake based on the input variables and obtained reasonable results.

A novel nonlinear functional expansion based PLS (FEPLS) and its soft sensor application

A novel robust nonlinear partial least square model is proposed to handle the nonlinearity and collinearity problems of process data. The proposed model integrates a nonlinear functional link artificial neural network (FLANN) with a traditional partial least square (PLS). There are two parts in the proposed model: a nonlinear mapping part and a linear regression part. In the nonlinear mapping part, the input space is effectively extended to nonlinear space through the functional expansion block of FLANN. The PLS regression (PLSR) is adopted in the linear part. Thus, a novel robust nonlinear PLS integrated with functional expansion (FEPLS) is built. The proposed FEPLS model is very easy to construct. First, a traditional FLANN is selected. Second, the input space is expanded to nonlinear space using the functional expansion block. Third, the collinearity among the expanded variables and the expected outputs is eliminated by extracting input latent variables and output latent variables through PLS projection, respectively. Finally, an optimal regression model between the expanded variables and the expected outputs is established by using PLSR. To evaluate the performance of the proposed model, case studies of modeling two complex chemical processes are provided. Four more models of FLANN, extreme learning machine based PLS (ELM-PLS), kernel PLS (KPLS), and PLSR are also developed for comparisons. Simulation results illustrated that the proposed FEPLS model could improve the prediction performance.

Process modeling based on nonlinear PLS models using a prior knowledge-driven time difference method

In this study, we proposed a new nonlinear partial least squares (NPLS) method using time difference (TD) method for modeling chemical and environmental processes. The TD-NPLS method embeds the TD method into the NPLS modeling, where a prior knowledge related to key process information can be utilized. The NPLS can model the nonlinear inner relationships in a reduced data dimension and the TD method captures the gradual changes, which results in the deterioration of the predictive accuracy. The TD-NPLS method is aimed at achieving a high predictive accuracy by simultaneously capturing the characteristics of nonlinearity and dynamics in a single modeling framework. The results in three case studies, one simulated case and two industrial environmental cases showed that the TD-NPLS method absolutely has a higher predictive accuracy in modeling nonlinear and auto-correlated processes than the linear PLS, NPLS, and TD-LPLS models.

Discharge Coefficient of Triangular and Asymmetric Labyrinth Side Weirs Using the Nonlinear PLS Method

AbstractSide weirs are hydraulic control structures widely used in irrigation and drainage systems as well as wastewater treatment plants. These structures also serve as adjusting/diverting flow structures with a minimum head loss. In the present study, by applying experimental data of more than 400 laboratory tests and using the multivariable nonlinear partial least-squares (PLS) method, three nonlinear equations are proposed for discharge coefficient, CM, of triangular and asymmetric labyrinth side weirs. These equations relate the discharge coefficient to the relevant geometrical and hydraulic dimensionless characteristics of flow. Results indicate that the new proposed equations are more accurate compared to those of former studies, being the best tools to estimate the diverted flow discharge over the side weirs.