Locally Weighted Partial Least Squares Research Articles

Rapid Determination of the Total Petroleum Hydrocarbon Content of Soils by Handheld Fourier Transform Near-Infrared Spectroscopy.

For successful soil remediation and hydrocarbon exploration operations, determining the total petroleum hydrocarbon (TPH) content of soils is an indispensable process step. This paper reports on the performance of a handheld Fourier transform near-infrared (FT-NIR) spectrometer for rapid and quantitative determination of TPH content of soils from two different sites by diffuse reflection measurements. For rapid decisions for exploration work or environmental site assessment projects, a quick─preferably on-site─determination of TPH content is valuable. Diffuse reflection NIR spectra were recorded from soil samples of two different sites with TPH reference values ranging from 350 to 30,000 ppm, as determined by capillary gas chromatography and flame ionization detection with hydrocarbon fingerprinting C1-C44. However, this paper not only addresses the development of site-specific partial-least squares (PLS) calibrations but also demonstrates the locally-weighted PLS (LW-PLS) technique, which can be used to develop global, site-independent PLS calibrations without significant penalty in calibration performance. As a first step, the diffuse reflection spectra were used to develop conservative, site-specific PLS calibration models with root-mean-square calibration/cross-validation errors (RMSEC/RMSECV) of 1043/1106 and 741/785 ppm TPH, respectively, and the average absolute prediction errors for samples not contained in the calibration set were 451 and 293 ppm for the two sites, respectively. In a further step, significant degradation of the RMSE values of a conservative PLS model based on the NIR spectra of both sites was then compared to the application of the LW-PLS method, with only a slight loss of the prediction accuracy relative to the site-independent models. This study confirms the ability of next-generation portable FT-NIR spectrometers to predict low TPH levels in various soil types through both─soil-specific and site-independent─calibrations, giving these spectrometers the potential to become rapid screening tools in the field.

A Comparative Study of Different Kernel Functions Applied to LW-KPLS Model for Nonlinear Processes

Soft sensors are inferential estimators when the employment of hardware sensors is inapplicable, expensive, or difficult in industrial plant processes. Currently, a simple soft sensor, namely locally weighted partial least squares (LW-PLS), which can cope with the nonlinearity of the process, has been developed. However, LW-PLS exhibits the disadvantages of handling strong nonlinear process data. To address this problem, Kernel functions are integrated into LW-PLS to form locally weighted Kernel partial least squares (LW-KPLS). Notice that a minimal study was carried out on the impact of different kernel functions that have not been integrated with the LW-KPLS, in which this model has the potential to be applied to different chemical-related nonlinear processes. Thus, this study investigates the predictive performance of LW-KPLS with several different Kernel functions using three nonlinear case studies. As the results, the predictive performances of LW-KPLS with Polynomial Kernel are better than other Kernel functions. The values of root-mean-square errors (RMSE) and error of approximation (Ea) for the training and testing dataset by utilizing this Kernel function are the lowest in their respective case studies, which are 34.60% to 95.39% lower for RMSEs values and 68.20% to 95.49% smaller for Ea values.

Adaptive soft sensor ensemble for selecting both process variables and dynamics for multiple process states

To improve the predictive ability of soft sensors in chemical and industrial plants, the selection of process variables and consideration of dynamics in the processes have been studied. When multiple process states exist in plants, process variables that are related to an objective variable (y) and their corresponding time delays can be different in each process state. In this paper, we propose a method to optimize process variables and dynamics for each process state, and predict the objective variable values using multiple adaptive soft sensors according to process states. First, a dataset is clustered using Gaussian mixture models; then, time-delayed process variables are selected as explanatory variables (x) for each cluster using genetic algorithm-based process variable and dynamics selection. For each set of explanatory variables, a nonlinear adaptive soft sensor is constructed. An ensemble prediction is performed by assigning a weight to each locally weighted partial least squares (LWPLS) model according to its predictive ability. The novelty of the proposed method is the construction of adaptive soft sensors that are optimized for each process state in a plant where multiple process states exist, and the accurate prediction of objective variable values through integrating the constructed adaptive soft sensors as the process state changes. The effectiveness of the proposed method was tested using two datasets from actual plants, and we confirmed that it could accurately predict the objective variable values in each process state.

A comparative study between PCR, PLSR, and LW-PLS on the predictive performance at different data splitting ratios

Principal component regression (PCR), partial least squares regression (PLSR), and locally weighted partial least squares (LW-PLS) models are supervised learning methods in which a labeled dataset is used to train the model. The split-sample validation is normally used to train these models where a dataset is split into training and testing datasets to develop and evaluate the model. However, a limited study is done to evaluate the prediction performance of PCR, PLSR, and LW-PLS models at the different data splitting ratios. Hence, to address this research gap, this submitted work is conducted to investigate the predictive performance of the abovementioned regression models at the different split sample ratios for the data. Meanwhile, this study also serves to determine the optimal splitting ratios for PCR, PLSR, and LW-PLS models via a simple data splitting method where a minimum of 50% of the entire dataset is allocated to train the model. The optimal split is determined by evaluating the root mean squared error, coefficient of determination, and error of approximation (Ea ) for five case studies. For PCR, PLSR, and LW-PLS models, LW-PLS performed better in most of the case studies since it copes better with the nonlinear data. Among these best models in each case study, it was found that the split-sample ratios of above 70% of training data had allowed major improvements in terms of predictive performance as compared to their base scenarios which have the largest Ea values.

Estimating the reliability of predictions in locally weighted partial least‐squares modeling

AbstractIt is important to predict the reliability of the estimation results given by adaptive soft sensors. In this study, a locally weighted partial least‐squares (LWPLS) method, which is a just‐in‐time‐based adaptive soft sensor, is analyzed, and the reliability of LWPLS modeling is predicted as the standard deviation of the estimated values of an objective variable y. The relationship between the minimum of Euclidean distance (MinED) and the standard deviation of y errors (SDYE) is constructed using training samples, giving the proposed y‐error model. For the test samples, the MinED from a query to the training samples is input into the y‐error model, allowing the SDYE for the query to be predicted. The proposed LWPLS model can estimate the y values with associated error bars, which indicate the reliability of the estimated y values. The effectiveness of the proposed method is demonstrated through two case studies using datasets from industrial plants.

Fast Locally Weighted PLS Modeling for Large-Scale Industrial Processes

Locally weighted partial least-squares (LW-PLS) is an efficient just-in-time (JIT) modeling method, which can handle process collinearity, nonlinearity, and time-varying characteristics. However, it is not suitable for modeling large-scale industrial processes due to its huge computational cost. To solve this issue, the present work proposes a novel fast LW-PLS (FLW-PLS) method that can handle large-scale process data well. FLW-PLS is designed based on the exact Euclidean locality-sensitive hashing algorithm. Unlike standard LW-PLS which implements a brute-force linear search of similar samples over a reference data set, FLW-PLS searches for similar samples in sublinear time. Thus, significant computational savings can be obtained by FLW-PLS. The effectiveness of the proposed FLW-PLS method was validated through the case studies of predicting the silicon content and phosphorus content in the ironmaking process. The application results have shown that when faced with large-scale data, FLW-PLS can significantly reduce the prediction time without significantly reducing the prediction accuracy in comparison to the conventional LW-PLS method.

A Novel Just-in-Time Learning Strategy for Soft Sensing with Improved Similarity Measure Based on Mutual Information and PLS.

In modern industrial process control, just-in-time learning (JITL)-based soft sensors have been widely applied. An accurate similarity measure is crucial in JITL-based soft sensor modeling since it is not only the basis for selecting the nearest neighbor samples but also determines sample weights. In recent years, JITL similarity measure methods have been greatly enriched, including methods based on Euclidean distance, weighted Euclidean distance, correlation, etc. However, due to the different influence of input variables on output, the complex nonlinear relationship between input and output, the collinearity between input variables, and other complex factors, the above similarity measure methods may become inaccurate. In this paper, a new similarity measure method is proposed by combining mutual information (MI) and partial least squares (PLS). A two-stage calculation framework, including a training stage and a prediction stage, was designed in this study to reduce the online computational burden. In the prediction stage, to establish the local model, an improved locally weighted PLS (LWPLS) with variables and samples double-weighted was adopted. The above operations constitute a novel JITL modeling strategy, which is named MI-PLS-LWPLS. By comparison with other related JITL methods, the effectiveness of the MI-PLS-LWPLS method was verified through case studies on both a synthetic Friedman dataset and a real industrial dataset.

A spatial-temporal LWPLS for adaptive soft sensor modeling and its application for an industrial hydrocracking process

Locally weighted partial least squares (LWPLS) is a widely used just-in-time learning (JITL) modeling algorithm for adaptive soft sensor development. In LWPLS, spatial variable distance is used to measure similarity and assign weights for historical samples, which is very effective to handle process time-varying problems of abrupt changes. However, the gradual process changes are not effectively handled in traditional LWPLS. To cope with this problem, a novel similarity is proposed for temporal distance measurement by introducing a temporal variable of sampling instant, in which newest sampled data can get large weights since they represent the more recent process state. Then, both spatial and temporal similarities are considered to construct a spatial-temporal adaptive LWPLS modeling framework in this paper. The effectiveness of the proposed algorithm is validated on an industrial hydrocracking process.

Multivariate Regression Model for Industrial Process Measurement Based on Double Locally Weighted Partial Least Squares

Multivariate regression models are commonly used in industrial process measurements. Locally weighted partial least squares (LWPLS) is a just-in-time learning (JITL) method for nonlinear multivariate regression models. However, it considers only the sample importance. In this article, a novel JITL multivariate regression model termed double LWPLS (D-LWPLS) is proposed to solve complex nonlinear problems and promote the model accuracy. Both the sample importance and variable importance are taken into consideration for the weighted model. Variable contribution is modified according to the variable importance. Sample similarity measurement is a significant basis for the model sample selection and sample weight calculation. The traditional sample similarity measurement methods take only the input information of samples as the index. In the D-LWPLS, a weighted similarity measurement is employed. The information on both input and output is considered by weighting the input variables according to the correlation between them. The effectiveness and the flexibility of the D-LWPLS model are demonstrated by the prediction results of a numerical example, a low-dimensional data set of an industrial process, and a high-dimensional data set of UV-Vis spectra of a polymetallic ion solution, which represent three kinds of nonlinear problems of classical, time-varying, and blind background fluctuation.

An Improved Locally Weighted PLS Based on Particle Swarm Optimization for Industrial Soft Sensor Modeling.

In industrial production, soft sensors play very important roles in ensuring product quality and production safety. Traditionally, global modeling methods, which use historical data to construct models offline, are often used to develop soft sensors. However, because of various complex and unknown changes in industrial production processes, the performance of global models deteriorates over time, and frequent model maintenance is difficult. In this study, locally weighted partial least squares (LWPLS) is adopted as a just-in-time learning method for industrial soft sensor modeling. In LWPLS, the bandwidth parameter h has an important impact on the performance of the algorithm, since it decides the range of the neighborhood and affects how the weight changes. Therefore, we propose a two-phase bandwidth optimization strategy that combines particle swarm optimization (PSO) and LWPLS. A numerical simulation example and an industrial application case were studied to estimate the performance of the proposed PSO–LWPLS method. The results show that, compared to the traditional global methods and the LWPLS with a fixed bandwidth, the proposed PSO–LWPLS can achieve a better prediction performance. The results also prove that the proposed method has apparent advantages over other methods in the case of data density changes.

Missing RRI Interpolation Algorithm based on Locally Weighted Partial Least Squares for Precise Heart Rate Variability Analysis.

The R-R interval (RRI) fluctuation in electrocardiogram (ECG) is called heart rate variability (HRV), which reflects activities of the autonomic nervous system (ANS) and has been used for various health monitoring services. Accurate R wave detection is crucial for success in HRV-based health monitoring services; however, ECG artifacts often cause missing R waves and deteriorate the accuracy of HRV analysis. The present work proposes a new missing RRI interpolation technique based on Just-In-Time (JIT) modeling. In the JIT modeling framework, a local regression model is built by weighing samples stored in the database according to the distance from a query and output is estimated only when an estimate is requested. The proposed method builds a local model and estimates missing RRI only when an RRI detection error is detected. Locally weighted partial least squares (LWPLS) is adopted for local model construction. The proposed method is referred to as LWPLS-based RRI interpolation (LWPLS-RI). The performance of the proposed LWPLS-RI was evaluated through its application to RRI data with artificial missing RRIs. We used the MIT-BIH Normal Sinus Rhythm Database for nominal RRI dataset construction. Missing RRIs were artificially introduced and they were interpolated by the proposed LWPLS-RI. In addition, MEAN that replaces the missing RRI by a mean of the past RRI data was compared as a conventional method. The result showed that the proposed LWPLS-RI improved root mean squared error (RMSE) of RRI by about 70% in comparison with MEAN. In addition, the proposed method realized precise HRV analysis. The proposed method will contribute to the realization of precise HRV-based health monitoring services.

Hybrid Online Multi-Sensor Error Detection and Functional Redundancy for Artificial Pancreas Control Systems

Sensor errors limit the performance of a supervision and control system. Sensor accuracy can be affected by many factors such as extreme working conditions, sensor deterioration and interferences from other devices. It may be difficult to distinguish sensor errors and real dynamic changes in a system. A hybrid online multi-sensor error detection and functional redundancy (HOMSED&FR) algorithm is developed to monitor the performance of multiple sensors and reconcile the erroneous sensor signals. The algorithm relies on two methods, outlier-robust Kalman filter (ORKF) and a locally-weighted partial least squares (LW-PLS) regression model. The two methods have different way of using data, ORKF is comparing current signal samples with the signal trace indicated by previous samples and LW-PLS is comparing samples in the past window with the samples from a database and uses the samples with the most similarity to build a model to predict the current signal values. The performance of this system is illustrated with a clinical case involving artificial pancreas experiments, which include data from a continuous glucose monitoring (CGM) sensor, and energy expenditure (EE) and Galvanic Skin Response (GSR) information based on wearable sensors that collect data from people with type 1 diabetes. The results indicate that the proposed method can successfully detect most of the erroneous signals and substitute them with reasonably estimated values computed by the functional redundancy system.

Hybrid online sensor error detection and functional redundancy for systems with time-varying parameters

Supervision and control systems rely on signals from sensors to receive information to monitor the operation of a system and adjust manipulated variables to achieve the control objective. However, sensor performance is often limited by their working conditions and sensors may also be subjected to interference by other devices. Many different types of sensor errors such as outliers, missing values, drifts and corruption with noise may occur during process operation. A hybrid online sensor error detection and functional redundancy system is developed to detect errors in online signals, and replace erroneous or missing values detected with model-based estimates. The proposed hybrid system relies on two techniques, an outlier-robust Kalman filter (ORKF) and a locally-weighted partial least squares (LW-PLS) regression model, which leverage the advantages of automatic measurement error elimination with ORKF and data-driven prediction with LW-PLS. The system includes a nominal angle analysis (NAA) method to distinguish between signal faults and large changes in sensor values caused by real dynamic changes in process operation. The performance of the system is illustrated with clinical data continuous glucose monitoring (CGM) sensors from people with type 1 diabetes. More than 50,000 CGM sensor errors were added to original CGM signals from 25 clinical experiments, then the performance of error detection and functional redundancy algorithms were analyzed. The results indicate that the proposed system can successfully detect most of the erroneous signals and substitute them with reasonable estimated values computed by functional redundancy system.

Missing RRI interpolation for HRV analysis using locally-weighted partial least squares regression.

The R-R interval (RRI) fluctuation in electrocardiogram (ECG) is called heart rate variability (HRV). Since HRV reflects autonomic nervous function, HRV-based health monitoring services, such as stress estimation, drowsy driving detection, and epileptic seizure prediction, have been proposed. In these HRV-based health monitoring services, precise R wave detection from ECG is required; however, R waves cannot always be detected due to ECG artifacts. Missing RRI data should be interpolated appropriately for HRV analysis. The present work proposes a missing RRI interpolation method by utilizing using just-in-time (JIT) modeling. The proposed method adopts locally weighted partial least squares (LW-PLS) for RRI interpolation, which is a well-known JIT modeling method used in the filed of process control. The usefulness of the proposed method was demonstrated through a case study of real RRI data collected from healthy persons. The proposed JIT-based interpolation method could improve the interpolation accuracy in comparison with a static interpolation method.

Proximate Analysis of Sawdust Using Near Infrared Spectroscopy and Locally Weighted Partial Least Squares

In recently years, biomass as a renewable and widely available source of energy has increasingly been used in power generation industry and the concept of bio-power is widely accepted. However, conventional methods for proximate analysis are time consuming and can only be performed in laboratory. In this paper, 110 biomass samples are collected and near infrared spectroscopy (NIRS) technology is applied to predict proximate analysis of samples. The data show that NIRS combined with the locally weighted partial least squares (LW-PLS) obtained better prediction results comparing to conventional methods like principal component regression (PCR) and partial least squares (PLS).

Sparse Sample Regression Based Just-In-Time Modeling (SSR-JIT): Beyond Locally Weighted Approach

In the present work, a new method for just-in-time (JIT) modeling is proposed. To develop virtual sensors or soft-sensors that can cope with changes in process characteristics as well as nonlinearity, JIT modeling such as locally weighted regression (LWR) and locally weighted partial least squares (LW-PLS) has been investigated and successfully used in various industries. The conventional JIT modeling methods predict output variables by constructing a local model by using past samples located in the neighborhood around the new target sample (query) every time when the output prediction is required; the modeling samples are selected or weighted according to the similarity between the samples and the query. The similarity is usually determined on the basis of the distance from the query. However, the use of distance does not assure the high prediction accuracy. To overcome this limitation of the conventional JIT methods, the proposed method selects past samples that are useful for constructing an accurate local model by using elastic net, which builds a sparse regression model to estimate the query, and uses the derived regression coefficients to evaluate the similarity for conducting LW-PLS. This sparse sample regression based just-in-time modeling (SSR-JIT) has a potential for surpassing the conventional distance-based JIT modeling. In fact, it was demonstrated that SSR-JIT outperformed LW-PLS in the prediction accuracy through two case studies with real industrial data.

Hybrid Online Sensor Error Detection and Functional Redundancy for Artificial Pancreas Control Systems

Artificial pancreas (AP) control systems rely on signals from glucose sensors to collect glucose concentration (GC) information from people with Type 1 diabetes and compute insulin infusion rates to maintain GC within a desired range Sensor performance is often limited by sensor errors, communication interruptions and noise A hybrid online sensor error detection and functional redundancy system is developed to detect errors in online signals, and replace erroneous or missing values detected with model-based estimates. The proposed hybrid system relies on two techniques, an outlier-robust Kalman filter (ORKF) and a locally-weighted partial least squares (LW-PLS) regression model. This leverages the advantages of automatic measurement error elimination with ORKF and data-driven prediction with LW-PLS. A novel method called nominal angle analysis is proposed to distinguish between signal faults and large changes in sensor values caused by real dynamic changes in the metabolism The performance of the system is illustrated with clinical data from continuous glucose monitoring sensors collected from people with Type 1 diabetes.

Ensemble locally weighted partial least squares as a just‐in‐time modeling method

The predictive ability of soft sensors, which estimate values of an objective variable y online, decreases due to process changes in chemical plants. To reduce the decrease of predictive ability, adaptive soft sensors have been developed. We focused on just‐in‐time soft sensors, especially locally weighted partial least squares (LWPLS) regression. Since a set of hyperparameters in an LWPLS model has to be set beforehand and there is only onedataset, a traditional LWPLS model is difficult to accurately predict y‐values in multiple process states. In this study, we propose to combine LWPLS and ensemble learning, and predict y‐values with multiple LWPLS models, whose datasets and sets of hyperparameters are different. The weights of LWPLS models are determined based on Bayes’ theorem, considering their predictive ability. We confirmed that the proposed model has higher predictive accuracy than traditional models through numerical simulation data and two industrial data analyses. © 2015 American Institute of Chemical Engineers AIChE J, 62: 717–725, 2016

Covariance-based locally weighted partial least squares for high-performance adaptive modeling

Locally weighted partial least squares (LW-PLS) is one of Just-in-Time (JIT) modeling methods; PLS is used to build a local linear regression model every time when output variables need to be estimated. The prediction accuracy of local models strongly depends on the definition of similarity between a newly obtained sample and past samples stored in a database. To calculate the similarity, the Euclidean distance and the Mahalanobis distance have been widely used, but they do not take account of the relationship between input and output variables. This fact limits the achievable performance of LW-PLS and other locally weight regression methods. Thus, in the present work, covariance-based locally weighted PLS (CbLW-PLS) is proposed by integrating LW-PLS and a new similarity index based on the covariance between input and output variables. CbLW-PLS was applied to two industrial problems: soft-sensor design for estimating unreacted NaOH concentration in an alkali washing tower in a petrochemical process, and process analytical technology (PAT) for estimating concentration of a residual drug substance in a pharmaceutical process. The proposed similarity index was compared with six conventional indexes based on distances, correlations, or regression coefficients. The results have demonstrated that CbLW-PLS achieved the best prediction performance of all in both case studies.

Adaptive Virtual Metrology Design for Semiconductor Dry Etching Process Through Locally Weighted Partial Least Squares

In semiconductor manufacturing processes, virtual metrology (VM) has been investigated as a promising tool to predict important characteristics of products. Although partial least squares (PLS) is a well-known modeling technique that can cope with collinearity and therefore applied to construction of VM, its prediction performance deteriorates due to changes in process characteristics. In particular, maintenance of equipment strongly affects the process characteristics and the prediction performance. In this paper, VM was developed by using locally weighted PLS (LW-PLS), which is a type of just-in-time modeling technique, and it was used to predict the etching conversion differential of an actual dry etching process. The industrial application results have shown that the developed VM based on LW-PLS is superior to the conventional VM based on the sequential update model and artificial neural network model. In particular, it has been confirmed that the LW-PLS-based VM can keep its high prediction performance even after the maintenance, i.e., replacement of parts.