Adversarial Sample Based Semi-Supervised Learning for Industrial Soft Sensor

Adversarial smoothing tri-regression for robust semi-supervised industrial soft sensor

Semi-supervised ensemble support vector regression based soft sensor for key quality variable estimation of nonlinear industrial processes with limited labeled data

Nowadays, soft sensor techniques have become promising solutions for enabling real-time estimation of difficult-to-measure quality variables in industrial processes. However, labeled data are often scarce in many real-world applications, which poses a significant challenge when building accurate soft sensor models. Therefore, this paper proposes a novel semi-supervised soft sensor method, referred to as ensemble semi-supervised negative correlation learning extreme learning machine (EnSSNCLELM), for industrial processes with limited labeled data. First, an improved supervised regression algorithm called NCLELM is developed, by integrating the philosophy of negative correlation learning into extreme learning machine (ELM). Then, with NCLELM as the base learning technique, a multi-learner pseudo-labeling optimization approach is proposed, by converting the estimation of pseudo labels as an explicit optimization problem, in order to obtain high-confidence pseudo-labeled data. Furthermore, a set of diverse semi-supervised NCLELM models (SSNCLELM) are developed from different enlarged labeled sets, which are obtained by combining the labeled and pseudo-labeled training data. Finally, those SSNCLELM models whose prediction accuracies were not worse than their supervised counterparts were combined using a stacking strategy. The proposed method can not only exploit both labeled and unlabeled data, but also combine the merits of semi-supervised and ensemble learning paradigms, thereby providing superior predictions over traditional supervised and semi-supervised soft sensor methods. The effectiveness and superiority of the proposed method were demonstrated through two chemical applications.

Quantum statistic based semi-supervised learning approach for industrial soft sensor development

Nonlinear industrial soft sensor development based on semi-supervised probabilistic mixture of extreme learning machines

Soft sensors have become reliable tools for estimating difficult-to-measure target variables in modern industrial processes. In order to make full use of labeled and unlabeled samples, an active semi-supervised soft sensor modeling method is proposed, which combines active learning and semi-supervised learning to maximize model performance and minimize the laboratory analysis cost of expanding the labeled sample data set. First, manifold regularization is introduced into the deep extreme learning machine (DELM) algorithm to form a semi-supervised DELM that improves the performance of a model trained with unlabeled samples. Then, considering non-Gaussian processes and the error information between the predicted and true values, an active sample selection strategy based on error Gaussian mixture model is developed. Using this strategy, the most uncertain and representative unlabeled samples are selected for labeling, and thereby expanding the labeled sample data set. Finally, the effectiveness of the proposed method is verified using industrial debutanizer process data.

Neural-network-based soft sensors are widely employed in the industrial process. Such models have great significance to smart manufacturing. Considering the strict requirements of industrial production, it is vital to ensure the safety and robustness of these models in their actual deployment. However, recent research has shown that neural networks are quite vulnerable to adversarial attacks. By imposing tiny perturbation to the original sample, the fabricated adversarial sample can cheat these models to make wrong decisions. Such a phenomenon may bring serious trouble to the practical application of soft sensors. This article focuses on the adversarial attacks on industrial soft sensors. For the first time, we verify and analyze the effectiveness and deficiencies of the existing attack methods in the industrial soft sensor scenario. Based on solving these defects, this article proposes a novel perspective for attacking soft sensors. We analyze the optimization mechanism behind this new idea and then design two algorithms to perform attacks. The proposed methods more conform to the actual situation. Besides, compared with the existing approaches, the proposed methods have potentials to cause severer damages since their attacks are not only more concealed but also more likely to cheat the technicians to execute wrong operations. The research and analyses of the proposed methods lay a solid foundation for more thorough defenses against various attacks, which is quite necessary for making the deployed soft sensors more robust and secure.

In industrial processes, inferential sensors have been extensively applied for prediction of quality variables that are difficult to measure online directly by hard sensors. Deep learning is a recently developed technique for feature representation of complex data, which has great potentials in soft sensor modeling. However, it often needs a large number of representative data to train and obtain a good deep network. Moreover, layer-wise pretraining often causes information loss and generalization degradation of high hidden layers. This greatly limits the implementation and application of deep learning networks in industrial processes. In this article, a layer-wise data augmentation (LWDA) strategy is proposed for the pretraining of deep learning networks and soft sensor modeling. In particular, the LWDA-based stacked autoencoder (LWDA-SAE) is developed in detail. Finally, the proposed LWDA-SAE model is applied to predict the 10% and 50% boiling points of the aviation kerosene in an industrial hydrocracking process. The results show that the LWDA-SAE-based soft sensor is superior to multilayer perceptron, traditional SAE, and the SAE with data augmentation only for its input layer (IDA-SAE). Moreover, LWDA-SAE can converge at a faster speed with a lower learning error than the other methods.

Despite their high prediction accuracy, deep learning-based soft sensor (DLSS) models face challenges related to adversarial robustness against malicious adversarial attacks, which hinder their widespread deployment and safe application. Although adversarial training is the primary method for enhancing adversarial robustness, existing adversarial-training-based defense methods often struggle with accurately estimating transfer gradients and avoiding adversarial robust overfitting. To address these issues, we propose a novel adversarial training approach, namely domain-adaptive adversarial training (DAAT). DAAT comprises two stages: historical gradient-based adversarial attack (HGAA) and domain-adaptive training. In the first stage, HGAA incorporates historical gradient information into the iterative process of generating adversarial samples. It considers gradient similarity between iterative steps to stabilize the updating direction, resulting in improved transfer gradient estimation and stronger adversarial samples. In the second stage, a soft sensor domain-adaptive training model is developed to learn common features from adversarial and original samples through domain-adaptive training, thereby avoiding excessive leaning toward either side and enhancing the adversarial robustness of DLSS without robust overfitting. To demonstrate the effectiveness of DAAT, a DLSS model for crystal quality variables in silicon single-crystal growth manufacturing processes is used as a case study. Through DAAT, the DLSS achieves a balance between defense against adversarial samples and prediction accuracy on normal samples to some extent, offering an effective approach for enhancing the adversarial robustness of DLSS.

Semi-supervised LSTM with historical feature fusion attention for temporal sequence dynamic modeling in industrial processes

A semi-supervised soft sensor method based on vine copula regression and tri-training algorithm for complex chemical processes

The growth of data collection in industrial processes has led to a renewed emphasis on the development of data-driven soft sensors. A key step in building an accurate, reliable soft sensor is feature representation. Deep networks have shown great ability to learn hierarchical data features using unsupervised pretraining and supervised fine-tuning. For typical deep networks like stacked auto-encoder (SAE), the pretraining stage is unsupervised, in which some important information related to quality variables may be discarded. In this article, a new quality-driven regularization (QR) is proposed for deep networks to learn quality-related features from industrial process data. Specifically, a QR-based SAE (QR-SAE) is developed, which changes the loss function to control the weights of the different input variables. By choosing an appropriate inductive bias for the weight matrix, the model provides quality-relevant information for predictive modeling. Finally, the proposed QR-SAE is used to predict the quality of a real industrial hydrocracking process. Comparative experiments show that QR-SAE can extract quality-related features and achieve accurate prediction performance.

Prediction uncertainty has rarely been integrated into traditional soft sensors in industrial processes. In this work, a novel autoswitch probabilistic soft sensor modeling method is proposed for online quality prediction of a whole industrial multigrade process with several steady-state grades and transitional modes. It is different from traditional deterministic soft sensors. Several single Gaussian process regression (GPR) models are first constructed for each steady-state grade. A new index is proposed to evaluate each GPR-based steady-state grade model. For the online prediction of a new sample, a prediction variance-based Bayesian inference method is proposed to explore the reliability of existing GPR-based steady-state models. The prediction can be achieved using the related steady-state GPR model if its reliability using this model is large enough. Otherwise, the query sample can be treated as in transitional modes and a local GPR model in a just-in-time manner is online built. Moreover, to improve the efficiency, detailed implementation steps of the autoswitch GPR soft sensors for a whole multigrade process are developed. The superiority of the proposed method is demonstrated and compared with other soft sensors in an industrial process in Taiwan, in terms of online quality prediction.

Gaussian mixture regression (GMR) is an effective tool in developing soft sensors for online estimating difficult-to-measure variables in industrial processes with multiple operating modes. However, the GMR usually requires a sufficient amount of labeled samples to guarantee accurate probability density function (PDF) estimations because of its supervised learning process. Unfortunately, in soft-sensor applications, labeled samples could be very infrequent due to technical or economic limitations, which may lead the GMR-based soft sensors to unreliable parameter estimation and model selection, resulting in poor prediction performance. To tackle this problem, a semisupervised GMR (S2GMR) was proposed, where both labeled and unlabeled samples were effective. In the S2GMR, the PDFs of Gaussian components in input space and the functional dependence between input and output variables were learned simultaneously based on the expectation–maximization algorithm. Moreover, the Bayesian information criterion was employed to automatically determine the number of Gaussians for the S2GMR. The S2GMR was first investigated by a numerical example, and then applied to a real-life ammonia synthesis process for estimating the oxygen concentration at the top of the primary reformer. The two case studies verified the effectiveness of the proposed method.

Deep learning with neighborhood preserving embedding regularization and its application for soft sensor in an industrial hydrocracking process