Multivariate Deep Reconstruction Neural Network for Multi-step-ahead Prediction of Industrial Process Quality Variables

Deep learning has been growing in popularity for soft sensor modeling of nonlinear industrial processes, infeuality-related variables. However, applications may be highly nonlinear, and the quantity of labeled samples is considerably limited. The extraction of relevant information from abundant unlabeled data is becoming an area of increasing interest in soft-sensor development. A novel ensemble deep relevant learning soft sensor (EDRLSS) modeling framework based on stacked autoencoder (SAE), mutual information (MI), and bagging-based strategy is proposed. SAE is trained layer-by-layer with MI analysis conducted between targeted outputs and learned hidden representations to evaluate and weight the current layer representations. The proposed method eliminates irrelevant information and weights the retained features to highlight the most relevant representations. Thus, the approach extracts deep representative information. Besides, a bagging-based ensemble strategy is applied to improve the soft-sensor performance and reliability. Two real-world industrial nonlinear processes are used to evaluate the EDRLSS framework performance. The results show enhanced prediction performance compared to other state-of-the-art and traditional methods.

Inter-Relational Mahalanobis SAE with semi-supervised strategy for fault classification in chemical processes

Nonlinear industrial soft sensor development based on semi-supervised probabilistic mixture of extreme learning machines

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

Soft sensors based on deep learning have been growing in industrial process applications, inferring hard-to-measure but crucial quality-related variables. However, applications may present strong non-linearity, dynamicity, and a lack of labeled data. To deal with the above-cited problems, the extraction of relevant features is becoming a field of interest in soft-sensing. A novel deep representative learning soft-sensor modeling approach is proposed based on stacked autoencoder (SAE), mutual information (MI), and long-short term memory (LSTM). SAE is trained layer by layer with MI evaluation performed between extracted features and targeted output to evaluate the relevance of learned representation in each layer. This approach highlights relevant information and eliminates irrelevant information from the current layer. Thus, deep output-related representative features are retrieved. In the supervised fine-tuning stage, an LSTM is coupled to the tail of the SAE to address system inherent dynamic behavior. Also, a k-fold cross-validation ensemble strategy is applied to enhance the soft-sensor reliability. Two real-world industrial non-linear processes are employed to evaluate the proposed method performance. The obtained results show improved prediction performance in comparison to other traditional and state-of-art methods. Compared to the other methods, the proposed model can generate more than 38.6% and 39.4% improvement of RMSE for the two analyzed industrial cases.

Soft sensors based on Gaussian mixture models (GMM) have been widely used in industrial process systems for modeling the nonlinearity, non-Gaussianity, and uncertainties. However, there are still some challenging issues in developing high-accuracy GMM-based soft sensors. First, labeled samples are usually scarce due to technical or economical limitations, causing traditional supervised GMM-based soft sensing methods fail to provide satisfactory performance. Second, tremendous amounts of unlabeled samples are gathered, nevertheless, how to fully exploit those unlabeled samples in terms of improving both the predictive accuracy and computational efficiency remains unresolved. In this paper, in order to deal with these issues, two computationally efficient soft sensing methods, namely the parallel computing-based semisupervised Dirichlet process mixture models (P–S $^2$ DPMM) and stochastic gradient descent-based S $^2$ DPMM (SGD–S $^2$ DPMM), are proposed. The S $^2$ DPMM is first developed to mine information contained in both labeled and unlabeled samples for predictive accuracy enhancement, and subsequently is extended to the P–S $^2$ DPMM and SGD–S $^2$ DPMM to handle large-scale process data with sufficient and limited computing resources, respectively. Two case studies are carried out on real-world industrial processes, and the results obtained demonstrate the effectiveness of the proposed methods.

Autoencoder-based nonlinear Bayesian locally weighted regression for soft sensor development

Purpose: Although deep learning (DL) models have shown increasing ability to accurately classify diagnostic images in oncology, significantly large amounts of well-curated data are often needed to match human level performance. Given the relative paucity of imaging datasets for less prevalent cancer types, there is an increasing need for methods which can improve the performance of deep learning models trained using limited diagnostic images. Deep metric learning (DML) is a potential method which can improve accuracy in deep learning models trained on limited datasets. Leveraging a triplet-loss function, DML exponentially increases training data compared to a traditional DL model. In this study, we investigated the utility of DML to improve the accuracy of DL models trained to classify cancerous lesions found on screening mammograms. Methods: Using a dataset of 2620 lesions found on routine screening mammogram, we trained both a traditional DL and DML models to classify suspicious lesions as cancerous or benign. The VGG16 architecture was used as the basis for the DL and DML models. Model performance was compared by calculating model accuracy, sensitivity, and specificity on a blinded test set of 378 lesions. In addition to individual model performance, we also measured agreement accuracy when both the DL and DML models were combined. Sub-analyses were conducted to identify phenotypes which were best suited for each model type. Both models underwent hyperparameters optimization to identify ideal batch size, learning rate, and regularization to prevent overfitting. Results: We found that the combination of the traditional DL model with DML model resulted in the highest overall accuracy (78.7%) representing a 7.1% improvement compared to the traditional DL model (p<.001). Alone, the traditional DL model had an improved accuracy compared to the DML model (71.4% vs 66.4%). The traditional DL model had a higher sensitivity (94.8% vs 73.6 %) , but lower specificity (34.7% vs 55.1%) compared the DML model. Sub-analyses suggested the traditional DL model was more accurate on higher density breasts, whereas the DML model was more accurate on lower density breasts. Additionally, the traditional DL model had the highest accuracy on oval shaped lesions, compared to the DML model which was most accurate on irregularly shaped breast lesions. Conclusion: Our study suggests that addition of DML models with traditional DL models can improve diagnostic image classification performance in cancer. Our results suggest DML models may provide increased specificity and help with classification of unique populations often misclassified by traditional DL models. Further studied investigating the utility of DML on other cancer imaging tasks are necessary to successfully build more robust DL models in cancer imaging. Citation Format: Justin Du, Sachin Umrao, Enoch Chang, Marina Joel, Aidan Gilson, Guneet Janda, Rachel Choi, Yongfeng Hui, Sanjay Aneja. The utility of deep metric learning for breast cancer identification on mammographic images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr 184.

Process mining is a new budding study in recent years which uses event logs and data. It is widely used in business organizations and it helps to improve the understanding of business processes, based on data. When a business process integrated with information systems provides the basis for new approach in data analysis. Process mining has a very good relationship with deep learning models that enables a strong relationship between business process management and business intelligence approach. Since the event data is available for process discovery and conformance checking techniques, process mining focuses on the entire process model. Process mining monitors and improves the real process by mining facts from event logs. While extracting knowledge there is no delay in today's information systems. The mining process models are used in various analysis. By applying deep learning techniques like deep neural networks and long short-term memory, risk management in a business process has been easily tracked and visualizing different activities involved in process prediction is also possible. Long Short-Term Memory of deep neural networks will be used for industrial process optimization, process improvement and process discovery in a Chemical Processes. It is well-suited to classify, process and predict time series given time lags of unknown duration. The revealed process models can be used for a large variety of analysis purposes. Deep fuzzy neural network framework that integrates the hidden feature of the CNN model and the LSTM model is proposed to improve the forecasting accuracy of business process outcome prediction.

A Causal Model-Inspired Automatic Feature-Selection Method for Developing Data-Driven Soft Sensors in Complex Industrial Processes

With the advancement of computer and sensor technology, soft sensors have been more and more extensively used in industrial processes. Soft sensors based on deep learning often need to redesign the structure and retrain the model when the prediction results are poor, which consumes a lot of time. Therefore, a deep cascade-gated broad learning system with fast update capability is proposed for industrial process soft sensor modeling. Being inspired by deep learning, the hidden layer features extracted by the autoencoder (AE) are used in the feature nodes of the broad learning system (BLS) to obtain the deep-BLS (D-BLS), which can circumvent the problem of insufficient feature extraction caused by stochastically generated weights in the feature nodes of BLS. On this basis, each feature node is integrated and sent to the enhancement nodes through the gated neurons. The enhancement nodes are cascaded to construct the deep cascaded-gated BLS (DC-GBLS), which can improve the prediction effect of the model while enhancing the utilization rate of the hidden layer features. Finally, a fast update method is developed for the model when the accuracy is insufficient. The validity and superiority of proposed model are demonstrated by two industrial processes.

Soft sensors have been widely accepted for online estimating key quality-related variables in industrial processes. The Gaussian mixture models (GMM) is one of the most popular soft sensing methods for the non-Gaussian industrial processes. However, in industrial applications, the quantity of samples with known labels is usually quite limited because of the technical limitations or economical reasons. Traditional GMM-based soft sensor models solely depending on labeled samples may easily suffer from singular covariances, overfitting, and difficulties in model selection, which results in the performance deterioration. To tackle these issues, we propose a semisupervised Bayesian GMM (S2BGMM). In the S2BGMM, we first propose a semisupervised fully Bayesian model, which enables learning from both the labeled and unlabeled datasets for remedying the deficiency of infrequent labeled samples. Subsequently, a general framework of weighted variational inference is developed to train the S2BGMM, such that the rate of learning from unlabeled samples can be controlled by penalizing the unlabeled dataset. Case studies are carried out to evaluate the performance of the S2BGMM through a numerical example and two real-world industrial processes, which demonstrate the effectiveness and reliability of the proposed approach.

Unsupervised and supervised deep learning extract effective and abstract features from different perspectives, which have been successfully applied in soft sensors. However, few studies have fused them and explored the complementary effect between the two kinds of features, which limits the utilization of comprehensive prediction information. To address the problem, a novel random subspace method with stacked auto-encoder (SAE) and bidirectional long short-term memory (Bi-LSTM), named RS-SBL, is proposed for soft sensors. Firstly, unsupervised and supervised deep representation features are extracted by SAE and Bi-LSTM, respectively. Secondly, to leverage the complementarity of the fusion features, an improved random subspcae (RS) method with a structure sparsity learning model is designed to discriminate the relative importance of different features and generate ensemble prediction results. Finally, the experiments on two real-world industrial nonlinear processes demonstrate that the proposed RS-SBL with the feature fusion strategy improves the prediction performance, and outperforms the other comparison soft sensor models.

The need for intubation in methanol-poisoned patients, if not predicted in time, can lead to irreparable complications and even death. Artificial intelligence (AI) techniques like machine learning (ML) and deep learning (DL) greatly aid in accurately predicting intubation needs for methanol-poisoned patients. So, our study aims to assess Explainable Artificial Intelligence (XAI) for predicting intubation necessity in methanol-poisoned patients, comparing deep learning and machine learning models. This study analyzed a dataset of 897 patient records from Loghman Hakim Hospital in Tehran, Iran, encompassing cases of methanol poisoning, including those requiring intubation (202 cases) and those not requiring it (695 cases). Eight established ML (SVM, XGB, DT, RF) and DL (DNN, FNN, LSTM, CNN) models were used. Techniques such as tenfold cross-validation and hyperparameter tuning were applied to prevent overfitting. The study also focused on interpretability through SHAP and LIME methods. Model performance was evaluated based on accuracy, specificity, sensitivity, F1-score, and ROC curve metrics. Among DL models, LSTM showed superior performance in accuracy (94.0%), sensitivity (99.0%), specificity (94.0%), and F1-score (97.0%). CNN led in ROC with 78.0%. For ML models, RF excelled in accuracy (97.0%) and specificity (100%), followed by XGB with sensitivity (99.37%), F1-score (98.27%), and ROC (96.08%). Overall, RF and XGB outperformed other models, with accuracy (97.0%) and specificity (100%) for RF, and sensitivity (99.37%), F1-score (98.27%), and ROC (96.08%) for XGB. ML models surpassed DL models across all metrics, with accuracies from 93.0% to 97.0% for DL and 93.0% to 99.0% for ML. Sensitivities ranged from 98.0% to 99.37% for DL and 93.0% to 99.0% for ML. DL models achieved specificities from 78.0% to 94.0%, while ML models ranged from 93.0% to 100%. F1-scores for DL were between 93.0% and 97.0%, and for ML between 96.0% and 98.27%. DL models scored ROC between 68.0% and 78.0%, while ML models ranged from 84.0% to 96.08%. Key features for predicting intubation necessity include GCS at admission, ICU admission, age, longer folic acid therapy duration, elevated BUN and AST levels, VBG_HCO3 at initial record, and hemodialysis presence. This study as the showcases XAI's effectiveness in predicting intubation necessity in methanol-poisoned patients. ML models, particularly RF and XGB, outperform DL counterparts, underscoring their potential for clinical decision-making.

This paper aimed to develop and validate a deep learning (DL) model for automated detection of the laterality of the eye on anterior segment photographs. Anterior segment photographs for training a DL model were collected with the Scheimpflug anterior segment analyzer. We applied transfer learning and fine-tuning of pre-trained deep convolutional neural networks (InceptionV3, VGG16, MobileNetV2) to develop DL models for determining the eye laterality. Testing datasets, from Scheimpflug and slit-lamp digital camera photography, were employed to test the DL model, and the results were compared with a classification performed by human experts. The performance of the DL model was evaluated by accuracy, sensitivity, specificity, operating characteristic curves, and corresponding area under the curve values. A total of 14,468 photographs were collected for the development of DL models. After training for 100 epochs, the DL models of the InceptionV3 mode achieved the area under the receiver operating characteristic curve of 0.998 (with 95% CI 0.924–0.958) for detecting eye laterality. In the external testing dataset (76 primary gaze photographs taken by a digital camera), the DL model achieves an accuracy of 96.1% (95% CI 91.7%–100%), which is better than an accuracy of 72.3% (95% CI 62.2%–82.4%), 82.8% (95% CI 78.7%–86.9%) and 86.8% (95% CI 82.5%–91.1%) achieved by human graders. Our study demonstrated that this high-performing DL model can be used for automated labeling for the laterality of eyes. Our DL model is useful for managing a large volume of the anterior segment images with a slit-lamp camera in the clinical setting.