Latent Variable Space Research Articles

Data driven latent variable adaptive control for nonlinear multivariable processes

This article aims at a new partial least squares (PLS) control design and analysis in a data-driven framework for nonlinear multivariable processes whose mechanistic models are completely unknown. First, a general nonlinear autoregressive moving average with the exogenous input (NARMAX) model is used as the dynamic nonlinear PLS model. Then, by introducing a dynamic linearisation approach in each latent variable (LV) space, the unknown NARMAX-based PLS model is transformed to a linear dynamic PLS data model (dPLSDM), which can be improved in real time by estimating its unknown parameter using the latent input and output (I/O) data. Next, a data-driven latent variable adaptive control (DDLVAC) is proposed in each LV loop. By virtue of the dPLSDM, the multivariable nonlinear process is decoupled into multiple single-loop systems and the high dimensions of the process data are reduced such that the corresponding DDLVAC is simplified. Further, the DDLVAC only depends on the I/O data without requiring any model information of the original process. Theoretical analysis confirms the validity of the DDLVAC. The simulation study demonstrates the advantages of the DDLVAC such as less storage space, smaller computation burden, less control cost, as well as more robustness against uncertainties.

Estimating Latent-Variable Panel Data Models Using Parameter-Expanded SEM Methods

This article presents new estimation algorithms for three types of dynamic panel data models with latent variables: factor models, discrete choice models, and persistent-transitory quantile processes. The new methods combine the parameter expansion (PX) ideas of Liu, Rubin, and Wu with the stochastic expectation-maximization (SEM) algorithm in likelihood and moment-based contexts. The goal is to facilitate convergence in models with a large space of latent variables by improving algorithmic efficiency. This is achieved by specifying expanded models within the M step. Effectively, we are proposing new estimators for the pseudo-data within iterations that take into account the fact that the model of interest is misspecified for draws based on parameter values far from the truth. We establish the asymptotic equivalence of the likelihood-based PX-SEM to an alternative SEM algorithm with a smaller expected fraction of missing information compared to the standard SEM based on the original model, implying a faster global convergence rate. Finally, in simulations we show that the new algorithms significantly improve the convergence speed relative to standard SEM algorithms, sometimes dramatically so, by reducing the total computing time from hours to a few minutes.

Bearing degradation prediction based on deep latent variable state space model with differential transformation

Rolling bearings are a critical component of mechanical transmission equipment. Predicting their degradation trend is crucial for ensuring safe and stable equipment operation. Most existing bearing degradation prediction methods based on state space models (SSMs) use either linear functions or limited nonlinear functions (e.g., exponential/power laws) to construct the state and measurement equations. As such, these models fail to adapt to the complex and varied nonlinear degradation processes that occur in real-world environments. To address this limitation, we developed a deep latent variable-driven state space degradation model and employed it for bearing degradation prediction. Owing to the powerful nonlinear modeling ability of deep learning models, the proposed method extends the applicability of state space equations. In addition, it inherits the advantages of SSMs and can model uncertainties in a structured manner. Furthermore, the model was integrated with differential pre-transformation to improve its long-term prediction performance. Finally, to validate the effectiveness of the proposed model in predicting bearing degradation, experiments were conducted using a bearing dataset from the PRONOSTIA platform and real wind turbine bearing data. Results showed that the proposed method yielded higher bearing degradation prediction accuracy than existing methods, thus demonstrating the superior performance of the proposed model in predicting bearing degradation.

Real-time prediction of fuel consumption and emissions based on deep autoencoding support vector regression for cylinder pressure-based feedback control of marine diesel engines

Predictive models serving as virtual sensors for online optimization and feedback control of diesel engines is gaining increasing attention. However, existing prediction models fall short in simultaneously achieving high prediction accuracy and fast computational speed. In this paper, a novel machine learning algorithm called deep autoencoding support vector regression (DASVR) was proposed, which combines the powerful non-linear feature extraction capability of artificial neural network (ANN) with the good adaptability of support vector regression (SVR) to low-dimensional input spaces. ANN-based autoencoder is firstly employed to extract features from the original high-dimensional input space, forming a low-dimensional latent variable space. SVR is then employed to perform non-linear mapping between latent variables and target output variables. Experiments were conducted on a 16-cylinder marine diesel engine under different load, rail pressure, and injection timing conditions. Prediction models for fuel consumption, NOx, PM, HC, and PM emissions were established based on experimental data and DASVR algorithm. The maximum error of DASVR-based models for the five desired output variables is less than 3.8 % under different operating conditions. DAVSR-based predictive models outperform conventional ANN-based and SVR-based models in terms of both prediction accuracy and real-time capability, and can theoretically meet the real-time requirement below 3000 r/min.

A novel strategy for the quality control of carbonized Typhae pollen using colorimeter, liquid chromatography-mass spectrometry, and efficacy evaluation coupled with multivariate statistical analysis.

In this study, a novel quality control strategy was proposed, aiming to establish a multivariate specification for the processing step by exploring the correlation between colors, chemical components, and hemostatic effects of the carbonized Typhae pollen (CTP) using multivariate statistical analysis. The CTP samples were stir-fried at different durations. Afterward, the colorimeter and LC-MS techniques were applied to characterize the CTP samples, followed by the determination of bleeding time and clotting time using mice to evaluate their hemostatic effect. Then, principal component analysis, hierarchical cluster analysis, and multi-block partial least squares were used for data analysis on colors, chemical components, and their correlation with the hemostatic effect. Consequently, 13 critical quality attributes (CQAs) of CTP were identified via multivariate statistical analysis-L*, a*, b*, 3,4-dihydroxybenzoic acid, 4-hydroxybenzoic acid, 3-hydroxybenzoic acid, quercetin-3-O-glucoside, azelaic acid, kaempferol-3-O-glucoside, quercetin, naringenin, kaempferol, and isorhamnetin. The multivariate specification method involving the 13 CQAs was developed and visualized in the latent variable space of the partial least squares model, indicating that the proposed method was successfully applied to assess the quality of CTP and the degree of carbonization. Most importantly, this study offers a novel insight into the control of processing for carbonized Chinese herbal medicines.

Multi-mode industrial soft sensor method based on mixture Laplace variational auto-encoder

The industrially collected process data usually exhibit non-Gaussian and multi-mode characteristics. Due to sensor failures, irregular disturbances, and transmission problems, there are unavoidable outliers that make the data exhibit heavy-tailed characteristics. To this end, a variational auto-encoder regression method based on the mixture Laplacian distribution (MLVAER) is proposed, by introducing a type-II multivariate Laplacian distribution in the latent variable space for robust modeling, and further extending it to the mixture form to accommodate multi-mode processes, the corresponding reparameterization trick is finally proposed for the mixture form of this distribution for neural network gradient descent training. The model based on this distribution assumption has higher degrees of freedom than the model based on the traditional multivariate Laplace distribution assumption when the network structure is the same. Numerical simulation and experiments on two industrial examples demonstrate that the proposed algorithm reduces the root mean square error by over 15% compared to other algorithms.

Image‐based characterization of flocculation processes through PLS inspired representation learning in convolutional neural networks

AbstractMonitoring of flocculation processes such as those used in downstream processing of a fermentation broth is essential for process control. One approach is to apply microscopic imaging combined with image analysis for characterizing the state of the process. In this work, we investigate and compare the use of supervised feedforward convolutional neural network (CNN) architectures to predict the process states from the image information and compare the results with the traditional alternative of characterizing flocs based on manually engineered image features guided by human expertise. From a well‐defined image data set representing six process states, the objective is to establish end‐to‐end classification models which are accurate but at the same time learn meaningful latent variable space representations. Specifically, we evaluate three different CNN architectures with varying degrees of regularization and compare results with logistic regression models based on inputs from two different traditional feature engineering methods. By applying global average pooling as a structural regularizer to the CNN architecture, we significantly improve the generalization performance in comparison with the classification accuracies of the traditional feature engineered models. Furthermore, we show that by imposing a projection to latent structures (PLS) like regularization framework onto the CNN, it can also learn a latent variable representation that mimics the features selected by human expertise.

Investigation of the structure of the latent variable space in Sentence-BERT sentence vectors using an image generation model

Investigation of the structure of the latent variable space in Sentence-BERT sentence vectors using an image generation model

Domain-informed variational neural networks and support vector machines based leakage detection framework to augment self-healing in water distribution networks

The reduction of water leakage is essential for ensuring sustainable and resilient water supply systems. Despite recent investments in sensing technologies, pipe leakage remains a significant challenge for the water sector, particularly in developed nations like the UK, which suffer from aging water infrastructure. Conventional models and analytical methods for detecting pipe leakage often face reliability issues and are generally limited to detecting leaks during nighttime hours. Moreover, leakages are frequently detected by the customers rather than the water companies. To achieve substantial reductions in leakage and enhance public confidence in water supply and management, adopting an intelligent detection method is crucial. Such a method should effectively leverage existing sensor data for reliable leakage identification across the network. This not only helps in minimizing water loss and the associated energy costs of water treatment but also aids in steering the water sector towards a more sustainable and resilient future. As a step towards ‘self-healing’ water infrastructure systems, this study presents a novel framework for rapidly identifying potential leakages at the district meter area (DMA) level. The framework involves training a domain-informed variational autoencoder (VAE) for real-time dimensionality reduction of water flow time series data and developing a two-dimensional surrogate latent variable (LV) mapping which sufficiently and efficiently captures the distinct characteristics of leakage and regular (non-leakage) flow. The domain-informed training employs a novel loss function that ensures a distinct but regulated LV space for the two classes of flow groupings (i.e., leakage and non-leakage). Subsquently, a binary SVM classifier is used to provide a hyperplane for separating the two classes of LVs corresponding to the flow groupings. Hence, the proposed framework can be efficiently utilised to classify the incoming flow as leakage or non-leakage based on the encoded surrogates LVs of the flow time series using the trained VAE encoder. The framework is trained and tested on a dataset of over 2000 DMAs in North Yorkshire, UK, containing water flow time series recorded at 15-minute intervals over one year. The framework performs exceptionally well for both regular and leakage water flow groupings with a classification accuracy of over 98 % on the unobserved test dataset

An online transfer kernel recursive algorithm for soft sensor modeling with variable working conditions

Soft sensor technology has found widespread application in the real-time detection of challenging variables like product quality and key process parameters. However, changes in working conditions can lead to alterations in data distribution, resulting in model inaccuracies. Furthermore, data from new working conditions arrive sequentially and online. Due to technical difficulties, time-consuming measurements, and high costs, there is often a lack of sufficient new data for modeling in the initial phase. Moreover, industrial processes often exhibit dynamic, nonlinear, variable coupling and other factors, posing significant challenges for model establishment. To address these challenges, this study proposed an online transfer kernel recursive algorithm for soft sensor modeling under changing working conditions. The algorithm consists of an offline phase and an online phase. In the offline phase, the algorithm considers the dynamic features of the process and variable coupling, resulting in the establishment of a dynamic inner model. The original data is then projected onto a latent variable space. In the online phase, the proposed method incorporates the concept of online transfer learning and takes into account the nonlinearity of the process. This allows for the establishment of a kernel online recursive method in the latent variable space, integrating parameter transfer and sample transfer. Experimental results conducted on multiple industrial datasets validate the effectiveness of the proposed approach.

Semi-Supervised Deep Learning-Based Multi-component Spectral Calibration Modeling for UV-vis and Near-Infrared Spectroscopy without Information Loss.

Spectral analysis is an important method for characterizing and identifying chemical species. However, quantitative spectral analysis of multiple chemical properties in the real world has always been a challenging problem due to the strong correlation, massive noise, and serious information overlapping of the spectral features. Here, we present a new semi-supervised spectral calibration method based on information lossless decoupling of spectral features named NICEM. To realize the separation and extraction of key latent features, the method uses the flow-based model non-linear independent component estimation (NICE) to learn the sample distribution. The spectral data information is transformed into independent latent variables obeying Gaussian distribution by the reversible structure of deep network without information loss, so as to find the essential properties and realize the feature nonlinear decomposition. Moreover, the association between the input latent feature variables and attributes is evaluated by the maximum mutual information coefficient to eliminate the adverse effects of irrelevant information in the latent variable space and mine key information. Since the latent variables are independent in each dimension, the NICEM method is easier to establish an accurate semi-supervised multi-component calibration model even for high overlapping and complex spectral data. The applicability of the proposed spectral modeling method is demonstrated by using three ultraviolet-visible and near-infrared spectral data sets with 15 physical and chemical properties including diesel fuels, corn, and multi-metal ions solution. Results show that the proposed NICEM method has the highest determination coefficient (R2) and significantly improves extrapolation compared with the seven state-of-the-art methods. The proposed method is intuitive because it obviates complex feature engineering and prior knowledge and is a promising spectral calibration tool for quantitative analysis in other spectroscopy applications.

Diversified Kernel Latent Variable Space and Multi-Objective Optimization for Selective Ensemble Learning-Based Soft Sensor

The improvement of data-driven soft sensor modeling methods and techniques for the industrial process has strongly promoted the development of the intelligent process industry. Among them, ensemble learning is an excellent modeling framework. Accuracy and diversity are two key factors that run through the entire stage of building an ensemble learning-based soft sensor. Existing base model generating methods or ensemble pruning methods always consider the two factors separately, which has limited the development of high-performance but low-complexity soft sensors. To work out this issue, a selective ensemble learning-based soft sensor modeling method based on multi-kernel latent variable space and evolutionary multi-objective optimization is proposed, referred to as MOSE-MLV-VSPLS. This method designs a multiple diversity enhancement mechanism in the base model generation stage. Diversified input variable subspaces are first constructed using the maximum information coefficient on the bootstrapping random resampling subset. Then a set of base models that combine accuracy and diversity are generated on supervised latent variable subspaces under multiple kernel function perturbations. Further, two quantifiable parameters are designed for accuracy and diversity, and the multi-objective gray wolf optimization algorithm is used to select the base models that maximize these two important parameters to achieve effective ensemble pruning at the model ensemble stage. The MOSE-MLV-VSPLS method is applied to two typical industry processes, and the experimental results show that the method is effective and superior in selective ensemble-based soft sensor modeling.

Improved dynamic kernelPCAbased on local preserving projections and its application for electric submersible pump fault diagnosis

AbstractDynamic kernel principal component analysis (DKPCA) has been frequently implemented for nonlinear and dynamic process monitoring of complex industrial processes. However, traditional DKPCA focuses only on the global structural analysis of data sets and strongly neglects the local information, which is equally essential for process detection and identification. In this paper, an improved DKPCA, referred to as the local DKPCA (LDKPCA), is proposed based on local preserving projections (LPP) for nonlinear dynamic process fault diagnosis. The method combines the advantages of LPP and DKPCA by utilizing the local structure feature to maintain the geometric structure of the data in a unified framework. To achieve a highly comprehensive feature extraction, the local characteristics are fused in DKPCA to produce an optimization objective. The neighbouring points of the new objective function projection in the feature space are still maintained in proximity, and the variance information is retained simultaneously. For the purpose of fault detection, two statistics, known as theT2and squared prediction error (SPE) statistics, are constructed, based on the LDKPCA model, and used to monitor the latent variable space and the residual space, respectively. In addition, the sensitivity analysis is brought in for fault identification of the two statistics. Based on the experimental analysis using the shaft breakage data of an offshore oilfield electric submersible pump (ESP), the proposed method outperforms the conventional DKPCA in terms of fault monitoring performance. The experimental results demonstrate the potential of the method in nonlinear dynamic process fault diagnosis.

Operating Performance Assessment Based on Semi-Supervised Cluster Generative Adversarial Networks for Gold Flotation Process

Operating performance assessment of gold flotation process plays an important role in improving the metallurgical performances and pursues the best comprehensive economic benefits. The appearance of froth is a good indicator of flotation performance; however, labeling the data is costly. To assess the operational performance of flotation process with a small amount of labeled data, this study proposes a semi-supervised cluster generative adversarial network (SSClusterGAN). First, latent variables are sampled from one-hot encoded variables and continuous normal distribution variables. While projecting all data into the latent variable space, the specific types of clustering in the latent space is realized by controlling the type of distribution of the learned latent codes, so that the training objectives of labeled data and unlabeled data are consistent. Then, by adapting the generative adversarial networks (GANs) framework, a pair of stacked discriminators are used to learn the conditional distribution of attributes for labeled and unlabeled data, respectively. In addition, using semi-supervised learning (SSL) as the basic learning technology, the high-confidence pseudo-labels obtained by training with unlabeled data are combined with the labeled training data to increase the training samples and effectively improve the accuracy of the model. Finally, the method was applied to the gold flotation process. The experimental results show that the method effectively utilizes unlabeled samples to expand the labeled dataset, combines the advantages of SSL and GAN, obtains accurate assessment results, and effectively improves the comprehensive economic benefits of the gold flotation process.

Mode Recognition of Multifunction Radars for Few-Shot Learning Based on Compound Alignments

The multifunctional radars can switch among a variety of fine-grained working modes, which often have flexible modulation types and programmable parameters. In an electromagnetic reconnaissance system, the process of identifying different working modes in pulse sequences guarantees the subsequent intention analysis and assists in devising a jamming strategy. Most of the existing working mode recognition methods attempt to establish a machine learning mechanism by training a model using a large number of annotated samples. However, this is hardly applicable in the real-world scenarios where only a few samples can be intercepted in advance. As the labeled signal samples are expensive, one direction is to augment the dataset by generating either samples or signal features in an embedding space. In this article, inspired by the fact that different modalities of the same working modes are generated from a set of fixed parameters, a few-shot learning framework based on compound alignments is proposed. Three branches, which take observations of long windows, observations of short windows, and coded semantic attributes as inputs, are aligned in both the latent variable space and the reconstruction space to learn a shared embedded space. A simple soft-max classifier is trained by sampling sufficient instances in the learned space to realize the final identification process. The experimental results and analysis show that the proposed method achieves excellent performance for fine-grained working mode recognition even for a small number of observed pulses. In addition, the proposed method is robust under different nonideal conditions, such as noise contamination, incomplete sequences, and spurious pulses.

Iterative learning model predictive control for multivariable nonlinear batch processes based on dynamic fuzzy PLS model

This paper proposes a latent variable nonlinear iterative learning model predictive control method (LV-NILMPC) based on the dynamic fuzzy partial least squares (DFPLS) model to achieve trajectory tracking and process disturbance suppression in multivariable nonlinear batch processes. The dynamic and nonlinear characteristics of the physical system are constructed by integrating the T-S fuzzy model into the regression framework of the dynamic partial least squares (PLS) inner model. The decoupling and dimensionality reduction characteristics of the DFPLS model automatically decompose a multivariable nonlinear system into multiple univariate subsystems operating independently in the latent variable space. Based on the DFPLS model, we design LV-NILMPC controllers corresponding to each latent variable subspace to track the projection of the reference trajectories. Compared with the previous control method, the method proposed in this paper has a faster convergence rate and smaller tracking error. The method is suitable for nonlinear, multivariable and strong coupling batch processes. Finally, the application of two cases shows that the method is effective.

Evaluating a face generator from a human perspective

StyleGAN2 is able to generate very realistic and high-quality faces of humans using a training set (FFHQ). Instead of using one of the many commonly used metrics to evaluate the performance of a face generator (e.g., FID, IS and P&R), this paper uses a more humanlike approach providing a different outlook on the performance of StyleGAN2. The generator within StyleGAN2 tries to learn the distribution of the input dataset. However, this does not necessarily mean that higher-level human concepts are preserved. We examine if general human attributes, such as age and gender, are transferred to the output dataset and if StyleGAN2 is able to generate actual new persons according to facial recognition methods. It is crucial for practical implementations that a face generator not only generates new humans, but that these humans are not clones of the original identities. This article addresses these questions. Although our approach can be used for other face generators, we only focused on StyleGAN2. First, multiple models are used to predict general human attributes. This shows that the generated images have the same attribute distributions as the input dataset. However, if truncation is applied to limit the latent variable space, the attribute distributions change towards the attributes corresponding with the latent variable used in truncation. Second, by clustering using face recognition models, we demonstrate that the generated images do not belong to an existing person from the input dataset. Thus, StyleGAN2 is able to generate new persons with similar human characteristics as the input dataset.

Variational inference with NoFAS: Normalizing flow with adaptive surrogate for computationally expensive models

Fast inference of numerical model parameters from data is an important prerequisite to generate predictive models for a wide range of applications. Use of sampling-based approaches such as Markov chain Monte Carlo may become intractable when each likelihood evaluation is computationally expensive. New approaches combining variational inference with normalizing flow are characterized by a computational cost that grows only linearly with the dimensionality of the latent variable space, and rely on gradient-based optimization instead of sampling, providing a more efficient approach for Bayesian inference about the model parameters. Moreover, the cost of frequently evaluating an expensive likelihood can be mitigated by replacing the true model with an offline trained surrogate model, such as neural networks. However, this approach might generate significant bias when the surrogate is insufficiently accurate around the posterior modes. To reduce the computational cost without sacrificing inferential accuracy, we propose Normalizing Flow with Adaptive Surrogate (NoFAS), an optimization strategy that alternatively updates the normalizing flow parameters and surrogate model parameters. We also propose an efficient sample weighting scheme for surrogate model training that preserves global accuracy while effectively capturing high posterior density regions. We demonstrate the inferential and computational superiority of NoFAS against various benchmarks, including cases where the underlying model lacks identifiability. The source code and numerical experiments used for this study are available at https://github.com/cedricwangyu/NoFAS.

Region- and Strength-Controllable GAN for Defect Generation and Segmentation in Industrial Images

Deep learning for computer vision has achieved remarkable results based on massive, diverse, and well-annotated training sets. However, it is difficult to collect defect datasets that cover all possible features, especially for small, weak defects. Therefore, in this article, a defect image generation method with controllable defect regions and strength is proposed. Regarded as image inpainting that uses a generative adversarial network, generated defect regions are controlled by using defect masks. Moreover, the defect direction vector is constructed in the latent variable space based on the feature continuity between defects and nondefects to control the defect strength, which enables a one-to-many correspondence between defect masks and images. Moreover, a defect attention loss is also designed to force the generation model to focus on the defect regions. Experimentally, our method yields generated images of better quality and diversity and thus significantly improves defect segmentation performance (an intersection over union of 63.20% and 61.86% on the Kolektor surface-defect and the metal hook defect datasets, respectively), especially for small, weak defects.

Ecological niche differentiation on the example of bottom communities of the Middle and Lower Volga regions

The authors discuss the use of the Greenell concept of ecological niches to analyze the taxonomic structure of communities and their relationship with environmental factors. The modeling of ecological niches was carried out using the results of 30-year studies of macrozoobenthos’ communities on 90 small and 12 medium-sized plain rivers in the region of the Kuibyshev, Saratov and Volgograd reservoirs. Geoclimatic indicators, relief characteristics and hydrochemical assessments of water quality at sampling points were considered as abiotic factors, where 11 variables with the least collinearity were taken. The multidimensional space of the initial abiotic factors was projected onto the orthogonal axes of the first two principal components; there were constructed ordination diagrams with plotted points of presence of species. The model of the potential ecological niche of each species was represented as an area in the space of latent variables, in which the habitat suitability index Z, estimated by the probability density of occurrence, corresponds to the given constraints. The authors used the Schoener and Hellinger indices to quantify the proportion of overlapping niches belonging to two different benthos taxa. The matrix of distances between overlapping niches was formed for all possible paired combinations of 40 most ecologically significant macrozoobenthos species. The cluster analysis of the obtained distance matrix was carried out by the methods of hierarchical agglomeration and fuzzy k-means. In the general multidimensional space of abiotic variables of the studied region, 4 areas of collective niches for groups of species with maximum distances between their centroids were identified. The analysis of intergroup variation of environmental factors is given and the characteristic features of each group are discussed: ecological strategy, biological characteristics and tolerance level.