Probabilistic Latent Variable Model Research Articles

Comparative evaluation of VAE-based monitoring statistics for real-time anomaly detection in AIS data

ABSTRACT Real-time tracking and anomaly detection in vessel navigation using AIS (Automatic Identification System) data is a critical issue in maritime logistics. However, the substantial volume, inherent noise, and irregular temporal intervals of AIS data pose challenges in applying traditional Statistical Process Monitoring (SPM) methods. To overcome these challenges, recent research has proposed the use of deep probabilistic latent variable models, specifically variational autoencoders (VAE). However, the monitoring statistics based on VAE employed in previous studies may vary. In this study, we conduct a comprehensive comparison and evaluation of various monitoring statistics based on VAE within the context of statistical process monitoring. Furthermore, we propose a new real-time monitoring method by integrating VAE-based monitoring statistics with the CUSUM chart for monitoring AIS data. By utilizing both simulated and real-world AIS data, our proposed method demonstrates superior detection performance and robustness compared to traditional methods.

Deep Probabilistic Principal Component Analysis for Process Monitoring.

Probabilistic latent variable models (PLVMs), such as probabilistic principal component analysis (PPCA), are widely employed in process monitoring and fault detection of industrial processes. This article proposes a novel deep PPCA (DePPCA) model, which has the advantages of both probabilistic modeling and deep learning. The construction of DePPCA includes a greedy layer-wise pretraining phase and a unified end-to-end fine-tuning phase. The former establishes a hierarchical deep structure based on cascading multiple layers of the PPCA module to extract high-level features. The latter builds an end-to-end connection between the raw inputs and the final outputs to further improve the representation of the model to high-level features. After constructing the model structure of DePPCA, we first present the detailed training processes of the pretraining and fine-tuning stages, then clarify the theoretical merits of the proposed model from the perspective of variational inference. For process monitoring purposes, we develop two statistics based on the established DePPCA. The monitoring performance of these two statistics can remain superior even if the features extracted by DePPCA are significantly compressed to univariate. This makes the feature extraction process and online monitoring procedure of DePPCA quite fast. In other words, the proposed DePPCA can achieve accurate and efficient process monitoring by only extracting one feature for each sample. Finally, the effectiveness of DePPCA is evaluated on the Tennessee Eastman (TE) process and the multiphase flow (MPF) facility.

M 2 D-VAE: Self-Supervised Probabilistic Temporal- Spatial Latent Representation Learning for Unsupervised Industrial Operational Applications Under Missing Value Interference.

Due to sensor malfunctions and data transmission corruptions, the industrial process data collected commonly contain missing values. It poses a significant challenge for data-driven approaches in aggregating temporal-spatial correlations that reflect dependencies across both variables and times, which makes it difficult to directly carry out downstream industrial operational applications. In this study, a self-supervised representation learning model is proposed to extract probabilistic temporal-spatial latent variables (LVs) from sequential data under missing value interference. The extracted LVs can be utilized for typical industrial operational applications through a unified framework. First, a novel deep dynamic probabilistic latent variable model, named Markov dynamic variational autoencoder (MD-VAE), is proposed to explicitly model the temporal-spatial dependencies between LVs. The latent posteriors are Bayesian smoothed by global sequence information for effective variational inference (VI). Second, a self-supervised learning approach, termed masked MD-VAE (), is proposed to address the challenge of directly extracting temporal-spatial LVs under missing value interference. Controllable constraints with practical interpretations are introduced to balance the latent bottleneck capacity with reconstruction accuracy during model optimization. A unified framework is proposed to utilize the latent representations for typical industrial downstream tasks. Case studies conducted on a real-world multiphase flow process demonstrate the superiority of in unsupervised industrial operational applications including missing value imputation and dynamic process monitoring under missing value interference.

Generative models for anomaly detection and design-space dimensionality reduction in shape optimization

Our work presents a novel approach to shape optimization, with the twofold objective to improve the efficiency of global optimization algorithms while promoting the generation of high-quality designs during the optimization process free of geometrical anomalies. This is accomplished by reducing the number of the original design variables defining a new reduced subspace where the geometrical variance is maximized and modeling the underlying generative process of the data via probabilistic linear latent variable models such as factor analysis and probabilistic principal component analysis. We show that the data follows approximately a Gaussian distribution when the shape modification method is linear and the design variables are sampled uniformly at random, due to the direct application of the central limit theorem. The degree of anomalousness is measured in terms of Mahalanobis distance, and the paper demonstrates that abnormal designs tend to exhibit a high value of this metric. This enables the definition of a new optimization model where anomalous geometries are penalized and consequently avoided during the optimization loop. The procedure is demonstrated for hull shape optimization of the DTMB 5415 model, extensively used as an international benchmark for shape optimization problems. The global optimization routine is carried out using Bayesian optimization and the DIRECT algorithm. From the numerical results, the new framework improves the convergence of global optimization algorithms, while only designs with high-quality geometrical features are generated through the optimization routine thereby avoiding the wastage of precious computationally expensive simulations.

A data-driven soft-sensing approach using probabilistic latent variable model with functional data framework

Functional principal component analysis (FPCA) and functional partial least squares (FPLS) are two mainstream functional data analysis (FDA) methods, which have been commonly used to extract deep information hidden in the original data space. However, the process data always contain random noise, which affects the performance of FDA models. To overcome this issue, two functional probabilistic latent variable models (FPLVMs), including functional probabilistic principal component analysis (FPPCA) and functional probabilistic partial least squares (FPPLS) are proposed in this work. First, the process data are converted into functional data using the FDA. Subsequently, a log-likelihood function considering the noise factor and functional latent variables is designed. Finally, the regression model parameters are estimated using an expectation–maximisation algorithm. In contrast to FPPCA, FPPLS decomposes the process data and the key variable with constrained latent variables, which is similar to the partial least squares (PLS) and the principal component analysis (PCA). Moreover, the degeneration mechanism from FPLVMs into probabilistic latent variable models and latent variable models is discussed. An adaptive strategy with functional covariance is used to satisfy the online predictive capabilities of the model. Finally, the proposed approach is validated using a numerical case, the Tennessee Eastman process and an industrial o-xylene distillation column for evaluation.

Learning to Learn With Variational Inference for Cross-Domain Image Classification

Learning models that can generalize to previously unseen domains to which we have no access is a fundamental yet challenging problem in machine learning. In this paper, we propose meta variational inference (MetaVI), a variational Bayesian framework of meta-learning for cross domain image classification. Within the meta learning setting, MetaVI is derived to learn a probabilistic latent variable model by maximizing a meta evidence lower bound (Meta ELBO) for knowledge transfer across domains. To enhance the discriminative ability of the model, we further introduce a Wasserstein distance based constraint to the variational objective, leading to the Wasserstein MetaVI, which largely improves classification performance. By casting into a probabilistic inference problem, MetaVI offers the first, principled variational meta-learning framework for cross domain learning. In addition, we collect a new visual recognition dataset to contribute a more challenging benchmark for cross domain learning, which will be released to the public. Extensive experimental evaluation and ablation studies on four benchmarks show that our Wasserstein MetaVI achieves new state-of-the-art performance and surpasses previous methods, demonstrating its great effectiveness.

Uncertainty-aware report generation for chest X-rays by variational topic inference.

Automating report generation for medical imaging promises to minimize labor and aid diagnosis in clinical practice. Deep learning algorithms have recently been shown to be capable of captioning natural photos. However, doing a similar thing for medical data, is difficult due to the variety in reports written by different radiologists with fluctuating levels of knowledge and experience. Current methods for automatic report generation tend to merely copy one of the training samples in the created report. To tackle this issue, we propose variational topic inference, a probabilistic approach for automatic chest X-ray report generation. Specifically, we introduce a probabilistic latent variable model where a latent variable defines a single topic. The topics are inferred in a conditional variational inference framework by aligning vision and language modalities in a latent space, with each topic governing the generation of one sentence in the report. We further adopt a visual attention module that enables the model to attend to different locations in the image while generating the descriptions. We conduct extensive experiments on two benchmarks, namely Indiana U. Chest X-rays and MIMIC-CXR. The results demonstrate that our proposed variational topic inference method can generate reports with novel sentence structure, rather than mere copies of reports used in training, while still achieving comparable performance to state-of-the-art methods in terms of standard language generation criteria.

Virtual Sensor Modeling for Nonlinear Dynamic Processes Based on Local Weighted PSFA

Modern industrial processes are featured with complex dynamic, nonlinear, and noisy characteristics. It is of great significance to apply the probabilistic latent variable models (LVMs) to mine the pivotal features of the industrial processes. A probabilistic slow feature analysis (PSFA) can extract slowly varying features in rapidly changing data sequences as a dynamic LVM. However, the performance of PSFA is limited because of its linear assumption. In this article, a locally weighted PSFA (LWPSFA) is proposed for nonlinear dynamic modeling of industrial data with random noises. Two different kinds of weighting techniques are designed to approximate the nonlinear slow feature transition and emission functions. After that, the expectation maximum (EM) algorithm is adopted to estimate the parameters of LWPSFA with a weighted log-likelihood function (W-LLF). Eventually, a debutanizer column and a hydrocracking process are used to validate the effectiveness of LWPSFA.

Multi-omic single-cell velocity models epigenome-transcriptome interactions and improves cell fate prediction.

Multi-omic single-cell datasets, in which multiple molecular modalities are profiled within the same cell, offer an opportunity to understand the temporal relationship between epigenome and transcriptome. To realize this potential, we developed MultiVelo, a differential equation model of gene expression that extends the RNA velocity framework to incorporate epigenomic data. MultiVelo uses a probabilistic latent variable model to estimate the switch time and rate parameters of chromatin accessibility and gene expression and improves the accuracy of cell fate prediction compared to velocity estimates from RNA only. Application to multi-omic single-cell datasets from brain, skin and blood cells reveals two distinct classes of genes distinguished by whether chromatin closes before or after transcription ceases. We also find four types of cell states: two states in which epigenome and transcriptome are coupled and two distinct decoupled states. Finally, we identify time lags between transcription factor expression and binding site accessibility and between disease-associated SNP accessibility and expression of the linked genes. MultiVelo is available on PyPI, Bioconda and GitHub ( https://github.com/welch-lab/MultiVelo ).

Learning and Inference in Sparse Coding Models With Langevin Dynamics.

We describe a stochastic, dynamical system capable of inference and learning in a probabilistic latent variable model. The most challenging problem in such models-sampling the posterior distribution over latent variables-is proposed to be solved by harnessing natural sources of stochasticity inherent in electronic and neural systems. We demonstrate this idea for a sparse coding model by deriving a continuous-time equation for inferring its latent variables via Langevin dynamics. The model parameters are learned by simultaneously evolving according to another continuous-time equation, thus bypassing the need for digital accumulators or a global clock. Moreover, we show that Langevin dynamics lead to an efficient procedure for sampling from the posterior distribution in the L0 sparse regime, where latent variables are encouraged to be set to zero as opposed to having a small L1 norm. This allows the model to properly incorporate the notion of sparsity rather than having to resort to a relaxed version of sparsity to make optimization tractable. Simulations of the proposed dynamical system on both synthetic and natural image data sets demonstrate that the model is capable of probabilistically correct inference, enabling learning of the dictionary as well as parameters of the prior.

Probabilistic dynamic-controlled latent variable model for pattern-space modelling and pattern-based stochastic model predictive control

Industrial processes are measured and controlled using high-dimensional process variables, but its overall operation is usually characterised by low-dimensional patterns. The changes in the pattern are dominated by three features: free motion, controlled motion, and uncertainty. In this paper, all three features are taken into consideration to propose a new probabilistic dynamic-controlled latent variable (PDCLV) model structure using a dynamic Bayesian network for process modelling in the pattern space. To this end, the linear dynamic system characterised by control inputs is introduced, and the expectation maximisation algorithm is specially designed for learning the PDCLV model. Benefitting from the dynamic causality between control inputs and the explicit modelling of the pattern, a method for pattern-based stochastic model predictive control (SMPC) is implemented successfully to realise process optimisation. A case study on an industrial boiler combustion process demonstrates the benefits of the proposed PDCLV structure for pattern-space modelling and pattern-based SMPC.

Variational Abnormal Behavior Detection With Motion Consistency.

Abnormal crowd behavior detection has recently attracted increasing attention due to its wide applications in computer vision research areas. However, it is still an extremely challenging task due to the great variability of abnormal behavior coupled with huge ambiguity and uncertainty of video contents. To tackle these challenges, we propose a new probabilistic framework named variational abnormal behavior detection (VABD), which can detect abnormal crowd behavior in video sequences. We make three major contributions: (1) We develop a new probabilistic latent variable model that combines the strengths of the U-Net and conditional variational auto-encoder, which also are the backbone of our model; (2) We propose a motion loss based on an optical flow network to impose the motion consistency of generated video frames and input video frames; (3) We embed a Wasserstein generative adversarial network at the end of the backbone network to enhance the framework performance. VABD can accurately discriminate abnormal video frames from video sequences. Experimental results on UCSD, CUHK Avenue, IITB-Corridor, and ShanghaiTech datasets show that VABD outperforms the state-of-the-art algorithms on abnormal crowd behavior detection. Without data augmentation, our VABD achieves 72.24% in terms of AUC on IITB-Corridor, which surpasses the state-of-the-art methods by nearly 5%.

Gaussian mixture deep dynamic latent variable model with application to soft sensing for multimode industrial processes

The data-driven deep probabilistic latent variable model (DPLVM) has attracted much attention for industrial process soft sensing in recent years. The DPLVM has well handled the nonlinear characteristics of the processes with powerful feature extracting capability. However, the multimode process property and the dynamic data features seldom be considered in those applications. To tackle the two issues, the article starts from the basic DPLVM, i.e., the Variational Autoencoder (VAE), to build a deep dynamic latent variable regression model (i.e., the Gated Recurrent Unit-based VAE regression, GVAER), where GRU cells are utilized to capture dynamic features from the process time sequence data. With the GVAER, a Gaussian Mixture GVAER (GM-GVAER) model is proposed. The Gaussian Mixture priors are used in the latent space to characterize the multimode process data features. In particular, a semi-supervised learning scheme is also proposed for the model to deal with the unequal scale of input and output data. A numerical example and a real chemical process case are provided to verify the feasibility and effectiveness of the proposed soft sensor model.

Predicting Emotional States Using Behavioral Markers Derived From Passively Sensed Data: Data-Driven Machine Learning Approach.

BackgroundMental health disorders affect multiple aspects of patients’ lives, including mood, cognition, and behavior. eHealth and mobile health (mHealth) technologies enable rich sets of information to be collected noninvasively, representing a promising opportunity to construct behavioral markers of mental health. Combining such data with self-reported information about psychological symptoms may provide a more comprehensive and contextualized view of a patient’s mental state than questionnaire data alone. However, mobile sensed data are usually noisy and incomplete, with significant amounts of missing observations. Therefore, recognizing the clinical potential of mHealth tools depends critically on developing methods to cope with such data issues.ObjectiveThis study aims to present a machine learning–based approach for emotional state prediction that uses passively collected data from mobile phones and wearable devices and self-reported emotions. The proposed methods must cope with high-dimensional and heterogeneous time-series data with a large percentage of missing observations.MethodsPassively sensed behavior and self-reported emotional state data from a cohort of 943 individuals (outpatients recruited from community clinics) were available for analysis. All patients had at least 30 days’ worth of naturally occurring behavior observations, including information about physical activity, geolocation, sleep, and smartphone app use. These regularly sampled but frequently missing and heterogeneous time series were analyzed with the following probabilistic latent variable models for data averaging and feature extraction: mixture model (MM) and hidden Markov model (HMM). The extracted features were then combined with a classifier to predict emotional state. A variety of classical machine learning methods and recurrent neural networks were compared. Finally, a personalized Bayesian model was proposed to improve performance by considering the individual differences in the data and applying a different classifier bias term for each patient.ResultsProbabilistic generative models proved to be good preprocessing and feature extractor tools for data with large percentages of missing observations. Models that took into account the posterior probabilities of the MM and HMM latent states outperformed those that did not by more than 20%, suggesting that the underlying behavioral patterns identified were meaningful for individuals’ overall emotional state. The best performing generalized models achieved a 0.81 area under the curve of the receiver operating characteristic and 0.71 area under the precision-recall curve when predicting self-reported emotional valence from behavior in held-out test data. Moreover, the proposed personalized models demonstrated that accounting for individual differences through a simple hierarchical model can substantially improve emotional state prediction performance without relying on previous days’ data.ConclusionsThese findings demonstrate the feasibility of designing machine learning models for predicting emotional states from mobile sensing data capable of dealing with heterogeneous data with large numbers of missing observations. Such models may represent valuable tools for clinicians to monitor patients’ mood states.

Variational Bayesian probabilistic modeling framework for data-driven distributed process monitoring

Data-driven process monitoring has gained increasing attention because of the increasing demand in process safety and the rapid advancement of data gathering techniques. When monitoring a plant-wide multiunit process, establishing a monitor for each unit individually ignores the correlations among units, whereas establishing a global monitor for the entire process ignores the local process behavior. A variational Bayesian-based probabilistic modeling approach is proposed for efficient distributed process monitoring. A novel probabilistic latent variable model is developed to characterize the variable relationship in each local unit and among units. First, variational Bayesian-based latent variable extraction is performed in each local unit, through which variable relationship within a local unit is characterized. Second, variational Bayesian-based regression model is established between the latent variables and neighboring variables, through which the variable relationship among units is characterized. Then, modeling residuals and monitoring statistics are generated, through which the process status and the type of a detected fault are identified. The effectiveness of the proposed probabilistic modeling and monitoring method is verified by three case studies, including a numerical example, the Tennessee Eastman benchmark process, and a laboratory distillation process.

Supervised Nonlinear Dynamic System for Soft Sensor Application Aided by Variational Auto-Encoder

Dynamic data modeling has been attracting much attention from researchers and has been introduced into the probabilistic latent variable model in the process industry. It is a huge challenge to extend these dynamic probabilistic latent variable models to nonlinear forms. In this article, a supervised nonlinear dynamic system (NDS) based on variational auto-encoder (VAE) is introduced for processes with dynamic behaviors and nonlinear characteristics. Based on the framework of VAE, which has a probabilistic data representation and a high fitting ability, the supervised NDS can extract effective nonlinear features for latent variable regression. The feasibility of the proposed supervised NDS is tested on two numerical examples and an industrial case. Detailed comparisons verify the effectiveness and superiority of the proposed model.

Industrial Big Data Modeling and Monitoring Framework for Plant-Wide Processes

This article proposes a distributed parallel modeling and monitoring framework for plant-wide processes with big data. The “distributed” contains two layers of meaning. One is the spatially distributed modeling and hierarchical monitoring for the plant-wide process with multiple operating units. The other represents the distributed parallel modeling for big process data with various features. Under the framework, the distributed parallel mixture probabilistic latent variable model is proposed based on the stochastic variational inference algorithm and the parameter server architecture to cope with the big process data. Then, the model is utilized to develop the plant-wide hierarchical and distributed process monitoring algorithms, where the multilevel monitoring indexes and fault contribution indexes are established based on the Bayesian fusion algorithm for process fault detection and diagnosis. The performance comparison and visualization for the industrial plant-wide process case has demonstrated the reliability and superiority of the proposed algorithm and framework.

Imitating by Generating: Deep Generative Models for Imitation of Interactive Tasks.

To coordinate actions with an interaction partner requires a constant exchange of sensorimotor signals. Humans acquire these skills in infancy and early childhood mostly by imitation learning and active engagement with a skilled partner. They require the ability to predict and adapt to one's partner during an interaction. In this work we want to explore these ideas in a human-robot interaction setting in which a robot is required to learn interactive tasks from a combination of observational and kinesthetic learning. To this end, we propose a deep learning framework consisting of a number of components for (1) human and robot motion embedding, (2) motion prediction of the human partner, and (3) generation of robot joint trajectories matching the human motion. As long-term motion prediction methods often suffer from the problem of regression to the mean, our technical contribution here is a novel probabilistic latent variable model which does not predict in joint space but in latent space. To test the proposed method, we collect human-human interaction data and human-robot interaction data of four interactive tasks “hand-shake,” “hand-wave,” “parachute fist-bump,” and “rocket fist-bump.” We demonstrate experimentally the importance of predictive and adaptive components as well as low-level abstractions to successfully learn to imitate human behavior in interactive social tasks.

Modern Soft-Sensing Modeling Methods for Fermentation Processes.

For effective monitoring and control of the fermentation process, an accurate real-time measurement of important variables is necessary. These variables are very hard to measure in real-time due to constraints such as the time-varying, nonlinearity, strong coupling, and complex mechanism of the fermentation process. Constructing soft sensors with outstanding performance and robustness has become a core issue in industrial procedures. In this paper, a comprehensive review of existing data pre-processing approaches, variable selection methods, data-driven (black-box) soft-sensing modeling methods and optimization techniques was carried out. The data-driven methods used for the soft-sensing modeling such as support vector machine, multiple least square support vector machine, neural network, deep learning, fuzzy logic, probabilistic latent variable models are reviewed in detail. The optimization techniques used for the estimation of model parameters such as particle swarm optimization algorithm, ant colony optimization, artificial bee colony, cuckoo search algorithm, and genetic algorithm, are also discussed. A comprehensive analysis of various soft-sensing models is presented in tabular form which highlights the important methods used in the field of fermentation. More than 70 research publications on soft-sensing modeling methods for the estimation of variables have been examined and listed for quick reference. This review paper may be regarded as a useful source as a reference point for researchers to explore the opportunities for further enhancement in the field of soft-sensing modeling.

Batch-to-Batch and Within-Batch Input Trajectory Adjustment Based on the Probabilistic Latent Variable Model

The input trajectory is critical to the output quality of batch processes; therefore, the input trajectory of an entire batch run should be designed according to the desired output before manufactu...