Variational Bayesian Principal Component Analysis Research Articles

Virtual Sensing of Key Variables in the Hydrogen Production Process: A Comparative Study of Data-Driven Models.

Hydrogen is an ideal energy carrier manufactured mainly by the natural gas steam reforming hydrogen production process. The concentrations of CH4, CO, CO2, and H2 in this process are key variables related to product quality, which thus need to be controlled accurately in real-time. However, conventional measurement methods for these concentrations suffer from significant delays or huge acquisition and upkeep costs. Virtual sensors effectively compensate for these shortcomings. Unfortunately, previously developed virtual sensors have not fully considered the complex characteristics of the hydrogen production process. Therefore, a virtual sensor model, called "moving window-based dynamic variational Bayesian principal component analysis (MW-DVBPCA)" is developed for key gas concentration estimation. The MW-DVBPCA considers complicated characteristics of the hydrogen production process, involving dynamics, time variations, and transportation delays. Specifically, the dynamics are modeled by the finite impulse response paradigm, the transportation delays are automatically determined using the differential evolution algorithm, and the time variations are captured by the moving window method. Moreover, a comparative study of data-driven virtual sensors is carried out, which is sporadically discussed in the literature. Meanwhile, the performance of the developed MW-DVBPCA is verified by the real-life natural gas steam reforming hydrogen production process.

No-Delay Multimodal Process Monitoring Using Kullback-Leibler Divergence-Based Statistics in Probabilistic Mixture Models

The primary goal of multimodal process monitoring is to detect abnormalities or occurrence of faults. However, the profound challenge in the multimodal monitoring problem is that it is difficult to quickly distinguish the fault occurrence on a process mode from other operating modes. In this work, a Gaussian mixture model based variational Bayesian principal component analysis (GMM-VBPCA) is proposed. GMM is used to capture the global multimodal information where each Gaussian component of GMM represents a corresponding normal operating mode. VBPCA is employed to construct a probabilistic model for each operating mode. Using the weights of posterior probabilities from global GMM, local VBPCA models can then be fused to characterize the normal multimodal processes. In order to detect the occurrence of faults, Kullback-Leibler (KL) divergence of latents and model residuals of the multimodal process are used as the monitoring statistics that measure the deviation from the normal multimodal distribution. Owing to the variational local model, the posterior distribution of latents and model residuals of the GMM-VBPCA can characterize the process behavior for every test sample. Finally, GMM-VBPCA based monitoring statistics are compared with existing process monitoring methods through a simulated numerical example and an industrial hydrocracking process. Note to Practitioners—In this paper, a novel process monitoring statistics has been proposed that can aid the practitioners in accurately identifying the process faults in near real-time with minimal false alarm. Also, the sensitivity to small bias faults is higher that the traditional methods, thus enabling higher fault detection rate. Based on the proposed statistics, an online monitoring scheme has been proposed. Hence, it is useful for practitioners in quickly taking preventive measures to avoid catastrophe, and also taking corrective measures to bring the plant to normal operating range or scheduling maintenance in case of early detection of sensor or equipment failures.

Identification of Carbonate Cave Reservoirs Based on Variational Bayesian Principal Component Analysis

Identification of Carbonate Cave Reservoirs Based on Variational Bayesian Principal Component Analysis

Principal Component Regression Modelling with Variational Bayesian Approach to Overcome Multicollinearity at Various Levels of Missing Data Proportion

This study aims to model Principal Component Regression (PCR) using Variational Bayesian Principal Component Analysis (VBPCA) with Ordinary Least Square (OLS) as a method of estimating regression parameters to overcome multicollinearity at various levels of the proportion of missing data. The data used in this study are secondary data and simulation data contaminated with collinearity in the predictor variables with various missing data proportions of 1%, 5%, and 10%. The secondary data used is the Human Depth Index in Java in 2021, complete data without missing values. The results indicate that the multicollinearity in secondary and original data can be optimally overcome as indicated by the smaller standard error value of the regression parameter for the PCR using VBPCA method which is smaller and has a relative efficiency value of less than 1. VBPCA can handle the proportion of missing data to less than 10%. The proportion of missing data causes information from the original variable to decrease, as evidenced by immense MAPE value and the parameter estimation bias that gets bigger. Then the cross validation (Q^2 ) value and the coefficient of determination (adjusted R^2 ) are get smaller as the proportion of missing data increases.

Characterizing Spatiotemporal Patterns of Terrestrial Water Storage Variations Using GNSS Vertical Data in Sichuan, China

AbstractWe invert daily GNSS vertical position time series for terrestrial water storage (TWS) changes and characterize their spatiotemporal patterns using various geodetic and hydrological data in Sichuan which features complex climatic and geographical conditions. Our inversion strategy depends on a variational Bayesian principal component analysis (VBPCA), which recovers the equivalent water height (EWH) changes using three principal components that explain 82% of the data variance. Various data sets from GNSS, GRACE, and GLDAS indicate an increasing seasonal water variability from northeast to southwest Sichuan. The GNSS modeling results reveal significant annual EWH amplitudes (up to 300 mm), which are largely underestimated in both GRACE‐EWH (∼120 mm) and GLDAS‐EWH (∼100 mm) products. The GNSS‐EWH also shows considerable seasonal changes in the high‐altitude plateau but subtle seasonal oscillations in the lowland basin, whereas this sharp contrast is invisible in GRACE‐ and GLDAS‐EWH primarily due to coarse spatial resolution. The GNSS inferred water products are used to determine the scaling factor for GRACE, with a value of 2.1 in Sichuan. The regional average precipitation peaks in July, about 1–2 months ahead of the maximum of monthly water heights. A bimodal feature in daily precipitation data is most notable in the low‐elevation Sichuan Basin and results in two peaks in daily GNSS‐EWH time series. Our results demonstrate that GNSS can serve as an independent tool for measuring water storage with high spatiotemporal resolution and provides additional constraints for understanding hydrological dynamics.

Multimodal process monitoring based on variational Bayesian PCA and Kullback-Leibler divergence between mixture models

In modern industrial processes, multimodality is a common characteristic and process monitoring tools should be capable of detecting the occurrence of abnormalities in the presence of process mode changes. Although Gaussian mixture model (GMM) is often used to depict the multimodal characteristics of processes, the conventional monitoring schemes rely on identifying the process mode and applying the unimodal statistical methods to each Gaussian component. However, the process mode of any given observation is typically unknown and it may belong to any process mode. Thus, multimodal process cannot be well modeled by unimodal models. In this paper, a multimodal process monitoring method using variational Bayesian principal component analysis (VBPCA) and Kullback-Leibler (KL) divergence between mixture models is proposed. GMM-VBPCA is used to capture multimodal process information. KL divergence between mixture Gaussians is used as statistics of both latent and noise variables to measure the dissimilarity between the reference mixture model and the monitored mixture model with respect to each process mode. Then, Bayesian inference is employed to fuse the statistics and control limits, and the final monitoring result is obtained by considering both latent and noise statistics. Finally, the proposed method and three other representative methods are evaluated through a simulated Continuous Stirred Tank Reactor (CSTR) and an industrial hydrocracking process.

Extracting Common Mode Errors of Regional GNSS Position Time Series in the Presence of Missing Data by Variational Bayesian Principal Component Analysis.

Removal of the common mode error (CME) is very important for the investigation of global navigation satellite systems’ (GNSS) error and the estimation of an accurate GNSS velocity field for geodynamic applications. The commonly used spatiotemporal filtering methods normally process the evenly spaced time series without missing data. In this article, we present the variational Bayesian principal component analysis (VBPCA) to estimate and extract CME from the incomplete GNSS position time series. The VBPCA method can naturally handle missing data in the Bayesian framework and utilizes the variational expectation-maximization iterative algorithm to search each principal subspace. Moreover, it could automatically select the optimal number of principal components for data reconstruction and avoid the overfitting problem. To evaluate the performance of the VBPCA algorithm for extracting CME, 44 continuous GNSS stations located in Southern California were selected. Compared to previous approaches, VBPCA could achieve better performance with lower CME relative errors when more missing data exists. Since the first principal component (PC) extracted by VBPCA is remarkably larger than the other components, and its corresponding spatial response presents nearly uniform distribution, we only use the first PC and its eigenvector to reconstruct the CME for each station. After filtering out CME, the interstation correlation coefficients are significantly reduced from 0.43, 0.46, and 0.38 to 0.11, 0.10, and 0.08, for the north, east, and up (NEU) components, respectively. The root mean square (RMS) values of the residual time series and the colored noise amplitudes for the NEU components are also greatly suppressed, with average reductions of 27.11%, 28.15%, and 23.28% for the former, and 49.90%, 54.56%, and 49.75% for the latter. Moreover, the velocity estimates are more reliable and precise after removing CME, with average uncertainty reductions of 51.95%, 57.31%, and 49.92% for the NEU components, respectively. All these results indicate that the VBPCA method is an alternative and efficient way to extract CME from regional GNSS position time series in the presence of missing data. Further work is still required to consider the effect of formal errors on the CME extraction during the VBPCA implementation.

Neighborhood Variational Bayesian Multivariate Analysis for Distributed Process Monitoring With Missing Data

Conventional methods for distributed monitoring commonly assume that complete process measurements are available. However, the problem of missing data is often encountered in the monitoring of large-scale multiunit processes. This paper proposes an approach based on a neighborhood variational Bayesian principal component analysis (NVBPCA) and canonical correlation analysis (CCA) for the efficient distributed monitoring of multiunit processes in the presence of missing data. Missing observations for a local unit are reconstructed through NVBPCA by considering information from both local and neighboring units. A CCA-based local monitor, which identifies the status of the local unit and the type of a detected fault using information from both the local and neighboring units, is then developed. The NVBPCA–CCA approach has a better performance since its missing data handling and local monitor construction consider information from both the local and neighboring units. The efficiency of the proposed monitoring method is demonstrated through its application in a numerical example and an industrial tail gas treatment process.

Estimation of missing values in a food property database by matrix completion using PCA-based approaches

In this work, five matrix completion algorithms were investigated for the estimation of missing values in a food property database: iterative PCA with (IPCAE) and without (IPCA) early stopping, trimmed scores regression with (TSRE) and without (TSR) early stopping and variational Bayesian PCA (VBPCA). Matrix completion was applied in the context of a food property database (31 properties×663 observations) developed by meta-analysis for new food product development, a novel application of matrix completion. The database contained 68.7% of missing values. VBPCA and TSRE were the most accurate algorithms and explained on average 42% and 40%, respectively, of the variance of the missing values. The incorporation of an early stopping step in the TSR and IPCA algorithms decreased overfitting and improved significantly their accuracy. The accuracy of the missing value estimates varied significantly according to the property, and the coefficient of determination for each property with VBPCA ranged from 0.02 to 0.84. The accuracy of the missing value estimates was higher when the property known for only a few observations were included in the database, indicating that the matrix completion algorithms successfully used the additional information that those properties provided to improve the estimation of the other properties in the database. For 17% of the database, the matrix completion algorithms could identify if the missing value was above or below the average value of the property with a confidence level above 90%, providing additional information for product characterization at no experimental cost.

A Probabilistic Just-in-Time Learning Framework for Soft Sensor Development With Missing Data

Just-in-time learning (JITL) is one of the most widely used strategies for soft sensor modeling in nonlinear processes. However, traditional JITL methods have difficulty in dealing with data samples that contain missing values. Meanwhile, data noises and uncertainties have not been taken into consideration for relevant sample selection in existing JITL approaches. To overcome these problems, a new probabilistic JITL (P-JITL) framework is proposed in this brief. In P-JITL, variational Bayesian principal component analysis is first utilized to handle missing values and extract Gaussian posterior distributions of latent variables. Then, symmetric Kullback-Leibler divergence is creatively employed to measure the dissimilarity of two distributions for relevant sample selection in the JITL framework. Finally, a nonlinear regression model, Gaussian process regression, is carried out to model the nonlinear relationship between the output and the extracted latent variables. In this way, the proposed probabilistic JITL (P-JITL) is able to deal with missing data and select relevant samples more accurately. To evaluate the effectiveness and flexibility of P-JITL, comparative studies between P-JITL and traditional deterministic JITL (D-JITL) are carried out on a numerical example and an industrial application example, in which missing data are simulated with percentages from 0% to 50%. The results show that P-JITL can provide more accurate prediction accuracy than D-JITL in each scenario considered.

Matrix and Tensor Based Methods for Missing Data Estimation in Large Traffic Networks

Intelligent transportation systems (ITSs) gather information about traffic conditions by collecting data from a wide range of on-ground sensors. The collected data usually suffer from irregular spatial and temporal resolution. Consequently, missing data is a common problem faced by ITSs. In this paper, we consider the problem of missing data in large and diverse road networks. We propose various matrix and tensor based methods to estimate these missing values by extracting common traffic patterns in large road networks. To obtain these traffic patterns in the presence of missing data, we apply fixed-point continuation with approximate singular value decomposition, canonical polyadic decomposition, least squares, and variational Bayesian principal component analysis. For analysis, we consider different road networks, each of which is composed of around 1500 road segments. We evaluate the performance of these methods in terms of estimation accuracy, variance of the data set, and the bias imparted by these methods.

Statistical process monitoring with integration of data projection and one-class classification

One-class classification (OCC) has attracted a great deal of attentions from various disciplines. Few attempts are made to extend the scope of such application for process monitoring. In the present work, the Principal Component Analysis (PCA) and Variational Bayesian Principal Component Analysis (VBPCA) approach provides a powerful tool to project original data into lower data set as well as spreading different types of faults with different directions. This, along with multiple types of one-class classifiers (density-based, boundary-based, reconstruction-based and combination-based) that are able to isolate abnormal data from normal one, supported the design of process monitoring. These methodologies have been validated by process data collected from a Wastewater Treatment Plant (WWTP). The results showed that the proposed methodology is capable of detecting sensor faults and process faults with good accuracy under different scenarios.

Condition for perfect dimensionality recovery by variational Bayesian PCA

Having shown its good performance in many applications, variational Bayesian (VB) learning is known to be one of the best tractable approximations to Bayesian learning. However, its performance was not well understood theoretically. In this paper, we clarify the behavior of VB learning in probabilistic PCA (or fully-observed matrix factorization). More specifically, we establish a necessary and sufficient condition for perfect dimensionality (or rank) recovery in the large-scale limit when the matrix size goes to infinity. Our result theoretically guarantees the performance of VB-PCA. At the same time, it also reveals the conservative nature of VB learning--it offers a low false positive rate at the expense of low sensitivity. By contrasting with an alternative dimensionality selection method, we characterize VB learning in PCA. In our analysis, we obtain bounds of the noise variance estimator, and a new and simple analytic-form solution for the other parameters, which themselves are useful for implementation of VB-PCA.

A probabilistic self-validating soft-sensor with application to wastewater treatment

In the wastewater treatment plants (WWTPs), soft sensors are viewed as a simple signal estimator for hard-to-measure quantities. However, the presence of unreliable data, coupled with increasing demands for measurement quality assurance, has rendered inadequate such a simplistic view. In this paper, a probabilistic self-validating soft-sensor is proposed with the capability of performing self-diagnostics, self-reconstruction and online uncertainty measurement. In this framework, data collecting for soft-sensor modeling (easy-to-measure data) is validated by a Variational Bayesian Principal Component Analysis (VBPCA) model before performing a soft-sensor model construction. By integrating Relevant Vector Machine (RVM) as a predictive model, not only prediction values are obtained, but also the credibility of information for easy-to-measure and hard-to-measure quantities can be generated. The performance of the proposed soft-sensor is validated through two simulation studies of WWTPs with different process characteristics. The results suggest that the proposed strategy significantly improves the prediction performance.

Statistical Monitoring of Wastewater Treatment Plants Using Variational Bayesian PCA

Multivariate statistical projection methods such as principal component analysis (PCA) are the most common strategy for process monitoring in wastewater treatment plants (WWTPs). Such monitoring strategies can indeed recognize faults and achieve better control performance for fully observed data sets but can be more difficult in the case of having missing data. This study presents a variational Bayesian PCA (VBPCA) based methodology for fault detection in the WWTPs. This methodology not only is robust against missing data but also reconstructs missing data. Furthermore, a novel historical data preprocess method is proposed to deal with diurnal behaviors in the WWTPs with fast sample rate. These methodologies have been validated by process data collected from two WWTPs with different process characteristics and different sample rates. The results showed that the proposed methodology is capable of detecting sensor faults and process faults with good accuracy under different scenarios (highly and lowly instr...

Speech feature analysis using variational Bayesian PCA

In most hidden Markov model-based automatic speech recognition systems, one of the fundamental questions is to determine the intrinsic speech feature dimensionality and the number of clusters used on the Gaussian mixture model. We analyzed mel-frequency band energies using a variational Bayesian principal component analysis method to estimate the feature dimensionality as well as the number of Gaussian mixtures by learning a maximum lower bound of the evidence instead of maximizing the likelihood function as used in conventional speech recognition systems. In analyzing the Texas Instruments/Massachusetts Institute of Technology (TIMIT) speech database, our method revealed the intrinsic structures of vowels and consonants. The usefulness of this method is demonstrated in the superior classification performance for the most difficult phonemes /b/, /d/, and /g/.