Variational Bayesian Inference Research Articles

Variational Bayesian Inference for Robust Identification of PWARX Systems With Time-Varying Time-Delays.

This article presents a robust variational Bayesian (VB) algorithm for identifying piecewise autoregressive exogenous (PWARX) systems with time-varying time-delays. To alleviate the adverse effects caused by outliers, the probability distribution of noise is taken to follow a t -distribution. Meanwhile, a solution strategy for more accurately classifying undecidable data points is proposed, and the hyperplanes used to split data are determined by a support vector machine (SVM). In addition, maximum-likelihood estimation (MLE) is adopted to re-estimate the unknown parameters through the classification results. The time-delay is regarded as a hidden variable and identified through the VB algorithm. The effectiveness of the proposed algorithm is illustrated by two simulation examples.

Attentional prototype inference for few-shot segmentation

This paper aims to address few-shot segmentation. While existing prototype-based methods have achieved considerable success, they suffer from uncertainty and ambiguity caused by limited labeled examples. In this work, we propose attentional prototype inference (API), a probabilistic latent variable framework for few-shot segmentation. We define a global latent variable to represent the prototype of each object category, which we model as a probabilistic distribution. The probabilistic modeling of the prototype enhances the model’s generalization ability by handling the inherent uncertainty caused by limited data and intra-class variations of objects. To further enhance the model, we introduce a local latent variable to represent the attention map of each query image, which enables the model to attend to foreground objects while suppressing the background. The optimization of the proposed model is formulated as a variational Bayesian inference problem, which is established by amortized inference networks. We conduct extensive experiments on four benchmarks, where our proposal obtains at least competitive and often better performance than state-of-the-art prototype-based methods. We also provide comprehensive analyses and ablation studies to gain insight into the effectiveness of our method for few-shot segmentation.

Bayesian Analysis Methods for Two-Level Diagnosis Classification Models

Understanding whether or not different types of students master various attributes can aid future learning remediation. In this study, two-level diagnostic classification models (DCMs) were developed to represent the probabilistic relationship between external latent classes and attribute mastery patterns. Furthermore, variational Bayesian (VB) inference and Gibbs sampling Markov chain Monte Carlo methods were developed for parameter estimation of the two-level DCMs. The results of a parameter recovery simulation study show that both techniques appropriately recovered the true parameters; Gibbs sampling in particular was slightly more accurate than VB, whereas VB performed estimation much faster than Gibbs sampling. The two-level DCMs with the proposed Bayesian estimation methods were further applied to fourth-grade data obtained from the Trends in International Mathematics and Science Study 2007 and indicated that mathematical activities in the classroom could be organized into four latent classes, with each latent class connected to different attribute mastery patterns. This information can be employed in educational intervention to focus on specific latent classes and elucidate attribute patterns.

A Novel Robust IMM Filtering Method for Surface-Maneuvering Target Tracking with Random Measurement Delay

A proper filtering method for jump Markov system (JMS) is an effective approach for tracking a maneuvering target. Since the coexisting of heavy-tailed measurement noises (HTMNs) and one-step random measurement delay (OSRMD) in the complex scenarios of the surface maneuvering target tracking, the effectiveness of typical interacting multiple model (IMM) techniques may decline severely. To solve the state estimation problem in JMSs with HTMN and OSRMD simultaneously, this article designs a novel robust IMM filter utilizing the variational Bayesian (VB) inference framework. This algorithm models the HTMNs as student’s t-distribuitons, and presents a random Bernoulli variable to describe the OSRMD in JMSs. By transforming measurement likelihood function form from weighted summation to exponential product, this paper constructs hierarchical Gaussian state space models. Then, the state vectors, random Bernoulli vairable, and model probability are inferred jointly according to VB inference. The surface maneuvering target tracking simulation example result indicates that the presented IMM filter achieves superior target state estimation accuracy among existing IMM filters.

Scalable Variational Inference for Low-Rank Spatiotemporal Receptive Fields.

An important problem in systems neuroscience is to characterize how a neuron integrates sensory inputs across space and time. The linear receptive field provides a mathematical characterization of this weighting function and is commonly used to quantify neural response properties and classify cell types. However, estimating receptive fields is difficult in settings with limited data and correlated or high-dimensional stimuli. To overcome these difficulties, we propose a hierarchical model designed to flexibly parameterize low-rank receptive fields. The model includes gaussian process priors over spatial and temporal components of the receptive field, encouraging smoothness in space and time. We also propose a new temporal prior, temporal relevance determination, which imposes a variable degree of smoothness as a function of time lag. We derive a scalable algorithm for variational Bayesian inference for both spatial and temporal receptive field components and hyperparameters. The resulting estimator scales to high-dimensional settings in which full-rank maximum likelihood or a posteriori estimates are intractable. We evaluate our approach on neural data from rat retina and primate cortex and show that it substantially outperforms a variety of existing estimators. Our modeling approach will have useful extensions to a variety of other high-dimensional inference problems with smooth or low-rank structure.

A learning- and scenario-based MPC design for nonlinear systems in LPV framework with safety and stability guarantees

ABSTRACT This paper presents a learning- and scenario-based model predictive control (MPC) design approach for systems modelled in the linear parameter-varying (LPV) framework. Using input-output data collected from the system, a state-space LPV model with uncertainty quantification is first learned through the variational Bayesian inference Neural Network (BNN) approach. The learned probabilistic model is assumed to contain the true dynamics of the system with a high probability and is used to generate scenarios that ensure safety for a scenario-based MPC. Moreover, to guarantee stability and enhance the performance of the closed-loop system, a parameter-dependent terminal cost and controller, as well as a terminal robust positive invariant set are designed. Numerical examples will be used to demonstrate that the proposed control design approach can ensure safety and achieve desired control performance.

Real-time hydrogen release and dispersion modelling of hydrogen refuelling station by using deep learning probability approach

Hydrogen release and dispersion from hydrogen refuelling stations have the potential to cause explosion disaster and bring significant causalities and economic losses to the surroundings. Real-time spatial hydrogen plume concentration prediction is essential for the quick emergency response planning to dissipate such flammable vapor cloud and prevent explosion disaster. Deep learning approaches have recently been applied to real-time gas release and dispersion modeling, however, are ‘over-confident’ for spatial plume concentration and boundary estimation, which could not support the robust decision-makings. This study proposes a hybrid deep probability learning-based spatial hydrogen plume concentration prediction model, namely DPL_H2Plume by integrating deep learning and Variational Bayesian Inference. Numerical model of hydrogen release and dispersion from hydrogen refuelling station is built to construct the benchmark dataset. By using such dataset, two pre-defined parameters, namely Monte Carlo sampling number m = 300 and dropout probability p = 0.1 are determined to ensure the model's tradeoff between inference accuracy and efficiency. Comparison between our proposed model and the state-of-the-art model is also conducted. The results demonstrate that our model exhibits a competitive accuracy of R2 = 0.97 as well as an inference time 3.32 s. In addition, our model gives the comprehensive estimations including not only spatial hydrogen plume concentration but also its uncertainty. Also, our model provides the more accurate estimation at plume boundary compared to the state-of-the-art model. Overall, our proposed model could provide reliable alternative for constructing a digital twin for emergency management of hydrogen refuelling station.

Neighborhood spatial correlations and machine learning classification of fatigue hot-spots in Ti–6Al–4V

The identification of grains with a higher propensity to form a fatigue crack in polycrystalline microstructures is crucial to the design and deployment of materials with increased fatigue resilience. The probability of fatigue crack formation within a grain has typically been quantified with the help of extreme value analyses of Fatigue Indicator Parameters (FIPs). Due to the immense computational cost associated with performing the cyclic crystal plasticity simulations necessary to compute these FIPs, a more efficient strategy to identify the fatigue critical neighborhoods in a microstructure is desirable. In this work, we present a novel surrogate modeling approach to isolate the grains that are likely to exhibit extreme values of the FIPs. This approach significantly extends the previously established Materials Knowledge Systems (MKS) framework to compute neighborhood statistics that capture the salient information about the grain and its neighbors. Variational Bayesian Inference along with Feedforward Neural Networks are utilized to build robust, uncertainty quantified linkages between the local neighborhood statistics and the probability of fatigue crack formation. In doing so, we refine the experimental observations pertaining to the identification of grains that are likely to form a fatigue crack. The approach presented herein is shown to generalize well for both the High Cycle and Transition Fatigue regimes, thereby providing a versatile protocol for multiscale materials design efforts.

VIGA: A variational graph autoencoder model to infer user interest representations for recommendation

Learning representations of both user interests and item characteristics is essentially important for recommendation tasks. Although graph neural network-based methods have significant advantages, there are still two inherent limitations: (i) users or items are often modeled as single embedding points in the vector space, and (ii) these methods usually focus more on the graph structure of user–item interactions, rather than the attribute-related information and useful attribute interactions. These limitations make it difficult to model complex user interests and learn high-quality representations, thus fail to obtain high recommendation accuracy. To overcome these limitations, we propose a variational inference-based graph autoencoder (VIGA) model to explore a multivariate distribution over latent representations for recommendation. In brief, to identify both of the complex users' interests and their interactive behaviors, VIGA first encodes users and items into latent variables, and subjects them to a multivariate Gaussian distribution via a graph autoencoder framework based on Bayesian variational inference. Then a feature fusion layer is established for the exchange and integration of both the topological structure and nodes attributes in the user–item interaction graph. Experiments on three real-world datasets show that the proposed VIGA method significantly outperforms state-of-the-art recommendation methods on recommendation accuracy.

Data-driven nested robust optimization for generation maintenance scheduling considering temporal correlation

Traditional power grids are gradually transitioning to smart grids with high penetration of renewable energy, which can realize the efficient utilization of power resources and low carbon emissions. However, the uncertainties of renewable energy (e.g., wind power) and load demand pose considerable challenges to secure operation and cost-effective planning in smart grids, such as generation maintenance scheduling (GMS). In this context, conventional methods including stochastic optimization and robust optimization are adopted to cope with the uncertainties and formulate the GMS plan. Unfortunately, these methods fail to consider the temporal information in uncertain variables, which can introduce extra operational costs brought by the uncertainties. To address this issue, we consider the temporal correlation of the uncertain wind power and load demand, and develop a data-driven two-stage nested robust optimization (NRO) approach for GMS to minimize the total costs of power system operation under uncertain scenarios. In our proposed approach, a temporal correlation Dirichlet process mixture model (TCDPMM) is developed to investigate the temporal information in the wind power and load demand datasets. Then, variational Bayesian inference (VBI) is employed to construct the data-driven uncertainty set, in which the temporal information for the uncertain variables and the correlations between the uncertain variables are considered. Subsequently, combined with this uncertainty set, a two-stage GMS problem is converted to a “min–max-max–min” optimization problem which is solved by the parallel Benders’ decomposition algorithm. The effectiveness and superiority of the proposed approach are demonstrated with a six-bus power system and a practical power system in China.

Sparse Bayesian learning approach for discrete signal reconstruction

This study addresses the problem of discrete signal reconstruction from the perspective of sparse Bayesian learning (SBL). Generally, it is intractable to perform the Bayesian inference with the ideal discretization prior under the SBL framework. To overcome this challenge, we introduce a novel discretization enforcing prior to exploit the knowledge of the discrete nature of the signal-of-interest. By integrating the discretization enforcing prior into the SBL framework and applying the variational Bayesian inference (VBI) methodology, we devise an alternating optimization algorithm to jointly characterize the finite-alphabet feature and reconstruct the unknown signal. When the measurement matrix is i.i.d. Gaussian per component, we further embed the generalized approximate message passing (GAMP) into the VBI-based method, so as to directly adopt the ideal prior and significantly reduce the computational burden. Simulation results demonstrate substantial performance improvement of the two proposed methods over existing schemes. Moreover, the GAMP-based variant outperforms the VBI-based method with i.i.d. Gaussian measurement matrices but it fails to work for non i.i.d. Gaussian matrices.

A variational Bayesian inference technique for model updating of structural systems with unknown noise statistics

Dynamic models of structural and mechanical systems can be updated to match the measured data through a Bayesian inference process. However, the performance of classical (non-adaptive) Bayesian model updating approaches decreases significantly when the pre-assumed statistical characteristics of the model prediction error are violated. To overcome this issue, this paper presents an adaptive recursive variational Bayesian approach to estimate the statistical characteristics of the prediction error jointly with the unknown model parameters. This approach improves the accuracy and robustness of model updating by including the estimation of model prediction error. The performance of this approach is demonstrated using numerically simulated data obtained from a structural frame with material non-linearity under earthquake excitation. Results show that in the presence of non-stationary noise/error, the non-adaptive approach fails to estimate unknown model parameters, whereas the proposed approach can accurately estimate them.

Sparse angular reciprocity learning for massive MIMO channel estimation

Sparse angular reciprocity refers to the fact that downlink and uplink channels share common sparsity in angular domain. This sophisticated sparsity can be exploited to improve the channel estimation performance for massive multiple-input multiple-output (MIMO) systems. All the existing sparse angular reciprocity methods solve the uplink channel estimation problem independently with an LS estimator in the first stage, and then assist the downlink channel estimation with the sparse angular reciprocity in the second stage. Such a two-stage scheme, as well as the prior LS estimator, will inevitably result in performance loss. To overcome the drawback, we consider the downlink and uplink channel estimation problems as a whole one, and present a new sparse angular reciprocity learning framework to leverage the sparse angular reciprocity. Since the framework provides a joint solution for the downlink and uplink channel estimation in the sense of Bayesian optimality, it is expected to yield higher channel estimation accuracy. Moreover, we devise an efficient variational Bayesian inference (VBI)-based algorithm to automatically decouple the uplink channel. The proposed method can further enhance the channel estimation performance as the prior LS estimator is avoided. Another advantage is that it can be applied to the underdetermined uplink channel estimation problem, where the number of uplink pilot symbols is less than the number of MUs. Simulation results verify the superiority of the proposed method.

A robust Cubature Kalman filter for nonlinear systems subject to randomly occurring measurement anomalies without a priori statistic

In this work, we investigate the problem of state estimation for a class of nonlinear systems subjected to randomly occurring measurement anomalies (ROMAs) without a priori statistic. To address the problem, first, a novel measurement model is constructed, in which the anomalous measurements and anomaly probability are modeled as Gaussian mixture distribution (GMD) and Beta distribution, respectively. Different from the existing researches assuming that the statistical information of anomalous measurements is known in advance, the model does not require a priori statistical knowledge of anomalous measurements. Moreover, by adaptive learning of the anomaly probability, the measurement model is identical with the classical cubature Kalman filter (CKF) in the absence of measurement anomalies. Then, the variational Bayesian inference (VBI) is employed to approximately calculate the joint posterior distribution of the system state and unknown parameters, and a robust filter is derived. Finally, the effectiveness of our filter is demonstrated by the numerical simulation.

Density-adaptive registration of pointclouds based on Dirichlet Process Gaussian Mixture Models

We propose an algorithm for rigid registration of pre- and intra-operative patient anatomy, represented as pointclouds, during minimally invasive surgery. This capability is essential for development of augmented reality systems for guiding such interventions. Key challenges in this context are differences in the point density in the pre- and intra-operative pointclouds, and potentially low spatial overlap between the two. Solutions, correspondingly, must be robust to both of these phenomena. We formulated a pointclouds registration approach which considers the pointclouds after rigid transformation to be observations of a global non-parametric probabilistic model named Dirichlet Process Gaussian Mixture Model. The registration problem is solved by minimizing the Kullback-Leibler divergence in a variational Bayesian inference framework. By this means, all unknown parameters are recursively inferred, including, importantly, the optimal number of mixture model components, which ensures the model complexity efficiently matches that of the observed data. By presenting the pointclouds as KDTrees, both the data and model are expanded in a coarse-to-fine style. The scanning weight of each point is estimated by its neighborhood, imparting the algorithm with robustness to point density variations. Experiments on several datasets with different levels of noise, outliers and pointcloud overlap show that our method has a comparable accuracy, but higher efficiency than existing Gaussian Mixture Model methods, whose performance is sensitive to the number of model components.

Local convexity of the TAP free energy and AMP convergence for Z2-synchronization

We study mean-field variational Bayesian inference using the TAP approach, for Z2-synchronization as a prototypical example of a high-dimensional Bayesian model. We show that for any signal strength λ>1 (the weak-recovery threshold), there exists a unique local minimizer of the TAP free energy functional near the mean of the Bayes posterior law. Furthermore, the TAP free energy in a local neighborhood of this minimizer is strongly convex. Consequently, a natural-gradient/mirror-descent algorithm achieves linear convergence to this minimizer from a local initialization, which may be obtained by a constant number of iterations of Approximate Message Passing (AMP). This provides a rigorous foundation for variational inference in high dimensions via minimization of the TAP free energy. We also analyze the finite-sample convergence of AMP, showing that AMP is asymptotically stable at the TAP minimizer for any λ>1, and is linearly convergent to this minimizer from a spectral initialization for sufficiently large λ. Such a guarantee is stronger than results obtainable by state evolution analyses, which only describe a fixed number of AMP iterations in the infinite-sample limit. Our proofs combine the Kac–Rice formula and Sudakov–Fernique Gaussian comparison inequality to analyze the complexity of critical points that satisfy strong convexity and stability conditions within their local neighborhoods.

Real-time natural gas explosion modeling of offshore platforms by using deep learning probability approach

Natural gas explosion of offshore platform is prone to cause accidental disaster such as platform collapse and casualties etc. Real-time natural gas explosion consequence reconstruction is essential to support a quick accidental emergency response planning to prevent the accidental escalation to disaster. The widely-used CFD is computationally intensive and thereby has a significant delay. Machine/deep learning-based models offer a potential real-time alternative, which however are not able to quantify the uncertainty of spatial overpressure prediction. This study aims to propose a hybrid deep learning probability model to real-time predict spatial explosion overpressure of offshore platform by using sparsely-observed overpressures. In this hybrid model, Variational Bayesian inference is incorporated into deep learning backbone. Both natural gas explosion experimental and numerical modeling of offshore platform are conducted to construct the benchmark dataset. By using this benchmark dataset, sensitivity analysis of Monte Carlo sampling number N, drop probability p on model's performance is also conducted. The results demonstrated our model exhibits high accuracy with R2 = 0.955 and real-time capability with inference time of 2.9s. Compared to the state-of-the-art model, the additional uncertainty estimation improves the accuracy and robustness of spatial overpressure prediction, which contributes to the reliable explosion accidental emergency decision-making. Overall, this study provides a reliable alternative for constructing digital twin emergency management system to effectively manage natural gas explosion risk of offshore platforms.

Robust sparse Bayesian learning based on the Bernoulli-Gaussian model of impulsive noise

Traditional sparse reconstruction methods suffer from performance degradation in the presence of impulsive noise. In this paper, we adopt the Bernoulli-Gaussian distribution to model the impulsive noise and introduce the hidden state variables to indicate the state of the noise. Based on the new hierarchical Bayesian model, a robust sparse reconstruction algorithm called Bernoulli-Gaussian robust sparse Bayesian learning (BG-RSBL) is developed through the variational Bayesian inference framework. Different from existing robust reconstruction methods, BG-RSBL does not make sparse assumptions about the impulses and outliers. The employment of the Bernoulli-Gaussian model enables BG-RSBL to adapt to environments with a wider range of impulse densities. Comparative analysis shows that the reconstruction performance can be improved by including the outliers in posterior inference with appropriate weightings. Simulations performed under both Bernoulli-Gaussian and symmetric α-stable distributed impulsive noise demonstrate the robust and outstanding reconstruction performance of the proposed algorithm.

Compressive Sensing via Variational Bayesian Inference under Two Widely Used Priors: Modeling, Comparison and Discussion.

Compressive sensing is a sub-Nyquist sampling technique for efficient signal acquisition and reconstruction of sparse or compressible signals. In order to account for the sparsity of the underlying signal of interest, it is common to use sparsifying priors such as Bernoulli-Gaussian-inverse Gamma (BGiG) and Gaussian-inverse Gamma (GiG) priors on the components of the signal. With the introduction of variational Bayesian inference, the sparse Bayesian learning (SBL) methods for solving the inverse problem of compressive sensing have received significant interest as the SBL methods become more efficient in terms of execution time. In this paper, we consider the sparse signal recovery problem using compressive sensing and the variational Bayesian (VB) inference framework. More specifically, we consider two widely used Bayesian models of BGiG and GiG for modeling the underlying sparse signal for this problem. Although these two models have been widely used for sparse recovery problems under various signal structures, the question of which model can outperform the other for sparse signal recovery under no specific structure has yet to be fully addressed under the VB inference setting. Here, we study these two models specifically under VB inference in detail, provide some motivating examples regarding the issues in signal reconstruction that may occur under each model, perform comparisons and provide suggestions on how to improve the performance of each model.

Covariance-Free Variational Bayesian Learning for Correlated Block Sparse Signals

We consider the problem of estimating channel in massive machine type communication (mMTC) systems. The sparse device activity in a mMTC system makes the channel block-sparse, with intra-block correlation. Block-sparse Bayesian learning (B-SBL) is a powerful framework for estimating such signals. The existing B-SBL algorithms become computationally expensive for high-dimensional problems, which is common in mMTC systems. This is because of large number of devices in a mMTC system, they invert a large-dimensional matrix to calculate the covariance matrix. To address this problem, we exploit variational Bayesian inference, and design a novel covariance-free variational B-SBL algorithm which inverts multiple small-sized block matrices, instead of inverting a complete big-sized matrix. The complexity is further reduced by avoiding explicit computation of the covariance matrix. The proposed algorithm, instead of performing costly matrix inversions, solves multiple linear systems to calculate an unbiased estimate of the posterior statistics. The proposed algorithm is numerically shown to estimate the mMTC channel with a much lesser complexity, and that too without compromising the reconstruction performance.