Variational Bayesian Approximation Research Articles

Variational Bayesian Approximation (VBA): Implementation and Comparison of Different Optimization Algorithms.

In any Bayesian computations, the first step is to derive the joint distribution of all the unknown variables given the observed data. Then, we have to do the computations. There are four general methods for performing computations: Joint MAP optimization; Posterior expectation computations that require integration methods; Sampling-based methods, such as MCMC, slice sampling, nested sampling, etc., for generating samples and numerically computing expectations; and finally, Variational Bayesian Approximation (VBA). In this last method, which is the focus of this paper, the objective is to search for an approximation for the joint posterior with a simpler one that allows for analytical computations. The main tool in VBA is to use the Kullback-Leibler Divergence (KLD) as a criterion to obtain that approximation. Even if, theoretically, this can be conducted formally, for practical reasons, we consider the case where the joint distribution is in the exponential family, and so is its approximation. In this case, the KLD becomes a function of the usual parameters or the natural parameters of the exponential family, where the problem becomes parametric optimization. Thus, we compare four optimization algorithms: general alternate functional optimization; parametric gradient-based with the normal and natural parameters; and the natural gradient algorithm. We then study their relative performances on three examples to demonstrate the implementation of each algorithm and their efficiency performance.

Dynamical system identification, model selection, and model uncertainty quantification by Bayesian inference.

This study presents a Bayesian maximum a posteriori (MAP) framework for dynamical system identification from time-series data. This is shown to be equivalent to a generalized Tikhonov regularization, providing a rational justification for the choice of the residual and regularization terms, respectively, from the negative logarithms of the likelihood and prior distributions. In addition to the estimation of model coefficients, the Bayesian interpretation gives access to the full apparatus for Bayesian inference, including the ranking of models, the quantification of model uncertainties, and the estimation of unknown (nuisance) hyperparameters. Two Bayesian algorithms, joint MAP and variational Bayesian approximation, are compared to the least absolute shrinkage and selection operator (LASSO), ridge regression, and the sparse identification of nonlinear dynamics (SINDy) algorithms for sparse regression by application to several dynamical systems with added Gaussian or Laplace noise. For multivariate Gaussian likelihood and prior distributions, the Bayesian formulation gives Gaussian posterior and evidence distributions, in which the numerator terms can be expressed in terms of the Mahalanobis distance or "Gaussian norm" ||y-y^||M-12=(y-y^)⊤M-1(y-y^), where y is a vector variable, y^ is its estimator, and M is the covariance matrix. The posterior Gaussian norm is shown to provide a robust metric for quantitative model selection for the different systems and noise models examined.

Passive underwater tracking with unknown measurement noise statistics using variational Bayesian approximation

This paper considers a bearings-only tracking problem with unknown measurement noise statistics. It is assumed that the measurement noise follows a Gaussian probability density function where the mean and the covariance of the noise are unknown. Here, an adaptive nonlinear filtering technique is proposed, where the joint distribution of the measurement noise mean and its covariance are considered to follow a normal inverse Wishart (NIW) distribution. Using the variational Bayesian (VB) approximation, joint distribution of the target state, the measurement noise mean and covariance is factorized as the product of their individual probability density function (pdf). Minimizing the Kullback-Leibler divergence (KLD) between the factorized and true joint pdfs, probability distributions of the noise mean, covariance and the target states are evaluated. The estimation of states with the proposed VB based method is compared with the maximum a posteriori (MAP) and the maximum likelihood estimation (MLE) based adaptive filtering. Deterministic sigma points are used to realize the filtering algorithms. The proposed adaptive filter with VB approximation is found to be more accurate compared to their corresponding MAP-MLE based counterparts.

Primal-Dual Decomposed State Estimation for Multi-Energy Systems Leveraging Variational Bayesian Approximation

Primal-Dual Decomposed State Estimation for Multi-Energy Systems Leveraging Variational Bayesian Approximation

Ensemble variational Bayesian approximation for the inversion and uncertainty quantification of Darcy flows in heterogeneous porous media with random parameters

The inversion and uncertainty quantification of physical parameters are important for many science and engineering problems. For nonlinear parameter inversion problems of flows in hydrocarbon reservoirs, estimating the true posterior probability distribution function for the random parameter is difficult when the dimension of the parameter is relatively high. Inversion methods such as gradient-based or heuristic optimization methods searching for the maximum a posterior are generally point-estimate, while uncertainty quantification using MCMC is computationally expensive. In the current study, a new ensemble variational Bayesian approximation method is developed to estimate the posterior probability density using an ensemble of realizations regarded as a trial sample distribution. The objective is to optimize the trial sample distribution such that the evidence lower bound is maximized indicating that the distance between the trial and true posterior distribution is minimized. A subspace is built using principle component analysis based on finite samples to obtain a diagonal positive-definite covariance matrix, and ensemble-based data assimilation is used to optimize the trial distribution efficiently. Heterogeneous 2D and 3D test cases of transient single- and two-phase Darcy flows are presented for validation.

Simultaneous estimation of a model-derived input function for quantifying cerebral glucose metabolism with [18F]FDG PET

BackgroundQuantification of the cerebral metabolic rate of glucose (CMRGlu) by dynamic [18F]FDG PET requires invasive arterial sampling. Alternatives to using an arterial input function (AIF) include the simultaneous estimation (SIME) approach, which models the image-derived input function (IDIF) by a series of exponentials with coefficients obtained by fitting time activity curves (TACs) from multiple volumes-of-interest. A limitation of SIME is the assumption that the input function can be modelled accurately by a series of exponentials. Alternatively, we propose a SIME approach based on the two-tissue compartment model to extract a high signal-to-noise ratio (SNR) model-derived input function (MDIF) from the whole-brain TAC. The purpose of this study is to present the MDIF approach and its implementation in the analysis of animal and human data.MethodsSimulations were performed to assess the accuracy of the MDIF approach. Animal experiments were conducted to compare derived MDIFs to measured AIFs (n = 5). Using dynamic [18F]FDG PET data from neurologically healthy volunteers (n = 18), the MDIF method was compared to the original SIME-IDIF. Lastly, the feasibility of extracting parametric images was investigated by implementing a variational Bayesian parameter estimation approach.ResultsSimulations demonstrated that the MDIF can be accurately extracted from a whole-brain TAC. Good agreement between MDIFs and measured AIFs was found in the animal experiments. Similarly, the MDIF-to-IDIF area-under-the-curve ratio from the human data was 1.02 ± 0.08, resulting in good agreement in grey matter CMRGlu: 24.5 ± 3.6 and 23.9 ± 3.2 mL/100 g/min for MDIF and IDIF, respectively. The MDIF method proved superior in characterizing the first pass of [18F]FDG. Groupwise parametric images obtained with the MDIF showed the expected spatial patterns.ConclusionsA model-driven SIME method was proposed to derive high SNR input functions. Its potential was demonstrated by the good agreement between MDIFs and AIFs in animal experiments. In addition, CMRGlu estimates obtained in the human study agreed to literature values. The MDIF approach requires fewer fitting parameters than the original SIME method and has the advantage that it can model the shape of any input function. In turn, the high SNR of the MDIFs has the potential to facilitate the extraction of voxelwise parameters when combined with robust parameter estimation methods such as the variational Bayesian approach.

Variational Bayesian Modal Composition Beamforming for fast-rotating axial-fan blade-noise localization and its application condition

The Modal Composition Beamforming (MCB) method can quickly achieve the localization of fast-rotating sound sources; however, its low resolution and unclear applied conditions severely limit its industrial application. This study aims to investigate the MCB-based high-resolution localization method for multiple rotating sound sources and apply it to the localization and identification of axial-fan blade-noise. In this paper, the Variational Bayesian Approximation (VBA) and the Subspace Variational Bayesian Approximation (SVB) methods are used to solve the MCB-based rotating sound power propagation (RSP) model, denoted as RSP-VBA and RSP-SVB, respectively. The effectiveness of the proposed RSP-VBA and RSP-SVB are experimentally validated for the first time. Compared to the conventional MCB method, the resolution is significantly improved, and the proposed high-resolution methods are even faster than the classical Rotating Source Identifier (ROSI) method in most conditions. More importantly, the applied condition of the MCB, RSP-VBA, and RSP-SVB methods is given and verified by using three evaluation indicators. Then a schematic of the applied condition with examples is provided. With the guide of the applied conditions, the RSP-VBA and RSP-SVB methods are applied to the blade-noise localization of various multiblade high-speed axial fans.

Variational Bayesian-based robust adaptive filtering for GNSS/INS tightly coupled positioning in urban environments

Kalman filter (KF) is an efficient approach for state estimation in integrated positioning systems of global navigation satellite systems and inertial navigation systems (GNSS/INS). Unfortunately, GNSS/INS integrated positioning systems are susceptible to state model perturbations and measurement outliers, which can severely degrade KF performance, especially in complex urban environments. To solve this problem, a robust adaptive filtering algorithm based on variational Bayesian (VBRAKF) is presented that can be employed in GNSS/INS tightly coupled positioning systems. In VBRAKF, the robust estimation approach is first introduced into the filter, followed by modeling the predicted covariance matrix using the inverse Wishart distribution, and finally the state is estimated using Variational Bayesian. The robustness is achieved by adjusting the noise by an equivalent weight matrix to reduce gross errors in observations, while the adaptiveness is realized by the variational Bayesian estimation method for the uncertain predicted noise covariance matrix. The positioning performance of the proposed algorithm in urban environments is verified through two sets of land vehicle experiments. The experimental results demonstrate that the 3D positioning accuracy of the VBRAKF algorithm is improved by 32.1 % and 23.5 % in experiment 1 and by 37.1 % and 17.3 % in experiment 2 when compared to KF and Robust Kalman Filter (RKF). The proposed VBRAKF algorithm has better positioning performance in all schemes, can simultaneously control the influence of measurement outliers and inaccurate noise statistics and improves the accuracy of filter estimation.

Joint Latent Space Model for Social Networks with Multivariate Attributes.

In social, behavioral and economic sciences, researchers are interested in modeling a social network among a group of individuals, along with their attributes. The attributes can be responses to survey questionnaires and are often high dimensional. We propose a joint latent space model (JLSM) that summarizes information from the social network and the multivariate attributes in a person-attribute joint latent space. We develop a variational Bayesian expectation-maximization estimation algorithm to estimate the attribute and person locations in the joint latent space. This methodology allows for effective integration, informative visualization and prediction of social networks and attributes. Using JLSM, we explore the French financial elites based on their social networks and their career, political views and social status. We observe a division in the social circles of the French elites in accordance with the differences in their attributes. We analyze user networks and behaviors in multimodal social media systems like YouTube. A R package "jlsm" is developed to fit the models proposed in this paper and is publicly available from the CRAN repository https://cran.r-project.org/web/packages/jlsm/jlsm.pdf .

A Tight Coupling Algorithm for Strapdown Inertial Navigation System (SINS)/Global Positioning System (GPS) Adaptive Integrated Navigation Based on Variational Bayesian

Multi-source nonlinear noise exists in the process of multi-source navigation information fusion in long-endurance positioning systems in complex environments. In such engineering applications, the classical Kalman filter (KF) and the extended Kalman filter (EKF) have the phenomena of noise instability and parameter drift, which lead to the divergence of filtering results and reductions in accuracy over long periods of time. Aiming at the above problems, this paper proposes a fusion algorithm of the variational Bayesian (VB) and the cubature Kalman filter (CKF). Firstly, the system is modeled through nonlinear filtering, and the CKF error equation is established by taking the position difference and velocity difference between SINS and GPS as observation variables. Then, to address the problem of poor self-adaptation of the CKF algorithm, the variational Bayesian adaptive estimation method is introduced into the CKF algorithm, and a measurement noise variance estimation model is introduced to the process of time and measurement updates of the CKF algorithm to finally obtain the adaptive VB–CKF algorithm. The simulation results from the experimental platform show that the proposed fusion algorithm improves the combined SINS/GPS system by about 30% in terms of attitude angle accuracy and reduces speed and position estimation errors (RMSE) by about 45%. At the same time, comprehensive experiments on multiple types of sites show that compared with the CKF algorithm, the VB–CKF algorithm improves the positioning accuracy by 10% when the GPS signal is stable and improves the accuracy by about 38% when the GPS measurement noise changes dramatically in complex terrain, which effectively suppresses the accuracy divergence of the CKF algorithm and has high value for engineering applications.

A Bayesian Framework of Non-Synchronous Measurements at Coprime Positions for Sound Source Localization With High Resolution

The noise distribution in the mechanical system is tightly connected to the structure design and particular operating conditions. The noise source localization can effectively assist with the operating condition monitoring and noise reduction design of the mechanical device. The performance limitation of the array aperture for lower frequency acoustic localization is broken up by the non-synchronous measurement at coprime positions (CP-NSM), while this method has high uncertainty and requires an efficient adaptive regularization method to solve its corresponding acoustic inverse problem. The algorithms under the Bayesian framework with Student-t priors (variational Bayesian approximation and subspace variational Bayesian) are derived and deployed to solve the acoustic inverse problem in the CP-NSM method, and the results are compared with those obtained by the interior-point method and the alternating direction method of the multipliers algorithm. The proposed Bayesian methods have the advantages of adaptive regularization parameter estimation, which can reduce the influence of various interferences in CP-NSM. At the same time, in addition to using simulations and experiments in the anechoic chambers, the proposed Bayesian algorithms are validated further in real industrial applications. The proposed method is applied to improve the noise reduction design of the centrifugal fan, a mechanical device containing more information from noise sources.

Fused Microwave and Ultrasonic Breast Imaging Within the Framework of a Joint Variational Bayesian Approximation

Early diagnosis of breast tumors can, in principle, be achieved by jointly running electromagnetic and acoustic probings, more precisely microwave (MW) and ultrasound (US). Indeed, such modalities are noninvasive, nonionizing, and low cost and proceed without registration for free pending breasts, not like other joint modalities that impose compression. Because of the strongly contrasted electromagnetic parameters of breast constituents, MW yields high-contrast images of low resolution and converse with the US when faced with weakly refracting elements. The key benefit is the common breast structure, and fusion should produce images with both high contrast and resolution. A Bayesian formalism is chosen, and an unsupervised joint variational Bayesian approximation (JVBA) is developed. In that, edges’ hidden variables and hyperparameters are automatically tuned along with the optimization. The mathematics is detailed in a general setting of fusion, and then, one proposes imaging of realistic MRI-derived breast slices. A wealth of numerical simulations from noisy single-frequency MW and multiple-frequency US data preserved from inverse crime yields means (i.e., breast maps) and variances of the unknown electromagnetic and acoustic parameter distributions as probabilistic realizations, evolutions of hyperparameters, as well as global indicators of accuracy. Comparisons with a joint contrast-source inversion edge-preserving (JCSI-EP) developed earlier in a deterministic framework illustrate the methodology.

An improved adaptive robust information filter for spacecraft relative navigation

This work concentrates on addressing the problem of time-varying measurement noise covariance and outliers (contaminated Gaussian distribution), which can significantly deteriorate the estimation accuracy of conventional Kalman filter. Adaptive filters or robust filters are only suitable for solving one of the above-mentioned problems. Meanwhile, the existing adaptive robust filters, performing unsatisfactorily, also can not meet the requirements of spacecraft relative navigation, which needs an accurate and reliable filter. This paper proposes an improved adaptive robust information filter, referred to as variational Bayesian (VB) adaptive dynamic-covariance-scaling (DCS)-based cubature information filter (VB-DCSCIF). The proposed filter, based on the cubature information filter framework, uses VB approximation to track measurement noise covariance with time-varying statistical characteristics, and relies on the improved DCS kernel function to suppress outliers in measurements. Therefore, VB-DCSCIF incorporates the advantages of VB approximation and the improved DCS kernel, exhibiting adaptivity and robustness. Spacecraft relative navigation simulations and Monte Carlo simulations demonstrate that VB-DCSCIF outperforms other filters for providing high-precision state estimation under time-varying measurement noise covariance and outliers.

Adaptive maximum correntropy based robust CKF with variational Bayesian for covariance estimation

To address the interference of outliers on the estimation of state and measurement noise covariance matrix, an adaptive maximum correntropy cubature Kalman filter with variational Bayesian approximation over a sliding window is proposed. The multiple kernel size is adjusted for different noise within a reasonable range based on the squared Mahalanobis distance of innovation, which overcomes the excessive convergence problem in the adjustment process. The correntropy matrix is established using the adaptive multiple kernel size to achieve measurement-specific outliers processing. Then the measurement noise covariance matrix is updated as inverse Wishart distribution exploiting the posterior smoothing-based variational Bayesian approximations with correntropy matrix, suppressing the disturbance of measurement outliers to the modification of the measurement noise covariance matrix. Finally, the target tracking simulation and cooperative positioning experiment demonstrate that the proposed method can effectively achieve the robust state estimation with accurate modification of MNCM in the presence of outliers.

EXTENDING THE LEE–CARTER MODEL WITH VARIATIONAL AUTOENCODER: A FUSION OF NEURAL NETWORK AND BAYESIAN APPROACH

AbstractIn this study, we propose a nonlinear Bayesian extension of the Lee–Carter (LC) model using a single-stage procedure with a dimensionality reduction neural network (NN). LC is originally estimated using a two-stage procedure: dimensionality reduction of data by singular value decomposition followed by a time series model fitting. To address the limitations of LC, which are attributed to the two-stage estimation and insufficient model fitness to data, single-stage procedures using the Bayesian state-space (BSS) approaches and extensions of flexibility in modeling by NNs have been proposed. As a fusion of these two approaches, we propose a NN extension of LC with a variational autoencoder that performs the variational Bayesian estimation of a state-space model and dimensionality reduction by autoencoding. Despite being a NN model that performs single-stage estimation of parameters, our model has excellent interpretability and the ability to forecast with confidence intervals, as with the BSS models, without using Markov chain Monte Carlo methods.

Variational Bayesian Estimator for Mobile Robot Localization With Unknown Noise Covariance

This article studies mobile robot localization with unknown noise covariance. The AprilTag is used as landmarks and observed using the onboard camera. The system model is created based on the mobile robot motion and AprilTag measurements. The unknown measurement noise covariance is considered as a random matrix satisfying an inverse Wishart distribution. A variational Bayesian estimator is proposed to estimate the robot pose, AprilTag locations, and measurement noise covariance, where the Rao–Blackwellized estimator is developed for robot pose and AprilTag location estimation, and variational Bayesian approximation is adopted for measurement noise covariance estimation. Simulations and experiments are conducted to validate the effectiveness of the proposed method.

Trajectory PHD Filter for Adaptive Measurement Noise Covariance Based on Variational Bayesian Approximation

In order to solve the problem that the measurement noise covariance may be unknown or change with time in actual multi-target tracking, this paper brings the variational Bayesian approximation method into the trajectory probability hypothesis density (TPHD) filter and proposes a variational Bayesian TPHD (VB-TPHD) filter to obtain measurement noise covariance adaptively. By modeling the unknown covariance as the random matrix that obeys the inverse gamma distribution, VB-TPHD filter minimizes the Kullback–Leibler divergence (KLD) and estimates the sequence of multi-trajectory states with noise covariance matrices simultaneously. We propose the Gaussian mixture VB-TPHD (AGM-VB-TPHD) filter under adaptive newborn intensity for linear Gaussian models and also give the extended Kalman (AEK-VB-TPHD) filter and unscented Kalman (AUK-VB-TPHD) filter in nonlinear Gaussian models. The simulation results prove the effectiveness of the idea that the VB-TPHD filter can form robust and stable trajectory filtering while learning adaptive measurement noise statistics. Compared with the tag-VB-PHD filter, the estimated error of the VB-TPHD filter is greatly reduced, and the estimation of the trajectory number is more accurate.

Attitude Estimation of SINS on Underwater Dynamic Base With Variational Bayesian Robust Adaptive Kalman Filter

Kalman filter (KF), as an effective method to solve the attitude estimation problem in the initial alignment process of strapdown inertial navigation system (SINS), has been widely used in engineering practice. Unfortunately, in the complex underwater environment, the measurement is easily contaminated by outliers, and the measurement noise covariance is unknown or time-varying, which will degrade the performance of KF severely. To solve this problem, this paper proposes an improved KF with both robustness and adaptivity (namely VBRAKF), and applies it to the in-motion attitude estimation for Doppler velocity log (DVL) aided SINS. In the VBRAKF, the robustness is achieved by suppressing the outliers based on Mahalanobis distance (MD) of the innovation, and the adaptivity is achieved by estimating the uncertain measurement noise covariance based on variational Bayesian (VB) approximation simultaneously. The SINS/DVL shipboard tests were carried out when the outputs of DVL are contaminated by outliers or Gaussian mixture distribution noise, respectively. The tests results demonstrate the superiority of the proposed method over the traditional methods.

State of charge estimation of lithium-ion battery under time-varying noise based on Variational Bayesian Estimation Methods

Non-linear filters such as UKF, CKF are often used to estimate the State of Charge (SoC) of lithium batteries, but the premise is that the noise is Gaussian noise, but the actual noise cannot be Gaussian noise, such as the process noise or measurement noise changed with time, which will affect the estimated effect. In this paper, a second-order battery model is established, and the Kalman filter is used to identify the model parameters, and then three methods are used to obtain the SOC-OCV mapping curve. Next, to improve the performance of SoC estimation under the influence of time-varying measurement noise or process noise, the idea of Variable Bayesian iteration is introduced into the nonlinear filter to obtain the Variable Bayesian unscented Kalman filter and the Variable Bayesian square-root cubature filter. Then the two methods are used to estimate the SoC under four working conditions, two temperatures, and two kinds of covariance time-varying noise. Experimental results show that the proposed method can effectively improve the estimation performance of SoC, and the SOC-OCV mapping curve used in this paper is more stable.

Fault-tolerant SINS/HSB/DVL underwater integrated navigation system based on variational Bayesian robust adaptive Kalman filter and adaptive information sharing factor

To solve the problem that position information obtained by SINS/DVL integrated navigation system is divergent in the underwater navigation applications, this paper introduces hydroacoustic single beacon (HSB) into the underwater integrated system and proposes a novel SINS/HSB/DVL fault-tolerant federated variational Bayesian robust adaptive Kalman filter (FVBRAKF). In the FVBRAKF, the traditional KF is replaced by VBRAKF, which not only suppress the adverse influence of outliers based on Mahalanobis distance (MD) effectively, but also estimate the unknown measurement noise covariance based on variational Bayesian (VB) approximation adaptively. Meanwhile, a novel adaptive information sharing factor (ISF) method is proposed during the information fusion process to form an improved FVBRAKF（IFVBRAKF）, which can adjust the fusion weights in real-time according to the accuracy of local filter. The semi-physical simulation experiments for SINS/HSB/DVL underwater integrated navigation system based on the test data are carried out to verify the adaptive ability of the ISF and the robust adaptive ability of the method, respectively. Experimental results demonstrate that the proposed algorithm can not only improve the estimation accuracy, but also enhance the fault-tolerant performance.