Variational Inference Framework Research Articles

GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework.

Particle-based Variational Inference (ParVI) methods have been widely adopted in deep Bayesian inference tasks such as Bayesian neural networks or Gaussian Processes, owing to their efficiency in generating high-quality samples given the score of the target distribution. Typically, ParVI methods evolve a weighted-particle system by approximating the first-order Wasserstein gradient flow to reduce the dissimilarity between the particle system's empirical distribution and the target distribution. Recent advancements in ParVI have explored sophisticated gradient flows to obtain refined particle systems with either accelerated position updates or dynamic weight adjustments. In this paper, we introduce the semi-Hamiltonian gradient flow on a novel Information-Fisher-Rao space, known as the SHIFR flow, and propose the first ParVI framework that possesses both accelerated position update and dynamical weight adjustment simultaneously, named the General Accelerated Dynamic-Weight Particle-based Variational Inference (GAD-PVI) framework. GAD-PVI is compatible with different dissimilarities between the empirical distribution and the target distribution, as well as different approximation approaches to gradient flow. Moreover, when the appropriate dissimilarity is selected, GAD-PVI is also suitable for obtaining high-quality samples even when analytical scores cannot be obtained. Experiments conducted under both the score-based tasks and sample-based tasks demonstrate the faster convergence and reduced approximation error of GAD-PVI methods over the state-of-the-art.

FUSION3D: Multimodal data fusion for 3D shape reconstruction: A soft-sensing approach

Traditional high-fidelity imaging techniques, such as X-ray computer tomography (CT), excel in capturing intricate shape details through high-resolution two-dimensional (2D) images. However, the extensive cost or impracticality of obtaining X-ray images poses a significant challenge in various applications (e.g., dense metal components in manufacturing). To overcome this limitation, we propose a soft sensing approach, named FUSION3D, aimed at reconstructing three-dimensional (3D) shapes from 2D sliced images and heterogenous process inputs using a novel model based on multi-modal process knowledge. The developed FUSION3D method is built upon a deep learning framework that utilizes continuous normalizing flow to model the complex shape distribution of 3D objects. We propose a principled probabilistic framework to regularize 3D point clouds by modeling them as a distribution of distributions. Specifically, we learn a two-level hierarchy of distributions where the first level is the distribution of shapes, and the second level is the distribution of points given a shape. Our generative regularization model learns each level of the distribution with a continuous normalizing flow. The invertibility of normalizing flows enables the computation of the likelihood during training and allows us to train our model in the variational inference framework. Our methodology is validated through a case study in additive manufacturing, demonstrating its effectiveness as a soft sensing approach for accurate 3D shape reconstruction without relying on high-fidelity imaging techniques like CT during model deployment.

ESVI-GaMM: A fast network intrusion detection approach based on the Bayesian gamma mixture model

As the application of networks permeates various aspects of daily life, maintaining network security has become a crucial challenge. A network intrusion detection system (NIDS) functions as a critical technique for securing cyberspace and has gained considerable attention. Although researchers have made significant progress in developing NIDSs, challenges still exist in high-speed networks with overwhelming network traffic. Existing methods largely focus on improving model detection accuracy and often overlook speed and computational efficiency. This oversight renders most current methods impractical for real-world high-speed network scenarios. To address this issue, we propose an innovative and efficient network intrusion detection algorithm, namely, the Bayesian gamma mixture model (GaMM) classifier. With the recently proposed extended stochastic variational inference (ESVI) framework, we introduce lower-bound approximations to the evidence lower bound (ELBO), namely, the original variational object function. An analytically tractable Bayesian estimation algorithm for a GaMM is derived through stochastic optimization of the obtained lower bound and we validate its performance and computational efficiency on three publicly available datasets (CICMalmem2022, OPCUA, and CICIDS2018). The experimental results indicate that the proposed classifier not only achieves a detection performance comparable to that of other benchmark models but also significantly reduces both the training and detection times.

Bayesian Estimation of Inverted Beta Mixture Models With Extended Stochastic Variational Inference for Positive Vector Classification.

The finite inverted beta mixture model (IBMM) has been proven to be efficient in modeling positive vectors. Under the traditional variational inference framework, the critical challenge in Bayesian estimation of the IBMM is that the computational cost of performing inference with large datasets is prohibitively expensive, which often limits the use of Bayesian approaches to small datasets. An efficient alternative provided by the recently proposed stochastic variational inference (SVI) framework allows for efficient inference on large datasets. Nevertheless, when using the SVI framework to address the non-Gaussian statistical models, the evidence lower bound (ELBO) cannot be explicitly calculated due to the intractable moment computation. Therefore, the algorithm under the SVI framework cannot directly use stochastic optimization to optimize the ELBO, and an analytically tractable solution cannot be derived. To address this problem, we propose an extended version of the SVI framework with more flexibility, namely, the extended SVI (ESVI) framework. This framework can be used in many non-Gaussian statistical models. First, some approximation strategies are applied to further lower the ELBO to avoid intractable moment calculations. Then, stochastic optimization with noisy natural gradients is used to optimize the lower bound. The excellent performance and effectiveness of the proposed method are verified in real data evaluation.

GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework

Particle-based Variational Inference (ParVI) methods approximate the target distribution by iteratively evolving finite weighted particle systems. Recent advances of ParVI methods reveal the benefits of accelerated position update strategies and dynamic weight adjustment approaches. In this paper, we propose the first ParVI framework that possesses both accelerated position update and dynamical weight adjustment simultaneously, named the General Accelerated Dynamic-Weight Particle-based Variational Inference (GAD-PVI) framework. Generally, GAD-PVI simulates the semi-Hamiltonian gradient flow on a novel Information-Fisher-Rao space, which yields an additional decrease on the local functional dissipation. GAD-PVI is compatible with different dissimilarity functionals and associated smoothing approaches under three information metrics. Experiments on both synthetic and real-world data demonstrate the faster convergence and reduced approximation error of GAD-PVI methods over the state-of-the-art.

Kernel Bayesian nonlinear matrix factorization based on variational inference for human–virus protein–protein interaction prediction

Identification of potential human–virus protein–protein interactions (PPIs) contributes to the understanding of the mechanisms of viral infection and to the development of antiviral drugs. Existing computational models often have more hyperparameters that need to be adjusted manually, which limits their computational efficiency and generalization ability. Based on this, this study proposes a kernel Bayesian logistic matrix decomposition model with automatic rank determination, VKBNMF, for the prediction of human–virus PPIs. VKBNMF introduces auxiliary information into the logistic matrix decomposition and sets the prior probabilities of the latent variables to build a Bayesian framework for automatic parameter search. In addition, we construct the variational inference framework of VKBNMF to ensure the solution efficiency. The experimental results show that for the scenarios of paired PPIs, VKBNMF achieves an average AUPR of 0.9101, 0.9316, 0.8727, and 0.9517 on the four benchmark datasets, respectively, and for the scenarios of new human (viral) proteins, VKBNMF still achieves a higher hit rate. The case study also further demonstrated that VKBNMF can be used as an effective tool for the prediction of human–virus PPIs.

Posterior sampling with convolutional neural network-based plug-and-play regularization with applications to poststack seismic inversion

Uncertainty quantification is a crucial component in any geophysical inverse problem, as it provides decision makers with valuable information about the inversion results. Seismic inversion is a notoriously ill-posed inverse problem, due to the band-limited and noisy nature of seismic data; as such, quantifying the uncertainties associated with the ill-posed nature of this inversion process is essential for qualifying the subsequent interpretation and decision-making processes. Selecting appropriate prior information is a crucial — yet nontrivial — step in probabilistic inversion because it influences the ability of sampling-based inference algorithms to provide geologically plausible posterior samples. However, the necessity to encapsulate prior knowledge into a probability distribution can greatly limit our ability to define expressive priors. To address this limitation and following in the footsteps of the plug-and-play (PnP) methodology for deterministic inversion, we develop a regularized variational inference framework that performs posterior inference by implicitly regularizing the Kullback-Leibler divergence loss — a measure of the distance between the approximated and target probabilistic distributions — with a convolutional neural network-based denoiser. We call this new algorithm PnP Stein variational gradient descent and determine its ability to produce high-resolution trustworthy samples that realistically represent subsurface structures. Our method is validated on synthetic and field poststack seismic data.

Reduced-order model-based variational inference with normalizing flows for Bayesian elliptic inverse problems

We propose a reduced-order model-based variational inference method with normalizing flows for Bayesian elliptic inverse problems. The coefficient of the elliptic PDE is represented by a finite number of parameters. We aim to estimate the posterior distributions of these parameters in the framework of variational inference, and the approximation of the posterior distribution is constructed through a normalizing flow. Moreover, as data of inputs and outputs of the forward problem are accumulated during the training of the normalizing flow, we can naturally exploit the low-dimensional intrinsic structure of the forward elliptic PDE using reduced-order models based on these data, in which information of the true posterior is incorporated. We construct a low-dimensional set of data-driven basis functions in the solution space using proper orthogonal decomposition (POD) and train a neural network that maps the parameters to the coefficients of these data-driven basis functions. The surrogate forward map, which is the combination of the reduced-order model and the parameter-to-coefficient neural network, is then applied to the Bayesian elliptic inverse problem, where in each iteration the forward problem is evaluated in the space spanned by the POD basis functions using the surrogate forward map. Thus the inversion will be significantly accelerated as the surrogate forward map is essentially a low-dimensional model. We present numerical examples to show the accuracy and efficiency of the proposed method.

Image restoration for optical synthetic aperture system via variational physics-informed network

Optical synthetic aperture with homogeneous circular sub-mirrors greatly improves the spatial resolution of space telescopes; however, the discrete and sparse characteristics of the sub-mirrors reduce the mid-frequency modulation transfer function (MTF), resulting in blurred images being obtained. In this paper, a method combining variational physics-informed with deep learning is presented, which shows blind image restoration without complex priors. The constraint effect of traditional maximum a posterior (MAP) framework is removed by variational inference framework, which is embedded into Variational Physics-informed Network (VPIN) to optimize neural network training. Residual dense blocks (RDBs) construction is contributed to image feature extraction. Networks with SSIM-corrected loss functions can be trained at the feature level to help with convergence. When SNR = 30 dB, the PSNR of Golay-6 remote sensing test set increases from 20.16 dB to 23.90 dB, SSIM is from 0.610 to 0.842, and MS-SSIM is from 0.930 to 0.955.

Hyperspectral anomaly detection based on variational background inference and generative adversarial network

Hyperspectral anomaly detection is aimed at detecting targets with significant spectral differences from their surroundings. Recently, deep generative models have been applied to anomaly detections, while the existing generative adversarial network (GAN)-based methods have difficulty in accurately modeling the background and achieving spectrum reconstruction. In this article, a hyperspectral anomaly detection network based on variational background inference and generative adversarial framework (VBIGAN-AD) is proposed. The proposed VBIGAN model can learn the background distribution characteristics of HSIs and enhance the detection performance by the use of reconstruction errors. Specifically, the VBIGAN framework consists of sample and latent GANs, which establishes the relationship between data samples and latent samples through two sub-networks to capture the data distribution. Furthermore, the variational inference method is introduced and the hyperspectral background distribution can be converged to a multivariate normal distribution. To accurately learn the background distribution characteristics and reconstruct the background spectra, the coupling loss is conducted by enforcing feature match in the two discriminators on the basis of composite loss, and the results show that the additional loss can promote the detection performance. As a result, the reconstruction errors generated by the VBIGAN-AD method is utilized to detect abnormal targets. The experiments conducted on five datasets proved the robustness and applicability of the proposed VBIGAN-AD method.

Neural extended Kalman filters for learning and predicting dynamics of structural systems

Accurate structural response prediction forms a main driver for structural health monitoring and control applications. This often requires the proposed model to adequately capture the underlying dynamics of complex structural systems. In this work, we utilize a learnable extended Kalman filter (EKF), named the neural extended Kalman filter (neural EKF) throughout this article, for learning the latent evolution dynamics of complex physical systems. The neural EKF is a generalized version of the conventional EKF, where the modeling of process dynamics and sensory observations can be parameterized by neural networks, therefore learned by end-to-end training. The method is implemented under the variational inference framework with the EKF conducting inference from sensing measurements. Typically, conventional variational inference models are parameterized by neural networks independent of the latent dynamics models. This characteristic makes the inference and reconstruction accuracy weakly based on the dynamics models and renders the associated training inadequate. In this work, we show that the structure imposed by the Neural EKF is beneficial to the learning process. We demonstrate the efficacy of the framework on both simulated and real-world structural monitoring datasets, with the results indicating significant predictive capabilities of the proposed scheme.

Joint Noise Model Learning and Source Localization With Unknown Transmit Power

Instead of assuming the accurate noise model and transmit power, we address the received signal strength (RSS) localization problem under a more practical assumption that the exact noise model and transmit power are unavailable. In this letter, we propose to use a universal mixture of Gaussian (MoG) distribution to model the unknown noise due to the fact that MoG distribution can adaptively fit any distribution by adjusting the involved parameters. We propose a novel variational inference framework to achieve joint automatical noise model learning and target localization with unknown transmit power. Numerical experiments are conducted to prove the robustness and effectiveness of the proposed algorithm.

Tensor train factorization under noisy and incomplete data with automatic rank estimation

As a powerful tool in analyzing multi-dimensional data, tensor train (TT) decomposition shows superior performance compared to other tensor decomposition formats. Existing TT decomposition methods, however, either easily overfit with noise, or require substantial fine-tuning to strike a balance between recovery accuracy and model complexity. To avoid the above shortcomings, this paper treats the TT decomposition in a fully Bayesian perspective, which includes automatic TT rank determination and noise power estimation. Theoretical justification on adopting the Gaussian-product-Gamma priors for inducing sparsity on the slices of the TT cores is provided, thus allowing the model complexity to be automatically determined even when the observed tensor data is noisy and contains many missing values. Furthermore, using the variational inference framework, an effective learning algorithm on the probabilistic model parameters is derived. Simulations on synthetic data demonstrate that the proposed algorithm accurately recovers the underlying TT structure from incomplete noisy observations. Further experiments on image and video data also show its superior performance to other existing TT decomposition algorithms.

DragNet: Learning-based deformable registration for realistic cardiac MR sequence generation from a single frame.

Deformable image registration (DIR) can be used to track cardiac motion. Conventional DIR algorithms aim to establish a dense and non-linear correspondence between independent pairs of images. They are, nevertheless, computationally intensive and do not consider temporal dependencies to regulate the estimated motion in a cardiac cycle. In this paper, leveraging deep learning methods, we formulate a novel hierarchical probabilistic model, termed DragNet, for fast and reliable spatio-temporal registration in cine cardiac magnetic resonance (CMR) images and for generating synthetic heart motion sequences. DragNet is a variational inference framework, which takes an image from the sequence in combination with the hidden states of a recurrent neural network (RNN) as inputs to an inference network per time step. As part of this framework, we condition the prior probability of the latent variables on the hidden states of the RNN utilised to capture temporal dependencies. We further condition the posterior of the motion field on a latent variable from hierarchy and features from the moving image. Subsequently, the RNN updates the hidden state variables based on the feature maps of the fixed image and the latent variables. Different from traditional methods, DragNet performs registration on unseen sequences in a forward pass, which significantly expedites the registration process. Besides, DragNet enables generating a large number of realistic synthetic image sequences given only one frame, where the corresponding deformations are also retrieved. The probabilistic framework allows for computing spatio-temporal uncertainties in the estimated motion fields. Our results show that DragNet performance is comparable with state-of-the-art methods in terms of registration accuracy, with the advantage of offering analytical pixel-wise motion uncertainty estimation across a cardiac cycle and being a motion generator. We will make our code publicly available.

Ensemble kalman variational objective: a variational inference framework for sequential variational auto-encoders

Ensemble kalman variational objective: a variational inference framework for sequential variational auto-encoders

Two-level Bayesian interaction analysis for survival data incorporating pathway information.

Genetic interactions play an important role in the progression of complex diseases, providing explanation of variations in disease phenotype missed by main genetic effects. Comparatively, there are fewer studies on survival time, given its challenging characteristics such as censoring. In recent biomedical research, two-level analysis of both genes and their involved pathways has received much attention and been demonstrated as more effective than single-level analysis. However, such analysis is usually limited to main effects. Pathways are not isolated, and their interactions have also been suggested to have important contributions to the prognosis of complex diseases. In this paper, we develop a novel two-level Bayesian interaction analysis approach for survival data. This approach is the first to conduct the analysis of lower-level gene-gene interactions and higher-level pathway-pathway interactions simultaneously. Significantly advancing from the existing Bayesian studies based on the Markov Chain Monte Carlo (MCMC) technique, we propose a variational inference framework based on the accelerated failure time model with effective priors to accommodate two-level selection as well as censoring. Its computational efficiency is much desirable for high-dimensional interaction analysis. We examine performance of the proposed approach using extensive simulation. The application to TCGA melanoma and lung adenocarcinoma data leads to biologically sensible findings with satisfactory prediction accuracy and selectionstability.

Reconfigurable Intelligent Surface-Aided 6G Massive Access: Coupled Tensor Modeling and Sparse Bayesian Learning

This paper investigates a reconfigurable intelligent surface (RIS)-aided unsourced random access (URA) scheme for the sixth-generation (6G) wireless networks with massive sporadic traffic devices. First of all, this paper proposes a novel joint active device separation (the message recovery of active device) and channel estimation architecture for the RIS-aided URA. Specifically, the RIS passive reflection is optimized before the successful device separation. Then, by associating the data sequences to multiple rank-one tensors and exploiting the angular sparsity of the RIS-BS channel, the detection problem is cast as a high-order coupled tensor decomposition problem without the need of exploiting pilot sequences. However, the inherent coupling among multiple sparse device-RIS channels, together with the unknown number of active devices make the detection problem at hand deviate from the widely-used coupled tensor decomposition format. To overcome this challenge, this paper judiciously devises a probabilistic model that captures both the element-wise sparsity from the angular channel model and the low-rank property due to the sporadic nature of URA. Then, based on such a probabilistic model, a iterative detection algorithm is developed under the framework of sparse variational inference, where each update iteration is obtained in a closed-form and the number of active devices can be automatically estimated for effectively avoiding the overfitting of noise. Extensive simulation results confirm the excellence of the proposed URA algorithm, especially for the case of a large number of reflecting elements for accommodating a significantly large number of devices.

Dirichlet Process Mixture of Generalized Inverted Dirichlet Distributions for Positive Vector Data With Extended Variational Inference.

A Bayesian nonparametric approach for estimation of a Dirichlet process (DP) mixture of generalized inverted Dirichlet distributions [i.e., an infinite generalized inverted Dirichlet mixture model (InGIDMM)] has been proposed. The generalized inverted Dirichlet distribution has been proven to be efficient in modeling the vectors that contain only positive elements. Under the classical variational inference (VI) framework, the key challenge in the Bayesian estimation of InGIDMM is that the expectation of the joint distribution of data and variables cannot be explicitly calculated. Therefore, numerical methods are usually applied to simulate the optimal posterior distributions. With the recently proposed extended VI (EVI) framework, we introduce lower bound approximations to the original variational objective function in the VI framework such that an analytically tractable solution can be derived. Hence, the problem in numerical simulation has been overcome. By applying the DP mixture technique, an InGIDMM can automatically determine the number of mixture components from the observed data. Moreover, the DP mixture model with an infinite number of mixture components also avoids the problems of underfitting and overfitting. The performance of the proposed approach is demonstrated with both synthesized data and real-life data applications.

Uncertainty-aware report generation for chest X-rays by variational topic inference.

Automating report generation for medical imaging promises to minimize labor and aid diagnosis in clinical practice. Deep learning algorithms have recently been shown to be capable of captioning natural photos. However, doing a similar thing for medical data, is difficult due to the variety in reports written by different radiologists with fluctuating levels of knowledge and experience. Current methods for automatic report generation tend to merely copy one of the training samples in the created report. To tackle this issue, we propose variational topic inference, a probabilistic approach for automatic chest X-ray report generation. Specifically, we introduce a probabilistic latent variable model where a latent variable defines a single topic. The topics are inferred in a conditional variational inference framework by aligning vision and language modalities in a latent space, with each topic governing the generation of one sentence in the report. We further adopt a visual attention module that enables the model to attend to different locations in the image while generating the descriptions. We conduct extensive experiments on two benchmarks, namely Indiana U. Chest X-rays and MIMIC-CXR. The results demonstrate that our proposed variational topic inference method can generate reports with novel sentence structure, rather than mere copies of reports used in training, while still achieving comparable performance to state-of-the-art methods in terms of standard language generation criteria.

Correlated Chained Gaussian Processes for Modelling Citizens Mobility Using a Zero-Inflated Poisson Likelihood

Modelling the mobility of people in a city depends on counting data with inherent problems of overdispersion. Such dispersion issues are caused by massive amounts of data with zero values. Though traditional machine learning models have been used to overcome said problems, they lack the ability to appropriately model the spatio-temporal correlations in data. To improve the modelling of such spatio-temporal correlations, in this work we propose to model the citizens mobility, for the Chinese city of Guangzhou, by means of a Zero-inflated Poisson likelihood in conjunction with Gaussian process priors generated from convolution processes. We follow the idea of chaining the likelihood’s parameters to latent functions drawn from Gaussian process priors; this way allowing a higher flexibility to model heteroscedasticity. Additionally, we derive a stochastic variational inference framework that allow us to use two types of convolution process models in the context of large datasets: correlated chained Gaussian processes with a convolution processes model, and correlated chained Gaussian processes with variational inducing kernels. We present quantitative and qualitative results comparing the performance between Negative Binomial and Zero-inflated Poisson likelihoods, both in combination with three types of Gaussian process priors: a linear model of coregionalisation, and our two proposed methods based on a convolution processes model and variational inducing kernels.