Variational Inference Technique Research Articles

Leveraging normalizing flows for orbital-free density functional theory

Abstract Orbital-free density functional theory (OF-DFT) for real-space systems has historically depended on Lagrange optimization techniques, primarily due to the inability of previously proposed electron density approaches to ensure the normalization constraint. This study illustrates how leveraging contemporary generative models, notably normalizing flows (NFs), can surmount this challenge. We develop a Lagrangian-free optimization framework by employing these machine learning models for the electron density. This diverse approach also integrates cutting-edge variational inference techniques and equivariant deep learning models, offering an innovative reformulation to the OF-DFT problem. We demonstrate the versatility of our framework by simulating a one-dimensional diatomic system, LiH, and comprehensive simulations of hydrogen, lithium hydride, water, and four hydrocarbon molecules. The inherent flexibility of NFs facilitates initialization with promolecular densities, markedly enhancing the efficiency of the optimization process.

Variational image registration with learned prior using multi-stage VAEs

Variational Autoencoders (VAEs) are an efficient variational inference technique coupled with the generated network. Due to the uncertainty provided by variational inference, VAEs have been applied in medical image registration. However, a critical problem in VAEs is that the simple prior cannot provide suitable regularization, which leads to the mismatch between the variational posterior and prior. An optimal prior can close the gap between the evidence’s real and variational posterior. In this paper, we propose a multi-stage VAE to learn the optimal prior, which is the aggregated posterior. A lightweight VAE is used to generate the aggregated posterior as a whole. It is an effective way to estimate the distribution of the high-dimensional aggregated posterior that commonly exists in medical image registration based on VAEs. A factorized telescoping classifier is trained to estimate the density ratio of a simple given prior and aggregated posterior, aiming to calculate the KL divergence between the variational and aggregated posterior more accurately. We analyze the KL divergence and find that the finer the factorization, the smaller the KL divergence is. However, too fine a partition is not conducive to registration accuracy. Moreover, the diagonal hypothesis of the variational posterior’s covariance ignores the relationship between latent variables in image registration. To address this issue, we learn a covariance matrix with low-rank information to enable correlations with each dimension of the variational posterior. The covariance matrix is further used as a measure to reduce the uncertainty of deformation fields. Experimental results on four public medical image datasets demonstrate that our proposed method outperforms other methods in negative log-likelihood (NLL) and achieves better registration accuracy.

Generative Calibration of Inaccurate Annotation for Label Distribution Learning

Label distribution learning (LDL) is an effective learning paradigm for handling label ambiguity. When applying LDL, it typically requires datasets annotated with label distributions. However, obtaining supervised data for LDL is a challenging task. Due to the randomness of label annotation, the annotator can produce inaccurate annotation results for the instance, affecting the accuracy and generalization ability of the LDL model. To address this problem, we propose a generative approach to calibrate the inaccurate annotation for LDL using variational inference techniques. Specifically, we assume that instances with similar features share latent similar label distributions. The feature vectors and label distributions are generated by Gaussian mixture and Dirichlet mixture, respectively. The relationship between them is established through a shared categorical variable, which effectively utilizes the label distribution of instances with similar features, and achieves a more accurate label distribution through the generative approach. Furthermore, we use a confusion matrix to model the factors that contribute to the inaccuracy during the annotation process, which captures the relationship between label distributions and inaccurate label distributions. Finally, the label distribution is used to calibrate the available information in the noisy dataset to obtain the ground-truth label distribution.

Variational Hierarchical Mixtures for Probabilistic Learning of Inverse Dynamics.

Well-calibrated probabilistic regression models are a crucial learning component in robotics applications as datasets grow rapidly and tasks become more complex. Unfortunately, classical regression models are usually either probabilistic kernel machines with a flexible structure that does not scale gracefully with data or deterministic and vastly scalable automata, albeit with a restrictive parametric form and poor regularization. In this paper, we consider a probabilistic hierarchical modeling paradigm that combines the benefits of both worlds to deliver computationally efficient representations with inherent complexity regularization. The presented approaches are probabilistic interpretations of local regression techniques that approximate nonlinear functions through a set of local linear or polynomial units. Importantly, we rely on principles from Bayesian nonparametrics to formulate flexible models that adapt their complexity to the data and can potentially encompass an infinite number of components. We derive two efficient variational inference techniques to learn these representations and highlight the advantages of hierarchical infinite local regression models, such as dealing with non-smooth functions, mitigating catastrophic forgetting, and enabling parameter sharing and fast predictions. Finally, we validate this approach on large inverse dynamics datasets and test the learned models in real-world control scenarios.

A Design Methodology for Fault-Tolerant Neuromorphic Computing Using Bayesian Neural Network.

Memristor crossbar arrays are a promising platform for neuromorphic computing. In practical scenarios, the synapse weights represented by the memristors for the underlying system are subject to process variations, in which the programmed weight when read out for inference is no longer deterministic but a stochastic distribution. It is therefore highly desired to learn the weight distribution accounting for process variations, to ensure the same inference performance in memristor crossbar arrays as the design value. In this paper, we introduce a design methodology for fault-tolerant neuromorphic computing using a Bayesian neural network, which combines the variational Bayesian inference technique with a fault-aware variational posterior distribution. The proposed framework based on Bayesian inference incorporates the impacts of memristor deviations into algorithmic training, where the weight distributions of neural networks are optimized to accommodate uncertainties and minimize inference degradation. The experimental results confirm the capability of the proposed methodology to tolerate both process variations and noise, while achieving more robust computing in memristor crossbar arrays.

Robust Deep Learning for Uplink Channel Estimation in Cellular Network Under Inter-Cell Interference

Deep learning (DL)-based channel estimation has achieved remarkable success. However, most existing works focus on the white Gaussian noise which are inapplicable for cell-edge users under inter-cell interference (ICI). In this paper, we address this issue by proposing a novel DL-based channel estimation solution with a cascaded model-based and model-free deep neural network (DNN) structure. Specifically, the model-based module is designed by the variational Bayesian inference (VBI) technique to suppress the time-varying ICI, and the model-free module is designed by the Denoising Sparse Autoencoder (DSAE) structure to further refine the channel estimation. The proposed DNN is firstly pre-trained by offline supervised training, and various channel statistics are encapsulated in the DNN weights with the assist of a hyper-prior net modelling different sparse priors for different training samples. Then, an online Bayesian learning algorithm is proposed to train the model-based VBI module based on real-time pilot samples to track the online channel statistics. Simulation results verify that the proposed solution outperforms various state-of-the-art baseline schemes in a large SINR range with comparable performance to the estimator with genie-aided channel statistics.

VariFi: Variational Inference for Indoor Pedestrian Localization and Tracking Using IMU and WiFi RSS

Accurate indoor pedestrian localization and tracking are crucial in many practical applications. One efficient yet low-cost sensing scheme is the integration of IMU and WiFi RSS due to the popularity of smart devices and WiFi networks. Many approaches have been proposed to enhance the localization performance. However, they heavily rely on prerequisites including prior knowledge (e.g. map information) and beacon corrections, which degrades the generalization of the approaches and their accuracy in complex environments. To address this issue, in this paper, we propose a novel localization approach named VariFi, which incorporates variational inference techniques to estimate the location of pedestrian. Variational inference is applied in this work, whose inference network can produce accurate estimates as its parameters are optimized in terms of the reconstruction loss and regularization loss in real time. A signal map is constructed to provide a conditional RSS distribution at any given location, which is further applied to generate the reconstruction loss based on the real measurements. Also, a filtering mechanism is designed to reduce local optimum cases in optimization by utilizing the prior estimate and RSS fingerprinting estimate. In addition, VariFi can be further applied to conduct online optimization following the existing localization approaches. We conduct experiments including static localization and trajectory estimation scenarios to validate the performance of our approach. The trajectory estimation results show that our approach outperforms the mainstream approaches in terms of both localization accuracy and robustness, respectively. Furthermore, the combination of existing approaches and VariFi has also been validated effectively in the experiments of two environments, where VariFi has the ability to bring enhanced localization accuracy.

Physics-Informed Sparse Gaussian Process for Probabilistic Stability Analysis of Large-Scale Power System With Dynamic PVs and Loads

This paper proposes a physics-informed sparse Gaussian process (SGP) for probabilistic stability assessment of large-scale power systems in the presence of uncertain dynamic PVs and loads. The differential and algebraic equations considering uncertainties from dynamic PVs and loads are reformulated to a nonlinear mapping relationship that allows the application of SGP. Thanks to the nonparametric characteristic of Gaussian process, the proposed framework does not require distributions of uncertain inputs and this distinguishes it from existing approaches. As the original Gaussian process is not scalable to large-scale systems with high dimensional uncertain inputs, this paper develops the SGP with a stochastic variational inference technique. It leads to approximately two orders of complex reduction. A data pre-processing step is also introduced to tackle the coexistence of stable and unstable cases by sample clustering and constructing separate SGPs. The probabilistic transient stability index is analyzed to assess system stability under different uncertain dynamics loads and PVs. Comparisons are performed with the sampling-based, the polynomial chaos expansion-based, and traditional Gaussian process-based methods on the modified IEEE 118-bus and Texas 2000-bus systems under various scenarios, including different levels of uncertainties and the existence of nonlinear correlations among dynamic PVs. The impacts of data quality and quantity issues are also investigated. It is shown that the proposed SGP achieves significantly improved computational efficiency while maintaining high accuracy with a limited number of data.

A variational Bayesian inference technique for model updating of structural systems with unknown noise statistics

Dynamic models of structural and mechanical systems can be updated to match the measured data through a Bayesian inference process. However, the performance of classical (non-adaptive) Bayesian model updating approaches decreases significantly when the pre-assumed statistical characteristics of the model prediction error are violated. To overcome this issue, this paper presents an adaptive recursive variational Bayesian approach to estimate the statistical characteristics of the prediction error jointly with the unknown model parameters. This approach improves the accuracy and robustness of model updating by including the estimation of model prediction error. The performance of this approach is demonstrated using numerically simulated data obtained from a structural frame with material non-linearity under earthquake excitation. Results show that in the presence of non-stationary noise/error, the non-adaptive approach fails to estimate unknown model parameters, whereas the proposed approach can accurately estimate them.

Fast and accurate Bayesian polygenic risk modeling with variational inference

The advent of large-scale genome-wide association studies (GWASs) has motivated the development of statistical methods for phenotype prediction with single-nucleotide polymorphism (SNP) array data. These polygenic risk score (PRS) methods use a multiple linear regression framework to infer joint effect sizes of all genetic variants on the trait. Among the subset of PRS methods that operate on GWAS summary statistics, sparse Bayesian methods have shown competitive predictive ability. However, most existing Bayesian approaches employ Markov chain Monte Carlo (MCMC) algorithms, which are computationally inefficient and do not scale favorably to higher dimensions, for posterior inference. Here, we introduce variational inference of polygenic risk scores (VIPRS), a Bayesian summary statistics-based PRS method that utilizes variational inference techniques to approximate the posterior distribution for the effect sizes. Our experiments with 36 simulation configurations and 12 real phenotypes from the UK Biobank dataset demonstrated that VIPRS is consistently competitive with the state-of-the-art in prediction accuracy while being more than twice as fast as popular MCMC-based approaches. This performance advantage is robust across a variety of genetic architectures, SNP heritabilities, and independent GWAS cohorts. In addition to its competitive accuracy on the "White British" samples, VIPRS showed improved transferability when applied to other ethnic groups, with up to 1.7-fold increase in R2 among individuals of Nigerian ancestry for low-density lipoprotein (LDL) cholesterol. To illustrate its scalability, we applied VIPRS to a dataset of 9.6 million genetic markers, which conferred further improvements in prediction accuracy for highly polygenic traits, such as height.

Tuning-Free Bayesian Estimation Algorithms for Faulty Sensor Signals in State-Space

Sensors provide insights into the industrial processes, while misleading sensor outputs may result in inappropriate decisions or even catastrophic accidents. In this article, the Bayesian estimation algorithms are developed to estimate unforeseen signals in sensor outputs without tuning. The optimal Bayesian estimation method is first derived by incorporating a Gaussian distribution specifying potential unmodeled dynamics into the measurement equation. Since its performance depends on tuning parameters, an iterative Bayesian estimation algorithm is developed using the variational inference technique. Specifically, an inverse Wishart distribution is introduced to describe the predicted covariance of abnormal signals. We then estimate it together with the other independent Gaussian distributions to conditionally approximate the joint posterior distribution, by which the effects of tuning parameters can be replaced adaptively. Testing the proposed algorithms through a simulated electromechanical brake model and a real experimental system shows that the proposed algorithm can satisfactorily estimate additive sensor faults online and services as a sensor monitor that simultaneously provides the locations and magnitudes of faulty signals without tuning.

Bayesian Deep-Learning-Based Prognostic Model for Equipment Without Label Data Related to Lifetime

Deep learning has become a promising tool for processing the massive data and attracted an increasing attention in the fields of degradation modeling and remaining useful life (RUL) prediction. The existing deep-learning-based methods are generally faced with the two aspects of problems. On the one hand, the prediction results are represented by the point estimates instead of the probabilistic distribution and, thus, the prognostic uncertainty in RUL prediction cannot be characterized. On the other hand, there exist plenty of the engineering assets without the label data related to lifetime, posing a great challenge for training the deep learning network. Toward this end, we propose a prognostic model under the framework of Bayesian deep learning for equipment lacking the label data related to lifetime. First, the monitoring data of the historical equipment and the historical data of field equipment in the database are preprocessed to generate the samples regarding degradation information as a label. Second, the bidirectional recurrent neural network (RNN) is employed as the candidate network for the advantages in handling the sequential monitoring data. On the basis of this, the idea of Bayesian deep learning is incorporated into the bidirectional RNN; thus, we can characterize the uncertainty of the predicted degradation level at any future time via utilizing the variational inference technique in the Bayesian neural networks. Furthermore, the failure probability for the concerned equipment at any time can be determined, by which the degradation uncertainty can be converted into the RUL uncertainty from the point of the reliability theory. Finally, we provide the case study associated with lithium-ion batteries to verify the proposed prognostic model.

A variational inference based learning approach for decentralized building fire estimation

Real-time estimation of the fire status is critical for reliable decision-making in building fire emergencies. Learning-based approaches for fire estimation have achieved great success, benefited from exploration of sensor data. However, existing works are mostly designed for specific building structures and seldom account for false alarms. Such flaws affect their generalization in broader environments and robustness in practical use. In this paper, we propose a decentralized learning-based approach for building fire estimation. The estimation is conducted in a decentralized scheme, where building zones conduct fire estimation respectively. The approach is built on variational inference technique, and can infer the location and dynamic intensity of the fires concurrently. Specifically, the fire estimation problem is interpreted in a latent variable model (LVM), with latent variables for fire intensity features and location features. The solution is then derived from variational inference (VI), and approximated via data learning on a novel deep network (namely FireEst-VAE). The network can produce accurate estimation of the fire intensity, while performing false alarm identification as a bonus effect. We evaluate the decentralized approach on full-scale experimental and simulation-generated data from a multi-compartment residence. The results illustrate that the proposed approach performs well on a wide range of fire scenarios with one or multiple fire locations, while preventing false alarms from common nuisance sources. The decentralized learning-based approach for fire estimation shows great potential in smart firefighting for large-scale buildings. • A decentralized learning approach based on variational inference for fire estimation. • Integrating interpretability from model methods and efficiency from learning methods. • The decentralized scheme benefits edge computing and generalization in practical use. • Promising performances on fire intensity estimation and false alarm identification.

Deep Gaussian Processes for Calibration of Computer Models (with Discussion)

Bayesian calibration of black-box computer models offers an established framework for quantification of uncertainty of model parameters and predictions. Traditional Bayesian calibration involves the emulation of the computer model and an additive model discrepancy term using Gaussian processes; inference is then carried out using Markov chain Monte Carlo. This calibration approach is limited by the poor scalability of Gaussian processes and by the need to specify a sensible covariance function to deal with the complexity of the computer model and the discrepancy. In this work, we propose a novel calibration framework, where these challenges are addressed by means of compositions of Gaussian processes into Deep Gaussian processes and scalable variational inference techniques. Thanks to this formulation, it is possible to obtain a flexible calibration approach, which is easy to implement in development environments featuring automatic differentiation and exploiting GPU-type hardware. We show how our proposal yields a powerful alternative to the state-of-the-art by means of experimental validations on various calibration problems. We conclude the paper by showing how we can carry out adaptive experimental design, and by discussing the identifiability properties of the proposed calibration model.

Characterization of uncertainties and model generalizability for convolutional neural network predictions of uranium ore concentrate morphology

As the capabilities of convolutional neural networks (CNNs) for image classification tasks have advanced, interest in applying deep learning techniques for determining the natural and anthropogenic origins of uranium ore concentrates (UOCs) and other unknown nuclear materials by their surface morphology characteristics has grown. But before CNNs can join the nuclear forensics toolbox along more traditional analytical techniques – such as scanning electron microscopy (SEM), X-ray diffractometry, mass spectrometry, radiation counting, and any number of spectroscopic methods – a deeper understanding of “black box” image classification will be required. This paper explores uncertainty quantification for convolutional neural networks and their ability to generalize to out-of-distribution (OOD) image data sets. For prediction uncertainty, Monte Carlo (MC) dropout and random image crops as variational inference techniques are implemented and characterized. Convolutional neural networks and classifiers using image features from unsupervised vector-quantized variational autoencoders (VQ-VAE) are trained using SEM images of pure, unaged, unmixed uranium ore concentrates considered “unperturbed.” OOD data sets are developed containing perturbations from the training data with respect to the chemical and physical properties of the UOCs or data collection parameters; predictions made on the perturbation sets identify where significant shortcomings exist in the current training data and techniques used to develop models for classifying uranium process history, and provides valuable insights into how datasets and classification models can be improved for better generalizability to out-of-distribution examples.

Collective Variational Inference for Personalized and Generative Physiological Modeling: A Case Study on Hemorrhage Resuscitation.

Individual physiological experiments typically provide useful but incomplete information about a studied physiological process. As a result, inferring the unknown parameters of a physiological model from experimental data is often challenging. The objective of this paper is to propose and illustrate the efficacy of a collective variational inference (C-VI) method, intended to reconcile low-information and heterogeneous data from a collection of experiments to produce robust personalized and generative physiological models. To derive the C-VI method, we utilize a probabilistic graphical model to impose structure on the available physiological data, and algorithmically characterize the graphical model using variational Bayesian inference techniques. To illustrate the efficacy of the C-VI method, we apply it to a case study on the mathematical modeling of hemorrhage resuscitation. In the context of hemorrhage resuscitation modeling, the C-VI method could reconcile heterogeneous combinations of hematocrit, cardiac output, and blood pressure data across multiple experiments to obtain (i) robust personalized models along with associated measures of uncertainty and signal quality, and (ii) a generative model capable of reproducing the physiological behavior of the population. The C-VI method facilitates the personalized and generative modeling of physiological processes in the presence of low-information and heterogeneous data. The resulting models provide a solid basis for the development and testing of interpretable physiological monitoring, decision-support, and closed-loop control algorithms.

Variational Label Enhancement.

Multi-label learning focuses on the ambiguity at the label side, i.e., one instance is associated with multiple class labels, where the logical labels are always adopted to partition class labels into relevant labels and irrelevant labels rigidly. However, the relevance or irrelevance of each label corresponding to one instance is essentially relative in real-world tasks and the label distribution is more fine-grained than the logical labels by denoting one instance with a certain number of the description degrees of all class labels. As the label distribution is not explicitly available in most training sets, a process named label enhancement emerges to recover the label distributions in training datasets. By inducing the generative model of the label distribution and adopting the variational inference technique, the approximate posterior density of the label distributions should maximize the variational lower bound. Following the above consideration, LEVI is proposed to recover the label distributions from the training examples. In addition, the multi-label predictive model is induced for multi-label learning by leveraging the recovered label distributions along with a specialized objective function. The recovery experiments on fourteen label distribution datasets and the predictive experiments on fourteen multi-label learning datasets validate the advantage of our approach over the state-of-the-art approaches.

Predicting plastic anisotropy using crystal plasticity and Bayesian neural network surrogate models

This work presents an efficient data-driven protocol to accurately predict plastic anisotropy from initial crystallographic texture. In this work, we integrated feed forward neural networks with Variational Bayesian Inference techniques to establish an accurate low-computational cost surrogate model capable of predicting the anisotropic constants based on the texture of the polycrystalline material with quantifiable uncertainty. The developed model was trained on the results of 54,480 crystal plasticity simulations. The performed simulations parametrized Hill’s anisotropic yield model for single crystals and polycrystalline textures, which were robustly represented using generalized spherical harmonics (GSH). Subsequently, the GSH-based representation of the different textures was linked to its corresponding Hill’s anisotropic coefficients using a variational Bayesian neural network. The efficacy and accuracy of the developed surrogate model were critically validated with the results of 20,000 new textures. The predictions from the Bayesian neural network model showed excellent agreement with results obtained from experiments and high-fidelity crystal plasticity finite element simulations.

Geometric Variational Inference.

Efficiently accessing the information contained in non-linear and high dimensional probability distributions remains a core challenge in modern statistics. Traditionally, estimators that go beyond point estimates are either categorized as Variational Inference (VI) or Markov-Chain Monte-Carlo (MCMC) techniques. While MCMC methods that utilize the geometric properties of continuous probability distributions to increase their efficiency have been proposed, VI methods rarely use the geometry. This work aims to fill this gap and proposes geometric Variational Inference (geoVI), a method based on Riemannian geometry and the Fisher information metric. It is used to construct a coordinate transformation that relates the Riemannian manifold associated with the metric to Euclidean space. The distribution, expressed in the coordinate system induced by the transformation, takes a particularly simple form that allows for an accurate variational approximation by a normal distribution. Furthermore, the algorithmic structure allows for an efficient implementation of geoVI which is demonstrated on multiple examples, ranging from low-dimensional illustrative ones to non-linear, hierarchical Bayesian inverse problems in thousands of dimensions.

Approximate Bayesian Uncertainties on Deep Learning Dynamical Mass Estimates of Galaxy Clusters

Abstract We study methods for reconstructing Bayesian uncertainties on dynamical mass estimates of galaxy clusters using convolutional neural networks (CNNs). We discuss the statistical background of approximate Bayesian neural networks and demonstrate how variational inference techniques can be used to perform computationally tractable posterior estimation for a variety of deep neural architectures. We explore how various model designs and statistical assumptions impact prediction accuracy and uncertainty reconstruction in the context of cluster mass estimation. We measure the quality of our model posterior recovery using a mock cluster observation catalog derived from the MultiDark simulation and UniverseMachine catalog. We show that approximate Bayesian CNNs produce highly accurate dynamical cluster mass posteriors. These model posteriors are log-normal in cluster mass and recover 68% and 90% confidence intervals to within 1% of their measured value. We note how this rigorous modeling of dynamical mass posteriors is necessary for using cluster abundance measurements to constrain cosmological parameters.