High-dimensional Probability Distributions Research Articles

Mean force based temperature accelerated sliced sampling: Efficient reconstruction of high dimensional free energy landscapes.

Temperature accelerated sliced sampling (TASS) is an efficient method to compute high dimensional free energy landscapes. The original TASS method employs the weighted histogram analysis method (WHAM) which is an iterative post-processing to reweight and stitch high dimensional probability distributions in sliced windows that are obtained in the presence of restraining biases. The WHAM necessitates that TASS windows lie close to each other for proper overlap of distributions and span the collective variable space of interest. On the other hand, increase in number of TASS windows implies more number of simulations, and thus it affects the efficiency of the method. To overcome this problem, we propose herein a new mean-force (MF) based reweighting scheme called TASS-MF, which enables accurate computation with a fewer number of windows devoid of the WHAM post-processing. Application of the technique is demonstrated for alanine di- and tripeptides in vacuo to compute their two- and four-dimensional free energy landscapes, the latter of which is formidable in conventional umbrella sampling and metadynamics. The landscapes are computed within a kcal mol-1 accuracy, ensuring a safe usage for broad applications in computational chemistry.

Geometric Variational Inference.

Efficiently accessing the information contained in non-linear and high dimensional probability distributions remains a core challenge in modern statistics. Traditionally, estimators that go beyond point estimates are either categorized as Variational Inference (VI) or Markov-Chain Monte-Carlo (MCMC) techniques. While MCMC methods that utilize the geometric properties of continuous probability distributions to increase their efficiency have been proposed, VI methods rarely use the geometry. This work aims to fill this gap and proposes geometric Variational Inference (geoVI), a method based on Riemannian geometry and the Fisher information metric. It is used to construct a coordinate transformation that relates the Riemannian manifold associated with the metric to Euclidean space. The distribution, expressed in the coordinate system induced by the transformation, takes a particularly simple form that allows for an accurate variational approximation by a normal distribution. Furthermore, the algorithmic structure allows for an efficient implementation of geoVI which is demonstrated on multiple examples, ranging from low-dimensional illustrative ones to non-linear, hierarchical Bayesian inverse problems in thousands of dimensions.

Dynamical mass inference of galaxy clusters with neural flows

ABSTRACT We present an algorithm for inferring the dynamical mass of galaxy clusters directly from their respective phase-space distributions, that is, the observed line-of-sight velocities and projected distances of galaxies from the cluster centre. Our method employs normalizing flows, a deep neural network capable of learning arbitrary high-dimensional probability distributions, and inherently accounts, to an adequate extent, for the presence of interloper galaxies which are not bounded to a given cluster, the primary contaminant of dynamical mass measurements. We validate and showcase the performance of our neural flow approach to robustly infer the dynamical mass of clusters from a realistic mock cluster catalogue. A key aspect of our novel algorithm is that it yields the probability density function of the mass of a particular cluster, thereby providing a principled way of quantifying uncertainties, in contrast to conventional machine learning (ML) approaches. The neural network mass predictions, when applied to a contaminated catalogue with interlopers, have a mean overall logarithmic residual scatter of 0.028 dex, with a lognormal scatter of 0.126 dex, which goes down to 0.089 dex for clusters in the intermediate- to high-mass range. This is an improvement by nearly a factor of 4 relative to the classical cluster mass scaling relation with the velocity dispersion, and outperforms recently proposed ML approaches. We also apply our neural flow mass estimator to a compilation of galaxy observations of some well-studied clusters with robust dynamical mass estimates, further substantiating the efficacy of our algorithm.

Controlled sequential Monte Carlo

Sequential Monte Carlo methods, also known as particle methods, are a popular set of techniques for approximating high-dimensional probability distributions and their normalizing constants. These methods have found numerous applications in statistics and related fields; e.g. for inference in non-linear non-Gaussian state space models, and in complex static models. Like many Monte Carlo sampling schemes, they rely on proposal distributions which crucially impact their performance. We introduce here a class of controlled sequential Monte Carlo algorithms, where the proposal distributions are determined by approximating the solution to an associated optimal control problem using an iterative scheme. This method builds upon a number of existing algorithms in econometrics, physics, and statistics for inference in state space models, and generalizes these methods so as to accommodate complex static models. We provide a theoretical analysis concerning the fluctuation and stability of this methodology that also provides insight into the properties of related algorithms. We demonstrate significant gains over state-of-the-art methods at a fixed computational complexity on a variety of applications.

Neural networks of different species, brain areas and states can be characterized by the probability polling state.

Cortical networks are complex systems of a great many interconnected neurons that operate from collective dynamical states. To understand how cortical neural networks function, it is important to identify their common dynamical operating states from the probabilistic viewpoint. Probabilistic characteristics of these operating states often underlie network functions. Here, using multi-electrode data from three separate experiments, we identify and characterize a cortical operating state (the "probability polling" or "p-polling" state), common across mouse and monkey with different behaviors. If the interaction among neurons is weak, the p-polling state provides a quantitative understanding of how the high dimensional probability distribution of firing patterns can be obtained by the low-order maximum entropy formulation, effectively utilizing a low dimensional stimulus-coding structure. These results show evidence for generality of the p-polling state and in certain situations its advantage of providing a mathematical validation for the low-order maximum entropy principle as a coding strategy.

Thermodynamic inference of data manifolds

The Gibbs-Boltzmann distribution offers a physically interpretable way to massively reduce the dimensionality of high dimensional probability distributions where the extensive variables are `features' and the intensive variables are `descriptors'. However, not all probability distributions can be modeled using the Gibbs-Boltzmann form. Here, we present TMI: TMI, {\bf T}hermodynamic {\bf M}anifold {\bf I}nference; a thermodynamic approach to approximate a collection of arbitrary distributions. TMI simultaneously learns from data intensive and extensive variables and achieves dimensionality reduction through a multiplicative, positive valued, and interpretable decomposition of the data. Importantly, the reduced dimensional space of intensive parameters is not homogeneous. The Gibbs-Boltzmann distribution defines an analytically tractable Riemannian metric on the space of intensive variables allowing us to calculate geodesics and volume elements. We discuss the applications of TMI with multiple real and artificial data sets. Possible extensions are discussed as well.

Solving Bayesian inverse problems from the perspective of deep generative networks

Deep generative networks have achieved great success in high dimensional density approximation, especially for applications in natural images and language. In this paper, we investigate their approximation capability in capturing the posterior distribution in Bayesian inverse problems by learning a transport map. Because only the unnormalized density of the posterior is available, training methods that learn from posterior samples, such as variational autoencoders and generative adversarial networks, are not applicable in our setting. We propose a class of network training methods that can be combined with sample-based Bayesian inference algorithms, such as various MCMC algorithms, ensemble Kalman filter and Stein variational gradient descent. Our experiment results show the pros and cons of deep generative networks in Bayesian inverse problems. They also reveal the potential of our proposed methodology in capturing high dimensional probability distributions.

Adversarial quantum circuit learning for pure state approximation

Adversarial learning is one of the most successful approaches to modeling high-dimensional probability distributions from data. The quantum computing community has recently begun to generalize this idea and to look for potential applications. In this work, we derive an adversarial algorithm for the problem of approximating an unknown quantum pure state. Although this could be done on universal quantum computers, the adversarial formulation enables us to execute the algorithm on near-term quantum computers. Two parametrized circuits are optimized in tandem: one tries to approximate the target state, the other tries to distinguish between target and approximated state. Supported by numerical simulations, we show that resilient backpropagation algorithms perform remarkably well in optimizing the two circuits. We use the bipartite entanglement entropy to design an efficient heuristic for the stopping criterion. Our approach may find application in quantum state tomography.

Scalable Importance Tempering and Bayesian Variable Selection

SummaryWe propose a Monte Carlo algorithm to sample from high dimensional probability distributions that combines Markov chain Monte Carlo and importance sampling. We provide a careful theoretical analysis, including guarantees on robustness to high dimensionality, explicit comparison with standard Markov chain Monte Carlo methods and illustrations of the potential improvements in efficiency. Simple and concrete intuition is provided for when the novel scheme is expected to outperform standard schemes. When applied to Bayesian variable-selection problems, the novel algorithm is orders of magnitude more efficient than available alternative sampling schemes and enables fast and reliable fully Bayesian inferences with tens of thousand regressors.

Joint regression modeling for missing categorical covariates in generalized linear models

Missing covariates data is a common issue in generalized linear models (GLMs). A model-based procedure arising from properly specifying joint models for both the partially observed covariates and the corresponding missing indicator variables represents a sound and flexible methodology, which lends itself to maximum likelihood estimation as the likelihood function is available in computable form. In this paper, a novel model-based methodology is proposed for the regression analysis of GLMs when the partially observed covariates are categorical. Pair-copula constructions are used as graphical tools in order to facilitate the specification of the high-dimensional probability distributions of the underlying missingness components. The model parameters are estimated by maximizing the weighted log-likelihood function by using an EM algorithm. In order to compare the performance of the proposed methodology with other well-established approaches, which include complete-cases and multiple imputation, several simulation experiments of Binomial, Poisson and Normal regressions are carried out under both missing at random and non-missing at random mechanisms scenarios. The methods are illustrated by modeling data from a stage III melanoma clinical trial. The results show that the methodology is rather robust and flexible, representing a competitive alternative to traditional techniques.

A BIRTH AND DEATH PROCESS FOR BAYESIAN NETWORK STRUCTURE INFERENCE

Bayesian networks are convenient graphical expressions for high-dimensional probability distributions which represent complex relationships between a large number of random variables. They have been employed extensively in areas such as bioinformatics, artificial intelligence, diagnosis, and risk management. The recovery of the structure of a network from data is of prime importance for the purposes of modeling, analysis, and prediction. There has been a great deal of interest in recent years in the NP-hard problem of learning the structure of a Bayesian network from observed data. Typically, one assigns a score to various structures and the search becomes an optimization problem that can be approached with either deterministic or stochastic methods. In this paper, we introduce a new search strategy where one walks through the space of graphs by modeling the appearance and disappearance of edges as a birth and death process. We compare our novel approach with the popular Metropolis–Hastings search strategy and give empirical evidence that the birth and death process has superior mixing properties.

S2JSD-LSH: A Locality-Sensitive Hashing Schema for Probability Distributions

To compare the similarity of probability distributions, the information-theoretically motivated metrics like Kullback-Leibler divergence (KL) and Jensen-Shannon divergence (JSD) are often more reasonable compared with metrics for vectors like Euclidean and angular distance. However, existing locality-sensitive hashing (LSH) algorithms cannot support the information-theoretically motivated metrics for probability distributions. In this paper, we first introduce a new approximation formula for S2JSD-distance, and then propose a novel LSH scheme adapted to S2JSD-distance for approximate nearest neighbors search in high-dimensional probability distributions. We define the specific hashing functions, and prove their local-sensitivity. Furthermore, extensive empirical evaluations well illustrate the effectiveness of the proposed hashing schema on six public image datasets and two text datasets, in terms of mean Average Precision, Precision@N and Precision-Recall curve.

Inference of Functionally-Relevant N-acetyltransferase Residues Based on Statistical Correlations.

Over evolutionary time, members of a superfamily of homologous proteins sharing a common structural core diverge into subgroups filling various functional niches. At the sequence level, such divergence appears as correlations that arise from residue patterns distinct to each subgroup. Such a superfamily may be viewed as a population of sequences corresponding to a complex, high-dimensional probability distribution. Here we model this distribution as hierarchical interrelated hidden Markov models (hiHMMs), which describe these sequence correlations implicitly. By characterizing such correlations one may hope to obtain information regarding functionally-relevant properties that have thus far evaded detection. To do so, we infer a hiHMM distribution from sequence data using Bayes’ theorem and Markov chain Monte Carlo (MCMC) sampling, which is widely recognized as the most effective approach for characterizing a complex, high dimensional distribution. Other routines then map correlated residue patterns to available structures with a view to hypothesis generation. When applied to N-acetyltransferases, this reveals sequence and structural features indicative of functionally important, yet generally unknown biochemical properties. Even for sets of proteins for which nothing is known beyond unannotated sequences and structures, this can lead to helpful insights. We describe, for example, a putative coenzyme-A-induced-fit substrate binding mechanism mediated by arginine residue switching between salt bridge and π-π stacking interactions. A suite of programs implementing this approach is available (psed.igs.umaryland.edu).

High-dimensional Bayesian inference via the unadjusted Langevin algorithm

We consider in this paper the problem of sampling a high-dimensional probability distribution $\pi$ having a density w.r.t. the Lebesgue measure on $\mathbb{R}^{d}$, known up to a normalization constant $x\mapsto\pi(x)=\mathrm{e}^{-U(x)}/\int_{\mathbb{R}^{d}}\mathrm{e}^{-U(y)}\,\mathrm{d}y$. Such problem naturally occurs for example in Bayesian inference and machine learning. Under the assumption that $U$ is continuously differentiable, $\nabla U$ is globally Lipschitz and $U$ is strongly convex, we obtain non-asymptotic bounds for the convergence to stationarity in Wasserstein distance of order $2$ and total variation distance of the sampling method based on the Euler discretization of the Langevin stochastic differential equation, for both constant and decreasing step sizes. The dependence on the dimension of the state space of these bounds is explicit. The convergence of an appropriately weighted empirical measure is also investigated and bounds for the mean square error and exponential deviation inequality are reported for functions which are measurable and bounded. An illustration to Bayesian inference for binary regression is presented to support our claims.

Fast Hamiltonian Monte Carlo Using GPU Computing

In recent years, the Hamiltonian Monte Carlo (HMC) algorithm has been found to work more efficiently compared to other popular Markov chain Monte Carlo (MCMC) methods (such as random walk Metropolis–Hastings) in generating samples from a high-dimensional probability distribution. HMC has proven more efficient in terms of mixing rates and effective sample size than previous MCMC techniques, but still may not be sufficiently fast for particularly large problems. The use of GPUs promises to push HMC even further greatly increasing the utility of the algorithm. By expressing the computationally intensive portions of HMC (the evaluations of the probability kernel and its gradient) in terms of linear or element-wise operations, HMC can be made highly amenable to the use of graphics processing units (GPUs). A multinomial regression example demonstrates the promise of GPU-based HMC sampling. Using GPU-based memory objects to perform the entire HMC simulation, most of the latency penalties associated with transferring data from main to GPU memory can be avoided. Thus, the proposed computational framework may appear conceptually very simple, but has the potential to be applied to a wide class of hierarchical models relying on HMC sampling. Models whose posterior density and corresponding gradients can be reduced to linear or element-wise operations are amenable to significant speed ups through the use of GPUs. Analyses of datasets that were previously intractable for fully Bayesian approaches due to the prohibitively high computational cost are now feasible using the proposed framework.

Inferring time-varying brain connectivity graph based on a new method for link estimation

ABSTRACTCausal interaction estimation among neuronal groups plays an important role in the assessment of brain functions. These directional relations can be best illustrated by means of graphical modeling which is a mathematical representation of a network. Here, we propose an efficient framework to derive a graphical model for the statistical analysis of multivariate processes from observed time series in a data-driven pipeline to explore the interregional brain interactions. A major part of this analysis is devoted to the graph link estimation, which is a measure capable of dealing with the multivariate analysis obstacles. In this paper, we use the Transfer Entropy (TE) measure and focus on its calculation that requires efficient estimation of high dimensional conditional probability distributions. Our method is based on the simplification of high dimensional parts of the conventional TE definition and especially devoted to the reduction of estimation dimension through searching for the most informative contents of the high dimensional parts. To this end, we exploit the causal Markov properties for time series graphs and prove that only a specified subset of involved variables plays an important role in multivariate TE estimation. We demonstrate the performance of our method for stationary processes using some numerical simulated examples as well as real neurophysiological data.

Comparison Theorems for Gibbs Measures

The Dobrushin comparison theorem is a powerful tool to bound the difference between the marginals of high-dimensional probability distributions in terms of their local specifications. Originally introduced to prove uniqueness and decay of correlations of Gibbs measures, it has been widely used in statistical mechanics as well as in the analysis of algorithms on random fields and interacting Markov chains. However, the classical comparison theorem requires validity of the Dobrushin uniqueness criterion, essentially restricting its applicability in most models to a small subset of the natural parameter space. In this paper we develop generalized Dobrushin comparison theorems in terms of influences between blocks of sites, in the spirit of Dobrushin-Shlosman and Weitz, that substantially extend the range of applicability of the classical comparison theorem. Our proofs are based on the analysis of an associated family of Markov chains. We develop in detail an application of our main results to the analysis of sequential Monte Carlo algorithms for filtering in high dimension.

A probabilistic graphical model approach to stochastic multiscale partial differential equations

We develop a probabilistic graphical model based methodology to efficiently perform uncertainty quantification in the presence of both stochastic input and multiple scales. Both the stochastic input and model responses are treated as random variables in this framework. Their relationships are modeled by graphical models which give explicit factorization of a high-dimensional joint probability distribution. The hyperparameters in the probabilistic model are learned using sequential Monte Carlo (SMC) method, which is superior to standard Markov chain Monte Carlo (MCMC) methods for multi-modal distributions. Finally, we make predictions from the probabilistic graphical model using the belief propagation algorithm. Numerical examples are presented to show the accuracy and efficiency of the predictive capability of the developed graphical model.

Visualization of High-Dimensional Point Clouds Using Their Density Distribution's Topology

We present a novel method to visualize multidimensional point clouds. While conventional visualization techniques, like scatterplot matrices or parallel coordinates, have issues with either overplotting of entities or handling many dimensions, we abstract the data using topological methods before presenting it. We assume the input points to be samples of a random variable with a high-dimensional probability distribution which we approximate using kernel density estimates on a suitably reconstructed mesh. From the resulting scalar field we extract the join tree and present it as a topological landscape, a visualization metaphor that utilizes the human capability of understanding natural terrains. In this landscape, dense clusters of points show up as hills. The nesting of hills indicates the nesting of clusters. We augment the landscape with the data points to allow selection and inspection of single points and point sets. We also present optimizations to make our algorithm applicable to large data sets and to allow interactive adaption of our visualization to the kernel window width used in the density estimation.

Particle Markov Chain Monte Carlo Methods

SummaryMarkov chain Monte Carlo and sequential Monte Carlo methods have emerged as the two main tools to sample from high dimensional probability distributions. Although asymptotic convergence of Markov chain Monte Carlo algorithms is ensured under weak assumptions, the performance of these algorithms is unreliable when the proposal distributions that are used to explore the space are poorly chosen and/or if highly correlated variables are updated independently. We show here how it is possible to build efficient high dimensional proposal distributions by using sequential Monte Carlo methods. This allows us not only to improve over standard Markov chain Monte Carlo schemes but also to make Bayesian inference feasible for a large class of statistical models where this was not previously so. We demonstrate these algorithms on a non-linear state space model and a Lévy-driven stochastic volatility model.