Neural Density Estimation Research Articles

Inference on the Macroscopic Dynamics of Spiking Neurons.

The process of inference on networks of spiking neurons is essential to decipher the underlying mechanisms of brain computation and function. In this study, we conduct inference on parameters and dynamics of a mean-field approximation, simplifying the interactions of neurons. Estimating parameters of this class of generative model allows one to predict the system's dynamics and responses under changing inputs and, indeed, changing parameters. We first assume a set of known state-space equationsand address the problem of inferring the lumped parameters from observed time series. Crucially, we consider this problem in the setting of bistability, random fluctuations in system dynamics, and partial observations, in which some states are hidden. To identify the most efficient estimation or inversion scheme in this particular system identification, we benchmark against state-of-the-art optimization and Bayesian estimation algorithms, highlighting their strengths and weaknesses. Additionally, we explore how well the statistical relationships between parameters are maintained across different scales. We found that deep neural density estimators outperform other algorithms in the inversion scheme, despite potentially resulting in overestimated uncertainty and correlation between parameters. Nevertheless, this issue can be improved by incorporating time-delay embedding. We then eschew the mean-field approximation and employ deep neural ODEs on spiking neurons, illustrating prediction of system dynamics and vector fields from microscopic states. Overall, this study affords an opportunity to predict brain dynamics and responses to various perturbations or pharmacological interventions using deep neural networks.

Isolated Pulsar Population Synthesis with Simulation-based Inference

We combine pulsar population synthesis with simulation-based inference (SBI) to constrain the magnetorotational properties of isolated Galactic radio pulsars. We first develop a framework to model neutron star birth properties and their dynamical and magnetorotational evolution. We specifically sample initial magnetic field strengths, B, and spin periods, P, from lognormal distributions and capture the late-time magnetic field decay with a power law. Each lognormal is described by a mean, μlogB,μlogP , and standard deviation, σlogB,σlogP , while the power law is characterized by the index, a late. We subsequently model the stars’ radio emission and observational biases to mimic detections with three radio surveys, and we produce a large database of synthetic P– Ṗ diagrams by varying our five magnetorotational input parameters. We then follow an SBI approach that focuses on neural posterior estimation and train deep neural networks to infer the parameters’ posterior distributions. After successfully validating these individual neural density estimators on simulated data, we use an ensemble of networks to infer the posterior distributions for the observed pulsar population. We obtain μlogB=13.10−0.10+0.08 , σlogB=0.45−0.05+0.05 and μlogP=−1.00−0.21+0.26 , σlogP=0.38−0.18+0.33 for the lognormal distributions and alate=−1.80−0.61+0.65 for the power law at the 95% credible interval. We contrast our results with previous studies and highlight uncertainties of the inferred a late value. Our approach represents a crucial step toward robust statistical inference for complex population synthesis frameworks and forms the basis for future multiwavelength analyses of Galactic pulsars.

EFTofLSS meets simulation-based inference: σ 8 from biased tracers

Cosmological inferences typically rely on explicit expressions for the likelihood and covariance of the data vector, which normally consists of a set of summary statistics. However, in the case of nonlinear large-scale structure, exact expressions for either likelihood or covariance are unknown, and even approximate expressions can become very cumbersome, depending on the scales and summary statistics considered. Simulation-based inference (SBI), in contrast, does not require an explicit form for the likelihood but only a prior and a simulator, thereby naturally circumventing these issues. In this paper, we explore how this technique can be used to infer σ 8 from a Lagrangian effective field theory (EFT) based forward model for biased tracers. The power spectrum and bispectrum are used as summary statistics to obtain the posterior of the cosmological, bias and noise parameters via neural density estimation. We compare full simulation-based inference with cases where the data vector is drawn from a Gaussian likelihood with sample and analytical covariances. We conclude that, for k max = 0.1hMpc-1 and 0.2hMpc-1, the form of the covariance is more important than the non-Gaussianity of the likelihood, although this conclusion is expected to depend on the cosmological parameter inferred, the summary statistics considered and range of scales probed.

Cosmology with Galaxy Photometry Alone

We present the first cosmological constraints using only the observed photometry of galaxies. Villaescusa-Navarro et al. recently demonstrated that the internal physical properties of a single simulated galaxy contain a significant amount of cosmological information. These physical properties, however, cannot be directly measured from observations. In this work, we present how we can go beyond theoretical demonstrations to infer cosmological constraints from actual galaxy observables (e.g., optical photometry) using neural density estimation and the CAMELS suite of hydrodynamical simulations. We find that the cosmological information in the photometry of a single galaxy is limited. However, we combine the constraining power of photometry from many galaxies using hierarchical population inference and place significant cosmological constraints. With the observed photometry of ∼20,000 NASA-Sloan Atlas galaxies, we constrain Ωm=0.323−0.095+0.075 and σ8=0.799−0.085+0.088 .

Evidence Networks: simple losses for fast, amortized, neural Bayesian model comparison

Evidence Networks can enable Bayesian model comparison when state-of-the-art methods (e.g. nested sampling) fail and even when likelihoods or priors are intractable or unknown. Bayesian model comparison, i.e. the computation of Bayes factors or evidence ratios, can be cast as an optimization problem. Though the Bayesian interpretation of optimal classification is well-known, here we change perspective and present classes of loss functions that result in fast, amortized neural estimators that directly estimate convenient functions of the Bayes factor. This mitigates numerical inaccuracies associated with estimating individual model probabilities. We introduce the leaky parity-odd power (l-POP) transform, leading to the novel ‘l-POP-Exponential’ loss function. We explore neural density estimation for data probability in different models, showing it to be less accurate and scalable than Evidence Networks. Multiple real-world and synthetic examples illustrate that Evidence Networks are explicitly independent of dimensionality of the parameter space and scale mildly with the complexity of the posterior probability density function. This simple yet powerful approach has broad implications for model inference tasks. As an application of Evidence Networks to real-world data we compute the Bayes factor for two models with gravitational lensing data of the Dark Energy Survey. We briefly discuss applications of our methods to other, related problems of model comparison and evaluation in implicit inference settings.

PopSED: Population-level Inference for Galaxy Properties from Broadband Photometry with Neural Density Estimation

We present PopSED , a framework for the population-level inference of galaxy properties from photometric data. Unlike the traditional approach of first analyzing individual galaxies and then combining the results to determine the physical properties of the entire galaxy population, we directly make the population distribution the inference objective. We train normalizing flows to approximate the population distribution by minimizing the Wasserstein distance between the synthetic photometry of the galaxy population and the observed data. We validate our method using mock observations and apply it to galaxies from the GAMA survey. PopSED reliably recovers the redshift and stellar mass distribution of 105 galaxies using broadband photometry within <1 GPU hr, being 105–6 times faster than the traditional spectral energy distribution modeling method. From the population posterior, we also recover the star-forming main sequence for GAMA galaxies at z < 0.1. With the unprecedented number of galaxies in upcoming surveys, our method offers an efficient tool for studying galaxy evolution and deriving redshift distributions for cosmological analyses.

Fast and efficient identification of anomalous galaxy spectra with neural density estimation

ABSTRACT Current large-scale astrophysical experiments produce unprecedented amounts of rich and diverse data. This creates a growing need for fast and flexible automated data inspection methods. Deep learning algorithms can capture and pick up subtle variations in rich data sets and are fast to apply once trained. Here, we study the applicability of an unsupervised and probabilistic deep learning framework, the probabilistic auto-encoder, to the detection of peculiar objects in galaxy spectra from the SDSS survey. Different to supervised algorithms, this algorithm is not trained to detect a specific feature or type of anomaly, instead it learns the complex and diverse distribution of galaxy spectra from training data and identifies outliers with respect to the learned distribution. We find that the algorithm assigns consistently lower probabilities (higher anomaly score) to spectra that exhibit unusual features. For example, the majority of outliers among quiescent galaxies are E+A galaxies, whose spectra combine features from old and young stellar population. Other identified outliers include LINERs, supernovae, and overlapping objects. Conditional modelling further allows us to incorporate additional information. Namely, we evaluate the probability of an object being anomalous given a certain spectral class, but other information such as metrics of data quality or estimated redshift could be incorporated as well. We make our code publicly available.

Efficient Non-parametric Neural Density Estimation and Its Application to Outlier and Anomaly Detection

The main goal of this thesis is to develop efficient non-parametric density estimation methods that can be integrated with deep learning architectures, for instance, convolutional neural networks and transformers. Density estimation methods can be applied to different problems in statistics and machine learning. They may be used to solve tasks such as anomaly detection, generative models, semi-supervised learning, compression, text-to-speech, among others. The present work will mainly focus on the application of the method in anomaly and outlier detection tasks such as medical anomaly detection, fraud detection, video surveillance, time series anomaly detection, industrial damage detection, among others. A recent approach to non-parametric density estimation is neural density estimation. One advantage of these methods is that they can be integrated with deep learning architectures and trained using gradient descent. Most of these methods are based on neural network implementations of normalizing flows which transform an original simpler distribution to a more complex one. The approach of this thesis is based on a different idea that combines random Fourier features with density matrices to estimate the underlying distribution function. The method can be seen as an approximation of the popular kernel density estimation method but without the inherent computational cost.

Concurrent Multi-Label Prediction in Event Streams

Streams of irregularly occurring events are commonly modeled as a marked temporal point process. Many real-world datasets such as e-commerce transactions and electronic health records often involve events where multiple event types co-occur, e.g. multiple items purchased or multiple diseases diagnosed simultaneously. In this paper, we tackle multi-label prediction in such a problem setting, and propose a novel Transformer-based Conditional Mixture of Bernoulli Network (TCMBN) that leverages neural density estimation to capture complex temporal dependence as well as probabilistic dependence between concurrent event types. We also propose potentially incorporating domain knowledge in the objective by regularizing the predicted probability. To represent probabilistic dependence of concurrent event types graphically, we design a two-step approach that first learns the mixture of Bernoulli network and then solves a least-squares semi-definite constrained program to numerically approximate the sparse precision matrix from a learned covariance matrix. This approach proves to be effective for event prediction while also providing an interpretable and possibly non-stationary structure for insights into event co-occurrence. We demonstrate the superior performance of our approach compared to existing baselines on multiple synthetic and real benchmarks.

Amortized Bayesian inference on generative dynamical network models of epilepsy using deep neural density estimators

Whole-brain modeling of epilepsy combines personalized anatomical data with dynamical models of abnormal activities to generate spatio-temporal seizure patterns as observed in brain imaging data. Such a parametric simulator is equipped with a stochastic generative process, which itself provides the basis for inference and prediction of the local and global brain dynamics affected by disorders. However, the calculation of likelihood function at whole-brain scale is often intractable. Thus, likelihood-free algorithms are required to efficiently estimate the parameters pertaining to the hypothetical areas, ideally including the uncertainty. In this study, we introduce the simulation-based inference for the virtual epileptic patient model (SBI-VEP), enabling us to amortize the approximate posterior of the generative process from a low-dimensional representation of whole-brain epileptic patterns. The state-of-the-art deep learning algorithms for conditional density estimation are used to readily retrieve the statistical relationships between parameters and observations through a sequence of invertible transformations. We show that the SBI-VEP is able to efficiently estimate the posterior distribution of parameters linked to the extent of the epileptogenic and propagation zones from sparse intracranial electroencephalography recordings. The presented Bayesian methodology can deal with non-linear latent dynamics and parameter degeneracy, paving the way for fast and reliable inference on brain disorders from neuroimaging modalities.

Bayesian model comparison for simulation-based inference

Abstract Comparison of appropriate models to describe observational data is a fundamental task of science. The Bayesian model evidence, or marginal likelihood, is a computationally challenging, yet crucial, quantity to estimate to perform Bayesian model comparison. We introduce a methodology to compute the Bayesian model evidence in simulation-based inference (SBI) scenarios (often called likelihood-free inference). In particular, we leverage the recently proposed learned harmonic mean estimator and exploit the fact that it is decoupled from the method used to generate posterior samples, i.e. it requires posterior samples only, which may be generated by any approach. This flexibility, which is lacking in many alternative methods for computing the model evidence, allows us to develop SBI model comparison techniques for the three main neural density estimation approaches, including neural posterior estimation, neural likelihood estimation, and neural ratio estimation. We demonstrate and validate our SBI evidence calculation techniques on a range of inference problems, including a gravitational wave example. Moreover, we further validate the accuracy of the learned harmonic mean estimator, implemented in the harmonic software, in likelihood-based settings. These results highlight the potential of harmonic as a sampler-agnostic method to estimate the model evidence in both likelihood-based and simulation-based scenarios.

DEMANDE: Density Matrix Neural Density Estimation

Density estimation is a fundamental task in statistics and machine learning that aims to estimate, from a set of samples, the probability density function of the distribution that generated them. There are different methods for addressing this problem but recently deep-neural density estimation methods have emerged as a powerful alternative. This paper presents a novel method for neural density estimation based on density matrices and adaptive Fourier features. Density matrices are commonly used in quantum mechanics to represent the quantum state of a physical system. In this work, they are used to estimate probability densities using an operation called quantum measurement. The proposed method can be trained without optimization using an averaging operation over the samples of the training dataset. It can also be integrated with deep learning architectures and trained using gradient descent. The performance of the proposed method was evaluated on a range of synthetic and real datasets and compared with fast kernel density estimation and state-of-the-art neural density estimation methods. The results demonstrate that the proposed method achieves competitive performance while being faster and more efficient than existing methods.

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Inferring the mutation rate of an RNA virus from time-series haplotype data via neural density estimation

Classifying anomalies through outer density estimation

We propose a new model-agnostic search strategy for physics beyond the standard model (BSM) at the LHC, based on a novel application of neural density estimation to anomaly detection. Our approach, which we call Classifying Anomalies THrough Outer Density Estimation (CATHODE), assumes the BSM signal is localized in a signal region (defined e.g. using invariant mass). By training a conditional density estimator on a collection of additional features outside the signal region, interpolating it into the signal region, and sampling from it, we produce a collection of events that follow the background model. We can then train a classifier to distinguish the data from the events sampled from the background model, thereby approaching the optimal anomaly detector. Using the LHC Olympics R&D dataset, we demonstrate that CATHODE nearly saturates the best possible performance, and significantly outperforms other approaches that aim to enhance the bump hunt (CWoLa Hunting and ANODE). Finally, we demonstrate that CATHODE is very robust against correlations between the features and maintains nearly-optimal performance even in this more challenging setting.

Flexible and efficient simulation-based inference for models of decision-making.

Inferring parameters of computational models that capture experimental data are a central task in cognitive neuroscience. Bayesian statistical inference methods usually require the ability to evaluate the likelihood of the model-however, for many models of interest in cognitive neuroscience, the associated likelihoods cannot be computed efficiently. Simulation-based inference (SBI) offers a solution to this problem by only requiring access to simulations produced by the model. Previously, Fengler et al. introduced likelihood approximation networks (LANs, Fengler et al., 2021) which make it possible to apply SBI to models of decision-making, but require billions of simulations for training. Here, we provide a new SBI method that is substantially more simulation efficient. Our approach, mixed neural likelihood estimation (MNLE), trains neural density estimators on model simulations to emulate the simulator, and is designed to capture both the continuous (e.g., reaction times) and discrete (choices) data of decision-making models. The likelihoods of the emulator can then be used to perform Bayesian parameter inference on experimental data using standard approximate inference methods like Markov Chain Monte Carlo sampling. We demonstrate MNLE on two variants of the drift-diffusion model and show that it is substantially more efficient than LANs: MNLE achieves similar likelihood accuracy with six orders of magnitude fewer training simulations, and is significantly more accurate than LANs when both are trained with the same budget. Our approach enables researchers to perform SBI on custom-tailored models of decision-making, leading to fast iteration of model design for scientific discovery.

Machine-guided exploration and calibration of astrophysical simulations

ABSTRACT We apply a novel method with machine learning to calibrate sub-grid models within numerical simulation codes to achieve convergence with observations and between different codes. It utilizes active learning and neural density estimators. The hyper parameters of the machine are calibrated with a well-defined projectile motion problem. Then, using a set of 22 cosmological zoom simulations, we tune the parameters of a popular star formation and feedback model within Enzo to match observations. The parameters that are adjusted include the star formation efficiency, coupling of thermal energy from stellar feedback, and volume into which the energy is deposited. This number translates to a factor of more than three improvements over manual calibration. Despite using fewer simulations, we obtain a better agreement to the observed baryon makeup of a Milky Way (MW)-sized halo. Switching to a different strategy, we improve the consistency of the recommended parameters from the machine. Given the success of the calibration, we then apply the technique to reconcile metal transport between grid-based and particle-based simulation codes using an isolated galaxy. It is an improvement over manual exploration while hinting at a less-known relation between the diffusion coefficient and the metal mass in the halo region. The exploration and calibration of the parameters of the sub-grid models with a machine learning approach is concluded to be versatile and directly applicable to different problems.

Inverting brain grey matter models with likelihood-free inference: a tool for trustable cytoarchitecture measurements

Effective characterisation of the brain grey matter cytoarchitecture with quantitative sensitivity to soma density and volume remains an unsolved challenge in diffusion MRI (dMRI). Solving the problem of relating the dMRI signal with cytoarchitectural characteristics calls for the definition of a mathematical model that describes brain tissue via a handful of physiologically-relevant parameters and an algorithm for inverting the model. To address this issue, we propose a new forward model, specifically a new system of equations, requiring a few relatively sparse b-shells. We then apply modern tools from Bayesian analysis known as likelihood-free inference (LFI) to invert our proposed model. As opposed to other approaches from the literature, our algorithm yields not only an estimation of the parameter vector θ that best describes a given observed data point x0 , but also a full posterior distribution p(θ|x0) over the parameter space. This enables a richer description of the model inversion, providing indicators such as credible intervals for the estimated parameters and a complete characterization of the parameter regions where the model may present indeterminacies. We approximate the posterior distribution using deep neural density estimators, known as normalizing flows, and fit them using a set of repeated simulations from the forward model. We validate our approach on simulations using dmipy and then apply the whole pipeline on two publicly available datasets.