Explicit Likelihood Function Research Articles

Multivariate adaptive regression splines-assisted approximate Bayesian computation for calibration of complex hydrological models

Abstract Approximate Bayesian computation (ABC) relaxes the need to derive explicit likelihood functions required by formal Bayesian analysis. However, the high computational cost of evaluating models limits the application of Bayesian inference in hydrological modeling. In this paper, multivariate adaptive regression splines (MARS) are used to expedite the ABC calibration process. The MARS model is trained using 6,561 runoff simulations generated by the soil and water assessment tool (SWAT) model and subsequently replaces the SWAT model to calculate the objective functions in ABC and multi-objective evolutionary algorithm (MOEA). In experiments, MARS can successfully reproduce the runoff time series simulations of the SWAT model at a low time cost, with a runoff variance determination coefficient of 0.90 as compared to the Monte Carlo method. MARS-assisted ABC can quickly and accurately estimate the parameter distributions of the SWAT model. The comparison of ABC with non-Bayesian MOEAs helps in the selection of an appropriate calibration approach.

An improved uncertainty autoencoder with blurred measurements

AbstractCompressed sensing (CS) techniques have enabled efficient acquisition and recovery of sparse high‐dimensional data via succinct low‐dimensional projections, which usually consist of an encoder and a decoder. Unlike conventional CS techniques with the encoding–decoding architecture, the uncertainty autoencoder (UAE) can sample from the learned input data distribution without an explicit likelihood function and hence avoids potential uninformative latent representations. However, existing works on UAE mainly focus on the encoders and maximize the lower bound of the mutual information between input and measurements, rather than the decoders, which brings the shortcoming that the two may not cope well. In this work, the authors propose a novel training scheme for UAE that blurs the measurements to learn the encoder and decoder simultaneously. Experimental results show that the proposed method improves the reconstruction performances when applied to UAE.

Bayesian Analysis of Multivariate Longitudinal Ordinal Data Using Multiple Multivariate Probit Models

Multivariate longitudinal ordinal data are often involved in longitudinal studies with each individual having more than one longitudinal ordinal measure. However, due to complicated correlation structures within each individual and no explicit likelihood functions, analyzing multivariate longitudinal ordinal data is quite challenging. In this paper, Markov chain Monte Carlo (MCMC) sampling methods are developed to analyze multivariate longitudinal ordinal data by extending multivariate probit (MVP) models for univariate longitudinal ordinal data to multiple multivariate probit models (MMVP) for multivariate longitudinal ordinal data. The identifiable MVP models require the covariance matrix of the latent multivariate normal variables underlying the longitudinal ordinal variables to be a correlation matrix, thus a Metropolis-Hastings (MH) algorithm is usually necessitated, which brings a rigorous task to develop efficient MCMC sampling methods. In contrast to the identifiable MVP models, the non-identifiable MVP models can be constructed to circumvent a MH algorithm to sample a correlation matrix by a Gibbs sampling to sample a covariance matrix, and hence improve the mixing and convergence of the MCMC components. Therefore, both the identifiable MMVP models and the non-identifiable MMVP models for multivariate longitudinal ordinal data are presented, and their corresponding MCMC sampling methods are developed. The performances of these methods are illustrated through simulation studies and an application using data from the Russia Longitudinal Monitoring Survey-Higher School of Economics (RLMS-HSE).

On Maximum a Posteriori Estimation with Plug & Play Priors and Stochastic Gradient Descent

Bayesian methods to solve imaging inverse problems usually combine an explicit data likelihood function with a prior distribution that explicitly models expected properties of the solution. Many kinds of priors have been explored in the literature, from simple ones expressing local properties to more involved ones exploiting image redundancy at a non-local scale. In a departure from explicit modelling, several recent works have proposed and studied the use of implicit priors defined by an image denoising algorithm. This approach, commonly known as Plug & Play (PnP) regularization, can deliver remarkably accurate results, particularly when combined with state-of-the-art denoisers based on convolutional neural networks. However, the theoretical analysis of PnP Bayesian models and algorithms is difficult and works on the topic often rely on unrealistic assumptions on the properties of the image denoiser. This papers studies maximum a posteriori (MAP) estimation for Bayesian models with PnP priors. We first consider questions related to existence, stability and well-posedness and then present a convergence proof for MAP computation by PnP stochastic gradient descent (PnP-SGD) under realistic assumptions on the denoiser used. We report a range of imaging experiments demonstrating PnP-SGD as well as comparisons with other PnP schemes.

Bayesian Imaging Using Plug & Play Priors: When Langevin Meets Tweedie

Since the seminal work of Venkatakrishnan et al. in 2013, Plug & Play (PnP) methods have become ubiquitous in Bayesian imaging. These methods derive Minimum Mean Square Error (MMSE) or Maximum A Posteriori (MAP) estimators for inverse problems in imaging by combining an explicit likelihood function with a prior that is implicitly defined by an image denoising algorithm. The PnP algorithms proposed in the literature mainly differ in the iterative schemes they use for optimisation or for sampling. In the case of optimisation schemes, some recent works guarantee the convergence to a fixed point, albeit not necessarily a MAP estimate. In the case of sampling schemes, to the best of our knowledge, there is no known proof of convergence. There also remain important open questions regarding whether the underlying Bayesian models and estimators are well defined, well-posed, and have the basic regularity properties required to support these numerical schemes. To address these limitations, this paper develops theory, methods, and provably convergent algorithms for performing Bayesian inference with PnP priors. We introduce two algorithms: 1) PnP-ULA (Unadjusted Langevin Algorithm) for Monte Carlo sampling and MMSE inference; and 2) PnP-SGD (Stochastic Gradient Descent) for MAP inference. Using recent results on the quantitative convergence of Markov chains, we establish detailed convergence guarantees for these two algorithms under realistic assumptions on the denoising operators used, with special attention to denoisers based on deep neural networks. We also show that these algorithms approximately target a decision-theoretically optimal Bayesian model that is well-posed. The proposed algorithms are demonstrated on several canonical problems such as image deblurring, inpainting, and denoising, where they are used for point estimation as well as for uncertainty visualisation and quantification.

Non-equilibrium time-dependent solution to discrete choice with social interactions.

We solve the binary decision model of Brock and Durlauf (2001) in time using a method reliant on the resolvent of the master operator of the stochastic process. Our solution is valid when not at equilibrium and can be used to exemplify path-dependent behaviours of the binary decision model. The solution is computationally fast and is indistinguishable from Monte Carlo simulation. Well-known metastable effects are observed in regions of the model's parameter space where agent rationality is above a critical value, and we calculate the time scale at which equilibrium is reached using a highly accurate method based on first passage time theory. In addition to considering selfish agents, who only care to maximise their own utility, we consider altruistic agents who make decisions on the basis of maximising global utility. Curiously, we find that although altruistic agents coalesce more strongly on a particular decision, thereby increasing their utility in the short-term, they are also more prone to being subject to non-optimal metastable regimes as compared to selfish agents. The method used for this solution can be easily extended to other binary decision models, including Kirman's model of ant recruitment Kirman (1993), and under reinterpretation also provides a time-dependent solution to the mean-field Ising model. Finally, we use our time-dependent solution to construct a likelihood function that can be used on non-equilibrium data for model calibration. This is a rare finding, since often calibration in economic agent based models must be done without an explicit likelihood function. From simulated data, we show that even with a well-defined likelihood function, model calibration is difficult unless one has access to data representative of the underlying model.

Efficient Bayesian inference using adversarial machine learning and low-complexity surrogate models

Bayesian inference is a key method for estimating parametric uncertainty from data. However, most Bayesian inference methods require the explicit likelihood function or many samples, both of which are unrealistic to provide for complex first-principles-based models. Here, we propose a novel Bayesian inference methodology for estimating uncertain parameters of computationally intensive first-principles-based models. Our approach exploits both low-complexity surrogate models and variational inference with arbitrarily expressive inference models. The proposed methodology indirectly predicts output responses and casts Bayesian inference as an optimization problem. We demonstrate its performance via synthetic problems, computational fluid dynamics, and kinetic Monte Carlo simulation to verify its applicability. This fast and reliable methodology enables us to capture multimodality and the shape of complicated posterior distributions with the quality of state-of-the-art Hamiltonian Monte Carlo methods but with much lower computation cost.

Reliability analysis of a complex system with hybrid structures and multi-level dependent life metrics

In practical engineering, the presence of dependent evidence is not rare due to various imperfections. Misuse of such information in reliability analysis will lead to conflicting or even erroneous results. In this paper, we propose a Bayesian reliability approach for complex systems with dependent life metrics. Notions such as explicit evidence and implicit evidence are established based on an identification of different roles of multiple dependent evidence in the likelihood construction. A likelihood decomposition method is developed to convert the overall likelihood into a product of Explicit Evidence-based Likelihood (EEL) function and Implicit Evidence-based Likelihood (IEL) function. An inferential diagram is developed to intuitively generate the required implicit evidence taking both outer-source information and the system configuration into consideration. An algorithm is then presented for implementation. The contribution of our work is a systematic investigation of the role of dependent evidence in system reliability evaluation and a full Bayesian approach that is applied to various system reliability models. Extensive numerical cases and a practical engineering case are demonstrated for validation and to illustrate the benefits of our approach.

BayesFlow: Learning Complex Stochastic Models With Invertible Neural Networks.

Estimating the parameters of mathematical models is a common problem in almost all branches of science. However, this problem can prove notably difficult when processes and model descriptions become increasingly complex and an explicit likelihood function is not available. With this work, we propose a novel method for globally amortized Bayesian inference based on invertible neural networks that we call BayesFlow. The method uses simulations to learn a global estimator for the probabilistic mapping from observed data to underlying model parameters. A neural network pretrained in this way can then, without additional training or optimization, infer full posteriors on arbitrarily many real data sets involving the same model family. In addition, our method incorporates a summary network trained to embed the observed data into maximally informative summary statistics. Learning summary statistics from data makes the method applicable to modeling scenarios where standard inference techniques with handcrafted summary statistics fail. We demonstrate the utility of BayesFlow on challenging intractable models from population dynamics, epidemiology, cognitive science, and ecology. We argue that BayesFlow provides a general framework for building amortized Bayesian parameter estimation machines for any forward model from which data can be simulated.

Modal linear regression using log-concave distributions

The modal linear regression suggested by Yao and Li (Scand J Stat 41(3):656–671, 2014) models the conditional mode of a response Y given a vector of covariates $$\mathbf{z }$$ as a linear function of $$\mathbf{z }$$ . To identify the conditional mode of Y given $$\mathbf{z }$$ , existing methods utilize a kernel density estimator to obtain the distribution of Y given $$\mathbf{z }$$ . Like other kernel-based methods, these require a suitable choice of tuning parameters, and no unified objective function exists for estimating regression parameters. In this paper, we propose a model-based modal linear regression using a family of log-concave distributions. The proposed method does not require tuning parameters and enables us to construct an explicit likelihood function. To estimate the regression parameters with an estimated log-concave density, we turned the log-likelihood into a sum of affine functions using a dual representation of piecewise linear concave functions so that well-known linear programming techniques can be adopted. Simulation studies reveal that the proposed method produces more efficient estimators than kernel-based methods. A real data example is also presented to illustrate the applicability of the proposed method.

Bayesian rank-based hypothesis testing for the rank sum test, the signed rank test, and Spearman's ρ

Bayesian inference for rank-order problems is frustrated by the absence of an explicit likelihood function. This hurdle can be overcome by assuming a latent normal representation that is consistent with the ordinal information in the data: the observed ranks are conceptualized as an impoverished reflection of an underlying continuous scale, and inference concerns the parameters that govern the latent representation. We apply this generic data-augmentation method to obtain Bayes factors for three popular rank-based tests: the rank sum test, the signed rank test, and Spearman's .

A hybrid omnibus test for generalized semiparametric single-index models with high-dimensional covariate sets.

Numerous statistical methods have been developed for analyzing high-dimensional data. These methods often focus on variable selection approaches but are limited for the purpose of testing with high-dimensional data. They are often required to have explicit-likelihood functions. In this article, we propose a "hybrid omnibus test" for high-dicmensional data testing purpose with much weaker requirements. Our hybrid omnibus test is developed under a semiparametric framework where a likelihood function is no longer necessary. Our test is a version of a frequentist-Bayesian hybrid score-type test for a generalized partially linear single-index model, which has a link function being a function of a set of variables through a generalized partially linear single index. We propose an efficient score based on estimating equations, define local tests, and then construct our hybrid omnibus test using local tests. We compare our approach with an empirical-likelihood ratio test and Bayesian inference based on Bayes factors, using simulation studies. Our simulation results suggest that our approach outperforms the others, in terms of type I error, power, and computational cost in both the low- and high-dimensional cases. The advantage of our approach is demonstrated by applying it to genetic pathway data for type II diabetes mellitus.

Model selection and parameter estimation of dynamical systems using a novel variant of approximate Bayesian computation

Abstract Model selection is a challenging problem that is of importance in many branches of the sciences and engineering, particularly in structural dynamics. By definition, it is intended to select the most likely model among a set of competing models that best matches the dynamic behaviour of a real structure and better predicts the measured data. The Bayesian approach which is based essentially on the evaluation of a likelihood function is one of the most popular approach to deal with model selection and parameter estimation issues. However, in some circumstances, the likelihood function is either intractable or not available even in a closed form. To overcome this issue, the likelihood-free or approximate Bayesian computation (ABC) algorithm has been introduced in the literature, which relaxes the need for an explicit likelihood function to measure the level of agreement between model predictions and measurements. However, ABC algorithms suffer from a low acceptance rate of samples which is actually a common problem with the traditional Bayesian methods. To overcome this shortcoming and alleviate the computational burden, a new variant of the ABC algorithm based on an ellipsoidal Nested Sampling (NS) technique is introduced in this paper; it has been called ABC-NS. Through this paper, it will be shown how the new algorithm is a promising alternative to deal with parameter estimation and model selection issues. It promises drastic speedups and provides a good approximation of the posterior distributions. To demonstrate its robust computational efficiency, four illustrative examples are given. Firstly, the efficiency of the algorithm is demonstrated to deal with parameter estimation. Secondly, two examples based on simulated and real data are given to demonstrate the efficiency of the algorithm to deal with model selection in structural dynamics.

Emulator-enabled approximate Bayesian computation (ABC) and uncertainty analysis for computationally expensive groundwater models

Bayesian inference provides a mathematically elegant and robust approach to constrain numerical model predictions with system knowledge and observations. Technical challenges, such as evaluating a large number of models with long runtimes, have restricted the application of Bayesian inference to groundwater modeling. To overcome such technical challenges, we use Gaussian process emulators to replace a transient regional groundwater MODFLOW model for computing objective functions during model constraining. The regional model is designed to assess the potential impact of a proposed coal seam gas (CSG) development on groundwater levels in the Richmond River catchment, Clarence-Moreton Basin, Australia. The emulators were trained using 4000 snapshots derived from the MODFLOW model and subsequently used to replace the MODFLOW model in an Approximate Bayesian Computation (ABC) scheme. ABC was deemed the more appropriate choice as it relaxes the need to derive an explicit likelihood function that the formal Bayesian analysis requires. The study demonstrated the flexibility of the Gaussian process emulators, which can accurately reproduce the original model behavior at a fraction of the computational cost (from hours to seconds). The gain in computational efficiency using the proposed approach allows the global calibration and uncertainty algorithms to become more feasible for computationally demanding groundwater models. Based on the ABC analysis, the probability for the simulated CSG development causing a water table change of more than 0.2 m was less than 5%. In addition to a probabilistic estimate of the prediction, an added value of emulator-assisted ABC inference is providing information on the extent to which observations can constrain parameters and predictions, as well as the flexibility to include various quantitative and qualitative parameter constraining information.

Bayesian Inference for Kendall’s Rank Correlation Coefficient

ABSTRACTThis article outlines a Bayesian methodology to estimate and test the Kendall rank correlation coefficient τ. The nonparametric nature of rank data implies the absence of a generative model and the lack of an explicit likelihood function. These challenges can be overcome by modeling test statistics rather than data. We also introduce a method for obtaining a default prior distribution. The combined result is an inferential methodology that yields a posterior distribution for Kendall’s τ.

Parametric inference for discrete observations of diffusion processes with mixed effects

Stochastic differential equations with mixed effects provide means to model intra-individual and inter-individual variability in repeated experiments leading to longitudinal data. We consider N i.i.d. stochastic processes defined by a stochastic differential equation with linear mixed effects which are discretely observed. We study a parametric framework with distributions leading to explicit approximate likelihood functions and investigate the asymptotic behavior of estimators under the asymptotic framework : the number N of individuals (trajectories) and the number n of observations per individual tend to infinity within a fixed time interval. The estimation method is assessed on simulated data for various models.

Inferring Rates and Length-Distributions of Indels Using Approximate Bayesian Computation.

The most common evolutionary events at the molecular level are single-base substitutions, as well as insertions and deletions (indels) of short DNA segments. A large body of research has been devoted to develop probabilistic substitution models and to infer their parameters using likelihood and Bayesian approaches. In contrast, relatively little has been done to model indel dynamics, probably due to the difficulty in writing explicit likelihood functions. Here, we contribute to the effort of modeling indel dynamics by presenting SpartaABC, an approximate Bayesian computation (ABC) approach to infer indel parameters from sequence data (either aligned or unaligned). SpartaABC circumvents the need to use an explicit likelihood function by extracting summary statistics from simulated sequences. First, summary statistics are extracted from the input sequence data. Second, SpartaABC samples indel parameters from a prior distribution and uses them to simulate sequences. Third, it computes summary statistics from the simulated sets of sequences. By computing a distance between the summary statistics extracted from the input and each simulation, SpartaABC can provide an approximation to the posterior distribution of indel parameters as well as point estimates. We study the performance of our methodology and show that it provides accurate estimates of indel parameters in simulations. We next demonstrate the utility of SpartaABC by studying the impact of alignment errors on the inference of positive selection. A C ++ program implementing SpartaABC is freely available in http://spartaabc.tau.ac.il.

Software reliability growth modeling and analysis with dual fault detection and correction processes

ABSTRACTComputer software is widely applied in safety-critical systems. The ever-increasing complexity of software systems makes it extremely difficult to ensure software reliability, and this problem has drawn considerable attention from both industry and academia. Most software reliability models are built on a common assumption that the detected faults are immediately corrected; thus, the fault detection and correction processes can be regarded as the same process. In this article, a comprehensive study is conducted to analyze the time dependencies between the fault detection and correction processes. The model parameters are estimated using the Maximum Likelihood Estimation (MLE) method, which is based on an explicit likelihood function combining both the fault detection and correction processes. Numerical case studies are conducted under the proposed modeling framework. The obtained results demonstrate that the proposed MLE method can be applied to more general situations and provide more accurate results. Furthermore, the predictive capability of the MLE method is compared with that of the Least Squares Estimation (LSE) method. The prediction results indicate that the proposed MLE method performs better than the LSE method when the data are not large in size or are collected in the early phase of software testing.

Reply to comment by Chu et al. on “High‐dimensional posterior exploration of hydrologic models using multiple‐try DREAM (ZS) and high‐performance computing”

PUBLICATIONS Water Resources Research COMMENTARY 10.1002/2013WR014425 This article is a reply to Chu et al. [2014] doi:10.1002/2012WR013341. Reply to comment by Chu et al. on ‘‘High-dimensional posterior exploration of hydrologic models using multiple-try DREAM (ZS) and high-performance computing’’ Jasper A. Vrugt 1,2,3 and Eric Laloy 4 Correspondence to: J. A. Vrugt, jasper@uci.edu Citation: Vrugt, J. A., and E. Laloy (2014), Reply to comment Chu et al. on ‘‘High- dimensional posterior exploration of hydrologic models using multiple-try DREAM (ZS) and high-performance computing,’’ Water Resour. Res., 50, 2781–2786, doi:10.1002/ 2013WR014425. Received 15 JULY 2013 Accepted 8 FEB 2014 Accepted article online 15 FEB 2014 Published online 21 MAR 2014 Department of Civil and Environmental Engineering, University of California, Irvine, California, USA, 2 Department of Earth System Science, University of California, Irvine, California, USA, 3 Institute for Biodiversity and Ecosystems Dynamics, University of Amsterdam, Amsterdam, Netherlands, 4 Institute for Environment, Health and Safety, Belgian Nuclear Research Centre, Mol, Belgium 1. Introduction The comment of Chu et al. [2014], hereafter referred to as CYG, raises questions about some of the results presented in our paper (Laloy and Vrugt [2012], hereinafter LV, which is not to be confused with Las Vegas, although appropriate concerning the subject of this work on Monte Carlo simulation). As a preamble, we would like to remark that the work presented in Chu et al. [2010] and LV (2012) concern two different fields of study. CYG view the model calibration as an optimization problem, and use common concepts to efficiently find a single realization of the parameter values that minimizes (or maximizes, if appropriate) some user-defined objective function. Our work, on the contrary, uses Bayesian principles coupled with MCMC simulation to derive a distribution of parameter values that honor the observed data. This distribution summarizes parameter and model predictive (simulation) uncertainty, a requirement for probabilistic analysis, operational forecasting, disentangling error sources, and decision making. The maxi- mum a posteriori density (MAP) parameter values derived with MCMC simulation should reside in close vicinity of the ‘‘best’’ solution found with an optimization algorithm, if the exact same data set, prior distribu- tion, and likelihood (objective) function are used. In this reply, we assume that CYG used a correct imple- mentation of the MT-DREAM (ZS) algorithm and similar data set, prior and likelihood function as LV. Otherwise, the comparative analysis is meaningless. We emphasize this for three reasons. First, the results presented herein contradict CYG and are similar to those reported in LV but now with more trials plotted. Second, contributions in physics [Horowitz et al., 2012; Toyli et al., 2012; Yale et al., 2013] and geophysics [Linde and Vrugt, 2013; Laloy et al., 2012, Rosas-Carbajal et al., 2014; T. Lochbuehler et al., Summary statistics from training images as model constraints in probabilistic inversion, Geophysical Journal International, in review, 2014] demonstrate proper convergence behavior of MT-DREAM (ZS) on complex and high- dimensional targets involving hundreds of parameters. Third, to justify their SP-UCI algorithm the original paper of Chu et al. [2010] portrays misleading results of the predecessor of DREAM, called SCEM-UA. Section 4 of this reply will address this latter issue in more detail. We now respond to the comments of CYG. We use different sections with numbering corresponding to CYG. 2. Computational Time Unit CYG find the Computational Time Unit (CTU) diagnostic to be a poor indicator of the performance of MT- DREAM (ZS) . They suggest using the number of function evaluations or clock time instead. The CPU-time (s) scales linearly with CTU, or CPU 5 aCTU, where a (s) denotes the average time it takes to complete a sin- gle function (model) evaluation. As a is dependent on the processor speed (hardware), LV purposely reported the CTU values. Note that we neglect the actual run time of MT-DREAM (ZS) , in the determination of a, which is appropriate given the intended application of this algorithm to CPU-intensive forward models. We purposely do not use the number of function evaluations as performance diagnostic. This metric does not properly convey the CPU-time (CTU) of parallel algorithms such as DREAM (ZS) or MT-DREAM (ZS) . These VRUGT AND LALOY C 2014. American Geophysical Union. All Rights Reserved. V

Bridging the gap between GLUE and formal statistical approaches: approximate Bayesian computation

Abstract. In recent years, a strong debate has emerged in the hydrologic literature regarding how to properly treat nontraditional error residual distributions and quantify parameter and predictive uncertainty. Particularly, there is strong disagreement whether such uncertainty framework should have its roots within a proper statistical (Bayesian) context using Markov chain Monte Carlo (MCMC) simulation techniques, or whether such a framework should be based on a quite different philosophy and implement informal likelihood functions and simplistic search methods to summarize parameter and predictive distributions. This paper is a follow-up of our previous work published in Vrugt and Sadegh (2013) and demonstrates that approximate Bayesian computation (ABC) bridges the gap between formal and informal statistical model–data fitting approaches. The ABC methodology has recently emerged in the fields of biology and population genetics and relaxes the need for an explicit likelihood function in favor of one or multiple different summary statistics that measure the distance of each model simulation to the data. This paper further studies the theoretical and numerical equivalence of formal and informal Bayesian approaches using discharge and forcing data from different watersheds in the United States, in particular generalized likelihood uncertainty estimation (GLUE). We demonstrate that the limits of acceptability approach of GLUE is a special variant of ABC if each discharge observation of the calibration data set is used as a summary diagnostic.