True Likelihood Function Research Articles

Bayesian estimation of large-scale simulation models with Gaussian process regression surrogates

Large scale, computationally expensive simulation models pose a particular challenge when it comes to estimating their parameters from empirical data. Most simulation models do not possess closed-form expressions for their likelihood function, requiring the use of simulation-based inference, such as simulated method of moments, indirect inference, likelihood-free inference or approximate Bayesian computation. However, given the high computational requirements of large-scale models, it is often difficult to run these estimation methods, as they require more simulated runs that can feasibly be carried out. The aim is to address the problem by providing a full Bayesian estimation framework where the true but intractable likelihood function of the simulation model is replaced by one generated by a surrogate model trained on the limited simulated data. This is provided by a Linear Model of Coregionalization, where each latent variable is a sparse variational Gaussian process, chosen for its desirable convergence and consistency properties. The effectiveness of the approach is tested using both a simulated Bayesian computing analysis on a known data generating process, and an empirical application in which the free parameters of a computationally demanding agent-based model are estimated on US macroeconomic data.

Nested Variational Chain and Its Application in Massive MIMO Detection for High-Order Constellations.

Multiple input multiple output (MIMO) technology necessitates detection methods with high performance and low complexity; however, the detection problem becomes severe when high-order constellations are employed. Variational approximation-based algorithms prove to deal with this problem efficiently, especially for high-order MIMO systems. Two typical algorithms named Gaussian tree approximation (GTA) and expectation consistency (EC) attempt to approximate the true likelihood function under discrete finite-set constraints with a new distribution by minimizing the Kullback-Leibler (KL) divergence. As the KL divergence is not a true distance measure, 'exclusive' and 'inclusive' KL divergences are utilized by GTA and EC, respctively, demonstrating different performances. In this paper, we further combine the two asymmetric KL divergences in a nested way by proposing a generic algorithm framework named nested variational chain. Acting as an initial application, a MIMO detection algorithm named Gaussian tree approximation expectation consistency (GTA-EC) can thus be presented along with its alternative version for better understanding. With less computational burden compared to its counterparts, GTA-EC is able to provide better detection performance and diversity gain, especially for large-scale high-order MIMO systems.

Bayesian evidence-driven likelihood selection for sky-averaged 21-cm signal extraction

AbstractWe demonstrate that the Bayesian evidence can be used to find a good approximation of the ground truth likelihood function of a dataset, a goal of the likelihood-free inference (LFI) paradigm. As a concrete example, we use forward modelled sky-averaged 21-cm signal antenna temperature datasets where we artificially inject noise structures of various physically motivated forms. We find that the Gaussian likelihood performs poorly when the noise distribution deviates from the Gaussian case, for example, heteroscedastic radiometric or heavy-tailed noise. For these non-Gaussian noise structures, we show that the generalised normal likelihood is on a similar Bayesian evidence scale with comparable sky-averaged 21-cm signal recovery as the ground truth likelihood function of our injected noise. We therefore propose the generalised normal likelihood function as a good approximation of the true likelihood function if the noise structure is a priori unknown.

Estimation in tensor Ising models

AbstractThe $p$-tensor Ising model is a one-parameter discrete exponential family for modeling dependent binary data, where the sufficient statistic is a multi-linear form of degree $p \geqslant 2$. This is a natural generalization of the matrix Ising model that provides a convenient mathematical framework for capturing, not just pairwise, but higher-order dependencies in complex relational data. In this paper, we consider the problem of estimating the natural parameter of the $p$-tensor Ising model given a single sample from the distribution on $N$ nodes. Our estimate is based on the maximum pseudolikelihood (MPL) method, which provides a computationally efficient algorithm for estimating the parameter that avoids computing the intractable partition function. We derive general conditions under which the MPL estimate is $\sqrt N$-consistent, that is, it converges to the true parameter at rate $1/\sqrt N$. Our conditions are robust enough to handle a variety of commonly used tensor Ising models, including spin glass models with random interactions and models where the rate of estimation undergoes a phase transition. In particular, this includes results on $\sqrt N$-consistency of the MPL estimate in the well-known $p$-spin Sherrington–Kirkpatrick model, spin systems on general $p$-uniform hypergraphs and Ising models on the hypergraph stochastic block model (HSBM). In fact, for the HSBM we pin down the exact location of the phase transition threshold, which is determined by the positivity of a certain mean-field variational problem, such that above this threshold the MPL estimate is $\sqrt N$-consistent, whereas below the threshold no estimator is consistent. Finally, we derive the precise fluctuations of the MPL estimate in the special case of the $p$-tensor Curie–Weiss model, which is the Ising model on the complete $p$-uniform hypergraph. An interesting consequence of our results is that the MPL estimate in the Curie–Weiss model saturates the Cramer–Rao lower bound at all points above the estimation threshold, that is, the MPL estimate incurs no loss in asymptotic statistical efficiency in the estimability regime, even though it is obtained by minimizing only an approximation of the true likelihood function for computational tractability.

Cosmoabc: Likelihood-free inference via Population Monte Carlo Approximate Bayesian Computation

Approximate Bayesian Computation (ABC) enables parameter inference for complex physical systems in cases where the true likelihood function is unknown, unavailable, or computationally too expensive. It relies on the forward simulation of mock data and comparison between observed and synthetic catalogues. Here we present cosmoabc, a Python ABC sampler featuring a Population Monte Carlo (PMC) variation of the original ABC algorithm, which uses an adaptive importance sampling scheme. The code is very flexible and can be easily coupled to an external simulator, while allowing to incorporate arbitrary distance and prior functions. As an example of practical application, we coupled cosmoabc with the numcosmo library and demonstrate how it can be used to estimate posterior probability distributions over cosmological parameters based on measurements of galaxy clusters number counts without computing the likelihood function. cosmoabc is published under the GPLv3 license on PyPI and GitHub and documentation is available at http://goo.gl/SmB8EX

Error Propagation in Gossip-Based Distributed Particle Filters

This paper examines the impact of the gossip procedure on distributed particle filters that employ averaging to estimate the global likelihood function. We consider a model where a gossip-driven algorithm leads to the use of a slightly distorted version of the likelihood function, in lieu of its true value. Under standard regularity conditions, and a mild assumption on the true likelihood function, we derive a time-uniform bound on the weak-sense $L_p$ error of the filter. Furthermore, we present an associated exponential inequality for the large deviations of the filter. These bounds capture the combined effects of sampling and consensus-based approximation. The results allow us to evaluate the impact of such approximations on the overall performance of the distributed particle filter, and analyze its stability. Finally, through numerical experiments, we demonstrate the practical implications of these results and explore the relationship of the performance of the filter with these theoretical error bounds.

Network Estimation in Cognition-Empowered Wireless Networks

An approach to parametric identification of the transmission processes of the terminals in a wireless network is proposed, presenting a tradeoff between accuracy of capturing the temporal dependencies in observations of transmission processes and the time complexity of the estimation procedure. The maximum likelihood estimator is built for an approximation of the true likelihood function for the observed network activity. A complex network where terminals store packets in a finite buffer and implement a backoff-based random channel access protocol is considered. Minimal information is available for observation to the cognitive terminals, in the form of energy readings mapped to the number of transmitting nodes in each time instant. The entanglement of the transmission processes induced by interference and the filtering effect of packet buffering make this task particularly difficult. It is shown how, based on the estimated parameters, the cognitive terminals, operating in the same channel resource, can predict the transmission trajectories of the other nodes and devise smart transmission strategies controlling the interference generated to the network.

Exact Cramer–Rao Bounds for Semiblind Channel Estimation in Amplify-and-Forward Two-Way Relay Networks Employing Square QAM

In this paper, we derive the Cramer-Rao bound (CRB) for semiblind channel estimation in amplify-and-forward two-way relay networks employing square QAM, assuming flat-fading channel conditions. The derived bound is exact as it is based on the true likelihood function that takes into account the statistics of the transmitted data symbols. Using the new bound, we show that exploiting even a limited number of transmitted data symbols in addition to the pilot symbols leads to substantial estimation accuracy improvements over conventional pilot-based estimation. We also propose a semiblind expectation-maximization-based estimation algorithm that performs very close to the exact CRB at an affordable computational cost. The superior accuracy of the semiblind approach makes it possible to significantly reduce the training overhead for channel estimation, thus offering a higher throughput and a better tradeoff between accuracy and spectral efficiency. We also derive the modified CRB, which approximates the exact CRB at high SNR for low modulation orders.

A loss function approach to model specification testing and its relative efficiency

The generalized likelihood ratio (GLR) test proposed by Fan, Zhang and Zhang [Ann. Statist. 29 (2001) 153-193] and Fan and Yao [Nonlinear Time Series: Nonparametric and Parametric Methods (2003) Springer] is a generally applicable nonparametric inference procedure. In this paper, we show that although it inherits many advantages of the parametric maximum likelihood ratio (LR) test, the GLR test does not have the optimal power property. We propose a generally applicable test based on loss functions, which measure discrepancies between the null and nonparametric alternative models and are more relevant to decision-making under uncertainty. The new test is asymptotically more powerful than the GLR test in terms of Pitman's efficiency criterion. This efficiency gain holds no matter what smoothing parameter and kernel function are used and even when the true likelihood function is available for the GLR test.

Closed-form likelihood expansions for multivariate time-inhomogeneous diffusions

The aim of this paper is to find approximate log-transition density functions for multivariate time-inhomogeneous diffusions in closed form. There are many empirical evidences supporting that the data generating process governing dynamics of many economics variables might vary over time because of economic climate changes or time effects. One possible way to explain the time-dependent dynamics of state variables is to model the drift or volatility terms as functions of time t as well as state variables. A way to find closed-form likelihood expansion for a multivariate time-homogeneous diffusion has been developed by Aït-Sahalia (2008). This research is built on his work and extends his results to time-inhomogeneous cases. We conduct Monte Carlo simulation studies to examine performance of the approximate transition density function when it is used to obtain ML estimates. The results reveal that our method yields a very accurate approximate likelihood function, which can be a good candidate when the true likelihood function is unavailable as is often the case.

Closed-form likelihood expansions for multivariate diffusions

This paper provides closed-form expansions for the log-likelihood function of multivariate diffusions sampled at discrete time intervals. The coefficients of the expansion are calculated explicitly by exploiting the special structure afforded by the diffusion model. Examples of interest in financial statistics and Monte Carlo evidence are included, along with the convergence of the expansion to the true likelihood function.

A Note on Whittle's Likelihood

The approximate likelihood function introduced by Whittle has been used to estimate the spectral density and certain parameters of a variety of time series models. In this note we attempt to empirically quantify the loss of efficiency of Whittle's method in nonstandard settings. A recently developed representation of some first-order non-Gaussian stationary autoregressive process allows a direct comparison of the true likelihood function with that of Whittle. The conclusion is that Whittle's likelihood can produce unreliable estimates in the non-Gaussian case, even for moderate sample sizes. Moreover, for small samples, and if the autocorrelation of the process is high, Whittle's approximation is not efficient even in the Gaussian case. While these facts are known to some extent, the present study sheds more light on the degree of efficiency loss incurred by using Whittle's likelihood, in both Gaussian and non-Gaussian cases.

Tree Search Detection Based on LLR Using M Algorithm in MC-CDMA Systems

In this paper, we propose a new multiuser detection scheme using Maximum Likelihood (ML) criterion and the M algorithm for Multi Carrier (MC)-Code Division Multiple Access (CDMA) systems in the down-link channel. We first describe an implementation of ML detection separating In- and Quadrature-phase components and using well-known linear filters. In the proposed algorithm, we produce hypothesis symbol vectors in a tree structure by partly changing the sub-optimum hard decisions based on the linear filter output. At each stage, we adopt the best M likely paths with respect to the true log likelihood or distance function as survivors. We determine the symbol vector which minimizes the distance function at the final stage. Although the complexity of ML detector is exponentially increasing as a function of the number of users, the proposed scheme requires by far less complexity. We demonstrate that the proposed scheme achieves equivalent Bit Error Rate (BER) performance with lower complexity in comparison with ML detector by computer simulations. Moreover we compare the proposed detection scheme with QRD-M algorithm which is based on QR decomposition combined with M algorithm.

An online sequential algorithm for the estimation of transition probabilities for jump Markov linear systems

This paper describes a new method to estimate the transition probabilities associated with a jump Markov linear system. The new algorithm uses stochastic approximation type recursions to minimize the Kullback–Leibler divergence between the likelihood function of the transition probabilities and the true likelihood function. Since the calculation of the likelihood function of the transition probabilities is impossible, an incomplete data paradigm, which has been previously applied to a similar problem for hidden Markov models, is used. The algorithm differs from the existing algorithms in that it assumes that the transition probabilities are deterministic quantities whereas the existing approaches consider them to be random variables with prior distributions.

Maximum-Likelihood Estimation of Discretely Sampled Diffusions: A Closed-Form Approach

When a continuous-time diffusion is observed only at discrete dates, not necessarily close together, the likelihood function of the observations is in most cases not explicitly computable. Researchers have relied on simulations of sample paths in between the observations points, or numerical solutions of partial differential equations, to obtain estimates of the function to be maximized. By contrast, we construct a sequence of fully explicit functions which we show converge under very general conditions, including non-ergodicity, to the true (but unknown) likelihood function of the discretely-sampled diffusion. We document that the rate of convergence of the sequence is extremely fast for a number of examples relevant in finance. We then show that maximizing the sequence instead of the true function results in an estimator which converges to the true maximum-likelihood estimator and shares its asymptotic properties of consistency, asymptotic normality and efficiency. Applications to the valuation of derivative securities are also discussed.

Improved profile likelihood inference

Inference about a multiparameter of interest in the presence of nuisance parameters is oftentimes based on the profile likelihood function. However, it does not behave as a true likelihood function and several adjustments to the profile likelihood function have been proposed. In this paper, we consider an additive adjustment that aims at reducing the score and information biases of the score function from order O ( 1 ) to order O ( n - 1 ) . The adjustment is applicable in wide generality since it allows both the interest and nuisance parameters to be vector-valued. We derive a Bartlett correction for the adjusted profile likelihood ratio statistic and obtain closed-form expressions for the class of the generalized linear models. A simulation study compares the performance of the usual profile likelihood ratio test, the adjusted profile likelihood ratio test and the associated Bartlett-corrected test.

Estimating Nonlinear Dynamic Equilibrium Economies: A Likelihood Approach

Abstract: This paper presents a framework to undertake likelihood-based inference in nonlinear dynamic equilibrium economies. The authors develop a sequential Monte Carlo algorithm that delivers an estimate of the likelihood function of the model using simulation methods. This likelihood can be used for parameter estimation and for model comparison. The algorithm can deal both with nonlinearities of the economy and with the presence of non-normal shocks. The authors show consistency of the estimate and its good performance in finite simulations. This new algorithm is important because the existing empirical literature that wanted to follow a likelihood approach was limited to the estimation of linear models with Gaussian innovations. The authors apply their procedure to estimate the structural parameters of the neoclassical growth model. JEL classification: C63, C68, E37 Key words: dynamic equilibrium economies, likelihood function, nonlinear solution methods 1. Introduction This paper presents a method to undertake likelihood based inference in nonlinear dynamic equilibrium models. We show how we can use sequential monte carlo methods to estimate the structural parameters of the model, those describing preferences and technology, and to compare different economies. Both tasks can be implemented from either a Bayesian or a classical perspective. Economist now routinely use dynamic general equilibrium economies to answer quantitative questions. However they employ much less often formal econometrics to take these models to the data. Part of the reason might have been the shortcomings of existing tools. To estimates these economies, the empirical literature has been forced to use either limited-information moment methods or likelihood techniques on linearized versions of the model. This situation is unsatisfactory. Moment procedures may suffer from strong small samples biases and may not use efficiently all the existing information. Linearization techniques depend crucially on the shape of the true policy function being accurately approximated by a linear relation and on the presence of gaussian shocks. The main obstacle for a more standard likelihood-based inference is the difficulty in evaluating the likelihood function implied by a nonlinear dynamic equilibrium economy. Beyond a few particular cases,(1) it is not possible to evaluate this function. Moment methods avoid the problem by moving away from full information approaches to inference. Linearization renounces evaluating the true likelihood function of the model and concentrates instead on the likelihood of an associated, more tractable, linear approximation to the economy. We propose a Sequential Monte Carlo method to solve this problem. We describe how this technique can be applied to evaluate the likelihood function implied by the nonlinear solution of a dynamic equilibrium economy even if the driving shocks of the model are non-normal (although the algorithm is general enough that it can also deal with linear models with or without normal shocks). To do so we borrow from a growing literature on nonlinear filtering(see the seminal paper by Gordon, Salmond and Smith, 1993 and the review of the literature in Doucet, de Freitas and Gordon, 2001). We adapt this know-how to deal with the likelihood functions of dynamic equilibrium models and we show how we get accurate and stable evaluations of the likelihood function. With these evaluations available, the door for likelihood-based inference is open, either by searching for a maximum of the function(Quasi-Maximum Likelihood estimation) or by simulating the posterior distribution of the parameters using a Markov Chain Monte Carlo algorithm(Bayesian estimation). The general idea of the procedure is as follows. First, for a given set of parameter values, we compute the equilibrium policy functions of the model. Since we want to conduct inference in the nonlinear model and not in a linear approximation, we rely on a nonlinear solution method to find the policy functions. …

Maximum Likelihood Estimation of Time-Inhomogeneous Diffusions

We extend the maximum likelihood estimation method of Ait-Sahalia (2002) for time-homogeneous diffusions to time-inhomogeneous ones. We derive a closed-form approximation of the likelihood function for discretely sampled time-inhomogeneous diffusions, and prove that this approximation converges to the true likelihood function and yields consistent parameter estimates. Monte Carlo simulations for several financial models reveal that our method largely outperforms other widely used numerical procedures in approximating the likelihood function. Furthermore, parameter estimates produced by our method are very close to the parameter estimates obtained by maximizing the true likelihood function, and superior to estimates obtained from the Euler approximation.

Maximum likelihood estimation of time-inhomogeneous diffusions

We extend the maximum likelihood estimation method of Ait-Sahalia (Econometrica 70 (2002) 223) for time-homogeneous diffusions to time-inhomogeneous ones. We derive a closed-form approximation of the likelihood function for discretely sampled time-inhomogeneous diffusions, and prove that this approximation converges to the true likelihood function and yields consistent parameter estimates. Monte Carlo simulations for several financial models reveal that our method largely outperforms other widely used numerical procedures in approximating the likelihood function. Furthermore, parameter estimates produced by our method are very close to the parameter estimates obtained by maximizing the true likelihood function, and superior to estimates obtained from the Euler approximation.

Maximum Likelihood Estimation of Discretely Sampled Diffusions: A Closed-form Approximation Approach

When a continuous-time diffusion is observed only at discrete dates, in most cases the transition distribution and hence the likelihood function of the observations is not explicitly computable. Using Hermite polynomials, I construct an explicit sequence of closed-form functions and show that it converges to the true (but unknown) likelihood function. I document that the approximation is very accurate and prove that maximizing the sequence results in an estimator that converges to the true maximum likelihood estimator and shares its asymptotic properties. Monte Carlo evidence reveals that this method outperforms other approximation schemes in situations relevant for financial models.