True Maximum Likelihood Estimator Research Articles

Enhanced Laplace approximation

The Laplace approximation has been proposed as a method for approximating the marginal likelihood of statistical models with latent variables. However, the approximate maximum likelihood estimators derived from the Laplace approximation are often biased for binary or temporally and/or spatially correlated data. Additionally, the corresponding Hessian matrix tends to underestimates the standard errors of these approximate maximum likelihood estimators. While higher-order approximations have been suggested, they are not applicable to complex models, such as correlated random effects models, and fail to provide consistent variance estimators. In this paper, we propose an enhanced Laplace approximation that provides the true maximum likelihood estimator and its consistent variance estimator. We study its relationship with the variational Bayes method. We also define a new restricted maximum likelihood estimator for estimating dispersion parameters and study their asymptotic properties. Enhanced Laplace approximation generally demonstrates how to obtain the true restricted maximum likelihood estimators and their variance estimators. Our numerical studies indicate that the enhanced Laplace approximation provides a satisfactory maximum likelihood estimator and restricted maximum likelihood estimator, as well as their variance estimators in the frequentist perspective. The maximum likelihood estimator and restricted maximum likelihood estimator can be also interpreted as the posterior mode and marginal posterior mode under flat priors, respectively. Furthermore, we present some comparisons with Bayesian procedures under different priors.

Asymptotic accuracy of the saddlepoint approximation for maximum likelihood estimation

The saddlepoint approximation gives an approximation to the density of a random variable in terms of its moment generating function. When the underlying random variable is itself the sum of n unobserved i.i.d. terms, the basic classical result is that the relative error in the density is of order 1/n. If instead the approximation is interpreted as a likelihood and maximised as a function of model parameters, the result is an approximation to the maximum likelihood estimate (MLE) that can be much faster to compute than the true MLE. This paper proves the analogous basic result for the approximation error between the saddlepoint MLE and the true MLE: subject to certain explicit identifiability conditions, the error has asymptotic size O(1/n2) for some parameters and O(1/n3/2) or O(1/n) for others. In all three cases, the approximation errors are asymptotically negligible compared to the inferential uncertainty. The proof is based on a factorisation of the saddlepoint likelihood into an exact and approximate term, along with an analysis of the approximation error in the gradient of the log-likelihood. This factorisation also gives insight into alternatives to the saddlepoint approximation, including a new and simpler saddlepoint approximation, for which we derive analogous error bounds. As a corollary of our results, we also obtain the asymptotic size of the MLE approximation error when the saddlepoint approximation is replaced by the normal approximation.

Joint Frame Detection and Channel Parameter Estimation for OOK Free-Space Optical Communications

We consider a free-space optical (FSO) communication link for packet-based transmissions over a turbulence fading channel. The modulation format is on-off keying (OOK) and a unique-word (UW) composed by a known synch pattern is periodically inserted in the data stream to identify the start of frame. Since an avalanche photo-diode (APD) is used for direct detection of the OOK symbols, the photocurrent signal provided by the APD is plagued by a mixture of thermal and shot noise with signal-dependent power. Our goal is the detection of the UW position in the received stream, along with the estimation of the unknown channel attenuation and noise variances. The aforementioned problem has recently been studied by ignoring any information conveyed by data symbols surrounding the UW. In this work, further investigation is conducted in order to determine the maximum likelihood (ML) solution that, in addition to the UW, exploits all the information-bearing symbols belonging to the observation window. The relevant Cramér-Rao bound (CRB) is also evaluated to establish the ultimate accuracy achievable in the estimation process. Since the true ML estimator leads to a computationally intractable multi-dimensional optimization problem, we develop suitable approximations that enable accurate frame synchronization with affordable complexity. We also present an iterative solution to refine the channel and noise power estimates provided by the UW detection procedure. Numerical simulations demonstrate the superiority of the proposed synchronization and estimation schemes with respect to existing alternatives.

Nonparametric estimation of the lifetime and disease onset distributions for a survival-sacrifice model

In carcinogenicity experiments with animals where the tumor is not palpable it is common to observe only the time of death of the animal, the cause of death (the tumor or another independent cause, as sacrifice) and whether the tumor was present at the time of death. These last two indicator variables are evaluated after an autopsy. Defining the non-negative variables $T_{1}$ (time of tumor onset), $T_{2}$ (time of death from the tumor) and $C$ (time of death from an unrelated cause), we observe $(Y,\Delta_{1},\Delta_{2})$, where $Y=\min\left\{T_{2},C\right\}$, $\Delta_{1}=1_{\left\{T_{1}\leq C\right\}}$, and $\Delta_{2}=1_{\left\{T_{2}\leq C\right\}}$. The random variables $T_{1}$ and $T_{2}$ are independent of $C$ and have a joint distribution such that $P(T_{1}\leq T_{2})=1$. Some authors call this model a “survival-sacrifice model”. [20] (generally to be denoted by LJP (1997)) proposed a Weighted Least Squares estimator for $F_{1}$ (the marginal distribution function of $T_{1}$), using the Kaplan-Meier estimator of $F_{2}$ (the marginal distribution function of $T_{2}$). The authors claimed that their estimator is more efficient than the MLE (maximum likelihood estimator) of $F_{1}$ and that the Kaplan-Meier estimator is more efficient than the MLE of $F_{2}$. However, we show that the MLE of $F_{1}$ was not computed correctly, and that the (claimed) MLE estimate of $F_{1}$ is even undefined in the case of active constraints. In our simulation study we used a primal-dual interior point algorithm to obtain the true MLE of $F_{1}$. The results showed a better performance of the MLE of $F_{1}$ over the weighted least squares estimator in LJP (1997) for points where $F_{1}$ is close to $F_{2}$. Moreover, application to the model, used in the simulation study of LJP (1997), showed smaller variances of the MLE estimators of the first and second moments for both $F_{1}$ and $F_{2}$, and sample sizes from 100 up to 5000, in comparison to the estimates, based on the weighted least squares estimator for $F_{1}$, proposed in LJP (1997), and the Kaplan-Meier estimator for $F_{2}$. R scripts are provided for computing the estimates either with the primal-dual interior point method or by the EM algorithm. In spite of the long history of the model in the biometrics literature (since about 1982), basic properties of the real maximum likelihood estimator (MLE) were still unknown. We give necessary and sufficient conditions for the MLE (Theorem 3.1), as an element of a cone, where the number of generators of the cone increases quadratically with sample size. From this and a self-consistency equation, turned into a Volterra integral equation, we derive the consistency of the MLE (Theorem 4.1). We conjecture that (under some natural conditions) one can extend the methods, used to prove consistency, to proving that the MLE is $\sqrt{n}$ consistent for $F_{2}$ and cube root $n$ convergent for $F_{1}$, but this has presently not yet been proved.

Approximate Maximum-likelihood Identification of Linear Systems from Quantized Measurements

We analyze likelihood-based identification of systems that are linear in the parameters from quantized output data; in particular, we propose a method to find approximate maximum-likelihood and maximum-a-posteriori solutions. The method consists of appropriate least-squares projections of the middle point of the active quantization intervals. We show that this approximation maximizes a variational approximation of the likelihood and we provide an upper bound for the approximation error. In a simulation study, we compare the proposed method with the true maximum-likelihood estimate of a finite impulse response model.

Tightness of the maximum likelihood semidefinite relaxation for angular synchronization

Maximum likelihood estimation problems are, in general, intractable optimization problems. As a result, it is common to approximate the maximum likelihood estimator (MLE) using convex relaxations. In some cases, the relaxation is tight: it recovers the true MLE. Most tightness proofs only apply to situations where the MLE exactly recovers a planted solution (known to the analyst). It is then sufficient to establish that the optimality conditions hold at the planted signal. In this paper, we study an estimation problem (angular synchronization) for which the MLE is not a simple function of the planted solution, yet for which the convex relaxation is tight. To establish tightness in this context, the proof is less direct because the point at which to verify optimality conditions is not known explicitly. Angular synchronization consists in estimating a collection of $n$ phases, given noisy measurements of the pairwise relative phases. The MLE for angular synchronization is the solution of a (hard) non-bipartite Grothendieck problem over the complex numbers. We consider a stochastic model for the data: a planted signal (that is, a ground truth set of phases) is corrupted with non-adversarial random noise. Even though the MLE does not coincide with the planted signal, we show that the classical semidefinite relaxation for it is tight, with high probability. This holds even for high levels of noise.

Maximum likelihood estimation in nonlinear structured fisheries models using survey and catch-at-age data

Age-structured population dynamics models play an important role in fisheries assessments. Such models have traditionally been estimated using crude likelihood approximations or more recently using Bayesian techniques. We contribute to this literature with three main messages. Firstly, we demonstrate how to estimate such models efficiently by simulated maximum likelihood using Laplace importance samplers for the likelihood function. Secondly, we demonstrate how simulated maximum likelihood estimates may be validated using different importance samplers known to approach the exact likelihood function in different regions of the parameter space. Thirdly, we show that our method works in practice by Monte Carlo simulations using parameter values as estimated from data on the Northeast Arctic cod ( Gadus morhua ) stock. The simulations suggest that we are able to recover the unknown true maximum likelihood estimates using moderate importance sample sizes and show that we are able to adequately recover the true parameter values.

Estimating affine multifactor term structure models using closed-form likelihood expansions

We develop and implement a technique for closed-form maximum likelihood estimation (MLE) of multifactor affine yield models. We derive closed-form approximations to likelihoods for nine Dai and Singleton (2000) affine models. Simulations show our technique very accurately approximates true (but infeasible) MLE. Using US Treasury data, we estimate nine affine yield models with different market price of risk specifications. MLE allows non-nested model comparison using likelihood ratio tests; the preferred model depends on the market price of risk. Estimation with simulated and real data suggests our technique is much closer to true MLE than Euler and quasi-maximum likelihood (QML) methods.

Fixed-domain asymptotic properties of tapered maximum likelihood estimators

When the spatial sample size is extremely large, which occurs in many environmental and ecological studies, operations on the large covariance matrix are a numerical challenge. Covariance tapering is a technique to alleviate the numerical challenges. Under the assumption that data are collected along a line in a bounded region, we investigate how the tapering affects the asymptotic efficiency of the maximum likelihood estimator (MLE) for the microergodic parameter in the Matern covariance function by establishing the fixed-domain asymptotic distribution of the exact MLE and that of the tapered MLE. Our results imply that, under some conditions on the taper, the tapered MLE is asymptotically as efficient as the true MLE for the microergodic parameter in the Matern model.

Estimating parameters in diffusion processes using an approximate maximum likelihood approach

We present an approximate Maximum Likelihood estimator for univariate Ito stochastic differential equations driven by Brownian motion, based on numerical calculation of the likelihood function. The transition probability density of a stochastic differential equation is given by the Kolmogorov forward equation, known as the Fokker-Planck equation. This partial differential equation can only be solved analytically for a limited number of models, which is the reason for applying numerical methods based on higher order finite differences. The approximate likelihood converges to the true likelihood, both theoretically and in our simulations, implying that the estimator has many nice properties. The estimator is evaluated on simulated data from the Cox-Ingersoll-Ross model and a non-linear extension of the Chan-Karolyi-Longstaff-Sanders model. The estimates are similar to the Maximum Likelihood estimates when these can be calculated and converge to the true Maximum Likelihood estimates as the accuracy of the numerical scheme is increased. The estimator is also compared to two benchmarks; a simulation-based estimator and a Crank-Nicholson scheme applied to the Fokker-Planck equation, and the proposed estimator is still competitive.

Parametric estimation of discretely sampled Gamma-OU processes

The stationary Gamma-OU processes are recommended to be the volatility of the financial assets. A parametric estimation for the Gamma-OU processes based on the discrete observations is considered in this paper. The estimator of an intensity parameter λ and its convergence result are given, and the simulations show that the estimation is quite accurate. Assuming that the parameter λ is estimated, the maximum likelihood estimation of shape parameter c and scale parameter α, whose likelihood function is not explicitly computable, is considered. By means of the Gaver-Stehfest algorithm, we construct an explicit sequence of approximations to the likelihood function and show that it converges the true (but unknown) one. Maximizing the sequence results in an estimator that converges to the true maximum likelihood estimator and the approximation shares the asymptotic properties of the true maximum likelihood estimator. Some simulation experiments reveal that this method is still quite accurate in most of rational situations for the background of volatility.

Maximum-Likelihood Estimation of Discretely Sampled Diffusions: A Closed-Form Approach

When a continuous-time diffusion is observed only at discrete dates, not necessarily close together, the likelihood function of the observations is in most cases not explicitly computable. Researchers have relied on simulations of sample paths in between the observations points, or numerical solutions of partial differential equations, to obtain estimates of the function to be maximized. By contrast, we construct a sequence of fully explicit functions which we show converge under very general conditions, including non-ergodicity, to the true (but unknown) likelihood function of the discretely-sampled diffusion. We document that the rate of convergence of the sequence is extremely fast for a number of examples relevant in finance. We then show that maximizing the sequence instead of the true function results in an estimator which converges to the true maximum-likelihood estimator and shares its asymptotic properties of consistency, asymptotic normality and efficiency. Applications to the valuation of derivative securities are also discussed.

On the Nature of SEM Estimates of ARMA Parameters

Several papers have been devoted to the use of structural equation modeling (SEM) software in fitting autoregressive moving average (ARMA) models to a univariate series observed in a single subject. Van Buuren (1997) went beyond specification and examined the nature of the estimates obtained with SEM software. Although the results were mixed, he concluded that these parameter estimates resemble true maximum likelihood (ML) estimates. Molenaar (1999) argued that the negative findings for pure moving average models might be due to the absence of invertibility constraints in Van Buuren's simulation experiment. The aim of this article is to (a) reexamine the nature of SEM estimates of ARMA parameters; (b) replicate Van Buuren's simulation experiment in light of Molenaar's comment; and (c) examine the behavior of the log-likelihood ratio test. We conclude that estimates of ARMA parameters obtained with SEM software are identical to those obtained by univariate stochastic model preliminary estimation, and are not true ML estimates. Still, these estimates, which may be viewed as moment estimates, have the same asymptotic properties as ML estimates for pure autoregressive (AR) processes. For pure moving average (MA) processes, they are biased and less efficient. The estimates from SEM software for mixed processes seem to have the same asymptotic properties as ML estimates. Furthermore, the log-likelihood ratio is reliable for pure AR processes, but this is not the case for pure MA processes. For mixed processes, the behavior of the log-likelihood ratio varies, and in this case these statistics should be handled with caution.

Maximum Likelihood Estimation of Discretely Sampled Diffusions: A Closed-form Approximation Approach

When a continuous-time diffusion is observed only at discrete dates, in most cases the transition distribution and hence the likelihood function of the observations is not explicitly computable. Using Hermite polynomials, I construct an explicit sequence of closed-form functions and show that it converges to the true (but unknown) likelihood function. I document that the approximation is very accurate and prove that maximizing the sequence results in an estimator that converges to the true maximum likelihood estimator and shares its asymptotic properties. Monte Carlo evidence reveals that this method outperforms other approximation schemes in situations relevant for financial models.

Pointwise and functional approximations in Monte Carlo maximum likelihood estimation

We consider the use of Monte Carlo methods to obtain maximum likelihood estimates for random effects models and distinguish between the pointwise and functional approaches. We explore the relationship between the two approaches and compare them with the EM algorithm. The functional approach is more ambitious but the approximation is local in nature which we demonstrate graphically using two simple examples. A remedy is to obtain successively better approximations of the relative likelihood function near the true maximum likelihood estimate. To save computing time, we use only one Newton iteration to approximate the maximiser of each Monte Carlo likelihood and show that this is equivalent to the pointwise approach. The procedure is applied to fit a latent process model to a set of polio incidence data. The paper ends by a comparison between the marginal likelihood and the recently proposed hierarchical likelihood which avoids integration altogether.

Estimating the sky map in gamma-ray astronomy with a Compton telescope

The author offers a formal mathematical formulation of the problem of estimating the sky-map from low-level Compton telescope data. A structured approach is used to derive an iterative algorithm, termed the EM ML (expectation-maximization maximum-likelihood) algorithm, which yields the rigorously optimal estimate of the sky-map given the formulation. The EM ML algorithm converges to the global maximum of the likelihood function, resulting in a true maximum likelihood estimate of the sky map. The result is rigorously optimal given the Poisson assumption. Initial Monte Carlo simulations indicate that maximum likelihood estimation of the sky map may offer improved contrast and all ability to resolve multiple sources within a diffuse background. The computational requirements of the EM ML algorithm are, however, roughly 60 times greater than that of the event circle method. >

Pseudolikelihood Estimation for Social Networks

Abstract Interest in log-linear modeling for social-network data has grown steadily since Holland and Leinhardt (1981) proposed their p 1 model. That model was designed for a single binary relationship (directed graph) representing interactions between individuals. It assumed that interactions between pairs of individuals are mutually independent. Subsequent work has extended the model in various ways, including block-modeling and the case of dependence between pairs of individuals. In empirical work it would often be desirable to fit a wide variety of these models, as the differences in predictions or goodness of fit are likely to provide insights into the data. This has not been common practice, however, because estimation for some of the models has been difficult, and the maximum likelihood schemes developed for others involve different computer programs not always available in standard packages. The focus of this article is on a general estimation technique that maximizes the pseudolikelihood, the product of the probabilities of the binary variables, with each probability conditional on the rest of the data. The method is shown to be equivalent to a weighted least squares procedure and thus can be carried out with standard computer packages. In cases where true maximum likelihood estimation is available for comparison the two methods seem to work about equally well. The pseudolikelihood estimation is used in an example where the fits of a large number of different models are compared. Some of these models, such as various Markov block models, have not previously been proposed. In this example (as in others considered) it appears that the p 1-type models are overparameterized, and that much more parsimonious models give tolerable fits.

On the exact maximum likelihood estimation of Gaussian autoregressive processes

It is shown that the true maximum-likelihood estimates of a Gaussian zero-mean AR process can be calculated. The steps toward their obtainment with standard subroutines are calculating extrema are given. The likelihood of the data is maximized in contrast to current approximate methods. However, the computational cost is significantly higher. Also, the order p of the process is presumed known.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Simple Parallel Hierarchical and Relaxation Algorithms for Segmenting Noncausal Markovian Random Fields

The modeling and segmentation of images by MRF's (Markov random fields) is treated. These are two-dimensional noncausal Markovian stochastic processes. Two conceptually new algorithms are presented for segmenting textured images into regions in each of which the data are modeled as one of C MRF's. The algorithms are designed to operate in real time when implemented on new parallel computer architectures that can be built with present technology. A doubly stochastic representation is used in image modeling. Here, a Gaussian MRF is used to model textures in visible light and infrared images, and an autobinary (or autoternary, etc.) MRF to model a priori information about the local geometry of textured image regions. For image segmentation, the true texture class regions are treated either as a priori completely unknown or as a realization of a binary (or ternary, etc.) MRF. In the former case, image segmentation is realized as true maximum likelihood estimation. In the latter case, it is realized as true maximum a posteriori likelihood segmentation. In addition to providing a mathematically correct means for introducing geometric structure, the autobinary (or ternary, etc.) MRF can be used in a generative mode to generate image geometries and artificial images, and such simulations constitute a very powerful tool for studying the effects of these models and the appropriate choice of model parameters. The first segmentation algorithm is hierarchical and uses a pyramid-like structure in new ways that exploit the mutual dependencies among disjoint pieces of a textured region.

Recursive maximum likelihood estimation of autoregressive processes

A new method of autoregressive parameter estimation is presented. The technique is a closer approximation to the true maximum likelihood estimator than that obtained using linear prediction techniques. The advantage of the new algorithm is mainly for short data records and/or sharply peaked spectra. Simulation results indicate that the parameter bias as well as the variance is reduced over the Yule-Walker and the forward-backward approaches of linear prediction. Also, spectral estimates exhibit more resolution and less spurious peaks. A stable all-pole filter estimate is guaranteed. The algorithm operates in a recursive model order fashion, which allows one to successively fit higher order models to the data.