Exponential Family Models Research Articles

Modified maximum likelihood estimation in one-parameter exponential family models

We propose a new pivotal quantity which is a function of the maximum likelihood estimate of a scalar parameter θ and whose distribution is standard normal excluding terms of order and smaller, where n is the sample size. The proposed pivot is a polynomial transformation of the standardized maximum likelihood estimate of at most third degree. We apply our main result to the one-parameter exponential family model and to a number of special distributions of this family. Some simulation results illustrate the superiority of our pivotal quantity over the usual standardized maximum likelihood estimate with regard to third-order asymptotic theory.

Directional tests for one-sided alternatives in multivariate models

Consider one-sided testing problems for a multivariate exponential family model. Through conditioning or other considerations, the problem oftentimes reduces to testing a null hypothesis that the natural parameter is a zero vector against the alternative that the natural parameter lies in a closed convex cone $\mathscr{C}$. The problems include testing homogeneity of parameters, testing independence in contingency tables, testing stochastic ordering of distributions and many others. A test methodology is developed that directionalizes the usual test procedures such as likelihood ratio, chi square, Fisher, and so on. The methodology can be applied to families of tests where the family is indexed by a size parameter so as to enable nonrandomized testing by $p$-values. For discrete models, a refined family of tests provides a refined grid for better testing by $p$-values. The tests have essential monotonicity properties that are required for admissibility and for desirable power properties. Two examples are given.

An approximation to the modified profile likelihood function

An approximation to the modified profile likelihood function is proposed. This approximation is invariant under interest-respecting reparameterisations, it agrees with the modified profile likelihood function to order O(n -1 ) in the moderate-deviation sense and to order O(n -1 /2) in the large-deviation sense, and it is stable in the sense of conditional inference. For the case of a curved exponential family model this approximation agrees with the one proposed by Barndorff-Nielsen (1995). A general expression for the approximate modified profile likelihood function is given for nonlinear regression models and for generalised linear models with known dispersion parameter.

Miscellanea. Estimation in parameter-redundant models

The likelihood surface resulting from a parameter-redundant stochastic model is maximised along a completely flat ridge. This ridge may be orthogonal to some parameter axes, so that these parameters have unique maximum likelihood estimates. For exponential-family models, we show how to determine which parameter combinations are estimable. The approach requires the calculation of a derivative matrix and the determination of its null space, both of which are readily achieved in computer algebra packages. Illustrative examples are drawn from the areas of compartment modelling and ring-recovery analysis.

On the LRT for the multidimensional exponential family under restricted alternative space

The LRT for testing a simple hypothesis against one sided alternatives will be derived for the exponential family model and it will be shown that it is admissible.

Bias reduction in one-parameter exponential family models

This paper gives closed-form expressions for the second and third order biases of maximum likelihood estimates for a number of distributions in the one-parameter exponential family. Approximations based on asymptotic expansions for some bias-corrections that require the evaluation of unusual functions and long expressions are given. A graphical analysis is also performed to show how such biases and the mean squared errors of both the maximum likelihood estimate and its bias-corrected version vary with the parameter that indexes some distributions. Finally, we present simulation results comparing the performance of the maximum likelihood estimator and its bias-corrected version.

Exact sample size calculations for exponential family models

A simple general formula is given for exact calculations of sample size for any member of the linear exponential family, which includes generalized linear models with known or fixed dispersion parameter.

Exponential and bayesian conjugate families: Review and extensions

The notion of a conjugate family of distributions plays a very important role in the Bayesian approach to parametric inference. One of the main features of such a family is that it is closed under sampling, but a conjugate family often provides prior distributions which are tractable in various other respects. This paper is concerned with the properties of conjugate families for exponential family models. Special attention is given to the class of natural exponential families having a quadratic variance function, for which the theory is particularly fruitful. Several classes of conjugate families have been considered in the literature and here we describe some of their most interesting features. Relationships between such classes are also discussed. Our aim is to provide a unified approach to the theory of conjugate families for exponential family likelihoods. An important aspect of the theory concerns reparameterisations of the exponential family under consideration. We briefly review the concept of a conjugate parameterisation, which provides further insight into many of the properties discussed throughout the paper. Finally, further implications of these results for Bayesian conjugate analysis of exponential families are investigated.

Estimating Mixing Densities in Exponential Family Models for Discrete Variables

This paper is concerned with estimating a mixing density gusing a random sample from the mixture distribution f(x)=∫f x | θ)g(θ)dθ where f(· | θ)is a known discrete exponen tial family of density functions. Recently two techniques for estimatingghave been proposed. The first uses Fourier analysis and the method of kernels and the second uses orthogonal polynomials. It is known that the first technique is capable of yielding estimators that achieve (or almost achieve) the minimax convergence rate. We show that this is true for the technique based on orthogonal polynomials as well. The practical implementation of these estimators is also addressed. Computer experiments indicate that the kernel estimators give somewhat disappoint ing finite sample results. However, the orthogonal polynomial estimators appear to do much better. To improve on the finite sample performance of the orthogonal polynomial estimators, a way of estimating the optimal truncation parameter is proposed. The resultant estimators retain the convergence rates of the previous estimators and a Monte Carlo finite sample study reveals that they perform well relative to the ones based on the optimal truncation parameter.

Estimating Mixing Densities in Exponential Family Models for Discrete Variables

This paper is concerned with estimating a mixing density gusing a random sample from the mixture distribution f(x)=∫f x | θ)g(θ)dθ where f(· | θ)is a known discrete exponen tial family of density functions. Recently two techniques for estimatingghave been proposed. The first uses Fourier analysis and the method of kernels and the second uses orthogonal polynomials. It is known that the first technique is capable of yielding estimators that achieve (or almost achieve) the minimax convergence rate. We show that this is true for the technique based on orthogonal polynomials as well. The practical implementation of these estimators is also addressed. Computer experiments indicate that the kernel estimators give somewhat disappoint ing finite sample results. However, the orthogonal polynomial estimators appear to do much better. To improve on the finite sample performance of the orthogonal polynomial estimators, a way of estimating the optimal truncation parameter is proposed. The resultant estimators retain the convergence rates of the previous estimators and a Monte Carlo finite sample study reveals that they perform well relative to the ones based on the optimal truncation parameter.

Moment-based oscillation properties of mixture models

Consider finite mixture models of the form $g(x; Q) = \int f(x; \theta) dQ(\theta)$, where f is a parametric density and Q is a discrete probability measure. An important and difficult statistical problem concerns the determination of the number of support points (usually known as components) of Q from a sample of observations from g. For an important class of exponential family models we have the following result: if P has more than p components and Q is an appropriately chosen p-component approximation of P, then $g(x; P) - g(x; Q)$ demonstrates a prescribed sign change behavior, as does the corresponding difference in the distribution functions. These strong structural properties have implications for diagnostic plots for the number of components in a finite mixture.

Discussion of the Paper by Lee and Nelder

Discussion of the Paper by Lee and Nelder

Second- and third-order bias reduction for one-parameter family models

In this paper we derive second- and third-order bias-corrected maximum likelihood estimates in general uniparametric models. We compare the corrected estimates and the usual maximum likelihood estimate in terms of their mean squared errors. We also obtain closed-form expressions for bias-corrected estimates in one-parameter exponential family models. Our results cover many important and commonly used distributions. Simulation results are also given.

The Cost of Adding Parameters to a Model

SUMMARY For a general regression model with n independent observations we consider the variance of the estimate of a quantity of interest under two scenarios. One scenario is where all the parameters are estimated from the data; the other scenario is where a subset of the parameters is assumed known at their true values and the remaining parameters are estimated. We focus on quantities of interest which are defined on the scale of the response variable. We show that, under certain conditions, the ratio of a weighted sum across the design points of the variance of the quantity of interest is given by q/p, where q and p are the number of free parameters in the two scenarios. Thus, in this average sense, the inflation in variance associated with adding parameters, also interpreted as the cost of adding parameters to a model, is directly proportional to the number of parameters. We study models involving power transformations, non-linear models and exponential family models.

Improved score tests for one-parameter exponential family models

Under suitable regularity conditions, an improved score test was derived by Cordeiro and Ferrari (1991). The test is based on a corrected score statistic which has a chi-squared distribution to order n −1 under the null hypothesis, where n is the sample size. In this paper we follow their approach and obtain a Bartlett-corrected score statistic for testing H o : θ = θ ( o) , where θ is the scalar parameter of a one-parameter exponential family model. We apply our main result to a number of special cases and derive approximations for corrections that involve unusual functions. We also obtain Bartlett-type corrections for natural exponential families.

MARKOV CHAIN MONTE CARLO METHODS FOR MODELING THE SPATIAL PATTERN OF DISEASE SPREAD IN BELL PEPPER

With exponential family models for dependent data, such as the autologistic model for binary spatial lattice data, maximum likelihood estimates can be obtained using Markov chain sampling methods by simulating an ergodic Markov chain which converges weakly to the equilibrium distribution of the model. This Markov chain Monte Carlo maximum likelihood (MCMCML) procedure provides a competitor to the usual pseudolikelihood estimation method often used for modeling discrete lattice data. Within this MCMCML framework, it is also possible to conduct formal inference using MCMC analogues to the usual likelihood ratio, Wald, and Lagrange multiplier tests, for which the asymptotic distributions are known subject to some mild regularity conditions. Here, the MCMC methodology will be discussed as it pertains to the autologistic model for binary data and will be used to model the spatial pattern of disease spread in bell pepper caused by the pathogen Phytophthora capsici.

Projected score methods for approximating conditional scores

SUMMARY This paper extends the projected score methods of Small & McLeish (1989). It is shown that the conditional score function may be approximated, with arbitrarily small stochastic error, in terms of a natural basis for the space of centred likelihood ratios. The utility of using this basis is established by identifying a U-statistic representation theorem and a class of expectation identities for the basis elements, making higher order asymptotics more tractable. The results are applied to a canonical exponential family model, where it is shown that the projected scores with estimated nuisance parameters can provide an accurate approximation to the conditional score function.

A Simple and Accurate Method for Approximate Conditional Inference Applied to Exponential Family Models

SUMMARY A simple method for approximate conditional inference is described. The methodology is applied to natural exponential family models where it is shown to provide accurate approximations to fully conditional estimates. The approximation technique can be applied much more generally than in this particular class of models. The technique depends only on the construction of certain 'projected scores' derived from higher order likelihood derivatives and their covariances and so can be used in many problems where it is relevant to control for the estimation of nuisance parameters.

Bartlett corrections for one-parameter exponential family models

In this paper we derive a general closed-form expression for the Bartlett correction for the test of Ho:λ = λ(0), where λ is a scalar parameter of a one-parameter exponential family model. Our results are general enough to cover many important and commonly used distributions. Several special cases and classes of variance functions of considerable importance are discussed, and some approximations based on asymptotic expansions are given. We also use a graphical analysis to examine how the correction varies with λ in some special cases. Stimulation results are also given.

On Estimating Mixing Densities in Discrete Exponential Family Models

This paper concerns estimating a mixing density function $g$ and its derivatives based on iid observations from $f(x) = \int f(x \mid \theta)g(\theta)d\theta$, where $f(x \mid \theta)$ is a known exponential family of density functions with respect to the counting measure on the set of nonnegative integers. Fourier methods are used to derive kernel estimators, upper bounds for their rate of convergence and lower bounds for the optimal rate of convergence. If $f(x \mid \theta_0) \geq \varepsilon^{x + 1} \forall x$, for some positive numbers $\theta_0$ and $\varepsilon$, then our estimators achieve the optimal rate of convergence $(\log n)^{-\alpha + m}$ for estimating the $m$th derivative of $g$ under a Lipschitz condition of order $\alpha > m$. The optimal rate of convergence is almost achieved when $(x!)^\beta f(x \mid \theta_0) \geq \varepsilon^{x + 1}$. Estimation of the mixing distribution function is also considered.