Parameter Exponential Family Research Articles

Robust estimation of a regression function in exponential families

We observe n pairs of independent (but not necessarily i.i.d.) random variables X1=(W1,Y1),…,Xn=(Wn,Yn) and tackle the problem of estimating the conditional distributions Qi⋆(wi) of Yi given Wi=wi for all i∈{1,…,n}. Even though these might not be true, we base our estimator on the assumptions that the data are i.i.d. and the conditional distributions of Yi given Wi=wi belong to a one parameter exponential family Q¯ with parameter space given by an interval I. More precisely, we pretend that these conditional distributions take the form Qθ(wi)∈Q¯ for some θ that belongs to a VC-class Θ¯ of functions with values in I. For each i∈{1,…,n}, we estimate Qi⋆(wi) by a distribution of the same form, i.e. Qθ̂(wi)∈Q¯, where θ̂=θ̂(X1,…,Xn) is a well-chosen estimator with values in Θ¯. We establish non-asymptotic exponential inequalities for the upper deviations of a Hellinger-type distance between the true conditional distributions of the data and the estimated one based on the exponential family Q¯ and the class of functions Θ¯ we chose. We show that our estimation strategy is robust to model misspecification, contamination and the presence of outliers. Besides, when the data are truly i.i.d., the exponential family Q¯ is suitably parametrized and the conditional distributions Qi⋆(wi) of the form Qθ⋆(wi)∈Q¯ for some unknown Hölderian function θ⋆ with values in I, we prove that the estimator θ̂ of θ⋆ is minimax (up to a logarithmic factor). Finally, we provide an algorithm for calculating θ̂ when Θ¯ is a VC-class of functions of low or moderate dimension and we carry out a simulation study to compare its performance to that of the MLE and median-based estimators. The proof of our main result relies on an upper bound, with explicit numerical constants, on the expectation of the supremum of an empirical process over a VC-subgraph class. This bound can be of independent interest.

Statistics of the two star ERGM

In this paper, we explore the two star Exponential Random Graph Model, which is a two parameter exponential family on the space of simple labeled graphs. We introduce auxiliary variables to express the two star model as a mixture of the β model on networks. Using this representation, we study asymptotic distribution of the number of edges, and the sampling variance of the degrees. In particular, the limiting distribution for the number of edges has similar phase transition behavior to that of the magnetization in the Curie-Weiss Ising model of Statistical Physics. Using this, we show existence of consistent estimates for both parameters. Finally, we prove that the centered partial sum of degrees converges as a process to a Brownian bridge in all parameter domains, irrespective of the phase transition.

Multiple values-inflated time series of counts: modeling and inference based on INGARCH scheme

ABSTRACT In this study, we consider integer-valued autoregressive conditional heteroscedastic (INGARCH) time series models of counts whose conditional density follows an multiple values-inflated one parameter exponential family. We check the stationarity and ergodicity of the models and verify the consistency and asymptotic normality of the conditional maximum likelihood estimator (CMLE) under regularity conditions. Then, as an application, we consider a parameter change test using the cumulative sum (CUSUM) test based on (standardized) residuals and squares of those residuals, then subsequently derive their limiting null distributions. A simulation study is conducted focusing on the 0-1-inflated time series of counts, which affirms the validity of the proposed methods. Moreover, a real data analysis using the number of monthly drug cases in the 32nd police car beat in Pittsburgh is also provided to further strengthen the validity of our methods.

Estimation of P(X ≤ Y) for discrete distributions with non-identical support

Abstract The Uniformly Minimum Variance Unbiased (UMVU) and the Maximum Likelihood (ML) estimations of R = P(X ≤ Y) and the associated variance are considered for independent discrete random variables X and Y. Assuming a discrete uniform distribution for X and the distribution of Y as a member of the discrete one parameter exponential family of distributions, theoretical expressions of such quantities are derived. Similar expressions are obtained when X and Y interchange their roles and both variables are from the discrete uniform distribution. A simulation study is carried out to compare the estimators numerically. A real application based on demand-supply system data is provided.

Most Powerful Test Sequences with Early Stopping Options.

We extended the application of uniformly most powerful tests to sequential tests with different stage-specific sample sizes and critical regions. In the one parameter exponential family, likelihood ratio sequential tests are shown to be uniformly most powerful for any predetermined α-spending function and stage-specific sample sizes. To obtain this result, the probability measure of a group sequential design is constructed with support for all possible outcome events, as is useful for designing an experiment prior to having data. This construction identifies impossible events that are not part of the support. The overall probability distribution is dissected into components determined by the stopping stage. These components are the sub-densities of interim test statistics first described by Armitage, McPherson and Rowe (1969) that are commonly used to create stopping boundaries given an α-spending function and a set of interim analysis times. Likelihood expressions conditional on reaching a stage are given to connect pieces of the probability anatomy together. The reduction of support caused by the adoption of an early stopping rule induces sequential truncation (not nesting) in the probability distributions of possible events. Multiple testing induces mixtures on the adapted support. Even asymptotic distributions of inferential statistics that are typically normal, are not. Rather they are derived from mixtures of truncated multivariate normal distributions.

Exponential family QMLE-based CUSUM test for integer-valued time series

This study considers testing for parameter changes in integer-valued time series models based on one parameter exponential family quasi-maximum likelihood estimates. We employ the cumulative sum (CUSUM) test as a change point test, based on score vectors, residuals, standardized residuals, and squares of residuals, focusing on performance comparison of negative binomial (NB) quasi-maximum likelihood estimate (QMLE)-based tests. Monte Carlo simulations are conducted with Poisson, NB, zero-inflated Poisson and NB, generalized Poisson INGARCH(1,1) models and Poisson and NB INAR(1) models. Real data analysis using the Korean stock price index (KOSPI) 200 is also included for illustration. The findings validate QMLE-based CUSUM tests using standardized residuals for INGARCH models and residuals for INAR models. Zero proportional change is well detected by the test using squares of residuals.

Monitoring Parameter Change for Time Series Models of Counts Based on Minimum Density Power Divergence Estimator.

In this study, we consider an online monitoring procedure to detect a parameter change for integer-valued generalized autoregressive heteroscedastic (INGARCH) models whose conditional density of present observations over past information follows one parameter exponential family distributions. For this purpose, we use the cumulative sum (CUSUM) of score functions deduced from the objective functions, constructed for the minimum power divergence estimator (MDPDE) that includes the maximum likelihood estimator (MLE), to diminish the influence of outliers. It is well-known that compared to the MLE, the MDPDE is robust against outliers with little loss of efficiency. This robustness property is properly inherited by the proposed monitoring procedure. A simulation study and real data analysis are conducted to affirm the validity of our method.

Fairness for Robust Log Loss Classification

Developing classification methods with high accuracy that also avoid unfair treatment of different groups has become increasingly important for data-driven decision making in social applications. Many existing methods enforce fairness constraints on a selected classifier (e.g., logistic regression) by directly forming constrained optimizations. We instead re-derive a new classifier from the first principles of distributional robustness that incorporates fairness criteria into a worst-case logarithmic loss minimization. This construction takes the form of a minimax game and produces a parametric exponential family conditional distribution that resembles truncated logistic regression. We present the theoretical benefits of our approach in terms of its convexity and asymptotic convergence. We then demonstrate the practical advantages of our approach on three benchmark fairness datasets.

DeconvolveR: A G-Modeling Program for Deconvolution and Empirical Bayes Estimation

Empirical Bayes inference assumes an unknown prior density g(θ) has yielded (unobservables) Θ₁, Θ₂, ..., Θ_N, and each Θ_i produces an independent observation X_i from p_i (X_i | Θ_i). The marginal density f_i (X_i) is a convolution of the prior g and p_i. The Bayes deconvolution problem is one of recovering g from the data. Although estimation of g - so called g-modeling - is difficult, the results are more encouraging if the prior g is restricted to lie within a parametric family of distributions. We present a deconvolution approach where g is restricted to be in a parametric exponential family, along with an R package deconvolveR designed for the purpose.

Semiparametric Generalized Linear Models with the gldrm Package

This paper introduces a new algorithm to estimate and perform inferences on a recently proposed and developed semiparametric generalized linear model (glm). Rather than selecting a particular parametric exponential family model, such as the Poisson distribution, this semiparametric glm assumes that the response is drawn from the more general exponential tilt family. The regression coefficients and unspecified reference distribution are estimated by maximizing a semiparametric likelihood. The new algorithm incorporates several computational stability and efficiency improvements over the algorithm originally proposed. In particular, the new algorithm performs well for either small or large support for the nonparametric response distribution. The algorithm is implemented in a new R package called gldrm.

A note on compatibility of conditional autoregressive models

Suppose that, to assess the joint distribution of a random vector (X1,…,Xn), one selects the kernels Q1,…,Qn with Qi to be regarded as a possible conditional distribution for Xi given (Xj:j≠i); Q1,…,Qn are compatible if there exists a joint distribution for (X1,…,Xn) with conditionals Q1,…,Qn. Similarly, Q1,…,Qn are improperly compatible if they can be obtained, according to the usual rule, with an improper distribution in place of a probability distribution. In this paper, compatibility and improper compatibility of Q1,…,Qn are characterized under some assumptions on their functional form. The characterization applies, in particular, if each Qi belongs to a one parameter exponential family. Special attention is paid to Gaussian conditional autoregressive models.

Determination of the optimal sample size for a clinical trial accounting for the population size.

The problem of choosing a sample size for a clinical trial is a very common one. In some settings, such as rare diseases or other small populations, the large sample sizes usually associated with the standard frequentist approach may be infeasible, suggesting that the sample size chosen should reflect the size of the population under consideration. Incorporation of the population size is possible in a decision‐theoretic approach either explicitly by assuming that the population size is fixed and known, or implicitly through geometric discounting of the gain from future patients reflecting the expected population size. This paper develops such approaches. Building on previous work, an asymptotic expression is derived for the sample size for single and two‐arm clinical trials in the general case of a clinical trial with a primary endpoint with a distribution of one parameter exponential family form that optimizes a utility function that quantifies the cost and gain per patient as a continuous function of this parameter. It is shown that as the size of the population, N, or expected size, N∗ in the case of geometric discounting, becomes large, the optimal trial size is O(N1/2) or O(N∗1/2). The sample size obtained from the asymptotic expression is also compared with the exact optimal sample size in examples with responses with Bernoulli and Poisson distributions, showing that the asymptotic approximations can also be reasonable in relatively small sample sizes.

Estimation in exponential families on permutations

Asymptotics of the normalizing constant is computed for a class of one parameter exponential families on permutations which includes Mallows model with Spearmans's Footrule and Spearman's Rank Correlation Statistic. The MLE, and a computable approximation of the MLE are shown to be consistent. The pseudo-likelihood estimator of Besag is shown to be $\sqrt{n}$-consistent. An iterative algorithm (IPFP) is proved to converge to the limiting normalizing constant. The Mallows model with Kendall's Tau is also analyzed to demonstrate flexibility of the tools of this paper.

Empirical Bayes deconvolution estimates

An unknown prior density $g(\theta )$ has yielded realizations $\Theta _1,\ldots ,\Theta _N$. They are unobservable, but each $\Theta _i$ produces an observable value $X_i$ according to a known probability mechanism, such as $X_i\sim {\rm Po}(\Theta _i)$. We wish to estimate $g(\theta )$ from the observed sample $X_1,\ldots ,X_N$. Traditional asymptotic calculations are discouraging, indicating very slow nonparametric rates of convergence. In this article we show that parametric exponential family modelling of $g(\theta )$ can give useful estimates in moderate-sized samples. We illustrate the approach with a variety of real and artificial examples. Covariate information can be incorporated into the deconvolution process, leading to a more detailed theory of generalized linear mixed models.

On the Admissibility of Estimators of Two Ordered Gamma Scale Parameters under Entropy Loss Function

Suppose that a random sample of size ni is drawn from a gamma distribution with known shape parameteri > 0 and unknown scale parameteri > 0, i=1,2, satis- fying 0 < �1 ≤ �2. In estimation of �1 and �2 under the entropy loss function, we consider the class of mixed estimators of �1 and �2. It is shown that a subclass of mixed estimators ofi beats the usual estimators Xi/�i, i=1,2, and the inadmissible estimators in the class of mixed estimators are derived. Also the asymptotic efficiency of mixed estimators relative to the usual estimators are obtained. Finally the results are extended to a subclass of the scale parameter exponential family and the family of transformed chi-square distributions.

Improved estimation of the population parameters when some additional information is available

Estimation of population parameters is considered by several statisticians when additional information such as coefficient of variation, kurtosis or skewness is known. Recently Wencheko and Wijekoon (Stat Papers 46:101–115, 2005) have derived minimum mean square error estimators for the population mean in one parameter exponential families when coefficient of variation is known. In this paper the results presented by Gleser and Healy (J Am Stat Assoc 71:977–981, 1976) and Arnholt and Hebert ( http://interstat.statjournals.net/YEAR/2001/articles/0103002.pdf , 2001) were generalized by considering T (X) as a minimal sufficient estimator of the parametric function g(θ) when the ratio $${\tau^{2}=[ {g(\theta )} ]^{-2}{\rm Var}[ {T(\boldsymbol{X} )} ]}$$ is independent of θ. Using these results the minimum mean square error estimator in a certain class for both population mean and variance can be obtained. When T (X) is complete and minimal sufficient, the ratio τ2 is called “WIJLA” ratio, and a uniformly minimum mean square error estimator can be derived for the population mean and variance. Finally by applying these results, the improved estimators for the population mean and variance of some distributions are obtained.

On estimation error using maximum entropy density estimates

PurposeIn many problems involving decision‐making under uncertainty, the underlying probability model is unknown but partial information is available. In some approaches to this problem, the available prior information is used to define an appropriate probability model for the system uncertainty through a probability density function. When the prior information is available as a finite sequence of moments of the unknown probability density function (PDF) defining the appropriate probability model for the uncertain system, the maximum entropy (ME) method derives a PDF from an exponential family to define an approximate model. This paper, aims to investigate some optimality properties of the ME estimates.Design/methodology/approachFor n&gt;m, when the exact model can be best approximated by one of an infinite number of unknown PDFs from an n parameter exponential family. The upper bound of the divergence distance between any PDF from this family and the m parameter exponential family PDF defined by the ME method are derived. A measure of adequacy of the model defined by ME method is thus provided.FindingsThese results may be used to establish confidence intervals on the estimate of a function of the random variable when the ME approach is employed. Additionally, it is shown that when working with large samples of independent observations, a probability density function (PDF) can be defined from an exponential family to model the uncertainty of the underlying system with measurable accuracy. Finally, a relationship with maximum likelihood estimation for this case is established.Practical implicationsThe so‐called known moments problem addressed in this paper has a variety of applications in learning, blind equalization and neural networks.Originality/valueAn upper bound for error in approximating an unknown density function, f(x) by its ME estimate based on m moment constraints, obtained as a PDF p(x, α) from an m parameter exponential family is derived. The error bound will help us decide if the number of moment constraints is adequate for modeling the uncertainty in the system under study. In turn, this allows one to establish confidence intervals on an estimate of some function of the random variable, X, given the known moments. It is also shown how, when working with a large sample of independent observations, instead of precisely known moment constraints, a density from an exponential family to model the uncertainty of the underlying system with measurable accuracy can be defined. In this case, a relationship to ML estimation is established.

Non-parametric empirical Bayes procedure

Abstract In this paper we introduce an empirical Bayes procedure for estimating an unknown parameter, say θ. This procedure gives the empirical Bayes estimator for θ and its associated minimum posterior risk in closed forms without estimating the unknown prior density function of θ. In such procedure the posterior probability density function of θ is not required. A sufficient statistic for θ with conditional probability density function in the one parameter exponential family is required. Instead of estimating the unknown prior density function, the marginal density function of the sufficient statistic must be estimated. As special cases the empirical Bayes estimators and their respective minimum posterior risks of the failure rate for the exponential distribution, the unknown scale parameters of Weibull and gamma distributions are obtained in simple forms as special cases. Numerical results and a simulation study are introduced to (i) investigate how the number of available past experiments and the sample size of each influence the accuracy of the empirical Bayes estimator, (ii) make a comparison between the presented procedure and the Bayes procedure when the prior probability density function of the parameter θ is gamma.

The Bayes procedure in exponential reliability family models using conjugate convex tent prior family

A Bayes procedure for estimating the unknown parameter indexed to some of one parameter exponential family distributions is presented. In such a procedure, we shall use a new conjugate prior family that we shall call a conjugate convex tent family. A member of this family could be constructed by assuming a little information about the unknown parameter. Some of the needed parameters for constructing prior density function are the values r, p and q. Bayes estimators for using a priori symmetrical convex tent density can be obtained as special cases of the present work. Numerical simulation study is introduced by using the Monte Carlo method. In this study we have investigated the influence of the prior parameters r, p and q on the accuracy of the Bayes estimators.

Improving the EM algorithm for mixtures

One of the estimating equations of the Maximum Likelihood Estimation method, for finite mixtures of the one parameter exponential family, is the first moment equation. This can help considerably in reducing the labor and the cost of calculating the Maximum Likelihood estimates. In this paper it is shown that the EM algorithm can be substantially improved by using this result when applied for mixture models. A short discussion about other methods proposed for the calculation of the Maximum Likelihood estimates are also reported showing that the above findings can help in this direction too.