James-Stein Estimator Research Articles

Minimum Message Length shrinkage estimation

This note considers estimation of the mean of a multivariate Gaussian distribution with known variance within the Minimum Message Length (MML) framework. Interestingly, the resulting MML estimator exactly coincides with the positive-part James–Stein estimator under the choice of an uninformative prior. A new approach for estimating parameters and hyperparameters in general hierarchical Bayes models is also presented.

Integral Inequality for Minimaxity in the Stein Problem

In the estimation of a multivariate normal mean, it is shown that the problem of deriving shrinkage estimators improving on the maximum likelihood estimator can be reduced to that of solving an integral inequality. The integral inequality not only provides a more general condition than a conventional differential inequality studied in the literature, but also handles non-differentiable or discontinuous estimators. The paper also gives general conditions on prior distributions such that the resulting generalized Bayes estimators are minimax. Finally, a simple proof for constructing a class of estimators improving on the James-Stein estimator is given based on the integral expression of the risk.

Shrinkage estimation in general linear models

We propose a James–Stein-type shrinkage estimator for the parameter vector in a general linear model when it is suspected that some of the parameters may be restricted to a subspace. The James–Stein estimator is shown to demonstrate asymptotically superior risk performance relative to the conventional least squares estimator under quadratic loss. An extensive simulation study based on a multiple linear regression model and a logistic regression model further demonstrates the improved performance of this James–Stein estimator in finite samples. The application of this new estimator is illustrated using Ontario newborn infants data spanning four fiscal years.

Estimation of a mean vector under quartic loss

Let X = ( X 1 , … , X p ) be a p-variate normal random vector with unknown mean θ = ( θ 1 , … , θ p ) and identity covariance matrix. Estimators δ = ( δ 1 , … , δ p ) of θ are considered under the quartic loss ∑ i = 1 p ( δ i - θ i ) 4 . For p ⩾ 3 , we develop sufficient conditions on δ ( X ) = X + g ( X ) to improve upon the usual estimator δ 0 ( X ) = X . To this end, we yield an unbiased estimator O g ( X ) of the risk difference between δ ( X ) and δ 0 ( X ) . An interesting feature is that, to obtain adequate dominating estimators, O g ( X ) is used in two ways. First, we search estimators such that O g ( x ) ⩽ 0 for any x ∈ R p , which guarantees the desired domination. Then, to enlarge the class of improved estimators, we investigate conditions for which this inequality is satisfied in mean, that is, E θ [ O g ( X ) ] ⩽ 0 . In particular, no James–Stein estimator satisfies O g ( X ) < 0 for all X, but E θ [ O g ( X ) ] < 0 for p ⩾ 5 and a shrinkage factor 0 < a ⩽ 2 ( p - 4 ) .

Estimation, prediction and the Stein phenomenon under divergence loss

We consider two problems: (1) estimate a normal mean under a general divergence loss introduced in [S. Amari, Differential geometry of curved exponential families — curvatures and information loss, Ann. Statist. 10 (1982) 357–387] and [N. Cressie, T.R.C. Read, Multinomial goodness-of-fit tests, J. Roy. Statist. Soc. Ser. B. 46 (1984) 440–464] and (2) find a predictive density of a new observation drawn independently of observations sampled from a normal distribution with the same mean but possibly with a different variance under the same loss. The general divergence loss includes as special cases both the Kullback–Leibler and Bhattacharyya–Hellinger losses. The sample mean, which is a Bayes estimator of the population mean under this loss and the improper uniform prior, is shown to be minimax in any arbitrary dimension. A counterpart of this result for predictive density is also proved in any arbitrary dimension. The admissibility of these rules holds in one dimension, and we conjecture that the result is true in two dimensions as well. However, the general Baranchick [A.J. Baranchick, a family of minimax estimators of the mean of a multivariate normal distribution, Ann. Math. Statist. 41 (1970) 642–645] class of estimators, which includes the James–Stein estimator and the Strawderman [W.E. Strawderman, Proper Bayes minimax estimators of the multivariate normal mean, Ann. Math. Statist. 42 (1971) 385–388] class of estimators, dominates the sample mean in three or higher dimensions for the estimation problem. An analogous class of predictive densities is defined and any member of this class is shown to dominate the predictive density corresponding to a uniform prior in three or higher dimensions. For the prediction problem, in the special case of Kullback–Leibler loss, our results complement to a certain extent some of the recent important work of Komaki [F. Komaki, A shrinkage predictive distribution for multivariate normal observations, Biometrika 88 (2001) 859–864] and George, Liang and Xu [E.I. George, F. Liang, X. Xu, Improved minimax predictive densities under Kullbak–Leibler loss, Ann. Statist. 34 (2006) 78–92]. While our proposed approach produces a general class of predictive densities (not necessarily Bayes, but not excluding Bayes predictors) dominating the predictive density under a uniform prior. We show also that various modifications of the James–Stein estimator continue to dominate the sample mean, and by the duality of estimation and predictive density results which we will show, similar results continue to hold for the prediction problem as well.

Methods for improvement in estimation of a normal mean matrix

This paper is concerned with the problem of estimating a matrix of means in multivariate normal distributions with an unknown covariance matrix under invariant quadratic loss. It is first shown that the modified Efron–Morris estimator is characterized as a certain empirical Bayes estimator. This estimator modifies the crude Efron–Morris estimator by adding a scalar shrinkage term. It is next shown that the idea of this modification provides a general method for improvement of estimators, which results in the further improvement on several minimax estimators. As a new method for improvement, an adaptive combination of the modified Stein and the James–Stein estimators is also proposed and is shown to be minimax. Through Monte Carlo studies of the risk behaviors, it is numerically shown that the proposed, combined estimator inherits the nice risk properties of both individual estimators and thus it has a very favorable risk behavior in a small sample case. Finally, the application to a two-way layout MANOVA model with interactions is discussed.

Asymptotic Expansion of the Coverage Probability of James–Stein Estimators

This paper provides a new approach to the asymptotic expansion construction of the coverage probability of the confidence sets recentered in [W. James and C. Stein, Estimation with quadratic loss, in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, Univ. California Press, Berkeley, CA, 1961, pp. 361–379] and its positive-part Stein estimators [C. Stein, J. Roy. Statist. Soc. Ser. B, 24 (1962), pp. 265–296]. The coverage probability of these confidence sets depends on the noncentrality parameter $\tau^2$ as in the case of risks of these estimators. The new approach (which is different than Berger's [J. O. Berger, Ann. Statist., 8 (1980), pp. 716–761] and Hwang and Casella's [J. T. Hwang and G. Casella, Statist. Decisions, suppl. 1 (1984), pp. 3–16]) allows us to obtain the asymptotics analysis of the coverage probabilities for the two cases, namely, when $\tau^2 \to 0$ and $\tau^2 \to\infty$. For both cases we provide a simple approximation of the coverage probabilities. Some graphical and tabular results are provided to assess the accuracy of our approximations.

James–Stein estimators for time series regression models

The least squares (LS) estimator seems the natural estimator of the coefficients of a Gaussian linear regression model. However, if the dimension of the vector of coefficients is greater than 2 and the residuals are independent and identically distributed, this conventional estimator is not admissible. James and Stein [Estimation with quadratic loss, Proceedings of the Fourth Berkely Symposium vol. 1, 1961, pp. 361–379] proposed a shrinkage estimator (James–Stein estimator) which improves the least squares estimator with respect to the mean squares error loss function. In this paper, we investigate the mean squares error of the James–Stein (JS) estimator for the regression coefficients when the residuals are generated from a Gaussian stationary process. Then, sufficient conditions for the JS to improve the LS are given. It is important to know the influence of the dependence on the JS. Also numerical studies illuminate some interesting features of the improvement. The results have potential applications to economics, engineering, and natural sciences.

Approximating bayesian inference by weighted likelihood

AbstractThe author proposes to use weighted likelihood to approximate Bayesian inference when no external or prior information is available. He proposes a weighted likelihood estimator that minimizes the empirical Bayes risk under relative entropy loss. He discusses connections among the weighted likelihood, empirical Bayes and James‐Stein estimators. Both simulated and real data sets are used for illustration purposes.

PERFORMANCE LIMITS FOR ESTIMATORS OF THE RISK OR DISTRIBUTION OF SHRINKAGE-TYPE ESTIMATORS, AND SOME GENERAL LOWER RISK-BOUND RESULTS

We consider the problem of estimating measures of precision of shrinkage-type estimators such as their risk or distribution. The notion of shrinkage-type estimators here refers to estimators such as the James–Stein estimator and Lasso-type estimators, in addition to “thresholding” estimators such as, e.g., Hodges' so-called superefficient estimator. Although the precision measures of such estimators typically can be estimated consistently, we show that they cannot be estimated uniformly consistently (even locally). This follows as a corollary to (locally) uniform lower bounds on the performance of estimators of the precision measures that we obtain in the paper. These lower bounds are typically quite large (e.g., they approach ½ or 1 depending on the situation considered). The analysis is based on some general lower risk bounds and related general results on the (non)existence of uniformly consistent estimators also obtained in the paper. A preliminary draft of the results in Section 3 of this paper was written in 1999.

Performance Limits for Estimators of the Risk or Distribution of Shrinkage-Type Estimators, and Some General Lower Risk-Bound Results

We consider the problem of estimating measures of precision of shrinkage-type estimators like their risk or distribution. The notion of shrinkage-type estimators here refers to estimators like the James-Stein estimator or Lasso-type estimators, as well as to 'thresholding' estimators such as, e.g., Hodges' so-called superefficient estimator. While the precision measures of such estimators typically can be estimated consistently, we show that they cannot be estimated uniformly consistently (even locally). This follows as a corollary to (locally) uniform lower bounds on the performance of estimators of the precision measures that we also obtain. These lower bounds are typically quite large (e.g., they approach 1/2 or 1 depending on the situation considered). The analysis is based on some general lower risk bounds and related general results on the (non)existence of uniformly consistent estimators.

Necessary conditions for dominating the James-Stein estimator

This paper develops necessary conditions for an estimator to dominate the James-Stein estimator and hence the James-Stein positive-part estimator. The ultimate goal is to find classes of such dominating estimators which are admissible. While there are a number of results giving classes of estimators dominating the James-Stein estimator, the only admissible estimator known to dominate the James-Stein estimator is the generalized Bayes estimator relative to the fundamental harmonic function in three and higher dimension. The prior was suggested by Stein and the domination result is due to Kubokawa. Shao and Strawderman gave a class of estimators dominating the James-Stein positive-part estimator but were unable to demonstrate admissiblity of any in their class. Maruyama, following a suggestion of Stein, has studied generalized Bayes estimators which are members of a point mass at zero and a prior similar to the harmonic prior. He finds a subclass which is minimax and admissible but is unable to show that any in his class with positive point mass at zero dominate the James-Stein estimator. The results in this paper show that a subclass of Maruyama's procedures including the class that Stein conjectured might contain members dominating the James-Stein estimator cannot dominate the James-Stein estimator. We also show that under reasonable conditions, the “constant” in shrinkage factor must approachp-2 for domination to hold.

Minimax estimation with thresholding and its application to wavelet analysis

Many statistical practices involve choosing between a full model and reduced models where some coefficients are reduced to zero. Data were used to select a model with estimated coefficients. Is it possible to do so and still come up with an estimator always better than the traditional estimator based on the full model? The James–Stein estimator is such an estimator, having a property called minimaxity. However, the estimator considers only one reduced model, namely the origin. Hence it reduces no coefficient estimator to zero or every coefficient estimator to zero. In many applications including wavelet analysis, what should be more desirable is to reduce to zero only the estimators smaller than a threshold, called thresholding in this paper. Is it possible to construct this kind of estimators which are minimax? In this paper, we construct such minimax estimators which perform thresholding. We apply our recommended estimator to the wavelet analysis and show that it performs the best among the well-known estimators aiming simultaneously at estimation and model selection. Some of our estimators are also shown to be asymptotically optimal.

General empirical Bayes wavelet methods and exactly adaptive minimax estimation

In many statistical problems, stochastic signals can be represented as a sequence of noisy wavelet coefficients. In this paper, we develop general empirical Bayes methods for the estimation of true signal. Our estimators approximate certain oracle separable rules and achieve adaptation to ideal risks and exact minimax risks in broad collections of classes of signals. In particular, our estimators are uniformly adaptive to the minimum risk of separable estimators and the exact minimax risks simultaneously in Besov balls of all smoothness and shape indices, and they are uniformly superefficient in convergence rates in all compact sets in Besov spaces with a finite secondary shape parameter. Furthermore, in classes nested between Besov balls of the same smoothness index, our estimators dominate threshold and James-Stein estimators within an infinitesimal fraction of the minimax risks. More general block empirical Bayes estimators are developed. Both white noise with drift and nonparametric regression are considered.

Estimation of the mean vector of a multivariate normal distribution: subspace hypothesis

This paper considers the estimation of the mean vector θ of a p-variate normal distribution with unknown covariance matrix Σ when it is suspected that for a p × r known matrix B the hypothesis θ = B η , η ∈ R r may hold. We consider empirical Bayes estimators which includes (i) the unrestricted unbiased (UE) estimator, namely, the sample mean vector (ii) the restricted estimator (RE) which is obtained when the hypothesis θ = B η holds (iii) the preliminary test estimator (PTE), (iv) the James–Stein estimator (JSE), and (v) the positive-rule Stein estimator (PRSE). The biases and the risks under the squared loss function are evaluated for all the five estimators and compared. The numerical computations show that PRSE is the best among all the five estimators even when the hypothesis θ = B η is true.

On Bayes and unbiased estimators of loss

We consider estimation of loss for generalized Bayes or pseudo-Bayes estimators of a multivariate normal mean vector, θ. In 3 and higher dimensions, the MLEX is UMVUE and minimax but is inadmissible. It is dominated by the James-Stein estimator and by many others. Johnstone (1988, On inadmissibility of some unbiased estimates of loss,Statistical Decision Theory and Related Topics, IV (eds. S. S. Gupta and J. O. Berger), Vol. 1, 361–379, Springer, New York) considered the estimation of loss for the usual estimatorX and the James-Stein estimator. He found improvements over the Stein unbiased estimator of risk. In this paper, for a generalized Bayes point estimator of θ, we compare generalized Bayes estimators to unbiased estimators of loss. We find, somewhat surprisingly, that the unbiased estimator often dominates the corresponding generalized Bayes estimator of loss for priors which give minimax estimators in the original point estimation problem. In particular, we give a class of priors for which the generalized Bayes estimator of θ is admissible and minimax but for which the unbiased estimator of loss dominates the generalized Bayes estimator of loss. We also give a general inadmissibility result for a generalized Bayes estimator of loss.

Asymptotic mean squared error of constrained James–Stein estimators

In this paper, we consider the asymptotic expansion of the MSE of constrained James–Stein estimators. We provide an estimator of the MSE which is asymptotically valid upto O( m −1). A simulation study is undertaken to evaluate the performance of these estimators.

Improved Estimation of Regression Parameters in Measurement Error Models

This paper deals with improved estimation of the regrssion parameters in measurement error models when the reliability ratio matrix is known. We consider empirical Bayes estimators which include (i) the unrestricted unbiased (UE) estimator, (ii) the preliminary test estimator (PTE), (iii) the James-Stein estimator (SE), and (iv) the Positive-Rule Stein estimator (PRSE). The biases and the risks under the squared loss function are evaluated for all the four estimaton; and compared.

Stein's idea and minimax admissible estimation of a multivariate normal mean

We consider estimation of a multivariate normal mean vector under sum of squared error loss.We propose a new class of minimax admissible estimator which are generalized Bayes with respect to a prior distribution which is a mixture of a point prior at the origin and a continuous hierarchical type prior. We also study conditions under which these generalized Bayes minimax estimators improve on the James–Stein estimator and on the positive-part James–Stein estimator.

On the inevitability of a paradox in shrinkage estimation for scale mixtures of normals

Let ( X, U) be a random vector with spherically symmetric distribution about ( θ,0) where dim X= dim θ=p and dim U= dim 0=k (for p⩾3). Consider estimation of θ with loss || θ− d|| 2. Fourdrinier and Strawderman (J. Multivariate Anal. 59 (1996) 109) considered two classes of James–Stein type estimators δ α a ( X, U)=(1− a[( U′ U) α / X′ X]) X for α=0 and 1. The case α=0 is referred to as a “classical” James–Stein estimator and the case α=1 as a “robust” James–Stein type estimator since α=0 corresponds to the original James–Stein estimator in the normal case with known variance. In contrast, the case α=1 corresponds to an estimator which, for a=( p−2)/( k+2), simultaneously and uniformly dominates X for all spherically symmetric distributions. Fourdrinier and Strawderman showed that, for certain spherically symmetric distributions, the optimal ( a=( p−2)/( k+2)) James–Stein estimator for α=1 simultaneously dominates all James–Stein estimators with α=0. They term this situation a paradox since an estimated value of σ 2 in the shrinkage constant leads to an estimator which improves over the entire class of estimators which use the known value of σ 2 in the shrinkage constant. We show in this paper that this paradox is inevitable whenever the underlying distribution is a nondegenerate mixture of normal distributions and the dimension, k, of the residual vector U is sufficiently large. We also calculate the “critical dimension” k 0 for a variety of examples including the Student- t.