Stein's Estimate Research Articles

Regularization through variable selection and conditional MLE with application to classification in high dimensions

It is often the case that high-dimensional data consist of only a few informative components. Standard statistical modeling and estimation in such a situation is prone to inaccuracies due to overfitting, unless regularization methods are practiced. In the context of classification, we propose a class of regularization methods through shrinkage estimators. The shrinkage is based on variable selection coupled with conditional maximum likelihood. Using Stein's unbiased estimator of the risk, we derive an estimator for the optimal shrinkage method within a certain class. A comparison of the optimal shrinkage methods in a classification context, with the optimal shrinkage method when estimating a mean vector under a squared loss, is given. The latter problem is extensively studied, but it seems that the results of those studies are not completely relevant for classification. We demonstrate and examine our method on simulated data and compare it to feature annealed independence rule and Fisher's rule.

Audio Denoising by Time-Frequency Block Thresholding

Removing noise from audio signals requires a nondiagonal processing of time-frequency coefficients to avoid producing ldquomusical noise.rdquo State of the art algorithms perform a parameterized filtering of spectrogram coefficients with empirically fixed parameters. A block thresholding estimation procedure is introduced, which adjusts all parameters adaptively to signal property by minimizing a Stein estimation of the risk. Numerical experiments demonstrate the performance and robustness of this procedure through objective and subjective evaluations.

Estimation of a mean vector under quartic loss

Let X = ( X 1 , … , X p ) be a p-variate normal random vector with unknown mean θ = ( θ 1 , … , θ p ) and identity covariance matrix. Estimators δ = ( δ 1 , … , δ p ) of θ are considered under the quartic loss ∑ i = 1 p ( δ i - θ i ) 4 . For p ⩾ 3 , we develop sufficient conditions on δ ( X ) = X + g ( X ) to improve upon the usual estimator δ 0 ( X ) = X . To this end, we yield an unbiased estimator O g ( X ) of the risk difference between δ ( X ) and δ 0 ( X ) . An interesting feature is that, to obtain adequate dominating estimators, O g ( X ) is used in two ways. First, we search estimators such that O g ( x ) ⩽ 0 for any x ∈ R p , which guarantees the desired domination. Then, to enlarge the class of improved estimators, we investigate conditions for which this inequality is satisfied in mean, that is, E θ [ O g ( X ) ] ⩽ 0 . In particular, no James–Stein estimator satisfies O g ( X ) < 0 for all X, but E θ [ O g ( X ) ] < 0 for p ⩾ 5 and a shrinkage factor 0 < a ⩽ 2 ( p - 4 ) .

A unified and generalized set of shrinkage bounds on minimax Stein estimates

Consider the problem of estimating the mean vector θ of a random variable X in R p , with a spherically symmetric density f ( ‖ x − θ ‖ 2 ) , under loss ‖ δ − θ ‖ 2 . We give an increasing sequence of bounds on the shrinkage constant of Stein-type estimators depending on properties of f ( t ) that unify and extend several classical bounds from the literature. The basic way to view the conditions on f ( t ) is that the distribution of X arises as the projection of a spherically symmetric vector ( X , U ) in R p + k . A second way is that f ( t ) satisfies ( − 1 ) j f ( j ) ( t ) ≥ 0 for 0 ≤ j ≤ ℓ and that ( − 1 ) ℓ f ( ℓ ) ( t ) is non-increasing where k = 2 ( ℓ + 1 ) . The case ℓ = 0 ( k = 2 ) corresponds to unimodality, while the case ℓ = k = ∞ corresponds to complete monotonicity of f ( t ) (or equivalently that f ( ‖ x − θ ‖ 2 ) is a scale mixture of normals). The bounds on the minimax shrinkage constant in this paper agree with the classical bounds in the literature for the case of spherical symmetry, spherical symmetry and unimodality, and scale mixtures of normals. However, they extend these bounds to an increasing sequence (in k or ℓ ) of minimax bounds.

Estimation, prediction and the Stein phenomenon under divergence loss

We consider two problems: (1) estimate a normal mean under a general divergence loss introduced in [S. Amari, Differential geometry of curved exponential families — curvatures and information loss, Ann. Statist. 10 (1982) 357–387] and [N. Cressie, T.R.C. Read, Multinomial goodness-of-fit tests, J. Roy. Statist. Soc. Ser. B. 46 (1984) 440–464] and (2) find a predictive density of a new observation drawn independently of observations sampled from a normal distribution with the same mean but possibly with a different variance under the same loss. The general divergence loss includes as special cases both the Kullback–Leibler and Bhattacharyya–Hellinger losses. The sample mean, which is a Bayes estimator of the population mean under this loss and the improper uniform prior, is shown to be minimax in any arbitrary dimension. A counterpart of this result for predictive density is also proved in any arbitrary dimension. The admissibility of these rules holds in one dimension, and we conjecture that the result is true in two dimensions as well. However, the general Baranchick [A.J. Baranchick, a family of minimax estimators of the mean of a multivariate normal distribution, Ann. Math. Statist. 41 (1970) 642–645] class of estimators, which includes the James–Stein estimator and the Strawderman [W.E. Strawderman, Proper Bayes minimax estimators of the multivariate normal mean, Ann. Math. Statist. 42 (1971) 385–388] class of estimators, dominates the sample mean in three or higher dimensions for the estimation problem. An analogous class of predictive densities is defined and any member of this class is shown to dominate the predictive density corresponding to a uniform prior in three or higher dimensions. For the prediction problem, in the special case of Kullback–Leibler loss, our results complement to a certain extent some of the recent important work of Komaki [F. Komaki, A shrinkage predictive distribution for multivariate normal observations, Biometrika 88 (2001) 859–864] and George, Liang and Xu [E.I. George, F. Liang, X. Xu, Improved minimax predictive densities under Kullbak–Leibler loss, Ann. Statist. 34 (2006) 78–92]. While our proposed approach produces a general class of predictive densities (not necessarily Bayes, but not excluding Bayes predictors) dominating the predictive density under a uniform prior. We show also that various modifications of the James–Stein estimator continue to dominate the sample mean, and by the duality of estimation and predictive density results which we will show, similar results continue to hold for the prediction problem as well.

A Littlewood–Paley–Stein estimate on graphs and groups

We establish the boundedness in $L^q$ spaces, $1< q\leq 2$, of a &#8220;vertical" Littlewood&#8211;Paley&#8211;Stein operator associated with a reversible random walk on a graph. This result extends to certain non-reversible random walks, including center

Blind Minimax Estimation

We consider the linear regression problem of estimating an unknown, deterministic parameter vector based on measurements corrupted by colored Gaussian noise. We present and analyze blind minimax estimators (BMEs), which consist of a bounded parameter set minimax estimator, whose parameter set is itself estimated from measurements. Thus, one does not require any prior assumption or knowledge, and the proposed estimator can be applied to any linear regression problem. We demonstrate analytically that the BMEs strictly dominate the least-squares estimator, i.e., they achieve lower mean-squared error for any value of the parameter vector. Both Stein's estimator and its positive-part correction can be derived within the blind minimax framework. Furthermore, our approach can be readily extended to a wider class of estimation problems than Stein's estimator, which is defined only for white noise and non-transformed measurements. We show through simulations that the BMEs generally outperform previous extensions of Stein's technique.

Methods for improvement in estimation of a normal mean matrix

This paper is concerned with the problem of estimating a matrix of means in multivariate normal distributions with an unknown covariance matrix under invariant quadratic loss. It is first shown that the modified Efron–Morris estimator is characterized as a certain empirical Bayes estimator. This estimator modifies the crude Efron–Morris estimator by adding a scalar shrinkage term. It is next shown that the idea of this modification provides a general method for improvement of estimators, which results in the further improvement on several minimax estimators. As a new method for improvement, an adaptive combination of the modified Stein and the James–Stein estimators is also proposed and is shown to be minimax. Through Monte Carlo studies of the risk behaviors, it is numerically shown that the proposed, combined estimator inherits the nice risk properties of both individual estimators and thus it has a very favorable risk behavior in a small sample case. Finally, the application to a two-way layout MANOVA model with interactions is discussed.

Covariance complexity and rates of return on assets

This paper considers the estimation of the expected rate of return on a set of risky assets. The approach to estimation focuses on the covariance matrix for the returns. The structure in the covariance matrix determines shared information which is useful in estimating the mean return for each asset. An empirical Bayes estimator is developed using the covariance structure of the returns distribution. The estimator is an improvement on the maximum likelihood and Bayes–Stein estimators in terms of mean squared error. The effect of reduced estimation error on accumulated wealth is analyzed for the portfolio choice model with constant relative risk aversion utility.

Asymptotic Expansion of the Coverage Probability of James–Stein Estimators

This paper provides a new approach to the asymptotic expansion construction of the coverage probability of the confidence sets recentered in [W. James and C. Stein, Estimation with quadratic loss, in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, Univ. California Press, Berkeley, CA, 1961, pp. 361–379] and its positive-part Stein estimators [C. Stein, J. Roy. Statist. Soc. Ser. B, 24 (1962), pp. 265–296]. The coverage probability of these confidence sets depends on the noncentrality parameter $\tau^2$ as in the case of risks of these estimators. The new approach (which is different than Berger's [J. O. Berger, Ann. Statist., 8 (1980), pp. 716–761] and Hwang and Casella's [J. T. Hwang and G. Casella, Statist. Decisions, suppl. 1 (1984), pp. 3–16]) allows us to obtain the asymptotics analysis of the coverage probabilities for the two cases, namely, when $\tau^2 \to 0$ and $\tau^2 \to\infty$. For both cases we provide a simple approximation of the coverage probabilities. Some graphical and tabular results are provided to assess the accuracy of our approximations.

Stein estimation for infinitely divisible laws

Unbiased risk estimation, a la Stein, is studied for infinitely divisible laws with finite second moment.

James–Stein estimators for time series regression models

The least squares (LS) estimator seems the natural estimator of the coefficients of a Gaussian linear regression model. However, if the dimension of the vector of coefficients is greater than 2 and the residuals are independent and identically distributed, this conventional estimator is not admissible. James and Stein [Estimation with quadratic loss, Proceedings of the Fourth Berkely Symposium vol. 1, 1961, pp. 361–379] proposed a shrinkage estimator (James–Stein estimator) which improves the least squares estimator with respect to the mean squares error loss function. In this paper, we investigate the mean squares error of the James–Stein (JS) estimator for the regression coefficients when the residuals are generated from a Gaussian stationary process. Then, sufficient conditions for the JS to improve the LS are given. It is important to know the influence of the dependence on the JS. Also numerical studies illuminate some interesting features of the improvement. The results have potential applications to economics, engineering, and natural sciences.

Improved Estimation of Dissimilarities by Presmoothing Functional Data

We examine the effect of presmoothing functional data on estimating the dissimilarities among objects in a dataset, with applications to cluster analysis and other distance methods, such as multidimensional scaling and statistical matching. We prove that a shrinkage method of smoothing results in a better estimator of the dissimilarities among a set of noisy curves. For a model with independent noise structure, the smoothed-data dissimilarity estimator dominates the observed-data estimator. For a dependent-error model—often applicable when the functional data are measured nearly continuously over some domain—an asymptotic domination result is given for the smoothed-data estimator. A simulation study indicates the magnitude of improvement provided by the shrinkage estimator and examines its behavior for heavy-tailed noise structure. The shrinkage estimator presented here combines Stein estimation and basis function-based linear smoothers in a novel manner. Statisticians increasingly analyze sizable sets of functional data, and the results in this article are a useful contribution to the theory of the effect of presmoothing on functional data analysis.

PERFORMANCE LIMITS FOR ESTIMATORS OF THE RISK OR DISTRIBUTION OF SHRINKAGE-TYPE ESTIMATORS, AND SOME GENERAL LOWER RISK-BOUND RESULTS

We consider the problem of estimating measures of precision of shrinkage-type estimators such as their risk or distribution. The notion of shrinkage-type estimators here refers to estimators such as the James–Stein estimator and Lasso-type estimators, in addition to “thresholding” estimators such as, e.g., Hodges' so-called superefficient estimator. Although the precision measures of such estimators typically can be estimated consistently, we show that they cannot be estimated uniformly consistently (even locally). This follows as a corollary to (locally) uniform lower bounds on the performance of estimators of the precision measures that we obtain in the paper. These lower bounds are typically quite large (e.g., they approach ½ or 1 depending on the situation considered). The analysis is based on some general lower risk bounds and related general results on the (non)existence of uniformly consistent estimators also obtained in the paper. A preliminary draft of the results in Section 3 of this paper was written in 1999.

The Constraining Power of International Treaties: Theory and Methods

We acknowledge the contribution of von Stein (2005) in calling attention to the very real problem of selection bias in estimating treaty effects. Nonetheless, we dispute both von Stein's theoretical and empirical conclusions. Theoretically, we contend that treaties can both screen and constrain simultaneously, meaning that findings of screening do nothing to undermine the claim that treaties constrain state behavior as well. Empirically, we question von Stein's estimator on several grounds, including its strong distributional assumptions and its statistical inconsistency. We then illustrate that selection bias does not account for much of the difference between Simmons's (2000) and von Stein's (2005) estimated treaty effects, and instead reframe the problem as one of model dependency. Using a preprocessing matching step to reduce that dependency, we then illustrate treaty effects that are both substantively and statistically significant—and that are quite close in magnitude to those reported by Simmons.

ASP fits to multi-way layouts

The balanced complete multi-way layout with ordinal or nominal factors is a fundamental data-type that arises in medical imaging, agricultural field trials, DNA microassays, and other settings where analysis of variance (ANOVA) is an established tool. ASP algorithms weigh competing biased fits in order to reduce risk through variance-bias tradeoff. The acronym ASP stands for Adaptive Shrinkage of Penalty bases. Motivating ASP is a penalized least squares criterion that associates a separate quadratic penalty term with each main effect and each interaction in the general ANOVA decomposition of means. The penalty terms express plausible conjecture about the mean function, respecting the difference between ordinal and nominal factors. Multiparametric asymptotics under a probability model and experiments on data elucidate how ASP dominates least squares, sometimes very substantially. ASP estimators for nominal factors recover Stein's superior shrinkage estimators for one- and two-way layouts. ASP estimators for ordinal factors bring out the merits of smoothed fits to multi-way layouts, a topic broached algorithmically in work by Tukey.

Small Confidence Sets for the Mean of a Spherically Symmetric Distribution

Summary Suppose that X has a k-variate spherically symmetric distribution with mean vector θ and identity covariance matrix. We present two spherical confidence sets for θ, both centred at a positive part Stein estimator TS+(X). In the first, we obtain the radius by approximating the upper α-point of the sampling distribution of ‖TS+(X)−θ‖2 by the first two non-zero terms of its Taylor series about the origin. We can analyse some of the properties of this confidence set and see that it performs well in terms of coverage probability, volume and conditional behaviour. In the second method, we find the radius by using a parametric bootstrap procedure. Here, even greater improvement in terms of volume over the usual confidence set is possible, at the expense of having a less explicit radius function. A real data example is provided, and extensions to the unknown covariance matrix and elliptically symmetric cases are discussed.

From Unit Root to Stein’s Estimator to Fisher’s k Statistics: If You Have a Moment, I Can Tell You More

Any general textbook that discusses moment generating functions (MGFs) shows how to obtain a moment of positive-integer order via differentiation, although usually the presented examples are only illustrative, because the corresponding moments can be calculated in more direct ways. It is thus somewhat unfortunate that very few textbooks discuss the use of MGFs when it becomes the simplest, and sometimes the only, approach for analytic calculation and manipulation of moments. Such situations arise when we need to evaluate the moments of ratios and logarithms, two of the most common transformations in statistics. Such moments can be obtained by differentiating and integrating a joint MGF of the underlying untransformed random variables in appropriate ways. These techniques are examples of multivariate Laplace transform methods and can also be derived from the fact that moments of negative orders can be obtained by integrating an MGF. This article reviews, extends and corrects various results scattered in the literature on this joint-MGF approach, and provides four applications of independent interest to demonstrate its power and beauty. The first application, which motivated this article, is for the exact calculation of the moments of a well-known limiting distribution under the unit-root AR(1) model. The second, which builds on Stigler’s Galtonian perspective, reveals a straightforward, non-Bayesian constructive derivation of the Stein estimator, as well as convenient expressions for studying its risk and bias. The third finds an exceedingly simple bound for the bias of a sample correlation from a bivariate normal population, namely the magnitude of the relative bias is not just of order n−1, but actually is bounded above by n−1 for all sample sizes n≥2. The fourth tackles the otherwise intractable problem of studying the finite-sample optimal bridge in the context of bridge sampling for computing normalizing constants. A by-product of the joint-MGF approach is that positive-order fractional moments can be easily obtained from an MGF without invoking the concept of fractional differentiation, a method used by R. A. Fisher in his study of k statistics 45 years before it reappeared in the probability literature.

Using James–Stein Estimators in Homogeneity Tests of the Risk Difference

ABSTRACT In multi-center clinical trials in which the success/failure of two treatments are measured, 2 × 2 × K contingency data are obtained, where K is the number of centers in the study. In this context, the risk difference may be preferred (over the odds ratio or relative risk) as a measure of the efficacy of the new treatment. To summarize the risk difference across centers, the estimated risk difference for each center must be comparable. Although several weighted least squares (WLS) tests of homogeneity of the risk difference have been proposed for the sparse data situation (1,2), none of these tests perform satisfactorily when each center has small samples. In the sparse data situation, the weights given to each center's risk difference estimate are imprecise and may be undefined. James–Stein estimates have been shown to be more precise (in terms of mean squared error) than their maximum likelihood counterparts. In this article, we investigate the use of James–Stein estimates for these weights. These estimates shrink the individual risk estimates towards the overall center mean and thus avoid problems encountered when risk estimates are zero or one. We use Monte Carlo simulations to show that using these estimates in the WLS test of homogeneity of risk differences improves their performance in terms of Type I error.

Minimax estimation with thresholding and its application to wavelet analysis

Many statistical practices involve choosing between a full model and reduced models where some coefficients are reduced to zero. Data were used to select a model with estimated coefficients. Is it possible to do so and still come up with an estimator always better than the traditional estimator based on the full model? The James–Stein estimator is such an estimator, having a property called minimaxity. However, the estimator considers only one reduced model, namely the origin. Hence it reduces no coefficient estimator to zero or every coefficient estimator to zero. In many applications including wavelet analysis, what should be more desirable is to reduce to zero only the estimators smaller than a threshold, called thresholding in this paper. Is it possible to construct this kind of estimators which are minimax? In this paper, we construct such minimax estimators which perform thresholding. We apply our recommended estimator to the wavelet analysis and show that it performs the best among the well-known estimators aiming simultaneously at estimation and model selection. Some of our estimators are also shown to be asymptotically optimal.