Stein's Estimate Research Articles

Shrinkage Estimators for Prediction Out-of-Sample: Conditional Performance

We find that, in a linear model, the James–Stein estimator, which dominates the maximum-likelihood estimator in terms of its in-sample prediction error, can perform poorly compared to the maximum-likelihood estimator in out-of-sample prediction. We give a detailed analysis of this phenomenon and discuss its implications. When evaluating the predictive performance of estimators, we treat the regressor matrix in the training data as fixed, i.e., we condition on the design variables. Our findings contrast those obtained by Baranchik (1973) and, more recently, by Dicker (2012) in an unconditional performance evaluation.

Bayesian regression based on principal components for high-dimensional data

The Gaussian sequence model can be obtained from the high-dimensional regression model through principal component analysis. It is shown that the Gaussian sequence model is equivalent to the original high-dimensional regression model in terms of prediction. Under a sparsity condition, we investigate the posterior consistency and convergence rates of the Gaussian sequence model. In particular, we examine two different modeling strategies: Bayesian inference with and without covariate selection. For Bayesian inferences without covariate selection, we obtain the consistency results of the estimators and posteriors with normal priors with constant and decreasing variances, and the James–Stein estimator; for Bayesian inference with covariate selection, we obtain convergence rates of Bayesian model averaging (BMA) and median probability model (MPM) estimators, and the posterior with variable selection prior. Based on these results, we conclude that variable selection is essential in high-dimensional Bayesian regression. A simulation study also confirms the conclusion. The methodologies are applied to a climate prediction problem.

Stein-type estimation using ranked set sampling

In this study, we consider the application of the James–Stein estimator for population means from a class of arbitrary populations based on ranked set sample (RSS). We consider a basis for optimally combining sample information from several data sources. We succinctly develop the asymptotic theory of simultaneous estimation of several means for differing replications based on the well-defined shrinkage principle. We showcase that a shrinkage-type estimator will have, under quadratic loss, a substantial risk reduction relative to the classical estimator based on simple random sample and RSS. Asymptotic distributional quadratic biases and risks of the shrinkage estimators are derived and compared with those of the classical estimator. A simulation study is used to support the asymptotic result. An over-riding theme of this study is that the shrinkage estimation method provides a powerful extension of its traditional counterpart for non-normal populations. Finally, we will use a real data set to illustrate the computation of the proposed estimators.

Mean squared error of James–Stein estimators for measurement error models

Whittemore (1989) in an interesting paper considered estimation of regression coefficients in measurement error models. She had the very interesting result which showed how to overcome the inconsistency of the usual least squares estimator of the regression coefficient in a measurement error model by an alternative James–Stein (James and Stein, 1961) estimator. The author did not provide a rigorous proof for the consistency of the suggested estimator, nor did she provide MSE of her estimator. Our objective is to provide a rigorous second order expansion of the mean squared error of the proposed James–Stein estimator under both known measurement variance and unknown measurement variance.

Nonparametric hierarchical Bayes analysis of binomial data via Bernstein polynomial priors

AbstractFor binomial data analysis, many methods based on empirical Bayes interpretations have been developed, in which a variance‐stabilizing transformation and a normality assumption are usually required. To achieve the greatest model flexibility, we conduct nonparametric Bayesian inference for binomial data and employ a special nonparametric Bayesian prior—the Bernstein–Dirichlet process (BDP)—in the hierarchical Bayes model for the data. The BDP is a special Dirichlet process (DP) mixture based on beta distributions, and the posterior distribution resulting from it has a smooth density defined on [0, 1]. We examine two Markov chain Monte Carlo procedures for simulating from the resulting posterior distribution, and compare their convergence rates and computational efficiency. In contrast to existing results for posterior consistency based on direct observations, the posterior consistency of the BDP, given indirect binomial data, is established. We study shrinkage effects and the robustness of the BDP‐based posterior estimators in comparison with several other empirical and hierarchical Bayes estimators, and we illustrate through examples that the BDP‐based nonparametric Bayesian estimate is more robust to the sample variation and tends to have a smaller estimation error than those based on the DP prior. In certain settings, the new estimator can also beat Stein's estimator, Efron and Morris's limited‐translation estimator, and many other existing empirical Bayes estimators. The Canadian Journal of Statistics 40: 328–344; 2012 © 2012 Statistical Society of Canada

Nonparametric empirical Bayes estimator in simultaneous estimation of Poisson means with application to mass spectrometry data

We consider the problem of simultaneous Poisson mean vector estimation and discuss the performance of nonparametric empirical Bayes (NPEB) estimator from the view point of risk consistency. We define the structural uniform risk consistency with respect to some classes of priors and show that the NPEB estimator achieves a structural uniform risk consistency with respect to some class of priors. It is shown that the NPEB estimator performs better than the maximum-likelihood estimator (MLE) and James–Stein estimators from the view point of structural uniform risk consistency. We also present numerical studies which support the asymptotic results and compare with the MLE and James–Stein-type estimators. We provide a real example of mass spectrometry data from a breast cancer study in Sauter et al. [Sauter, E.R., Davis, W., Qin, W., Scanlon, S., Mooney, B., Bromert, K., and Folk, W.R. (2009), ‘Identification of a β-Casein-Like Peptide in Breast Nipple Aspirate Fluid that is Associated with Breast Cancer’, Biomarkers in Medicine, 3, 577–588] with comparison of various estimators.

Stein Estimation for Spherically Symmetric Distributions: Recent Developments

This paper reviews advances in Stein-type shrinkage estimation for spherically symmetric distributions. Some emphasis is placed on developing intuition as to why shrinkage should work in location problems whether the underlying population is normal or not. Considerable attention is devoted to generalizing the "Stein lemma" which underlies much of the theoretical development of improved minimax estimation for spherically symmetric distributions. A main focus is on distributional robustness results in cases where a residual vector is available to estimate an unknown scale parameter, and, in particular, in finding estimators which are simultaneously generalized Bayes and minimax over large classes of spherically symmetric distributions. Some attention is also given to the problem of estimating a location vector restricted to lie in a polyhedral cone.

Asymptotically optimal shrinkage estimates for non-normal data

Motivated by several practical issues, we consider the problem of estimating the mean of a p-variate population (not necessarily normal) with unknown finite covariance. A quadratic loss function is used. We give a number of estimators (for the mean) with their loss functions admitting expansions to the order of p −1/2 as p→∞. These estimators contain Stein's [Inadmissibility of the usual estimator for the mean of a multivariate normal population, in Proceedings of the Third Berkeley Symposium in Mathematical Statistics and Probability, Vol. 1, J. Neyman, ed., University of California Press, Berkeley, 1956, pp. 197–206] estimate as a particular case and also contain ‘multiple shrinkage’ estimates improving on Stein's estimate. Finally, we perform a simulation study to compare the different estimates.

Tweedie’s Formula and Selection Bias

We suppose that the statistician observes some large number of estimates zi, each with its own unobserved expectation parameter μi. The largest few of the zi’s are likely to substantially overestimate their corresponding μi’s, this being an example of selection bias, or regression to the mean. Tweedie’s formula, first reported by Robbins in 1956, offers a simple empirical Bayes approach for correcting selection bias. This article investigates its merits and limitations. In addition to the methodology, Tweedie’s formula raises more general questions concerning empirical Bayes theory, discussed here as “relevance” and “empirical Bayes information.” There is a close connection between applications of the formula and James–Stein estimation.

Preliminary test and Stein estimations in simultaneous linear equations

In modeling of an economic system, there may occur some stochastic constraints, that can cause some changes in the estimators and their respective behaviors. In this approach we formulate the simultaneous equation models into the problem of estimating the regression parameters of a multiple regression model, under elliptical errors. We define five different sorts of estimators for the vector-parameter. Their exact risk expressions are also derived under the balanced loss function. Comparisons are then made to clarify the performance of the proposed estimators. It is shown that the shrinkage factor of the Stein estimator is robust with respect to departures from normality assumption.

A new estimator of covariance matrix

The problem of estimating a covariance matrix is considered in this paper. Using the so-called partial Iwasawa coordinates of the covariance matrix, a new improved estimator dominating the James–Stein estimator is proposed. The results of a simulation study verifies that the new estimator provides a substantial improvement in risk under Stein's loss.

Expansions for the risk of Stein type estimates for non-normal data

Abstract We consider the James–Stein problem for non-normal data for estimating a p-vector θ . It is shown how the risk may be expanded in powers of p -1 . The factor 1-2/p that distinguishes the James–Stein estimate from the Stein estimate is shown to have only O(p -2) effect on the risk. The case, where the variance must be estimated is studied for the one-way unbalanced ANOVA problem.

Application of wavelet‐based denoising techniques to remote sensing very low frequency signals

In this paper, we apply wavelet‐based denoising techniques to experimental remote sensing very low frequency (VLF) signals obtained from the Holographic Array for Ionospheric/Lightning research system and the Elazig VLF receiver system. The wavelet‐based denoising techniques are tested by soft, hard, hyperbolic and nonnegative garrote wavelet thresholding with the threshold selection rule based on Stein's unbiased estimate of risk, the fixed form threshold, the mixed threshold selection rule and the minimax‐performance threshold selection rule. The aim of this study is to find out the direct (early/fast) and indirect (lightning‐induced electron precipitation) effects of lightning in noisy VLF transmitter signals without discomposing the nature of signal. The appropriate results are obtained by fixed form threshold selection rule with soft thresholding using Symlet wavelet family.

Estimation of the Mean Vector of a Multivariate Elliptically Contoured Distribution

Abtsrcat This paper deals with the estimation of the mean vector θ of a p-variate elliptically contoured distribution, Ep(θ,Σ, f) based on the sample Y1 Y2,..., YN of size N of size N when it is suspected that for a p× r known matrix B, the hypothesis θ = Bη, η∈ Rr may hold. We consider the following estimators, (i) the unrestricted estimator (UE), (ii) the restricted estimator (RE), (iii) the preliminary test estimator (PTE), (iv) the James—Stein estimator (JSE), and (v) the positive-rule Stein estimator (PRSE). The bias and the risk expressions under the squared loss function are obtained for the five estimators and compared. It is noted that the dominance properties of these estimators remain the same as under normal theory. Further, it is shown that the shrinkage factor of the Stein-type estimators is robust with respect to the mean and unknown mixing distributions.

A SURE Approach for Digital Signal/Image Deconvolution Problems

In this paper, we are interested in the classical problem of restoring data degraded by a convolution and the addition of a white Gaussian noise. The originality of the proposed approach is twofold. First, we formulate the restoration problem as a nonlinear estimation problem leading to the minimization of a criterion derived from Stein's unbiased quadratic risk estimate. Secondly, the deconvolution procedure is performed using any analysis and synthesis frames that can be overcomplete or not. New theoretical results concerning the calculation of the variance of the Stein's risk estimate are also provided in this work. Simulations carried out on natural images show the good performance of our method with respect to conventional wavelet-based restoration methods.

Sharp adaptive estimation by a blockwise method

We consider a blockwise James–Stein estimator for nonparametric function estimation in suitable wavelet or Fourier bases. The estimator can be readily explained and implemented. We show that the estimator is asymptotically sharp adaptive in minimax risk over any Sobolev ball containing the true function. Further, for a moderately broad range of bounded sets in the Besov space our estimator is asymptotically nearly sharp adaptive in the sense that it comes within the Donoho–Liu constant, 1.24, of being exactly sharp adaptive. Other parameter spaces are also considered. The paper concludes with a Monte-Carlo study comparing the performance of our estimator with that of three other popular wavelet estimators. Our procedure generally (but not always) outperforms two of these and is overall comparable, or perhaps slightly superior, with the third.

High order approximation for the coverage probability by a confident set centered at the positive-part James–Stein estimator

In this paper we continue our investigation connected with the new approach developed in Ahmed et al. [Ahmed, S.E., Saleh, A.K.Md.E., Volodin, A., Volodin, I., 2006. Asymptotic expansion of the coverage probability of James–Stein estimators. Theory Probab. Appl. 51 (4) 1–14] for asymptotic expansion construction of coverage probabilities, for confidence sets centered at James–Stein and positive-part James–Stein estimators. The coverage probabilities for these confidence sets depend on the noncentrality parameter τ 2 , the same as the risks of these estimators. In this paper we consider only the confidence set centered at the positive-part James–Stein estimator. As is shown in the above-mentioned reference, the new approach provides a method to obtain for the given confidence set, an asymptotic expansion of the coverage probability as one formula for both cases τ → 0 and τ → ∞ . We obtain the third terms of the asymptotic expansion for both mentioned cases, that is, the coefficients at τ 2 and τ − 2 . Numerical illustrations show that the third term has only a small influence on the accuracy of the asymptotic estimation of coverage probability.

Minimum Message Length shrinkage estimation

This note considers estimation of the mean of a multivariate Gaussian distribution with known variance within the Minimum Message Length (MML) framework. Interestingly, the resulting MML estimator exactly coincides with the positive-part James–Stein estimator under the choice of an uninformative prior. A new approach for estimating parameters and hyperparameters in general hierarchical Bayes models is also presented.

Shrinkage estimation in general linear models

We propose a James–Stein-type shrinkage estimator for the parameter vector in a general linear model when it is suspected that some of the parameters may be restricted to a subspace. The James–Stein estimator is shown to demonstrate asymptotically superior risk performance relative to the conventional least squares estimator under quadratic loss. An extensive simulation study based on a multiple linear regression model and a logistic regression model further demonstrates the improved performance of this James–Stein estimator in finite samples. The application of this new estimator is illustrated using Ontario newborn infants data spanning four fiscal years.

Estimation of parameters of parallelism model with elliptically distributed errors

In this paper we consider some improved estimators of the intercept and slope parameters in a parallelism model with errors belonging to a sub-class of elliptically contoured distributions. We derive the exact bias, MSE matrices and quadratic risk expressions for these estimators. It is shown that the dominance properties of these estimators are the same as under normal theory. Further, it is shown that the shrinkage factor of the Stein estimators is robust with respect to the regression parameters and unknown mixing distributions.