Stein Rule Research Articles

Machine learning and the James–Stein estimator

It is now 62 years since the publication of James and Stein’s seminal article on the estimation of a multivariate normal mean vector. The paper made a spectacular first impression on the statistical community through its demonstration of inadmissability of the maximum likelihood estimator. It continues to be influential, but not for the initial reasons. Empirical Bayes shrinkage estimation, now a major topic, found its early justification in the James–Stein formula. Less obvious downstream topics include Tweedie’s formula and Benjamini and Hochberg’s false discovery rate algorithm. This is a short and mainly non-technical review of the James–Stein rule and its effects on the machine learning era of statistical innovation.

Generalized Bayes estimation for a SAR model with linear restrictions binding the coefficients

The Spatial Autoregressive (SAR) models have drawn considerable attention in recent econometrics literature because of their capability to model the spatial spill overs in a feasible way. While considering the Bayesian analysis of these models, one may face the problem of lack of robustness with respect to underlying prior assumptions. The generalized Bayes estimators provide a viable alternative to incorporate prior belief and are more robust with respect to underlying prior assumptions. The present paper considers the SAR model with a set of linear restrictions binding the regression coefficients and derives restricted generalized Bayes estimator for the coefficients vector. The minimaxity of the restricted generalized Bayes estimator has been established. Using a simulation study, it has been demonstrated that the estimator dominates the restricted least squares as well as restricted Stein rule estimators.

Post estimation and prediction strategies in negative binomial regression model

ABSTRACT We addressed parameter estimation for low-dimensional and high-dimensional negative binomial regression models in the presence of overfitting and uncertainty about the subspace information. We proposed novelty parameter estimation based on linear shrinkage, preliminary test, James–Stein rule, and penalty strategies, outperforming the classical maximum likelihood. The asymptotic distributional bias and risk were derived to explore and compare the theoretical predictions of the proposed estimators. A numerical comparison of the performance of the proposed estimators was also studied via Monte Carlo simulations and real application to confirm the theoretical results. Based on our findings, estimators based on the preliminary test and James–Stein rule strategies were most effective at addressing the overfitting problem when the accuracy of the subspace information was unknown.

Generalized Bayes Estimation for a SAR Model with Linear Restrictions Binding the Coefficients

Considering spatial autoregressive model, this paper derives restricted generalized Bayes estimator for the regression coefficients vector in the presence of a set of linear restrictions binding coefficients. The minimaxity of the restricted generalized Bayes estimator has been established. Using a simulation study, it has been demonstrated that the estimator dominates the restricted least squares as well as restricted Stein rule estimators.

MSE performance of the weighted average estimators consisting of shrinkage estimators

ABSTRACTIn this paper, we consider a regression model and propose estimators which are the weighted averages of two estimators among three estimators; the Stein-rule (SR), the minimum mean squared error (MMSE), and the adjusted minimum mean-squared error (AMMSE) estimators. It is shown that one of the proposed estimators has smaller mean-squared error (MSE) than the positive-part Stein-rule (PSR) estimator over a moderate region of parameter space when the number of the regression coefficients is small (i.e., 3), and its MSE performance is comparable to the PSR estimator even when the number of the regression coefficients is not so small.

Estimation of Missing Data Using Convoluted Weighted Method in Nigeria Household Survey

The analysis of survey data becomes difficult in the presence of missing data. By the use of Least Squares and Stein Rule method, estimator for the parameters of interest can be obtained. In this study, proposed convoluted Weighted Least Squares and Stein Rule method is compared with some existing techniques where the data is considered missing completely at random (MCAR). The results show that other techniques are occasionally useful in estimating most of the parameter, but proposed (LSSR) technique perform better regardless of the percentage of the missing data under MCAR assumption.

Comparisons of estimators for regression coefficient in a misspecified linear model with elliptically contoured errors

In a misspecified linear regression model with elliptically contoured errors, the exact risks of generalized least squares (GLS), restricted least squares (RLS), preliminary test (PT), Stein-rule (SR) and positive-rule shrinkage (PRS) estimators of regression coefficient are derived. Risk superiority conditions dependent on prior constraint error and the model specification error are obtained. When the model is misspecified and the error terms obey the elliptically contoured distribution, it is shown analytically that the PRS estimator dominates uniformly the SR estimator. However, the PRS and SR estimators do not dominate uniformly the GLS estimator. Furthermore, the dominance of the RLS estimator over the GLS estimator does not hold necessarily even if the model has linear constraint.

Risk Comparison of Improved Estimators in a Linear Regression Model with MultivariatetErrors under Balanced Loss Function

Under a balanced loss function, we derive the explicit formulae of the risk of the Stein-rule (SR) estimator, the positive-part Stein-rule (PSR) estimator, the feasible minimum mean squared error (FMMSE) estimator, and the adjusted feasible minimum mean squared error (AFMMSE) estimator in a linear regression model with multivariateterrors. The results show that the PSR estimator dominates the SR estimator under the balanced loss and multivariateterrors. Also, our numerical results show that these estimators dominate the ordinary least squares (OLS) estimator when the weight of precision of estimation is larger than about half, and vice versa. Furthermore, the AFMMSE estimator dominates the PSR estimator in certain occasions.

Confidence ellipsoids based on a general family of shrinkage estimators for a linear model with non-spherical disturbances

This paper considers a general family of Stein rule estimators for the coefficient vector of a linear regression model with nonspherical disturbances, and derives estimators for the Mean Squared Error (MSE) matrix, and risk under quadratic loss for this family of estimators. The confidence ellipsoids for the coefficient vector based on this family of estimators are proposed, and the performance of the confidence ellipsoids under the criterion of coverage probability and expected volumes is investigated. The results of a numerical simulation are presented to illustrate the theoretical findings, which could be applicable in the area of economic growth modeling.

An application of shrinkage estimation to the nonlinear regression model

Various large sample estimation techniques in a nonlinear regression model are presented. These estimators are based around preliminary tests of significance, and the James–Stein rule. The properties of these estimators are studied when estimating regression coefficients in the multiple nonlinear regression model when it is a priori suspected that the coefficients may be restricted to a subspace. A simulation based on a demand for money model shows the superiority of the positive-part shrinkage estimator, in terms of standard measures of asymptotic distributional quadratic bias and risk measures, over a range of economically meaningful parameter values. Further work remains in analysing the use of these estimators in economic applications, relative to the inferential approach which is best to use in these circumstances.

The Stein hull

We are interested in the statistical linear inverse problem Y=Af+εξ, where A denotes a compact operator and εξ a stochastic noise. In this setting, the risk hull point of view provides interesting tools for the construction of adaptive estimators. It sheds light on the processes governing the behaviour of linear estimators. In this article, we investigate the link between some threshold estimators and this risk hull point of view. The penalised blockwise Stein rule plays a central role in this study. In particular, this estimator may be considered as a risk hull minimisation method, provided the penalty is well chosen. Using this perspective, we study the properties of the threshold and propose an admissible range for the penalty leading to accurate results. We eventually propose a penalty close to the lower bound of this range.

Robustness of Stein-type estimators under a non-scalar error covariance structure

The Stein-rule (SR) and positive-part Stein-rule (PSR) estimators are two popular shrinkage techniques used in linear regression, yet very little is known about the robustness of these estimators to the disturbances’ deviation from the white noise assumption. Recent studies have shown that the OLS estimator is quite robust, but whether this is so for the SR and PSR estimators is less clear as these estimators also depend on the F statistic which is highly susceptible to covariance misspecification. This study attempts to evaluate the effects of misspecifying the disturbances as white noise on the SR and PSR estimators by a sensitivity analysis. Sensitivity statistics of the SR and PSR estimators are derived and their properties are analyzed. We find that the sensitivity statistics of these estimators exhibit very similar properties and both estimators are extremely robust to MA(1) disturbances and reasonably robust to AR(1) disturbances except for the cases of severe autocorrelation. The results are useful in light of the rising interest of the SR and PSR techniques in the applied literature.

The effects of the proxy information on the iterative Stein-rule estimator of the disturbance variance

In a regression model with proxy variables, we consider the iterative estimator of the disturbance variance to obtain more precise estimates. In the formula of the estimator of the disturbance variance, the estimator is obtained by using Stein-rule (SR) estimator instead of OLS (ordinary least squares) estimator is called Iterative estimator of the disturbance variance. It is shown that, in a regression model with proxy variables the mean square error (MSE) of the iterative estimator of the disturbance variance is greater than the MSE of the disturbance variance related to the OLS estimator under certain conditions.

Risk comparison of the Stein-rule estimator in a linear regression model with omitted relevant regressors and multivariatet errors under the Pitman nearness criterion

In this paper we consider a linear regression model with omitted relevant regressors and multivariatet error terms. The explicit formula for the Pitman nearness criterion of the Stein-rule (SR) estimator relative to the ordinary least squares (OLS) estimator is derived. It is shown numerically that the dominance of the SR estimator over the OLS estimator under the Pitman nearness criterion can be extended to the case of the multivariatet error distribution when the specification error is not severe. It is also shown that the dominance of the SR estimator over the OLS estimator cannot be extended to the case of the multivariatet error distribution when the specification error is severe.

The existence of secondary orbital interactions

B3LYP/6-311+G** (and MP2/6-311+G**) computations, performed for a series of Diels-Alder (DA) reactions, confirm that the endo transition states (TS) and the related Cope-TSs are favored energetically over the respective exo-TSs. Likewise, the computed magnetic properties (nucleus-independent chemical shifts and magnetic susceptibililties) of the endo- (as well as the Cope) TS's reveal their greater electron delocalization and greater aromaticity than the exo-TS's. However, Woodward and Hoffmann's original example is an exception: their endo-TS model, involving the DA reaction of a syn- with an anti-butadiene (BD), actually is disfavored energetically over the corresponding exo-TS; magnetic criteria also do not indicate the existence of SOI delocalization in either case. Instead, a strong energetic preference for endo-TSs due to SOI is found when both BDs are in the syn conformations. This is in accord with Alder and Stein's rule of "maximum accumulation of double bonds:" both the dienophile and the diene should have syn conformations. Plots along the IRC's show that the magnetic properties typically are most strongly exalted close to the energetic TS. Because of SOI, all the points along the endo reaction coordinates are more diatropic than along the corresponding exo pathways. We find weak SOI effects to be operative in the endo-TSs involved in the cycloadditions of cyclic alkenes, cyclopropene, aziridine, cyclobutene, and cyclopentene, with cyclopentadiene. While the endo-TSs are only slightly lower in energy than the respective exo-TSs, the magnetic properties of the endo-TS's are significantly exalted over those for the exo-TS's and the Natural Bond Orbitals indicate small stabilizing interactions between the methylene cycloalkene hydrogen orbitals (and lone pairs in case of aziridine) with pi-character and the diene pi MOs.

PMSE performance of the Stein-rule and positive-part Stein-rule estimators in a regression model with or without proxy variables

Consider a linear regression model with some relevant regressors are unobservable. In such a situation, we estimate the model by using the proxy variables as regressors or by simply omitting the relevant regressors. In this paper, we derive the explicit formula of the predictive mean squared error (PMSE) of the Stein-rule (SR) estimator and the positive-part Stein-rule (PSR) estimator for the regression coefficients when the proxy variables are used. We examine the effect of using the proxy variables on the risk performances of the SR and PSR estimators. It is shown analytically that the PSR estimator dominates the SR estimator even when the proxy variables are used. Also, our numerical results show that using the proxy variables is preferable to omitting the relevant regressors.

PMSE PERFORMANCE OF THE BIASED ESTIMATORS IN A LINEAR REGRESSION MODEL WHEN RELEVANT REGRESSORS ARE OMITTED

In this paper, we consider a linear regression model when relevant regressors are omitted. We derive the explicit formulae for the predictive mean squared errors (PMSEs) of the Stein-rule (SR) estimator, the positive-part Stein-rule (PSR) estimator, the minimum mean squared error (MMSE) estimator, and the adjusted minimum mean squared error (AMMSE) estimator. It is shown analytically that the PSR estimator dominates the SR estimator in terms of PMSE even when there are omitted relevant regressors. Also, our numerical results show that the PSR estimator and the AMMSE estimator have much smaller PMSEs than the ordinary least squares estimator even when the relevant regressors are omitted.

Sharp adaptation for inverse problems with random noise

We consider a heteroscedastic sequence space setup with polynomially increasing variances of observations that allows to treat a number of inverse problems, in particular multivariate ones. We propose an adaptive estimator that attains simultaneously exact asymptotic minimax constants on every ellipsoid of functions within a wide scale (that includes ellipoids with polynomially and exponentially decreasing axes) and, at the same time, satisfies asymptotically exact oracle inequalities within any class of linear estimates having monotone non-increasing weights. The construction of the estimator is based on a properly penalized blockwise Stein's rule, with weakly geometically increasing blocks. As an application, we construct sharp adaptive estimators in the problems of deconvolution and tomography.

ON THE USE OF THE STEIN VARIANCE ESTIMATOR IN THE DOUBLE k-CLASS ESTIMATOR IN REGRESSION

This paper investigates the predictive mean squared error performance of a modified double k-class estimator by incorporating the Stein variance estimator. Recent studies show that the performance of the Stein rule estimator can be improved by using the Stein variance estimator. However, as we demonstrate below, this conclusion does not hold in general for all members of the double k-class estimators. On the other hand, an estimator is found to have smaller predictive mean squared error than the Stein variance-Stein rule estimator, over quite large parts of the parameter space.

ON THE USE OF THE STEIN VARIANCE ESTIMATOR IN THE DOUBLE k-CLASS ESTIMATOR IN REGRESSION

This paper investigates the predictive mean squared error performance of a modified double k-class estimator by incorporating the Stein variance estimator. Recent studies show that the performance of the Stein rule estimator can be improved by using the Stein variance estimator. However, as we demonstrate below, this conclusion does not hold in general for all members of the double k-class estimators. On the other hand, an estimator is found to have smaller predictive mean squared error than the Stein variance-Stein rule estimator, over quite large parts of the parameter space.