Shrinking toward submodels in regression

Abstract

A normal-equation approach is introduced for converting any shrinkage estimator, such as a ridge or Stein estimator, into an estimator that shrinks toward a subspace. The resulting subspace-Stein estimator has been derived previously, in a different way, but the resulting subspace-ridge estimator is new. Unified treatments of subspace-shrinkage estimation are presented in terms of generalized ridge estimation and in terms of Bayesian estimation. The choice of shrinkage coefficient and the choice of subspace are discussed. We have considered both prior and data-adaptive selection of the subspace. Subspace-shrinkage estimation can perform substantially better than ordinary least-squares estimation if variable selection is appropriate and, moreover, does not entail as much risk of poor performance in case variable selection is not appropriate.