On Shrinkage Estimation: Non-orthogonal Case

Abstract

In this paper, we consider the estimation of the parameters of the non-orthogonal regression model, when we suspect a sparsity condition. We provide with a comparative performance characteristics of the primary penalty estimators, namely, the ridge and the LASSO, with the least square estimator, restricted LSE, preliminary test and Stein-type of estimators, when the dimension of the parameter space is less than the dimension of the sample space. Using the principle of marginal distribution theory, the analysis of risks leads to the following conclusions: (i) ridge estimator outperforms least squares, preliminary test and Stein-type estimators uniformly, (ii) The restricted least squares estimator and LASSO are competitive, although LASSO lags behind the restricted least squares estimator uniformly. Both estimators outperform the least squares, preliminary test, and Stein-type estimators in a subspace, respectively. (iii) The lower bound risk expression of LASSO does not depend on the threshold parameter. (iv) Performance of the estimators depends upon the size of numbers of active coefficients, non-active coefficients, and the divergence parameter. In support of our conclusion, we prepare some tables and graphs relevant to the properties of the estimators.