On shrinkage and selection: ANOVA model

Abstract

This paper considers the estimation of the parameters of an ANOVA model when sparsity is suspected. Accordingly, we consider the least square estimator (LSE), restricted LSE, preliminary test and Stein-type estimators, together with three penalty estimators, namely, the ridge estimator, subset selection rules (hard threshold estimator) and the LASSO (soft threshold estimator). We compare and contrast the L2-risk of all the estimators with the lower bound of L2-risk of LASSO in a family of diagonal projection scheme which is also the lower bound of the exact L2-risk of LASSO. The result of this comparison is that neither LASSO nor the LSE, preliminary test, and Stein-type estimators outperform each other uniformly. However, when the model is sparse, LASSO outperforms all estimators except “ridge” estimator since both LASSO and ridge are L2-risk equivalent under sparsity. We also find that LASSO and the restricted LSE are L2-risk equivalent and both outperform all estimators (except ridge) depending on the dimension of sparsity. Finally, ridge estimator outperforms all estimators uniformly. Our finding are based on L2-risk of estimators and lower bound of the risk of LASSO together with tables of efficiency and graphical display of efficiency and not based on simulation.