Generalized Cross Validation in variable selection with and without shrinkage

Abstract

This paper investigates two types of results that support the use of Generalized Cross Validation (GCV) for variable selection under the assumption of sparsity. The first type of result is based on the well established links between GCV on one hand and Mallows’s Cp and Stein Unbiased Risk Estimator (SURE) on the other hand. The result states that GCV performs as well as Cp or SURE in a regularized or penalized least squares problem as an estimator of the prediction error for the penalty in the neighborhood of its optimal value. This result can be seen as a refinement of an earlier result in GCV for soft thresholding of wavelet coefficients. The second novel result concentrates on the behavior of GCV for penalties near zero. Good behavior near zero is of crucial importance to ensure successful minimization of GCV as a function of the regularization parameter. Understanding the behavior near zero is important in the extension of GCV from ℓ1 towards ℓ0 regularized least squares, i.e., for variable selection without shrinkage, or hard thresholding. Several possible implementations of GCV are compared with each other and with SURE and Cp. These simulations illustrate the importance of the fact that GCV has an implicit and robust estimator of the observational variance.