Variable Selection and Shrinkage: Comparison of Some Approaches

Abstract

A common strategy within the framework of regression models is the selection of variables with possible predictive value, which are incorporated in the regression model. Two recently proposed methods, Breiman's Garotte (Breiman, 1995) and Tibshirani's Lasso (Tibshirani, 1996) try to combine variable selection and shrinkage. We compare these with pure variable selection and shrinkage procedures. We consider the backward elimination procedure as a typical variable selection procedure and as an example of a shrinkage procedure an approach of Van Houwelingen and Le Cessie (1990). Additionally an extension of van Houwelingens and le Cessies approach proposed by Sauerbrei (1999) is considered. The ordinary least squares method is used as a reference.With the help of a simulation study we compare these approaches with respect to the distribution of the complexity of the selected model, the distribution of the shrinkage factors, selection bias, the bias and variance of the effect estimates and the average prediction error.