On tuning parameter selection of lasso-type methods - a monte carlo study

Abstract

In regression analysis, variable selection is a challenging task. Over the last decade, the lasso-type methods have become popular method for variable selection due to their property of shrinking some of the model coefficients to exactly zero. Theory says that lasso-type methods are able to do consistent variable selection but it is hard to achieve this property in practice. This consistent variable selection highly depends on the right choice of the tuning parameter. In this paper, we show that selection of tuning parameter by cross validation almost always fail to achieve consistent variable selection. We have also shown that lasso-type methods with a BIC-type tuning parameter selector, under certain conditions, can do the consistent variable selection. We have also made a novel suggestion for choosing the value of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , a weight on estimated model size, in BIC. Our results show that with this choice of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , the lasso-type methods can do consistent variable selection.In regression analysis, variable selection is a challenging task. Over the last decade, the lasso-type methods have become popular method for variable selection due to their property of shrinking some of the model coefficients to exactly zero. Theory says that lasso-type methods are able to do consistent variable selection but it is hard to achieve this property in practice. This consistent variable selection highly depends on the right choice of the tuning parameter. In this paper, we show that selection of tuning parameter by cross validation almost always fail to achieve consistent variable selection. We have also shown that lasso-type methods with a BIC-type tuning parameter selector, under certain conditions, can do the consistent variable selection. We have also made a novel suggestion for choosing the value of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , a weight on estimated model size, in BIC. Our results show that with this choice of C <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> , the lasso-type methods can do consistent variable selection.