Two tales of variable selection for high dimensional regression: Screening and model building

Abstract

AbstractVariable selection plays an important role in high‐dimensional regression problems where a large number of variables are given as potential predictors of a response of interest. Typically, it arises at two stages of statistical modeling, namely screening and formal model building, with different goals. Screening aims at filtering out irrelevant variables prior to model building where a formal description of a functional relation between the variables screened for relevance and the response is sought. Accordingly, proper comparison of variable selection methods calls for evaluation criteria that reflect the differential goals: accuracy in ranking order of variables for screening and prediction accuracy for formal modeling.Without delineating the difference in the two aspects, confounding comparisons of various screening and selection methods have often been made in the literature, which may lead to misleading conclusions. In this paper, we present comprehensive numerical studies for comparison of four commonly used screening and selection procedures: correlation screening (also known as sure independence screening), forward selection, LASSO and SCAD. By clearly differentiating screening and model building, we highlight the situations where the performance of these procedures might differ. In addition, we propose a new method for cross‐validation for LASSO. Furthermore, we discuss connections to relevant comparison studies that appeared in the recent literature to clarify different findings and conclusions.