Variable selection by searching for good subsets

Abstract

Machine learning and statistical models are increasingly used in a prediction context and in the process of building these models the question of which variables to include often arises. Over the last 50 years a number of procedures have been proposed, especially in the statistical literature. In this paper a newvariable selection procedure is introduced for linear models. A subset of variables is defined here to be “good at margin λ” if it has two properties, namely (i) its associated criterion of fit will be improved in relative terms by less than λ if any variable is added to it, and (ii) its criterion of fit will deteriorate in relative terms by at least λ if any variable inside it, is dropped from it. Thus, such a subset contains all variables that are individually important and none that are unimportant at a given margin λ ≥ 0. This paper discusses calculation of such λ-good subsets. The “good” approach extends readily to generalised linear and many other models by using an appropriate criterion of performance. The approach is illustrated on an artificial data set and a number of real data sets. Keywords: Good subsets, Linear regression, Logistic regression, Robust regression, Subset selection, Variable importance, Variable selection