Optimizing feature selection methods by removing irrelevant features using sparse least squares

Abstract

Feature (variable) selection is recognized as an integral part of model construction in machine learning. One can use feature selection to remove redundant and irrelevant features. This in turn can help overcome the curse of dimensionality, reduce overfitting, and come up with interpretable models. In this paper, we propose Sparse Least Squares method (SLS) based on singular value decomposition and least squares to remove irrelevant features. We show that augmenting well-known feature selection methods with SLS significantly reduces the running time while improving or maintaining the prediction accuracy of the model.