On the role of sparsity in feature selection and an innovative method LRMI

Abstract

Abstract Feature selection is used in many applications in machine learning and bioinformatics. As a popular approach, feature selection can be implemented in the filter-manner based on the sparse solution of the l1 regularization. Most study of the l1 regularization concentrates on investigating the iteration solution of the problem or focuses on adapting sparsity to different applications. It is necessary to explore more deeply about how the sparsity learned with the l1 regularization contributes to feature selection. In this paper, we make an effort to analyze the role of the l1 regularization in feature selection from the perspective of information theory. We discover that the l1 regularization contributes to minimizing the redundancy in feature selection. To avoid the complex computation of the l1 optimization, we propose a novel feature selection algorithm, i.e. the Laplacian regularization based on mutual information (LRMI), which realizes the minimization of the redundancy in a new way, and incorporates the l2 norm to achieve automatic grouping. Extensive experimental results demonstrate the superiority of LRMI over several traditional l1 regularization based feature selection algorithms with less time consumption.