Information Theoretic-based Feature Selection (IFS) for Machine Learning

Abstract

Mutual Information (MI) has been a popular choice for the selection criterion function of the feature selection techniques. However, only a handful of methods of formulating MI estimator for regression tasks exist, despite the existence of numerous methods of MI estimation for classification tasks in the literature. This work proposed a feature selection method based on mutual information for both classification and regression tasks. Unlike the existing MI estimator for regression tasks which are widely non-parametric, the proposed method provides a suitable way to estimate MI based on conventional kernel density estimation numerically. Two different experiments were conducted to evaluate the proposed IFS method. The first test is designed to assess the correctness of the formulated MI estimator by comparing it with various existing benchmarks. The second experiment evaluates the proposed method using natural domain data. Overall results obtained from these experiments show that the proposed method was successfully able to select relevant features.