An Effective Two-Stage Feature Selection Method with Parameters Optimized by Simulated Annealing Algorithm

Abstract

With the increase of the dimension of the collected data, the research of feature selection has gained more and more attention in recent years. As an essential preprocessing method, feature selection can not only reduce the cost of modeling by deleting those unnecessary features, but also improve the precision of the model in some degree. In this paper, an effective feature selection method is proposed, which is based on Mutual Information and Cosine Distance with parameters optimized by Simulated Annealing algorithm (MICD-SA). The proposed method achieves the goal of dimension reduction through two-stage feature selection operations. First, features are selected based on the mutual information between features and labels. Then, cosine distance between each feature is introduced to further reduce the redundancy of the selected features in stage one. Finally, the simulated annealing algorithm is adopted to automatically optimize the two thresholds used in the previous two-stage feature selection process. In addition, the proposed method is also applied to four high dimensional datasets to test its feasibility and effectiveness. Three feature selection methods combined with four different classifiers (KNN, CART, SVM, NB)are tested and their results are compared. The proposed method's effectiveness both in accuracy and feature number are well proved by the comparison.