Designing genetic programming classifiers with feature selection and feature construction

Abstract

Due to the flexibility of Genetic Programming (GP), GP has been used for feature construction, feature selection and classifier construction. In this paper, GP classifiers with feature selection and feature construction are investigated to obtain simple and effective classification rules. During the construction of a GP classifier, irrelevant and redundant features affect the search ability of GP, and make GP easily fall into local optimum. This paper proposes two new GP classifier construction methods to restrict bad impact of irrelevant and redundant features on GP classifier. The first is to use a multiple-objective fitness function that decreases both classification error rate and the number of selected features, which is named as GPMO. The second is to first use a feature selection method, i.e., linear forward selection (LFS) to remove irrelevant and redundant features and then use GPMO to construct classifiers, which is named as FSGPMO. Experiments on twelve datasets show that GPMO and FSGPMO have advantages over GP classifiers with a single-objective fitness function named GPSO in term of classification performance, the number of selected features, time cost and function complexity. The proposed FSGPMO can achieve better classification performance than GPMO on higher dimension datasets, however, FSGPMO may remove potential effective features for GP classifier and achieve much lower classification performance than GPMO on some datasets. Compared with two other GP-based classifiers, GPMO can significantly improve the classification performance. Comparisons with other classification algorithms show that GPMO can achieve better or comparable classification performance on most selected datasets. Our proposed GPMO can achieve better performance than wrapper-based feature construction methods using GP on applications with insufficient instances. Further investigations show that bloat phenomena exists in the process of GP evolution and overfitting phenomena is not obvious. Moreover, the benefits of GP over other machine learning algorithms are discussed.