Filter Feature Selection Algorithms Research Articles

A new filter feature selection algorithm for classification task by ensembling pearson correlation coefficient and mutual information

Feature selection is widely used in various fields as a key means of data dimension reduction. The existing feature selection algorithms only use one linear or nonlinear correlation indicator when evaluating variables relationships, which lacks diversity. Considering the complexity of the relationship between features, a novel feature selection evaluation function CONMI is constructed, which ensembles Pearson correlation coefficient (liner) and normalized mutual information (non-linear) to comprehensively portrays the dependencies between features and class variables. We further propose the CONMI_FS algorithm based on CONMI, which selects the optimal subset of features that has high correlation with the class variables and low redundancy between the selected features. CONMI_FS is compared with four methods on 20 datasets and evaluated by reduction rate, classification accuracy, precision and recall metrics on KNN, SVM and DT classifiers. The experimental results show that CONMI_FS obtains the highest reduction rate of 80.04%, and achieves the best classification accuracy on KNN and SVM classifiers, which are 88.83% and 88.98%, respectively. These results indicate that CONMI_FS has good competitiveness.

FG-HFS: A feature filter and group evolution hybrid feature selection algorithm for high-dimensional gene expression data

High dimensional and small samples characterize gene expression data and contain a large number of genes unrelated to disease. Feature selection improves the efficiency of disease diagnosis by selecting a small number of important genes. Unfortunately, existing algorithms do not consider the correlation between features, and search algorithms tend to fall into the local optimal solution in the feature search process. To this end, this paper proposes a feature filter and group evolution hybrid feature selection algorithm (FG-HFS) for high-dimensional gene expression data. Unlike existing algorithms, we propose using spectral clustering to group redundant features into a group. Then, we propose a redundant feature filter algorithm. According to the principle of approximate Markov blanket, grouped feature groups are filtered to delete these redundant features. Among them, filtered features are evenly divided by density according to the feature exponential strategy. Most importantly, we propose using the group evolution multi-objective genetic algorithm to search the filtered feature subsets and evaluate the candidate feature subsets according to the in-group and out-group so as to select the feature subsets with the highest accuracy and the least number. Experimental results show that the average accuracy (ACC) and Matthews correlation coefficient (MCC) indexes of the selected feature subsets (FSs) by the FG-HFS algorithm on 5 gene expression datasets are 92.76% and 88.76%, respectively, which are significantly better than the existing algorithms. In addition, the FSs and ACC/FSs indexes of the FG-HFS algorithm are also better than the existing algorithms, which fully proves the superiority of the FG-HFS algorithm. More importantly, the Wilcoxon and Friedman statistical experiments results show that the feature selection effect of FG-HFS algorithm is significantly better than that of existing algorithms, no matter in pairwise comparison or multiple comparison.

Evaluation of Boulder Characteristics for Improved Boulder Detection Based on Machine Learning Techniques

Detailed maps of the seabed and knowledge of its habitats are critical for a wide range of tasks, such as sustainable development, and environmental protection. Boulders on the seabed form an important environment for ecosystems, but the detection of them is challenging. In this study, we aim to improve the understanding of boulder predictors and to determine connections between predictors and boulder environments on different spatial scales. The Relief-F filter feature selection algorithm was used on four 30 m × 30 m areas in Rødsand lagoon, containing one boulder each, to determine the most relevant predictors. The predictors could be divided into four groups detecting different boulder characteristics: colour contrast, height, boulder boundaries, and spherical geometry. Twelve different types of boulder environments were evaluated. Bare, spherical boulders on sandy seabeds can be predicted from all four predictor groups. It is not possible to detect non-spherical boulders on seabed covered by vegetation. The best predictors for boulder detection depend on the shape and size of the boulder and the surrounding sediment and vegetation. The predictors were evaluated on a larger 400 × 2500 m area. When up-scaling the boulder detection area, larger seabed structures may affect the results. Therefore, knowledge about these structures can be used to remove errors and uncertainties from machine learning input data.

A Novel Nonparametric Feature Selection Approach Based on Mutual Information Transfer Network.

The filter feature selection algorithm is habitually used as an effective way to reduce the computational cost of data analysis by selecting and implementing only a subset of original features into the study. Mutual information (MI) is a popular measurement adopted to quantify the dependence among features. MI-based greedy forward methods (MIGFMs) have been widely applied to escape from computational complexity and exhaustion of high-dimensional data. However, most MIGFMs are parametric methods that necessitate proper preset parameters and stopping criteria. Improper parameters may lead to ignorance of better results. This paper proposes a novel nonparametric feature selection method based on mutual information and mixed-integer linear programming (MILP). By forming a mutual information network, we transform the feature selection problem into a maximum flow problem, which can be solved with the Gurobi solver in a reasonable time. The proposed method attempts to prevent negligence on obtaining a superior feature subset while keeping the computational cost in an affordable range. Analytical comparison of the proposed method with six feature selection methods reveals significantly better results compared to MIGFMs, considering classification accuracy.

A multi-label feature selection method based on an approximation of interaction information

High-dimensional multi-label data is widespread in practical applications, which brings great challenges to the research field of pattern recognition and machine learning. Many feature selection algorithms have been proposed in recent years, among which the filtering feature selection algorithm is the most popular one because of its simplicity. Therefore, filtering feature selection has become a hot research topic, especially the multi-label feature selection algorithm based on mutual information. In the algorithm, the computation cost of high dimensional mutual information is expensive. How to approximate high order mutual information based on low order mutual information has become a major research direction. To our best knowledge, all existing feature selection algorithms that consider the label correlation will increase the computational cost greatly. Therefore, this paper proposes an approximation method of three-dimensional interaction information, which is applied to the calculation of correlation and redundancy. It can take the correlation of labels into account and don’t increase the computation cost significantly at the same time. Experiments analysis results show that the proposed method is effective.

A novel filter feature selection algorithm based on relief

A novel filter feature selection algorithm based on relief

FILTER FEATURE SELECTION ANALYSIS TO DETERMINE THE CHARACTERISTICS OF DEMENTIA

Dementias are known as neuropsychiatric disorders. As getting old, the chance of coming down with a dementia disease increases. Two-dimensional sliced brain scans can be generated via magnetic resonance imaging. Three-dimensional measurements of regions can be reached from those scans. For the samples in the ADNI dataset, the brain features are extracted through operating the Freesurfer brain analyzing tool. Parametrizing those features and demographic information in learning algorithms can label an unknown sample as healthy or dementia. On the other hand, some of the features in the initial set may be less practical than others. In this research, the aim is to decrease the feature-size, not the feature-dimension, as a first step to determine the most distinctive dementia characteristics. To that end, a total of 2264 samples (471 AD, 428 lMCI, 669 eMCI, 696 healthy controls) are divided into two sets: 65% training set (1464 samples) and 35% test set (800 samples). Various filter feature selection algorithms are tested over different parameters together with multiple Bayesian-based and tree-based classifiers. Test performance accuracy rates up to 76.50% are analyzed in detail. Instead of processing the whole feature set, the overall performance tends to increase with correctly fewer attributes taken.

TAGA: Tabu Asexual Genetic Algorithm embedded in a filter/filter feature selection approach for high-dimensional data

Feature selection is the process of selecting an optimal subset of features required for maintaining or improving the performance of data mining models. Recently, hybrid filter/wrapper feature selection methods have shown promising results for high-dimensional data. However, filter/wrapper methods lack of generalisation power, which enables the selected features to be trainable over different classifiers without having to repeat the feature selection process. To address the generalisation power problem, this paper proposes a novel evolutionary-based filter feature selection algorithm that is sequentially hybridised with the Fisher score filter algorithm in a new hybrid framework called filter/filter. The proposed algorithm is based on a long-term memory Tabu Search combined with an Asexual (i.e. mutation-based) Genetic Algorithm (TAGA). TAGA benefits from a new integer-encoded solution representation, a novel mutation operator, a new tabu list encoding scheme, and uses a minimum redundancy maximum relevance information theory-based criterion as the fitness function. Experiments were carried out on various high-dimensional datasets including image, text, and biological data. The goodness of the selected subsets was evaluated using different classifiers and the experimental results demonstrate that TAGA outperforms other conventional and state-of-the-art feature selection algorithms.

Attack Transferability Against Information-Theoretic Feature Selection

Machine learning (ML) is vital to many application-driven fields, such as image and signal classification, cyber-security, and health sciences. Unfortunately, many of these fields can easily have their training data tampered with by an adversary to thwart an ML algorithm’s objective. Further, the adversary can impact any stage in an ML pipeline (e.g., preprocessing, learning, and classification). Recent work has shown that many models can be attacked by poisoning the training data, and the impact of the poisoned data can be quite significant. Prior works on adversarial feature selection have shown that the attacks can damage feature selection (FS). Filter FS algorithms, a type of FS, are widely used for their ability to model nonlinear relationships, classifier independence and lower computational requirements. One important question from the security perspective of these widely used approaches is, whether filter FS algorithms are robust against other FS attacks. In this work, we focus on the task of information-theoretic filter FS such MIM, MIFS, and mRMR, and the impact that gradient-based attack can have on these selections. The experiments on five benchmark datasets demonstrate that the stability of different information-theoretic algorithms can be significantly degraded by injecting poisonous data into the training dataset.

An Effective Filter Method Towards the Performance Improvement of FF-SVM Algorithm

Fining effective and informative biomarker genes form microarray is very challenging. In order to develop an hybrid gene selection algorithm, numerous filter feature selection algorithms have been previously reported. This research paper aims to identify the filter method that will improve the performance of our previously proposed FF-SVM algorithm to find the minimum number of accurate genes that achieves high accuracy performance. Therefore, an experiment was conducted using four different filter methods: Maximum Relevance Minimum Redundancy (mRMR), Joint Mutual Information (JMI), F-score, and Double Input Symmetrical Relevance (DISR). This experiment was undertaken in two phases: the first phase was filter to SVM , to identify the minimum number of features (genes) which served to maximize the SVM classifier; the second phase was filter to FF-SVM , to ascertain the best suite filter method to our previously proposed FF-SVM algorithm. The result of this experiment would be the most suited filter method to the FF-SVM. In conclusion, we found that the f-score method outperformed other filter methods when combined with FF-SVM.

A multi-objective algorithm for multi-label filter feature selection problem

Feature selection is an important data preprocessing method before classification. Multi-objective optimization algorithms have been proved an effective way to solve feature selection problems. However, there are few studies on multi-objective optimization feature selection methods for multi-label data. In this paper, a multi-objective multi-label filter feature selection algorithm based on two particle swarms (MOMFS) is proposed. We use mutual information to measure the relevance between features and label sets, and the redundancy between features, which are taken as two objectives. In order to avoid Particle Swarm Optimization (PSO) from falling into the local optimum and obtaining a false Pareto front, we employ two swarms to optimize the two objectives separately and propose an improved hybrid topology based on particle’s fitness value. Furthermore, an archive maintenance strategy is introduced to maintain the distribution of archive. In order to study the effectiveness of the proposed algorithm, we select five multi-label evaluation criteria and perform experiments on seven multi-label data sets. MOMFS is compared with classic single-objective multi-label feature selection algorithms, multi-objective filter and wrapper feature selection algorithms. The experimental results show that MOMFS can effectively reduce the multi-label data dimension and perform better than other approaches on five evaluation criteria.

Measuring Success of Heterogeneous Ensemble Filter Feature Selection Models

One problem in utilizing ensemble feature selection models is machine learning is the fact that there is no guarantee that an ensemble model will improve machine learning classification performance. This implies that different ensemble models have different success probability, i.e. have different probability in improving the performance of machine learning. This paper introduces the concept of success probability for heterogeneous ensemble models and stated the definitions, notations, and algorithms necessary to the mathematical formulation and computation of the success probability. To show how the theory applied, we create an ensemble filter feature selection model that uses four filter feature selection algorithms (Correlation, Gain Ratio, Info Gain, and One R) as base filters and the Max as a combination method. The experimental results showed that the success probability of the developed ensemble filter model using a set of 9 machine learning algorithms is found to be 0.58.

Classification of high dimensional biomedical data based on feature selection using redundant removal.

High dimensional biomedical data contain tens of thousands of features, accurate and effective identification of the core features in these data can be used to assist diagnose related diseases. However, there are often a large number of irrelevant or redundant features in biomedical data, which seriously affect subsequent classification accuracy and machine learning efficiency. To solve this problem, a novel filter feature selection algorithm based on redundant removal (FSBRR) is proposed to classify high dimensional biomedical data in this paper. First of all, two redundant criteria are determined by vertical relevance (the relationship between feature and class attribute) and horizontal relevance (the relationship between feature and feature). Secondly, to quantify redundant criteria, an approximate redundancy feature framework based on mutual information (MI) is defined to remove redundant and irrelevant features. To evaluate the effectiveness of our proposed algorithm, controlled trials based on typical feature selection algorithm are conducted using three different classifiers, and the experimental results indicate that the FSBRR algorithm can effectively reduce the feature dimension and improve the classification accuracy. In addition, an experiment of small sample dataset is designed and conducted in the section of discussion and analysis to clarify the specific implementation process of FSBRR algorithm more clearly.

Driver Drowsiness Detection Based on Steering Wheel Data Applying Adaptive Neuro-Fuzzy Feature Selection.

This paper presents a novel feature selection method to design a non-invasive driver drowsiness detection system based on steering wheel data. The proposed feature selector can select the most related features to the drowsiness level to improve the classification accuracy. This method is based on the combination of the filter and wrapper feature selection algorithms using adaptive neuro-fuzzy inference system (ANFIS). In this method firstly, four different filter indexes are applied on extracted features from steering wheel data. After that, output values of each filter index are imported as inputs to a fuzzy inference system to determine the importance degree of each feature and select the most important features. Then, the selected features are imported to a support vector machine (SVM) for binary classification to classify the driving conditions in two classes of drowsy and awake. Finally, the classifier accuracy is exploited to adjust parameters of an adaptive fuzzy system using a particle swarm optimization (PSO) algorithm. The experimental data were collected from about 20.5 h of driving in the simulator. The results show that the drowsiness detection system is working with a high accuracy and also confirm that this method is more accurate than the recent available algorithms.

User identification by keystroke dynamics using improved binary particle swarm optimisation

As a kind of behavioural characteristic, keystroke features are crucial to the accuracy of user identification system using shallow machine learning algorithms. Filter and wrapper feature selection algorithms are the two most important methods. The information gain and particle swarm optimisation algorithm represent the two feature optimisation methods, respectively. In this paper, new hybrid binary particle swarm optimisation methods combined with information gain theory are proposed in association with opposite-based learning and distributed techniques. The converted information gain values act as weight coefficients to adaptively adjust the flight speed of particles. The support vector machine (SVM) algorithm is applied to evaluate the performance of feature optimisation in terms of user identification accuracy and feature reduction rate. Experimental results of three public keystroke datasets show that the proposed optimisation methods achieve better classification accuracy with fewer features than four existing optimisation methods.

User identification by keystroke dynamics using improved binary particle swarm optimisation

As a kind of behavioural characteristic, keystroke features are crucial to the accuracy of user identification system using shallow machine learning algorithms. Filter and wrapper feature selection algorithms are the two most important methods. The information gain and particle swarm optimisation algorithm represent the two feature optimisation methods, respectively. In this paper, new hybrid binary particle swarm optimisation methods combined with information gain theory are proposed in association with opposite-based learning and distributed techniques. The converted information gain values act as weight coefficients to adaptively adjust the flight speed of particles. The support vector machine (SVM) algorithm is applied to evaluate the performance of feature optimisation in terms of user identification accuracy and feature reduction rate. Experimental results of three public keystroke datasets show that the proposed optimisation methods achieve better classification accuracy with fewer features than four existing optimisation methods.

Coupling Multivariate Adaptive Regression Spline (MARS) and Random Forest (RF)

In this article, a new algorithm to select the relevant features is proposed for handling microarray data with the specific aim of increasing classification accuracy. In particular, the optimal genes are extracted using filter and wrapper feature selection algorithms. Here, the use of non-parametric regression algorithm called Multivariate Adaptive Regression Spline (MARS) followed by proposed Random Forest Statistical Test (RFST) algorithm are being studied. The study evaluates the comparative performance of the results of RFST and MARS with existing algorithms on ten standard microarray datasets. For performance analysis, three parameters are taken into consideration, namely, the number of selected features, runtime, and classification accuracy. Experimental results indicate that different feature selection algorithms yield different candidate gene subset; therefore, a Hybrid approach is applied to determine the best candidate genes to provide maximum information about the disease. The findings foretell that the RFST is performing better on six out of ten datasets whereas MARS is performing better on other datasets.

Mrmr+ and Cfs+ feature selection algorithms for high-dimensional data

Feature selection is a central issue in machine learning and applied mathematics. Filter feature selection algorithms aim to solve the optimization problem of selecting a set of features that maximize the correlation feature-class and minimize the correlation feature-feature. Mrmr (Minimum Redundancy Maximum Relevance) and Cfs (Correlation-based Feature Selection) are one of the most well-known algorithms that can find an approximate solution to this optimization problem. However, as time passes, the availability of data becomes greater, which makes the feature selection process more challenging. In this paper, we propose two new versions of Mrmr and Cfs that output the same feature set as the original algorithms, but are considerably much faster. Our novel algorithms are based on the solution of the duplication and the redundancy problems intrinsic in the original algorithms. We applied our proposals to thirty datasets related to the field of microarray and cancer analysis. Experiments revealed that the proposed algorithms Mrmr+ and Cfs+ are on average fourteen and three times faster than the original algorithms respectively.

A Filter Feature Selection Algorithm Based on Mutual Information for Intrusion Detection

For a large number of network attacks, feature selection is used to improve intrusion detection efficiency. A new mutual information algorithm of the redundant penalty between features (RPFMI) algorithm with the ability to select optimal features is proposed in this paper. Three factors are considered in this new algorithm: the redundancy between features, the impact between selected features and classes and the relationship between candidate features and classes. An experiment is conducted using the proposed algorithm for intrusion detection on the KDD Cup 99 intrusion dataset and the Kyoto 2006+ dataset. Compared with other algorithms, the proposed algorithm has a much higher accuracy rate (i.e., 99.772%) on the DOS data and can achieve better performance on remote-to-login (R2L) data and user-to-root (U2R) data. For the Kyoto 2006+ dataset, the proposed algorithm possesses the highest accuracy rate (i.e., 97.749%) among the other algorithms. The experiment results demonstrate that the proposed algorithm is a highly effective feature selection method in the intrusion detection.

An OMIC biomarker detection algorithm TriVote and its application in methylomic biomarker detection.

Transcriptomic and methylomic patterns represent two major OMIC data sources impacted by both inheritable genetic information and environmental factors, and have been widely used as disease diagnosis and prognosis biomarkers. Modern transcriptomic and methylomic profiling technologies detect the status of tens of thousands or even millions of probing residues in the human genome, and introduce a major computational challenge for the existing feature selection algorithms. This study proposes a three-step feature selection algorithm, TriVote, to detect a subset of transcriptomic or methylomic residues with highly accurate binary classification performance. TriVote outperforms both filter and wrapper feature selection algorithms with both higher classification accuracy and smaller feature number on 17 transcriptomes and two methylomes. Biological functions of the methylome biomarkers detected by TriVote were discussed for their disease associations. An easy-to-use Python package is also released to facilitate the further applications.