Well-known Feature Selection Methods Research Articles

EDDE–LNS: a new hybrid ensemblist approach for feature selection

Feature selection is the process of selecting a subset of relevant, non-redundant features from the original ones. It is an NP-hard combinatorial optimization problem. In this paper, we propose a new feature selection method, abbreviated as EDDE–LNS, using a combination of large neighbourhood search (LNS) and a new Ensemblist Discrete Differential Evolution (EDDE). Each solution of the search space represents a feature subset of predefined size K. EDDE–LNS explores this search space by evolving a population of individuals in two phases. During the first phase, the LNS strategy is used to improve each feature subset by alternately destroying and repairing it. The proposed accuracy rate difference measure is used to determine irrelevant and redundant features that are removed during the application of the destruction process. In the second phase, the individuals resulting from the application of LNS are used as inputs to the proposed EDDE approach. EDDE is a discrete algorithm inspired by the differential evolution (DE) method. Whereas the original DE method attempts to find the best feature subset in a multidimensional space by applying simple and fast arithmetic operators to each dimension (feature) separately, the EDDE approach proposed in this paper attempts to find the best feature subset in a single dimension space by applying new ensemblist operators to a set of K features. In this way, EDDE will consider the possible interactions between features. Experiments are conducted on intrusion detection and other machine learning datasets. The results indicate that the proposed approach is able to achieve good accuracies in comparison with other well-known feature selection methods.

Group-penalized feature selection and robust twin SVM classification via second-order cone programming

Selecting the relevant factors in a particular domain is of utmost interest in the machine learning community. This paper concerns the feature selection process for twin support vector machine (TWSVM), a powerful classification method that constructs two nonparallel hyperplanes in order to define a classification rule. Besides the Euclidean norm, our proposal includes a second regularizer that aims at eliminating variables in both twin hyperplanes in a synchronized fashion. The baseline classifier is a twin SVM implementation based on second-order cone programming, which confers robustness to the approach and leads to potentially better predictive performance compared to the standard TWSVM formulation. The proposal is studied empirically and compared with well-known feature selection methods using microarray datasets, on which it succeeds at finding low-dimensional solutions with highest average performance among all the other methods studied in this work.

Enhancing Effectiveness of Dimension Reduction in Text Classification

Nowadays, text is one prevalent forms of data and text classification is a widely used data mining task, which has various application fields. One mass-produced instance of text is email. As a communication medium, despite having a lot of advantages, email suffers from a serious problem. The number of spam emails has steadily increased in the recent years, leading to considerable irritation. Therefore, spam detection has emerged as a separate field of text classification. A primary challenge of text classification, which is more severe in spam detection and impedes the process, is high-dimensionality of feature space. Various dimension reduction methods have been proposed that produce a lower dimensional space compared to the original. These methods are divided mainly into two groups: feature selection and feature extraction. This research deals with dimension reduction in the text classification task and especially performs experiments in the spam detection field. We employ Information Gain (IG) and Chi-square Statistic (CHI) as well-known feature selection methods. Also, we propose a new feature extraction method called Sprinkled Semantic Feature Space (SSFS). Furthermore, this paper presents a new hybrid method called IG_SSFS. In IG_SSFS, we combine the selection and extraction processes to reap the benefits from both. To evaluate the mentioned methods in the spam detection field, experiments are conducted on some well-known email datasets. According to the results, SSFS demonstrated superior effectiveness over the basic selection methods in terms of improving classifiers’ performance, and IG_SSFS further enhanced the performance despite consuming less processing time.

RJMIM: A New Feature Selection Method Based On Joint Mutual Information

Feature selection based on information theory plays an important role in classification algorithm due to its computational efficiency and independent from classification method. It is widely used in many application areas like datamining, bioinformatics and machine learning. But drawbacks of these methods are the neglect of the feature interaction and overestimation of features significance due to the limitations of goal functions criterion. To address this problem, we proposed a new feature goal function RJMIM. The method employed joint mutual information and information interaction,which alleviates the shortcomings of overestimation of the feature significance as demonstrated both theoretically and experimentally. The experiments conducted to verify the performance of the proposed method, it compared with four well-known feature selection methods use three publically available datasets from UCI.The average classification accuracy and C4.5 classifier is used to assess the effectiveness of RJMIM method.

Using an ensemble classifier based on sequential floating forward selection for financial distress prediction problem

Financial distress prediction (FDP) is a significant issue investigated by researchers, credit institutions and banks. Although extensive research has been conducted in this area, applications of combined feature selection (FS) methods and classification models are subjects that have been addressed intensely in recent years. One of the most important issues in the FDP problem is to employ an effective FS algorithm, leading to an acceptable level of performance accuracy in the implementation stage. Hence, this study primarily attempted to introduce a precise FS model and compared the obtained results with those of other conventional models tackling FDP in terms of accuracy. The proposed method involved the sequential floating forward selection (SFFS) algorithm applied as a wrapper FS technique to determine the best subset of features. At the classification stage, the support vector machine (SVM), owing to its good performance, demonstrated in numerous studies, in solving classification problems, was deployed. The performance of the proposed method was compared with those of other current well-known FS methods including artificial bee colony (ABC), genetic algorithm (GA) and sequential forward selection (SFS) (all of which are categorized under wrapper methods), and principal component analysis (PCA), relief and information gain (IG) (best known as filter techniques) for our given datasets. The results indicated that a combined model of SVM based on the SFFS approach can yield greater accuracy than the other methods applied for our defined domestic and foreign datasets. Therefore, the SFFS-SVM ensemble classifier can be considered a promising addition to existent models when confronting the FDP issue.

Fuzzy clustering-based feature extraction method for mental task classification

A brain computer interface (BCI) is a communication system by which a person can send messages or requests for basic necessities without using peripheral nerves and muscles. Response to mental task-based BCI is one of the privileged areas of investigation. Electroencephalography (EEG) signals are used to represent the brain activities in the BCI domain. For any mental task classification model, the performance of the learning model depends on the extraction of features from EEG signal. In literature, wavelet transform and empirical mode decomposition are two popular feature extraction methods used to analyze a signal having non-linear and non-stationary property. By adopting the virtue of both techniques, a theoretical adaptive filter-based method to decompose non-linear and non-stationary signal has been proposed known as empirical wavelet transform (EWT) in recent past. EWT does not work well for the signals having overlapped in frequency and time domain and failed to provide good features for further classification. In this work, Fuzzy c-means algorithm is utilized along with EWT to handle this problem. It has been observed from the experimental results that EWT along with fuzzy clustering outperforms in comparison to EWT for the EEG-based response to mental task problem. Further, in case of mental task classification, the ratio of samples to features is very small. To handle the problem of small ratio of samples to features, in this paper, we have also utilized three well-known multivariate feature selection methods viz. Bhattacharyya distance (BD), ratio of scatter matrices (SR), and linear regression (LR). The results of experiment demonstrate that the performance of mental task classification has improved considerably by aforesaid methods. Ranking method and Friedman’s statistical test are also performed to rank and compare different combinations of feature extraction methods and feature selection methods which endorse the efficacy of the proposed approach.

Feature selection for ECG signal processing using improved genetic algorithm and empirical mode decomposition

Abstract This paper proposes a novel scheme of feature selection, which employs a modified genetic algorithm that uses a variable-range searching strategy and empirical mode decomposition (EMD). Combined with support vector machines (SVMs), a new pattern recognition method for electrocardiograph (ECG) is developed. First, the ECG signal is decomposed into intrinsic mode functions (IMFs) that represent signal characteristics with sample oscillatory modes. Then, the modified genetic algorithm with variable-range encoding and dynamic searching strategy is used to optimize statistical feature subsets. Next, a statistical model based on receiver operating characteristic (ROC) analysis is developed to select the dominant features. Finally, the SVM-based pattern recognition model is used to classify different ECG patterns. Comparative studies with peer-reviewed results and two other well-known feature selection methods demonstrate that the proposed method can select dominant features in processing ECG signal, and achieve better classification performance with lower feature dimensionality.

Hybrid feature selection based on enhanced genetic algorithm for text categorization

This paper proposes hybrid feature selection approaches based on the Genetic Algorithm (GA). This approach uses a hybrid search technique that combines the advantages of filter feature selection methods with an enhanced GA (EGA) in a wrapper approach to handle the high dimensionality of the feature space and improve categorization performance simultaneously. First, we propose EGA by improving the crossover and mutation operators. The crossover operation is performed based on chromosome (feature subset) partitioning with term and document frequencies of chromosome entries (features), while the mutation is performed based on the classifier performance of the original parents and feature importance. Thus, the crossover and mutation operations are performed based on useful information instead of using probability and random selection. Second, we incorporate six well-known filter feature selection methods with the EGA to create hybrid feature selection approaches. In the hybrid approach, the EGA is applied to several feature subsets of different sizes, which are ranked in decreasing order based on their importance, and dimension reduction is carried out. The EGA operations are applied to the most important features that had the higher ranks. The effectiveness of the proposed approach is evaluated by using naïve Bayes and associative classification on three different collections of Arabic text datasets. The experimental results show the superiority of EGA over GA, comparisons of GA with EGA showed that the latter achieved better results in terms of dimensionality reduction, time and categorization performance. Furthermore, six proposed hybrid FS approaches consisting of a filter method and the EGA are applied to various feature subsets. The results showed that these hybrid approaches are more effective than single filter methods for dimensionality reduction because they were able to produce a higher reduction rate without loss of categorization precision in most situations.

Feature selection using feature dissimilarity measure and density-based clustering: Application to biological data

Reduction of dimensionality has emerged as a routine process in modelling complex biological systems. A large number of feature selection techniques have been reported in the literature to improve model performance in terms of accuracy and speed. In the present article an unsupervised feature selection technique is proposed, using maximum information compression index as the dissimilarity measure and the well-known density-based cluster identification technique DBSCAN for identifying the largest natural group of dissimilar features. The algorithm is fast and less sensitive to the user-supplied parameters. Moreover, the method automatically determines the required number of features and identifies them. We used the proposed method for reducing dimensionality of a number of benchmark data sets of varying sizes. Its performance was also extensively compared with some other well-known feature selection methods.

Minimal neighborhood redundancy maximal relevance: Application to the diagnosis of Alzheimer׳s disease

Existing feature selection methods are able to choose discriminative features with low redundancy but are computationally too expensive for neuroimaging applications. This occurs because they analyze every brain voxel while trying to reduce the redundancy between the selected features. We propose a significantly faster method that focuses on the main source of redundancy which is neighboring voxels and compare this new approach with four other well-known feature selection methods, evaluating them extensively on three datasets. We start by using an artificial dataset to study the robustness of our approach to noisy features, erroneous labels and small number of samples, which are problems that are often encountered when building a CAD system that takes brain images as its input. Then, we analyze the computational complexity of our method and study its usefulness for the diagnosis of Alzheimer׳s disease and Mild Cognitive Impairment using FDG-PET images and tissue probability maps of Gray-Matter extracted from MR images. Experimental results on synthetic and real data clearly validate our approach as a very efficient algorithm for the selection of non-redundant features applicable to a variety of neuroimaging techniques. In fact, the major computational gains come at no cost in either performance or robustness.

A review of microarray datasets and applied feature selection methods

Microarray data classification is a difficult challenge for machine learning researchers due to its high number of features and the small sample sizes. Feature selection has been soon considered a de facto standard in this field since its introduction, and a huge number of feature selection methods were utilized trying to reduce the input dimensionality while improving the classification performance. This paper is devoted to reviewing the most up-to-date feature selection methods developed in this field and the microarray databases most frequently used in the literature. We also make the interested reader aware of the problematic of data characteristics in this domain, such as the imbalance of the data, their complexity, or the so-called dataset shift. Finally, an experimental evaluation on the most representative datasets using well-known feature selection methods is presented, bearing in mind that the aim is not to provide the best feature selection method, but to facilitate their comparative study by the research community.

Improved feature-selection method considering the imbalance problem in text categorization.

The filtering feature-selection algorithm is a kind of important approach to dimensionality reduction in the field of the text categorization. Most of filtering feature-selection algorithms evaluate the significance of a feature for category based on balanced dataset and do not consider the imbalance factor of dataset. In this paper, a new scheme was proposed, which can weaken the adverse effect caused by the imbalance factor in the corpus. We evaluated the improved versions of nine well-known feature-selection methods (Information Gain, Chi statistic, Document Frequency, Orthogonal Centroid Feature Selection, DIA association factor, Comprehensive Measurement Feature Selection, Deviation from Poisson Feature Selection, improved Gini index, and Mutual Information) using naïve Bayes and support vector machines on three benchmark document collections (20-Newsgroups, Reuters-21578, and WebKB). The experimental results show that the improved scheme can significantly enhance the performance of the feature-selection methods.

Image steganalysis using a bee colony based feature selection algorithm

Feature selection is one of the most significant phases of pre-analysis processing, which can influence the performance of steganalysis. In this paper, we have proposed a new feature-based blind steganalysis method for detecting stego images from the cover images in JPEG images using a feature selection technique based on artificial bee colony (IFAB). Most usual techniques for feature selection are wrapper methods and filter methods which IFAB is one of the wrapper based feature selection methods. Artificial bee colony (ABC) algorithm is inspired by honey bees' social behavior in their search for perfect food sources. However, in the suggested algorithm, classifier performance and the dimension of the selected feature vector are dependent on heuristic information for ABC. As a result, we can choose the adaptive feature subset with respect to the shortest feature dimension and the improved performance of the classifier. The experimental results show that the proposed approach is easy to be employed for steganalysis purposes. Moreover, since IFAB is used as one of wrapper methods, as a result, its overall performance is better than several recent and well-known feature selection methods.

Feature Interaction Maximisation

Feature selection plays an important role in classification algorithms. It is particularly useful in dimensionality reduction for selecting features with high discriminative power. This paper introduces a new feature-selection method called Feature Interaction Maximisation (FIM), which employs three-way interaction information as a measure of feature redundancy. It uses a forward greedy search to select features which have maximum interaction information with the features already selected, and which provide maximum relevance. The experiments conducted to verify the performance of the proposed method use three datasets from the UCI repository. The method is compared with four other well-known feature-selection methods: Information Gain (IG), Minimum Redundancy Maximum Relevance (mRMR), Double Input Symmetrical Relevance (DISR), and Interaction Gain Based Feature Selection (IGFS). The average classification accuracy of two classifiers, Naïve Bayes and K-nearest neighbour, is used to assess the performance of the new feature-selection method. The results show that FIM outperforms the other methods.

Incremental filter and wrapper approaches for feature discretization

Discrete data representations are necessary, or at least convenient, in many machine learning problems. While feature selection (FS) techniques aim at finding relevant subsets of features, the goal of feature discretization (FD) is to find concise (quantized) data representations, adequate for the learning task at hand. In this paper, we propose two incremental methods for FD. The first method belongs to the filter family, in which the quality of the discretization is assessed by a (supervised or unsupervised) relevance criterion. The second method is a wrapper, where discretized features are assessed using a classifier. Both methods can be coupled with any static (unsupervised or supervised) discretization procedure and can be used to perform FS as pre-processing or post-processing stages. The proposed methods attain efficient representations suitable for binary and multi-class problems with different types of data, being competitive with existing methods. Moreover, using well-known FS methods with the features discretized by our techniques leads to better accuracy than with the features discretized by other methods or with the original features.

The Combined Effect of Applying Feature Selection and Parameter Optimization on Machine Learning Techniques for Solar Power Prediction

This paper empirically shows that the combined effect of applying the selected feature subsets and optimized parameters on machine learning techniques significantly improves the accuracy for solar power prediction. To provide evidence, experiments are carried on in two phases. For all the experiments the machine learning techniques namely Least Median Square (LMS), Multilayer Perceptron (MLP) and Support Vector Machine (SVM) are used. In the first phase five well-known wrapper feature selection methods are used to obtain the prediction accuracy of machine learning techniques with selected feature subsets and default parameter settings. The experiments from the first phase demonstrate that holding the default parameters, LMS, MLP and SVM provides better prediction accuracy (i.e. reduced MAE and MASE) with selected feature subsets rather than without selected feature subsets. After getting improved prediction accuracy from the first phase, the second phase continues the experiments to optimize machine learning parameters and the prediction accuracy of those machine learning techniques are re-evaluated through adopting both the optimized parameter settings and selected feature subsets. The comparison between the results of two phases clearly shows that the later phase (i.e. machine learning techniques with selected feature subsets and optimized parameters) provides substantial improvement in the accuracy for solar power prediction than the earlier phase (i.e. machine learning techniques with selected feature subsets and default parameters). Experiments are carried out using reliable and real life historical meteorological data. The machine learning accuracy of solar radiation prediction is justified in terms of statistical error measurement and validation metrics. Experimental results of this paper facilitate to make a concrete verdict that providing more attention and effort towards the feature subset selection and machine learning parameter optimization (e.g. combined effect of selected feature subsets and optimized parameters on prediction accuracy which is investigated in this paper) can significantly contribute to improve the accuracy of solar power prediction.

The Effectiveness of Feature Selection Method in Solar Power Prediction

This paper empirically shows that the effect of applying selected feature subsets on machine learning techniques significantly improves the accuracy for solar power prediction. Experiments are performed using five well-known wrapper feature selection methods to obtain the solar power prediction accuracy of machine learning techniques with selected feature subsets. For all the experiments, the machine learning techniques, namely, least median square (LMS), multilayer perceptron (MLP), and support vector machine (SVM), are used. Afterwards, these results are compared with the solar power prediction accuracy of those same machine leaning techniques (i.e., LMS, MLP, and SVM) but without applying feature selection methods (WAFS). Experiments are carried out using reliable and real life historical meteorological data. The comparison between the results clearly shows that LMS, MLP, and SVM provide better prediction accuracy (i.e., reduced MAE and MASE) with selected feature subsets than without selected feature subsets. Experimental results of this paper facilitate to make a concrete verdict that providing more attention and effort towards the feature subset selection aspect (e.g., selected feature subsets on prediction accuracy which is investigated in this paper) can significantly contribute to improve the accuracy of solar power prediction.

Feature subset selection using differential evolution and a wheel based search strategy

Differential evolution has started to attract a lot of attention as a powerful search method and has been successfully applied to a variety of applications including pattern recognition. One of the most important tasks in many pattern recognition systems is to find an informative subset of features that can effectively represent the underlying problem. Specifically, a large number of features can affect the system's classification accuracy and learning time. In order to overcome such problems, we propose a new feature selection method that utilizes differential evolution in a novel manner to identify relevant feature subsets. The proposed method aims to reduce the search space using a simple, yet powerful, procedure that involves distributing the features among a set of wheels. Two versions of the method are presented. In the first one, the desired feature subset size is predefined by the user, while in the second the user only needs to set an upper limit to the feature subset size. Experiments on a number of datasets with different sizes proved that the proposed method can achieve remarkably good results when compared with some of the well-known feature selection methods.

Feature selection based on max–min-associated indices for classification of remotely sensed imagery

This article proposes two novel feature selection methods for dimension reduction according to max–min-associated indices derived from Cramer's V-test coefficient. The proposed methods incrementally select features simultaneously satisfying the criteria of a statistically maximal association (A) between target labels and features and a minimal association (R) among selected features with respect to Cramer's V-test value. Two indices are developed by different combinations of the A and R conditions. One index is to maximize A/R and the other is to maximize A–λR, which are referred to as the MMAIQ and MMAIS methods, respectively. Since the proposed feature selection algorithms are feature filter methods, how to determine the best number of features is another important issue. This article adopts an information lost criterion by measuring the variation between χ2 and β statistics to optimize the number of features selected associated with the Gaussian maximal likelihood classifier (GMLC). To validate the proposed methods, experiments are conducted with both a hyperspectral image data set and a high spatial resolution image data set. The results demonstrate that the proposed methods can provide an effective tool for feature selection and improve classification accuracy significantly. Furthermore, the proposed methods with well-known feature selection methods, i.e. mutual information-based max-dependency criterion (mRMR) and sequential forward selection (SFS), are evaluated and compared. The experiments demonstrate that the proposed methods can offer better results in terms of kappa coefficient and overall classification accuracy measurements.

Power transformer protection scheme based on time-frequency analysis

SUMMARY This article presents a new approach for power transformer protection based on a hybrid pattern recognition scheme. The hyperbolic S-transform (HST) is a very powerful tool for analysis of the nonstationary waveforms because it is able to extract the information from transient signals simultaneously in both time and frequency domains. Magnitude, frequency, and HST contours are the main attributes obtained from the output matrix of HST. At first, differential current waveforms of different conditions such as normal, internal and external faults, inrush, and over-excitation conditions are analyzed by HST and some potential useful features are extracted from the abovementioned contours. To decrease the dimension of feature vector and to increase the classification accuracy of the proposed algorithm, the most effective features are selected by using well-known feature selection methods namely sequential forward selection, sequential backward selection, and genetic algorithm. Selected features are trained by a probabilistic neural network as an effective classifier core, which has advantages regarding learning speed and generalization capability compared with feed-forward neural network. The classification accuracy of the proposed algorithm has been used as a criterion function for the selection of the best subset features. The proposed protection scheme is evaluated for various operating conditions of three different transformers using the PSCAD/EMTDC package. Extensive simulation results show that the proposed algorithm relies only on the waveshape properties, and it is independent of the value of transformer parameters and consumed power. Copyright © 2012 John Wiley & Sons, Ltd.