Representative Feature Selection Research Articles

A fuzzy rough set-based feature selection method using representative instances

The fuzzy rough set theory has been widely used to deal with uncertainty in real-valued or even complex data, in which one of the most concerned issues is feature selection. Since a real-world data set generally contains redundant data objects (or instances) and errors which lead to the fact that not all the instances are of equal importance, focusing on the representative instances for feature selection can not only acquire more convincing analysis results but also alleviate computational complexity in mining knowledge. At present, however, little attention has been paid on using representative instances to select features. In this paper, the issue of selecting features by using representative instances is investigated based on fuzzy rough sets and a representative instance-based feature selection approach is proposed. First, the fuzzy granular rule is employed to describe the discriminating information of an instance. Then, the representative instances are selected according to the coverage ability of the fuzzy granular rules induced by all of the instances. Furthermore, an implication relationship-preserved reduction is presented to maintain the discriminating information of the selected instances, and then a heuristic algorithm is presented to search for such a feature subset. Finally, a filter-wrapper approach is suggested to select the best subset of the features. Some numerical experiments are further conducted to show the performance of the proposed feature selection method and the results are satisfactory in terms of both efficiency and effectiveness.

Feature selection for software effort estimation with localized neighborhood mutual information

Feature selection is usually employed before applying case based reasoning (CBR) for Software Effort Estimation (SEE). Unfortunately, most feature selection methods treat CBR as a black box method so there is no guarantee on the appropriateness of CBR on selected feature subset. The key to solve the problem is to measure the appropriateness of CBR assumption for a given feature set. In this paper, a measure called localized neighborhood mutual information (LNI) is proposed for this purpose and a greedy method called LNI based feature selection (LFS) is designed for feature selection. Experiment with leave-one-out cross validation (LOOCV) on 6 benchmark datasets demonstrates that: (1) CBR makes effective estimation with the LFS selected subset compared with a randomized baseline method. Compared with three representative feature selection methods, (2) LFS achieves optimal MAR value on 3 out of 6 datasets with a 14% average improvement and (3) LFS achieves optimal MMRE on 5 out of 6 datasets with a 24% average improvement.

An effective approach for causal variables analysis in diesel engine production by using mutual information and network deconvolution

The effective control of the power consistency, which is one of the most important quality indicators of diesel engine, plays a decisive role for improving the competitiveness of the products. The widely used sensors and other data acquisition equipment make the “data-driven quality control” become possible. However, how to determine the highly related parameters with the engine power from massive captured manufacturing data and effectively discriminated the direct and indirect dependencies between these variables are still challenging. This paper proposed a feature selection algorithm named NMI-ND which uses network deconvolution (ND) to infer causal correlations among various diesel engine manufacturing parameters from the observed correlations based on normalized mutual information (NMI). The proposed algorithm is thoroughly evaluated through the experimental study by comparing it with other representative feature selection algorithms. The comparison demonstrates that NMI-ND performs better in both effectiveness and efficiency.

Hierarchical co-evolutionary clustering tree-based rough feature game equilibrium selection and its application in neonatal cerebral cortex MRI

A wide variety of feature selection methods have been developed as promising solutions to find the classification pattern inside increasing applications. But the exploring efficient, flexible and robust feature selection method to handle the rising big data is still an exciting challenge. This paper presents a novel hierarchical co-evolutionary clustering tree-based rough feature game equilibrium selection algorithm (CTFGES). It aims to select out the high-quality feature subsets, which can enrich the research of feature selection and classification in the heterogeneous big data. Firstly, we construct a flexible hierarchical co-evolutionary clustering tree model to speed up the process of feature selection, which can effectively extract the features from the parent and children branches of four-layer co-evolutionary clustering tree. Secondly, we design a mixed co-evolutionary game equilibrium scheme with adaptive dynamics to guide parent and children branch subtrees to approach the optimal equilibrium regions, and enable their feature sets to converge stably to the Nash equilibrium. So both noisy heterogeneous features and non-identified redundant ones can be further eliminated. Finally, the extensive experiments on various big datasets are conducted to demonstrate the more excellent performance of CTFGES, in terms of accuracy, efficiency and robustness, compared with the representative feature selection algorithms. In addition, the proposed CTFGES algorithm has been successfully applied into the feature segmentation of large-scale neonatal cerebral cortex MRI with varying noise ratios and intensity non-uniformity levels. The results indicate that it can be adaptive to derive from the cortical folding surfaces and achieves the satisfying consistency with medical experts, which will be potential significance for successfully assessing the impact of aberrant brain growth on the neurodevelopment of neonatal cerebrum.

Unsupervised Representative Feature Selection Algorithm Based on Information Entropy and Relevance Analysis

Feature selection plays an important role in preprocessing in pattern recognition and data mining, especially in large scale image, digital text, and biological data. Specifically, class label information is unavailable for conducting the selection of a minimal feature subset in unsupervised learning, which is full of challenging and interesting problems. In this paper, we present an unsupervised REPresentative Feature Selection (REPFS) algorithm based on information entropy and relevance analysis. The proposed method seeks to find a high-quality feature subset through feature clustering without using any learning algorithms. More importantly, the features’ relevance will be computed based on an information metric of the relevance gain, which provides an information theoretical foundation for finding a minimum of the redundancy between features. Our results on nine benchmark data sets demonstrate that REPFS can significantly improve upon state-of-the-art unsupervised algorithms.

Rapid and Efficient Bug Assignment Using ELM for IOT Software

The reliable implementation of software in an Internet system directly influences information transmission especially for the Internet of Things (IoT) system. Once defects in the system are found, the communication between things and things and the interaction between people and things in the IoT will be greatly affected. Therefore, rapid and effective defect assignment to the right developer is the key to ensuring software quality and bringing down time consumption in the IoT software life cycle. However, as the size of the software becomes increasing larger, the requirement of users grows, and a large number of software bugs will be found every day. It is difficult for managers to assign the software defects to the appropriate developers. In this paper, a novel hybrid method based on a diversified feature selection and an ensemble extreme learning machine (ELM) is proposed. First, the useful information is extracted from defect reports; then, the data are preprocessed to establish a vector space model; and the diversified feature selection is preprocessed in order to select a smallest set of representative non-redundant features with maximal statistical information. Finally, an ensemble GA-based ELM training classifier is used. Experimental results show that, compared to SVM, C4.5, NaiveBayes, and KNN classifiers, the proposed ELM-based bug triage approach with representative feature selection techniques in this paper significantly improves the efficiency and the effectiveness of bug triages.

Feature Selection

Feature selection, as a data preprocessing strategy, has been proven to be effective and efficient in preparing data (especially high-dimensional data) for various data-mining and machine-learning problems. The objectives of feature selection include building simpler and more comprehensible models, improving data-mining performance, and preparing clean, understandable data. The recent proliferation of big data has presented some substantial challenges and opportunities to feature selection. In this survey, we provide a comprehensive and structured overview of recent advances in feature selection research. Motivated by current challenges and opportunities in the era of big data, we revisit feature selection research from a data perspective and review representative feature selection algorithms for conventional data, structured data, heterogeneous data and streaming data. Methodologically, to emphasize the differences and similarities of most existing feature selection algorithms for conventional data, we categorize them into four main groups: similarity-based, information-theoretical-based, sparse-learning-based, and statistical-based methods. To facilitate and promote the research in this community, we also present an open source feature selection repository that consists of most of the popular feature selection algorithms (http://featureselection.asu.edu/). Also, we use it as an example to show how to evaluate feature selection algorithms. At the end of the survey, we present a discussion about some open problems and challenges that require more attention in future research.

Novel artificial bee colony based feature selection method for filtering redundant information

Feature selection, which can reduce the dimensions of feature space without sacrificing the performance of the classifier, is an effective technique for text classification. Because many classifiers cannot deal with the features with high dimensions, filtering the redundant information from the original feature space becomes one of the core goals in feature selection field. In this paper, the concept of equivalence word set is introduced and a set of equivalence word sets (represented as EWS 1) is constructed using the rich semantic information of the Open Directory Project (ODP). On this basis, an artificial bee colony based feature selection method is proposed for filtering the redundant information, and a feature subset FS is obtained by using an optimal feature selection (OFS) method and two predetermined thresholds. In order to obtain the best predetermined thresholds, an improved memory based artificial bee colony method (IABCM) is proposed. In the experiments, fuzzy support vector machine (FSVM) and Naive Bayesian (NB) classifiers are used on six datasets: LingSpam, WebKB, SpamAssian, 20-Newsgroups, Reuters21578 and TREC2007. Experimental results verify that when FSVM and NB are applied, the proposed method is efficient and achieves better accuracy than several representative feature selection methods.

Relevance popularity: A term event model based feature selection scheme for text classification

Feature selection is a practical approach for improving the performance of text classification methods by optimizing the feature subsets input to classifiers. In traditional feature selection methods such as information gain and chi-square, the number of documents that contain a particular term (i.e. the document frequency) is often used. However, the frequency of a given term appearing in each document has not been fully investigated, even though it is a promising feature to produce accurate classifications. In this paper, we propose a new feature selection scheme based on a term event Multinomial naive Bayes probabilistic model. According to the model assumptions, the matching score function, which is based on the prediction probability ratio, can be factorized. Finally, we derive a feature selection measurement for each term after replacing inner parameters by their estimators. On a benchmark English text datasets (20 Newsgroups) and a Chinese text dataset (MPH-20), our numerical experiment results obtained from using two widely used text classifiers (naive Bayes and support vector machine) demonstrate that our method outperformed the representative feature selection methods.

An Improved Combination of Spectral and Spatial Features for Vegetation Classification in Hyperspectral Images

Due to the advances in hyperspectral sensor technology, hyperspectral images have gained great attention in precision agriculture. In practical applications, vegetation classification is usually required to be conducted first and then the vegetation of interest is discriminated from the others. This study proposes an integrated scheme (SpeSpaVS_ClassPair_ScatterMatrix) for vegetation classification by simultaneously exploiting image spectral and spatial information to improve vegetation classification accuracy. In the scheme, spectral features are selected by the proposed scatter-matrix-based feature selection method (ClassPair_ScatterMatrix). In this method, the scatter-matrix-based class separability measure is calculated for each pair of classes and then averaged as final selection criterion to alleviate the problem of mutual redundancy among the selected features, based on the conventional scatter-matrix-based class separability measure (AllClass_ScatterMatrix). The feature subset search is performed by the sequential floating forward search method. Considering the high spectral similarity among different green vegetation types, Gabor features are extracted from the top two principal components to provide complementary spatial features for spectral features. The spectral features and Gabor features are stacked into a feature vector and then the ClassPair_ScatterMatrix method is used on the formed vector to overcome the over-dimensionality problem and select discriminative features for vegetation classification. The final features are fed into support vector machine classifier for classification. To verify whether the ClassPair_ScatterMatrix method could well avoid selecting mutually redundant features, the mean square correlation coefficients were calculated for the ClassPair_ScatterMatrix method and AllClass_ScatterMatrix method. The experiments were conducted on a widely used agricultural hyperspectral image. The experimental results showed that (1) the The proposed ClassPair_ScatterMatrix method could better alleviate the problem of selecting mutually redundant features, compared to the AllClass_ScatterMatrix method; (2) compared with the representative mutual information-based feature selection methods, the scatter-matrix-based feature selection methods generally achieved higher classification accuracies, and the ClassPair_ScatterMatrix method especially, produced the highest classification accuracies with respect to both data sets (87.2% and 90.1%); and (3) the proposed integrated scheme produced higher classification accuracy, compared with the decision fusion of spectral and spatial features and the methods only involving spectral features or spatial features. The comparative experiments demonstrate the effectiveness of the proposed scheme.

Joint Discriminative and Representative Feature Selection for Alzheimer's Disease Diagnosis.

Neuroimaging data have been widely used to derive possible biomarkers for Alzheimer's Disease (AD) diagnosis. As only certain brain regions are related to AD progression, many feature selection methods have been proposed to identify informative features (i.e., brain regions) to build an accurate prediction model. These methods mostly only focus on the feature-target relationship to select features which are discriminative to the targets (e.g., diagnosis labels). However, since the brain regions are anatomically and functionally connected, there could be useful intrinsic relationships among features. In this paper, by utilizing both the feature-target and feature-feature relationships, we propose a novel sparse regression model to select informative features which are discriminative to the targets and also representative to the features. We argue that the features which are representative (i.e., can be used to represent many other features) are important, as they signify strong "connection" with other ROIs, and could be related to the disease progression. We use our model to select features for both binary and multi-class classification tasks, and the experimental results on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset show that the proposed method outperforms other comparison methods considered in this work.

Feature selection with redundancy-complementariness dispersion

Feature selection has attracted significant attention in data mining and machine learning in the past decades. Many existing feature selection methods eliminate redundancy by measuring pairwise inter-correlation of features, whereas the complementariness of features and higher inter-correlation among more than two features are ignored. In this study, a modification item concerning feature complementariness is introduced in the evaluation criterion of features. Additionally, in order to identify the interference effect of already-selected False Positives (FPs), the redundancy-complementariness dispersion is also taken into account to adjust the measurement of pairwise inter-correlation of features. To illustrate the effectiveness of proposed method, classification experiments are applied with four frequently used classifiers on ten datasets. Classification results verify the superiority of proposed method compared with seven representative feature selection methods.

A novel feature selection method considering feature interaction

Interacting features are those that appear to be irrelevant or weakly relevant with the class individually, but when it combined with other features, it may highly correlate to the class. Discovering feature interaction is a challenging task in feature selection. In this paper, a novel feature selection algorithm considering feature interaction is proposed. Firstly, feature relevance, feature redundancy and feature interaction have been redefined in the framework of information theory. Then the interaction weight factor which can reflect the information of whether a feature is redundant or interactive is proposed. Afterwards, we bring forward an Interaction Weight based Feature Selection algorithm (IWFS). To evaluate the performance of the proposed algorithm, we compare IWFS with other five representative feature selection algorithms, including CFS, INTERACT, FCBF, MRMR and Relief-F, in terms of the classification accuracies and the number of selected features with three different types of classifiers including C4.5, IB1 and PART. The results on the six synthetic datasets show that IWFS can effectively identify irrelevant and redundant features while reserving interactive ones. The results on the eight real world datasets indicate that IWFS not only efficiently reduces the dimensionality of feature space, but also offers the highest average accuracy for all the three classification algorithms.

Efficient sequential feature selection based on adaptive eigenspace model

Though Fisher score is a representative and effective feature selection method, it has an unsolved drawback: it either evaluates the features individually and selects the top features, or selects features using the sequential search strategies. The individual-method ignores the mutual relationship among the selected features while the sequential-methods always suffer from heavy computation. In this work, we present an efficient sequential feature selection method. In the proposed method, the generalized Fisher score is used as a robust measurement of the discriminative ability of the features, which can naturally deal with the Small Size Sample problem. Besides, each feature is considered as a pattern vector and an adaptive eigenspace model is applied to update the generalized Fisher score. In the proposed adaptive eigenspace model, the size of the eigen-decomposition problems does not increase with the number of selected features, but is determined by the dimension of the adaptive eignespace. If the dimension of the adaptive eigenspace model is fixed, the proposed algorithm approximately consumes constant time to evaluate a candidate feature. Therefore, the proposed method is computationally more efficient than the traditional sequential methods. Experiments on six widely used face databases are conducted to demonstrate the efficacy of the proposed approach.

Fuzzy-rough feature selection accelerator

Fuzzy rough set method provides an effective approach to data mining and knowledge discovery from hybrid data including categorical values and numerical values. However, its time-consumption is very intolerable to analyze data sets with large scale and high dimensionality. Many heuristic fuzzy-rough feature selection algorithms have been developed however, quite often, these methods are still computationally time-consuming. For further improvement, we propose an accelerator, called forward approximation, which combines sample reduction and dimensionality reduction together. The strategy can be used to accelerate a heuristic process of fuzzy-rough feature selection. Based on the proposed accelerator, an improved algorithm is designed. Through the use of the accelerator, three representative heuristic fuzzy-rough feature selection algorithms have been enhanced. Experiments show that these modified algorithms are much faster than their original counterparts. It is worth noting that the performance of the modified algorithms becomes more visible when dealing with larger data sets.

Feature selection using data envelopment analysis

Feature selection has been attracting increasing attention in recent years for its advantages in improving the predictive efficiency and reducing the cost of feature acquisition. In this paper, we regard feature selection as an efficiency evaluation process with multiple evaluation indices, and propose a novel feature selection framework based on Data Envelopment Analysis (DEA). The most significant advantages of this framework are that it can make a trade-off among several feature properties or evaluation criteria and evaluate features from a perspective of “efficient frontier” without parameter setting. We then propose a simple feature selection method based on the framework to effectively search “efficient” features with high class-relevance and low conditional independence. Super-efficiency DEA is employed in our method to fully rank features according to their efficiency scores. Experimental results for twelve well-known datasets indicate that proposed method is effective and outperforms several representative feature selection methods in most cases. The results also show the feasibility of proposed DEA-based feature selection framework.

A novel feature subset selection algorithm based on association rule mining

In this paper, a novel feature selection algorithm FEAST is proposed based on association rule mining. The proposed algorithm first mines association rules from a data set; then, it identifies the relevant and interactive feature values with the constraint association rules whose consequent is the target concept, detects and eliminates the redundant feature values with the constraint association rules whose consequent and antecedent are both of single feature value. Finally, it obtains the feature subset by mapping the feature values to the corresponding features. As the support and confidence thresholds are two important parameters in association rule mining and play a vital role in FEAST, a partial least square regression PLSR based threshold prediction method is presented as well. The effectiveness of FEAST is tested on both synthetic and real world data sets, and the classification results of five different types of classifiers with seven representative feature selection algorithms are compared. The results on the synthetic data sets show that FEAST can effectively identify irrelevant and redundant features while reserving interactive ones. The results on the real world data sets show that FEAST outperforms other feature selection algorithms in terms of classification accuracies. In addition, the PLSR based threshold prediction method is performed on the real world data sets, and the results show it works well in recommending proper support and confidence thresholds for FEAST.

Feature Selection for Time Series Modeling

In machine learning, selecting useful features and rejecting redundant features is the prerequisite for better modeling and prediction. In this paper, we first study representative feature selection methods based on correlation analysis, and demonstrate that they do not work well for time series though they can work well for static systems. Then, theoretical analysis for linear time series is carried out to show why they fail. Based on these observations, we propose a new correlation-based feature selection method. Our main idea is that the features highly correlated with progressive response while lowly correlated with other features should be selected, and for groups of selected features with similar residuals, the one with a smaller number of features should be selected. For linear and nonlinear time series, the proposed method yields high accuracy in both feature selection and feature rejection.

A Fast Clustering-Based Feature Subset Selection Algorithm for High-Dimensional Data

Feature selection involves identifying a subset of the most useful features that produces compatible results as the original entire set of features. A feature selection algorithm may be evaluated from both the efficiency and effectiveness points of view. While the efficiency concerns the time required to find a subset of features, the effectiveness is related to the quality of the subset of features. Based on these criteria, a fast clustering-based feature selection algorithm (FAST) is proposed and experimentally evaluated in this paper. The FAST algorithm works in two steps. In the first step, features are divided into clusters by using graph-theoretic clustering methods. In the second step, the most representative feature that is strongly related to target classes is selected from each cluster to form a subset of features. Features in different clusters are relatively independent, the clustering-based strategy of FAST has a high probability of producing a subset of useful and independent features. To ensure the efficiency of FAST, we adopt the efficient minimum-spanning tree (MST) clustering method. The efficiency and effectiveness of the FAST algorithm are evaluated through an empirical study. Extensive experiments are carried out to compare FAST and several representative feature selection algorithms, namely, FCBF, ReliefF, CFS, Consist, and FOCUS-SF, with respect to four types of well-known classifiers, namely, the probability-based Naive Bayes, the tree-based C4.5, the instance-based IB1, and the rule-based RIPPER before and after feature selection. The results, on 35 publicly available real-world high-dimensional image, microarray, and text data, demonstrate that the FAST not only produces smaller subsets of features but also improves the performances of the four types of classifiers.

Divergence-based feature selection for separate classes

Feature selection is one of the core issues in designing pattern recognition systems and has attracted considerable attention in the literature. Most of the feature selection methods in the literature only handle relevance and redundancy analysis from the point of view of the whole class, which neglects the relation of features and the separate classes. In this paper, we propose a novel feature selection framework to explicitly handle the relevance and redundancy analysis for each class label. Then we propose two simple and effective feature selection algorithms based on this framework and Kullback–Leibler divergence. An empirical study is conducted to evaluate the efficiency and effectiveness of our algorithms comparing with five representative feature selection algorithms. Empirical results show that our proposed algorithms are efficient and outperform the selected algorithms in most cases, and show the superiority of our proposed feature selection framework.