Wrapper-based Feature Subset Selection Research Articles

Scalable feature subset selection for big data using parallel hybrid evolutionary algorithm based wrapper under apache spark environment.

Extant sequential wrapper-based feature subset selection (FSS) algorithms are not scalable and yield poor performance when applied to big datasets. Hence, to circumvent these challenges, we propose parallel and distributed hybrid evolutionary algorithms (EAs) based wrappers under Apache Spark. We propose two hybrid EAs based on the Binary Differential Evolution (BDE), and Binary Threshold Accepting (BTA), namely, (i) Parallel Binary Differential Evolution and Threshold Accepting (PB-DETA), where BDE and BTA work in tandem in every iteration, and (ii) its ablation variant, Parallel Binary Threshold Accepting and Differential Evolution (PB-TADE). Here, BTA is invoked to enhance the search capability and avoid premature convergence of BDE. For comparison purposes, we also parallelized two state-of-the-art algorithms: adaptive DE (ADE) and permutation based DE (DE-FSPM), and named them PB-ADE and P-DE-FSPM respectively. Throughout, logistic regression (LR) is employed to compute the fitness function, namely, area under the receiver operator characteristic curve (AUC). The effectiveness of the proposed algorithms is tested over the five big datasets of varying dimensions. It is noteworthy that the PB-TADE turned out to be statistically significant than the rest. All the algorithms have shown the repeatability property. The proposed parallel model attained a speedup of 2.2–2.9. We also reported feature subset with high AUC and least cardinality.

A novel wrapper-based feature subset selection method using modified binary differential evolution algorithm

In classification problems, normally there exists a large number of features, but not all of them contributing to the improvement of classification performance. These redundant features make the classification problem time consuming and often result in poor performance. Feature selection methods have been proposed to reduce the number of features, minimize computational cost, and maximize classification accuracy. As a wrapper-based approach, evolutionary algorithms have been widely applied in feature subset selection tasks. However, some of them trap into local minima, especially when the number of features increases, while others are not efficient in computational time. This paper proposes a Modified Differential Evolution (DE) approach to Feature Selection (MDEFS) by utilizing two new mutation strategies to create a feasible balance between exploration and exploitation and maintain the classification performance in an acceptable range concerning both the number of features and accuracy. Some modifications are made to the standard DE crossover and its key control parameters to enhance the proposed algorithm’s capabilities further. The proposed method has been compared to several state-of-the-art methods utilizing standard datasets from the UCI repository and results of the experiments demonstrate the superiority of the proposed approach.

Analysis of various transfer functions for binary owl search algorithm in feature selection problem

ABSTRACT Owl Search Algorithm (OSA) is a recently proposed nature-inspired meta-heuristic algorithm which is easily implementable and exhibits great potential for solving continuous optimization problems. In our earlier work, a binary version of owl search algorithm (BOSA), with transfer functions for mapping the continuous solution space into a binary one, has been developed and applied in optimal feature subset selection problem. In our preliminary simulation experiments, it was found that the performance of the solution depends on the type of transfer function used. In this work, an extensive analysis of various types of transfer functions and their respective effects on the selection of optimal feature subset has been studied by simulation experiments with multiple benchmark datasets. Transfer functions of three different families, S-shaped, V-shaped and quadratic, are used for designing eleven BOSA models, each of which is made by combining individual transfer function. The performances of the proposed wrapper based feature subset selection algorithm based on several BOSA models have been evaluated by simulation experiments with twenty datasets for finding out the best model. The best observed BOSA model has also been compared with other similar meta-heuristics algorithms for feature subset selection. Experimental results show that the feature subset selected by BOSA with quadratic transfer function produces the highest classification accuracy with the minimum number of selected features compared to other algorithms.

Binary coordinate ascent: An efficient optimization technique for feature subset selection for machine learning

Feature subset selection (FSS) has been an active area of research in machine learning. A number of techniques have been developed for selecting an optimal or sub-optimal subset of features, because it is a major factor to determine the performance of a machine-learning technique. In this paper, we propose and develop a novel optimization technique, namely, a binary coordinate ascent (BCA) algorithm that is an iterative deterministic local optimization that can be coupled with wrapper or filter FSS. The algorithm searches throughout the space of binary coded input variables by iteratively optimizing the objective function in each dimension at a time. We investigated our BCA approach in wrapper-based FSS under area under the receiver-operating-characteristic (ROC) curve (AUC) criterion for the best subset of features in classification. We evaluated our BCA-based FSS in optimization of features for support vector machine, multilayer perceptron, and Naïve Bayes classifiers with 12 datasets. Our experimental datasets are distinct in terms of the number of attributes (ranging from 18 to 11,340), and the number of classes (binary or multi-class classification). The efficiency in terms of the number of subset evaluations was improved substantially (by factors of 5–37) compared with two popular FSS meta-heuristics, i.e., sequential forward selection (SFS) and sequential floating forward selection (SFFS), while the classification performance for unseen data was maintained.

On the Stability of Feature Selection Methods in Software Quality Prediction: An Empirical Investigation

Software quality modeling is the process of using software metrics from previous iterations of development to locate potentially faulty modules in current under-development code. This has become an important part of the software development process, allowing practitioners to focus development efforts where they are most needed. One difficulty encountered in software quality modeling is the problem of high dimensionality, where the number of available software metrics is too large for a classifier to work well. In this case, many of the metrics may be redundant or irrelevant to defect prediction results, thereby selecting a subset of software metrics that are the best predictors becomes important. This process is called feature (metric) selection. There are three major forms of feature selection: filter-based feature rankers, which uses statistical measures to assign a score to each feature and present the user with a ranked list; filter-based feature subset evaluation, which uses statistical measures on feature subsets to find the best feature subset; and wrapper-based subset selection, which builds classification models using different subsets to find the one which maximizes performance. Software practitioners are interested in which feature selection methods are best at providing the most stable feature subset in the face of changes to the data (here, the addition or removal of instances). In this study we select feature subsets using fifteen feature selection methods and then use our newly proposed Average Pairwise Tanimoto Index (APTI) to evaluate the stability of the feature selection methods. We evaluate the stability of feature selection methods on a pair of subsamples generated by our fixed-overlap partitions algorithm. Four different levels of overlap are considered in this study. 13 software metric datasets from two real-world software projects are used in this study. Results demonstrate that ReliefF (RF) is the most stable feature selection method and wrapper based feature subset selection shows least stability. In addition, as the overlap of partitions increased, the stability of the feature selection strategies increased.

Accelerating wrapper-based feature selection with K-nearest-neighbor

Wrapper-based feature subset selection (FSS) methods tend to obtain better classification accuracy than filter methods but are considerably more time-consuming, particularly for applications that have thousands of features, such as microarray data analysis. Accelerating this process without degrading its high accuracy would be of great value for gene expression analysis. In this study, we explored how to reduce the time complexity of wrapper-based FSS with an embedded K-Nearest-Neighbor (KNN) classifier. Instead of considering KNN as a black box, we proposed to construct a classifier distance matrix and incrementally update the matrix to accelerate the calculation of the relevance criteria in evaluating the quality of the candidate features. Extensive experiments on eight publicly available microarray datasets were first conducted to demonstrate the effectiveness of the wrapper methods with KNN for selecting informative features. To demonstrate the performance gain in terms of time cost reduction, we then conducted experiments on the eight microarray datasets with the embedded KNN classifiers and analyzed the theoretical time/space complexity. Both the experimental results and theoretical analysis demonstrated that the proposed approach markedly accelerates the wrapper-based feature selection process without degrading the high classification accuracy, and the space complexity analysis indicated that the additional space overhead is affordable in practice.

A Novel Method for Classifying Body Mass Index on the Basis of Speech Signals for Future Clinical Applications: A Pilot Study

Obesity is a serious public health problem because of the risk factors for diseases and psychological problems. The focus of this study is to diagnose the patient BMI (body mass index) status without weight and height measurements for the use in future clinical applications. In this paper, we first propose a method for classifying the normal and the overweight using only speech signals. Also, we perform a statistical analysis of the features from speech signals. Based on 1830 subjects, the accuracy and AUC (area under the ROC curve) of age- and gender-specific classifications ranged from 60.4 to 73.8% and from 0.628 to 0.738, respectively. We identified several features that were significantly different between normal and overweight subjects (P < 0.05). Also, we found compact and discriminatory feature subsets for building models for diagnosing normal or overweight individuals through wrapper-based feature subset selection. Our results showed that predicting BMI status is possible using a combination of speech features, even though significant features are rare and weak in age- and gender-specific groups and that the classification accuracy with feature selection was higher than that without feature selection. Our method has the potential to be used in future clinical applications such as automatic BMI diagnosis in telemedicine or remote healthcare.

Identifying malignant transformations in recurrent low grade gliomas using high resolution magic angle spinning spectroscopy

The objective of this study was to determine whether metabolic parameters derived from ex vivo analysis of tissue samples are predictive of biologic characteristics of recurrent low grade gliomas (LGGs). This was achieved by exploring the use of multivariate pattern recognition methods to generate statistical models of the metabolic characteristics of recurrent LGGs that correlate with aggressive biology and poor clinical outcome. Statistical models were constructed to distinguish between patients with recurrent gliomas that had undergone malignant transformation to a higher grade and those that remained grade 2. The pattern recognition methods explored in this paper include three filter-based feature selection methods (chi-square, gain ratio, and two-way conditional probability), a genetic search wrapper-based feature subset selection algorithm, and five classification algorithms (linear discriminant analysis, logistic regression, functional trees, support vector machines, and decision stump logit boost). The accuracy of each pattern recognition framework was evaluated using leave-one-out cross-validation and bootstrapping. The population studied included fifty-three patients with recurrent grade 2 gliomas. Among these patients, seven had tumors that transformed to grade 4, twenty-four had tumors that transformed to grade 3, and twenty-two had tumors that remained grade 2. Image-guided tissue samples were obtained from these patients using surgical navigation software. Part of each tissue sample was examined by a pathologist for histological features and for consistency with the tumor grade diagnosis. The other part of the tissue sample was analyzed with ex vivo nuclear magnetic resonance (NMR) spectroscopy. Distinguishing between recurrent low grade gliomas that transformed to a higher grade and those that remained grade 2 was achieved with 96% accuracy, using areas of the ex vivo NMR spectrum corresponding to myoinositol, 2-hydroxyglutarate, hypo-taurine, choline, glycerophosphocholine, phosphocholine, glutathione, and lipid. Logistic regression and decision stump boosting models were able to distinguish between recurrent gliomas that transformed to a higher grade and those that did not with 100% training accuracy (95% confidence interval [93-100%]), 96% leave-one-out cross-validation accuracy (95% confidence interval [87-100%]), and 96% bootstrapping accuracy (95% confidence interval [95-97%]). Linear discriminant analysis, functional trees, and support vector machines were able to achieve leave-one-out cross-validation accuracy above 90% and bootstrapping accuracy above 85%. The three feature ranking methods were comparable in performance. This study demonstrates the feasibility of using quantitative pattern recognition methods for the analysis of metabolic data from brain tissue obtained during the surgical resection of gliomas. All pattern recognition techniques provided good diagnostic accuracies, though logistic regression and decision stump boosting slightly outperform the other classifiers. These methods identified biomarkers that can be used to detect malignant transformations in individual low grade gliomas, and can lead to a timely change in treatment for each patient.

Fast wrapper feature subset selection in high-dimensional datasets by means of filter re-ranking

This paper deals with the problem of supervised wrapper-based feature subset selection in datasets with a very large number of attributes. Recently the literature has contained numerous references to the use of hybrid selection algorithms: based on a filter ranking, they perform an incremental wrapper selection over that ranking. Though working fine, these methods still have their problems: (1) depending on the complexity of the wrapper search method, the number of wrapper evaluations can still be too large; and (2) they rely on a univariate ranking that does not take into account interaction between the variables already included in the selected subset and the remaining ones. Here we propose a new approach whose main goal is to drastically reduce the number of wrapper evaluations while maintaining good performance (e.g. accuracy and size of the obtained subset). To do this we propose an algorithm that iteratively alternates between filter ranking construction and wrapper feature subset selection (FSS). Thus, the FSS only uses the first block of ranked attributes and the ranking method uses the current selected subset in order to build a new ranking where this knowledge is considered. The algorithm terminates when no new attribute is selected in the last call to the FSS algorithm. The main advantage of this approach is that only a few blocks of variables are analyzed, and so the number of wrapper evaluations decreases drastically. The proposed method is tested over eleven high-dimensional datasets (2400–46,000 variables) using different classifiers. The results show an impressive reduction in the number of wrapper evaluations without degrading the quality of the obtained subset.

Wrapper-Based Feature Subset Selection for Rapid Image Information Mining

In a disaster, there is a need for rapid image-information retrieval in real or near real time from vast amounts of data coming from multiple remote-sensing sensors. In general, image information mining (IIM) approaches produce enormous amounts of features that are computationally expensive and inefficient to process before the actual information discovery takes place. Also, it is complicated because the combination of the features has little relevance to the hypothesis space. Hence, selecting a relevant subset of features is necessary to overcome these problems and to provide an efficient representation of the target class. In this letter, we propose feature selection and feature transformations based on a wrapper-based genetic algorithm approach. A support vector machine classification is applied for generating predictive models for those land-cover classes that are important in a coastal disaster event. The proposed system, rapid IIM, is a region-based approach where, in lieu of the prevalent pixel-based methods, it localizes interesting zones and enables rapid querying. Results from this study indicate that selecting relevant feature subsets increases the rate of correctly identifying a semantic class and also enables this process with less number of features.

Unsupervised feature dimension reduction for classification of MR spectra.

We present an unsupervised feature dimension reduction method for the classification of magnetic resonance spectra. The technique preserves spectral information, important for disease profiling. We propose to use this technique as a preprocessing step for computationally demanding wrapper-based feature subset selection. We show that the classification accuracy on an independent test set can be sustained while achieving considerable feature reduction. Our method is applicable to other classification techniques, such as neural networks, support vector machines, etc.