Feature Subset Size Research Articles

Differential evolution for feature selection: a fuzzy wrapper–filter approach

The selection of an optimal feature subset from all available features in the data is a vital task of data pre-processing used for several purposes such as the dimensionality reduction, the computational complexity reduction required for data processing (e.g., clustering, classification and regression) and the performance enhancement of a data processing technique. To serve such purposes, feature selection approaches which are fundamentally categorized into filters and wrappers try to eliminate irrelevant, redundant and erroneous features in the data. Each category comes with its own advantages and disadvantages. While wrappers can generally provide higher classification performance than filters, filters are computationally more efficient than wrappers. In order to bring the advantages of wrappers and filters together, i.e., to get higher classification performance with smaller feature subset size in a shorter time, this paper proposes a differential evolution approach combining filter and wrapper approaches through an improved information theoretic local search mechanism which is based on the concepts of fuzziness to cope with both continuous and discrete datasets. To show the superiority of the proposed approach, it is examined and compared with traditional and recent evolutionary feature selection approaches on several benchmarks from different well-known data repositories.

Selection of Optimal Object Features in Object-Based Image Analysis Using Filter-Based Algorithms

With the increase in spatial resolution of recent sensors, object-based image analysis (OBIA) has gained importance for producing detailed land use maps. One of the main advantages of OBIA is that a variety of spectral, spatial and textural features can be extracted for the segmented image objects that are later utilized in classification. However, using a large number of features not only increases the required computational time, but also requires a large number of ground samples, which is unavailable in most cases. For these reasons, feature selection (FS) has become an important research topic for OBIA based classification studies. In this study, three filter-based FS algorithms namely, Chi square, information gain and ReliefF were applied to determine the most effective object features that ensure high separability among landscape features. For this purpose, importance degree (i.e. ranks) of 110 input object features were firstly estimated by the algorithms, and correlation-based merit function was then applied to determine optimum feature subset size. Multi-resolution segmentation algorithm was applied for segmenting a WorldView-2 image. Support vector machine, random forest and nearest neighbour classifiers were all utilized to classify segmented image objects using the selected object features. Results revealed that the FS algorithms were effective for selecting the most relevant features. Also, the classifiers produced the highest performances with 24 out of 110 features selected by the information gain (IG) algorithm. Particularly, the support vector machine classifier produced the highest overall accuracy (92.00%) with 24 selected features determined by the IG algorithm. A significant improvement of about 4% was achieved by applying FS procedures that was found statistically significant in terms of Wilcoxon signed-ranks test.

Random subset feature selection for ecological niche models of wildfire activity in Western North America

Variable selection in ecological niche modelling can influence model projections to a degree comparable to variations in future climate scenarios. Consequently, it is important to select feature (variable) subsets for optimizing model performance and characterizing variability. We utilize a novel random subset feature selection algorithm (RSFSA) for niche modelling to select an ensemble of optimally sized feature subsets of limited correlation (|r| < 0.7) from 90 climatic, topographic and anthropogenic indices, generating wildfire activity models for western North America with higher performance. Monitoring Trends in Burn Severity and LANDFIRE wildfire data were used to develop thousands of MaxEnt, GLM and Glmnet models. The RSFSA-selected models performed better than random models, having higher accuracy (Area Under the Curve statistic; AUC), lower complexity (corrected Akaike Information Criterion; AICc), and, in some cases, lower overfitting (AUCdiff). The RSFSA-selected MaxEnt quadratic/hinge (β-regularization 2) feature models generally had higher AUC and lower AICc, outperforming other niche model parameterizations and methods. Feature subset ensembles of RSFSA-selected 15-variable MaxEnt quadratic/hinge models were used to characterize variability in projected areas of large wildfires for three burn severities under current, 2050, and 2070 climate scenarios. Expert screening of variables before RSFSA did not improve model performance. Widespread contemporary wildfire deficits and projected regional changes in wildfires highlight the need to manage fuel loads and restore natural fire regimes. The RSFSA is valuable for optimizing niche model performance and generating feature subset ensembles to characterize model variability across niche models of various feature subset sizes, modelling methods, and climate scenarios.

Ensemble support vector machine classification of dementia using structural MRI and mini-mental state examination

BackgroundThe International Challenge for Automated Prediction of MCI from MRI data offered independent, standardized comparison of machine learning algorithms for multi-class classification of normal control (NC), mild cognitive impairment (MCI), converting MCI (cMCI), and Alzheimer's disease (AD) using brain imaging and general cognition. New methodWe proposed to use an ensemble of support vector machines (SVMs) that combined bagging without replacement and feature selection. SVM is the most commonly used algorithm in multivariate classification of dementia, and it was therefore valuable to evaluate the potential benefit of ensembling this type of classifier. ResultsThe ensemble SVM, using either a linear or a radial basis function (RBF) kernel, achieved multi-class classification accuracies of 55.6% and 55.0% in the challenge test set (60 NC, 60 MCI, 60 cMCI, 60 AD), resulting in a third place in the challenge. Similar feature subset sizes were obtained for both kernels, and the most frequently selected MRI features were the volumes of the two hippocampal subregions left presubiculum and right subiculum. Post-challenge analysis revealed that enforcing a minimum number of selected features and increasing the number of ensemble classifiers improved classification accuracy up to 59.1%. Comparison with existing method(s)The ensemble SVM outperformed single SVM classifications consistently in the challenge test set. ConclusionsEnsemble methods using bagging and feature selection can improve the performance of the commonly applied SVM classifier in dementia classification. This resulted in competitive classification accuracies in the International Challenge for Automated Prediction of MCI from MRI data.

An integrated machine learning framework for hospital readmission prediction

Unplanned readmission (re-hospitalization) is the main source of cost for healthcare systems and is normally considered as an indicator of healthcare quality and hospital performance. Poor understanding of the relative importance of predictors and limited capacity of traditional statistical models challenge the development of accurate predictive models for readmission. This study aims to develop a robust and accurate risk prediction framework for hospital readmission, by combining feature selection algorithms and machine learning models. With regard to feature selection, an enhanced version of multi-objective bare-bones particle swarm optimization (EMOBPSO) is developed as the principal search strategy, and a new mutual information-based criterion is proposed to efficiently estimate feature relevancy and redundancy. A greedy local search strategy (GLS) is developed and merged into EMOBPSO to control the final feature subset size as desired. For the modeling process, manifold machine learning models, such as support vector machine, random forest, and deep neural network, are trained with preprocessed datasets and corresponding feature subsets. In the case study, the proposed methodology is applied to an actual hospital located in Northeast China, with various levels of data collected from the hospital information system. Results obtained from comparative experiments demonstrate the effectiveness of EMOBPSO and EMOBPSO-GLS feature selection algorithms. The combination of EMOBPSO (EMOBPSO-GLS) and deep neural network possesses robust predictive power among different datasets. Furthermore, insightful implications are abstracted from the obtained elite features and can be used by practitioners to determine the vulnerable patients for readmission and target the delivery of early resource-intensive interventions.

A novel hybrid genetic algorithm with granular information for feature selection and optimization

Feature selection has been a significant task for data mining and pattern recognition. It aims to choose the optimal feature subset with the minimum redundancy and the maximum discriminating ability. This paper analyzes the feature selection method from two aspects of data and algorithm. In order to deal with the redundant features and irrelevant features in high-dimensional & low-sample data and low-dimensional & high-sample data, the feature selection algorithm model based on the granular information is presented in this paper. Thus, our research examines experimentally how granularity level affects both the classification accuracy and the size of feature subset for feature selection. First of all, the improved binary genetic algorithm with feature granulation (IBGAFG) is used to select the significant features. Then, the improved neighborhood rough set with sample granulation (INRSG) is proposed under different granular radius, which further improves the quality of the feature subset. Finally, in order to find out the optimal granular radius, granularity λ optimization based on genetic algorithm (ROGA) is presented. The optimal granularity parameters are found adaptively according to the feedback of classification accuracy. The performance of the proposed algorithms is tested upon eleven publicly available data sets and is compared with other supervisory methods or evolutionary algorithms. Additionally, the ROGA algorithm is applied to the enterprise financial dataset, which can select the features that affect the financial status. Experiment results demonstrate that the approaches are efficient and can provide higher classification accuracy using granular information.

Fuzzy Mutual Information Feature Selection Based on Representative Samples

Building classification models from real-world datasets became a difficult task, especially in datasets with high dimensional features. Unfortunately, these datasets may include irrelevant or redundant features which have a negative effect on the classification performance. Selecting the significant features and eliminating undesirable features can improve the classification models. Fuzzy mutual information is widely used feature selection to find the best feature subset before classification process. However, it requires more computation and storage space. To overcome these limitations, this paper proposes an improved fuzzy mutual information feature selection based on representative samples. Based on benchmark datasets, the experiments show that the proposed method achieved better results in the terms of classification accuracy, selected feature subset size, storage, and stability.

Hybrid optimization for feature selection in opinion mining

A sub-discipline of Information Retrieval (IR) is opinion mining and the lexicon of computers is not concerned of the subject of thedocument, but about the opinion expressed.It has caused a large impact in the arena of academics and industry as it has a wide area of research and the applications are widespread.Feature selection is a vital step in opinion mining, as its individual feature decides the opinions expressed by the customers.Feature selection reduces the dimensionality of data by avoiding non-relevant features; it can be con-sidered as a necessary and excellent process for data mining applications.In this study, feature subset is optimized through Particle Swarm Optimization (PSO) algorithm, Cuckoo Search (CS) algorithm and hybridized PSO-CS algorithm.Classification is done through Naïve bayes and K-Nearest Neighbours (KNN) classifiers.Feature extraction has its basis on Term Frequency-Inverse DocumentFrequency (TF-IDF).The accuracy of classification precision is increased by the reduction in size of feature subset and computational com-plexity.

Five discrete symbiotic organisms search algorithms for simultaneous optimization of feature subset and neighborhood size of KNN classification models

This paper develops five new discrete Symbiotic Organisms Search (SOS) algorithms for simultaneous optimization of feature subset and neighborhood size of k-nearest neighbor model to improve classification accuracy. The first algorithm is a discrete version of the original SOS algorithm, named DSOS. The second is a hybrid derived from enhancing the first DSOS with paired swap local search, named DHSOS. The third is a cooperative hybrid between DSOS and a discrete particle swarm optimization (DPSO), named DSOSPSO. The fourth and fifth are modified from the second by adapting population size rather than fixing it, named APDHSOS and AP2DHSOS, respectively. Five existing metaheuristic algorithms are also implemented and extended for comparison. The performance of these algorithms employing the k-nearest neighbor classification model are evaluated in terms of classification error and computational time based on stratified k-fold cross validation with 11 datasets. The classification errors of five larger data sets are also obtained to further verify the test results. Based on the test results, it is found that: (1) feature selection with fixed neighborhood size yields lower errors than optimized neighborhood size without feature selection; (2) simultaneous optimization of feature subset and neighborhood size overall outperforms optimizing either feature subset or neighborhood size alone; and (3) among all the ten algorithms tested for simultaneous optimization, APDHSOS and AP2DHSOS emerge to tie the best in terms of classification error, implying that adaptive population size works better than fixing population size.

Differential evolution for filter feature selection based on information theory and feature ranking

Feature selection is an essential step in various tasks, where filter feature selection algorithms are increasingly attractive due to their simplicity and fast speed. A common filter is to use mutual information to estimate the relationships between each feature and the class labels (mutual relevancy), and between each pair of features (mutual redundancy). This strategy has gained popularity resulting a variety of criteria based on mutual information. Other well-known strategies are to order each feature based on the nearest neighbor distance as in ReliefF, and based on the between-class variance and the within-class variance as in Fisher Score. However, each strategy comes with its own advantages and disadvantages. This paper proposes a new filter criterion inspired by the concepts of mutual information, ReliefF and Fisher Score. Instead of using mutual redundancy, the proposed criterion tries to choose the highest ranked features determined by ReliefF and Fisher Score while providing the mutual relevance between features and the class labels. Based on the proposed criterion, two new differential evolution (DE) based filter approaches are developed. While the former uses the proposed criterion as a single objective problem in a weighted manner, the latter considers the proposed criterion in a multi-objective design. Moreover, a well known mutual information feature selection approach (MIFS) based on maximum-relevance and minimum-redundancy is also adopted in single-objective and multi-objective DE algorithms for feature selection. The results show that the proposed criterion outperforms MIFS in both single objective and multi-objective DE frameworks. The results also indicate that considering feature selection as a multi-objective problem can generally provide better performance in terms of the feature subset size and the classification accuracy.

Improving detection of untrustworthy online reviews using ensemble learners combined with feature selection

As fake reviews become more prominent on the web, a method to differentiate between untruthful and truthful reviews becomes increasingly necessary. However, detection of false reviews may be difficult, as determining the validity of a review based solely on text can be nearly impossible for a human. In this study, we aim to determine the effectiveness of machine learning techniques, specifically ensemble techniques and the combination of feature selection and ensemble techniques, for the detection of spam reviews. In addition to traditional ensemble techniques, such as Boosting and Bagging, we employ techniques that combine ensemble methods with a form of feature selection: Select-Boost, Select-Bagging and Random Forest. For Select-Boost and Select-Bagging, we combine the Boosting and Bagging approaches with three different feature rankers. Random Forest was performed using 100, 250, and 500 trees. Our results show a combination of Select-Boost, multinomial naive Bayes and, either Chi-squared or signal-to-noise, significantly outperforms all methods except Random Forest using 500 trees. There is no significant difference between the feature subset sizes tested when using Select-Boost with multinomial naive Bayes, regardless of the feature selection technique employed. To the best of our knowledge, this is the first study to examine the effect of a combination of ensemble techniques and feature selection in the domain of spam review detection.

Computer aided diagnosis of pulmonary hamartoma from CT scan images using ant colony optimization based feature selection

BackgroundComputer-aided diagnosis (CAD) systems for the detection of lung disorders play an important role in clinical decision making. CAD systems provide a second opinion to the physician in interpreting computed tomography (CT) images. In this work, a CAD system to diagnose pulmonary hamartoma nodules from chest CT images is proposed. MethodsSegmentation of lung parenchyma from CT images is carried out using Otsu’s thresholding method. Nodules are considered to be the region of interests (ROIs) in this work. Texture, shape and run length based features are extracted from the ROIs. Cosine similarity measure (CSM) and rough dependency measure (RDM) are used independently as filter evaluation functions with ant colony optimization (ACO) to select two subsets of features. The selected subsets are used to train two classifiers namely support vector machine (SVM) and Naive Bayes (NB) classifiers using 10-fold cross validation. All the four trained classifiers are tested and the performance measures are estimated. ResultsCT slices of patients affected with pulmonary cancer and hamartoma are used for experimentation. From the lung parenchymal tissues of 300 CT slices, 390 nodules are extracted. The feature selection algorithms, ACO-CSM and ACO-RDM are run for different feature subset sizes. The selected features are used to train SVM and NB classifiers. From the results obtained, it is inferred that SVM classifier with the feature subsets chosen by ACO-RDM feature selection approach yielded a maximum classification accuracy of 94.36% with 38 features. ConclusionFrom the results, it can be clearly inferred that selecting relevant features to train the classifier has a definite impact on the performance of the classifier.

Toward better public health reporting using existing off the shelf approaches: The value of medical dictionaries in automated cancer detection using plaintext medical data

ObjectivesExisting approaches to derive decision models from plaintext clinical data frequently depend on medical dictionaries as the sources of potential features. Prior research suggests that decision models developed using non-dictionary based feature sourcing approaches and “off the shelf” tools could predict cancer with performance metrics between 80% and 90%. We sought to compare non-dictionary based models to models built using features derived from medical dictionaries. Materials and methodsWe evaluated the detection of cancer cases from free text pathology reports using decision models built with combinations of dictionary or non-dictionary based feature sourcing approaches, 4 feature subset sizes, and 5 classification algorithms. Each decision model was evaluated using the following performance metrics: sensitivity, specificity, accuracy, positive predictive value, and area under the receiver operating characteristics (ROC) curve. ResultsDecision models parameterized using dictionary and non-dictionary feature sourcing approaches produced performance metrics between 70 and 90%. The source of features and feature subset size had no impact on the performance of a decision model. ConclusionOur study suggests there is little value in leveraging medical dictionaries for extracting features for decision model building. Decision models built using features extracted from the plaintext reports themselves achieve comparable results to those built using medical dictionaries. Overall, this suggests that existing “off the shelf” approaches can be leveraged to perform accurate cancer detection using less complex Named Entity Recognition (NER) based feature extraction, automated feature selection and modeling approaches.

Automatic ICD-10 multi-class classification of cause of death from plaintext autopsy reports through expert-driven feature selection.

ObjectivesWidespread implementation of electronic databases has improved the accessibility of plaintext clinical information for supplementary use. Numerous machine learning techniques, such as supervised machine learning approaches or ontology-based approaches, have been employed to obtain useful information from plaintext clinical data. This study proposes an automatic multi-class classification system to predict accident-related causes of death from plaintext autopsy reports through expert-driven feature selection with supervised automatic text classification decision models.MethodsAccident-related autopsy reports were obtained from one of the largest hospital in Kuala Lumpur. These reports belong to nine different accident-related causes of death. Master feature vector was prepared by extracting features from the collected autopsy reports by using unigram with lexical categorization. This master feature vector was used to detect cause of death [according to internal classification of disease version 10 (ICD-10) classification system] through five automated feature selection schemes, proposed expert-driven approach, five subset sizes of features, and five machine learning classifiers. Model performance was evaluated using precisionM, recallM, F-measureM, accuracy, and area under ROC curve. Four baselines were used to compare the results with the proposed system.ResultsRandom forest and J48 decision models parameterized using expert-driven feature selection yielded the highest evaluation measure approaching (85% to 90%) for most metrics by using a feature subset size of 30. The proposed system also showed approximately 14% to 16% improvement in the overall accuracy compared with the existing techniques and four baselines.ConclusionThe proposed system is feasible and practical to use for automatic classification of ICD-10-related cause of death from autopsy reports. The proposed system assists pathologists to accurately and rapidly determine underlying cause of death based on autopsy findings. Furthermore, the proposed expert-driven feature selection approach and the findings are generally applicable to other kinds of plaintext clinical reports.

Toward better public health reporting using existing off the shelf approaches: A comparison of alternative cancer detection approaches using plaintext medical data and non-dictionary based feature selection

ObjectivesIncreased adoption of electronic health records has resulted in increased availability of free text clinical data for secondary use. A variety of approaches to obtain actionable information from unstructured free text data exist. These approaches are resource intensive, inherently complex and rely on structured clinical data and dictionary-based approaches. We sought to evaluate the potential to obtain actionable information from free text pathology reports using routinely available tools and approaches that do not depend on dictionary-based approaches. Materials and methodsWe obtained pathology reports from a large health information exchange and evaluated the capacity to detect cancer cases from these reports using 3 non-dictionary feature selection approaches, 4 feature subset sizes, and 5 clinical decision models: simple logistic regression, naïve bayes, k-nearest neighbor, random forest, and J48 decision tree. The performance of each decision model was evaluated using sensitivity, specificity, accuracy, positive predictive value, and area under the receiver operating characteristics (ROC) curve. ResultsDecision models parameterized using automated, informed, and manual feature selection approaches yielded similar results. Furthermore, non-dictionary classification approaches identified cancer cases present in free text reports with evaluation measures approaching and exceeding 80–90% for most metrics. ConclusionOur methods are feasible and practical approaches for extracting substantial information value from free text medical data, and the results suggest that these methods can perform on par, if not better, than existing dictionary-based approaches. Given that public health agencies are often under-resourced and lack the technical capacity for more complex methodologies, these results represent potentially significant value to the public health field.

Deep Feature Selection: Theory and Application to Identify Enhancers and Promoters.

Sparse linear models approximate target variable(s) by a sparse linear combination of input variables. Since they are simple, fast, and able to select features, they are widely used in classification and regression. Essentially they are shallow feed-forward neural networks that have three limitations: (1) incompatibility to model nonlinearity of features, (2) inability to learn high-level features, and (3) unnatural extensions to select features in a multiclass case. Deep neural networks are models structured by multiple hidden layers with nonlinear activation functions. Compared with linear models, they have two distinctive strengths: the capability to (1) model complex systems with nonlinear structures and (2) learn high-level representation of features. Deep learning has been applied in many large and complex systems where deep models significantly outperform shallow ones. However, feature selection at the input level, which is very helpful to understand the nature of a complex system, is still not well studied. In genome research, the cis-regulatory elements in noncoding DNA sequences play a key role in the expression of genes. Since the activity of regulatory elements involves highly interactive factors, a deep tool is strongly needed to discover informative features. In order to address the above limitations of shallow and deep models for selecting features of a complex system, we propose a deep feature selection (DFS) model that (1) takes advantages of deep structures to model nonlinearity and (2) conveniently selects a subset of features right at the input level for multiclass data. Simulation experiments convince us that this model is able to correctly identify both linear and nonlinear features. We applied this model to the identification of active enhancers and promoters by integrating multiple sources of genomic information. Results show that our model outperforms elastic net in terms of size of discriminative feature subset and classification accuracy.

Budget constrained non-monotonic feature selection

Feature selection is an important problem in machine learning and data mining. We consider the problem of selecting features under the budget constraint on the feature subset size. Traditional feature selection methods suffer from the “monotonic” property. That is, if a feature is selected when the number of specified features is set, it will always be chosen when the number of specified feature is larger than the previous setting. This sacrifices the effectiveness of the non-monotonic feature selection methods. Hence, in this paper, we develop an algorithm for non-monotonic feature selection that approximates the related combinatorial optimization problem by a Multiple Kernel Learning (MKL) problem. We justify the performance guarantee for the derived solution when compared to the global optimal solution for the related combinatorial optimization problem. Finally, we conduct a series of empirical evaluation on both synthetic and real-world benchmark datasets for the classification and regression tasks to demonstrate the promising performance of the proposed framework compared with the baseline feature selection approaches.

Feature Selection for Multi-Class Imbalanced Data Sets Based on Genetic Algorithm

This paper presents an improved genetic algorithm based feature selection method for multi-class imbalanced data. This method improves the fitness function through using the evaluation criterion EG-mean instead of the global classification accuracy in order to choose the features which are favorable to recognize the minor classes. The method is evaluated using several benchmark data sets, and the experimental results show that, compared with the traditional feature selection method based on genetic algorithm, the proposed method has certain advantages in the size of feature subsets and improves the precision of the minor classes for multi-class imbalanced data sets.

Feature Selection using Multi-objective Genetic Algorith m: A Hybrid Approach

Feature selection is an important pre-processing task for building accurate and comprehensible classification models. Several researchers have applied filter, wrapper or hybrid approaches using genetic algorithms which are good candidates for optimization problems that involve large search spaces like in the case of feature selection. Moreover, feature selection is an inherently multi-objective problem with many competing objectives involving size, predictive power and redundancy of the feature subset under consideration. Hence, Multi-Objective Genetic Algorithms (MOGAs) are a natural choice for this problem. In this paper, we propose a hybrid approach (a wrapper guided by filter approach) for feature selection which employs a MOGA at filter phase and a simple GA at the wrapper phase. The MOGA at filter phase provides a non-dominated set of feature subsets optimized on several criteria as input to the wrapper phase. Now, Genetic Algorithm at wrapper phase does the classifier dependent optimization. We have used support vector machine (SVM) as the classification algorithm in the wrapper phase. The proposed hybrid approach has been validated on ten datasets from UCI Machine learning repository. A comparison is presented in terms of predictive accuracy, feature subset size and running time among the pure filter, pure wrapper, an earlier hybrid approach based on genetic algorithm and the proposed approach.

Integration of graph clustering with ant colony optimization for feature selection

Feature selection is an important preprocessing step in machine learning and pattern recognition. The ultimate goal of feature selection is to select a feature subset from the original feature set to increase the performance of learning algorithms. In this paper a novel feature selection method based on the graph clustering approach and ant colony optimization is proposed for classification problems. The proposed method’s algorithm works in three steps. In the first step, the entire feature set is represented as a graph. In the second step, the features are divided into several clusters using a community detection algorithm and finally in the third step, a novel search strategy based on the ant colony optimization is developed to select the final subset of features. Moreover the selected subset of each ant is evaluated using a supervised filter based method called novel separability index. Thus the proposed method does not need any learning model and can be classified as a filter based feature selection method. The proposed method integrates the community detection algorithm with a modified ant colony based search process for the feature selection problem. Furthermore, the sizes of the constructed subsets of each ant and also size of the final feature subset are determined automatically. The performance of the proposed method has been compared to those of the state-of-the-art filter and wrapper based feature selection methods on ten benchmark classification problems. The results show that our method has produced consistently better classification accuracies.