Feature Selection In Classification Problems Research Articles

Structural health monitoring via a group-theoretic WSA for optimal feature selection and data fusion

In this study, two binary versions of the Water Strider Algorithm (WSA) are proposed and applied to optimal feature selection in classification problems. In the new binary versions, the formulations of WSA in continuous space are converted into binary space using group-theoretic operators (in AWSA) and sigmoid function (in BWSA). AWSA, BWSA, genetic algorithm (GA), and binary particle swarm optimization (BPSO) are selected and compared over eighteen well-known datasets from the University of California, Irvine repository. The results of AWSA indicate its satisfactory performance compared to those of other algorithms. Then, they are applied to find optimal features of a structural health monitoring classification problem using two well-known machine learning classifiers, namely k-Nearest Neighbor (KNN) and Naïve Bayes (NB) algorithms. To further improve the accuracy of the classification models, a decision-level data fusion technique is proposed based on the improved Dempster-Shafer theory. It is demonstrated that the AWSA presents superior results compared to the other algorithms and the suggested decision-level data fusion provides a reliable detection of damage.

Hybrid firefly particle swarm optimisation algorithm for feature selection problems

AbstractFeature selection techniques play a vital role in the processes that deal with enormous amounts of data. These techniques have become extremely crucial and necessary for data mining and machine learning problems. Researchers have always been in a race to develop and provide libraries and frameworks to standardise this procedure. In this work, we propose a hybrid meta‐heuristic algorithm to facilitate the problem of feature selection for classification problems in machine learning. It is a python based, lucid and efficient algorithm geared towards optimising and striking a balance between the number of features selected and accuracy. The proposed work is a binary hybrid of existing meta‐heuristic algorithms, the particle swarm optimisation (PSO) algorithm, and the firefly algorithm (FA) such that it blends the best of each algorithm to provide an optimised and efficient way of solving the said problem. The suggested approach is assessed against six datasets from different domains that are publicly available at the UCI repository to demonstrate its validity. The datasets are Breast cancer, Iris, WBC, Mushroom, Glass ID, and Abalone. This approach has also been evaluated against similar, such evolutionary‐based approaches to prove its superiority. Various metrics such as accuracy, precision, recall, f1 score, number of selected features, and run time have been analysed, measured, and compared. The hybrid firefly particle swarm optimisation algorithm is found to be suitable for feature selection problems.

Learning the transfer function in binary metaheuristic algorithm for feature selection in classification problems

Learning the transfer function in binary metaheuristic algorithm for feature selection in classification problems

GRAFİKSEL LASSO İLE PORTFÖY OPTİMİZASYONU VE BORSA İSTANBUL’DA BİR UYGULAMA

Graphical Lasso (Least absolute shrinkage and selection operator) has become a popular tool in the field of machine learning in recent years. Although it has been deployed mainly for feature selection in classification problems, it is also used for covariance matrix estimation. Mean-variance portfolio optimization relies on sample covariance matrix for the calculation of the portfolio’s risk, whereas it has been most hardly criticized. The aim of this study is to demonstrate the effect of the covariance matrix estimation by Graphical Lasso algorithm with varying L1 penalty factors. Mean-variance portfolio optimization using empirical and estimated covariance matrices are applied to BIST 30 index and the results are compared.

Binary Horse Optimization Algorithm for Feature Selection

The bio-inspired research field has evolved greatly in the last few years due to the large number of novel proposed algorithms and their applications. The sources of inspiration for these novel bio-inspired algorithms are various, ranging from the behavior of groups of animals to the properties of various plants. One problem is the lack of one bio-inspired algorithm which can produce the best global solution for all types of optimization problems. The presented solution considers the proposal of a novel approach for feature selection in classification problems, which is based on a binary version of a novel bio-inspired algorithm. The principal contributions of this article are: (1) the presentation of the main steps of the original Horse Optimization Algorithm (HOA), (2) the adaptation of the HOA to a binary version called the Binary Horse Optimization Algorithm (BHOA), (3) the application of the BHOA in feature selection using nine state-of-the-art datasets from the UCI machine learning repository and the classifiers Random Forest (RF), Support Vector Machines (SVM), Gradient Boosted Trees (GBT), Logistic Regression (LR), K-Nearest Neighbors (K-NN), and Naïve Bayes (NB), and (4) the comparison of the results with the ones obtained using the Binary Grey Wolf Optimizer (BGWO), Binary Particle Swarm Optimization (BPSO), and Binary Crow Search Algorithm (BCSA). The experiments show that the BHOA is effective and robust, as it returned the best mean accuracy value and the best accuracy value for four and seven datasets, respectively, compared to BGWO, BPSO, and BCSA, which returned the best mean accuracy value for four, two, and two datasets, respectively, and the best accuracy value for eight, seven, and five datasets, respectively.

Advanced strategies on update mechanism of Sine Cosine Optimization Algorithm for feature selection in classification problems

Sine Cosine Algorithm (SCA) that is one of the population-based metaheuristic optimization algorithms basically consists of the updating mechanism based on sine and cosine functions. In this algorithm, a few random and adaptive variables are also utilized for more effective motions of the candidate solutions. SCA has some drawbacks like other some metaheuristic algorithms. SCA tends to be stuck into the local regions in the search space and this affects negatively on the computational effort required to find the best solution point in the search space. This paper presents four different improved versions of SCA. The proposed improvements on original SCA are the innovations on the updating mechanism of SCA. To evaluate the performances of Improved Sine Cosine Algorithms (ImpSCAs), well-known numerical optimization problems including CEC 2014 test suite are used. Firstly, different analyses of the proposed ImpSCAs are dealt with such as the convergence analysis, search history analysis, trajectory analysis, average distance analysis, and computational complexity analysis. Secondly, the proposed four versions of ImpSCAs are compared with the original SCA for CEC 2014 benchmark problems with dimension sizes of 10D, 30D and 50D. Finally, original SCA and ImpSCAs are adapted to select optimal feature combination and they are tested for 10 feature selection datasets taken from the UCI machine learning repository. The benchmark results show that the performances of the ImpSCA 1, ImpSCA 2, and ImpSCA 4 are better than that of the original SCA. From the feature selection results, it is observed that three versions of ImpSCAs (except ImpSCA 3) outperform the original SCA in 80% of the datasets. Source codes of ImpSCAs are publicly available at https://github.com/uguryuzgec/ImpSCAs.

Feature Selection Approach for Solving Imbalanced Data Problem in Single Nucleotide Polymorphism Discovery

Single Nucleotide Polymorphism (SNP) is a type of molecular marker which constitutes the phenotypic variations between individuals in certain species. In recent years, the advantages of SNP were widely considered in many fields, for instance in designing precision medicine in humans and assembling superior cultivars in plant breeding. The main challenge in SNP discovery is imbalanced data distribution between classes, where the number of true SNPs in question is much fewer than false SNPs. While the study in observing the benefit of feature selection in classification problem was widely reported, the use of this technique in solving imbalanced class problem still become interesting topic for research. In this study, we selected the features that most contribute in identifying SNP using Feature Assessment by Sliding Thresholds (FAST) method. FAST evaluates the contribution of each feature in identifying SNPs based on the Area under ROC Curve (AUC) value. SNP identification using 4 best features resulted in improved classifier performance in terms of G-Means compared to using 24 features. In addition, using feature selection techniques can reduce computational time and save resource needed.

Impact of Solution Representation in Nature-Inspired Algorithms for Feature Selection

This paper investigates how does the solution representation in nature-inspired algorithms impact the performance of feature selection in classification problems. Four most suitable nature-inspired algorithms for feature selection were considered in the analysis, namely the Differential Evolution, Artificial Bee Colony, Particle Swarm Optimization, and Genetic Algorithm. The binary-coded and real-coded variants of the mentioned algorithms were compared for filter-based and wrapper-based feature selection methodologies on datasets commonly used by the research community. Additionally, the algorithms' performance on reducing the feature subset size regarding different solution representations was compared. Statistical tests were performed for discovering any significant differences in the algorithms' performances.

Features Selection Based ABC-SVM and PSO-SVM in Classification Problem

Feature selection can be used to improve the performance of classification algorithms. This study aims to use algorithms for feature selection in classification problems. The method in feature selection use to the Artificial Bee Colony (ABC) algorithm and Particle Swarm Optimization (PSO) based on Support Vector Machine (SVM). The ABC-SVM algorithm functions as a method of feature selection to choose the optimal subset according to the objectives set to provide the results of the classification. Then, the PSO-SVM as a comparison method in other feature selection. The results of the classification conducted by PSO-SVM with the vowel dataset is good classification (AUC 0.873), when compared with the ABC-SVM with the classification result is superior (AUC 0.996). The results of the Precision Recall and F-Measure calculations on the PSO-VSM algorithm have good classification results for sonar and wavefrom data sets. Meanwhile, the results of tests conducted by ABC-SVM get superior value from the classifier quality efficiency in the vowel data set.

A new fusion of grey wolf optimizer algorithm with a two-phase mutation for feature selection

Because of their high dimensionality, dealing with large datasets can hinder the data mining process. Thus, the feature selection is a pre-process mandatory phase for reducing the dimensionality of datasets through using the most informative features and at the same time maximizing the classification accuracy. This paper proposes a new Grey Wolf Optimizer algorithm integrated with a Two-phase Mutation to solve the feature selection for classification problems based on the wrapper methods. The sigmoid function is used to transform the continuous search space to the binary one in order to match the binary nature of the feature selection problem. The two-phase mutation enhances the exploitation capability of the algorithm. The purpose of the first mutation phase is to reduce the number of selected features while preserving high classification accuracy. The purpose of the second mutation phase is to attempt to add more informative features that increase the classification accuracy. As the mutation phase can be time-consuming, the two-phase mutation can be done with a small probability. The wrapper methods can give high-quality solutions so we use one of the most famous wrapper methods which called k-Nearest Neighbor (k-NN) classifier. The Euclidean distance is computed to search for the k-NN. Each dataset is split into training and testing data using K-fold cross-validation to overcome the overfitting problem. Several comparisons with the most famous and modern algorithms such as flower algorithm, particle swarm optimization algorithm, multi-verse optimizer algorithm, whale optimization algorithm, and bat algorithm are done. The experiments are done using 35 datasets. Statistical analyses are made to prove the effectiveness of the proposed algorithm and its outperformance.

Mutual information and sensitivity analysis for feature selection in customer targeting: A comparative study

Feature selection is a highly relevant task in any data-driven knowledge discovery project. The present research focuses on analysing the advantages and disadvantages of using mutual information (MI) and data-based sensitivity analysis (DSA) for feature selection in classification problems, by applying both to a bank telemarketing case. A logistic regression model is built on the tuned set of features identified by each of the two techniques as the most influencing set of features on the success of a telemarketing contact, in a total of 13 features for MI and 9 for DSA. The latter performs better for lower values of false positives while the former is slightly better for a higher false-positive ratio. Thus, MI becomes a better choice if the intention is reducing slightly the cost of contacts without risking losing a high number of successes. However, DSA achieved good prediction results with less features.

A supervised filter method for multi-objective feature selection in EEG classification based on multi-resolution analysis for BCI

This paper proposes a supervised filter method for evolutionary multi-objective feature selection for classification problems in high-dimensional feature space, which is evaluated by comparison with wrapper approaches for the same application. The filter method based on a set of label-aided utility functions is compared with wrapper approaches using the accuracy and generalization properties in the effective searching of the most adequate subset of features through an evolutionary multi-objective optimization scheme. The target application corresponds to a brain–computer interface (BCI) classification task based on linear discriminant analysis (LDA) classifiers, where the properties of multi-resolution analysis (MRA) for signal analysis in temporal and spectral domains have been used to extract features from electroencephalogram (EEG) signals. The results, corresponding to a dataset obtained from the databases of the BCI Laboratory of the University of Essex, UK, including ten subjects with three different imagery movements, have allowed us to evaluate the advantages and drawbacks of the different approaches with respect to time consumption, accuracy and generalization capabilities.

Machine learning methods in the computational biology of cancer

The objectives of this Perspective paper are to review some recent advances in sparse feature selection for regression and classification, as well as compressed sensing, and to discuss how these might be used to develop tools to advance personalized cancer therapy. As an illustration of the possibilities, a new algorithm for sparse regression is presented and is applied to predict the time to tumour recurrence in ovarian cancer. A new algorithm for sparse feature selection in classification problems is presented, and its validation in endometrial cancer is briefly discussed. Some open problems are also presented.

Evolutionary computation for feature selection in classification problems

Feature subset selection (FSS) has received a great deal of attention in statistics, machine learning, and data mining. Real world data analyzed by data mining algorithms can involve a large number of redundant or irrelevant features or simply too many features for a learning algorithm to handle them efficiently. Feature selection is becoming essential as databases grow in size and complexity. The selection process is expected to bring benefits in terms of better performing models, computational efficiency, and simpler more understandable models. Evolutionary computation (EC) encompasses a number of naturally inspired techniques such as genetic algorithms, genetic programming, ant colony optimization, or particle swarm optimization algorithms. Such techniques are well suited to feature selection because the representation of a feature subset is straightforward and the evaluation can also be easily accomplished through the use of wrapper or filter algorithms. Furthermore, the capability of such heuristic algorithms to efficiently search large search spaces is of great advantage to the feature selection problem. Here, we review the use of different EC paradigms for feature selection in classification problems. We discuss details of each implementation including representation, evaluation, and validation. The review enables us to uncover the best EC algorithms for FSS and to point at future research directions. WIREs Data Mining Knowl Discov 2013, 3:381–407. doi: 10.1002/widm.1106This article is categorized under: Technologies > Classification Technologies > Computational Intelligence Technologies > Data Preprocessing

MULTI-OBJECTIVE EVOLUTIONARY ALGORITHMS FOR FILTER BASED FEATURE SELECTION IN CLASSIFICATION

Feature selection is a multi-objective problem with the two main conflicting objectives of minimising the number of features and maximising the classification performance. However, most existing feature selection algorithms are single objective and do not appropriately reflect the actual need. There are a small number of multi-objective feature selection algorithms, which are wrapper based and accordingly are computationally expensive and less general than filter algorithms. Evolutionary computation techniques are particularly suitable for multi-objective optimisation because they use a population of candidate solutions and are able to find multiple non-dominated solutions in a single run. However, the two well-known evolutionary multi-objective algorithms, non-dominated sorting based multi-objective genetic algorithm II (NSGAII) and strength Pareto evolutionary algorithm 2 (SPEA2) have not been applied to filter based feature selection. In this work, based on NSGAII and SPEA2, we develop two multi-objective, filter based feature selection frameworks. Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the two proposed frameworks. The proposed multi-objective algorithms are examined and compared with a single objective method and three traditional methods (two filters and one wrapper) on eight benchmark datasets. A decision tree is employed to test the classification performance. Experimental results show that the proposed multi-objective algorithms can automatically evolve a set of non-dominated solutions that include a smaller number of features and achieve better classification performance than using all features. NSGAII and SPEA2 outperform the single objective algorithm, the two traditional filter algorithms and even the traditional wrapper algorithm in terms of both the number of features and the classification performance in most cases. NSGAII achieves similar performance to SPEA2 for the datasets that consist of a small number of features and slightly better results when the number of features is large. This work represents the first study on NSGAII and SPEA2 for filter feature selection in classification problems with both providing field leading classification performance.

Estimating mutual information for feature selection in the presence of label noise

A way to achieve feature selection for classification problems polluted by label noise is proposed. The performances of traditional feature selection algorithms often decrease sharply when some samples are wrongly labelled. A method based on a probabilistic label noise model combined with a nearest neighbours-based entropy estimator is introduced to robustly evaluate the mutual information, a popular relevance criterion for feature selection. A backward greedy search procedure is used in combination with this criterion to find relevant sets of features. Experiments establish that (i) there is a real need to take a possible label noise into account when selecting features and (ii) the proposed methodology is effectively able to reduce the negative impact of the mislabelled data points on the feature selection process.

A New Feature Selection Method for One-Class Classification Problems

Feature selection is a data processing method that is used to select a few important features among many input features and to remove any irrelevant one. Although feature selection in classification problems has been the focus of much research, few feature selection methods are available for use in one-class classification problems (i.e., anomaly detection). In particular, existing feature selection methods cannot be applied for the feature selection of the one-class classification problem when there are no available observations for the anomaly (or the second class). In this study, we propose two support vector data description (SVDD)-based feature selection methods: SVDD-radius-recursive feature elimination (RFE) and SVDD dual-objective RFE. The SVDD-radius-RFE method can be used to minimize the size of the boundary of describing normal observations measured through the value of its radius squared and the SVDD-dual-objective-RFE method can be applied to obtain a compact description in the dual space of SVDD. Experimental results using both simulated and real-life datasets demonstrate that the proposed methods show the improved performance compared with existing support vector machine RFE methods even for the classification problems when available observations for the anomaly are few.

A multi-objective particle swarm optimisation for filter-based feature selection in classification problems

Feature selection has the two main objectives of minimising the classification error rate and the number of features. Based on binary particle swarm optimisation (BPSO), we develop two novel multi-objective feature selection frameworks for classification, which are multi-objective binary PSO using the idea of non-dominated sorting (NSBPSO) and multi-objective binary PSO using the ideas of crowding, mutation and dominance (CMDBPSO). Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the proposed frameworks. The proposed algorithms are examined and compared with a single objective method on eight benchmark data sets. Experimental results show that the proposed multi-objective algorithms can evolve a set of solutions that use a smaller number of features and achieve better classification performance than using all features. In most cases, NSBPSO achieves better results than the single objective algorithm and CMDBPSO outperforms all other methods mentioned above. This work represents the first study on multi-objective BPSO for filter-based feature selection.

Mutual information based input feature selection for classification problems

The elimination process aims to reduce the size of the input feature set and at the same time to retain the class discriminatory information for classification problems. This paper investigates the approaches to solve classification problems of the feature selection and proposes a new feature selection algorithm using the mutual information (MI) concept in information theory for the classification problems. The proposed algorithm calculates the MI between the combinations of input features and the class instead of the MI between a single input feature and the class for both continuous-valued and discrete-valued features. Three experimental tests are conducted to evaluate the proposed algorithm. Comparison studies of the proposed algorithm with the previously published classification algorithms indicate that the proposed algorithm is robust, stable and efficient.

Multi‐objective feature selection using a Bayesian artificial immune system

PurposeThe purpose of this paper is to apply a multi‐objective Bayesian artificial immune system (MOBAIS) to feature selection in classification problems aiming at minimizing both the classification error and cardinality of the subset of features. The algorithm is able to perform a multimodal search maintaining population diversity and controlling automatically the population size according to the problem. In addition, it is capable of identifying and preserving building blocks (partial components of the whole solution) effectively.Design/methodology/approachThe algorithm evolves candidate subsets of features by replacing the traditional mutation operator in immune‐inspired algorithms with a probabilistic model which represents the probability distribution of the promising solutions found so far. Then, the probabilistic model is used to generate new individuals. A Bayesian network is adopted as the probabilistic model due to its capability of capturing expressive interactions among the variables of the problem. In order to evaluate the proposal, it was applied to ten datasets and the results compared with those generated by state‐of‐the‐art algorithms.FindingsThe experiments demonstrate the effectiveness of the multi‐objective approach to feature selection. The algorithm found parsimonious subsets of features and the classifiers produced a significant improvement in the accuracy. In addition, the maintenance of building blocks avoids the disruption of partial solutions, leading to a quick convergence.Originality/valueThe originality of this paper relies on the proposal of a novel algorithm to multi‐objective feature selection.