Solve Feature Selection Problems Research Articles

Feature selection based on improved binary global harmony search for data classification

Harmony search (HS) is an effective meta-heuristic algorithm inspired by the music improvisation process, where musicians search for a pleasing harmony by adjusting their instruments’ pitches. The HS algorithm and its variants have been widely used to solve binary and continuous optimization problems. In this paper, we propose an improved binary global harmony search algorithm, called IBGHS, to undertake feature selection problems. A modified improvisation step is introduced to enhance the global search ability and increase the convergence speed of the algorithm. In addition, the K-nearest neighbor (KNN) is used as an underlying learning model to evaluate the effectiveness of the selected feature subsets. The experimental results on eighteen benchmark problems indicate that the proposed IBGHS algorithm is able to produce comparable results as compared with other state-of-the-art population-based methods such as genetic algorithm (GA), particle swarm optimization (PSO), antlion optimizer (ALO), novel global harmony search (NGHS) and whale optimization algorithm (WOA) in solving feature selection problems.

A Comprehensive Analysis of Nature-Inspired Meta-Heuristic Techniques for Feature Selection Problem

Meta-heuristics are problem-independent optimization techniques which provide an optimal solution by exploring and exploiting the entire search space iteratively. These techniques have been successfully engaged to solve distinct real-life and multidisciplinary problems. A good amount of literature has been already published on the design and role of various meta-heuristic algorithms and on their variants. The aim of this study is to present a comprehensive analysis of nature-inspired meta-heuristic utilized in the domain of feature selection. A systematic review methodology has been used for synthesis and analysis of one hundered and seventy six articles. It is one of the important multidisciplinary research areas that assist in finding an optimal set of features so that a better rate of classification can be achieved. The concept of feature selection process along with relevance and redundancy metric is briefly elucidated. A categorical list of nature-inspired meta-heuristic techniques has been presented. The major applications of these techniques are explored to highlight the least and most explored areas. The area of disease diagnosis has been extensively assessed. In addition, the special attention has been given on highlighting the role and performance of binary and chaotic variants of different nature-inspired meta-heuristic techniques. The summary of nature-inspired meta-heuristic methods and their variants along with datasets, performance (mean, best, worst, error rate and standard deviation) is also depicted. In addition, the detailed publication trend of meta-heuristic feature selection approaches has also been presented. The research gaps have been identified for the researcher who inclines to design or analyze the performance of divergent meta-heuristic techniques in solving feature selection problem.

Hybrid binary Sine Cosine Algorithm and Ant Lion Optimization (SCALO) approaches for feature selection problem

These days, a massive quantity of data is produced online and is incorporated into a variety of datasets in the form of features, however there are lot of features in these datasets that may not be relevant to the problem. In this perspective, feature selection aids to improve the classification accuracy with lesser number of features, which can be well thought-out as an optimization problem. In this paper, Sine Cosine Algorithm (SCA) hybridized with Ant Lion Optimizer (ALO) to form a hybrid Sine Cosine Ant Lion Optimizer (SCALO) is proposed. The proposed algorithm is mapped to its binary versions by using the concept of transfer functions, with the objective to eliminate the inappropriate features and to enhance the accuracy of the classification algorithm (or in any case remains the same). For the purpose of experimentation, this research considers 18 diverse datasets and moreover, the performance of the binary versions of SCALO is compared with some of the latest metaheuristic algorithms, on the basis of various criterions. It can be observed that the binary versions of SCALO perform better than the other algorithms on various evaluation criterions for solving feature selection problem.

Heart Disease Identification Method Using Machine Learning Classification in E-Healthcare

Heart disease is one of the complex diseases and globally many people suffered from this disease. On time and efficient identification of heart disease plays a key role in healthcare, particularly in the field of cardiology. In this article, we proposed an efficient and accurate system to diagnosis heart disease and the system is based on machine learning techniques. The system is developed based on classification algorithms includes Support vector machine, Logistic regression, Artificial neural network, K-nearest neighbor, Naive bays, and Decision tree while standard features selection algorithms have been used such as Relief, Minimal redundancy maximal relevance, Least absolute shrinkage selection operator and Local learning for removing irrelevant and redundant features. We also proposed novel fast conditional mutual information feature selection algorithm to solve feature selection problem. The features selection algorithms are used for features selection to increase the classification accuracy and reduce the execution time of classification system. Furthermore, the leave one subject out cross-validation method has been used for learning the best practices of model assessment and for hyperparameter tuning. The performance measuring metrics are used for assessment of the performances of the classifiers. The performances of the classifiers have been checked on the selected features as selected by features selection algorithms. The experimental results show that the proposed feature selection algorithm (FCMIM) is feasible with classifier support vector machine for designing a high-level intelligent system to identify heart disease. The suggested diagnosis system (FCMIM-SVM) achieved good accuracy as compared to previously proposed methods. Additionally, the proposed system can easily be implemented in healthcare for the identification of heart disease.

An Improved Harris Hawks Optimization Algorithm With Simulated Annealing for Feature Selection in the Medical Field

Harris Hawks Optimization (HHO) algorithm is a new metaheuristic algorithm, inspired by the cooperative behavior and chasing style of Harris’ Hawks in nature called surprise pounce. HHO demonstrated promising results compared to other optimization methods. However, HHO suffers from local optima and population diversity drawbacks. To overcome these limitations and adapt it to solve feature selection problems, a novel metaheuristic optimizer, namely Chaotic Harris Hawks Optimization (CHHO), is proposed. Two main improvements are suggested to the standard HHO algorithm. The first improvement is to apply the chaotic maps at the initialization phase of HHO to enhance the population diversity in the search space. The second improvement is to use the Simulated Annealing (SA) algorithm to the current best solution to improve HHO exploitation. To validate the performance of the proposed algorithm, CHHO was applied on 14 medical benchmark datasets from the UCI machine learning repository. The proposed CHHO was compared with the original HHO and some famous and recent metaheuristics algorithms, containing Grasshopper Optimization Algorithm (GOA), Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Butterfly Optimization Algorithm (BOA), and Ant Lion Optimizer (ALO). The used evaluation metrics include the number of selected features, classification accuracy, fitness values, Wilcoxon’s statistical test ( $P$ -value), and convergence curve. Based on the achieved results, CHHO confirms its superiority over the standard HHO algorithm and the other optimization algorithms on the majority of the medical datasets.

Self-adaptive parameter and strategy based particle swarm optimization for large-scale feature selection problems with multiple classifiers

Feature selection has been widely used in classification for improving classification accuracy and reducing computational complexity. Recently, evolutionary computation (EC) has become an important approach for solving feature selection problems. However, firstly, as the datasets processed by classifiers become increasingly large and complex, more and more irrelevant and redundant features may exist and there may be more local optima in the large-scale feature space. Therefore, traditional EC algorithms which have only one candidate solution generation strategy (CSGS) with fixed parameter values may not perform well in searching for the optimal feature subsets for large-scale feature selection problems. Secondly, many existing studies usually use only one classifier to evaluate feature subsets. To show the effectiveness of evolutionary algorithms for feature selection problems, more classifiers should be tested. Thus, in order to efficiently solve large-scale feature selection problems and to show whether the EC-based feature selection method is efficient for more classifiers, a self-adaptive parameter and strategy based particle swarm optimization (SPS-PSO) algorithm is proposed in this paper using multiple classifiers. In SPS-PSO, a representation scheme of solutions and five CSGSs have been used. To automatically adjust the CSGSs and their parameter values during the evolutionary process, a strategy self-adaptive mechanism and a parameter self-adaptive mechanism are employed in the framework of particle swarm optimization (PSO). By using the self-adaptive mechanisms, the SPS-PSO can adjust both CSGSs and their parameter values when solving different large-scale feature selection problems. Therefore, SPS-PSO has good global and local search ability when dealing with these large-scale problems. Moreover, four classifiers, i.e., k-nearest neighbor (KNN), linear discriminant analysis (LDA), extreme learning machine (ELM), and support vector machine (SVM), are individually used as the evaluation functions for testing the effectiveness of feature subsets generated by SPS-PSO. Nine datasets from the UCI Machine Learning Repository and Causality Workbench are used in the experiments. All the nine datasets have more than 600 dimensions, and two of them have more than 5,000 dimensions. The experimental results show that the strategy and parameter self-adaptive mechanisms can improve the performance of the evolutionary algorithms, and that SPS-PSO can achieve higher classification accuracy and obtain more concise solutions than those of the other algorithms on the large-scale feature problems selected in this research. In addition, feature selection can improve the classification accuracy and reduce computational time for various classifiers. Furthermore, KNN is a better surrogate model compared with the other classifiers used in these experiments.

Optimization Algorithms to Solve Feature Selection Problem: A Review

Optimization Algorithms to Solve Feature Selection Problem: A Review

Improved Salp Swarm Algorithm based on opposition based learning and novel local search algorithm for feature selection

Many fields such as data science, data mining suffered from the rapid growth of data volume and high data dimensionality. The main problems which are faced by these fields include the high computational cost, memory cost, and low accuracy performance. These problems will occur because these fields are mainly used machine learning classifiers. However, machine learning accuracy is affected by the noisy and irrelevant features. In addition, the computational and memory cost of the machine learning is mainly affected by the size of the used datasets. Thus, to solve these problems, feature selection can be used to select optimal subset of features and reduce the data dimensionality. Feature selection represents an important preprocessing step in many intelligent and expert systems such as intrusion detection, disease prediction, and sentiment analysis. An improved version of Salp Swarm Algorithm (ISSA) is proposed in this study to solve feature selection problems and select the optimal subset of features in wrapper-mode. Two main improvements were included into the original SSA algorithm to alleviate its drawbacks and adapt it for feature selection problems. The first improvement includes the use of Opposition Based Learning (OBL) at initialization phase of SSA to improve its population diversity in the search space. The second improvement includes the development and use of new Local Search Algorithm with SSA to improve its exploitation. To confirm and validate the performance of the proposed improved SSA (ISSA), ISSA was applied on 18 datasets from UCI repository. In addition, ISSA was compared with four well-known optimization algorithms such as Genetic Algorithm, Particle Swarm Optimization, Grasshopper Optimization Algorithm, and Ant Lion Optimizer. In these experiments four different assessment criteria were used. The rdemonstrate that ISSA outperforms all baseline algorithms in terms of fitness values, accuracy, convergence curves, and feature reduction in most of the used datasets. The wrapper feature selection mode can be used in different application areas of expert and intelligent systems and this is confirmed from the obtained results over different types of datasets.

Solving feature selection problem by hybrid binary genetic enhanced particle swarm optimization algorithm

Solving feature selection problem by hybrid binary genetic enhanced particle swarm optimization algorithm

Self-Adaptive Particle Swarm Optimization for Large-Scale Feature Selection in Classification

Many evolutionary computation (EC) methods have been used to solve feature selection problems and they perform well on most small-scale feature selection problems. However, as the dimensionality of feature selection problems increases, the solution space increases exponentially. Meanwhile, there are more irrelevant features than relevant features in datasets, which leads to many local optima in the huge solution space. Therefore, the existing EC methods still suffer from the problem of stagnation in local optima on large-scale feature selection problems. Furthermore, large-scale feature selection problems with different datasets may have different properties. Thus, it may be of low performance to solve different large-scale feature selection problems with an existing EC method that has only one candidate solution generation strategy (CSGS). In addition, it is time-consuming to find a suitable EC method and corresponding suitable parameter values for a given large-scale feature selection problem if we want to solve it effectively and efficiently. In this article, we propose a self-adaptive particle swarm optimization (SaPSO) algorithm for feature selection, particularly for large-scale feature selection. First, an encoding scheme for the feature selection problem is employed in the SaPSO. Second, three important issues related to self-adaptive algorithms are investigated. After that, the SaPSO algorithm with a typical self-adaptive mechanism is proposed. The experimental results on 12 datasets show that the solution size obtained by the SaPSO algorithm is smaller than its EC counterparts on all datasets. The SaPSO algorithm performs better than its non-EC and EC counterparts in terms of classification accuracy not only on most training sets but also on most test sets. Furthermore, as the dimensionality of the feature selection problem increases, the advantages of SaPSO become more prominent. This highlights that the SaPSO algorithm is suitable for solving feature selection problems, particularly large-scale feature selection problems.

Feature selection based on rough set approach, wrapper approach, and binary whale optimization algorithm

The principle of any approach for solving feature selection problem is to find a subset of the original features. Since finding a minimal subset of the features is an NP-hard problem, it is necessary to develop and propose practical and efficient heuristic algorithms. The whale optimization algorithm is a recently developed nature-inspired meta-heuristic optimization algorithm that imitates the hunting behavior of humpback whales to solve continuous optimization problems. In this paper, we propose a novel binary whale optimization algorithm (BWOA) to solve feature selection problem. BWOA is especially desirable and appealing for feature selection problem whenever there is no heuristic information that can lead the search to the optimal minimal subset. Nonetheless, whales can find the best features as they hunt the prey. Rough set theory (RST) is one of the effective algorithms for feature selection. We use RST with BWOA as the first experiment, and in the second experiment, we use a wrapper approach with BWOA on three different classifiers for feature selection. Also, we verify the performance and the effectiveness of the proposed algorithm by performing our experiments using 32 datasets from the UCI machine learning repository and comparing the proposed algorithm with some powerful existing algorithms in the literature. Furthermore, we employ two nonparametric statistical tests, Wilcoxon Signed-Rank test, and Friedman test, at 5% significance level. Our results show that the proposed algorithm can provide an efficient tool to find a minimal subset of the features.

An improved runner-root algorithm for solving feature selection problems based on rough sets and neighborhood rough sets

Solving the feature selection problem is considered an important issue when addressing data from real applications that contain a large number of features. However, not all of these features are important; therefore, the redundant features must be removed because they affect the accuracy of the data representation and introduce time complexity into the analysis of these data. For these reasons, the feature selection problem is considered an NP-complete nonlinearly constrained optimization problem. The rough set (RS) and neighborhood rough set (NRS) are the most powerful methods used to solve the feature selection problem; however, both approaches suffer from high time complexity. To avoid these limitations, we combined the RS and NRS with a new metaheuristic algorithm called the runner-root algorithm (RRA). The spirit of the RRA originated from real-life plants called running plants, which have roots and runners that spread the plants in search of minerals and water resources through their root and runner development. To validate the proposed algorithm, several UCI Machine Learning Repository datasets are used to compute the performance of our algorithm employing two effective classifiers, the random forest and the K-nearest neighbor, in addition to some other measures for the performance evaluation. The experimental results illustrate that the proposed algorithm is superior to the state-of-the-art metaheuristic algorithms in terms of the performance measures. Additionally, the NRS increases the performance of the proposed method more than the RS as an objective function.

An evolutionary gravitational search-based feature selection

With recent advancements in data collection tools and the widespread use of intelligent information systems, a huge amount of data streams with lots of redundant, irrelevant, and noisy features are collected and a large number of features (attributes) should be processed. Therefore, there is a growing demand for developing efficient Feature Selection (FS) techniques. Gravitational Search algorithm (GSA) is a successful population-based metaheuristic inspired by Newton’s law of gravity. In this research, a novel GSA-based algorithm with evolutionary crossover and mutation operators is proposed to deal with feature selection (FS) tasks. As an NP-hard problem, FS finds an optimal subset of features from a given set. For the proposed wrapper FS method, both K-Nearest Neighbors (KNN) and Decision Tree (DT) classifiers are used as evaluators. Eighteen well-known UCI datasets are utilized to assess the performance of the proposed approaches. In order to verify the efficiency of proposed algorithms, the results are compared with some popular nature-inspired algorithms (i.e. Genetic Algorithm (GA), Particle Swarm Optimizer (PSO), and Grey Wolf Optimizer (GWO)). The extensive results and comparisons demonstrate the superiority of the proposed algorithm in solving FS problems.

Hybrid Efficient Genetic Algorithm for Big Data Feature Selection Problems

Due to the huge amount of data being generating from different sources, the analyzing and extracting of useful information from these data becomes a very complex task. The difficulty of dealing with big data optimization problems comes from many factors such as the high number of features, and the existing of lost data. The feature selection process becomes an important step in many data mining and machine learning algorithms to reduce the dimensionality of the optimization problems and increase the performance of the classification or clustering algorithms. In this paper, a set of hybrid and efficient genetic algorithms are proposed to solve feature selection problem, when the handled data has a large feature size. The proposed algorithms use a new gene-weighted mechanism that can adaptively classify the features into strong relative features, weak or redundant features, and unstable features during the evolution of the algorithm. Based on this classification, the proposed algorithm gives the strong features high priority and the weak features less priority when generating new candidate solutions. In the same time, the proposed algorithm tries to more concentrate on unstable features that sometimes appear and sometimes disappear from the best solutions of the population. The performance of proposed algorithms is investigated by using different datasets and feature selection algorithms. The results show that our proposed algorithms can outperform the other feature selection algorithms and effectively enhance the classification performance over the tested datasets.

Discrete honey bees mating optimization (DHBMO) for solving environmental optimization problem using single and multi-objectives optimization

One of the recently proposed algorithms in the field of nature-inspired algorithm is the Honey Bees Mating Optimization (HBMO) algorithm by Haddad and Afshar. The HBMO algorithm simulates the marriage behaviour of bees. The solutions are incrementally constructed by the HBMO algorithm. Therefore, the HBMO algorithm is a swarm-based constructive optimization algorithm. However, no study has reported the modification of HBMO for feature selection. Therefore, we try to develop a discrete honey bees mating optimization (DHBMO) as the modification of HBMO for solving discrete optimization problem especially feature selection to improve the performance of machine vision for predicting water stress in plants. We test the algorithm to solve feature selection problem using single and multi-objectives optimization to optimize water usage in plants for environmental sustainability. The results showed DHBMO has better performance compared to the existed nature-inspired optimization algorithms such as genetic algorithms (GA), simulated annealing (SA) and discrete-particle swarm optimization (discrete-PSO).

High dimensional sentiment classification of product reviews using evolutionary computation

Feature selection is an important process in text classification. In general, traditional feature selection approaches are based on exhaustive search hence become inefficient due to a large search space. Further, this task becomes more challenging as the number of features increases. Recently, evolutionary computation (EC)-based search techniques have received a lot of attention in solving feature selection problem in high-dimensional feature space. This paper proposes a particle swarm optimisation (PSO)-based feature selection approach which is capable of generating the desired number of high-quality features from a large feature space. The proposed algorithm is tested on a large dataset and compared with several existing state-of-the-art algorithms used for feature selection. The accuracy of the underlying classifier has been considered as a measure of performance. Our obtained results demonstrated that the proposed PSO-based feature selection approach outperforms the other traditional feature selection algorithms in all the considered classifiers.

Binary Optimization Using Hybrid Grey Wolf Optimization for Feature Selection

A binary version of the hybrid grey wolf optimization (GWO) and particle swarm optimization (PSO) is proposed to solve feature selection problems in this paper. The original PSOGWO is a new hybrid optimization algorithm that benefits from the strengths of both GWO and PSO. Despite the superior performance, the original hybrid approach is appropriate for problems with a continuous search space. Feature selection, however, is a binary problem. Therefore, a binary version of hybrid PSOGWO called BGWOPSO is proposed to find the best feature subset. To find the best solutions, the wrapper-based method K-nearest neighbors classifier with Euclidean separation matric is utilized. For performance evaluation of the proposed binary algorithm, 18 standard benchmark datasets from UCI repository are employed. The results show that BGWOPSO significantly outperformed the binary GWO (BGWO), the binary PSO, the binary genetic algorithm, and the whale optimization algorithm with simulated annealing when using several performance measures including accuracy, selecting the best optimal features, and the computational time.

Feature Selection Using an Improved Gravitational Search Algorithm

Feature selection is an important issue in the field of machine learning, which can reduce misleading computations and improve classification performance. Generally, feature selection can be considered as a binary optimization problem. Gravitational Search Algorithm (GSA) is a population-based heuristic algorithm inspired by Newton's laws of gravity and motion. Although GSA shows good performance in solving optimization problems, it has a shortcoming of premature convergence. In this paper, the concept of global memory is introduced and the definition of exponential Kbest is used in an improved version of GSA called IGSA. In this algorithm, the position of the optimal solution obtained so far is memorized, which can effectively prevent particles from gathering together and moving slowly. In this way, the exploitation ability of the algorithm gets improved, and a proper balance between exploration and exploitation gets established. Besides, the exponential Kbest can significantly decrease the running time. In order to solve feature selection problem, a binary IGSA (BIGSA) is further introduced. The proposed algorithm is tested on a set of standard datasets and compared with other algorithms. The experimental results confirm the high efficiency of BIGSA for feature selection.

A New Hybrid Seagull Optimization Algorithm for Feature Selection

Hybrid algorithms have attracted more and more attention in the field of optimization algorithms. In this paper, three hybrid algorithms are proposed to solve feature selection problems based on seagull optimization algorithm (SOA) and thermal exchange optimization (TEO). In the first algorithm, we take the roulette wheel to choose one of the two algorithms for located updating. Another method is to join the TEO algorithm for optimization after SOA algorithm iteration. The last method is to adopt TEO algorithm's heat exchange formula to improve the seagull attack mode of SOA algorithm, so as to improve the exploitation ability of SOA algorithm. The performance of proposed methods is evaluated on 20 standard benchmark datasets in the UCI repository and compared with three well-known hybrid optimization feature selection methods in the literature. The experimental results illustrate that the proposed algorithm has high efficiency in improving classification accuracy, ensuring the ability of hybrid SOA algorithm in feature selection and classification task information attribute selection, and reducing the CPU time.

Ant Colony Optimization Based Subset Feature Selection in Speech Processing: Constructing Graphs with Degree Sequences

Feature selection or the process of selecting the most discriminating feature subset is an essential practice in speech processing that significantly affects the performance of classification. However, the volume of features that presents in speech processing makes the feature selection perplexing. Moreover, finding the optimal feature subset is a NP-hard problem (2n). Thus, a good searching strategy is required to avoid evaluating large number of combinations in the whole feature subsets. As a result, in recent years, many heuristic based search algorithms are developed to address this NP-hard problem. One of the several meta heuristic algorithms that is applied in many application domains to solve feature selection problem is Ant Colony Optimization (ACO) based algorithms. ACO based algorithms are nature-inspired from the foraging behavior of actual ants. The success of an ACO based feature selection algorithm depends on the choice of the construction graph with respect to runtime behavior. While most ACO based feature selection algorithms use fully connected graphs, this paper proposes ACO based algorithm that uses graphs with prescribed degree sequences. In this method, the degree of the graph representing the search space will be predicted and the construction graph that satisfies the predicted degree will be generated. This research direction on graph representation for ACO algorithms may offer possibilities to reduce computation complexity from O(n2) to O(nm) in which m is the number of edges. This paper outlines some popular optimization based feature selection algorithms in the field of speech processing applications and overviewed ACO algorithm and its main variants. In addition to that, ACO based feature selection is explained and its application in various speech processing tasks is reviewed. Finally, a degree based graph construction for ACO algorithms is proposed.