Evolutionary Feature Selection Research Articles

An improved evolutionary wrapper-filter feature selection approach with a new initialisation scheme

Treated as one of the popular measures in information theory, fuzzy mutual information quantifies the amount of information that one random variable has about another one. Different from standard mutual information, fuzzy mutual information can deal with not only discrete-valued but also real-valued variables. Therefore, fuzzy mutual information has been recently used in evolutionary filter feature selection approaches to measure the correlation between the classes and the features, and the dependencies within a feature set. Typically, this way can be considered as computationally efficient but sometimes it may not contribute to the performance of a classification algorithm. To address this issue, an improved evolutionary wrapper-filter approach which integrates an initialisation scheme and a local search module based on fuzzy mutual information in differential evolution is proposed. According to a number of experiments conducted on several real-world benchmark datasets, the proposed approach does not only significantly improve the computational efficiency of an evolutionary computation technique but also the performance of a classification algorithm.

Advanced feature selection to study the internationalization strategy of enterprises.

Firms face an increasingly complex economic and financial environment in which the access to international networks and markets is crucial. To be successful, companies need to understand the role of internationalization determinants such as bilateral psychic distance, experience, etc. Cutting-edge feature selection methods are applied in the present paper and compared to previous results to gain deep knowledge about strategies for Foreign Direct Investment. More precisely, evolutionary feature selection, addressed from the wrapper approach, is applied with two different classifiers as the fitness function: Bagged Trees and Extreme Learning Machines. The proposed intelligent system is validated when applied to real-life data from Spanish Multinational Enterprises (MNEs). These data were extracted from databases belonging to the Spanish Ministry of Industry, Tourism, and Trade. As a result, interesting conclusions are derived about the key features driving to the internationalization of the companies under study. This is the first time that such outcomes are obtained by an intelligent system on internationalization data.

Feature selection and regularization of interpretable soft sensors using evolutionary multi-objective optimization design procedures

Soft sensors are mathematical models that estimate hard-to-measure variables given easy-to-measure ones. This field of study has given the industry a valuable tool to enable a better control of different plants and processes. With such models, quality indicators and other variables that usually demand costly or slow sensors can be predicted in real time. However, one important factor for application in industry is model interpretability. Moreover, regression models have an inherent trade-off between bias and variance, which is not considered by usual learning algorithms. Therefore, this work proposes a novel multi-objective least squares based on evolutionary feature selection and regularization, which incorporates feature selection and regularization, as well as the search for a better trade-off between bias and variance, using evolutionary multi-objective optimization algorithms. Experiments on two famous soft sensor benchmark problems, the debutanizer column and the sulfur recovery unit, indicate that the proposed algorithm can train more robust interpretable linear models than the commonly used partial least squares and least absolute shrinkage and selection operator.

Estimation of natural streams longitudinal dispersion coefficient using hybrid evolutionary machine learning model

Among several indicators for river engineering sustainability, the longitudinal dispersion coefficient () is the main parameter that defines the transport of pollutants in natural streams. Accurate estimation of has been challenging for hydrologists due to the high stochasticity and non-linearity of this hydraulic-environmental parameter. This study presents a new hybrid machine learning (ML) model integrating a Gaussian Process Regression (GPR) and an evolutionary feature selection (FS) approach (i.e. Covariance Matrix Adaptation Evolution Strategy (CMAES)) to estimate in natural streams. The dataset consists of geometric and hydraulic river system parameters from 29 streams in the United States. The modeling results showed that the proposed model outperformed other models in the literature, producing more stable and accurate estimations. The FS approach evidenced the significance of the cross-sectional average flow velocity (U), channel width (B), and channel sinuosity σ to estimate the dispersion coefficient. In quantitative terms, the integrated GPR model with feature selection approach attained the minimum root mean square error () and maximum coefficient of determination (). The proposed hybrid evolutionary ML model arises as robust, flexible and reliable alternative computer aid technology for predicting the longitudinal dispersion coefficient in natural streams.

Multi-Objective Evolutionary Simultaneous Feature Selection and Outlier Detection for Regression

When investigating the causes of contamination in specific contexts, such as in underground water wells, multivariate regression is commonly used to establish possible links between the chemical-physical values of the samples and the levels of contaminant. Two issues often arise from such a statistical analysis: selecting the best predicting variables and detecting the instances that can be suspected to be outliers. In this paper, we propose a comprehensive, integrated, and general optimization model that solves these two problems simultaneously in such a way that outliers can be detected in reference to the specific variables that are selected for the regression, and we implement such an optimization model with a well-known evolutionary algorithm. We test our proposal on data extracted from a project whose aim is to establish the causes of the contamination of underwater water wells in a very specific area of northeastern Italy. The results show that our variable selection and outlier detection algorithm allows the synthesis of very reliable, interpretable, and clean regression models.

Toward a Fair Evaluation and Analysis of Feature Selection for Music Tag Classification

Accurate music tag classification of music clips has been attracting great attention recently, because it allows one to provide various music excerpts, including unpopular ones, to users based on the clips’ acoustic similarities. Given a user’s preferred music, acoustic features are extracted and then fed into the classifier, which outputs the related tag to recommend new music. Furthermore, the accuracy of the tag classifiers can be improved by selecting the best feature subset based on the domain to which the tag belongs. However, recent studies have struggled to evaluate the superiority of various classifiers because they utilize different feature extractors. In this study, to conduct a direct comparison of existing methods of classification, we create 20 music datasets with the same acoustic feature structure. In addition, we propose an effective evolutionary feature selection algorithm to evaluate the effectiveness of feature selection for tag classification. Our experiments demonstrate that the proposed method improves the accuracy of tag classification, and the analysis with multiple datasets provides valuable insights, such as the important features for general music tag classification in target domains.

Classification of Clustered Snow Off Dates Over British Columbia, Canada, from Mean Sea Level Pressure

ABSTRACT Atmosphere–ocean teleconnections influence the accumulation and melt of snow in western Canada and can be useful in seasonal forecasting of snowmelt and runoff. Interannual variation in these atmosphere–ocean modes has been shown to influence the accumulation and melt of snow within British Columbia (BC), Canada. We investigate fall mean sea level pressure (MSLP) globally as a predictor of remotely sensed snowmelt dates within BC. We use the last day of continuous snow cover () detected from time series satellite imagery acquired by the Moderate Resolution Imaging Spectroradiometer for the hydrological years 2000–2018. It has been shown that is correlated with continuous snow duration and is also of interest to seasonal forecasters. Global MSLP from the Fifth major global reanalysis produced by the European Centre for Medium-range Weather Forecasts was obtained over hydrological years 1979–2018. An S-mode (time versus location) principal component analysis was carried out on both datasets. The principal component scores were grouped using a k-means clustering routine. Using evolutionary feature selection, the subset of MSLP principal components that provided good linear discrimination of the clusters were found. We explore the atmospheric MSLP principal components that influence the timing of snowmelt over BC and use them to predict the clusters at a seasonal lead time.

Clinical data classification using an enhanced SMOTE and chaotic evolutionary feature selection

Class imbalance and the presence of irrelevant or redundant features in training data can pose serious challenges to the development of a classification framework. This paper proposes a framework for developing a Clinical Decision Support System (CDSS) that addresses class imbalance and the feature selection problem. Under this framework, the dataset is balanced at the data level and a wrapper approach is used to perform feature selection. The following three clinical datasets from the University of California Irvine (UCI) machine learning repository were used for experimentation: the Indian Liver Patient Dataset (ILPD), the Thoracic Surgery Dataset (TSD) and the Pima Indian Diabetes (PID) dataset. The Synthetic Minority Over-sampling Technique (SMOTE), which was enhanced using Orchard's algorithm, was used to balance the datasets. A wrapper approach that uses Chaotic Multi-Verse Optimisation (CMVO) was proposed for feature subset selection. The arithmetic mean of the Matthews correlation coefficient (MCC) and F-score (F1), which was measured using a Random Forest (RF) classifier, was used as the fitness function. After selecting the relevant features, a RF, which comprises 100 estimators and uses the Information Gain Ratio as the split criteria, was used for classification. The classifier achieved a 0.65 MCC, a 0.84 F1 and 82.46% accuracy for the ILPD; a 0.74 MCC, a 0.87 F1 and 86.88% accuracy for the TSD; and a 0.78 MCC, a 0.89 F1and 89.04% accuracy for the PID dataset. The effects of balancing and feature selection on the classifier were investigated and the performance of the framework was compared with the existing works in the literature. The results showed that the proposed framework is competitive in terms of the three performance measures used. The results of a Wilcoxon test confirmed the statistical superiority of the proposed method.

Neonatal EEG sleep stage classification based on deep learning and HMM

Objective. Automatic sleep stage scoring is of great importance for investigating sleep architecture during infancy. In this work, we introduce a novel multichannel approach based on deep learning networks and hidden Markov models (HMM) to improve the accuracy of sleep stage classification in term neonates. Approach. The classification performance was evaluated on quiet sleep (QS) and active sleep (AS) stages, each with two sub-states, using multichannel EEG data recorded from sixteen neonates with postmenstrual age of 38–40 weeks. A comprehensive set of linear and nonlinear features were extracted from thirty-second EEG segments. The feature space dimensionality was then reduced by using an evolutionary feature selection method called MGCACO (Modified Graph Clustering Ant Colony Optimization) based on the relevance and redundancy analysis. A bi-directional long-short time memory (BiLSTM) network was trained for sleep stage classification. The number of channels was optimized using the sequential forward selection method to reduce the spatial space. Finally, an HMM-based postprocessing stage was used to reduce false positives by incorporating the knowledge of transition probabilities between stages into the classification process. The method performance was evaluated using the K-fold (KFCV) and leave-one-out cross-validation (LOOCV) strategies. Main results. Using six-bipolar channels, our method achieved a mean kappa and an overall accuracy of 0.71–0.76 and 78.9%–82.4% using the KFCV and LOOCV strategies, respectively. Significance. The presented automatic sleep stage scoring method can be used to study the neurodevelopmental process and to diagnose brain abnormalities in term neonates.

Machine learning-based identification of radiofrequency electromagnetic radiation (RF-EMR) effect on brain morphology: a preliminary study.

The brain of a human and other organisms is affected by the electromagnetic field (EMF) radiations, emanating from the cell phones and mobile towers. Prolonged exposure to EMF radiations may cause neurological changes in the brain, which in turn may bring chemical as well as morphological changes in the brain. Conventionally, the identification of EMF radiation effect on the brain is performed using cellular-level analysis. In the present work, an automatic image processing-based approach is used where geometric features extracted from the segmented brain region has been analyzed for identifying the effect of EMF radiation on the morphology of a brain, using drosophila as a specimen. Genetic algorithm-based evolutionary feature selection algorithm has been used to select an optimal set of geometrical features, which, when fed to the machine learning classifiers, result in their optimal performance. The best classification accuracy has been obtained with the neural network with an optimally selected subset of geometrical features. A statistical test has also been performed to prove that the increase in the performance of classifier post-feature selection is statistically significant. This machine learning-based study indicates that there exists discrimination between the microscopic brain images of the EMF-exposed drosophila and non-exposed drosophila. Graphical abstract Proposed Methodology for identification of radiofrequency electromagnetic radiation (RF-EMR) effect on the morphology of brain of Drosophila.

An evolutionary decomposition-based multi-objective feature selection for multi-label classification.

Data classification is a fundamental task in data mining. Within this field, the classification of multi-labeled data has been seriously considered in recent years. In such problems, each data entity can simultaneously belong to several categories. Multi-label classification is important because of many recent real-world applications in which each entity has more than one label. To improve the performance of multi-label classification, feature selection plays an important role. It involves identifying and removing irrelevant and redundant features that unnecessarily increase the dimensions of the search space for the classification problems. However, classification may fail with an extreme decrease in the number of relevant features. Thus, minimizing the number of features and maximizing the classification accuracy are two desirable but conflicting objectives in multi-label feature selection. In this article, we introduce a multi-objective optimization algorithm customized for selecting the features of multi-label data. The proposed algorithm is an enhanced variant of a decomposition-based multi-objective optimization approach, in which the multi-label feature selection problem is divided into single-objective subproblems that can be simultaneously solved using an evolutionary algorithm. This approach leads to accelerating the optimization process and finding more diverse feature subsets. The proposed method benefits from a local search operator to find better solutions for each subproblem. We also define a pool of genetic operators to generate new feature subsets based on old generation. To evaluate the performance of the proposed algorithm, we compare it with two other multi-objective feature selection approaches on eight real-world benchmark datasets that are commonly used for multi-label classification. The reported results of multi-objective method evaluation measures, such as hypervolume indicator and set coverage, illustrate an improvement in the results obtained by the proposed method. Moreover, the proposed method achieved better results in terms of classification accuracy with fewer features compared with state-of-the-art methods.

Variable-Size Cooperative Coevolutionary Particle Swarm Optimization for Feature Selection on High-Dimensional Data

Evolutionary feature selection (FS) methods face the challenge of “curse of dimensionality” when dealing with high-dimensional data. Focusing on this challenge, this article studies a variable-size cooperative coevolutionary particle swarm optimization algorithm (VS-CCPSO) for FS. The proposed algorithm employs the idea of “divide and conquer” in cooperative coevolutionary approach, but several new developed problem-guided operators/strategies make it more suitable for FS problems. First, a space division strategy based on the feature importance is presented, which can classify relevant features into the same subspace with a low computational cost. Following that, an adaptive adjustment mechanism of subswarm size is developed to maintain an appropriate size for each subswarm, with the purpose of saving computational cost on evaluating particles. Moreover, a particle deletion strategy based on fitness-guided binary clustering, and a particle generation strategy based on feature importance and crossover both are designed to ensure the quality of particles in the subswarms. We apply VS-CCPSO to 12 typical datasets and compare it with six state-of-the-art methods. The experimental results show that VS-CCPSO has the capability of obtaining good feature subsets, suggesting its competitiveness for tackling FS problems with high dimensionality.

Knowledge management overview of feature selection problem in high-dimensional financial data: cooperative co-evolution and MapReduce perspectives

The term “big data” characterizes the massive amounts of data generation by the advanced technologies in different domains using 4Vs – volume, velocity, variety, and veracity - to indicate the amount of data that can only be processed via computationally intensive analysis, the speed of their creation, the different types of data, and their accuracy. High-dimensional financial data, such as time-series and space-time data, contain a large number of features (variables) while having a small number of samples, which are used to measure various real-time business situations for financial organizations. Such datasets are normally noisy, and complex correlations may exist between their features, and many domains, including financial, lack the al analytic tools to mine the data for knowledge discovery because of the high-dimensionality. Feature selection is an optimization problem to find a minimal subset of relevant features that maximizes the classification accuracy and reduces the computations. Traditional statistical-based feature selection approaches are not adequate to deal with the curse of dimensionality associated with big data. Cooperative co-evolution, a meta-heuristic algorithm and a divide-and-conquer approach, decomposes high-dimensional problems into smaller sub-problems. Further, MapReduce, a programming model, offers a ready-to-use distributed, scalable, and fault-tolerant infrastructure for parallelizing the developed algorithm. This article presents a knowledge management overview of evolutionary feature selection approaches, state-of-the-art cooperative co-evolution and MapReduce-based feature selection techniques, and future research directions.

Multi-objective feature selection based on artificial bee colony: An acceleration approach with variable sample size

Due to the need to repeatedly call a classifier to evaluate individuals in the population, existing evolutionary feature selection algorithms have the disadvantage of high computational cost. In view of it, this paper studies a multi-objective feature selection framework based on sample reduction strategy and evolutionary algorithm, significantly reducing the computational cost of algorithm without affecting optimal results. In the framework, a selection strategy of representative samples, called K-means clustering based differential selection, and a ladder-like sample utilization strategy are proposed to reduce the size of samples used in the evolutionary process. Moreover, a fast multi-objective evolutionary feature selection algorithm, called FMABC-FS, is proposed by embedding an improved artificial bee colony algorithm based on the particle update model into the framework. By applying FMABC-FS to several typical UCI datasets, and comparing with three multi-objective feature selection algorithms, experimental results show that the proposed variable sample size strategy is more suitable to FMABC-FS, and FMABC-FS can obtain better feature subsets with much less running time than those comparison algorithms.

Dimensionality reduction in evolutionary algorithms-based feature selection for motor imagery brain-computer interface

For the classification of motor imagery brain-computer interface (BCI) based on electroencephalography (EEG), appropriate features are crucial to obtain a high classification accuracy. Considering the characteristics of the EEG signals, the time-frequency-space three-dimensional features are extracted. Due to a considerable number of the extracted features, the performance of a classifier will degrade. Therefore, it is necessary to implement feature selection. However, existing feature selection methods are easy to fall into a local optimum of a high-dimensional feature selection problem. In this paper, a dimensionality reduction mechanism (called DimReM) is proposed, which gradually reduces the dimension of the search space by removing some unimportant features. In principle, DimReM transforms a high-dimensional feature selection problem into a low-dimensional one. DimReM does not introduce any additional parameters and its implementation is simple. To verify its effectiveness, DimReM is combined with different evolutionary algorithms and different classifiers to select features on various kinds of datasets. Compared with evolutionary algorithms without dimensionality reduction, their augmented versions equipped with DimReM can find feature subsets with higher classification accuracies while smaller numbers of selected features.

Hybrid intelligent phishing website prediction using deep neural networks with genetic algorithm‐based feature selection and weighting

In recent years, the web phishing attack has become one of the most serious web security problems, in which the phishers can steal significant financial information about the internet users to carry out financial thefts. Several blacklist-based conventional phishing website detection methods are used to predict the phishing websites. However, numerous phishing websites are not predicted precisely by these blacklist-based conventional methods since many new phishing websites are constantly developed and launched on the Web over time. In this study, hybrid intelligent phishing website prediction using deep neural networks (DNNs) with evolutionary algorithm-based feature selection and weighting methods are suggested to enhance the phishing website prediction. In the proposed hybrid intelligent phishing website prediction approaches, the most influential features and the optimal weights of website features are heuristically identified with the genetic algorithm (GA) to help in increasing the accuracy of phishing website prediction. Accordingly, the website features selected and weighted by the GA are utilised to train DNNs to accurately predict the phishing websites. The experimental results demonstrated that the proposed hybrid intelligent phishing website prediction approaches achieved significantly higher classification accuracy, sensitivity, specificity, and geometric mean in phishing website prediction compared to those proposed in other studies.

Spam detection on social networks using cost-sensitive feature selection and ensemble-based regularized deep neural networks

Spam detection on social networks is increasingly important owing to the rapid growth of social network user base. Sophisticated spam filters must be developed to deal with this complex problem. Traditional machine learning approaches such as neural networks, support vector machines and Naïve Bayes classifiers are not effective enough to process and utilize complex features present in high-dimensional data on social network spam. Moreover, the traditional objective criteria of social network spam filters cannot cope with different costs assigned to type I and type II errors. To overcome these problems, here we propose a novel cost-sensitive approach to social network spam filtering. The proposed approach is composed of two stages. In the first stage, multi-objective evolutionary feature selection is used to minimize both the misclassification cost of the proposed model and the number of attributes necessary for spam filtering. Then, the approach uses cost-sensitive ensemble learning techniques with regularized deep neural networks as base learners. We demonstrate that this approach is effective for social network spam filtering on two benchmark datasets. We also show that the proposed approach outperforms other popular algorithms used in social network spam filtering, such as random forest, Naïve Bayes or support vector machines.

Indian language identification using time-frequency image textural descriptors and GWO-based feature selection

ABSTRACTAn ability to categorise and recognise a spoken language is an essential task in a multi-lingual society like India. Language identification (LID) is the process of identifying the language spoken by some unknown speaker using a given speech sample. In this article, textural descriptors extracted from spectrogram image and evolutionary feature selection is presented for Indian language identification. Language-specific long-term cues and prosodic information present in various frequency zones of the spectrogram image can efficiently modelled using textural descriptors. Firstly, an input audio sample is converted into a spectrogram visual representation which characterises the band of frequencies of a signal with respect to time. Then, completed local binary pattern (CLBP), local binary pattern histogram Fourier (LBPHF) and discrete Wavelet transform based texture descriptors are used to extract the features from the spectrogram image. Later, using grey wolf optimiser (GWO) feature selection, irrelevant and redundant features are removed, and only optimal features are selected from the dataset. GWO-based feature selection supports to construct the classification model with optimal features and the performance of the classifier is optimised. Finally, using the artificial neural network classifier and Indic-TTS database 96.9659% accuracy was obtained.

An evolutionary gravitational search-based feature selection

With recent advancements in data collection tools and the widespread use of intelligent information systems, a huge amount of data streams with lots of redundant, irrelevant, and noisy features are collected and a large number of features (attributes) should be processed. Therefore, there is a growing demand for developing efficient Feature Selection (FS) techniques. Gravitational Search algorithm (GSA) is a successful population-based metaheuristic inspired by Newton’s law of gravity. In this research, a novel GSA-based algorithm with evolutionary crossover and mutation operators is proposed to deal with feature selection (FS) tasks. As an NP-hard problem, FS finds an optimal subset of features from a given set. For the proposed wrapper FS method, both K-Nearest Neighbors (KNN) and Decision Tree (DT) classifiers are used as evaluators. Eighteen well-known UCI datasets are utilized to assess the performance of the proposed approaches. In order to verify the efficiency of proposed algorithms, the results are compared with some popular nature-inspired algorithms (i.e. Genetic Algorithm (GA), Particle Swarm Optimizer (PSO), and Grey Wolf Optimizer (GWO)). The extensive results and comparisons demonstrate the superiority of the proposed algorithm in solving FS problems.

Multiobjective Evolutionary Feature Selection for Fuzzy Classification

The interpretability of classification systems refers to the ability of these to express their behavior in a way that is easily understandable by a user. Interpretable classification models allow for external validation by an expert and, in certain disciplines, such as medicine or business, providing information about decision making is essential for ethical and human reasons. Fuzzy rule based classification systems are consolidated powerful classification tools based on fuzzy logic and designed to produce interpretable models; however, in presence of a large number of attributes, even rule-based models tend to be too complex to be easily interpreted. In this paper, we propose a novel multivariate feature selection method in which both search strategy and classifier are based on multiobjective evolutionary computation. We designed a set of experiments to establish an acceptable setting with respect to the number of evaluations required by the search strategy and by the classifier. We tested our strategy on a real-life dataset and compared the results against a wide range of feature selection methods that includes filter, wrapper, multivariate, and univariate methods, with deterministic and probabilistic search strategies, and with evaluators of diverse nature. Finally, the fuzzy rule based classification model obtained with the proposed method has been evaluated with standard performance metrics and compared with other well-known fuzzy rule based classifiers. We have used two real-life datasets extracted from a contact center; in one case, with the proposed method, we obtained an accuracy of 0.7857 with eight rules, while the best fuzzy classifier compared obtained 0.7679 with eight rules, and in the second case, we obtained an accuracy of 0.7403 with five rules, while the best fuzzy classifier compared obtained 0.6364 with four rules.