Multi-objective Evolutionary Feature Selection Research Articles

Surrogate-assisted multi-objective evolutionary feature selection of generation-based fixed evolution control for time series forecasting with LSTM networks

Surrogate-assisted multi-objective evolutionary algorithms are powerful techniques to solve computationally expensive multi-objective optimization problems. In this paper, we propose a direct fitness replacement method with generation-based fixed evolution control to implement a multi-objective evolutionary algorithm that uses a surrogate model for wrapper-type feature selection, where long short-term memory is established as the learning algorithm. The importance of the work and its benefits lie in the need to reduce the excessive computational time required by conventional wrapper-type feature selection methods based on multi-objective evolutionary algorithms and LSTM networks, maintaining or improving the predictive capacity of the models. We analyze the use of incremental learning to update the surrogate model, in comparison with the conventional non-incremental learning approach. We applied these methods in real-life time series forecasting of air quality, indoor temperature in a smart building and oil temperature in electricity transformers. Multi-step ahead predictions of the forecast models obtained with different meta-learners of the surrogate model were compared by using the Diebold–Mariano statistical test on a multi-criteria performance metric. The proposed method outperformed other approaches for feature selection including, among others, methods based on surrogate-assisted multi-objective evolutionary algorithms developed by the authors in previous research, other surrogate-assisted deterministic methods for feature selection and the conventional wrapper-type feature selection method based on LSTM, improving the prediction on test dataset by 23.98%, 34.61% and 13.77%, respectively.

A Dynamic Tasking-Based Evolutionary Algorithm for Bi-Objective Feature Selection

Feature selection in classification is a complex optimization problem that cannot be solved in polynomial time. Bi-objective feature selection, aiming to minimize both selected features and classification errors, is challenging due to the conflict between objectives, while one of the most effective ways to tackle this is to use multi-objective evolutionary algorithms. However, very few of these have ever reflected an evolutionary multi-tasking framework, despite the implicit parallelism offered by the population-based search characteristic. In this paper, a dynamic multi-tasking-based multi-objective evolutionary algorithm (termed DTEA) is proposed for handling bi-objective feature selection in classification, which is not only suitable for datasets with relatively lower dimensionality of features, but is also suitable for datasets with relatively higher dimensionality of features. The role and influence of multi-tasking on multi-objective evolutionary feature selection were studied, and a dynamic tasking mechanism is proposed to self-adaptively assign multiple evolutionary search tasks by intermittently analyzing the population behaviors. The efficacy of DTEA is tested on 20 classification datasets and compared with seven state-of-the-art evolutionary algorithms. A component contribution analysis was also conducted by comparing DTEA with its three variants. The empirical results show that the dynamic-tasking mechanism works efficiently and enables DTEA to outperform other algorithms on most datasets in terms of both optimization and classification.

Surrogate-Assisted and Filter-Based Multiobjective Evolutionary Feature Selection for Deep Learning.

Feature selection (FS) for deep learning prediction models is a difficult topic for researchers to tackle. Most of the approaches proposed in the literature consist of embedded methods through the use of hidden layers added to the neural network architecture that modify the weights of the units associated with each input attribute so that the worst attributes have less weight in the learning process. Other approaches used for deep learning are filter methods, which are independent of the learning algorithm, which can limit the precision of the prediction model. Wrapper methods are impractical with deep learning due to their high computational cost. In this article, we propose new attribute subset evaluation FS methods for deep learning of the wrapper, filter and wrapper-filter hybrid types, where multiobjective and many-objective evolutionary algorithms are used as search strategies. A novel surrogate-assisted approach is used to reduce the high computational cost of the wrapper-type objective function, while the filter-type objective functions are based on correlation and an adaptation of the reliefF algorithm. The proposed techniques have been applied in a time series forecasting problem of air quality in the Spanish south-east and an indoor temperature forecasting problem in a domotic house, with promising results compared to other FS techniques used in the literature.

Intrusion detection in cyber–physical environment using hybrid Naïve Bayes—Decision table and multi-objective evolutionary feature selection

Researchers are motivated to build effective Intrusion Detection Systems because of the implications of malicious actions in computing, communication, and cyber–physical systems (IDSs). In order to develop signature-based intrusion detection techniques that are suitable for use in cyber–physical environments, state-of-the-art supervised learning algorithms are devised. The main contribution of this research is the introduction of a signature-based intrusion detection model that is based on a hybrid Decision Table and Naive Bayes technique. In addition, the contribution of the suggested method is evaluated by comparing it to the existing literature in the field. In the preprocessing stage, Multi-Objective Evolutionary Feature Selection (MOEFS) feature selection has been used to select only five attack features from the recent CICIDS017 dataset. Keeping in view the class imbalance nature of CICIDS2017 dataset, adequate attack samples has been selected with more weightage to the attack classes having a smaller number of instances in the dataset. A hybrid of Decision Table and Naive Bayes models were combined to train and detect intrusions. Detection of botnets, port scans, Denial of Service (DoS)/Distributed Denial of Service (DDoS) attacks, such as Golden-Eye, Hulk, Slow httptest, slowloris, Heartbleed, Brute Force attacks, such as Patator (FTP), Patator (SSH), and Web attacks such as Infiltration, Web Brute Force, SQL Injection, and XSS, are all successfully detected by the proposed hybrid detection model. The proposed approach shows an accuracy of 96.8% using five features of CICIDS2017, which is higher than the accuracy of methods discussed in the literatures.

Mining and Interpretation of Critical Aspects of Infant Health Status Using Multi-Objective Evolutionary Feature Selection Approaches

The rate of infant mortality (IMR) in a population under one year of age is a marker for infant mortality. It is a major sensitive marker of a community’s overall physical health. Protecting the lives of newborns has become a challenging issue in public health, development programs, and humanitarian initiatives. Almost 10.1% infants died in the United States of America (USA) in 2021. Therefore, this paper aims to extract and understand the various influential factors causing infant deaths in the USA. A crowding distance-based multi-objective ant lion optimization (MOALO-CD) is proposed here with statistical evidence for feature selection. The proposed technique is compared with competitive metaheuristic models such as multi-objective genetic algorithm based on crowding distance (MOGA-CD), multi-objective filter approaches, and recursive feature elimination. Various machine learning classifiers are applied to the selected feature subset obtained from MOALO-CD on the USA’s infant dataset. Extensive experimental results indicate that the proposed model outperforms the existing metaheuristic approaches in terms of Generational Distance, Inverted Generational Distance, Spread, and Hyper volume. Also, the comparative analysis of various machine learning models reveals that random forest achieves significantly better performance on the feature subset obtained from MOALO-CD.

Multi-objective feature selection based on artificial bee colony: An acceleration approach with variable sample size

Due to the need to repeatedly call a classifier to evaluate individuals in the population, existing evolutionary feature selection algorithms have the disadvantage of high computational cost. In view of it, this paper studies a multi-objective feature selection framework based on sample reduction strategy and evolutionary algorithm, significantly reducing the computational cost of algorithm without affecting optimal results. In the framework, a selection strategy of representative samples, called K-means clustering based differential selection, and a ladder-like sample utilization strategy are proposed to reduce the size of samples used in the evolutionary process. Moreover, a fast multi-objective evolutionary feature selection algorithm, called FMABC-FS, is proposed by embedding an improved artificial bee colony algorithm based on the particle update model into the framework. By applying FMABC-FS to several typical UCI datasets, and comparing with three multi-objective feature selection algorithms, experimental results show that the proposed variable sample size strategy is more suitable to FMABC-FS, and FMABC-FS can obtain better feature subsets with much less running time than those comparison algorithms.

Spam detection on social networks using cost-sensitive feature selection and ensemble-based regularized deep neural networks

Spam detection on social networks is increasingly important owing to the rapid growth of social network user base. Sophisticated spam filters must be developed to deal with this complex problem. Traditional machine learning approaches such as neural networks, support vector machines and Naïve Bayes classifiers are not effective enough to process and utilize complex features present in high-dimensional data on social network spam. Moreover, the traditional objective criteria of social network spam filters cannot cope with different costs assigned to type I and type II errors. To overcome these problems, here we propose a novel cost-sensitive approach to social network spam filtering. The proposed approach is composed of two stages. In the first stage, multi-objective evolutionary feature selection is used to minimize both the misclassification cost of the proposed model and the number of attributes necessary for spam filtering. Then, the approach uses cost-sensitive ensemble learning techniques with regularized deep neural networks as base learners. We demonstrate that this approach is effective for social network spam filtering on two benchmark datasets. We also show that the proposed approach outperforms other popular algorithms used in social network spam filtering, such as random forest, Naïve Bayes or support vector machines.

Multiobjective Evolutionary Feature Selection for Fuzzy Classification

The interpretability of classification systems refers to the ability of these to express their behavior in a way that is easily understandable by a user. Interpretable classification models allow for external validation by an expert and, in certain disciplines, such as medicine or business, providing information about decision making is essential for ethical and human reasons. Fuzzy rule based classification systems are consolidated powerful classification tools based on fuzzy logic and designed to produce interpretable models; however, in presence of a large number of attributes, even rule-based models tend to be too complex to be easily interpreted. In this paper, we propose a novel multivariate feature selection method in which both search strategy and classifier are based on multiobjective evolutionary computation. We designed a set of experiments to establish an acceptable setting with respect to the number of evaluations required by the search strategy and by the classifier. We tested our strategy on a real-life dataset and compared the results against a wide range of feature selection methods that includes filter, wrapper, multivariate, and univariate methods, with deterministic and probabilistic search strategies, and with evaluators of diverse nature. Finally, the fuzzy rule based classification model obtained with the proposed method has been evaluated with standard performance metrics and compared with other well-known fuzzy rule based classifiers. We have used two real-life datasets extracted from a contact center; in one case, with the proposed method, we obtained an accuracy of 0.7857 with eight rules, while the best fuzzy classifier compared obtained 0.7679 with eight rules, and in the second case, we obtained an accuracy of 0.7403 with five rules, while the best fuzzy classifier compared obtained 0.6364 with four rules.

Multiobjective evolutionary feature selection and fuzzy classification of contact centre data

AbstractIn this work, a data set describing phone interactions arising in a multichannel and multiskill contact centre is considered with the aim of classifying inbound sessions into those that will be eventually managed by an agent and those that, instead, will be abandoned before. More precisely, the goal of the work is to extract interpretable pieces of information that allow us to predict whether a user will or will not abandon a call, which may turn out to be very useful for the purpose of contact centre managing. To this end, the performance of two well‐known, state‐of‐the‐art evolutionary algorithms for feature selection (evolutionary nondominated radial slots based algorithm and nondominated sorted genetic algorithm) is compared for the task of feature selection, under the criteria of accuracy and cardinality of the selection, as well as for the task of fuzzy rule extraction, under the criteria of interpretability, accuracy, and hypervolume test. The best obtained fuzzy classifier, chosen after a decision making process, is validated and interpreted by domain experts.

Two-stage consumer credit risk modelling using heterogeneous ensemble learning

Modelling consumer credit risk is a crucial task for banks and non-bank financial institutions to support decision-making on granting loans. To model the overall credit risk of a consumer loan in terms of expected loss (EL), three key credit risk parameters must be estimated: probability of default (PD), loss given default (LGD) and exposure at default (EAD). Research to date has tended to model these parameters separately. Moreover, a neglected area in the field of LGD/EAD modelling is the application of ensemble learning, which by benefitting from diverse base learners reduces the over-fitting problem and enables modelling diverse risk profiles of defaulted loans. To overcome these problems, this paper proposes a two-stage credit risk model that integrates (1) class-imbalanced ensemble learning for predicting PD (credit scoring), and (2) an EAD prediction using a regression ensemble. Furthermore, multi-objective evolutionary feature selection is used to minimize both the misclassification cost (root mean squared error) of the PD and EAD models and the number of attributes necessary for modelling. For this task, we propose a misclassification cost metric suitable for consumer loans with fixed exposure because it combines opportunity cost and LGD. We show that the proposed credit risk model is not only more effective than single-stage credit risk models but also outperforms state-of-the-art methods used to model credit risk in terms of prediction and economic performance.

A methodology for evaluating multi-objective evolutionary feature selection for classification in the context of virtual screening

Virtual screening (VS) methods have been shown to increase success rates in many drug discovery campaigns, when they complement experimental approaches, such as high-throughput screening methods or classical medicinal chemistry approaches. Nevertheless, predictive capability of VS is not yet optimal, mainly due to limitations in the underlying physical principles describing drug binding phenomena. One approach that can improve VS methods is the aid of machine learning methods. When enough experimental data are available to train such methods, predictive capability can considerably increase. We show in this research work how a multi-objective evolutionary search strategy for feature selection, which can provide with small and accurate decision trees that can be very easily understood by chemists, can drastically increase the applicability and predictive ability of these techniques and therefore aid considerable in the drug discovery problem. With the proposed methodology, we find classification models with accuracy between 0.9934 and 1.00 and area under ROC between 0.96 and 1.00 evaluated in full training sets, and accuracy between 0.9849 and 0.9940 and area under ROC between 0.89 and 0.93 evaluated with tenfold cross-validation over 30 iterations, while substantially reducing the model size.

Multi-objective evolutionary feature selection for online sales forecasting

Sales forecasting uses historical sales figures, in association with products characteristics and peculiarities, to predict short-term or long-term future performance in a business, and it can be used to derive sound financial and business plans. By using publicly available data, we build an accurate regression model for online sales forecasting obtained via a novel feature selection methodology composed by the application of the multi-objective evolutionary algorithm ENORA (Evolutionary NOn-dominated Radial slots based Algorithm) as search strategy in a wrapper method driven by the well-known regression model learner Random Forest. Our proposal integrates feature selection for regression, model evaluation, and decision making, in order to choose the most satisfactory model according to an a posteriori process in a multi-objective context. We test and compare the performances of ENORA as multi-objective evolutionary search strategy against a standard multi-objective evolutionary search strategy such as NSGA-II (Non-dominated Sorted Genetic Algorithm), against a classical backward search strategy such as RFE (Recursive Feature Elimination), and against the original data set.

An SVM-Wrapped Multiobjective Evolutionary Feature Selection Approach for Identifying Cancer-MicroRNA Markers

MicroRNAs (miRNAs), have been shown to play important roles in gene regulation and various biological processes. Recent studies have revealed that abnormal expression of some specific miRNAs often results in the development of cancer. Microarray datasets containing the expression profiles of several miRNAs are being used for identification of miRNAs which are differentially expressed in normal and malignant tissue samples. In this article, a multiobjective feature selection approach is proposed for this purpose. The proposed method uses Genetic Algorithm for multiobjective optimization and support vector machine (SVM) classifier as a wrapper for evaluating the chromosomes that encode feature subsets. The performance has been demonstrated on real-life miRNA datasets for and the identified miRNA markers are reported. Moreover biological significance tests have been carried out for the obtained markers.

Multi-objective evolutionary feature selection for instrument recognition in polyphonic audio mixtures

Instrument recognition is one of the music information retrieval research topics. This task becomes very challenging if several instruments are played simultaneously because of their varying physical characteristics: inharmonic attack noise, energy development during attack---decay---sustain---release envelope or overtone distribution. In our framework, we treat instrument detection as a machine-learning task based on a large amount of preprocessed audio features with target to build classification models. Since classification algorithms are very sensitive to feature input and the optimal feature set differs from instrument to instrument, we propose to run a multi-objective feature selection procedure before building of classification models. Two objectives are considered for evaluation: classification mean-squared error and feature rate (smaller amount of features stands for reduced costs and decreased risk of overfitting). The analysis of the extensive experimental study confirms that application of an evolutionary multi-objective algorithm is a good choice to optimize feature selection for music instrument identification.