Multi-objective Feature Selection Research Articles

Computational Cost Reduction in Multi-Objective Feature Selection Using Permutational-Based Differential Evolution

Feature selection is a preprocessing step in machine learning that aims to reduce dimensionality and improve performance. The approaches for feature selection are often classified according to the evaluation of a subset of features as filter, wrapper, and embedded approaches. The high performance of wrapper approaches for feature selection is associated at the same time with the disadvantage of high computational cost. Cost-reduction mechanisms for feature selection have been proposed in the literature, where competitive performance is achieved more efficiently. This work applies the simple and effective resource-saving mechanisms of the fixed and incremental sampling fraction strategies with memory to avoid repeated evaluations in multi-objective permutational-based differential evolution for feature selection. The selected multi-objective approach is an extension of the DE-FSPM algorithm with the selection mechanism of the GDE3 algorithm. The results showed high resource savings, especially in computational time and the number of evaluations required for the search process. Nonetheless, it was also detected that the algorithm’s performance was diminished. Therefore, the results reported in the literature on the effectiveness of the strategies for cost reduction in single-objective feature selection were only partially sustained in multi-objective feature selection.

A Knowledge-Guided Competitive Co-Evolutionary Algorithm for Feature Selection

In real-world applications, feature selection is crucial for enhancing the performance of data science and machine learning models. Typically, feature selection is a complex combinatorial optimization problem and a multi-objective optimization problem. Its primary goals are to reduce the dimensionality of the dataset and enhance the performance of machine learning algorithms. The selection of features in high-dimensional datasets is challenging due to the intricate relationships between features, which pose significant challenges to the performance and computational efficiency of algorithms. This paper introduces a Knowledge-Guided Competitive Co-Evolutionary Algorithm (KCCEA) for feature selection, especially for high-dimensional features. In the proposed algorithm, we make improvements to the foundational dominance-based multi-objective evolutionary algorithm in two aspects. First, the use of feature correlation as knowledge to guide evolution enhances the search speed and quality of traditional multi-objective evolutionary algorithm solutions. Second, a dynamically allocated competitive–cooperative evolutionary mechanism is proposed, integrating the improved knowledge-guided evolution with traditional evolutionary algorithms, further enhancing the search efficiency and diversity of solutions. Through rigorous empirical testing on various datasets, the KCCEA demonstrates superior performance compared to basic multi-objective evolutionary algorithms, providing effective solutions to multi-objective feature selection problems while enhancing the interpretability and effectiveness of prediction models.

Multiple reference points-based multi-objective feature selection for multi-label learning

Multiple reference points-based multi-objective feature selection for multi-label learning

WSBCV: A data-driven cross-version defect model via multi-objective optimization and incremental representation learning

Cross-version defect prediction (CVDP) refers to training an excellent model on tagged defect data in a previously released project version, and then performing defect prediction on an unlabeled instance module in the current version. Nevertheless, the complicated internal intrinsic construction hidden behind the code defects makes it difficult for the previous cross-version defect models to capture more discriminative software features, and seriously restrains the CVDP performance. In this study, we propose an intelligent data-driven CVDP model named WSBCV based on multi-objective optimization and incremental representation learning. We firstly leverage an advanced deep generation adversarial network – WGAN-GP (Wasserstein GAN with Gradient Penalty) to perform data augmentation, including balancing defect classes and synthesizing abundant training instances. Secondly, a multi-objective SPEA/R (Strength Pareto-based Evolutionary Algorithm / Reference) feature selection optimization method is built to effectively search the fewest representative feature subsets while achieving the minimum error. Finally, a powerful defect predictor for CVDP based on the BLS (Broad Learning System) with incremental learning is built to learn excellent feature representations and achieve incremental online model update quickly. Experimental results across 32 cross-version pairs from 45 version demonstrate that the proposed SPEA/R, BLS and WSBCV all have statistically significant difference advantages compared to ten multi-objective feature selection approaches, six defect predictors and two CVDP models, respectively.

MOFS-REPLS: A large-scale multi-objective feature selection algorithm based on real-valued encoding and preference leadership strategy

Multi-objective feature selection (MOFS) has emerged as a crucial step in constructing efficient machine-learning models. While multi-objective evolutionary algorithms often yield satisfactory sub-optimal solutions, enhancing these algorithms' global optimization capacity remains a central challenge in the field of engineering optimization. To improve the quality of solutions to problems, there is an imperative need for an algorithm with superior optimization capability. This study introduces a large-scale MOFS algorithm based on real-valued encoding and a preference leadership strategy, named MOFS-REPLS, which aims to address the challenge of large-scale sparse feature selection (FS). First, we propose a novel encoding scheme to facilitate broader population exploration. During the population initialization phase, we integrate a ReliefF-guided approach with roulette wheel selection to create the initial population. Second, we introduce a preference leadership strategy that directs individuals toward their respective areas in the Pareto front. Finally, we devise an adaptive learning strategy incorporating ReliefF-guided methods to steer the evolution of the population, thereby mitigating performance deficiencies due to the algorithm's lack of prior knowledge. MOFS-REPLS employs a dual-archive mechanism to maintain diversity within the algorithm and to preserve non-dominated solutions for further exploration. Through experimental assessment using 20 UCI datasets and 10 state-of-the-art algorithms, we demonstrate the effectiveness of MOFS-REPLS. The results show that our proposed algorithm not only maintains high accuracy but also selects a smaller, more relevant set of features, significantly outperforming other FS algorithms in comparison.

A Multi-objective Feature Selection Method Considering the Interaction Between Features

A Multi-objective Feature Selection Method Considering the Interaction Between Features

Evolutionary Multi-Objective Feature Selection Algorithms on Multiple Smart Sustainable Community Indicator Datasets

The conceptual fusion of smart city and sustainability indicators has inspired the emergence of the smart sustainable city (SSC). Given the early stage of development in this field, most SSC studies have been primarily theoretical. Notably, existing empirical studies have overlooked the crucial aspect of feature engineering in the context of SSC, despite its significance in advancing SSC initiatives. This paper introduces an approach advocating for feature subset selection to maximize prediction accuracy and minimize computational time across diverse SSC indicators encompassing socio-cultural, economic, environmental, and governance categories. The study systematically collected multiple datasets on SSC indicators, covering various themes within the SSC framework. Employing six carefully chosen multiple-objective evolutionary feature selection algorithms, the research selected feature subsets. These subsets were then utilized in modeling algorithms to predict SSC indicators. The proposal enhanced prediction accuracy for life expectancy, online shopping intentions, energy consumption, air quality, water quality, and traffic flow for a smart and sustainable city by minimizing the subset features. The findings underscore the efficacy of feature subset selection in generating minimal features, thereby enhancing both prediction accuracy and computational efficiency in the realm of SSC indicators. For researchers aiming to develop sustainable systems for real-time data monitoring within SSC, the identified subset features offer a valuable resource, negating the necessity for extensive dataset collection. The provided SSC datasets are anticipated to serve as a catalyst, inspiring researchers to embark on empirical studies that explore SSC development from diverse perspectives, ultimately contributing to a more profound understanding of the SSC dynamics.

A Multiform Optimization Framework for Multi-objective Feature Selection in Classification

Feature selection in machine learning as a key data processing technique has two conflicting goals: minimizing the classification error rate and minimizing the number of features selected. However, most of the existing multi-objective feature selection methods face the problems of easily falling into local optima and slow convergence by virtue of their problem characteristics, such as partially conflicting objectives and highly discontinuous Pareto fronts. To solve these problems, this article proposes a multiform optimization framework to solve a multi-objective feature selection task together with several auxiliary single-objective feature selection tasks in a multitask environment. The proposed framework uses the problem-solving experience of single-objective tasks to assist the multi-objective feature selection task in exploring more promising regions and accelerating the convergence speed. Specifically, a knowledge transfer strategy based on the search experience of different tasks is developed to accomplish multiform optimization. In addition, a diversity enhancement mechanism is presented to improve search ability in promising decision space areas by considering historical information about the population. In most cases, the experiment results on 27 datasets demonstrate that the proposed technique can uncover more diversified feature subsets on the Pareto front in less time than existing state-of-the-art methods.

Solving Multi-objective Feature Selection Problems in Classification via Problem Reformulation and Duplication Handling

Reducing the number of selected features and improving the classification performance are two major objectives in feature selection, which can be viewed as a multi-objective optimization problem. Multi-objective feature selection in classification has its unique characteristics, such as it has a strong preference for the classification performance over the number of selected features. Besides, solution duplication often appears in both the search and the objective spaces, which degenerates the diversity and results in the premature convergence of the population. To deal with the above issues, in this paper, during the evolutionary training process, a multi-objective feature selection problem is reformulated and solved as a constrained multi-objective optimization problem, which adds a constraint on the classification performance for each solution (e.g., feature subset) according to the distribution of nondominated solutions, with the aim of selecting promising feature subsets that contain more informative and strongly relevant features, which are beneficial to improve the classification performance. Furthermore, based on the distribution of feature subsets in the objective space and their similarity in the search space, a duplication analysis and handling method is proposed to enhance the diversity of the population. Experimental results demonstrate that the proposed method outperforms six state-of-the-art algorithms and is computationally efficient on 18 classification datasets.

Balancing Different Optimization Difficulty Between Objectives in Multi-Objective Feature Selection

Balancing Different Optimization Difficulty Between Objectives in Multi-Objective Feature Selection

An Information-Based Elite-Guided Evolutionary Algorithm for Multi-Objective Feature Selection

An Information-Based Elite-Guided Evolutionary Algorithm for Multi-Objective Feature Selection

Learning to Preselection: A Filter-Based Performance Predictor for Multiobjective Feature Selection in Classification

Learning to Preselection: A Filter-Based Performance Predictor for Multiobjective Feature Selection in Classification

A Survey on Evolutionary Multiobjective Feature Selection in Classification: Approaches, Applications, and Challenges

A Survey on Evolutionary Multiobjective Feature Selection in Classification: Approaches, Applications, and Challenges

Benefiting From Single-Objective Feature Selection to Multiobjective Feature Selection: A Multiform Approach.

Evolutionary multiobjective feature selection (FS) has gained increasing attention in recent years. However, it still faces some challenges, for example, the frequently appeared duplicated solutions in either the search space or the objective space lead to the diversity loss of the population, and the huge search space results in the low search efficiency of the algorithm. Minimizing the number of selected features and maximizing the classification performance are two major objectives in FS. Usually, the fitness function of a single-objective FS problem linearly aggregates these two objectives through a weighted sum method. Given a predefined direction (weight) vector, the single-objective FS task can explore the specified direction or area extensively. Different direction vectors result in different search directions in the objective space. Motivated by this, this article proposes a multiform framework, which solves a multiobjective FS task combined with its auxiliary single-objective FS tasks in a multitask environment. By setting different direction vectors, promising feature subsets from single-objective FS tasks can be utilized, to boost the evolutionary search of the multiobjective FS task. By comparing with five classical and state-of-the-art multiobjective evolutionary algorithms, as well as four well-performing FS algorithms, the effectiveness and efficiency of the proposed method are verified via extensive experiments on 18 classification datasets. Furthermore, the effectiveness of the proposed method is also investigated in a noisy environment.

A novel enterprise credit risk rating model with the perspective of multiple regulatory demands

With the number of market participants rising, the issue of contradiction between heavy supervision task and insufficient supervision resource have become increasingly urgent for regulatory authorities to consider. The credit risk rating of enterprises is therefore introduced to improve the efficiency of supervision. By analyzing the purposes of enterprise credit risk rating and the multiple demands for regulatory authorities, this paper offers a multi-classification ensemble model with hybrid decomposition strategy and two-stage multi-objective feature selection. First, an adaptive resampling technique is introduced into the hybrid decomposition strategy for multi-classification, so to appropriately allocate supervision resources in each credit risk class. Then, a two-stage multi-objective feature selection method is developed to combine the various demands of regulatory authorities for feature selection. In the first stage, results of multiple Filter feature selection methods are integrated as output to improve the performance of feature information; In the second stage, the feature information from the first stage is used as input to guide the Wrapper feature selection, which is based on the Improved Multi-objective Ant Colony Optimization. Finally, a real listed enterprise credit dataset of China is used to evaluate the performance of each module and the entire model. The empirical results show that the design of hybrid decomposition strategy and multi-objective feature selection can not only deal with the class-imbalanced problem during the decomposition process, but also take into account the balance of each demand well. More importantly, a desirable framework is set in this paper to simplify the process from imbalanced multi-classification to a balanced binary one.

A Novel Multi-Objective Based Feature Selection Method for Response Prediction

Accurate response prediction is essential towards personalized treatment in radiation therapy. Excessive imaging features, extracted from medical images, pose a great challenge in radiomic analyses. Feature selection is an essential step to remove redundant and irrelevant features for model construction. We proposed a novel multi-objective based radiomic feature selection method (MRMOPSO), where the number of features, sensitivity, and specificity are jointly considered as optimization objectives for feature selection. The MRMOPSO innovated by three aspects: 1) Fisher score initialize the feature population to speed up the convergence; 2) Min-redundancy particle generation operations to reduce the redundancy between radiomic features, a truncation strategy was also introduced; 3) Particle selection operation guided by elitism strategies to improve local search ability of the algorithm. We evaluated the effectiveness of the proposed MRMOPSO method using a cohort of oropharyngeal cancer patients from The Cancer Imaging Archive (TCIA). 357 patients were used for model training and additional 64 patients were used for independent evaluation. The proposed methods were compared with (a) classical feature selection methods, i.e., Lasso, minimal-redundancy-maximal-relevance criterion (mRMR), F-score, and mutual information (MI), (b) single-objective feature selection methods, i.e., genetic algorithm (GA), particle swarm optimization algorithm (PSO) and (c) multi-objective feature selection methods, i.e., multiple objective particle swarm optimization (MOPSO), nondominated sorting genetic algorithm II (NSGA II). The other feature selection methods yielded AUCs, sensitivity, specificity of (0.48-0.71), (0.49-0.86), (0.33-0.67), respectively. The MRMOPSO achieved significantly highly AUC of 0.84 with smaller number of selected features on the independent dataset (Table 1). Additionally, the MRMOPSO remarkably improved the sensitivity (0.81), specificity (0.81) and achieved an excellent balance between sensitively and specificity. We demonstrated a novel multi-objective based radiomic feature selection method. The proposed algorithm effectively reduced feature dimension, and achieved superior AUC with simultaneous improved sensitivity and specificity, for radiomic response prediction.

Information-Theory-based Nondominated Sorting Ant Colony Optimization for Multiobjective Feature Selection in Classification.

Feature selection (FS) has received significant attention since the use of a well-selected subset of features may achieve better classification performance than that of full features in many real-world applications. It can be considered as a multiobjective optimization consisting of two objectives: 1) minimizing the number of selected features and 2) maximizing classification performance. Ant colony optimization (ACO) has shown its effectiveness in FS due to its problem-guided search operator and flexible graph representation. However, there lacks an effective ACO-based approach for multiobjective FS to handle the problematic characteristics originated from the feature interactions and highly discontinuous Pareto fronts. This article presents an Information-theory-based Nondominated Sorting ACO (called INSA) to solve the aforementioned difficulties. First, the probabilistic function in ACO is modified based on the information theory to identify the importance of features; second, a new ACO strategy is designed to construct solutions; and third, a novel pheromone updating strategy is devised to ensure the high diversity of tradeoff solutions. INSA's performance is compared with four machine-learning-based methods, four representative single-objective evolutionary algorithms, and six state-of-the-art multiobjective ones on 13 benchmark classification datasets, which consist of both low and high-dimensional samples. The empirical results verify that INSA is able to obtain solutions with better classification performance using features whose count is similar to or less than those obtained by its peers.

Enhanced SparseEA for large-scale multi-objective feature selection problems

Large-scale multi-objective feature selection problems are widely existing in the fields of text classification, image processing, and biological omics. Numerous features usually mean more correlation and redundancy between features, so effective features are usually sparse. SparseEA is an evolutionary algorithm for solving Large-scale Sparse Multi-objective Optimization Problems (i.e., most decision variables of the optimal solutions are zero). It determines feature Scores by calculating the fitness of individual features, which does not reflect the correlation between features well. In this manuscript, ReliefF was used to calculate the weights of features, with unimportant features being removed first. Then combine the weights calculated by ReliefF with Scores of SparseEA to guide the evolution process. Moreover, the Scores of features remain constant throughout all runs in SparseEA. Therefore, the fitness values of excellent and poor individuals in each iteration are used to update the Scores. In addition, difference operators of Differential Evolution are introduced into SparseEA to increase the diversity of solutions and help the algorithm jump out of the local optimal solution. Comparative experiments are performed on large-scale datasets selected from scikit-feature repository. The results show that the proposed algorithm is superior to the original SparseEA and the state-of-the-art algorithms.

Prediction of stock price movement using an improved NSGA-II-RF algorithm with a three-stage feature engineering process.

Prediction of stock price has been a hot topic in artificial intelligence field. Computational intelligent methods such as machine learning or deep learning are explored in the prediction system in recent years. However, making accurate predictions of stock price direction is still a big challenge because stock prices are affected by nonlinear, nonstationary, and high dimensional features. In previous works, feature engineering was overlooked. How to select the optimal feature sets that affect stock price is a prominent solution. Hence, our motivation for this article is to propose an improved many-objective optimization algorithm integrating random forest (I-NSGA-II-RF) algorithm with a three-stage feature engineering process in order to decrease the computational complexity and improve the accuracy of prediction system. Maximizing accuracy and minimizing the optimal solution set are the optimization directions of the model in this study. The integrated information initialization population of two filtered feature selection methods is used to optimize the I-NSGA-II algorithm, using multiple chromosome hybrid coding to synchronously select features and optimize model parameters. Finally, the selected feature subset and parameters are input to the RF for training, prediction, and iterative optimization. Experimental results show that the I-NSGA-II-RF algorithm has the highest average accuracy, the smallest optimal solution set, and the shortest running time compared to the unmodified multi-objective feature selection algorithm and the single target feature selection algorithm. Compared to the deep learning model, this model has interpretability, higher accuracy, and less running time.

Three-stage multi-objective feature selection with distributed ensemble machine and deep learning for processing of complex and large datasets

The need for analytical solutions using machine learning (ML) and deep learning (DL) has risen in this era of industry-wide informed decision-making. Data is now readily available and moving at fast speeds due to the maturation of data processing technologies and the development of new technologies in the Big Data, streaming, and IoT sectors. As a result, a transformation is required in how analytical systems handle data. A novel approach to learning, distributed learning, maybe a wise choice for managing huge and high-speed data while overcoming the difficulties presented by noisy and dynamic data sources. Data scientists can considerably increase their productivity using distributed learning by 1) distributing concurrent experiments across several devices and 2) significantly reducing training time by distributing a network's training across several devices. A learning strategy takes a long time, and the model's performance suffers in large datasets due to redundant features and the dimensionality curse. But most feature selection methods are unstable, selecting different subsets of features for different training datasets, resulting in varying levels of classification accuracy. A novel ensemble machine and deep learning technologies are also needed to improve accuracy in distributed learning. To analyze complex and large datasets, this paper proposed a three-stage multi-objective feature selection (TMFS) with distributed ensemble machine and deep learning (DEMDL). The proposed TMFS technique selects the best feature subset by combining multiple feature subsets. To achieve these goals, the TMFS technique employs 5 feature selection strategies in 3 stages: correlation coefficient, Fisher score, information gain, mean absolute deviation, and min-max normalization. SVM is used for machine learning, and Dl4jMlpCassifier is used for deep learning. This SVM and Dl4jMlpClassifier are combined in DEMDL using a stacking classifier. SVM and Dl4jMlpClassifier are utilized as base classifiers in this stacking classifier, and the RIPPER classifier is employed as the meta-classifier. The experimental finding demonstrates that compared to existing classification techniques, the TMFS with DEMDL technique offers higher accuracy, Precision, Recall, and F-measure.