Existing Feature Selection Algorithms Research Articles

Improve the Prediction in the Digital Era: Causal Feature Selection with Minimum Redundancy

Effective decision-making in complex environments requires discerning the relevant from the irrelevant, a challenge that becomes pronounced with large multivariate time-series data. However, existing feature selection algorithms often suffer from complexity and a lack of interpretability, making it difficult for decision-makers to extract value, manage risks, and adhere to compliance regulations in an easily explainable manner. To address these challenges, we propose a novel causality-based feature selection technique that embeds an explainable unsupervised feature selection algorithm. We refer to our proposed method as Causal Feature Selection with Minimum Redundancy (CFSMR). Our method yields a minimum viable feature set without compromising model performance while ensuring interpretability. We conduct a large-scale experimental study to compare the proposed technique with conventional feature selection techniques. Our results demonstrate that our proposed method outperforms or performs on par with existing techniques, making it a promising approach for decision-makers seeking an effective and interpretable feature selection method.

Non-convex feature selection based on feature correlation representation and dual manifold optimization

The existing feature selection algorithms often utilize local structure of data, but do not fully mine internal structure and ignore potential correlation information between samples. To address the problems and fully utilize manifold information of data, this paper proposes non-convex feature selection based on feature correlation representation and dual manifold optimization (FDNFS). Firstly, FDNFS constructs feature graph of original data, which can obtain feature correlation representation to represent the interconnection information. Based on the obtained interconnection information, FDNFS unifies feature correlation representation learning and feature selection through feature transformation matrix, so that the interconnection information between data guides feature selection. Secondly, to make multivariate frameworks guide feature selection, FDNFS introduces self-representation on the improved sparse regression model. Using self-representation can make basis matrix and reconstruction coefficient matrix reconstruct original data more accurately. Next, in order to preserve local structure information abundantly, FDNFS has two-part manifold regularization on the pseudo-label matrix in sparse regression model and the reconstructed coefficient matrix in self-representation framework. This can fully use the manifold information of data. In addition, FDNFS imposes the non-convex constraints. It can ensure the sparsity of feature selection matrix. In turn, this can select features with lower redundancy, and then select a better feature subset. Finally, this paper adopts an iterative optimization method. FDNFS is compared with nine algorithms on seven datasets. The clustering results reflect better performance of FDNFS.

A new filter feature selection algorithm for classification task by ensembling pearson correlation coefficient and mutual information

Feature selection is widely used in various fields as a key means of data dimension reduction. The existing feature selection algorithms only use one linear or nonlinear correlation indicator when evaluating variables relationships, which lacks diversity. Considering the complexity of the relationship between features, a novel feature selection evaluation function CONMI is constructed, which ensembles Pearson correlation coefficient (liner) and normalized mutual information (non-linear) to comprehensively portrays the dependencies between features and class variables. We further propose the CONMI_FS algorithm based on CONMI, which selects the optimal subset of features that has high correlation with the class variables and low redundancy between the selected features. CONMI_FS is compared with four methods on 20 datasets and evaluated by reduction rate, classification accuracy, precision and recall metrics on KNN, SVM and DT classifiers. The experimental results show that CONMI_FS obtains the highest reduction rate of 80.04%, and achieves the best classification accuracy on KNN and SVM classifiers, which are 88.83% and 88.98%, respectively. These results indicate that CONMI_FS has good competitiveness.

MCDM-EFS: A novel ensemble feature selection method for software defect prediction using multi-criteria decision making

Software defect prediction models are used for predicting high risk software components. Feature selection has significant impact on the prediction performance of the software defect prediction models since redundant and unimportant features make the prediction model more difficult to learn. Ensemble feature selection has recently emerged as a new methodology for enhancing feature selection performance. This paper proposes a new multi-criteria-decision-making (MCDM) based ensemble feature selection (EFS) method. This new method is termed as MCDM-EFS. The proposed method, MCDM-EFS, first generates the decision matrix signifying the feature’s importance score with respect to various existing feature selection methods. Next, the decision matrix is used as the input to well-known MCDM method TOPSIS for assigning a final rank to each feature. The proposed approach is validated by an experimental study for predicting software defects using two classifiers K-nearest neighbor (KNN) and naïve bayes (NB) over five open-source datasets. The predictive performance of the proposed approach is compared with existing feature selection algorithms. Two evaluation metrics – nMCC and G-measure are used to compare predictive performance. The experimental results show that the MCDM-EFS significantly improves the predictive performance of software defect prediction models against other feature selection methods in terms of nMCC as well as G-measure.

A new binary object-oriented programming optimization algorithm for solving high-dimensional feature selection problem

Feature selection (FS) is a crucial task in machine learning applications, which aims to select the most appropriate feature subset while maintaining high classification accuracy with the minimum number of selected features. Despite the widespread usage of metaheuristics as wrapper-based FS techniques, they show reduced effectiveness and increased computational cost when applied to high-dimensional datasets. This paper presents a novel Binary Object-Oriented Programming Optimization Algorithm (BOOPOA) for FS of high dimensional datasets, where the Object-Oriented Programming Optimization Algorithm (OOPOA) is a novel optimization technique inspired by the inheritance concept of Object-Oriented programming (OOP) languages. The effectiveness of this method in solving high dimensional FS problems is validated by using 26 datasets, most of which are of high dimension (large number of features). Seven existing FS algorithms are compared with the proposed OOPOA using various metrics, including best fitness, average fitness (AVG), selection size, and computational time. The results prove the superiority of the proposed algorithm over the other FS algorithms, having an average performance of %92.5, 0.078, 0.084, %38.9, and 8.6 min for classification accuracy, best fitness, average fitness, size reduction ratio, and computational time. The outcomes demonstrate the proposed FS approach's superiority over currently used methods.

Adapting Feature Selection Algorithms for the Classification of Chinese Texts

Text classification has been highlighted as the key process to organize online texts for better communication in the Digital Media Age. Text classification establishes classification rules based on text features, so the accuracy of feature selection is the basis of text classification. Facing fast-increasing Chinese electronic documents in the digital environment, scholars have accumulated quite a few algorithms for the feature selection for the automatic classification of Chinese texts in recent years. However, discussion about how to adapt existing feature selection algorithms for various types of Chinese texts is still inadequate. To address this, this study proposes three improved feature selection algorithms and tests their performance on different types of Chinese texts. These include an enhanced CHI square with mutual information (MI) algorithm, which simultaneously introduces word frequency and term adjustment (CHMI); a term frequency–CHI square (TF–CHI) algorithm, which enhances weight calculation; and a term frequency–inverse document frequency (TF–IDF) algorithm enhanced with the extreme gradient boosting (XGBoost) algorithm, which improves the algorithm’s ability of word filtering (TF–XGBoost). This study randomly chooses 3000 texts from six different categories of the Sogou news corpus to obtain the confusion matrix and evaluate the performance of the new algorithms with precision and the F1-score. Experimental comparisons are conducted on support vector machine (SVM) and naive Bayes (NB) classifiers. The experimental results demonstrate that the feature selection algorithms proposed in this paper improve performance across various news corpora, although the best feature selection schemes for each type of corpus are different. Further studies of the application of the improved feature selection methods in other languages and the improvement in classifiers are suggested.

A multi-measure feature selection method for decision systems with preference relation

Feature selection focuses on selecting important features that can improve the accuracy and simplification of the learning model. Nevertheless, for the ordered data in many real-world applications, most of the existing feature selection algorithms take the single-measure into consideration when selecting candidate features, which may affect the classification performance. Based on the insights obtained, a multi-measure feature selection algorithm is developed for ordered data, which not only considers the certain information by the dominance-based dependence, but also uses the discern information provided by the dominance-based information granularity. Extensive experiments are performed to evaluate the performance of the proposed algorithm on UCI data sets in terms of the number of selected feature subset and classification accuracy. The experimental results demonstrate that the proposed algorithm not only can find the relevant feature subset but also the classification performance is better than, or comparably well to other feature selection algorithms.

On k-means iterations and Gaussian clusters

Nowadays, k-means remains arguably the most popular clustering algorithm (Jain, 2010; Vouros et al., 2021). Two of its main properties are simplicity and speed in practice. Here, our main claim is that the average number of iterations k-means takes to converge (τ¯) is in fact very informative. We find this to be particularly interesting because τ¯ is always known when applying k-means but has never been, to our knowledge, used in the data analysis process. By experimenting with Gaussian clusters, we show that τ¯ is related to the structure of a data set under study. Data sets containing Gaussian clusters have a much lower τ¯ than those containing uniformly random data. In fact, we go considerably further and demonstrate a pattern of inverse correlation between τ¯ and the clustering quality. We illustrate the importance of our findings through two practical applications. First, we describe the cases in which τ¯ can be effectively used to identify irrelevant features present in a given data set or be used to improve the results of existing feature selection algorithms. Second, we show that there is a strong relationship between τ¯ and the number of clusters in a data set, and that this relationship can be used to find the true number of clusters it contains.

Electricity theft detection based on hybrid random forest and weighted support vector data description

Improving the detection rate of electricity theft users in smart grids is crucial to the safe operation of the power system and the economic efficiency of the grid. The traditional electricity theft detection relies on manual household inspection, which is costly and inefficient. With the application of smart meters, machine learning-based methods are gradually applied to detect electricity theft users. However, the training of the model is limited by the unbalanced electricity consumption dataset, and the performance of the model is limited in the process of processing high-dimensional data. Therefore, this paper adopts ensemble learning and imbalance data classification model to analyze the electricity theft detection problem in complex grid environment, and proposes an electricity theft detection model based on random forest and weighted support vector description. Firstly, the feature selection algorithm for mixed load data with statistical features based on random forest (FS-MLDRF) is used to extract the features that can best represent the customers’ electricity consumption behavior, so as to reduce the complexity of user data. Finally, an electricity theft detection based on hybrid random forest and weighted support vector data description is constructed (HFR-WSVDD), and the SVDD classifier is trained with the features weighted by feature importance to improve the performance of power theft detection in the case of unbalanced data. The simulation experiment results on the real electricity dataset show that compared with existing feature selection algorithms and anomaly detection algorithms, the algorithms proposed in this paper have excellent feature representation capability and better classification performance.

Multi-stage biomedical feature selection extraction algorithm for cancer detection

Cancer is a significant cause of death worldwide. Early cancer detection is greatly aided by machine learning and artificial intelligence (AI) to gene microarray data sets (microarray data). Despite this, there is a significant discrepancy between the number of gene features in the microarray data set and the number of samples. Because of this, it is crucial to identify markers for gene array data. Existing feature selection algorithms, however, generally use long-standing, are limited to single-condition feature selection and rarely take feature extraction into account. This work proposes a Multi-stage algorithm for Biomedical Deep Feature Selection (MBDFS) to address this issue. In the first, three feature selection techniques are combined for thorough feature selection, and feature subsets are obtained; in the second, an unsupervised neural network is used to create the best representation of the feature subset to enhance final classification accuracy. Using a variety of metrics, including a comparison of classification results before and after feature selection and the performance of alternative feature selection methods, we evaluate MBDFS's efficacy. The experiments demonstrate that although MBDFS uses fewer features, classification accuracy is either unchanged or enhanced.

Differential Evolution-Based Feature Selection: A Niching-Based Multiobjective Approach

Feature selection is to reduce both the dimensionality of data and the classification error rate (i.e., increase the classification accuracy) of a learning algorithm. The two objectives are often conflicting, thus a multiobjective feature selection method can obtain a set of nondominated feature subsets. Each solution in the set has a different size and a corresponding classification error rate. However, most existing feature selection algorithms have ignored that, for a given size, there can be different feature subsets with very similar or the same accuracy. This article introduces a niching-based multiobjective feature selection method that simultaneously minimizes the number of selected features and the classification error rate. The proposed method conceives to identify: 1) a set of feature subsets with good convergence and distribution and 2) multiple feature subsets choosing the same number of features with almost the same lowest classification error rate. The contributions of this article are threefold. First, a niching and global interaction mutation operator is proposed that can produce promising feature subsets. Second, a newly developed environmental selection mechanism allows equal informative feature subsets to be stored by relaxing the Pareto-dominance relationship. Finally, the proposed subset repairing mechanism can generate better feature subsets and further remove the redundant features. The proposed method is compared against seven multiobjective feature selection algorithms on 19 datasets, including both binary and multiclass classification tasks. The results show that the proposed method can evolve a rich and diverse set of nondominated solutions for different feature selection tasks, and their availability helps in understanding the relationships between features.

Designing Dispatching Rules via Novel Genetic Programming with Feature Selection in Dynamic Job-Shop Scheduling

Genetic Programming (GP) has been widely employed to create dispatching rules intelligently for production scheduling. The success of GP depends on a suitable terminal set of selected features. Specifically, techniques that consider feature selection in GP to enhance rule understandability for dynamic job shop scheduling (DJSS) have been successful. However, existing feature selection algorithms in GP focus more emphasis on obtaining more compact rules with fewer features than on improving effectiveness. This paper is an attempt at combining a novel GP method, GP via dynamic diversity management, with feature selection to design effective and interpretable dispatching rules for DJSS. The idea of the novel GP method is to achieve a progressive transition from exploration to exploitation by relating the level of population diversity to the stopping criteria and elapsed duration. We hypothesize that diverse and promising individuals obtained from the novel GP method can guide the feature selection to design competitive rules. The proposed approach is compared with three GP-based algorithms and 20 benchmark rules in the different job shop conditions and scheduling objectives. Experiments show that the proposed approach greatly outperforms the compared methods in generating more interpretable and effective rules for the three objective functions. Overall, the average improvement over the best-evolved rules by the other three GP-based algorithms is 13.28%, 12.57%, and 15.62% in the mean tardiness (MT), mean flow time (MFT), and mean weighted tardiness (MWT) objective, respectively.

Intelligent injury prediction for traumatic airway obstruction.

Airway obstruction is one of the crucial causes of death in trauma patients during the first aid. It is extremely challenging to accurately treat a great deal of casualties with airway obstruction in hospitals. The diagnosis of airway obstruction in an emergency mostly relies on the medical experience of physicians. In this paper, we propose the feature selection approach genetic algorithm-mean decrease impurity (GA-MDI) to effectively minimize the number of features as well as ensure the accuracy of prediction. Furthermore, we design a multi-modal neural network, called fully convolutional network with squeeze-and-excitation and multilayer perceptron (FCN-SE + MLP), to help physicians to predict the severity of airway obstruction. We validate the effectiveness of the proposed feature selection approach and multi-modal model on the emergency medical database from the Chinese General Hospital of the PLA. The experimental results show that GA-MDI outperforms the existing feature selection algorithms, while it is also validated that the model FCN-SE + MLP can effectively and accurately achieve the prediction of the severity of airway obstruction, which can assist clinicians in making treatment decisions for airway obstruction casualties.

Distributed Fuzzy Cognitive Maps for Feature Selection in Big Data Classification

The features of a dataset play an important role in the construction of a machine learning model. Because big datasets often have a large number of features, they may contain features that are less relevant to the machine learning task, which makes the process more time-consuming and complex. In order to facilitate learning, it is always recommended to remove the less significant features. The process of eliminating the irrelevant features and finding an optimal feature set involves comprehensively searching the dataset and considering every subset in the data. In this research, we present a distributed fuzzy cognitive map based learning-based wrapper method for feature selection that is able to extract those features from a dataset that play the most significant role in decision making. Fuzzy cognitive maps (FCMs) represent a hybrid computing technique combining elements of both fuzzy logic and cognitive maps. Using Spark’s resilient distributed datasets (RDDs), the proposed model can work effectively in a distributed manner for quick, in-memory processing along with effective iterative computations. According to the experimental results, when the proposed model is applied to a classification task, the features selected by the model help to expedite the classification process. The selection of relevant features using the proposed algorithm is on par with existing feature selection algorithms. In conjunction with a random forest classifier, the proposed model produced an average accuracy above 90%, as opposed to 85.6% accuracy when no feature selection strategy was adopted.

Feature selection for label distribution learning using dual-similarity based neighborhood fuzzy entropy

Label distribution learning (LDL) is a novel framework for handling label ambiguity problems and has been used widely in practice. However, dealing with high-dimensional data or data with redundant features in the LDL context is still an open problem. Existing feature selection algorithms cannot be directly applied to LDL due to the unique challenges caused by the label uncertainty nature. In this paper, we propose a novel LDL feature selection algorithm based on neighborhood rough sets. Specifically, we first introduce dual-similarity that is used to measure sample similarity in both the feature and the label spaces. Second, we invent a novel neighborhood fuzzy entropy as a feature evaluation metric, with which neighborhood rough sets can be applied to deal with LDL problems. Lastly, we complete a feature selection model that inherits the spirit of neighborhood rough sets and neighborhood fuzzy entropy. Extensive experiments have been conducted on twelve real-world LDL datasets, and the results demonstrate the superiority of our proposed model against to other six state-of-the-art algorithms.

A multi-label feature selection method based on an approximation of interaction information

High-dimensional multi-label data is widespread in practical applications, which brings great challenges to the research field of pattern recognition and machine learning. Many feature selection algorithms have been proposed in recent years, among which the filtering feature selection algorithm is the most popular one because of its simplicity. Therefore, filtering feature selection has become a hot research topic, especially the multi-label feature selection algorithm based on mutual information. In the algorithm, the computation cost of high dimensional mutual information is expensive. How to approximate high order mutual information based on low order mutual information has become a major research direction. To our best knowledge, all existing feature selection algorithms that consider the label correlation will increase the computational cost greatly. Therefore, this paper proposes an approximation method of three-dimensional interaction information, which is applied to the calculation of correlation and redundancy. It can take the correlation of labels into account and don’t increase the computation cost significantly at the same time. Experiments analysis results show that the proposed method is effective.

A new metric for feature selection on short text datasets

AbstractIn recent years, short texts are everywhere, especially in social media networks. Short text classification is an essential task for various applications related to the operations on short text documents. In many cases, using the entire feature set causes the high dimensionality problem in short text data. This problem reason of time‐consuming and negatively impacts the performance of classifiers. This study presents an effective feature selection algorithm called XY method, which represents the features on XY line and calculates the distance of a feature to the XY line. Also, a value named λ is calculated. According to this value, the terms are divided into different regions such as negative, positive, and third to determine their discrimination capability. The novel XY method aims to select as few terms as possible in the negative region. The proposed method is evaluated using four different short text datasets with Macro‐F1 success measure. In comparisons with other existing feature selection algorithms such as chi‐square, information gain, deviation from Poisson distribution, recently proposed max‐min ratio, and distinguishing feature selector demonstrate that the XY method achieves either better or competitive performance in significantly reduced various feature sizes.

A Hybrid Feature Selection Framework for Predicting Students Performance

Student performance prediction helps the educational stakeholders to take proactive decisions and make interventions, for the improvement of quality of education and to meet the dynamic needs of society. The selection of features for student's performance prediction not only plays significant role in increasing prediction accuracy, but also helps in building the strategic plans for the improvement of students’ academic performance. There are different feature selection algorithms for predicting the performance of students, however the studies reported in the literature claim that there are different pros and cons of existing feature selection algorithms in selection of optimal features. In this paper, a hybrid feature selection framework (using feature-fusion) is designed to identify the significant features and associated features with target class, to predict the performance of students. The main goal of the proposed hybrid feature selection is not only to improve the prediction accuracy, but also to identify optimal features for building productive strategies for the improvement in students’ academic performance. The key difference between proposed hybrid feature selection framework and existing hybrid feature selection framework, is two level feature fusion technique, with the utilization of cosine-based fusion. Whereas, according to the results reported in existing literature, cosine similarity is considered as the best similarity measure among existing similarity measures. The proposed hybrid feature selection is validated on four benchmark datasets with variations in number of features and number of instances. The validated results confirm that the proposed hybrid feature selection framework performs better than the existing hybrid feature selection framework, existing feature selection algorithms in terms of accuracy, f-measure, recall, and precision. Results reported in presented paper show that the proposed approach gives more than 90% accuracy on benchmark dataset that is better than the results of existing approach.

Android Malware Detection Based on Feature Selection and Weight Measurement

With the rapid development of Android devices, Android is currently one of the most popular mobile operating systems. However, it is also believed to be an entry point of many attack vectors. The existing Android malware detection method does not fare well when dealing with complex and intelligent malware applications, especially those based on feature detection systems which have become increasingly elusive. Therefore, we propose a novel feature selection algorithm called frequency differential selection (FDS) and weight measurement for Android malware detection. The purpose is to solve the shortcomings of the existing feature selection algorithms in detection and to filter out other effective features. Weight measurement is used to optimize the detection accuracy of the classifier and improve the accuracy of detection. We combine the optimized features and the detection model for verification and evaluation. Experiments were conducted on the OmniDroid dataset, which is a large and comprehensive dataset of features extracted from 22,000 real malware and benign samples. Theoretical analysis and experimental results showed that the FDS algorithm and weight measurement are effective, feasible, and exhibit advantages over other existing malware detection models. In detecting Android malware samples, the proposed method can achieve an accuracy of 99% and an F1-score of 98%.

A Feature Selection Algorithm Integrating Maximum Classification Information and Minimum Interaction Feature Dependency Information.

Feature selection is the key step in the analysis of high-dimensional small sample data. The core of feature selection is to analyse and quantify the correlation between features and class labels and the redundancy between features. However, most of the existing feature selection algorithms only consider the classification contribution of individual features and ignore the influence of interfeature redundancy and correlation. Therefore, this paper proposes a feature selection algorithm for nonlinear dynamic conditional relevance (NDCRFS) through the study and analysis of the existing feature selection algorithm ideas and method. Firstly, redundancy and relevance between features and between features and class labels are discriminated by mutual information, conditional mutual information, and interactive mutual information. Secondly, the selected features and candidate features are dynamically weighted utilizing information gain factors. Finally, to evaluate the performance of this feature selection algorithm, NDCRFS was validated against 6 other feature selection algorithms on three classifiers, using 12 different data sets, for variability and classification metrics between the different algorithms. The experimental results show that the NDCRFS method can improve the quality of the feature subsets and obtain better classification results.