Heuristic Feature Selection Algorithm Research Articles

Matrix-based incremental feature selection method using weight-partitioned multigranulation rough set

Incremental feature selection methods have gained increasing research attention as they improve the efficiency of feature selection for dynamic datasets. Multigranulation rough set, as an extension of rough set theory, allows for a comprehensive and rational analysis of problems from multiple hierarchical and granular perspectives. However, existing research on granularity partitioning relies on the decision maker's subjective experience, which lacks convincing power. In this paper, we propose a generalized multigranulation neighborhood rough set based on weight partition model, using a matrix form. We discuss several properties and define a new entropy measure to evaluate feature importance. A heuristic feature selection algorithm is developed based on this entropy to search for the optimal subset. Furthermore, we discuss dynamic updating mechanism and design two incremental feature selection algorithms. Finally, we conduct experiments on 12 public datasets to evaluate the performance of the proposed algorithms and validate their effectiveness and efficiency in feature selection for both static and dynamic datasets.

Feature selection considering feature relevance, redundancy and interactivity for neighbourhood decision systems

Feature selection is an effective method to simplify data analysis and obtain key features, which improves the accuracy and generalization ability of classifiers. Neighbourhood rough set is a typical granular computing model that enables data analysis at different granularity by setting different neighbourhood radii. In recent years, feature selection based on neighbourhood rough sets has received widespread attention from researchers. However, these models still have the following shortcomings. On the one hand, they rarely simultaneously consider the relevance, the redundancy and interactivity among features, making it difficult to accurately depict classification information in the data. On the other hand, there is still a lack of effective uncertainty measures to evaluate the classification ability of feature subsets. For these reasons, this article proposes a feature selection model that considers the relevance, redundancy, and interactivity of features. Firstly, the relevance, the redundancy and interactivity between features, are defined based on the neighbourhood, respectively. Secondly, a new feature evaluation function (MMM function) is proposed based on the principles of maximum relevance, minimum redundancy, and maximum interactivity. Finally, from the perspective of maintaining classification ability, a heuristic feature selection algorithm (FS_CRRI) was designed. To evaluate the performance of the proposed algorithm, it was compared with some representative feature selection algorithms on twelve common datasets. The experimental results on three different classifiers indicate that the FS_CRRI algorithm can effectively reduce data and achieve higher classification performance.

Noise-Tolerant Discrimination Indexes for Fuzzy ɣ Covering and Feature Subset Selection.

Fuzzy β covering (FBC) has attracted considerable attention in recent years. Nevertheless, as the basic information granularity of FBC, fuzzy β neighborhood does not satisfy reflexivity, which may lead to instability in classification learning and decision-making. Although a few studies have involved reflexive fuzzy β neighborhoods, they only focus on a single fuzzy covering and cannot effectively deal with the information representation and information fusion of multiple fuzzy coverings. Moreover, there is a lack of investigation on noise-tolerant uncertainty measures for FBC, as well as their application in feature selection. Motivated by these issues, we investigate a noise-tolerant variable precision discrimination index (VPDI) by means of a new reflexive fuzzy covering neighborhood. To this end, fuzzy ɣ neighborhood with reflexivity is introduced to characterize the information fusion of a fuzzy covering family. An uncertainty measure called fuzzy ɣ neighborhood discrimination index is then presented to reflect the discriminatory power of fuzzy covering families. Some variants of the uncertainty measure, such as variable precision joint discrimination index, variable precision conditional discrimination index, and variable precision mutual discrimination index, are then put forth by means of fuzzy decision. These VPDIs can be used as an evaluation metric for a family of fuzzy coverings. Finally, the knowledge reduction of fuzzy covering decision systems is addressed from the point of keeping the discriminatory power, and a heuristic feature selection algorithm is designed by means of the variable precision conditional discrimination index. The experiments on 16 public datasets exhibit that the proposed algorithm can effectively reduce redundant features and achieve competitive results compared with six state-of-the-art feature selection algorithms. Moreover, it demonstrates strong robustness to the interference of random noise.

Neighborhood multigranulation rough sets for cost-sensitive feature selection on hybrid data

Feature selection is a vital preprocessing step in real applications of data mining and machine learning. With the prevalence of high-dimensional hybrid data sets in real-world scenarios, along with the presence of test costs and misclassification costs, the need for effective feature selection methods has become more prominent. However, existing feature selection approaches mainly focus on cost-sensitive data from a single granularity perspective, and they are primarily applicable to single-typed data sets. To address these limitations, this paper presents a novel feature selection approach specifically designed for hybrid data, considering variable test costs and misclassification costs. The proposed method is based on neighborhood multigranulation rough sets (NMRS), which provides more comprehensive and multi-angle data analysis for cost-sensitive hybrid data. First, a novel multigranulation model is developed to effectively process cost-sensitive hybrid data. Building upon this model, a cost-based multi-criteria measure is proposed to evaluate the significance of features. This measure takes into account the comprehensive information of candidate features, including their power in algebraic view, information view, and associated costs. Furthermore, a heuristic feature selection algorithm based on NMRS is proposed to handle hybrid data with the test costs and misclassification costs. This algorithm leverages the benefits of the proposed multigranulation model and cost-based measure to identify the most discriminative features efficiently. Finally, the experimental results on twelve different datasets show that the proposed heuristic algorithm outperforms other compared algorithms, especially in total cost and classification accuracy.

Semisupervised Bacterial Heuristic Feature Selection Algorithm for High-Dimensional Classification with Missing Labels

Feature selection is a crucial method for discovering relevant features in high-dimensional data. However, most studies primarily focus on completely labeled data, ignoring the frequent occurrence of missing labels in real-world problems. To address high-dimensional and label-missing problems in data classification simultaneously, we proposed a semisupervised bacterial heuristic feature selection algorithm. To track the label-missing problem, a k-nearest neighbor semisupervised learning strategy is designed to reconstruct missing labels. In addition, the bacterial heuristic algorithm is improved using hierarchical population initialization, dynamic learning, and elite population evolution strategies to enhance the search capacity for various feature combinations. To verify the effectiveness of the proposed algorithm, three groups of comparison experiments based on eight datasets are employed, including two traditional feature selection methods, four bacterial heuristic feature selection algorithms, and two swarm-based heuristic feature selection algorithms. Experimental results demonstrate that the proposed algorithm has obvious advantages in terms of classification accuracy and selected feature numbers.

Cardiac Arrhythmia Disease Classifier Model Based on a Fuzzy Fusion Approach

Cardiac diseases are one of the greatest global health challenges. Due to the high annual mortality rates, cardiac diseases have attracted the attention of numerous researchers in recent years. This article proposes a hybrid fuzzy fusion classification model for cardiac arrhythmia diseases. The fusion model is utilized to optimally select the highest-ranked features generated by a variety of well-known feature-selection algorithms. An ensemble of classifiers is then applied to the fusion’s results. The proposed model classifies the arrhythmia dataset from the University of California, Irvine into normal/abnormal classes as well as 16 classes of arrhythmia. Initially, at the preprocessing steps, for the miss-valued attributes, we used the average value in the linear attributes group by the same class and the most frequent value for nominal attributes. However, in order to ensure the model optimality, we eliminated all attributes which have zero or constant values that might bias the results of utilized classifiers. The preprocessing step led to 161 out of 279 attributes (features). Thereafter, a fuzzy-based feature-selection fusion method is applied to fuse high-ranked features obtained from different heuristic feature-selection algorithms. In short, our study comprises three main blocks: (1) sensing data and preprocessing; (2) feature queuing, selection, and extraction; and (3) the predictive model. Our proposed method improves classification performance in terms of accuracy, F1 measure, recall, and precision when compared to state-of-the-art techniques. It achieves 98.5% accuracy for binary class mode and 98.9% accuracy for categorized class mode.

An optimized adaptive ensemble model with feature selection for network intrusion detection

SummaryNetwork intrusion detection system (NIDS) is a key component to identify abnormal behavior of network systems and plays an important role in preventing the occurrence of network attacks. Although a considerable number of machine learning methods have been applied in the field of intrusion detection, it is still a challenge for existing solutions to achieve a good classification performance. The existing traffic datasets generally have redundant and irrelevant features, which hinder classifiers from making more accurate predictions. Furthermore, a single classifier has limited classification performance and may not be able to achieve a better detection performance overall in the face of unbalanced multi‐category traffic data. Therefore, in order to improve the classification performance of intrusion detection models, this paper proposes an adaptive ensemble model by combining feature selection techniques and effective ensemble methods. Firstly, a heuristic feature selection algorithm (NRS‐SSA) is proposed by introducing the neighborhood dependency degree of the neighborhood rough set (NRS) into the salp swarm algorithm (SSA). Then, an improved adaptive weighted voting algorithm is designed. The SSA is introduced to optimize the weight matrix when setting the voting weight. Finally, we use the designed voting algorithm to combine the classification advantages of homogeneous classifiers and heterogeneous classifiers, respectively, and propose an M‐Tree algorithm and an adaptive ensemble model. The experimental results on multiple intrusion detection datasets show that the proposed adaptive ensemble model achieves an advanced detection level.

Noise-resistant multilabel fuzzy neighborhood rough sets for feature subset selection

Feature selection attempts to capture the more discriminative features and plays a significant role in multilabel learning. As an efficient mathematical tool to handle incomplete and uncertain information, fuzzy rough set theory has been widely used in feature selection. However, there are two pivotal issues to be addressed when adopting fuzzy rough sets for multilabel data analysis. One is how to effectively characterize the discriminative ability of the features to multilabel samples. Another one is how to alleviate the effect of data noise on feature selection. For these reasons, this study mainly studies the noise-tolerant fuzzy neighborhood rough set model for multilabel learning and its feature selection strategy. Firstly, a parameterized hybrid fuzzy similarity relation is introduced to granulate multilabel data, and the parameterized fuzzy decision is extended to multilabel learning. Then, a noise-resistant multilabel fuzzy neighborhood rough set model using inclusion relation is constructed for describing the discriminative ability of the features to multilabel samples. Furthermore, a noise-resistant heuristic feature selection algorithm titled NRFSFN is designed. Besides, motivated by local sampling and ensemble strategy, an efficient extended version of NRFSFN is designed from the perspective of data distribution, named ENFSFN. Finally, a series of comparison experiments on sixteen multilabel datasets demonstrate the effectiveness of the proposed two algorithms.

Feature selection based on double-hierarchical and multiplication-optimal fusion measurement in fuzzy neighborhood rough sets

In fuzzy neighborhood rough sets (FNRSs), uncertainty measurement performs mainly classification-hierarchical and multiplication-simple fusion, so the corresponding feature selection has advancement space. This paper aims to improve uncertainty measurement and feature selection via FNRSs. Two measurement strategies regarding class-hierarchical fusion and multiplication-optimal fusion are proposed, and three measure-based heuristic feature selection algorithms are developed. Concretely, fuzzy neighborhood self-information (FNSI) and joint entropy (FNJE) constitute two bases of heterogeneous fusion, and their multiplication fusion induces both the existing measure FNSIJE (which is based on classification-level fusion) and a new measure CFNSIJE (which is based on class-level fusion); furthermore, FNSIJE and CFNSIJE are extended to the optimal measures FNSIJEE and CFNSIJEE, respectively, by exponential parameterization. The four types of fusion measures acquire their calculation algorithms and granulation nonmonotonicity and systematically motivate four heuristic feature selection algorithms, i.e., the current FNSIJE-FS and the new CFNSIJE-FS, FNSIJEE-FS, and CFNSIJEE-FS. By using examples and experiments, relevant uncertainty measurement and granulation nonmonotonicity are validated, while the novel selection algorithms demonstrate better classification performances. This study establishes the hierarchical fusion and exponential expansion to acquire robust uncertainty measurement and optimal feature selection, and the measurement, nonmonotonicity, and selection have strong generalization for information fusion and rough-set learning.

Feature selection based on self-information and entropy measures for incomplete neighborhood decision systems

For incomplete datasets with mixed numerical and symbolic features, feature selection based on neighborhood multi-granulation rough sets (NMRS) is developing rapidly. However, its evaluation function only considers the information contained in the lower approximation of the neighborhood decision, which easily leads to the loss of some information. To solve this problem, we construct a novel NMRS-based uncertain measure for feature selection, named neighborhood multi-granulation self-information-based pessimistic neighborhood multi-granulation tolerance joint entropy (PTSIJE), which can be used to incomplete neighborhood decision systems. First, from the algebra view, four kinds of neighborhood multi-granulation self-information measures of decision variables are proposed by using the upper and lower approximations of NMRS. We discuss the related properties, and find the fourth measure-lenient neighborhood multi-granulation self-information measure (NMSI) has better classification performance. Then, inspired by the algebra and information views simultaneously, a feature selection method based on PTSIJE is proposed. Finally, the Fisher score method is used to delete uncorrelated features to reduce the computational complexity for high-dimensional gene datasets, and a heuristic feature selection algorithm is raised to improve classification performance for mixed and incomplete datasets. Experimental results on 11 datasets show that our method selects fewer features and has higher classification accuracy than related methods.

Electricity Network Security Monitoring Based on Bee Colony Algorithm

In order to effectively detect and discover network threats in the initial stage, this study proposes an electricity network security intrusion detection method based on feature selection. A heuristic feature selection algorithm based on the bee colony algorithm is proposed to overcome the shortcomings of existing feature evaluation methods. The algorithm uses average mutual information to measure the importance of features and more truly reflects the relationship between the selected features, the selected features, and the classification labels. Aiming at the problem that the algorithm is easy to fall into local optimization, a heuristic random search algorithm is proposed, which iteratively optimizes to generate smaller feature subsets, and improves the speed and accuracy of intrusion detection. The experimental results show that compared with the traditional algorithm, the proposed method can effectively evaluate the risk of attack path on the selected experimental data set, and the gap between the generation strategy and the optimal strategy is reduced by 71.3%, which enhances the practicability of the attack graph analysis method in a large-scale network environment. Conclusion. This method has good scalability and can be applied to large-scale network environments. It can effectively obtain attack paths that are more in line with the real threat situation in an acceptable time, so as to effectively find the network threats.

Incremental Feature Selection Using a Conditional Entropy Based on Fuzzy Dominance Neighborhood Rough Sets

Incremental feature selection approaches can improve the efficiency of feature selection used for dynamic datasets, which has attracted increasing research attention. Nevertheless, there is currently no work on incremental feature selection approaches for dynamic ordered data. Moreover, the monotonic classification effect of ordered data is easily affected by noise, so a robust feature evaluation metric is needed for feature selection algorithm. Motivated by these two issues, we investigate incremental feature selection approaches using a new conditional entropy with robustness for dynamic ordered data in this study. First, we propose a new rough set model, i.e., fuzzy dominance neighborhood rough sets (FDNRS). Second, a conditional entropy with robustness is defined based on FDNRS model, which is used as evaluation metric for features and combined with a heuristic feature selection algorithm. Finally, two incremental feature selection algorithms are designed on the basis of the above researches. Experiments are performed on ten public datasets to evaluate the robustness of the proposed metric and the performance of the incremental algorithms. Experimental results verify that the proposed metric is robust and our incremental algorithms are effective and efficient for updating reducts in dynamic ordered data.

Feature selection using Fisher score and multilabel neighborhood rough sets for multilabel classification

In recent years, feature selection for multilabel classification has attracted attention in machine learning and data mining. However, some feature selection methods ignore the correlations among labels, resulting in low performance, and most of them face challenges in determining an appropriate neighborhood radius for neighborhood systems and suffer from expensive time cost. To overcome the issues, we propose a novel feature selection method using Fisher score and multilabel neighborhood rough sets (MNRS) in multilabel neighborhood decision systems. First, to identify the correlations between labels under a binary distribution, two types of new mutual information between labels are considered, and their balance coefficients are defined. By enhancing strong correlations and weakening weak correlations between labels, a mutual information-based Fisher score model with a second-order correlation between labels is designed to fit multilabel data. Second, to address the problem of automatically choosing a neighborhood radius, a subset of heterogeneous and homogeneous samples is employed to develop a new classification margin as a neighborhood radius, and some concepts of neighborhood, neighborhood class, and upper and lower approximations are formulated for multilabel neighborhood decision systems. The weight and dependency degree are presented to effectively measure the uncertainty of samples in multilabel neighborhood decision systems. Thus, we further present a new classification margin-based MNRS model. Finally, a filter-wrapper preprocessing algorithm for feature selection using the improved Fisher score model is proposed to decrease the spatiotemporal complexity of multilabel data, and a heuristic feature selection algorithm is designed for improve classification performance on multilabel datasets. Experimental results on thirteen multilabel datasets show that the proposed algorithm is effective in selecting significant features, demonstrating its excellent classification ability in multilabel datasets.

Feature selection for dynamic interval-valued ordered data based on fuzzy dominance neighborhood rough set

Incremental learning strategy based feature selection approaches can improve the efficiency of reduction algorithm used for datasets with dynamic characteristic, which has attracted increasing research attention. Nevertheless, there is currently no work on incremental feature selection approaches for dynamic interval-valued ordered data. Interval-valued ordered data is a generalized form of single-valued ordered data, which is more widely used in practice. However, the endpoints of the interval numbers are easily polluted by noise, thereby the knowledge granules are very sensitive. Motivated by these two issues, we study incremental feature selection approaches based on a fuzzy dominance neighborhood rough set (FDNRS) for dynamic interval-valued ordered data in this work. First, we propose the FDNRS model for an interval-valued ordered decision system (IvODS) and investigate its related properties. Second, a conditional entropy with robustness is proposed based on the proposed model. This conditional entropy can measure the degree of monotonic consistency of the IvODS, so it is used as a metric and combined with a heuristic feature selection algorithm. Finally, two incremental feature selection algorithms are proposed on the basis of the above researches. Experiments are performed on nine public datasets to evaluate the robustness of the proposed metric and the performance of the incremental algorithms. Experimental results verify that the proposed metric is robust and our incremental algorithms are effective and efficient for updating reducts in dynamic IvODS.

Feature selection based on fuzzy-neighborhood relative decision entropy

Abstract Feature selection facilitates pattern recognition, and fuzzy neighborhood rough sets provide an effective tool. By fuzzy neighborhood rough sets, we propose a heuristic feature selection algorithm based on fuzzy-neighborhood relative decision entropy, called AFNRDE. At first, the fuzzy-neighborhood relative decision entropy is proposed by granulation extension and information fusion, and it acquires uncertainty measurement, integration computing, and granulation monotonicity; then, the corresponding feature selection and heuristic reduction algorithm are constructed; finally, the measure monotonicity and algorithm validity are verified by numerical example and data experiment. AFNRDE promotes initial algorithm FSMRDE based on relative decision entropy to numerical data processing, and it also outperforms two usual methods FNRS and FNGRS for classification performance.

Feature selection using Lebesgue and entropy measures for incomplete neighborhood decision systems

Feature selection for mixed and incomplete data in terms of numerical and categorical features with missing values has currently gained considerable attention. The development of the neighborhood rough sets-based feature selection method is an important step in improving classification performance, especially in incomplete data with mixed continuous numerical and categorical features. In this paper, a novel feature selection method based on the neighborhood rough sets using Lebesgue and entropy measures in incomplete neighborhood decision systems is proposed, and the method has the capacity to handle mixed and incomplete datasets; further, it can simultaneously maintain the original classification information. First, a Lebesgue measure based on the neighborhood tolerance class is developed to study the positive region and dependency degree. To thoroughly analyze the uncertainty, noise and incompleteness of incomplete neighborhood decision systems, some neighborhood tolerance entropy-based uncertainty measures are presented based on Lebesgue and entropy measures. Then, by combining an algebraic view with an information view in neighborhood rough sets, the neighborhood tolerance dependency joint entropy is defined in incomplete neighborhood decision systems. Moreover, all the corresponding properties are discussed, and the relationships among these measures are established to meaningfully convey the knowledge essence and investigate the uncertainty of incomplete neighborhood decision systems. Finally, for all high-dimensional datasets, the Fisher score method is used to preliminarily eliminate irrelevant features to significantly reduce the computational complexity, and a heuristic feature selection algorithm is designed to improve the classification performance of mixed and incomplete datasets. Experiments under an instance and fifteen public datasets demonstrate that the proposed feature selection method is effective in selecting the most relevant features, achieving great classification ability for incomplete neighborhood decision systems.

Feature selection using neighborhood entropy-based uncertainty measures for gene expression data classification

Gene expression data classification is an important technology for cancer diagnosis in bioinformatics and has been widely researched. Due to the large number of genes and the small sample size in gene expression data, feature selection based on neighborhood rough sets is a key step for improving the performance of gene expression data classification. However, some quantitative measures of feature sets may be nonmonotonic in neighborhood rough sets, and many feature selection methods based on evaluation functions yield high cardinality and low predictive accuracy. Therefore, investigating effective and efficient heuristic reduction algorithms is necessary. In this paper, a novel feature selection method based on neighborhood rough sets using neighborhood entropy-based uncertainty measures for cancer classification from gene expression data is proposed. First, some neighborhood entropy-based uncertainty measures are investigated for handling the uncertainty and noise of neighborhood decision systems. Then, to fully reflect the decision-making ability of attributes, the neighborhood credibility and neighborhood coverage degrees are defined and introduced into decision neighborhood entropy and mutual information, which are proven to be nonmonotonic. Moreover, some of the properties and relationships among these measures are derived, which is helpful for understanding the essence of the knowledge content and the uncertainty of neighborhood decision systems. Finally, the Fisher score method is employed to preliminarily eliminate irrelevant genes to significantly reduce complexity, and a heuristic feature selection algorithm with low computational complexity is presented to improve the performance of cancer classification using gene expression data. Experiments on ten gene expression datasets show that our proposed algorithm is indeed efficient and outperforms other related methods in terms of the number of selected genes and the classification accuracy, especially as the size of the genes increases.

A feature selection technique based on rough set and improvised PSO algorithm (PSORS-FS) for permission based detection of Android malwares

The set of permissions required by any Android app during installation time is considered as the feature set which are used in permission based detection of Android malwares. Those high dimensional feature set should be reduced to minimize computational overhead by choosing an optimal sub set of features. In recent times, selection of meaningful attributes is an inevitable step for mining of large dimensional data and the application of heuristic feature selection algorithms are the main research directions in this field. “Quality of classification” measure is inspired by rough set theory and can be combined with bio inspired heuristic search techniques (Particle swarm optimization, Genetic Algorithm etc.) in selecting optimal or near optimal subsets of features. In this work, a feature selection technique based on rough set and improvised particle swarm optimization (PSO) algorithm is proposed for selection of features in the permission based detection of Android malwares. The main contribution of this work is to recommend a new random key encoding method which is used in the proposed work (PSORS-FS) to convert classical PSO algorithm in discrete domain. It also reduces the issues related to maximum velocity of particles as well as sigmoid function which is related with binary PSO. PSORS-FS ensures diversity in the search process and it also reduces the tendency of premature convergence. Datasets of UCI, KEEL machine learning repository and two Android permission datasets have been used to evaluate the performance of the proposed method. Better classification performance has been yielded by proposed method over conventional filters and wrapper methods for most of the machine learning classifiers.

A hierarchical selective ensemble randomized neural network hybridized with heuristic feature selection for estimation of sea-ice thickness

In this paper, a hybrid intelligent system is developed to estimate sea-ice thickness along the Labrador coast of Canada. The developed intelligent system consists of two main parts. The first part is a heuristic feature selection algorithm used for processing a database to select the most effective features. The second part is a hierarchical selective ensemble randomized neural network (HSE-RNN) that is used to create a nonlinear map between the selected features and sea-ice thickness. The required data for processing have been collected from two sensors, i.e. moderate resolution imaging spectro-radiometer (MODIS), and advanced microwave scanning radiometer-earth (AMSR-E) observing system. To evaluate the computational advantages of the proposed intelligent framework, it is given brightness temperatures data captured at two different frequencies (low frequency, 6.9GHz, and high frequency, 36.5GHz) in addition to both atmospheric and oceanic variables from forecasting models. The obtained results demonstrate the computational power of the developed intelligent algorithm for the estimation of sea-ice thickness along the Labrador coast.

A relative decision entropy-based feature selection approach

Rough set theory has been proven to be an effective tool for feature selection. To avoid the exponential computation in exhaustive methods, many heuristic feature selection algorithms have been proposed in rough sets. However, these algorithms still suffer from high computational cost. In this paper, we propose a novel heuristic feature selection algorithm (called FSMRDE) in rough sets. To measure the significance of features in FSMRDE, we propose a new model of relative decision entropy, which is an extension of Shannon׳s information entropy in rough sets. Moreover, to test the effectiveness of FSMRDE, we apply it to intrusion detection and other application domains. Experimental results show that by using the relative decision entropy-based feature significance as heuristic information, FSMRDE is efficient for feature selection. In particular, FSMRDE is able to achieve good scalability for large data sets.