Evaluation Of Feature Selection Methods Research Articles

Empirical evaluation of feature selection methods for machine learning based intrusion detection in IoT scenarios

This paper delves into the critical need for enhanced security measures within the Internet of Things (IoT) landscape due to inherent vulnerabilities in IoT devices, rendering them susceptible to various forms of cyber-attacks. The study emphasizes the importance of Intrusion Detection Systems (IDS) for continuous threat monitoring. The objective of this study was to conduct a comprehensive evaluation of feature selection (FS) methods using various machine learning (ML) techniques for classifying traffic flows within datasets containing intrusions in IoT environments. An extensive benchmark analysis of ML techniques and FS methods was performed, assessing feature selection under different approaches including Filter Feature Ranking (FFR), Filter-Feature Subset Selection (FSS), and Wrapper-based Feature Selection (WFS). FS becomes pivotal in handling vast IoT data by reducing irrelevant attributes, addressing the curse of dimensionality, enhancing model interpretability, and optimizing resources in devices with limited capacity. Key findings indicate the outperformance for traffic flows classification of certain tree-based algorithms, such as J48 or PART, against other machine learning techniques (naive Bayes, multi-layer perceptron, logistic, adaptive boosting or k-Nearest Neighbors), showcasing a good balance between performance and execution time. FS methods' advantages and drawbacks are discussed, highlighting the main differences in results obtained among different FS approaches. Filter-feature Subset Selection (FSS) approaches such as CFS could be more suitable than Filter Feature Ranking (FFR), which may select correlated attributes, or than Wrapper-based Feature Selection (WFS) methods, which may tailor attribute subsets for specific ML techniques and have lengthy execution times. In any case, reducing attributes via FS has allowed optimization of classification without compromising accuracy. In this study, F1 score classification results above 0.99, along with a reduction of over 60% in the number of attributes, have been achieved in most experiments conducted across four datasets, both in binary and multiclass modes. This work emphasizes the importance of a balanced attribute selection process, taking into account threat detection capabilities and computational complexity.

Evaluation of Feature Selection Methods in Estimation of Precipitation Based on Deep Learning Artificial Neural Networks

Precipitation is the most important element of the water cycle and an indispensable element of water resources management. This paper’s aim is to model the monthly precipitation in 8 precipitation observation stations in the province of Hamadan, Iran. The effects and role of different feature weights pre-processing methods (Weight by deviation, Weight by PCA, Weight by correlation and Weight by Support Vector Machine) on artificial intelligence modeling were investigated. Deep learning method based on a multi-layer feed-forward artificial neural network that is trained with Stochastic Gradient Descent using back-propagation (DL-SGD) and Convolutional Neural Networks (CNN) modelling were applied. The precipitation of each station is modeled using the precipitation values of the other stations. The best result, among all scenarios, at the Vasaj station according to the DL-SGD method (CC = 0.9845, NS = 0.9543 and RMSE = 10.4169 mm) and at the Varayineh station according to the CNN method (CC = 0.9679, NS = 0.9362 and RMSE = 16.0988 mm) were estimated.

Building energy performance prediction: A reliability analysis and evaluation of feature selection methods

Building energy performance prediction: A reliability analysis and evaluation of feature selection methods

Evaluation of feature selection methods utilizing random forest and logistic regression for lung tissue categorization using HRCT images

AbstractCategorization of lung tissue patterns with interstitial lung diseases (ILD) utilize high‐resolution computed tomography (HRCT) lung images of the TALISMAN dataset which is challenging due to high intra‐class variation and inter‐class ambiguity. To tackle this, major contributions are made in three aspects. First, a novel shape‐based feature is proposed to quantify the amount of fibrotic and nodular components in a lung tissue pattern which helps to minimize intra‐class variation and inter‐class ambiguity. Second, we address the curse of dimensionality which often arises due to huge feature space. Third, to prevent an overfitting issue, the Grid Search optimization algorithm is utilized by tuning the Random Forest hyper‐parameters. In this manuscript, a framework is proposed to categorize lung tissue patterns by integrating four types of feature domains (a) intensity‐based, (b) texture‐based, (c) wavelet‐based, and (d) shape‐based along with the novel shape‐based feature. As a result, we encounter a large feature space (i.e., ), which leads to high dimensionality. To address this issue, we reduce the feature space using filter‐based f‐statistic, reliefF, minimum Redundancy Maximum Relevance (mRMR), and embedded‐based decision trees, regularization models. We found that the regularization model shrinks the feature space by 2.5 times in just 90 s whereas mRMR methods reduce the feature space by 10 times in 13 min. Using the proposed feature set, we employ Random Forest and Logistic Regression as potential classifiers to classify lung tissue patterns. Experiential results reveal that the proposed framework categorizes lung tissue patterns more effectively than state‐of‐the‐art hand‐crafted and deep learning‐based approaches.

Evaluation of Feature Selection Methods for Preserving Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine.

Temporal dataset shift can cause degradation in model performance as discrepancies between training and deployment data grow over time. The primary objective was to determine whether parsimonious models produced by specific feature selection methods are more robust to temporal dataset shift as measured by out-of-distribution (OOD) performance, while maintaining in-distribution (ID) performance. Our dataset consisted of intensive care unit patients from MIMIC-IV categorized by year groups (2008-2010, 2011-2013, 2014-2016, and 2017-2019). We trained baseline models using L2-regularized logistic regression on 2008-2010 to predict in-hospital mortality, long length of stay (LOS), sepsis, and invasive ventilation in all year groups. We evaluated three feature selection methods: L1-regularized logistic regression (L1), Remove and Retrain (ROAR), and causal feature selection. We assessed whether a feature selection method could maintain ID performance (2008-2010) and improve OOD performance (2017-2019). We also assessed whether parsimonious models retrained on OOD data performed as well as oracle models trained on all features in the OOD year group. The baseline model showed significantly worse OOD performance with the long LOS and sepsis tasks when compared with the ID performance. L1 and ROAR retained 3.7 to 12.6% of all features, whereas causal feature selection generally retained fewer features. Models produced by L1 and ROAR exhibited similar ID and OOD performance as the baseline models. The retraining of these models on 2017-2019 data using features selected from training on 2008-2010 data generally reached parity with oracle models trained directly on 2017-2019 data using all available features. Causal feature selection led to heterogeneous results with the superset maintaining ID performance while improving OOD calibration only on the long LOS task. While model retraining can mitigate the impact of temporal dataset shift on parsimonious models produced by L1 and ROAR, new methods are required to proactively improve temporal robustness.

Feature selection methods and predictive models in CT lung cancer radiomics

Radiomics is a technique that extracts quantitative features from medical images using data-characterization algorithms. Radiomic features can be used to identify tissue characteristics and radiologic phenotyping that is not observable by clinicians. A typical workflow for a radiomics study includes cohort selection, radiomic feature extraction, feature and predictive model selection, and model training and validation. While there has been increasing attention given to radiomic feature extraction, standardization, and reproducibility, currently, there is a lack of rigorous evaluation of feature selection methods and predictive models. Herein, we review the published radiomics investigations in CT lung cancer and provide an overview of the commonly used radiomic feature selection methods and predictive models. We also compare limitations of various methods in clinical applications and present sources of uncertainty associated with those methods. This review is expected to help raise awareness of the impact of radiomic feature and model selection methods on the integrity of radiomics studies.

Evaluation of Feature Selection Methods for Classification of Epileptic Seizure EEG Signals.

Epilepsy is a disease that decreases the quality of life of patients; it is also among the most common neurological diseases. Several studies have approached the classification and prediction of seizures by using electroencephalographic data and machine learning techniques. A large diversity of features has been extracted from electroencephalograms to perform classification tasks; therefore, it is important to use feature selection methods to select those that leverage pattern recognition. In this study, the performance of a set of feature selection methods was compared across different classification models; the classification task consisted of the detection of ictal activity from the CHB-MIT and Siena Scalp EEG databases. The comparison was implemented for different feature sets and the number of features. Furthermore, the similarity between selected feature subsets across classification models was evaluated. The best F1-score (0.90) was reported by the K-nearest neighbor along with the CHB-MIT dataset. Results showed that none of the feature selection methods clearly outperformed the rest of the methods, as the performance was notably affected by the classifier, dataset, and feature set. Two of the combinations (classifier/feature selection method) reporting the best results were K-nearest neighbor/support vector machine and random forest/embedded random forest.

Building Energy Efficiency Prediction for Eco Living: An Evaluation of Feature Selection Methods

Building Energy Efficiency Prediction for Eco Living: An Evaluation of Feature Selection Methods

Evaluation of Feature Selection Methods on Psychosocial Education Data Using Additive Ratio Assessment

Artificial intelligence, particularly machine learning, is the fastest-growing research trend in educational fields. Machine learning shows an impressive performance in many prediction models, including psychosocial education. The capability of machine learning to discover hidden patterns in large datasets encourages researchers to invent data with high-dimensional features. In contrast, not all features are needed by machine learning, and in many cases, high-dimensional features decrease the performance of machine learning. The feature selection method is one of the appropriate approaches to reducing the features to ensure machine learning works efficiently. Various selection methods have been proposed, but research to determine the essential subset feature in psychosocial education has not been established thus far. This research investigated and proposed methods to determine the best feature selection method in the domain of psychosocial education. We used a multi-criteria decision system (MCDM) approach with Additive Ratio Assessment (ARAS) to rank seven feature selection methods. The proposed model evaluated the best feature selection method using nine criteria from the performance metrics provided by machine learning. The experimental results showed that the ARAS is promising for evaluating and recommending the best feature selection method for psychosocial education data using the teacher’s psychosocial risk levels dataset.

An evaluation of feature selection methods for environmental data

We present a comprehensive experimental study of 12 individual as well as 6 ensemble methods for feature selection for classification tasks on environmental data, more specifically on the species distribution modeling domain. The individual methods span all 3 categories, i.e. filter, wrapper, and embedded feature selection. Experiments on 8 environmental datasets show that Shapley Additive Explanations (SHAP) and Permutation Importance are the most effective individual methods, both from the wrapper category. Generally, filter methods perform poorly, and embedded methods fall in-between. Of the 2 machine learning algorithms used, Random Forest and LightGBM, the latter prevailed. Of the 6 ensemble methods considered, i.e. Borda Count, Condorcet, Coombs, Bucklin, Instant Runoff, and Reciprocal Ranking, the last one performs best, outperforming every other method, individual or ensemble, and has a high stability.

Evaluation of feature selection methods based on artificial neural network weights

Weight-based feature selection (WBFS) are methods used to measure the contribution of input to output in a trained artificial neural network (ANN). Furthermore, algorithms such as Garson’s rely upon a single best neural network model or the mean importance value of several ANNs. However, different initialization weights lead to different importance values, as reported in other studies. These differences are misleading since each rank could result in different scores, altering the position of a variable in a given rank. Therefore, we propose a new methodology to assess the stability of a WBFS method. In essence, the idea is to use a voting approach to evaluate the importance of rankings. The results showed that Garson’s, Olden’s and Yoon’s algorithms are more stable methods when applied to artificial datasets. Nevertheless, its stability is considerably reduced when applied to real-world datasets. Hence, we concluded that future work should take into consideration the aforementioned instability of existing WBFS methods as applied to complex real-world data.

Performance evaluation of feature selection methods for aircraft hard landing incident Prediction

AbstractThis poster aims to find out a quick and accurate way for feature selection in hard landing prediction. To make the performance evaluation effective, a general evaluation framework is designed. Then seven classic feature selection methods are carried out by applying four classification algorithms, nine comparison metrics on a real QAR dataset with a total of 3,040 instances. At last, TOPSIS is employed for overall performance evaluation. Results indicate that the data‐driven category is preferable in feature selection of hard landing prediction. Moreover, Gradient Boosting Decision Tree combining with K‐Nearest Neighbor classifier on a balanced training dataset outperforms among 56 possible combination models.

Abstract 5475: Evaluation of feature selection methods used for cluster analysis in identification of novel cancer subtypes

Abstract Background: RNA-seq data from tumor samples can be used to identify novel cancer subtypes using cluster analysis. The number of features is often large compared to the number of samples and different clusters can appear in different subsets of the feature space. Feature selection techniques are therefore commonly used to reduce the dimension and remove redundant and irrelevant features before performing cluster analysis. An abundance of feature selection methods have been proposed in the literature, but it is unclear how the ability to identify novel subtypes of cancer using cluster analysis is affected by choice of feature selection method. Method: We evaluated 13 feature selection methods on four publicly available cancer data sets from The Cancer Genome Atlas. RNA-seq data and associated clinical data were retrieved from Broad institute GDAC Firehose. Overlap and characterization of the highest ranked features (i.e. genes) were studied for top 100, 1000 and 3000 genes. Performance was measured by comparing known cancer subtypes to partitions obtained using hierarchical clustering with Euclidean distance and Ward´s linkage, based on the selected features. The result was compared to both a random selection (negative control) and a supervised approach (positive control) using adjusted Rand index. Results: The relative performance of the feature selection methods varied heavily depending on number of included genes and the data set used. Based on all data, gene selection using the Dip-test statistic and the Bimodality index generated the overall highest clustering performance. The overlap of selected genes between the Bimodality index, the Dip-test and the supervised approach was relatively low. The Dip-test and Bimodality index tended to favor genes with relatively low expression values. Conclusions: The choice of feature selection method can have a huge impact on the ability to identify cancer subtypes using cluster analysis, and the relative performance is highly dependent on the data. Low overlap of selected genes between the highest ranked methods suggests that the methods identify genes that contain complementary information, and that it might be beneficial to combine two or more feature selection methods. Citation Format: Linda Vidman, David Källberg, Patrik Rydén. Evaluation of feature selection methods used for cluster analysis in identification of novel cancer subtypes [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 5475.

Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods

The evaluation of feature selection methods for text classification with small sample datasets must consider classification performance, stability, and efficiency. It is, thus, a multiple criteria decision-making (MCDM) problem. Yet there has been few research in feature selection evaluation using MCDM methods which considering multiple criteria. Therefore, we use MCDM-based methods for evaluating feature selection methods for text classification with small sample datasets. An experimental study is designed to compare five MCDM methods to validate the proposed approach with 10 feature selection methods, nine evaluation measures for binary classification, seven evaluation measures for multi-class classification, and three classifiers with 10 small datasets. Based on the ranked results of the five MCDM methods, we make recommendations concerning feature selection methods. The results demonstrate the effectiveness of the used MCDM-based method in evaluating feature selection methods.

Evaluation of Feature Selection Methods for Object-Based Land Cover Mapping of Unmanned Aerial Vehicle Imagery Using Random Forest and Support Vector Machine Classifiers

The increased feature space available in object-based classification environments (e.g., extended spectral feature sets per object, shape properties, or textural features) has a high potential of improving classifications. However, the availability of a large number of derived features per segmented object can also lead to a time-consuming and subjective process of optimizing the feature subset. The objectives of this study are to evaluate the effect of the advanced feature selection methods of popular supervised classifiers (Support Vector Machines (SVM) and Random Forest (RF)) for the example of object-based mapping of an agricultural area using Unmanned Aerial Vehicle (UAV) imagery, in order to optimize their usage for object-based agriculture pattern recognition tasks. In this study, several advanced feature selection methods were divided into both types of classifiers (SVM and RF) to conduct further evaluations using five feature-importance-evaluation methods and three feature-subset-evaluation methods. A visualization method was used to measure the change pattern of mean classification accuracy with the increase of features used, and a two-tailed t-test was used to determine the difference between two population means for both repeated ten classification accuracies. This study mainly contribute to the uncertainty analysis of feature selection for object-based classification instead of the per-pixel method. The results highlight that the RF classifier is relatively insensitive to the number of input features, even for a small training set size, whereby a negative impact of feature set size on the classification accuracy of the SVM classifier was observed. Overall, the SVM Recursive Feature Elimination (SVM-RFE) seems to be an appropriate method for both groups of classifiers, while the Correlation-based Feature Selection (CFS) is the best feature-subset-evaluation method. Most importantly, this study verified that feature selection for both classifiers is crucial for the evolving field of Object-based Image Analysis (OBIA): It is highly advisable for feature selection to be performed before object-based classification, even though an adverse impact could sometimes be observed from the wrapper methods.

Theoretical evaluation of feature selection methods based on mutual information

Feature selection methods are usually evaluated by wrapping specific classifiers and datasets in the evaluation process, resulting very often in unfair comparisons between methods. In this work, we develop a theoretical framework that allows obtaining the true feature ordering of two-dimensional sequential forward feature selection methods based on mutual information, which is independent of entropy or mutual information estimation methods, classifiers, or datasets, and leads to an undoubtful comparison of the methods. Moreover, the theoretical framework unveils problems intrinsic to some methods that are otherwise difficult to detect, namely inconsistencies in the construction of the objective function used to select the candidate features, due to various types of indeterminations and to the possibility of the entropy of continuous random variables taking null and negative values.

Performance Evaluation of Feature Selection Methods on Large Dimensional Databases

Data mining retrieves knowledge information from larger amounts of data. Clustering is an assemble of similar objects in to one class and dissimilar objects in to another class. When designing clustering ensemble on large dimensional data space, both time and space requirements for processing may be overinflated. This tends to impose feature selection methods to remove redundant features and handle the noise data. There are filter, wrapper and hybrid methods in feature selection. This paper shows a tour on types of feature selection techniques and numbers of experiments are conducted to compare feature selection techniques using different datasets with R tool, which gives better technique for clustering ensemble design.

An Evaluation of Feature Selection Methods for Multiclass Learning in Bio Informatics

Traditional data mining techniques such as classification or clustering have demonstrated achievement in datasets which has multiple instances in singly relation but while extreme point of dimensionality or complex dependencies presents in the data it fails to offer accuracy and correctness. In solution to this, Feature (attribute/variable) selection techniques since last two decades have verified its requisites to improve speed, prediction and reduce computational cost of machine learners. In this paper review of assorted feature selection methods named filter, wrapper and embedded with each classifier like support vector machines (SVM), averaged perceptron and neural network is presented. Additionally it conveys an assessment of which FS approach works better for which classifier for breast cancer dataset.

An Evaluation of Feature Selection Methods for Positive-Unlabeled Learning in Text Classification

Feature Selection is important in the processing of data in domains such as text because such data can be of very high dimension. Because in positive-unlabeled (PU) learning problems, there are no labeled negative data for training, we need unsupervised feature selection methods that do not use the class information in the training documents when selecting features for the classifier. There are few feature selection methods that are available for use in document classification with PU learning. In this paper we evaluate four unsupervised methods including, collection frequency (CF), document frequency (DF), collection frequency-inverse document frequency (CF-IDF) and term frequency-document frequency (TF-DF). We found DF most effective in our experiments.

Empirical evaluation of feature selection methods in classification

In the paper, we present an empirical evaluation of five feature selection methods: ReliefF, random forest feature selector, sequential forward selection, sequential backward selection, and Gini index. Among the evaluated methods, the random forest feature selector has not yet been widely compared to the other methods. In our evaluation, we test how the implemented feature selection can affect (i.e. improve) the accuracy of six different classifiers by performing feature selection. The results show that ReliefF and random forest enabled the classifiers to achieve the highest increase in classification accuracy on the average while reducing the number of unnecessary attributes. The achieved conclusions can advise the machine learning users which classifier and feature selection method to use to optimize the classification accuracy, which may be important especially in risk-sensitive applications of Machine Learning (e.g. medicine, business decisions, control applications) as well as in the aim to reduce costs of collecting, processing and storage of unnecessary data.