Feature Subset Selection Method Research Articles

An overlapping conscious relief-based feature subset selection method

Feature selection is considered as a fundamental prepossessing step in various data mining and machine learning based works. The quality of features is essential to achieve good classification performance and to have better data analysis experience. Among several feature selection methods, distance-based methods are gaining popularity because of their eligibility in capturing feature interdependency and relevancy with the endpoints. However, most of the distance-based methods only rank the features and ignore the class overlapping issues. Features with class overlapping data work as an obstacle during classification. Therefore, the objective of this research work is to propose a method named overlapping conscious MultiSURF (OMsurf) to handle data overlapping and select a subset of informative features discarding the noisy ones. Experimental results over 20 benchmark dataset demonstrates the superiority of OMsurf over six existing state-of-the-art methods

Identifying the Most Significant Features for Stress Prediction of Automobile Drivers: A Comprehensive Study

Objective: This paper used three feature selection methods on a Jordanian automobile drivers’ dataset to identify the most significant features for stress prediction algorithm performance. The dataset contains “stress” and “no-stress” classes with 30 features, categorised into physiological and contextual subsets. Methods: Eighteen classifiers from six prediction algorithm categories were evaluated: Rule-based, Tree-based, Ensemble-based, Function-based, Naïve Bayes-based and Lazy-based. Three Feature Subset Selection (FSS) methods were used: Gain Ratio, Chi-square and feature separation. Eight evaluation measures included [Formula: see text]1, Accuracy, Specificity, Sensitivity, Kappa Statistics, Mean Absolute Error (MAE), Area Under Curve (AUC) and Precision Recall Curve Area (PRCA). Results: Among the classifiers, Lazy-based LocalKNN performed significantly well in [Formula: see text]1, Accuracy, Kappa and MAE. Naïve Bayes-based Bayesian Network excelled in other measures. The original dataset with all features yielded the best overall performance, followed by the physiological-only subset. Gain Ratio and Chi-square FSS methods also showed promising results, though not significant. Conclusion: Four physiological (EMG, EMG Amplitude, Heart rate, Respiration Amplitude) and seven contextual (time range of driving, gender, age, driving skills, general accidents, last year’s accidents, stress frequency) features contributed to the best prediction outcomes. The study highlights the importance of proper feature selection and identifies optimal algorithms for specific measures.

A Comparative Analysis for Filter-Based Feature Selection Techniques with Tree-based Classification

The selection of features is crucial as an essential pre-processing method, used in the area of research as Data Mining, Text mining, and Image Processing. Raw datasets for machine learning, comprise a combination of multidimensional attributes which have a huge amount of size. They are used for making predictions. If these datasets are used for classification, due to the majority of the presence of features that are inconsistent and redundant, it occupies more resources according to time and produces incorrect results and effects on the classification. With the intention of improving the efficiency and performance of the classification, these features have to be eliminated. A variety of feature subset selection methods had been presented to find and eliminate as many redundant and useless features as feasible. A comparative analysis for filter-based feature selection techniques with tree-based classification is done in this research work. Several feature selection techniques and classifiers are applied to different datasets using the Weka Tool. In this comparative analysis, we evaluated the performance of six different feature selection techniques and their effects on decision tree classifiers using 10-fold cross-validation on three datasets. After the analysis of the result, It has been found that the feature selection method ChiSquaredAttributeEval + Ranker search with Random Forest classifier beats other methods for effective and efficient evaluation and it is applicable to numerous real datasets in several application domains

Linear adversarial vector modeling based intrusion detection using feature subset selection and representation methods

Linear adversarial vector modeling based intrusion detection using feature subset selection and representation methods

Active Learning Based Adversary Evasion Attacks Defense for Malwares in the Internet of Things

In this article, we study adversarial evasion attacks in the context of an active learning environment. To prevent evasion attacks in Internet of Things environments, a feature subset selection method is proposed. To train an independent classification model for a single Android application, the approach extracts application-specific data from that application. We compare and evaluate the performance of Android malware benchmarks using ensemble-based active learning, followed by the use of a collaborative machine learning classifier to protect against adversarial evasion attacks on a dataset of Android malware benchmarks. It was found that the proposed approach generates 0.91 receiver operating characteristic with 14 fabricated input features.

OSCAR: Optimal subset cardinality regression using the L0-pseudonorm with applications to prognostic modelling of prostate cancer.

In many real-world applications, such as those based on electronic health records, prognostic prediction of patient survival is based on heterogeneous sets of clinical laboratory measurements. To address the trade-off between the predictive accuracy of a prognostic model and the costs related to its clinical implementation, we propose an optimized L0-pseudonorm approach to learn sparse solutions in multivariable regression. The model sparsity is maintained by restricting the number of nonzero coefficients in the model with a cardinality constraint, which makes the optimization problem NP-hard. In addition, we generalize the cardinality constraint for grouped feature selection, which makes it possible to identify key sets of predictors that may be measured together in a kit in clinical practice. We demonstrate the operation of our cardinality constraint-based feature subset selection method, named OSCAR, in the context of prognostic prediction of prostate cancer patients, where it enables one to determine the key explanatory predictors at different levels of model sparsity. We further explore how the model sparsity affects the model accuracy and implementation cost. Lastly, we demonstrate generalization of the presented methodology to high-dimensional transcriptomics data.

The impact of feature selection techniques on effort‐aware defect prediction: An empirical study

AbstractEffort‐Aware Defect Prediction (EADP) methods sort software modules based on the defect density and guide the testing team to inspect the modules with high defect density first. Previous studies indicated that some feature selection methods could improve the performance of Classification‐Based Defect Prediction (CBDP) models, and the Correlation‐based feature subset selection method with the Best First strategy (CorBF) performed the best. However, the practical benefits of feature selection methods on EADP performance are still unknown, and blindly employing the best‐performing CorBF method in CBDP to pre‐process the defect datasets may not improve the performance of EADP models but possibly result in performance degradation. To assess the impact of the feature selection techniques on EADP, a total of 24 feature selection methods with 10 classifiers embedded in a state‐of‐the‐art EADP model (CBS+) on the 41 PROMISE defect datasets were examined. We employ six evaluation metrics to assess the performance of EADP models comprehensively. The results show that (1) The impact of the feature selection methods varies in classifiers and datasets. (2) The four wrapper‐based feature subset selection methods with forwards search, that is, AdaBoost with Forwards Search, Deep Forest with Forwards Search, Random Forest with Forwards Search, and XGBoost with Forwards Search (XGBF) are better than other methods across the studied classifiers and the used datasets. And XGBF with XGBoost as the embedded classifier in CBS+ performs the best on the datasets. (3) The best‐performing CorBF method in CBDP does not perform well on the EADP task. (4) The selected features vary with different feature selection methods and different datasets, and the features noc (number of children), ic (inheritance coupling), cbo (coupling between object classes), and cbm (coupling between methods) are frequently selected by the four wrapper‐based feature subset selection methods with forwards search. (5) Using AdaBoost, deep forest, random forest, and XGBoost as the base classifiers embedded in CBS+ can achieve the best performance. In summary, we recommend the software testing team should employ XGBF with XGBoost as the embedded classifier in CBS+ to enhance the EADP performance.

A mixed-integer exponential cone programming formulation for feature subset selection in logistic regression

Logistic regression is one of the widely-used classification tools to construct prediction models. For datasets with a large number of features, feature subset selection methods are considered to obtain accurate and interpretable prediction models, in which irrelevant and redundant features are removed. In this paper, we address the problem of feature subset selection in logistic regression using modern optimization techniques. To this end, we formulate this problem as a mixed-integer exponential cone program (MIEXP). To the best of our knowledge, this is the first time both nonlinear and discrete aspects of the underlying problem are fully considered within an exact optimization framework. We derive different versions of the MIEXP model by the means of regularization and goodness of fit measures including Akaike Information Criterion and Bayesian Information Criterion. Finally, we solve our MIEXP models using the solver MOSEK and evaluate the performance of our different versions over a set of toy examples and benchmark datasets. The results show that our approach is quite successful in obtaining accurate and interpretable prediction models compared to other methods from the literature.

MalFSM: Feature Subset Selection Method for Malware Family Classification

MalFSM: Feature Subset Selection Method for Malware Family Classification

A Method for Feature Subset Selection in Software Product Lines

Software product line (SPL) represents methods, tools, and techniques for creating a group of related software systems. Each product is a combination of multiple features. So, the task of production can be mapped to a feature subset selection problem, which is an NP-hard problem. This issue is very significant when the number of features in a software product line is huge. This chapter is aimed to address the feature subset selection in software product lines. Furthermore, the authors aim at studying the performance of a proposed multi-objective method in solving this NP-hard problem. Here, a multi-objective method (MOBAFS) is presented for feature selection in SPLs. The MOBAFS is a an optimization algorithm, which is inspired by the foraging behavior of honeybees. This technique is evaluated on five large-scale real-world software product lines in the range of 1,244 to 6,888 features. The proposed method is compared with the SATIBEA. According to the results of three solution quality indicators and two diversity metrics, the proposed method, in most cases, surpasses the other algorithm.

Initialization of Feature Selection Search for Classification

Selecting the best features in a dataset improves accuracy and efficiency of classifiers in a learning process. Datasets generally have more features than necessary, some of them being irrelevant or redundant to others. For this reason, numerous feature selection methods have been developed, in which different evaluation functions and measures are applied. This paper proposes the systematic application of individual feature evaluation methods to initialize search-based feature subset selection methods. An exhaustive review of the starting methods used by genetic algorithms from 2014 to 2020 has been carried out. Subsequently, an in-depth empirical study has been carried out evaluating the proposal for different search-based feature selection methods (Sequential forward and backward selection, Las Vegas filter and wrapper, Simulated Annealing and Genetic Algorithms). Since the computation time is reduced and the classification accuracy with the selected features is improved, the initialization of feature selection proposed in this work is proved to be worth considering while designing any feature selection algorithms.

Blood-based transcriptomic signature panel identification for cancer diagnosis: benchmarking of feature extraction methods.

Liquid biopsy has shown promise for cancer diagnosis due to its minimally invasive nature and the potential for novel biomarker discovery. However, the low concentration of relevant blood-based biosources and the heterogeneity of samples (i.e. the variability of relative abundance of molecules identified), pose major challenges to biomarker discovery. Moreover, the number of molecular measurements or features (e.g. transcript read counts) per sample could be in the order of several thousand, whereas the number of samples is often substantially lower, leading to the curse of dimensionality. These challenges, among others, elucidate the importance of a robust biomarker panel identification or feature extraction step wherein relevant molecular measurements are identified prior to classification for cancer detection. In this work, we performed a benchmarking study on 12 feature extraction methods using transcriptomic profiles derived from different blood-based biosources. The methods were assessed both in terms of their predictive performance and the robustness of the biomarker panels in diagnosing cancer or stratifying cancer subtypes. While performing the comparison, the feature extraction methods are categorized into feature subset selection methods and transformation methods. A transformation feature extraction method, namely partial least square discriminant analysis, was found to perform consistently superior in terms of classification performance. As part of the benchmarking study, a generic pipeline has been created and made available as an R package to ensure reproducibility of the results and allow for easy extension of this study to other datasets (https://github.com/VafaeeLab/bloodbased-pancancer-diagnosis).

A Spectral Feature Selection Approach With Kernelized Fuzzy Rough Sets

Feature evaluation is an important issue in constructing a feature selection algorithm in kernelized fuzzy rough sets, which has been proven to be an effective approach to deal with nonlinear classification tasks and uncertainty in learning problems. However, the feature evaluation function developed with kernelized fuzzy rough sets cannot better reflect the affinity relationship of samples and is time-consuming. To overcome these drawbacks, in this article, the problem of feature selection with kernelized fuzzy rough sets is studied based on the spectral graph theory. First, the within-class and between-class sample similarity matrices by using kernelized fuzzy approximation operators are constructed. Two operators, which can capture the affinity relationship of samples, are then introduced based on the sample similarity matrices. The proposed operator can be regarded as the sum of the weighted kernelized fuzzy approximation operators. Second, based on the ratio criterion, a feature evaluation function and its corresponding feature selection algorithm FRKF are presented, which can effectively evaluate the importance of features. Third, to illustrate the performance of the proposed algorithm, extensive experiments have been carried out to compare FRKF and other well-known feature selection methods, including the feature ranking methods and feature subset selection methods on various classification tasks. The experimental results on real-world datasets demonstrate that FRKF achieves the high performances in terms of the robustness, efficiency, and effectiveness.

Hand Motor Imagery Classification Using Effective Connectivity and Hierarchical Machine Learning in EEG Signals.

Background: Motor Imagery (MI) Brain Computer Interface (BCI) directly links central nervous system to a computer or a device. Most MI-BCI structures rely on features of a single channel of Electroencephalogram (EEG). However, to provide more valuable features, the relationships among EEG channels in the form of effective brain connectivity analysis must be considered.Objective: This study aims to identify a set of robust and discriminative effective connectivity features from EEG signals and to develop a hierarchical machine learning structure for discrimination of left and right hand MI task effectively.Material and Methods: In this analytical study, we estimated effective connectivity using Granger Causality (GC) methods namely, Generalized Partial Directed Coherence (GPDC), Directed Transfer Function (DTF) and direct Directed Transfer Function (dDTF). These measures determine the transient causal relation between different brain areas. Then a feature subset selection method based on Kruskal–Wallis test was performed to choose most significant directed causal connection between channels. Moreover, the minimal-redundancy-maximal-relevance feature selection method is applied to discard non-significance features. Finally, support vector machine method is used for classification.Results: The maximum value of the classification accuracies using GC methods over different frequency bands in 29 subjects in 60 trial is approximately 84% in Mu (8−12 Hz) - Beta1 (12 − 15 Hz) frequency band using GPDC method.Conclusion: This new hierarchical automated BCI system could be applied for discrimination of left and right hand MI tasks from EEG signal, effectively.

Dynamic Feature Subset Selection for Occluded Face Recognition

Accurate recognition of person identity is a critical task in civil society for various application and different needs. There are different well-established biometric modalities that can be used for recognition purposes such as face, voice, fingerprint, iris, etc. Recently, face images have been widely used for person recognition, since the human face is the most natural and user-friendly recognition method. However, in real-life applications, some factors may degrade the recognition performance, such as partial face occlusion, poses, illumination conditions, facial expressions, etc. In this paper, we propose two dynamic feature subset selection (DFSS) methods to achieve better recognition for occluded faces. The proposed DFSS methods are based on resilient algorithms attempting to minimize the negative influence of confusing and low-quality features extracted from occluded areas by either exclusion or weight reduction. Principal Component Analysis and Gabor filtering based approaches are initially used for face feature extraction, then the proposed DFSS methods are dynamically enforced. This is leading to more effective feature representation and an improved recognition performance. To validate their effectiveness, multiple experiments are conducted and the performance of different methods is compared. The experimental work is carried out using AR database and Extended Yale Face Database B. The obtained results of face identification and verification show that both proposed DFSS methods outperform the standard (static) use of the whole number of features and the equal feature weights.

A Two-Stage Feature Selection Approach for Fruit Recognition Using Camera Images With Various Machine Learning Classifiers

Fruit and vegetable identification and classification system is always necessary and advantageous for the agriculture business, the food processing sector, as well as the convenience shops and hypermarkets where these products are sold. Therefore, it is necessary to build an effective automated tool to meet the needs of the market by boosting the outcome, in order to improve economic efficiency. In this paper, a two-stage model is proposed to recognize fruits using camera images. We employed a Densnet121 to get the features from the fruits dataset in the first module. In the second stage, we utilize a feature subset selection method to choose the most significant features for recognizing fruits from the images of the fruits. In this study, Adaptive particle - Grey Wolf Optimization (APGWO) has been applied for choosing the most pertinent features. The final subset feature has been used for recognizing fruits using several machine learning classifiers, namely K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and Multilayer Perceptron (MLP). The proposed research’s experimental results are highly effective; the training time of proposed models is reduced to over 50%, and the classification accuracy reaches 99%.

A novel TMGWO–SLBNC‐based multidimensional feature subset selection and classification framework for frequent diagnosis of breast lesion abnormalities

The selection of optimal subset of features from high-dimensional data sets still remains a major challenge during breast cancer detection and categorization. There exist several research works regarding optimal feature subset selection from high-dimensional data sets, but the obtained results are not satisfying when multidimensional data sets (MDDs) are employed in large amount during disease analysis. In this article, an effective feature subset selection and classification method suitable for MDD is proposed. At first, the important and distinct features are extracted from the mammogram images using a Deep Neural Network with wrapper-based extraction technique. Then, a novel two-phase mutation strategy integrated with grey wolf optimizer algorithm is employed for selecting the most relevant feature subsets. Finally, a learning-based semilazy Bayesian network classifier with parallel implementation is proposed for the precise categorization of the breast cancer stages. The proposed method is executed in MATLAB platform and analyzed using mammogram images taken from MAMMOSET database. The proposed method is likened with three state-of-the-art existing feature subset selection and classification approaches for validating the efficiency of the proposed approach. For Data set 1, the proposed method shows an accuracy of 90%, 92% and 98%, which is better than the existing methods taken for comparison. Also for Data set 2, the proposed method shows an accuracy of 97%, 91% and 94%, which is much better than the accuracy achieved by the existing approaches. Thus, the proposed approach outperforms the compared existing approaches by providing better precision, recall, F-measure, specificity and accuracy.

Classification of Right/Left Hand Motor Imagery by Effective Connectivity Based on Transfer Entropy in Electroencephalogram Signal.

The right and left-hand motor imagery (MI) analysis based on the electroencephalogram (EEG) signal can directly link the central nervous system to a computer or a device. This study aims to identify a set of robust and nonlinear effective brain connectivity features quantified by transfer entropy (TE) to characterize the relationship between brain regions from EEG signals and create a hierarchical feature selection and classification for discrimination of right and lefthand MI tasks. TE is calculated among EEG channels as the distinctive, effective connectivity features. TE is a model-free method that can measure nonlinear effective connectivity and analyze multivariate dependent directed information flow among neural EEG channels. Then four feature subset selection methods namely relief-F, Fisher, Laplacian, and local learningbased clustering (LLCFS) algorithms are used to choose the most significant effective connectivity features and reduce redundant information. Finally, support vector machine (SVM) and linear discriminant analysis (LDA) methods are used for classification. Results show that the best performance in 29 healthy subjects and 60 trials is achieved using the TE method via the Relief-F algorithm as feature selection and support vector machine (SVM) classification with 91.02% accuracy. The TE index and a hierarchical feature selection and classification can be useful for the discrimination of right- and left-hand MI tasks from multichannel EEG signals. Effective connectivity features were extracted from electroencephalogram (EEG) to analyze relationships between regions.Four feature selection methods used to select most significant effective features.Support vector machine (SVM) used for discrimination of right and left hand motor imagery (MI) task. In this study, we investigated brain activity using effective connectivity during MI task based on EEG signals. The motor imagery task can accomplish the same goal as motor execution, since they are both activated by the same brain area. Transfer entropy, coherence, and Granger casualty were employed to extract the features. Differential patterns of activity between the left vs. right MI task showed activity around the motor area rather than other areas. In order to reduce redundant information and select the most significant effective connectivity features, four feature subset selection algorithms are used: Relief-F, Fisher, Laplacian, and learning-based clustering feature selection (LLCFS). Then, support vector machine (SVM) and linear discriminant analysis (LDA) are used to classify left and right hand MI task. Comparison of three different connectivity methods showed that TE index had the highest classification accuracy, and could be useful for the discrimination of right and left hand MI tasks from multichannel EEG signals.

An intelligent hybrid feature subset selection and production pattern recognition method for modeling ethylene plant

A data-driven model framework integrating Feature Subset Selection (FSS), production pattern clustering analysis and prediction was proposed for predicting ethylene yield of ethylene plant by using the massive sensing data recorded by the Distributed Control System (DCS) of petrochemical enterprises. Firstly, an Ensemble-Filter FSS model based on three different metrics is designed to initially filter all the steam cracking furnace features, and then a Wrapper FSS model based on GA-SVR is used to obtain the optimal subset of features affecting ethylene yield. The steam cracking furnace was identified based on the Density Peak Clustering (DPC) algorithm based on the production patterns embedded in the data. Ethylene yield prediction models were separately developed for each production pattern to summarize the final prediction results. The proposed model was validated against historical data from an industrial steam cracking furnace in northwest China. Results show that the number of features have a 93.4% reduction in the FSS stage. and a 40.6% reduction in predicted MSE. Compared with the benchmark ANN model, the proposed DPNN model has a 56.6% reduction in MSE based on the optimal cluster result. What’s more, the proposed framework has a strong generalization ability and with a modular structure which is easy to modify., which is expected to be used to guide the ethylene plant operating in reasonable intervals.

Intensity prediction of tropical cyclone using multilayer multi-block local binary pattern and tree-based classifiers over North Indian Ocean

Local binary pattern (LBP) is an extensively used method in image analysis and pattern recognition related works across various domains. In this paper, we explore cyclone cloud evolution patterns using a modified LBP. The multilayer multi-block local binary pattern (MMLBP) is an extended version of Completed LBP (CLBP) with multiple blocks and many layers. A tropical cyclone (TC) image is converted into 3 × 3 identical blocks to generate central pixels using CLBP magnitude descriptor block-wise (CLBP_MB). These central pixels are used to find the next layer, which is further divided into 3 × 3 blocks to generate central pixels by CLBP_MB descriptor. This technique will iterate until the dimension of the final layer reached to 1 × 1. These central pixels are extracted from each layer and gather in a vector called a feature vector. The proposed method is applied to 600 tropical cyclone (TC) images and each image generates one feature vector. Hence, 600 vectors are passed through tree-based classifiers to classify cyclone images of various classes. The proposed method of feature extraction and classification reaches a maximum of 84.66% accuracy using Random Forest (RF) classifier. The root mean squared error (RMSE) of the MMLBP method is 9.27 kt, which is better than the LBP and MBLBP approaches. Finally, the size of the original feature vector is reduced to 97.4% using correlation based feature subset selection method.