Wrapper Feature Selection Research Articles

Efficient traffic-based IoT device identification using a feature selection approach with Lévy flight-based sine chaotic sub-swarm binary honey badger algorithm

Internet of Things (IoT) refers to the various devices connected to the Internet, enabling them to communicate and transmit data with each other. The rapid development of the IoT also brings security and other problems in cyberspace. In this case, device identification is a crucial tool for IoT security issues, which can detect and prevent cyber-attacks. However, device identification has some challenges on IoT traffic datasets, such as large-scale and high-dimensional sparse datasets, features prone to security vulnerabilities, and identification of IP and non-IP devices, which affects the performance of classifiers. Feature selection can be seen as an effective data preprocessing technique in IoT device identification, which may improve the performance of classification and reduce the computational complexity of IoT device identification. In this paper, we propose a traffic-based IoT device identification model using a novel wrapper feature selection approach based on an improved and efficient method, which we call Lévy flight-based sine chaotic sub-swarm binary honey badger algorithm (LS2-BHBA). Specifically, four improved factors are employed in LS2-BHBA to expand the search scope, balance the exploration and exploitation phases, and enhance the search capability. In addition, a binary mechanism is implemented to enhance the suitability of the proposed LS2-BHBA for feature selection in IoT device identification. The experimental results on several real IoT traffic datasets denote that LS2-BHBA can reduce the number of features to 10%, and the classification accuracy can reach 98%, which outperforms some classical and latest comparison algorithms in the feature selection of IoT device identification.

Deep Curious Feature Selection: A Recurrent, Intrinsic-Reward Reinforcement Learning Approach to Feature Selection

Feature selection (FS) is an important step in the process of building machine learning-based models. The goal of the feature selection step is to find a small subset of features that will provide good prediction results by removing noisy, irrelevant, or repetitive features. Commonly used Wrapper methods use a machine learning model as a black box and its performance as the goal function for evaluating different features' subsets and selecting the best one. To avoid examining all possible subsets (an NP-hard problem), search algorithms are used to find the subsets to be examined, in a heuristic way. As exhaustive search methods are computationally complicated, most methods use simple and greedy search methods that yield only locally optimal results and are not sensitive to possible features interactions, which means that a feature may be chosen at the expense of two others that are more informative together. We analyze the problem of searching the features subset space with reference to two dimensions, namely, memory of past selected features subset and future selected features. We propose a new wrapper feature selection method based on the deep artificial curiosity framework, which implements intrinsic reward reinforcement learning, with long short-term memory unit (LSTM). This novel algorithm integrates these two elements of memory and future step. We show that our method, called the Deep Curious Feature Selection (DCFS) algorithm, deals with feature interactions, and provides a feature subset which improves the accuracy of learning models on artificial and real datasets.

Search space division method for wrapper feature selection on high-dimensional data classification

Feature selection (FS) is an essential pre-processing technique for high-dimensional data. Wrapper-based FS techniques are known for their superior performance over filter FS. However, when the dimensionality of data is very high the wrapper techniques become computationally very expensive. To solve this problem of scalability, this paper proposes the concept of search space division (SSD) which leads to smaller search spaces and hence reduced computational cost. The proposed SSD approach is generic in nature and can be integrated with any wrapper-based FS technique. The SSD approach divides search space into multiple parts and the wrapper-based FS is independently applied to each part. To facilitate the interaction of features over the complete search space, all feature subsets obtained from each part are combined and the wrapper-based FS is again applied to get the final feature subset. Moreover, a new wrapper FS technique named Binary Rao (BRAO) Algorithm has been proposed. BRAO is based on the metaphor-less, parameter-less meta-heuristic optimization algorithm Rao-1. The proposed combination of SSD and BRAO algorithm named SSDR has produced better classification accuracies with a smaller number of features in a much shorter time as compared to other well-regarded wrapper FS techniques as well as classical FS techniques over thirteen benchmark high-dimensional datasets.

Ensemble learning classifiers hybrid feature selection for enhancing performance of intrusion detection system

Feature selection (FS) plays an important role in the construction of efficient ensemble classifiers; particularly for intrusion detection system (IDS). An IDS is a utilized in a network architecture to protect the availability of sensitive information. However, existing IDSs suffer from redundancy, high dimensionality, and high false alarm rate (FAR). Also, lots of models are constructed for outdated datasets, which makes them less flexible to deal with new assaults. Therefore, this paper proposes a new IDS relies on hybrid FS and ensemble classifiers. A hybrid FS approach consists of two techniques, hard-voting and mean. In contrast to recent papers, we use three different FS approaches: extra tree classifier importance as an embedded FS, recursive feature elimination (RFE) as a wrapper FS, and mutual information (MI) as a filter FS. Then, a hard-voting technique has been used to fuse output of these approaches and obtain a reduced subset of features. Since each feature has three weights, a mean technique has been utilized to assign one weight to each feature and obtain an optimal subset of features. The experimental outcomes, utilizing the modern InSDN dataset, confirm that the proposed hybrid FS with ensemble soft voting classifier achieves better results than other ensemble and individual classifiers due to several measures.

Hybrid wrapper feature selection method based on genetic algorithm and extreme learning machine for intrusion detection

Intrusion detection systems play a critical role in the mitigation of cyber-attacks on the Internet of Things (IoT) environment. Due to the integration of many devices within the IoT environment, a huge amount of data is generated. The generated data sets in most cases consist of irrelevant and redundant features that affect the performance of the existing intrusion detection systems (IDS). The selection of optimal features plays a critical role in the enhancement of intrusion detection systems. This study proposes a sequential feature selection approach using an optimized extreme learning machine (ELM) with an SVM (support vector machine) classifier. The main challenge of ELM is the selection of the input parameters, which affect its performance. In this study, the genetic algorithm (GA) is used to optimize the weights of ELM to boost its performance. After the optimization, the algorithm is applied as an estimator in the sequential forward selection (wrapper technique) to select key features. The final obtained feature subset is applied for classification using SVM. The IoT_ToN network and UNSWNB15 datasets were used to test the model's performance. The performance of the model was compared with other existing state-of-the-art classifiers such as k-nearest neighbors, gradient boosting, random forest, and decision tree. The model had the best quality of the selected feature subset. The results indicate that the proposed model had a better intrusion detection performance with 99%, and 86% accuracy for IoT_ToN network dataset and UNSWNB15 datasets, respectively. The model can be used as a promising tool for enhancing the classification performance of IDS datasets.

Embedded Feature Selection Approach Using Penalized Logistic Regression for Universal Steganalysis

For universal steganalysis, the Spatial Rich Model (SRM) provides a robust feature set in the form of 106 feature submodels (34671-dimensional) to attack highly secure steganography methods. Nonetheless, the curse of dimensionality emerges as a drawback, prompting researchers to explore feature selection methods to achieve optimal performance for steganalyzers. In previous work, a wrapper feature selection method using global and local best particle swarm optimization (GLBPSO) has been used to select 28 submodels of SRM out of 106 submodels. The 28 submodels contain a total of 9152 features and an ensemble classifier has been used for the classification results. In this study, an embedded feature selection and classification approach using logistic regression (LR) with an elastic net penalty has been proposed and applied to the 9152 features. The proposed approach efficiently selects an informative subset of 9152 features while training the LR classifier. This not only reduces the number of features by a significant 75% but also boosts the classification performance when contrasted with the ensemble classifier. The proposed approach achieves an accuracy of 87.41%, surpassing the current state-of-the-art steganalysis methods based on deep learning by a substantial margin of 3-7%. It also outperforms a recent machine learning-based steganalysis method by 13%.

Comparison of Recursive Feature Elimination and Boruta as Feature Selection in Greenhouse Gas Emission Data Classification

Classification analysis is a supervised learning method that can be utilized to categorize levels of greenhouse gas emissions. Regular monitoring of greenhouse gas emissions is essential for relevant agencies to devise prevention and mitigation programs that address climate change. In classification analysis, enhancing model performance is correlated with the number of features or variables utilized, thus necessitating feature selection in its application. This study compares feature selection methods for classifying greenhouse gas emission levels, specifically wrapper feature selection, recursive feature elimination, and boruta. The Support Vector Machine (SVM) algorithm is employed to evaluate classification performance, focusing on binary classification into "high" and "low" categories in this study. The results indicate that classification performance improves with feature selection and recursive feature elimination compared to scenarios without feature selection or with Borutafeature selection. By employing three out of the thirty-nine features, accuracy, sensitivity, and specificity of 98.95%, 99%, and 97% were achieved, respectively.

Evolution of the random subset feature selection algorithm for classification problem

Datasets often include excessive or irrelevant data that affect the performance and complexity of the machine learning model. Feature selection is one of the most effective dimension-reduction technics in datasets to solve the problem. Feature reduction can lead to increased classification accuracy and decreased computational costs. Wrapper feature selection algorithms are a popular and effective category of these methods which considering the learning method's feedback. One of the well-known algorithms in this area is the random subset feature selection algorithm (RSFS). This study proposes an improved version that has higher convergence speed, lower feature selection rate, and higher classification accuracy. Improvements include considering the features' interaction in the selective subsets, continuous evaluation of the selected features to avoid being stuck in the local optima and enhancing the method's evolution phase. To comprehensively evaluate the proposed algorithm, its performance was applied to 20 standard and well-known datasets from public resources and compared with various recent related methods. Experimental results demonstrate that the proposed method reduced classification errors using fewer features compared to other methods, achieving the highest ranking in the Friedman and Wilcoxon rank-sum tests.

Machine learning approach for Migraine Aura Complexity Score prediction based on magnetic resonance imaging data

BackgroundPrevious studies have developed the Migraine Aura Complexity Score (MACS) system. MACS shows great potential in studying the complexity of migraine with aura (MwA) pathophysiology especially when implemented in neuroimaging studies. The use of sophisticated machine learning (ML) algorithms, together with deep profiling of MwA, could bring new knowledge in this field. We aimed to test several ML algorithms to study the potential of structural cortical features for predicting the MACS and therefore gain a better insight into MwA pathophysiology.MethodsThe data set used in this research consists of 340 MRI features collected from 40 MwA patients. Average MACS score was obtained for each subject. Feature selection for ML models was performed using several approaches, including a correlation test and a wrapper feature selection methodology. Regression was performed with the Support Vector Machine (SVM), Linear Regression, and Radial Basis Function network.ResultsSVM achieved a 0.89 coefficient of determination score with a wrapper feature selection. The results suggest a set of cortical features, located mostly in the parietal and temporal lobes, that show changes in MwA patients depending on aura complexity.ConclusionsThe SVM algorithm demonstrated the best potential in average MACS prediction when using a wrapper feature selection methodology. The proposed method achieved promising results in determining MwA complexity, which can provide a basis for future MwA studies and the development of MwA diagnosis and treatment.

Machine learning model for predicting immediate postoperative desaturation using spirometry signal data

Postoperative desaturation is a common post-surgery pulmonary complication. The real-time prediction of postoperative desaturation can become a preventive measure, and real-time changes in spirometry data can provide valuable information on respiratory mechanics. However, there is a lack of related research, specifically on using spirometry signals as inputs to machine learning (ML) models. We developed an ML model and postoperative desaturation prediction index (DPI) by analyzing intraoperative spirometry signals in patients undergoing laparoscopic surgery. We analyzed spirometry data from patients who underwent laparoscopic, robot-assisted gynecologic, or urologic surgery, identifying postoperative desaturation as a peripheral arterial oxygen saturation level below 95%, despite facial oxygen mask usage. We fitted the ML model on two separate datasets collected during different periods. (Datasets A and B). Dataset A (Normal 133, Desaturation 74) was used for the entire experimental process, including ML model fitting, statistical analysis, and DPI determination. Dataset B (Normal 20, Desaturation 4) was only used for verify the ML model and DPI. Four feature categories—signal property, inter-/intra-position correlation, peak value/interval variability, and demographics—were incorporated into the ML models via filter and wrapper feature selection methods. In experiments, the ML model achieved an adequate predictive capacity for postoperative desaturation, and the performance of the DPI was unbiased.

A novel hybrid binary whale optimization algorithm with chameleon hunting mechanism for wrapper feature selection in QSAR classification model:A drug-induced liver injury case study

A novel hybrid binary whale optimization algorithm with chameleon hunting mechanism for wrapper feature selection in QSAR classification model:A drug-induced liver injury case study

Genetic Algorithm for High-Dimensional Emotion Recognition from Speech Signals

Feature selection plays a crucial role in establishing an effective speech emotion recognition system. To improve recognition accuracy, people always extract as many features as possible from speech signals. However, this may reduce efficiency. We propose a hybrid filter–wrapper feature selection based on a genetic algorithm specifically designed for high-dimensional (HGA) speech emotion recognition. The algorithm first utilizes Fisher Score and information gain to comprehensively rank acoustic features, and then these features are assigned probabilities for inclusion in subsequent operations according to their ranking. HGA improves population diversity and local search ability by modifying the initial population generation method of genetic algorithm (GA) and introducing adaptive crossover and a new mutation strategy. The proposed algorithm clearly reduces the number of selected features in four common English speech emotion datasets. It is confirmed by K-nearest neighbor and random forest classifiers that it is superior to state-of-the-art algorithms in accuracy, precision, recall, and F1-Score.

Research on Feature Extraction of Underwater Acoustic Target Radiation Noise Based on Machine Learning Algorithm

Underwater acoustic target recognition is a very important technology in the field of underwater acoustics, with great economic and military value. Feature extraction technology for underwater acoustic target radiation noise signals is the key to achieving acoustic target recognition. This study aims at the feature extraction task of acoustic targets and extracts 10 types of 252-dimensional feature vectors from three domains: time domain, frequency domain, and auditory domain. Through 7 machine learning algorithms for classification and recognition experiments, the experimental results show that the recognition performance of the ensemble classifier is much better than that of a single classifier. For different types of features, this study combines three ensemble learning algorithms and feature selection algorithms to select the original 252-dimensional features. The feature selection experiment shows that the wrapper feature selection algorithm has the best effect, and the feature vector dimension can be reduced to 40 dimensions. The recognition accuracy rate is not less than 92.8%, which provides feature extraction guidance for acoustic target recognition based on feature extraction.

A novel enterprise credit risk rating model with the perspective of multiple regulatory demands

With the number of market participants rising, the issue of contradiction between heavy supervision task and insufficient supervision resource have become increasingly urgent for regulatory authorities to consider. The credit risk rating of enterprises is therefore introduced to improve the efficiency of supervision. By analyzing the purposes of enterprise credit risk rating and the multiple demands for regulatory authorities, this paper offers a multi-classification ensemble model with hybrid decomposition strategy and two-stage multi-objective feature selection. First, an adaptive resampling technique is introduced into the hybrid decomposition strategy for multi-classification, so to appropriately allocate supervision resources in each credit risk class. Then, a two-stage multi-objective feature selection method is developed to combine the various demands of regulatory authorities for feature selection. In the first stage, results of multiple Filter feature selection methods are integrated as output to improve the performance of feature information; In the second stage, the feature information from the first stage is used as input to guide the Wrapper feature selection, which is based on the Improved Multi-objective Ant Colony Optimization. Finally, a real listed enterprise credit dataset of China is used to evaluate the performance of each module and the entire model. The empirical results show that the design of hybrid decomposition strategy and multi-objective feature selection can not only deal with the class-imbalanced problem during the decomposition process, but also take into account the balance of each demand well. More importantly, a desirable framework is set in this paper to simplify the process from imbalanced multi-classification to a balanced binary one.

A Wrapper Feature Selection Based Hybrid Deep Learning Model for DDoS Detection in a Network with NFV Behaviors

A Wrapper Feature Selection Based Hybrid Deep Learning Model for DDoS Detection in a Network with NFV Behaviors

A Novel Wrapped Feature Selection Framework for Developing Power System Intrusion Detection Based on Machine Learning Methods

A Novel Wrapped Feature Selection Framework for Developing Power System Intrusion Detection Based on Machine Learning Methods

Motor imagery EEG task recognition using a nonlinear Granger causality feature extraction and an improved Salp swarm feature selection

In the study of motor imagery (MI) brain-computer interfaces (BCIs), how to improve task classification accuracy has been always one of major challenges in the applications of MI-BCIs. As a type of crucial temporal and spatial feature, nonlinear Granger Causality (NGC) analysis was applied to feature extraction of MI-electroencephalogram (EEG) signals because the constructed brain network features can reflect the causal relationship between different channels in various brain regions. However, the MI-BCI task recognition often suffer from the information redundancy of NGC features, and these redundant features will increase the complexity of the machine learning models and accordingly reduce the prediction accuracy of the classification algorithms. To address this problem, this paper proposes a step-by-step tent chaos simulated annealing salp swarm feature selection (STCSA_SaSFS) algorithm to select an optimal set of features in a wrapper feature selection model. Then, the effectiveness of this feature selection method is verified using a support vector machine (SVM) classifier. Through the study of task related MI-BCI EEG data from ten subjects, the experiments showed that the highest classification accuracy of NGC feature extraction plus STCSA_SaSFS reached 97.19%, and the average classification accuracy was 89.57%. This average classification accuracy was 20.07% higher than that of NGC feature extraction without any feature selection, and it is also 2.96% higher than that of NGC feature extraction plus a traditional SaSFS algorithm. The effectiveness of STCSA_SaSFS was also compared with that of other smart swarm optimization algorithms, such as the sparrow search feature selection algorithm (SpSFS). STCSA_SaSFS outperforms SpSFS with an average classification accuracy of 8.07%. The algorithm was validated using a public dataset validation consisting of 10 subjects, which ultimately showed that the feature selection method proposed in this paper (STSA_SaSAFS) has a large advantage in the classification performance of motor imagery brain-computer interface tasks.

Artificial Intelligence based wrapper for high dimensional feature selection

BackgroundFeature selection is important in high dimensional data analysis. The wrapper approach is one of the ways to perform feature selection, but it is computationally intensive as it builds and evaluates models of multiple subsets of features. The existing wrapper algorithm primarily focuses on shortening the path to find an optimal feature set. However, it underutilizes the capability of feature subset models, which impacts feature selection and its predictive performance.Method and ResultsThis study proposes a novel Artificial Intelligence based Wrapper (AIWrap) algorithm that integrates Artificial Intelligence (AI) with the existing wrapper algorithm. The algorithm develops a Performance Prediction Model using AI which predicts the model performance of any feature set and allows the wrapper algorithm to evaluate the feature subset performance in a model without building the model. The algorithm can make the wrapper algorithm more relevant for high-dimensional data. We evaluate the performance of this algorithm using simulated studies and real research studies. AIWrap shows better or at par feature selection and model prediction performance than standard penalized feature selection algorithms and wrapper algorithms.ConclusionAIWrap approach provides an alternative algorithm to the existing algorithms for feature selection. The current study focuses on AIWrap application in continuous cross-sectional data. However, it could be applied to other datasets like longitudinal, categorical and time-to-event biological data.

Hybrid distributed feature selection using particle swarm optimization-mutual information

Feature selection (FS) is a data preprocessing step in machine learning (ML) that selects a subset of relevant and informative features from a large feature pool. FS helps ML models improve their predictive accuracy at lower computational costs. Moreover, FS can handle the model overfitting problem on a high-dimensional dataset. A major problem with the filter and wrapper FS methods is that they consume a significant amount of time during FS on high-dimensional datasets. The proposed “HDFS(PSO-MI): hybrid distribute feature selection using particle swarm optimization-mutual information (PSO-MI)”, is a PSO-based hybrid method that can overcome the problem mentioned above. This method hybridizes the filter and wrapper techniques in a distributed manner. A new combiner is also introduced to merge the effective features selected from multiple data distributions. The effectiveness of the proposed HDFS(PSO-MI) method is evaluated using five ML classifiers, i.e., logistic regression (LR), k-NN, support vector machine (SVM), decision tree (DT), and random forest (RF), on various datasets in terms of accuracy and Matthew’s correlation coefficient (MCC). From the experimental analysis, we observed that HDFS(PSO-MI) method yielded more than 98%, 95%, 92%, 90%, and 85% accuracy for the unbalanced, kidney disease, emotions, wafer manufacturing, and breast cancer datasets, respectively. Our method shows promising results comapred to other methods, such as mutual information, gain ratio, Spearman correlation, analysis of variance (ANOVA), Pearson correlation, and an ensemble feature selection with ranking method (EFSRank).

Improved feature selection using a hybrid side-blotched lizard algorithm and genetic algorithm approach

<span lang="EN-US">Feature selection entails choosing the significant features among a wide collection of original features that are essential for predicting test data using a classifier. Feature selection is commonly used in various applications, such as bioinformatics, data mining, and the analysis of written texts, where the dataset contains tens or hundreds of thousands of features, making it difficult to analyze such a large feature set. Removing irrelevant features improves the predictor performance, making it more accurate and cost-effective. In this research, a novel hybrid technique is presented for feature selection that aims to enhance classification accuracy. A hybrid binary version of side-blotched lizard algorithm (SBLA) with genetic algorithm (GA), namely SBLAGA, which combines the strengths of both algorithms is proposed. We use a sigmoid function to adapt the continuous variables values into a binary one, and evaluate our proposed algorithm on twenty-three standard benchmark datasets. Average classification accuracy, average number of selected features and average fitness value were the evaluation criteria. According to the experimental results, SBLAGA demonstrated superior performance compared to SBLA and GA with regards to these criteria. We further compare SBLAGA with four wrapper feature selection methods that are widely used in the literature, and find it to be more efficient.</span>