Proposed Feature Selection Method Research Articles

Feature Selection Using Diversity-Based Multi-objective Binary Differential Evolution

By identifying relevant features from the original data, feature selection methods can maintain or improve the classification accuracy and reduce the dimensionality. Recently, many multi-objective evolutionary methods have been proposed for feature selection. However, effectively handling the trade-offs between convergence and diversity of the non-dominated solutions remains a major challenge, especially for high-dimensional datasets. To cover this issue, this work studies a diversity-based multi-objective differential evolution approach to feature selection. During the environmental selection process, each of the solutions in the candidate pool will have a diversity score, and solutions with large diversity score values will be preferred so as to improve the population diversity. To reduce the search space, irrelevant and weakly relevant features are detected and removed in the proposed method. A new binary mutation operator using the neighborhood information of individuals is also proposed, aiming to produce better feature subsets. Experimental results on 14 datasets with varying difficulties show that the proposed feature selection method can obtain significantly better feature selection performance than current popular multi-objective feature selection methods.

Feature Selection Based on a Sparse Neural-Network Layer With Normalizing Constraints.

Feature selection (FS) is an important step in machine learning since it has been shown to improve prediction accuracy while suppressing the curse of dimensionality of high-dimensional data. Neural networks have experienced tremendous success in solving many nonlinear learning problems. Here, we propose a new neural-network-based FS approach that introduces two constraints, the satisfaction of which leads to a sparse FS layer. We performed extensive experiments on synthetic and real-world data to evaluate the performance of our proposed FS method. In the experiments, we focus on high-dimensional, low-sample-size data since they represent the main challenge for FS. The results confirm that the proposed FS method based on a sparse neural-network layer with normalizing constraints (SNeL-FS) is able to select the important features and yields superior performance compared to other conventional FS methods.

Utilizing Artificial Bee Colony Algorithm as Feature Selection Method in Arabic Text Classification

A huge amount of crucial information is contained in documents. The vast increase in the number of E-documents available for user access makes the utilization of automated text classification essential. Classifying or arranging documents into predefined groups is called Text classification. Feature selection (FS) is needed for minimizing the dimensionality of high-dimensional data and extracting only the features that are most pertinent to a particular task. One of the widely used algorithms for feature selection in text classification is the Evolutionary algorithm. In this paper, the filter method chi-square and the Artificial Bee Colony (ABC) algorithm were both used as FS methods. The chi-square method is a useful technique for reducing the number of features and removing those that are superfluous or redundant. The ABC technique considers the chi-square method's chosen features as viable solutions (food sources). The ABC algorithm searches for the most efficient selection of features that increase classification performance. Support Vector Machine and Naïve Bayes classifiers were used as a fitness function for the ABC algorithm. The experiment results demonstrated that the proposed feature selection method was able of decreasing the number of features by approximately 89.5%, and 94%, respectively when NB and SVM were used as fitness functions in comparison to the original dataset, while also enhancing classification performance

Unsupervised Learning for Feature Selection: A Proposed Solution for Botnet Detection in 5G Networks

The world has seen exponential growth in deploying Internet of Things (IoT) devices. In recent years, connected IoT devices have surpassed the number of connected non-IoT devices. The number of IoT devices continues to grow and they are becoming a critical component of the national infrastructure. IoT devices' characteristics and inherent limitations make them attractive targets for hackers and cyber criminals. Botnet attack is one of the serious threats on the Internet today. This article proposes pattern-based feature selection methods as part of a machine learning (ML)-based botnet detection system. Specifically, two methods are proposed: the first is based on the most dominant pattern feature values and the second is based on maximal frequent itemset mining. The proposed feature selection method uses Gini impurity and an unsupervised clustering method to select the most influential features automatically. The evaluation results show that the proposed methods have improved the performance of the detection system. The developed system has a true positive rate of 100% and a false positive rate of 0% for best performing models. In addition, the proposed methods reduce the computational cost of the system as evidenced by the detection speed of the system.

Feature Selection Based on Two-stage Resampling Technique for Imbalanced Dataset

The conventional feature selection methods fail to work on imbalanced datasets due to the overfitting issue caused by the extremely imbalanced positive and negative samples. In addition, the large number of negative samples makes the feature selection operation inefficient. To address these issues, an automatic feature selection method for addressing feature selection issues on extremely imbalanced datasets is proposed. An undersampling operation is performed to generate a small balanced dataset at first. Then, the feature importance scores are estimated based on the occurrence of the feature columns in the feature combinations that have a higher score than the base score. Finally, a repeated feature removal operation and classification model retrain are performed on the new dataset generated with the oversampling technique until the removal of the unimportant feature column does not bring any score improvement. In the proposed approach, the oversampling operation can fully utilize the dataset to train the classification model properly and the importance score is easily interpreted as the frequency of each feature column in the satisfied feature combinations. The experiment shows the superior performance of the proposed feature selection method.

FEATURE ENGINEERING WITH SENTENCE SIMILARITY USING THE LONGEST COMMON SUBSEQUENCE FOR EMAIL CLASSIFICATION

Feature selection plays a prominent role in email classification since selecting the most relevant features enhances the accuracy and performance of the learning classifier. Due to the exponential increase rate in the usage of emails, the classification of such emails posed a fitting problem. Therefore, there is a requirement for a proper classification system. Such an email classification system requires an efficient feature selection method for the accurate classification of the most relevant features. This paper proposes a novel feature selection method for sentence similarity using the longest common subsequence for email classification. The proposed feature selection method works in two main phases: First, it builds the longest common subsequence vector of features by comparing each email with all other emails in the dataset. Later, a template is constructed for each class using the closest features of emails of a particular class. Further, email classification is tested for unseen emails using these templates. The performance of the proposed method is compared with traditional feature selection methods such as TF-IDF, Information Gain, Chi-square, and semantic approach. The experimental results showed that the proposed method performed well with 96.61% accuracy.

Anomaly Detection for Hydroelectric Generating Units by Fast Robust Random Cut Forest with Fast Feature Selection Considering Characteristics of Operating Data and Random Cut Trees

This paper proposes anomaly detection for hydroelectric generating units by Fast Robust Random Cut Forest with a fast feature selection method by considering characteristics of operating data and Random Cut Trees. Hydroelectric generating units are renewable electric generating sources for electricity supply. Therefore, it is crucial to accurately detect anomalies of hydroelectric generating units. Moreover, effective features for anomaly detection should be selected to reduce operation costs of anomaly detection services. The proposed anomaly detection method can detect anomalies approximately thirty times faster than the conventional Robust Random Cut Forest with an accuracy comparative to that of the conventional method. The proposed feature selection method can select effective features almost fifty times faster than the conventional feature selection method.

Identification of clinical factors related to prediction of alcohol use disorder from electronic health records using feature selection methods

BackgroundHigh dimensionality in electronic health records (EHR) causes a significant computational problem for any systematic search for predictive, diagnostic, or prognostic patterns. Feature selection (FS) methods have been indicated to be effective in feature reduction as well as in identifying risk factors related to prediction of clinical disorders. This paper examines the prediction of patients with alcohol use disorder (AUD) using machine learning (ML) and attempts to identify risk factors related to the diagnosis of AUD.MethodsA FS framework consisting of two operational levels, base selectors and ensemble selectors. The first level consists of five FS methods: three filter methods, one wrapper method, and one embedded method. Base selector outputs are aggregated to develop four ensemble FS methods. The outputs of FS method were then fed into three ML algorithms: support vector machine (SVM), K-nearest neighbor (KNN), and random forest (RF) to compare and identify the best feature subset for the prediction of AUD from EHRs.ResultsIn terms of feature reduction, the embedded FS method could significantly reduce the number of features from 361 to 131. In terms of classification performance, RF based on 272 features selected by our proposed ensemble method (Union FS) with the highest accuracy in predicting patients with AUD, 96%, outperformed all other models in terms of AUROC, AUPRC, Precision, Recall, and F1-Score. Considering the limitations of embedded and wrapper methods, the best overall performance was achieved by our proposed Union Filter FS, which reduced the number of features to 223 and improved Precision, Recall, and F1-Score in RF from 0.77, 0.65, and 0.71 to 0.87, 0.81, and 0.84, respectively. Our findings indicate that, besides gender, age, and length of stay at the hospital, diagnosis related to digestive organs, bones, muscles and connective tissue, and the nervous systems are important clinical factors related to the prediction of patients with AUD.ConclusionOur proposed FS method could improve the classification performance significantly. It could identify clinical factors related to prediction of AUD from EHRs, thereby effectively helping clinical staff to identify and treat AUD patients and improving medical knowledge of the AUD condition. Moreover, the diversity of features among female and male patients as well as gender disparity were investigated using FS methods and ML techniques.

Relevance-diversity algorithm for feature selection and modified Bayes for prediction

Big data analytics uncovers hidden patterns through classification, prediction and reinforcement of big datasets. In these datasets, some features have a negligible connection with other features and some may be insignificant as their presence does not impact the results of big data analytics. The algorithms of big data analytics generate better classification models when supplied with a dataset consisting of relevant, important and informative features. These features can be classified as important and unimportant. For the selection of important features, different filtrations techniques are used. These techniques filter features on different basis like information gain, information dispersion, Gini index, etc. and have a few drawbacks reviewed in this paper. The first contribution of this paper is to propose a new feature selection technique named “Relevance-diversity algorithm” for selecting important features based on two measures i.e. relevance and diversity for optimizing features as low as possible and reducing the search time used in feature selection. The second contribution of the paper is that it proposes a new supervised classification algorithm based on Naive Bayes classification. The assumption of naive i.e. feature independence is discarded from the algorithm of Naive Bayes classification. The features are considered to be dependent on each other and their combined impact on the class value is evaluated. The newly proposed classification algorithm is then applied to the features selected through the relevance-diversity based feature selection technique. The datasets of Weather, Tic-Tac-Toe, Lenses, Balance-scale and CarEval are used for the evaluation of both the techniques. The results of the proposed feature selection method are compared with the existing methods and the results of Modified-Bayes are compared with the existing Naive Bayes algorithm. Analysis revealed that the proposed method performed better in terms of the number of features, accuracy and time complexity.

Hierarchical Voting-Based Feature Selection and Ensemble Learning Model Scheme for Glioma Grading with Clinical and Molecular Characteristics.

Determining the aggressiveness of gliomas, termed grading, is a critical step toward treatment optimization to increase the survival rate and decrease treatment toxicity for patients. Streamlined grading using molecular information has the potential to facilitate decision making in the clinic and aid in treatment planning. In recent years, molecular markers have increasingly gained importance in the classification of tumors. In this study, we propose a novel hierarchical voting-based methodology for improving the performance results of the feature selection stage and machine learning models for glioma grading with clinical and molecular predictors. To identify the best scheme for the given soft-voting-based ensemble learning model selections, we utilized publicly available TCGA and CGGA datasets and employed four dimensionality reduction methods to carry out a voting-based ensemble feature selection and five supervised models, with a total of sixteen combination sets. We also compared our proposed feature selection method with the LASSO feature selection method in isolation. The computational results indicate that the proposed method achieves 87.606% and 79.668% accuracy rates on TCGA and CGGA datasets, respectively, outperforming the LASSO feature selection method.

Crop type mapping using time-series Sentinel-2 imagery and U-Net in early growth periods in the Hetao irrigation district in China

The information on spatial distribution pattern and area of different crops is particularly useful for monitoring and managing the sustainability of agricultural resources. However, there are still challenges in timely mapping of crop types and planting areas to support production during the present growing season. Here, we proposed to use time-series Sentinel-2 imagery and a deep learning method for major crop mapping (wheat, maize, sunflower, and squash) in the agricultural production areas in the Hetao Irrigation District (HID), Inner Mongolia Autonomous Region, China. A feature selection method that combines the global separability index and feature recursive elimination was proposed to optimize the band features of Sentinel-2 imagery. The Random Forest, Extreme Gradient Boosting, U-Net and deeplabv3+ algorithms were then used for image classification with all spectral bands and the selected band features, respectively. The selected bands and best method were used to determine the key time window for the early identification of four major crops. Integrate all image datasets in the early window period, and build mapping models suitable for images at different times in the window period by combining the selected bands and best method. The results showed that the proposed feature selection method effectively removed redundant features and improved the classification efficiency. The U-Net algorithm was more suitable for the classification of major crops in HID with a higher accuracy whether using all band features or the selected features. The key periods for early recognition of wheat, maize, sunflower, and squash in HID were mid-May, late June, early July, and late June, respectively. The constructed models for crop type mapping in the early period were tested in 2020 and 2021 in the HID, and the F1-score, the mean intersection-over-union and overall accuracy were 86.71 %, 84.66 %, and 98.89 %, respectively. The proposed feature selection method, U-Net classification algorithm, and screening of the early mapping window period in this study provide an efficient, accurate and timely crop mapping tool in HID. Future research can further verify the applicability of the early identification time window proposed in this study to support other agricultural management activities in the areas.

Predicting Employee Attrition Using Logistic Regression With Feature Selection

Employee attrition is a reduction in employees that happens gradually. Employee attrition can damage the organization of a company, including the projects and its employee structure. This study aims to predict employee attrition in a company using the logistic regression method. Employee attrition can be predicted using machine learning because the machine learning approach is not biased due to human interference. In addition, human resources in a company need to know the most influential factors that cause the occurrence of employee attrition. In this study, we proposed feature selection methods to identify those influential factors and simplify the data training. Our approach is to predict employee attrition with three kinds of feature selection methods, namely information gain, select k-best, and recursive feature elimination (RFE). The 10-fold cross-validation was performed as an evaluation method. Prediction of employee attrition using the logistic regression method without applying feature selection gets an accuracy value of 0.865 and an AUC score of 0.932. However, by applying the RFE feature selection showed the highest evaluation result than information gain and select k-best, with an accuracy value of 0.853 and an AUC score of 0.925

Performance Enhancement of the Unbalanced Text Classification Problem Through a Modified Chi Square-Based Feature Selection Technique

This paper proposes a modified chi square-based feature selection algorithm in conjunction with a random vector functional link network-based text classifier for improving the classification performance of multi-labeled text documents with unbalanced class distributions. In the proposed feature selection method, maximum features are selected from classes that have a great deal of training and testing documents as an improvement towards original chi-square method. On two benchmark datasets that are multi-labeled, multi-class, and unbalanced, a comparison of the model with three conventional selection techniques such as chi-square, term frequency-inverse document frequency, and mutual information is accumulated for assessing its effectiveness. Additionally, the proposed model is compared with four different classifiers. In the study, it was found that the proposed model performs better in terms of precision, recall, f-measure, and hamming losses and is able to select the majority of true positive documents despite an unbalanced class distribution for both the datasets.

A new feature selection method based on frequent and associated itemsets for text classification

SummaryFeature selection is one of the major issues in pattern recognition. The quality of selected features is important for classification as the low‐quality data can degrade the model construction performance. Due to the difficulty of dealing with the problem that selected features always contain redundant information, this article focuses on the association analysis theory in data mining to select important features. In this study, a novel feature selection method based on frequent and associated itemsets (FS‐FAI) for text classification is proposed. FS‐FAI seeks to find relevant features and also takes feature interaction into account. Moreover, it uses association as a metric to evaluate the relativity between the target concept and feature(s). To evaluate the efficacy of the proposed method, several experiments were conducted on a BBC dataset from the BBC news website and SMS spam collection dataset from the UCI machine learning repository. The obtained results were compared to well‐known feature selection methods. The reported results demonstrated the effectiveness of the proposed feature selection method in selecting high‐quality features and in handling redundant information in text classification.

FG-Droid: Grouping based feature size reduction for Android malware detection.

BackgroundThe number of applications prepared for use on mobile devices has increased rapidly with the widespread use of the Android OS. This has resulted in the undesired installation of Android application packages (APKs) that violate user privacy or are malicious. The increasing similarity between Android malware and benign applications makes it difficult to distinguish them from each other and causes a situation of concern for users.MethodsIn this study, FG-Droid, a machine-learning based classifier, using the method of grouping the features obtained by static analysis, was proposed. It was created because of experiments with machine learning (ML), deep neural network (DNN), recurrent neural network (RNN), long short-term memory (LSTM), and gated recurrent unit (GRU)-based models using Drebin, Genome, and Arslan datasets.ResultsThe experimental results revealed that FG-Droid achieved a 97.7% area under the receiver operating characteristic (ROC) curve (AUC) score with a vector including only 11 static features and the ExtraTree algorithm. While reaching a high classification rate, only 0.063 seconds were needed for analysis per application. This means that the proposed feature selection method is faster than all traditional feature selection methods, and FG-Droid is one of the tools to date with the shortest analysis time per application. As a result, an efficient classifier with few features, low analysis time, and high classification success was developed using a unique feature grouping method.

A comparison between classical and new proposed feature selection methods for attention level recognition in disordered children

Lack of attention is a chronic behavior in ADHD (Attention Deficit Hyperactivity Disorder) and ASD (Autism Spectrum Disorder). Our goal is to develop a reliable method for the detection of inattention with high accuracy and low time consumption to be used in real time neurofeedback. The new applied methods for inattention in children are EMD (Empirical Mode Decomposition) with difference time series (Dt) and MRA (Multi Resolution Analysis). EMD is a method of breaking down a signal into ‘modes’ (IMFs) representing its different frequency components. Furthermore, MRA strikes balance between temporal and frequency resolution through localizing the EEG signal in frequency domain of interest (beta range) by wavelet decomposition or EMD and then retains time domain information using FD. As the results demonstrate, in intermediate and severe level cases of inattention, EMD_Dt technique is the most accurate. In mild level cases of inattention MRA (wavelet + FD) technique performance is better than EMD_Dt. However, the time consumption of the MRA (wavelet + FD) technique is fifteen times larger than EMD_Dt technique. EMD_Dt is the best technique as it requires less processing time which is the most important factor in neurofeedback, furthermore, clinician concerned more with severe and intermediate level of inattention to be treated.

An Evolutionary Multitasking-Based Feature Selection Method for High-Dimensional Classification

Feature selection (FS) is an important data preprocessing technique in data mining and machine learning, which aims to select a small subset of information features to increase the performance and reduce the dimensionality. Particle swarm optimization (PSO) has been successfully applied to FS due to being efficient and easy to implement. However, most of the existing PSO-based FS methods face the problems of trapping into local optima and computationally expensive high-dimensional data. Multifactorial optimization (MFO), as an effective evolutionary multitasking paradigm, has been widely used for solving complex problems through implicit knowledge transfer between related tasks. Inspired by MFO, this study proposes a novel PSO-based FS method to solve high-dimensional classification via information sharing between two related tasks generated from a dataset. To be specific, two related tasks about the target concept are established by evaluating the importance of features. A new crossover operator, called assortative mating, is applied to share information between these two related tasks. In addition, two mechanisms, which are variable-range strategy and subset updating mechanism, are also developed to reduce the search space and maintain the diversity of the population, respectively. The results show that the proposed FS method can achieve higher classification accuracy with a smaller feature subset in a reasonable time than the state-of-the-art FS methods on the examined high-dimensional classification problems.

Visual Object Recognition From Single-Trial EEG Signals Using Machine Learning Wrapper Techniques.

Responses of the human brain to different visual stimuli elicit specific patterns in electroencephalography (EEG) signals. It is confirmed that by analyzing these patterns, we can recognize the category of the visited objects. However, high levels of noise and artifacts in EEG signals and the discrepancies between the recorded data from different subjects in visual object recognition task make classification of cognitive states of subjects a serious challenge. In this research, we present a framework for evaluating machine learning and wrapper channel selection algorithms used for classifying single-trial EEG signals recorded in response to photographic stimuli. It is shown that by correctly mapping the entire EEG data space to informative feature spaces (IFS), the performance of the classification methods can improve significantly. Results outperform the state-of-the-art results and confirm efficiency of the proposed feature selection methods in capturing the most informative EEG channels. This can help to achieve high separability of object categories in single-trial visual object recognition task.

A new PM2.5 forecasting model based on data preprocessing, reinforcement learning and gated recurrent unit network

Accurate PM2.5 forecasting is of great significance to atmosphere pollution monitoring and control. To accurately predict PM2.5 concentration, a novel hybrid model is proposed. Our novel model includes the following three modeling processes: In stage I, a novel secondary decomposition method is adopted to decompose the raw PM2.5 data into several subseries. In stage II, a feature selection method based on reinforcement learning selects optimal features of each subseries for the predictor. In stage III, the selected features are input into a gated recurrent unit network and output the final forecasting result of all subseries. The experimental results of the paper on different data sets have verified that: (1) The proposed feature selection method based on reinforcement learning can select the optimal features, and our method outperforms the traditional feature selection method in the forecasting accuracy; (2) The novel model has excellent prediction performance in all cases and can obtain the optimal forecasting accuracy compared with twenty benchmark models and three state-of-the-art models.

Guided regularized random forest feature selection for smartphone based human activity recognition.

Human activity recognition (HAR) is an eminent area of research due to its extensive scope of applications in remote health monitoring, sports, smart home, and many more. Smartphone-based HAR systems use high-dimensional sensor data to infer human physical activities. Researchers continuously endeavor to select pertinent and non-redundant features without compromising the classification accuracy. In this work, our aim is to build an efficient HAR model that not only extracts the most relevant features from the 3-axial accelerometer and gyroscope signal data but also enhances the classification accuracy of the HAR system, without data loss using time-frequency domain features. We propose a feature selection method based on guided regularized random forest (GRRF) to determine the most pertinent and non-redundant features to reduce the time to recognize the human activities efficiently. After selecting the most relevant features, a support vector machine (SVM) is used to identify various human physical activities. The UCI public dataset and a self-collected dataset are used to assess the generalization capability and performance of the proposed feature selection method. Eventually, the accuracy reached 99.10% and 99.30% on the self-collected and UCI HAR datasets, respectively.