Efficient Feature Selection Method Research Articles

FAStEN: An Efficient Adaptive Method for Feature Selection and Estimation in High-Dimensional Functional Regressions

Functional regression analysis is an established tool for many contemporary scientific applications. Regression problems involving large and complex data sets are ubiquitous, and feature selection is crucial for avoiding overfitting and achieving accurate predictions. We propose a new, flexible and ultra-efficient approach to perform feature selection in a sparse high dimensional function-on-function regression problem, and we show how to extend it to the scalar-on-function framework. Our method, called FAStEN, combines functional data, optimization, and machine learning techniques to perform feature selection and parameter estimation simultaneously. We exploit the properties of Functional Principal Components and the sparsity inherent to the Dual Augmented Lagrangian problem to significantly reduce computational cost, and we introduce an adaptive scheme to improve selection accuracy. In addition, we derive asymptotic oracle properties, which guarantee estimation and selection consistency for the proposed FAStEN estimator. Through an extensive simulation study, we benchmark our approach to the best existing competitors and demonstrate a massive gain in terms of CPU time and selection performance, without sacrificing the quality of the coefficients’ estimation. The theoretical derivations and the simulation study provide a strong motivation for our approach. Finally, we present an application to brain fMRI data from the AOMIC PIOP1 study. Complete FAStEN code is provided at https://github.com/IBM/funGCN. Supplementary materials for this article are available online.

Novel feature selection method for accurate breast cancer classification using Correlation coefficient and Modified GWO Algorithm

Breast cancer is perceived as the most common cause of mortality among women globally. Early detection of this disease is critical to reduce significantly the possibility of death. Machine learning techniques have been proved to be efficient and very successful for an accurate breast cancer diagnosis. In this paper, an efficient hybrid Feature Selection (FS) method named a Correlation technique-Modified Grey Wolf Optimizer (CMGWO) was proposed for accurate breast cancer classification based on dimensionality reduction. The suggested technique is based on two stages: the feature selection step and the classification step. Feature selection is the process of picking the most significant characteristics from a dataset. This stage is crucial in machine learning. Firstly, we focus on the filter method by using a Correlation technique for dimensionality reduction. This technique is intended to eliminate and reduce the number of features by selecting one feature from the other correlated features. Secondly, we use the Modified Grey Wolf Optimization algorithm (MGWO) to locate and determine the most significant features from uncorrelated features. After that, we use multiple classifiers to classify breast cancer disease based on the selected features. The Wisconsin Diagnostic Breast Cancer (WDBC) database was used to prove the performance of our proposed work. The experimental results show that the combination of the correlation method and MGWO for feature selection increases the accuracy rate of classification with a minimum number of features. The performances of different machine learning algorithms were evaluated, including Random Forest classifier (RF), Support Vector Machine (SVM) Classifier, and Naïve Bayes (NB) Classifier for the classification step. The suggested technique proves to be the best approach and reliable one among all studied approaches since it increases classification accuracy to 99.12\% obtained by CMGWO using Random Forest classifier and demonstrates its significance in detecting breast cancer.

Predictive modelling of stress, anxiety and depression: A network analysis and machine learning study.

This study assessed predictors of stress, anxiety and depression during the COVID-19 pandemic using a large number of demographic, COVID-19 context and psychological variables. Data from 741 adults were drawn from the Boston College daily sleep and well-being survey. Baseline demographics, the long version of the daily surveys and the round one assessment of the survey were utilized for the present study. A Gaussian graphical model (GGM) was estimated as a feature selection technique on a subset of ordinal/continuous variables. An ensemble Random Forest (RF) machine learning algorithm was used for prediction. GGM was found to be an efficient feature selection method and supported the findings derived from the RF machine learning model. Psychological variables were significant predictors of stress, anxiety and depression, while demographic and COVID-19-related factors had minimal predictive value. The outcome variables were mutually predictive of each other, and negative affect and subjective sleep quality were the common predictors of these outcomes of stress, anxiety, and depression. The study identifies risk factors for adverse mental health outcomes during the pandemic and informs interventions to mitigate the impact on mental health.

Towards federated feature selection: Logarithmic division for resource-conscious methods

Feature selection is a popular preprocessing step to reduce the dimensionality of the data while preserving the important information. In this paper, we propose an efficient and green feature selection method based on information theory, with the novelty of using the logarithmic division and resorting to fixed-point precision. Moreover, we extend these advancements by adapting the Mutual Information calculation for federated scenarios. The results of experiments conducted on several datasets indicate the potential of our proposal, as it does not incur significant information loss compared to the double-precision method, both in the features selected and in the subsequent classification step. Our method has shown significant potential when applied in federated scenarios, where experimentation demonstrated a lossless feature selection process and maintains classification results compared with centralised versions. These findings open up possibilities towards a new family of green feature selection methods, which would help to minimise energy consumption, lower carbon emissions and increase adaptability to Internet of Things environments.

Enhancing IoT security: A comparative study of feature reduction techniques for intrusion detection system

Internet of Things (IoT) devices are extensively utilized but are susceptible to cyberattacks, posing significant security challenges. To mitigate these threats, machine learning techniques have been implemented for network intrusion detection in IoT environments. These techniques commonly employ various feature reduction methods, prior to inputting data into models, in order to enhance the efficiency of detection processes to meet real-time requirements. This study provides a comprehensive comparison of feature selection (FS) and feature extraction (FE) techniques for network intrusion detection systems (NIDS) in IoT environments, utilizing the TON-IoT and BoT-IoT datasets for both binary and multi-class classification tasks. We evaluated FS methods, including Pearson correlation and Chi-square, and FE methods, such as Principal Component Analysis (PCA) and Autoencoders (AE), across five classic machine learning models: Decision Tree (DT), Random Forest (RF), Naive Bayes (NB), k-Nearest Neighbors (kNN), and Multi-Layer Perceptron (MLP). Our analysis revealed that FE techniques generally achieve higher accuracy and robustness compared to FS methods, with RF paired with AE delivering superior performance despite higher computational demands. DTs are most effective with smaller feature sets, while MLPs excel with larger sets. Chi-square is identified as the most efficient FS method, balancing performance and computational efficiency, whereas PCA outperforms AE in runtime efficiency. The study also highlights that FE methods are more effective for complex datasets and less sensitive to feature set size, whereas FS methods show significant performance improvements with more informative features. Despite the higher computational costs of FE methods, they demonstrate a greater capability to detect diverse attack types, making them particularly suitable for complex IoT environments. These findings are crucial for both academic research and industry applications, providing insights into optimizing detection performance and computational efficiency in NIDS for IoT networks.

Non-negative consistency affinity graph learning for unsupervised feature selection and clustering

Feature selection plays a crucial role in data mining and pattern recognition tasks. This paper proposes an efficient method for robust unsupervised feature selection, called joint local preserving and low-rank representation with nonnegative and symmetric constraint (JLPLRNS). This approach utilizes an indicator matrix instead of the row sparsity of the projection matrix to directly select some significant features from the original data and adaptively preserves the local geometric structure of original data into the low-dimensional embedded feature subspace via learned indicator matrix. By simultaneously imposing nonnegative symmetric and low-rank constraints on the representation coefficient matrix, it cannot only make this matrix discriminative, sparse and weight consistency for each pair of data, but also uncover the global structure of original data. These effectively will improve clustering performance. An algorithm based on the augmented Lagrange multiplier method with an alternating direction strategy is designed to resolve this model. Experimental results on various real datasets show that the proposed method can effectively identify some important features in data and outperforms many state-of-the-art unsupervised feature selection methods in terms of clustering performance.

A simple and efficient filter feature selection method via document-term matrix unitization

Text processing tasks commonly grapple with the challenge of high dimensionality. One of the most effective solutions to this challenge is to preprocess text data through feature selection methods. Feature selection can select the most advantageous features for subsequent operations (e.g., classification) from the native feature space of the text. This process effectively trims the feature space’s dimensionality, enhancing subsequent operations’ efficiency and accuracy. This paper proposes a straightforward and efficient filter feature selection method based on document-term matrix unitization (DTMU) for text processing. Diverging from previous filter feature selection methods that concentrate on scoring criteria definition, our method achieves more optimal feature selection by unitizing each column of the document-term matrix. This approach mitigates feature-to-feature influence and reinforces the role of the weighting proportion within the features. Subsequently, our scoring criterion subtracts the sum of weights for negative samples from positive samples and takes the absolute value. We conduct numerical experiments to compare DTMU with four advanced filter feature selection methods: max–min ratio metric, proportional rough feature selector, least loss, and relative discrimination criterion, along with two classical filter feature selection methods: Chi-square and information gain. The experiments are performed on four ten-thousand-dimensional feature space datasets: book, dvd, music, movie and two thousand-dimensional feature space datasets: imdb, amazon_cells, sourced from Amazon product reviews and movie reviews. Experimental findings demonstrate that DTMU selects more advantageous features for subsequent operations and achieves a higher dimensionality reduction rate than those of the other six methods used for comparison. Moreover, DTMU exhibits robust generalization capabilities across various classifiers and dimensional datasets. Notably, the average CPU time for a single run of DTMU is measured at 1.455 s.

Fast dominant feature selection with compensation for efficient image steganalysis

Image steganalysis has witnessed significant development but still encounters challenges in detection speed and accuracy. Based on this consideration, this paper proposes a simple yet efficient dominant feature selection method. First, a separability measurement is designed utilizing the principles of “intraclass aggregation and interclass dispersion” and “maximum interclass disparity”, by which the contribution of each feature is better evaluated. Second, a separability measurement is presented considering the “holistic interclass disparity”, and thus dominant features are directly determined. Moreover, a compensation strategy is proposed to reduce the possibility of missing dominant features, thereby further enhancing the accuracy of the selected features. The effectiveness of the proposed method is empirically verified. Extensive experiments are conducted on the BOSSbase 1.01 and ALASKA2 datasets. The results show that, compared to some state-of-the-art works, the proposed method achieves better performance in terms of computational cost, feature dimension, and detection accuracy. In particular, the computational cost of the proposed method is extremely low.

Ensemble learning-based stability improvement method for feature selection towards performance prediction

The uncertainty and complexity of real data collected in the industrial production process increase the difficulty in data-based knowledge discovering. Feature selection is an important step to remove redundant and irrelevant data, and thus it is essential to construct an efficient feature selection method. In this paper, an ensemble learning-driven stable feature selection method is proposed to improve the stability and accuracy of the feature selection. Firstly, datasets of different characteristics are generated to increase the diversity of data segments for feature selection. Secondly, two criteria (stability and prediction accuracy) are adopted to evaluate the performance weight of each feature selection algorithm, to ensure that the results of high-performance selectors have high priority in the algorithm aggregation process. Thirdly, the feature subsets are weighted and filtered based on expert experience to further ensure its stability. Finally, comparative experiments are conducted to show the effectiveness of the proposed method. Comparing with other methods, the proposed one can achieve the highest overall stability for feature selection (namely 0.936 measured by the Spearman rank correlation coefficient), and select the reasonable feature subset for data-driven prediction with the low mean absolute error (namely 0.315 as the average level).

A novel approach for classification of Tor and non-Tor traffic using efficient feature selection methods

In the dynamic realm of encrypted communications, traffic analysis and its classification are crucial for efficient resource utilization and network management. The prevalence of encryption technologies, The Onion Router (Tor) a globally recognized privacy-preserving network, poses a challenge for the task at hand by introducing complexity through its innovative onion routing mechanism. To overcome Tor’s limitations not only in terms of achieving better accuracy but also in performing classification in time-constrained scenarios, we propose a classification approach for Tor and non-Tor traffic classification, utilizing multiple models to enhance categorization and application identification. Leveraging the University of New Brunswick (UNB) Tor and non-Tor dataset, initially in a packet capture format, the preprocessing is done by transforming through CICFlowmeter. To expedite classification, we applied feature selection techniques like Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (tSNE). Machine learning algorithms like support vector machine (SVM), Gradient Boosting, Random Forest, and Artificial Neural Network (ANN) are applied. Our approach achieves a remarkable recall score ratio of 1.00, demonstrating high accuracy in Tor traffic identification. Notably, efficient feature selection has significantly reduced classification time. This work also contributes to effective Tor and non-Tor network traffic analysis, offering an efficient model for enhanced security and management.

Multi-ensemble machine learning framework for omics data integration: A case study using breast cancer samples

Integration of voluminous omics data aids to unravel biological complexities associated with different disease phenotypes. Machine learning (ML) approaches provide insightful techniques for systematic multi-omics data integration. In this study, survival prediction of breast cancer patients was undertaken using omics data of 302 female patients from The Cancer Genome Atlas (TCGA). The data included gene expression, miRNA expression, DNA methylation and copy number variation. Three computational multi-ensemble ML pipelines were tested using Support Vector Machine (SVM), Random Forest (RF) and Partial Least Squares-Discriminant Analysis (PLS-DA) algorithms. To overcome the limitations associated with univariate feature selection criteria, the ML pipelines were built along with latent factors obtained by multivariate dimension reduction method. This facilitated investigation of background genetic networks and identification of potential hub genes. Analysis of the results obtained revealed that SVM with PLS-DA method (integrated with gene expression, DNA methylation, and miRNA expression modalities) was the best-performing model with an Area Under Curve (AUC) of 89% and an accuracy of 83% for survival prediction. This study not only corroborated previously reported breast cancer-specific prognostic biomarkers but also predicted additional potential biomarkers. The work demonstrates the effective use of a multi-ensemble ML model with efficient feature selection methods as a robust protocol for cancer genotype to phenotype correlation.

Embedded EEG Feature Selection for Multi-Dimension Emotion Recognition via Local and Global Label Relevance.

Due to the problem of a small amount of EEG samples and relatively high dimensionality of electroencephalogram (EEG) features, feature selection plays an essential role in EEG-based emotion recognition. However, current EEG-based emotion recognition studies utilize a problem transformation approach to transform multi-dimension emotional labels into single-dimension labels, and then implement commonly used single-label feature selection methods to search feature subsets, which ignores the relations between different emotional dimensions. To tackle the problem, we propose an efficient EEG feature selection method for multi-dimension emotion recognition (EFSMDER) via local and global label relevance. First, to capture the local label correlations, EFSMDER implements orthogonal regression to map the original EEG feature space into a low-dimension space. Then, it employs the global label correlations in the original multi-dimension emotional label space to effectively construct the label information in the low-dimension space. With the aid of local and global relevance information, EFSMDER can conduct representational EEG feature subset selection. Three EEG emotional databases with multi-dimension emotional labels were used for performance comparison between EFSMDER and fourteen state-of-the-art methods, and the EFSMDER method achieves the best multi-dimension classification accuracies of 86.43, 84.80, and 97.86 percent on the DREAMER, DEAP, and HDED datasets, respectively.

Efficient and Stable Unsupervised Feature Selection Based on Novel Structured Graph and Data Discrepancy Learning.

Unsupervised feature selection is an important tool in data mining, machine learning, and pattern recognition. Although data labels are often missing, the number of data classes can be known and exploited in many scenarios. Therefore, a structured graph, whose number of connected components is identical to the number of data classes, has been proposed and is frequently applied in unsupervised feature selection. However, methods based on the structured graph learning face two problems. First, their structured graphs are not always guaranteed to maintain the same number of connected components as the data classes with existing optimization algorithms. Second, they usually lack strategies for choosing moderate hyperparameters. To solve these problems, an efficient and stable unsupervised feature selection method based on a novel structured graph and data discrepancy learning (ESUFS) is proposed. Specifically, the novel structured graph, consisting of a pairwise data similarity matrix and an indicator matrix, can be efficiently learned by solving a discrete optimization problem. Data discrepancy learning focuses on features that maximize the difference among data and helps in selecting discriminative features. Extensive experiments conducted on various datasets show that ESUFS outperforms state-of-the-art methods not only in accuracy (ACC) but also in stability and speed.

Feature redundancy removal for text classification using correlated feature subsets

AbstractThe curse of high dimensionality in text classification is a worrisome problem that requires efficient and optimal feature selection (FS) methods to improve classification accuracy and reduce learning time. Existing filter‐based FS methods evaluate features independently of other related ones, which can then lead to selecting a large number of redundant features, especially in high‐dimensional datasets, resulting in more learning time and less classification performance, whereas information theory‐based methods aim to maximize feature dependency with the class variable and minimize its redundancy for all selected features, which gradually becomes impractical when increasing the feature space. To overcome the time complexity issue of information theory‐based methods while taking into account the redundancy issue, in this article, we propose a new feature selection method for text classification termed correlation‐based redundancy removal, which aims to minimize the redundancy using subsets of features having close mutual information scores without sequentially seeking already selected features. The idea is that it is not important to assess the redundancy of a dominant feature having high classification information with another irrelevant feature having low classification information and vice‐versa since they are implicitly weakly correlated. Our method, tested on seven datasets using both traditional classifiers (Naive Bayes and support vector machines) and deep learning models (long short‐term memory and convolutional neural networks), demonstrated strong performance by reducing redundancy and improving classification compared to ten competitive metrics.

An efficient feature selection and explainable classification method for EEG-based epileptic seizure detection

Epilepsy is a prevalent neurological disorder that poses life-threatening emergencies. Early electroencephalogram (EEG) seizure detection can mitigate the risks and aid in the treatment of patients with epilepsy. EEG based epileptic seizure (ES) detection has significant applications in epilepsy treatment and medical diagnosis. Therefore, this paper presents an innovative framework for efficient ES detection, providing coefficient and distance correlation feature selection algorithms, a Bagged Tree-based classifer (BTBC), and Explainable Artificial Intelligence (XAI). Initially, the Butterworth filter is employed to eliminate various artifacts, and the discrete wavelet transform (DWT) is used to decompose the EEG signals and extract various eigenvalue features of the statistical time domain (STD) as linear and Fractal dimension-based non-linear (FD-NL). The optimal features are then identified through correlation coefficients with P-value and distance correlation analysis.These features are subsequently utilized by the Bagged Tree-based classifer (BTBC). The proposed model provides best performance in mitigating overfitting issues and improves the average accuracy by 2% using (CD, E), (AB, CD, E), and (A, B) experimental types as compared to other machine learning (ML) models using well-known Bonn and UCI-EEG benchmark datasets. Finally, SHapley additive exPlanation (SHAP) was used to interpret and explain the decision-making process of the proposed model. The results highlight the framework’s capability to accurately classify ES, thereby improving the diagnosis process in patients with brain dysfunctions.

Deep Learning Based Feature Selection and Ensemble Learning for Sintering State Recognition.

Sintering is a commonly used agglomeration process to prepare iron ore fines for blast furnace. The quality of sinter significantly impacts the blast furnace ironmaking process. In the vast majority of sintering plants, the judgment of sintering quality still relies on the intuitive observation of the cross section at sintering machine tail by operators, which is susceptible to the external environment and the experience of operators. In this paper, we propose a new sintering state recognition method using deep learning based feature selection and ensemble learning. First, features from the infrared thermal images of sinter cross section at the tail of the sinterer are extracted based on ResNeXt. Then, to eliminate the irrelevant, redundant and noisy features, an efficient feature selection method based on binary state transition algorithm (BSTA) is proposed to find the truly useful features. Subsequently, an ensemble learning (EL) method based on group decision making (GDM) is proposed to recognize the sintering states. Novel combination strategies considering the varying performance of the base learners are designed to further improve recognition accuracy. Industrial experiments conducted at a steel plant verify the effectiveness and superiority of the proposed method.

Survey Analysis on smart features selection for machine learning techniques mainly applied to EEG.

This research presents a survey for analyzing and classifying the EEG signal based on feature selection approaches. Moreover, The increasing complexity of high-dimensional medical datasets necessitates efficient feature selection methods for early disease detection and safeguarding public health. Intelligent feature selection represents an advanced stage in machine learning and innovative computer applications, as it reduces the number of features required for accurate classification. Generally, The main goal of feature selection is to improve the predictive model's performance and reduce the computational cost of modeling. This paper contains a survey of considerable research several on feature selection. The main measures to analyze this paper are Accuracy, precision, Recall, and F1-score assessment. In order to evaluate performance used stander dataset are EEG Bonn University. The results have proven that they have achieved the highest accuracy rate of around 99% compared with different techniques.

CInf-FS$$_S$$: an efficient infinite feature selection method using K-means clustering to partition large feature spaces

CInf-FS$$_S$$: an efficient infinite feature selection method using K-means clustering to partition large feature spaces

A COPRAS-based Approach to Multi-Label Feature Selection for Text Classification

In this article, we present a novel approach for text classification feature selection using a pseudo-relation matrix constructed with ridge regression. This approach involves generating a matrix that captures the relationship between features and target variables and then using that matrix to create a multi-criteria decision making (MCDM) problem. We then utilize the Complex Proportional Assessment (COPRAS) method to rank features in order of importance. To assess the performance of our proposed algorithm, we conduct tests on ten real-world text datasets and compare our results to the existing methods. Our findings indicate that the proposed approach outperforms other methods, and we provide statistical significance to support our claims. Our proposed algorithm provides a viable answer for researchers looking for efficient and effective feature selection methods for text data analysis, which can lead to a better understanding and interpretation of the underlying text.

RHDOFS: A Distributed Online Algorithm Towards Scalable Streaming Feature Selection

Feature selection is an important topic in data mining and machine learning, which aims to select an optimal feature subset for building effective and explainable prediction models. This paper introduces Rough Hypercuboid based Distributed Online Feature Selection (RHDOFS) method to tackle two critical challenges of Volume and Velocity associated with Big Data. By exploring the class separability in the boundary region of rough hypercuboid approach, a novel integrated feature evaluation criterion is proposed by examining not only the explicit patterns contained in the positive region but also the useful implicit patterns derived from the boundary region. An efficient online feature selection method for streaming feature scenario is developed to identify relevant and nonredundant features in an incremental iterative fashion. Furthermore, a parallel optimization mechanism by combining both data and computational independence is further employed to accelerate the original sequential implementation. An efficient distributed online feature selection algorithm is presented and implemented on the Apache Spark platform to scale for massive amount of data by exploiting the computational capabilities of multicore clusters. Encouraging results of extensive experiments indicate the superiority and notable advantages of the proposed algorithm over the relevant and representative online feature selection algorithms. Empirical tests on scalability and extensibility also demonstrate our distributed implementation significantly reduces the computational times requirements while maintaining the prediction accuracy, and is capable of scaling well in volume of data and number of computing nodes.