Feature Selection Stability Research Articles

Importance of feature selection stability in the classifier evaluation on high-dimensional genetic data

Classifiers trained on high-dimensional data, such as genetic datasets, often encounter situations where the number of features exceeds the number of objects. In these cases, classifiers typically rely on a small subset of features. For a robust algorithm, this subset should remain relatively stable with minor changes in the training data, such as the replacement of a few samples. While the stability of feature selection is a common focus in studies of feature selection algorithms, it is less emphasized in classifier evaluation, where only metrics such as accuracy are commonly used. We investigate the importance of feature selection stability through an empirical study of four classifiers (logistic regression, support vector machine, convex and piecewise Linear, and Random Forest) on seven high dimensional, publicly available, gene datasets. We measure the stability of feature selection using Lustgarten, Nogueira and Jaccard Index measures. We employed our own cross-validation procedure that guarantees a difference of exactly p objects between any two training sets which allows us to control the level of disturbance in the data. Our results show the existence of a drop in feature selection stability when we increase disturbance in the data in all 28 experiment configurations (seven datasets and four classifiers). The relationship is not linear, but resembles more of a hyperbolic pattern. In the evaluation of the tested classifiers, logistic regression demonstrated the highest stability. It was followed in order by support vector machine, convex and piecewise linear, with Random Forest exhibiting the lowest stability among them. This work provides evidence that all tested classifiers are very sensitive to even small changes in learning data in terms of features used by the model, while showing almost no sensitivity in terms of accuracy. The data and codes to reproduce the results in the article are available publicly on GitHub: https://github.com/tlukaszuk/feature-selection-stability-in-classifier-evaluation.

Improving the Feature Selection Stability of the Delta Test in Regression

Improving the Feature Selection Stability of the Delta Test in Regression

A new definition for feature selection stability analysis

Feature selection (FS) stability is an important topic of recent interest. Finding stable features is important for creating reliable, non-overfitted feature sets, which in turn can be used to generate machine learning models with better accuracy and explanations and are less prone to adversarial attacks. There are currently several definitions of FS stability that are widely used. In this paper, we demonstrate that existing stability metrics fail to quantify certain key elements of many datasets such as resilience to data drift or non-uniformly distributed missing values. To address this shortcoming, we propose a new definition for FS stability inspired by Lyapunov stability in dynamic systems. We show the proposed definition is statistically different from the classical record-stability on (n=90\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$n=90$$\\end{document}) datasets. We present the advantages and disadvantages of using Lyapunov and other stability definitions and demonstrate three scenarios in which each one of the three proposed stability metrics is best suited.

A Comprehensive Review of Feature Selection and Feature Selection Stability in Machine Learning

Feature selection is a dimension reduction technique used to select features that are relevant to machine learning tasks. Reducing the dataset size by eliminating redundant and irrelevant features plays a pivotal role in increasing the performance of machine learning algorithms, speeding up the learning process, and building simple models. The apparent need for feature selection has aroused considerable interest amongst researchers and has caused feature selection to find a wide range of application domains including text mining, pattern recognition, cybersecurity, bioinformatics, and big data. As a result, over the years, a substantial amount of literature has been published on feature selection and a wide variety of feature selection methods have been proposed. The quality of feature selection algorithms is measured not only by evaluating the quality of the models built using the features they select, or by the clustering tendencies of the features they select, but also by their stability. Therefore, this study focused on feature selection and feature selection stability. In the pages that follow, general concepts and methods of feature selection, feature selection stability, stability measures, and reasons and solutions for instability are discussed.

An Empirical Evaluation of Feature Selection Stability and Classification Accuracy

The performance of inductive learners can be negatively affected by high-dimensional datasets. To address this issue, feature selection methods are used. Selecting relevant features and reducing data dimensions is essential for having accurate machine learning models. Stability is an important criterion in feature selection. Stable feature selection algorithms maintain their feature preferences even when small variations exist in the training set. Studies have emphasized the importance of stable feature selection, particularly in cases where the number of samples is small and the dimensionality is high. In this study, we evaluated the relationship between stability measures, as well as, feature selection stability and classification accuracy, using the Pearson Product-Moment Correlation Coefficient. We conducted an extensive series of experiments using five filter and two wrapper feature selection methods, three classifiers for subset and classification performance evaluation, and eight real-world datasets taken from two different data repositories. We measured the stability of feature selection methods using a total of twelve stability metrics. Based on the results of correlation analyses, we have found that there is a lack of substantial evidence supporting a linear relationship between feature selection stability and classification accuracy. However, a strong positive correlation has been observed among several stability metrics.

Data Integration-Possibilities of Molecular and Clinical Data Fusion on the Example of Thyroid Cancer Diagnostics.

(1) Background: The data from independent gene expression sources may be integrated for the purpose of molecular diagnostics of cancer. So far, multiple approaches were described. Here, we investigated the impacts of different data fusion strategies on classification accuracy and feature selection stability, which allow the costs of diagnostic tests to be reduced. (2) Methods: We used molecular features (gene expression) combined with a feature extracted from the independent clinical data describing a patient’s sample. We considered the dependencies between selected features in two data fusion strategies (early fusion and late fusion) compared to classification models based on molecular features only. We compared the best accuracy classification models in terms of the number of features, which is connected to the potential cost reduction of the diagnostic classifier. (3) Results: We show that for thyroid cancer, the extracted clinical feature is correlated with (but not redundant to) the molecular data. The usage of data fusion allows a model to be obtained with similar or even higher classification quality (with a statistically significant accuracy improvement, a p-value below 0.05) and with a reduction in molecular dimensionality of the feature space from 15 to 3–8 (depending on the feature selection method). (4) Conclusions: Both strategies give comparable quality results, but the early fusion method provides better feature selection stability.

Radiomics for Discrimination between Early-Stage Nasopharyngeal Carcinoma and Benign Hyperplasia with Stable Feature Selection on MRI.

Simple SummaryIn this study, we investigated the potential of radiomic models to discriminate nasopharyngeal carcinoma from benign hyperplasia on MRI, which is important to enable screening programs to detect cancer early. We found that whereas radiomics showed promising performance, instability was presented by the feature selection step in the radiomics pipeline, which could undermine its reliability. Therefore, we built a radiomics model using 17 features selected from a pool of 422 features by a proposed ensemble technique that improved the feature selection stability using a combination of bagging and boosting. This radiomic model achieved an area under the curve of 0.85 and 0.80 for discriminating the two abnormalities on the training and testing data, respectively. In addition, the proposed feature selection technique significantly improved stability when compared to well-established techniques.Discriminating early-stage nasopharyngeal carcinoma (NPC) from benign hyperplasia (BH) on MRI is a challenging but important task for the early detection of NPC in screening programs. Radiomics models have the potential to meet this challenge, but instability in the feature selection step may reduce their reliability. Therefore, in this study, we aim to discriminate between early-stage T1 NPC and BH on MRI using radiomics and propose a method to improve the stability of the feature selection step in the radiomics pipeline. A radiomics model was trained using data from 442 patients (221 early-stage T1 NPC and 221 with BH) scanned at 3T and tested on 213 patients (99 early-stage T1 NPC and 114 BH) scanned at 1.5T. To verify the improvement in feature selection stability, we compared our proposed ensemble technique, which uses a combination of bagging and boosting (BB-RENT), with the well-established elastic net. The proposed radiomics model achieved an area under the curve of 0.85 (95% confidence interval (CI): 0.82–0.89) and 0.80 (95% CI: 0.74–0.86) in discriminating NPC and BH in the 3T training and 1.5T testing cohort, respectively, using 17 features selected from a pool of 422 features by the proposed feature selection technique. BB-RENT showed a better feature selection stability compared to the elastic net (Jaccard index = 0.39 ± 0.14 and 0.24 ± 0.06, respectively; p < 0.001).

On Feature Selection Algorithms and Feature Selection Stability Measures : A Comparative Analysis

On Feature Selection Algorithms and Feature Selection Stability Measures : A Comparative Analysis

Assesing The Stability And Selection Performance Of Feature Selection Methods Under Different Data Complexity

Our study aims to investigate the stability and the selection accuracy of feature selection performance under different data complexity. The motivation behind this investigation is that there are significant contributions in the research community from examining the effect of complex data characteristics such as overlapping classes or non-linearity of the decision boundaries on the classification algorithm's performance; however, relatively few studies have investigated the stability and the selection accuracy of feature selection methods with such data characteristics. Also, this study is interested in investigating the interactive effects of the classes overlapped with other data challenges such as small sample size, high dimensionality, and imbalance classes to provide meaningful insights into the root causes for feature selection methods misdiagnosing the relevant features among different real-world data challenges. This analysis will be extended to real-world data to guide the practitioners and researchers in choosing the correct feature selection methods that are more appropriate for a particular dataset. Our study outcomes indicate that using feature selection techniques with datasets of different characteristics may generate different subsets of features under variations to the training data showing that small sample size and overlapping classes have the highest impact on the stability and selection accuracy of feature selection performance, among other data challenges that have been investigated in this study. Also, in this study, we will provide a survey on the current state of research in the feature selection stability context to highlight the area that requires more attention for other researchers.

Stable feature selection based on brain storm optimisation for high‐dimensional data

Feature selection is a widely used data pre-processing method. However, the research on feature selection stability is very rare. Although there are some related studies that mainly focus on filters rather than wrappers, especially wrappers based on evolutionary algorithms. In this paper, a stable feature selection method called stable brain storm optimisation (SBSO) is proposed. It puts forward a new population initialisation strategy, which treats stable feature selection results as guide information for initialisation and utilises such chaos to initialise population. SBSO also sets up an information archive to store the historical optimal individuals dynamically. A large number of experiments show that SBSO gives excellent classification performance when compared with other methods, and superior feature selection stability when compared with other wrappers.

Benchmark of filter methods for feature selection in high-dimensional gene expression survival data.

Feature selection is crucial for the analysis of high-dimensional data, but benchmark studies for data with a survival outcome are rare. We compare 14 filter methods for feature selection based on 11 high-dimensional gene expression survival data sets. The aim is to provide guidance on the choice of filter methods for other researchers and practitioners. We analyze the accuracy of predictive models that employ the features selected by the filter methods. Also, we consider the run time, the number of selected features for fitting models with high predictive accuracy as well as the feature selection stability. We conclude that the simple variance filter outperforms all other considered filter methods. This filter selects the features with the largest variance and does not take into account the survival outcome. Also, we identify the correlation-adjusted regression scores filter as a more elaborate alternative that allows fitting models with similar predictive accuracy. Additionally, we investigate the filter methods based on feature rankings, finding groups of similar filters.

Individualized real‐time prediction of working memory performance by classifying electroencephalography signals

AbstractAn individualized model that predicts trial‐by‐trial working memory performance is instrumental for personalized interventions. Here, we propose a single‐trial electroencephalography (EEG) classification process predicting individuals' responses, that is, target correct versus target non‐correct during a working memory task, N‐back. We used event‐related (de‐)synchronization (ERD and ERS) prior to an anticipatory cue as features. The proposed comprehensive process addresses single‐trial EEG classification challenges such as temporal overlap between training and testing datasets, feature selection's stability, and significance of the classification accuracy which have been often overlooked. Our model identified for the first time a few (ranged between 4 and 10) brain regions and oscillations where ERD and ERS predicted an individual's performance. Mean (SD) prediction accuracy across 50 participants (mean age [SD] = 28.56 [7.55]) was 69.51% (8.41). Accuracy was significantly above chance in 34 participants. This machine learning‐based approach provides a proof of principle for individualizing EEG targets for potential interventions.

Improving nature-inspired algorithms for feature selection

Selecting highly discriminative features from a whole feature set has become an important research area. Not only can this improve the performance of classification, but it can also decrease the cost of system diagnoses when a large number of noisy, redundant features are excluded. Binary nature-inspired algorithms have been used as a feature selection procedure. Each of these algorithms requires an initial population to be set, and the appropriateness of the initialization plays a key role in the final result. At the stage of population initialization, the positions are initialized randomly by uniform distribution which leads to a high variability of the classification results. To avoid the randomness of the population generated and to take into account the relation between each feature and the class variable, parametric and non-parametric methods, such as the t-test and Wilcoxon rank sum test are proposed as an initial population in the binary nature-inspired algorithms. This modification can help these binary algorithms to enhance global exploration and local exploitation or exhibit a slow convergence speed compared with the standard procedure. The binary bat, gray wolf, and whale algorithms are considered. The performance of our proposed methods is evaluated on ten publicly available datasets with high-dimensional and low-dimensional data. The experimental results and statistical analysis confirm that the performance of our proposed methods compared with the standard algorithms is better in terms of classification accuracy, the number of selected features, running time, and feature selection stability.

Feature selection and threshold method based on fuzzy joint mutual information

Improving classification performance is one of the main challenges in a variety of real-world applications. Unfortunately, classification models are sensitive to undesirable features of data such as redundant and irrelevant features. Feature selection (FS) is a powerful solution to address the negative effect of these features. Among various methods, Feature selection based on mutual information (MI) is an effective method to select the significant features and deny the undesirable ones. Although most of the existing methods can estimate the feature relevancy efficiently, they may find difficulty to estimate the feature redundancy well. This is due to the individual estimation between the candidate feature and the features of pre-selected subset. To address this limitation, this paper introduces a novel feature selection method, called Fuzzy Joint Mutual Information (FJMI). The proposed method also overcomes common limitations as dealing with continuous feature without information loss and returns the best feature subset automatically without a user-defined threshold. To evaluate the effectiveness of the proposed method, FJMI is compared with six conventional and state-of-the-art feature selection methods. The experimental results on 16 benchmark datasets, with moderate size, show a promising improvement by the proposed method in terms of classification performance, feature selection stability, and number of selected features.

Feature Selection Stability and Accuracy of Prediction Models for Genomic Prediction of Residual Feed Intake in Pigs Using Machine Learning.

Feature selection (FS, i.e., selection of a subset of predictor variables) is essential in high-dimensional datasets to prevent overfitting of prediction/classification models and reduce computation time and resources. In genomics, FS allows identifying relevant markers and designing low-density SNP chips to evaluate selection candidates. In this research, several univariate and multivariate FS algorithms combined with various parametric and non-parametric learners were applied to the prediction of feed efficiency in growing pigs from high-dimensional genomic data. The objective was to find the best combination of feature selector, SNP subset size, and learner leading to accurate and stable (i.e., less sensitive to changes in the training data) prediction models. Genomic best linear unbiased prediction (GBLUP) without SNP pre-selection was the benchmark. Three types of FS methods were implemented: (i) filter methods: univariate (univ.dtree, spearcor) or multivariate (cforest, mrmr), with random selection as benchmark; (ii) embedded methods: elastic net and least absolute shrinkage and selection operator (LASSO) regression; (iii) combination of filter and embedded methods. Ridge regression, support vector machine (SVM), and gradient boosting (GB) were applied after pre-selection performed with the filter methods. Data represented 5,708 individual records of residual feed intake to be predicted from the animal’s own genotype. Accuracy (stability of results) was measured as the median (interquartile range) of the Spearman correlation between observed and predicted data in a 10-fold cross-validation. The best prediction in terms of accuracy and stability was obtained with SVM and GB using 500 or more SNPs [0.28 (0.02) and 0.27 (0.04) for SVM and GB with 1,000 SNPs, respectively]. With larger subset sizes (1,000–1,500 SNPs), the filter method had no influence on prediction quality, which was similar to that attained with a random selection. With 50–250 SNPs, the FS method had a huge impact on prediction quality: it was very poor for tree-based methods combined with any learner, but good and similar to what was obtained with larger SNP subsets when spearcor or mrmr were implemented with or without embedded methods. Those filters also led to very stable results, suggesting their potential use for designing low-density SNP chips for genome-based evaluation of feed efficiency.

Stable ant‐antlion optimiser for feature selection on high‐dimensional data

High-dimensional data exists widely in the real world, such as gene, magnetic resonance imaging (MRI), text, web data and so on. Feature selection is an effective and powerful method that is often adopted to reduce dimensions of high-dimensional data for promoting learning algorithm's ability to obtain useful information from them. However, feature selection stability lacks attention for a long time, which plays an important role in getting compelling results. This study proposes a stable feature selection approach called stable ant-antlion optimiser (SALO) that is a modified hybrid evolutionary method combining ant colony optimisation and antlion optimiser. Several relative stable filter methods are employed to rank features as guide information, fisher discriminant rate is used to evaluate features as heuristic information, and F1 indicator and the similarity between feature subset and stable ranked feature list are taken as optimisation objects. Comprehensive experiments show the fantasy stability and excellent classification performance of our proposed sophisticated method.

Stable bagging feature selection on medical data

In the medical field, distinguishing genes that are relevant to a specific disease, let’s say colon cancer, is crucial to finding a cure and understanding its causes and subsequent complications. Usually, medical datasets are comprised of immensely complex dimensions with considerably small sample size. Thus, for domain experts, such as biologists, the task of identifying these genes have become a very challenging one, to say the least. Feature selection is a technique that aims to select these genes, or features in machine learning field with respect to the disease. However, learning from a medical dataset to identify relevant features suffers from the curse-of-dimensionality. Due to a large number of features with a small sample size, the selection usually returns a different subset each time a new sample is introduced into the dataset. This selection instability is intrinsically related to data variance. We assume that reducing data variance improves selection stability. In this paper, we propose an ensemble approach based on the bagging technique to improve feature selection stability in medical datasets via data variance reduction. We conducted an experiment using four microarray datasets each of which suffers from high dimensionality and relatively small sample size. On each dataset, we applied five well-known feature selection algorithms to select varying number of features. The proposed technique shows a significant improvement in selection stability while at least maintaining the classification accuracy. The stability improvement ranges from 20 to 50 percent in all cases. This implies that the likelihood of selecting the same features increased 20 to 50 percent more. This is accompanied with the increase of classification accuracy in most cases, which signifies the stated results of stability.

Feature Selection and Feature Stability Measurement Method for High-Dimensional Small Sample Data Based on Big Data Technology.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Feature selection based on fuzzy joint mutual information maximization.

Nowadays, real-world applications handle a huge amount of data, especially with high-dimension features space. These datasets are a significant challenge for classification systems. Unfortunately, most of the features present are irrelevant or redundant, thus making these systems inefficient and inaccurate. For this reason, many feature selection (FS) methods based on information theory have been introduced to improve the classification performance. However, the current methods have some limitations such as dealing with continuous features, estimating the redundancy relations, and considering the outer-class information. To overcome these limitations, this paper presents a new FS method, called Fuzzy Joint Mutual Information Maximization (FJMIM). The effectiveness of our proposed method is verified by conducting an experimental comparison with nine of conventional and state-of-the-art feature selection methods. Based on 13 benchmark datasets, experimental results confirm that our proposed method leads to promising improvement in classification performance and feature selection stability.

Inferring Protein Sequence-Function Relationships with Large-Scale Positive-Unlabeled Learning.

Machine learning can infer how protein sequence maps to function without requiring a detailed understanding of the underlying physical or biological mechanisms. It is challenging to apply existing supervised learning frameworks to large-scale experimental data generated by deep mutational scanning (DMS) and related methods. DMS data often contain high-dimensional and correlated sequence variables, experimental sampling error and bias, and the presence of missing data. Notably, most DMS data do not contain examples of negative sequences, making it challenging to directly estimate how sequence affects function. Here, we develop a positive-unlabeled (PU) learning framework to infer sequence-function relationships from large-scale DMS data. Our PU learning method displays excellent predictive performance across ten large-scale sequence-function datasets, representing proteins of different folds, functions, and library types. The estimated parameters pinpoint key residues that dictate protein structure and function. Finally, we apply our statistical sequence-function model to design highly stabilized enzymes.