Supervised Feature Selection Methods Research Articles

Maximal cliques-based hybrid high-dimensional feature selection with interaction screening for regression

Studies on feature selection have been extensively conducted in the literature, as it plays a significant role in both supervised and unsupervised machine learning tasks. Since the bulk of features in high-dimensional data sets might not be significant, feature selection plays a key role in removing unimportant variables and improving prediction and data analysis performance. Many of the current feature selection methods, meanwhile, become ineffective when used on contemporary datasets, which exhibit an escalating number of features in relation to sample size. This paper introduces a novel supervised feature selection method for regression problems. The proposed algorithm is called maximal Clique with Interaction Screening (ISClique). The ISClique algorithm’s overall structure can be described in two steps. Initially, a filter approach is used to select relevant features from an initial feature space and examine the different interactions between them. This is done using an innovative coefficient based on Kendall’s tau and partial Kendall’s tau. Secondly, the maximal clique strategy is applied as a wrapper to the selected set from the previous step to construct subsets of features. The most optimal subset that minimizes prediction error is selected. The proposed method integrates the advantages of graph theory with feature screening. Additionally, because the criteria employed in developing the ISClique method accommodate variable heterogeneity, this method is equally suitable for classification tasks. The proposed hybrid approach has been evaluated through applications involving various simulation scenarios and real datasets. Experimental findings demonstrate the advantages of ISClique over comparable methods.

Integrating Multi-Omics Using Bayesian Ridge Regression with Iterative Similarity Bagging

Cancer research has increasingly utilized multi-omics analysis in recent decades to obtain biomolecular information from multiple layers, thereby gaining a better understanding of complex biological systems. However, the curse of dimensionality is one of the most significant challenges when handling omics or biological data. Additionally, integrating multi-omics by transforming different omics types into a new representation can reduce a model’s interpretability, as the extracted features may lose the biological context. This paper proposes Iterative Similarity Bagging (ISB), assisted by Bayesian Ridge Regression (BRR). BRR serves as a domain-oriented supervised feature selection method, choosing essential features by calculating the coefficients for each feature. Despite this, the BRR output datasets contain many features, leading to complexity and high dimensionality. To address this, ISB was introduced to dynamically reduce dimensionality and complexity without losing the biological integrity of the omics data, which often occurs with transformation-based integration approaches. The evaluation measures employed were Root Mean Square Error (RMSE), the Pearson Correlation Coefficient (PCC), and the coefficient of determination (R2). The results demonstrate that the proposed method outperforms some current models in terms of regression performance, achieving an RMSE of 0.12, a PCC of 0.879, and an R2 of 0.77 for the CCLE. For the GDSC, it achieved an RMSE of 0.029, a PCC of 0.90, and an R2 of 0.80.

Sparse Trace Ratio LDA for Supervised Feature Selection.

Classification is a fundamental task in the field of data mining. Unfortunately, high-dimensional data often degrade the performance of classification. To solve this problem, dimensionality reduction is usually adopted as an essential preprocessing technique, which can be divided into feature extraction and feature selection. Due to the ability to obtain category discrimination, linear discriminant analysis (LDA) is recognized as a classic feature extraction method for classification. Compared with feature extraction, feature selection has plenty of advantages in many applications. If we can integrate the discrimination of LDA and the advantages of feature selection, it is bound to play an important role in the classification of high-dimensional data. Motivated by the idea, we propose a supervised feature selection method for classification. It combines trace ratio LDA with l2,p -norm regularization and imposes the orthogonal constraint on the projection matrix. The learned row-sparse projection matrix can be used to select discriminative features. Then, we present an optimization algorithm to solve the proposed method. Finally, the extensive experiments on both synthetic and real-world datasets indicate the effectiveness of the proposed method.

Feature Selection With Partial Autoencoding for Zero-Sample Fault Diagnosis

Fault diagnosis is still a challenging task es- pecially for unseen faults, which could happen in the sys- tems. Zero-sample fault diagnosis alleviates this issue by utilizing information from seen faults and attributes defined by domain knowledge. However, existing methods suffer from irrelevant features in the original space while existing supervised feature selection methods yield unsatisfactory performance due to discrepancy caused by domain differ- ence. In this paper, we propose a novel feature selection method for zero-sample fault diagnosis, called concrete partial autoencoder. The concrete partial autoencoder se- lects features beneficial for both seen and unseen faults through striking a balance between classification accuracy and reconstruction errors of selected features. The con- crete partial autoencoder utilizes categorical reparameter- ization to efficiently solve the feature selection problem. The evaluation results on the Tennessee Eastman Process show that the proposed method improves classification ac- curacy and robustness against irrelevant features at zero- sample fault diagnosis.

Feature Selection Based on Intrusive Outliers Rather Than All Instances.

Feature selection (FS) has recently attracted considerable attention in many fields. Highly-overlapping classes and skewed distributions of data within classes have been found in various classification tasks. Most existing FS methods are all instance-based, which ignores the significant differences in characteristics between the particular outliers and the main body of the class, causing confusion for classifiers. In this paper, we propose a novel supervised FS method, Intrusive Outliers-based Feature Selection (IOFS), to find out what kind of outliers lead to misclassification and exploit the characteristics of such outliers. In order to accurately identify the intrusive outliers (IOs), we provide a density-mean center algorithm to obtain the appropriate representative of a class. A special distance threshold is given to obtain the candidate for IOs. Combining with several metrics, mathematical formulations are provided to evaluate the overlapping degree of the intrusive class pairs. Features with high overlapping degrees are assigned to low rankings in IOFS method. An extension of IOFS based on a small number of extreme IOs, called E-IOFS, is also proposed. Three theoretical proofs are provided for the essential theoretical basis of IOFS. Experiments comparing against various state-of-the-art methods on eleven benchmark datasets show that IOFS is rational and effective, especially on the datasets with higher overlapping classes. And E-IOFS almost always outperforms IOFS.

Supervised Feature Selection based on the Law of Total Variance

Feature selection is a fundamental pre-processing step in machine learning that decreases data dimensionality by removing superfluous and irrelevant features. This study proposes a supervised feature selection method based on feature relevance by employing the law of total variance (LTV). Specifically, the LTV is used to quantify the relevance of features by analysing the association between features and class label. Six classifiers were employed to evaluate the performance and reliability of the proposed method pertaining to classification accuracy. The results proved that a feature subset given by the proposed method has the capability to achieve comparable classification accuracy to the full feature set when just half or less than half of the original features are retained. The proposed method was also proven to be versatile as it can achieves adequate classification accuracy with all six classifiers with different learning schemes. In addition, a comparison with a similar type of feature selection method (AmRMR) shows that the proposed method yields a more accurate classification.

A NOVEL ALERT CORRELATION TECHNIQUE FOR FILTERING NETWORK ATTACKS

An alert correlation is a high-level alert evaluation technique for managing large volumes of irrelevant and redundant intrusion alerts raised by Intrusion Detection Systems (IDSs).Recent trends show that pure intrusion detection no longer can satisfy the security needs of organizations. One problem with existing alert correlation techniques is that they group related alerts together without putting their severity into consideration. This paper proposes a novel alert correlation technique that can filter unnecessary and low impact alerts from a large volume of intrusion. The proposed technique is based on a supervised feature selection method that usesclass type to define the correlation between alerts. Alerts of similar class type are identified using a class label. Class types are further classified based on their metric ranks of low, medium and high level. Findings show that the technique is able detect and report high level intrusions.

Supervised feature selection on gene expression microarray datasets using manifold learning

In recent decades, the ultimate output from microarray assay, has produced enormous numbers of microarray datasets, regardless of the used technology. These datasets include complex and high dimensional samples and genes that the number of samples is much smaller than the number of genes (features). Due to the redundant dimensions in these datasets, processing them directly not only leads to poor performance but also increases computation time and memory usage. Feature selection reduces computational expense while improving or maintaining diagnosis accuracy. In this study, we propose a new supervised feature selection method based on a manifold learning approach. We focus in two different directions to address this issue. First, maximum relevancy criterion that achieves by integrating Supervised Laplacian Eigenmaps (S-LE) and a matrix, which can realize the process of feature selection. The applied criterion simultaneously opts the features that make same-class samples closer to each other and ignores the features that cause different-class samples be near. Second, minimum redundancy among selected features by applying the Pearson correlation coefficient. In the test phase, the proposed method is compared with ten state-of-the-art algorithms on seven microarray datasets. Reported results show that the proposed method has more promising performance than the other methods.

Acoustic Structural Integrity Assessment of Ceramics using Supervised Machine Learning and Uncertainty-Based Rejection

Industry and Quality 4.0 pose the opportunity to integrate artificial intelligence-based technology into the quality management of products/services. Particularly, quality control procedures of tableware ceramics require a demanding and faulty human manual (visual and acoustic) inspection. In this paper, we propose an uncertainty-aware automated acoustic inspection using a supervised machine learning model based on a set of novel acoustic features to classify ceramic plates, as cracked and uncracked. We conducted experiments on a dataset of 31 ceramic plates (16 cracked and 15 uncracked), collected in the laboratory. Data quality check and augmentation strategies were also performed, resulting in 2900 samples. The main contributions of this paper are: 1) description of 192 features selected for the acoustic inspection of ceramic plates; 2) comparison of model calibration results regarding three different classifiers; 3) study of different sources of uncertainty for classification with rejection option, through uncertainty quantification measures, and the effect of feature selection on it. We performed two experiments that differ in the usage of a supervised feature selection method. We split the augmented dataset into train/test sets in a proportion of 90/10. The calibrated SVM was selected as the best classifier based on model calibration and cross-validation results and was used in the prediction on the test set. The uncertainty-based rejection improved the train and test sets' classification results. In the experiment with feature selection, the classification performance remained high, while the uncertainty about the predictions and the percentage of rejected samples decreased.

Neurodynamics-driven holistic approaches to semi-supervised feature selection

Feature selection is a crucial part of machine learning and pattern recognition, which aims at selecting a subset of informative features from the original dataset. Because of label information, supervised feature selection performs better than unsupervised feature selection without label information. However, in the presence of a small number of labeled data and a large number of unlabeled data, it is challenging for supervised feature selection methods to select relevant features. In this paper, we propose three neurodynamics-driven holistic approaches to semi-supervised feature selection via semi-supervised feature redundancy minimization and semi-supervised feature relevancy maximization. We first define information-theoretic semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector based on multi-information, unsupervised symmetric uncertainty, and entropy to measure feature redundancy and relevancy. We then formulate a fractional programming problem and an iteratively weighted quadratic programming problem based on the semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector for semi-supervised feature selection. To solve the formulated problems, we delineate three neurodynamic optimization approaches based on two projection neural networks. We elaborate on the experimental results on six benchmark datasets to demonstrate the superior classification performance of the proposed neurodynamic approaches against six existing supervised and semi-supervised feature selection methods.

Feature redundancy assessment framework for subject matter experts

Traditional feature removal techniques focus on showing how well the selected subset of features can perform in terms of model accuracy while neglecting the aspect of eliminating redundant features and incorporating Subject Matter Experts’ (SME) prior knowledge. This is important so that SMEs can leverage their prior knowledge to incorporate actionable or controllable features to build a downstream model with confidence and practical application. Furthermore, feature removal should include evidence on how similar the redundant features are with the selected features. We proposed a framework that incorporates SME prior knowledge to assess/augment the relevancy of the features with respect to the domain-specific problem. First, we rely on the Variance Inflation Factor (VIF) to iteratively remove the redundant features and measure their information loss. The quantifying of information loss will assist the SME in determining the number of features to be selected. Next, Partitions Around Medoids (PAM) is used to cluster redundant features to the closest selected feature. These clusters guide the SME in the augmentation process where the SME can retain, add, or swap the preferred features with those deemed non-redundant by the algorithm. We compared our result based on four commonly used benchmark datasets (Alate Adelges, Sonar, Wisconsin Diagnostic Breast Cancer, and Wine) with the features selected by domain experts, how they are being grouped, and the possible options to perform feature swaps. Our results show the similarity features between redundant features and their corresponding selected features. Also, we have demonstrated that our framework is able to maintain comparable retained information with those supervised feature selection methods, and demonstrate overall higher retained information of up to 3%.

Unsupervised and Supervised Feature Selection for Incomplete Data via L2,1-Norm and Reconstruction Error Minimization

Feature selection has been widely used in machine learning and data mining since it can alleviate the burden of the so-called curse of dimensionality of high-dimensional data. However, in previous works, researchers have designed feature selection methods with the assumption that all the information from a data set can be observed. In this paper, we propose unsupervised and supervised feature selection methods for use with incomplete data, further introducing an L2,1 norm and a reconstruction error minimization method. Specifically, the proposed feature selection objective functions take advantage of an indicator matrix reflecting unobserved information in incomplete data sets, and we present pairwise constraints, minimizing the L2,1-norm-robust loss functionand performing error reconstruction simultaneously. Furthermore, we derive two alternative iterative algorithms to effectively optimize the proposed objective functions and the convergence of the proposed algorithms is proven theoretically. Extensive experimental studies were performed on both real and synthetic incomplete data sets to demonstrate the performance of the proposed methods.

Sorting of Odor Dilutions Is a Meaningful Addition to Assessments of Olfactory Function as Suggested by Machine-Learning-Based Analyses.

Background: The categorization of individuals as normosmic, hyposmic, or anosmic from test results of odor threshold, discrimination, and identification may provide a limited view of the sense of smell. The purpose of this study was to expand the clinical diagnostic repertoire by including additional tests. Methods: A random cohort of n = 135 individuals (83 women and 52 men, aged 21 to 94 years) was tested for odor threshold, discrimination, and identification, plus a distance test, in which the odor of peanut butter is perceived, a sorting task of odor dilutions for phenylethyl alcohol and eugenol, a discrimination test for odorant enantiomers, a lateralization test with eucalyptol, a threshold assessment after 10 min of exposure to phenylethyl alcohol, and a questionnaire on the importance of olfaction. Unsupervised methods were used to detect structure in the olfaction-related data, followed by supervised feature selection methods from statistics and machine learning to identify relevant variables. Results: The structure in the olfaction-related data divided the cohort into two distinct clusters with n = 80 and 55 subjects. Odor threshold, discrimination, and identification did not play a relevant role for cluster assignment, which, on the other hand, depended on performance in the two odor dilution sorting tasks, from which cluster assignment was possible with a median 100-fold cross-validated balanced accuracy of 77–88%. Conclusions: The addition of an odor sorting task with the two proposed odor dilutions to the odor test battery expands the phenotype of olfaction and fits seamlessly into the sensory focus of standard test batteries.

ESRRG, ATP4A, and ATP4B as Diagnostic Biomarkers for Gastric Cancer: A Bioinformatic Analysis Based on Machine Learning.

Based on multiple bioinformatics methods and machine learning techniques, this study was designed to explore potential hub genes of gastric cancer with a diagnostic value. The novel biomarkers were detected through multiple databases of gastric cancer–related genes. The NCBI Gene Expression Omnibus (GEO) database was used to obtain gene expression files. Three hub genes (ESRRG, ATP4A, and ATP4B) were detected through a combination of weighted gene co-expression network analysis (WGCNA), gene–gene interaction network analysis, and supervised feature selection method. GEPIA2 was used to verify the differences in the expression levels of the hub genes in normal and cancer tissues in the RNA-seq levels of Genotype-Tissue Expression (GTEx) and The Cancer Genome Atlas (TCGA) databases. The objectivity of potential hub genes was also verified by immunohistochemistry in the Human Protein Atlas (HPA) database and transcription factor–hub gene regulatory network. Machine learning (ML) methods including data pre-processing, model selection and cross-validation, and performance evaluation were examined on the hub-gene expression profiles in five Gene Expression Omnibus datasets and verified on a GEO external validation (EV) dataset. Six supervised learning models (support vector machine, random forest, k-nearest neighbors, neural network, decision tree, and eXtreme Gradient Boosting) and one semi-supervised learning model (label spreading) were established to evaluate the diagnostic value of biomarkers. Among the six supervised models, the support vector machine (SVM) algorithm was the most effective one according to calculated performance metrics, including 0.93 and 0.99 area under the curve (AUC) scores on the test and external validation datasets, respectively. Furthermore, the semi-supervised model could also successfully learn and predict sample types, achieving a 0.986 AUC score on the EV dataset, even when 10% samples in the five GEO datasets were labeled. In conclusion, three hub genes (ATP4A, ATP4B, and ESRRG) closely related to gastric cancer were mined, based on which the ML diagnostic model of gastric cancer was conducted.

A Novel Deep Learning Method to Predict Lung Cancer Long-Term Survival With Biological Knowledge Incorporated Gene Expression Images and Clinical Data.

Lung cancer is the leading cause of the cancer deaths. Therefore, predicting the survival status of lung cancer patients is of great value. However, the existing methods mainly depend on statistical machine learning (ML) algorithms. Moreover, they are not appropriate for high-dimensionality genomics data, and deep learning (DL), with strong high-dimensional data learning capability, can be used to predict lung cancer survival using genomics data. The Cancer Genome Atlas (TCGA) is a great database that contains many kinds of genomics data for 33 cancer types. With this enormous amount of data, researchers can analyze key factors related to cancer therapy. This paper proposes a novel method to predict lung cancer long-term survival using gene expression data from TCGA. Firstly, we select the most relevant genes to the target problem by the supervised feature selection method called mutual information selector. Secondly, we propose a method to convert gene expression data into two kinds of images with KEGG BRITE and KEGG Pathway data incorporated, so that we could make good use of the convolutional neural network (CNN) model to learn high-level features. Afterwards, we design a CNN-based DL model and added two kinds of clinical data to improve the performance, so that we finally got a multimodal DL model. The generalized experiments results indicated that our method performed much better than the ML models and unimodal DL models. Furthermore, we conduct survival analysis and observe that our model could better divide the samples into high-risk and low-risk groups.

Comparison of five supervised feature selection algorithms leading to top features and gene signatures from multi-omics data in cancer

BackgroundAs many complex omics data have been generated during the last two decades, dimensionality reduction problem has been a challenging issue in better mining such data. The omics data typically consists of many features. Accordingly, many feature selection algorithms have been developed. The performance of those feature selection methods often varies by specific data, making the discovery and interpretation of results challenging.Methods and resultsIn this study, we performed a comprehensive comparative study of five widely used supervised feature selection methods (mRMR, INMIFS, DFS, SVM-RFE-CBR and VWMRmR) for multi-omics datasets. Specifically, we used five representative datasets: gene expression (Exp), exon expression (ExpExon), DNA methylation (hMethyl27), copy number variation (Gistic2), and pathway activity dataset (Paradigm IPLs) from a multi-omics study of acute myeloid leukemia (LAML) from The Cancer Genome Atlas (TCGA). The different feature subsets selected by the aforesaid five different feature selection algorithms are assessed using three evaluation criteria: (1) classification accuracy (Acc), (2) representation entropy (RE) and (3) redundancy rate (RR). Four different classifiers, viz., C4.5, NaiveBayes, KNN, and AdaBoost, were used to measure the classification accuary (Acc) for each selected feature subset. The VWMRmR algorithm obtains the best Acc for three datasets (ExpExon, hMethyl27 and Paradigm IPLs). The VWMRmR algorithm offers the best RR (obtained using normalized mutual information) for three datasets (Exp, Gistic2 and Paradigm IPLs), while it gives the best RR (obtained using Pearson correlation coefficient) for two datasets (Gistic2 and Paradigm IPLs). It also obtains the best RE for three datasets (Exp, Gistic2 and Paradigm IPLs). Overall, the VWMRmR algorithm yields best performance for all three evaluation criteria for majority of the datasets. In addition, we identified signature genes using supervised learning collected from the overlapped top feature set among five feature selection methods. We obtained a 7-gene signature (ZMIZ1, ENG, FGFR1, PAWR, KRT17, MPO and LAT2) for EXP, a 9-gene signature for ExpExon, a 7-gene signature for hMethyl27, one single-gene signature (PIK3CG) for Gistic2 and a 3-gene signature for Paradigm IPLs.ConclusionWe performed a comprehensive comparison of the performance evaluation of five well-known feature selection methods for mining features from various high-dimensional datasets. We identified signature genes using supervised learning for the specific omic data for the disease. The study will help incorporate higher order dependencies among features.

Multi-view learning for lymph node metastasis prediction using tumor and nodal radiomics in gastric cancer

Purpose. This study aims to develop and validate a multi-view learning method by the combination of primary tumor radiomics and lymph node (LN) radiomics for the preoperative prediction of LN status in gastric cancer (GC). Methods. A total of 170 contrast-enhanced abdominal CT images from GC patients were enrolled in this retrospective study. After data preprocessing, two-step feature selection approach including Pearson correlation analysis and supervised feature selection method based on test-time budget (FSBudget) was performed to remove redundance of tumor and LN radiomics features respectively. Two types of discriminative features were then learned by an unsupervised multi-view partial least squares (UMvPLS) for a latent common space on which a logistic regression classifier is trained. Five repeated random hold-out experiments were employed. Results. On 20-dimensional latent common space, area under receiver operating characteristic curve (AUC), precision, accuracy, recall and F1-score are 0.9531 ± 0.0183, 0.9260 ± 0.0184, 0.9136 ± 0.0174, 0.9468 ± 0.0106 and 0.9362 ± 0.0125 for the training cohort respectively, and 0.8984 ± 0.0536, 0.8671 ± 0.0489, 0.8500 ± 0.0599, 0.9118 ± 0.0550 and 0.8882 ± 0.0440 for the validation cohort respectively (reported as mean ± standard deviation). It shows a better discrimination capability than single-view methods, our previous method, and eight baseline methods. When the dimension was reduced to 2, the model not only has effective prediction performance, but also is convenient for data visualization. Conclusions. Our proposed method by integrating radiomics features of primary tumor and LN can be helpful in predicting lymph node metastasis in patients of GC. It shows multi-view learning has great potential for guiding the prognosis and treatment decision-making in GC.

Discriminative feature selection with directional outliers correcting for data classification

With the rapid development of multimedia technologies (e.g. deep learning), Feature Selection (FS) is now playing a critical role in acquiring discriminative features from massive data. Traditional FS methods score feature importance and select the top best features by treating all instances equally; Hence, valuable instances like directional outliers (DOs), which are specific outliers closer to other class centres than to their owns, seldom receive particular attention during feature selection. Based on our observation, DOs derive from “misclassified instances” which lead to misclassification. In this paper, we present a novel supervised feature selection method entitled Feature Selection via Directional Outliers Correcting (FSDOC), for accurate data classification. The proposed FSDOC includes an optimization algorithm to capture DOs, and two correcting algorithms to reasonably capture redundant features by correcting DOs with intraclass deviation minimization and interclass relative distance maximization. We give theoretical guarantees and adequate analysis on all algorithms to show the effectiveness of FSDOC. Extensive experiments on fifteen public datasets, and two case studies of deep features and very-high dimensional Fisher Vector selection, demonstrate the superior performance of FSDOC.

Efficient weakly supervised LIBS feature selection method in quantitative analysis of iron ore slurry

On-stream analysis of the element content in ore slurry plays an important role in the control of the mineral flotation process. Therefore, our laboratory developed a LIBS-based slurry analyzer named LIBSlurry, which can monitor the iron content in slurries in real time. However, achieving high-precision quantitative analysis results of the slurries is challenging. In this paper, a weakly supervised feature selection method named spectral distance variable selection was proposed for the raw spectral data. This method utilizes the prior information that multiple spectra of the same slurry sample have the same reference concentration to assess the important weight of spectral features, and features selected by this prior can avoid over-fitting compared with a traditional wrapper method. The spectral data were collected on-stream of iron ore concentrate slurry samples during the mineral flotation process. The results show that the prediction accuracy is greatly improved compared with the full-spectrum input and other feature selection methods; the root mean square error of the prediction of iron content can be decreased to 0.75%, which helps to realize the successful application of the analyzer.

A Novel Supervised Feature Selection Method Based on Gaussian Distribution

A Novel Supervised Feature Selection Method Based on Gaussian Distribution