Robust Feature Selection Research Articles

Robust Multi-Label Relief Feature Selection Based on Fuzzy Margin Co-Optimization

Feature extraction is one of the most important tasks in multi-label learning. The performance of multi-label classification can be effectively improved by reducing the dimension of multi-label datasets. Although research on multi-label feature extraction has received extensive attention and made significant progress, further improvement is still necessary. One of the concerns is robustness, where existing multi-label feature extraction algorithms are usually sensitive to noise and outliers. To address this issue, a robust multi-label relief feature selection algorithm based on fuzzy margin co-optimization, called ML-FS-FM, is proposed in this article. Under the multi-label learning framework, the classical fuzzy relief feature weighting algorithm is introduced to ML-FS-FM, which involves the mechanisms of fuzzy feature weighting, fuzzy nearest neighbor and fuzzy instance force coefficient, so as to effectively reduce the influence of noise and outliers, and to extract feature subsets that are beneficial to the classification task. The effectiveness of the proposed algorithm is verified by a large number of experiments on multi-label datasets.

Stand-Alone Composite Attention Network for Concrete Structural Defect Classification

Automation in structural health monitoring involves a critical step of automatic classification of concrete defect images/videos. Although interdisciplinary research community in AI has responded with some progress, immense challenges are still involved because of the predominant overlapping nature of the defect classes, exacerbated by the large variations in their visual appearance. However, current methodologies mostly consider single-class nonoverlapping defects and emphasize equally over the entire image plane; thus unable to focus on specific defect regions for robust feature selection. Thereby, the classification performance gets degraded and the methodologies became less suitable for real-world scenarios. In this work, we propose a novel stand-alone composite attention network that automatically exerts higher emphasis on the defective regions and less emphasis on the healthy regions to recognize multitarget multiclass and single-class concrete structural defects. This architecture stems from the novel GFGA mechanism as the building block to capture minute local features from visually-similar defect classes. We then propose the MSAM that encompasses multiscale discriminative information to capture variations in image properties. The MSAM incorporates BMAM to obtain crucial channel-spatial descriptors making the overall architecture an end-to-end-trainable network. Extensive experimentation and analysis on three large concrete defect datasets show the superiority of our proposed network as compared to the current state-of-the-art methodologies.

Less Is More: Robust and Novel Features for Malicious Domain Detection

Malicious domains are increasingly common and pose a severe cybersecurity threat. Specifically, many types of current cyber attacks use URLs for attack communications (e.g., C&C, phishing, and spear-phishing). Despite the continuous progress in detecting cyber attacks, there are still critical weak spots in the structure of defense mechanisms. Since machine learning has become one of the most prominent malware detection methods, a robust feature selection mechanism is proposed that results in malicious domain detection models that are resistant to evasion attacks. This mechanism exhibits a high performance based on empirical data. This paper makes two main contributions: First, it provides an analysis of robust feature selection based on widely used features in the literature. Note that even though the feature set dimensional space is cut by half, the performance of the classifier is still improved (an increase in the model’s F1-score from 92.92% to 95.81%). Second, it introduces novel features that are robust with regard to the adversary’s manipulation. Based on an extensive evaluation of the different feature sets and commonly used classification models, this paper shows that models based on robust features are resistant to malicious perturbations and concurrently are helpful in classifying non-manipulated data.

Classification of multiple sclerosis patients based on structural disconnection: A robust feature selection approach.

Background and PurposeAlthough structural disconnection represents the hallmark of multiple sclerosis (MS) pathophysiology, classification attempts based on structural connectivity have achieved low accuracy levels. Here, we set out to fill this gap, exploring the performance of supervised classifiers on features derived from microstructure informed tractography and selected applying a novel robust approach.MethodsUsing microstructure informed tractography with diffusion MRI data, we created quantitative connectomes of 55 MS patients and 24 healthy controls. We then used a robust approach—based on two classical methods of feature selection— to select relevant features from three network representations (whole connectivity matrices, node strength, and local efficiency). Classification accuracy of the selected features was tested with five different classifiers, while their meaningfulness was tested via correlation with clinical scales. As a comparison, the same classifiers were run on features selected with the standard procedure in network analysis (thresholding).ResultsOur procedure identified 11 features for the whole net, five for local efficiency, and seven for node strength. For all classifiers, the accuracy was in the range 64.5%‐91.1%, with features extracted from the whole net reaching the maximum, and overcoming results obtained with the standard procedure in all cases. Correlations with clinical scales were identified across functional domains, from motor and cognitive abilities to fatigue and depression.ConclusionApplying a robust feature selection procedure to quantitative structural connectomes, we were able to classify MS patients with excellent accuracy, while providing information on the white matter connections and gray matter regions more affected by MS pathology.

An Efficient Feature Selection Method for Video-Based Activity Recognition Systems

Human activity recognition (HAR) is the examination of gestures and actions of humans from various resources such as depth or RGB cameras. In this work, we have designed a dynamic and robust feature selection algorithm for a HAR system, through which the system accurately recognizes various kinds of activities. In the proposed approach, we employed mutual information algorithm, which selects the prominent features from the extracted features. The proposed algorithm is the expansion of two methods like max-relevance and min-redundancy, respectively. This method has the capability to gather the assets of various extraction algorithms. But the procedure of selection may be unfair due to the dissimilarity between the classification power and redundancy of the features. To resolve this type of unfair selection, we stabilize both parts through the proposed algorithm that has autonomous upper limit of the mutual information function. Likewise, for the feature extraction and recognition, we used the symlet wavelet transform and hidden Markov model, respectively, for action classification. The proposed algorithm has been justified on depth-based database which has thirteen kinds of activities under comprehensive set of experiments. We showed that the proposed feature selection method achieved best classification accuracy against existing works.

Explainable machine learning for knee osteoarthritis diagnosis based on a novel fuzzy feature selection methodology.

Knee Osteoarthritis (ΚΟΑ) is a degenerative joint disease of the knee that results from the progressive loss of cartilage. Due to KOA’s multifactorial nature and the poor understanding of its pathophysiology, there is a need for reliable tools that will reduce diagnostic errors made by clinicians. The existence of public databases has facilitated the advent of advanced analytics in KOA research however the heterogeneity of the available data along with the observed high feature dimensionality make this diagnosis task difficult. The objective of the present study is to provide a robust Feature Selection (FS) methodology that could: (i) handle the multidimensional nature of the available datasets and (ii) alleviate the defectiveness of existing feature selection techniques towards the identification of important risk factors which contribute to KOA diagnosis. For this aim, we used multidimensional data obtained from the Osteoarthritis Initiative database for individuals without or with KOA. The proposed fuzzy ensemble feature selection methodology aggregates the results of several FS algorithms (filter, wrapper and embedded ones) based on fuzzy logic. The effectiveness of the proposed methodology was evaluated using an extensive experimental setup that involved multiple competing FS algorithms and several well-known ML models. A 73.55% classification accuracy was achieved by the best performing model (Random Forest classifier) on a group of twenty-one selected risk factors. Explainability analysis was finally performed to quantify the impact of the selected features on the model’s output thus enhancing our understanding of the rationale behind the decision-making mechanism of the best model.

Intelligent fault diagnosis of wind turbine gearboxes based on refined generalized multi-scale state joint entropy and robust spectral feature selection

Intelligent fault diagnosis of wind turbine gearboxes based on refined generalized multi-scale state joint entropy and robust spectral feature selection

Correction to: A Fused Heterogeneous Deep Neural Network and Robust Feature Selection Framework for Human Actions Recognition

A Correction to this paper has been published: 10.1007/s13369-021-05881-4

Robust multi-class feature selection via l2,0-norm regularization minimization

Feature selection is an important data preprocessing in data mining and machine learning, that can reduce the number of features without deteriorating model’s performance. Recently, sparse regression has received considerable attention in feature selection task due to its good performance. However, because the l2,0-norm regularization term is non-convex, this problem is hard to solve, and most of the existing methods relaxed it by l2,1-norm. Unlike the existing methods, this paper proposes a novel method to solve the l2,0-norm regularized least squares problem directly based on iterative hard thresholding, which can produce exact row-sparsity solution for weights matrix, and features can be selected more precisely. Furthermore, two homotopy strategies are derived to reduce the computational time of the optimization method, which are more practical for real-world applications. The proposed method is verified on eight biological datasets, experimental results show that our method can achieve higher classification accuracy with fewer number of selected features than the approximate convex counterparts and other state-of-the-art feature selection methods.

Development of Data Mining Models Based on Features Ranks Voting (FRV)

Data size plays a significant role in the design and the performance of data mining models. A good feature selection algorithm reduces the problems of big data size and noise due to data redundancy. Features selection algorithms aim at selecting the best features and eliminating unnecessary ones, which in turn simplifies the structure of the data mining model as well as increases its performance. This paper introduces a robust features selection algorithm, named Features Ranking Voting Algorithm FRV. It merges the benefits of the different features selection algorithms to specify the features ranks in the dataset correctly and robustly; based on the feature ranks and voting algorithm. The FRV comprises of three different proposed techniques to select the minimum best feature set, the forward voting technique to select the best high ranks features, the backward voting technique, which drops the low ranks features (low importance feature), and the third technique merges the outputs from the forward and backward techniques to maximize the robustness of the selected features set. Different data mining models were built using obtained selected features sets from applying the proposed FVR on different datasets; to evaluate the success behavior of the proposed FRV. The high performance of these data mining models reflects the success of the proposed FRV algorithm. The FRV performance is compared with other features selection algorithms. It successes to develop data mining models for the Hungarian CAD dataset with Acc. of 96.8%, and with Acc. of 96% for the Z-Alizadeh Sani CAD dataset compared with 83.94% and 92.56% respectively in [48].

Advanced hybrid ensemble gain ratio feature selection model using machine learning for enhanced disease risk prediction

Currently, there is an increased need for employing machine learning (ML) and data mining in the healthcare system domain, applications of which play a pivotal role in providing beneficial knowledge to society by utilizing the available data. There are disease risk prediction models that either do not use feature selection or use traditional feature selection models—filter, wrapper, and embedded—which have limitations such as, not using ML, using single evaluation metric, and working on imbalanced datasets. Apart from this, there is a great scope for enhancement in the prediction performance. To address these issues, an advanced hybrid ensemble gain ratio feature selection (AHEG-FS) model is proposed, which consists of four major feature selection techniques: an ensemble feature selection, a gain ratio feature selection, a backward feature elimination, and an area under the curve (AUC)—an additional evaluation metric of the novel feature reduction along with accuracy, for robust feature selection aimed at effective disease risk prediction. The subsets of important and highly ranked features are obtained with the first two techniques. Then, the proposed model is aligned with nine ML algorithms. Additionally, in the third and fourth techniques of the proposed model, the AUCs are evaluated for the aforementioned ML algorithms and the backward feature elimination is applied to remove the redundant features, resulting in the acquisition of the best subsets of highly contributing features that produce the highest precision results. Thus, four benchmarked heart disease datasets—Cleveland, Hungarian, Statlog, and Switzerland of the University of California, Irvine ML repository—are used, and the results are encouraging. The highest AUC and accuracy of 99.00% and 95.47%, respectively, are achieved, with 46.15% of features reduced. A 6.18% higher accuracy than recent studies was achieved with convergent speed.

MAINE: a web tool for multi-omics feature selection and rule-based data exploration.

SummaryPatient multi-omics datasets are often characterized by a high dimensionality; however, usually only a small fraction of the features is informative, that is change in their value is directly related to the disease outcome or patient survival. In medical sciences, in addition to a robust feature selection procedure, the ability to discover human-readable patterns in the analyzed data is also desirable. To address this need, we created MAINE—Multi-omics Analysis and Exploration. The unique functionality of MAINE is the ability to discover multidimensional dependencies between the selected multi-omics features and event outcome prediction as well as patient survival probability. Learned patterns are visualized in the form of interpretable decision/survival trees and rules.Availability and implementationMAINE is freely available at maine.ibemag.pl as an online web application.Supplementary information Supplementary data are available at Bioinformatics online.

Orthogonally constrained matrix factorization for robust unsupervised feature selection with local preserving

Feature selection is a significant preprocessing technique that involves discarding redundant and irrelevant features, so as to reduce the data dimensionality and build a better understanding of data. Pruning superfluous features tend to build better generalization models while improving the computation efficiency extremely. In practices, obtaining the labels of data is time-consuming and labor-intensive, which brings great challenges for feature selection. In this paper, we present a novel robust unsupervised feature selection method for unlabeled dimensionality reduction, which obtains feature importance information by predicting cluster labels of data with matrix factorization. The orthogonal constraints on two decomposing matrices facilitate achieving more accurate class labels of clusters so as to select features with highly discrimination power. Then, the local preserving term is integrated into the projected data for selecting features with local retention ability. Independent of that, an alternative iterative algorithm is incorporated into the optimization of ℓ2,1-norm objective function for efficient and robust feature selection. Extensive comparative experiments are carried out on six benchmark datasets to evaluate the performance of the proposed method. The results show that our method outperforms several well-known unsupervised feature selection methods in terms of both clustering accuracy and normalized mutual information.

Effective feature learning of multi‐modal genetic and neuroimaging data for prediction of future conversion to Alzheimer’s disease: A machine learning based study

AbstractBackgroundDue to the high dimensionality of Single Nucleotide Polymorphism (SNP) data, current imaging genetic studies of Dementia of Alzheimer’s Type (DAT) usually only select a limited number of candidate SNPs that are reported in the literature, while there might be potentially important AD‐related risk factors in the remaining genes. In this study, we harness a robust feature selection technique that utilizes all available SNPs in the human genome to identify the relevant MRI and genetic features while overcoming feature redundancy.MethodA total of 543 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI) that had both MRI and genetic data available are included in the study. These subjects are divided into seven novel stratified groups [Table 1] based on their longitudinal clinical diagnosis. A subject is categorized as DAT+ if it has a follow‐up diagnosis of DAT, and is categorized as DAT‐ otherwise. We utilize a 2‐stage probabilistic multi‐kernel classifier. In the first stage, using 10‐fold cross‐validation: In each fold we select discriminative features by applying Fisher's Exact test on SNP data and Welch's t‐test on MRI data using subjects in the subgroups with the most certain longitudinal diagnosis (sNC in DAT‐ and sDAT in DAT+). In the second stage, using only the most frequently selected features above, we retrain on 80% of sNC and sDAT, validate on the remaining 20%, and test on subjects in the other subgroups with milder longitudinal diagnosis results.ResultOur study showed that although genetic features have a lower prediction power than MRI features, combining both modalities can improve the prediction of future conversion to AD. For subjects in pNC group, MRI features fail to indicate potential risk of developing AD while genetic data can accurately identify the risk [Figure 1]. Our feature selection method has successfully identified significant AD risk factors for both modalities.ConclusionOur study demonstrated that using effective feature selection methods eliminate the need for heuristic selection of AD‐related genes used for machine‐learning studies for image genetics. Our proposed method reveals genetic risk factors already mentioned in literature as well as some novel risk factors that could advance clinical diagnostics in the future.

Smart Robust Feature Selection (SoFt) for imbalanced and heterogeneous data

Designing a smart and robust predictive model that can deal with imbalanced data and a heterogeneous set of features is paramount to its widespread adoption by practitioners. By smart, we mean the model is either parameter-free or works well with default parameters, avoiding the challenge of parameter tuning. Furthermore, a robust model should consistently achieve high accuracy regardless of any dataset (imbalance, heterogeneous set of features) or domain (such as medical, financial). To this end, a computationally inexpensive and yet robust predictive model named smart robust feature selection (SoFt) is proposed. SoFt involves selecting a learning algorithm and designing a filtering-based feature selection algorithm named multi evaluation criteria and Pareto (MECP). Two state-of-the-art gradient boosting methods (GBMs), CatBoost and H2O GBM, are considered potential candidates for learning algorithms. CatBoost is selected over H2O GBM due to its robustness with both default and tuned parameters. The MECP uses multiple parameter-free feature scores to rank the features. SoFt is validated against CatBoost with a full feature set and wrapper-based CatBoost. SoFt is robust and consistent for imbalanced datasets, i.e., average value and standard deviation of log loss are low across different folds of K-fold cross-validation. Features selected by MECP are also consistent, i.e., features selected by SoFt and wrapper-based CatBoost are consistent across different folds, demonstrating the effectiveness of MECP. For balanced datasets, MECP selects too few features, and hence, the log loss of SoFt is significantly higher than CatBoost with a full feature set.

Feature selection and classification over the network with missing node observations.

Jointly analyzing transcriptomic data and the existing biological networks can yield more robust and informative feature selection results, as well as better understanding of the biological mechanisms. Selecting and classifying node features over genome-scale networks has become increasingly important in genomic biology and genomic medicine. Existing methods have some critical drawbacks. The first is they do not allow flexible modeling of different subtypes of selected nodes. The second is they ignore nodes with missing values, very likely to increase bias in estimation. To address these limitations, we propose a general modeling framework for Bayesian node classification (BNC) with missing values. A new prior model is developed for the class indicators incorporating the network structure. For posterior computation, we resort to the Swendsen-Wang algorithm for efficiently updating class indicators. BNC can naturally handle missing values in the Bayesian modeling framework, which improves the node classification accuracy and reduces the bias in estimating gene effects. We demonstrate the advantages of our methods via extensive simulation studies and the analysis of the cutaneous melanoma dataset from The Cancer Genome Atlas.

Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders

Major complications arise from the recent increase in the amount of high-dimensional data, including high computational costs and memory requirements. Feature selection, which identifies the most relevant and informative attributes of a dataset, has been introduced as a solution to this problem. Most of the existing feature selection methods are computationally inefficient; inefficient algorithms lead to high energy consumption, which is not desirable for devices with limited computational and energy resources. In this paper, a novel and flexible method for unsupervised feature selection is proposed. This method, named QuickSelection (The code is available at: https://github.com/zahraatashgahi/QuickSelection), introduces the strength of the neuron in sparse neural networks as a criterion to measure the feature importance. This criterion, blended with sparsely connected denoising autoencoders trained with the sparse evolutionary training procedure, derives the importance of all input features simultaneously. We implement QuickSelection in a purely sparse manner as opposed to the typical approach of using a binary mask over connections to simulate sparsity. It results in a considerable speed increase and memory reduction. When tested on several benchmark datasets, including five low-dimensional and three high-dimensional datasets, the proposed method is able to achieve the best trade-off of classification and clustering accuracy, running time, and maximum memory usage, among widely used approaches for feature selection. Besides, our proposed method requires the least amount of energy among the state-of-the-art autoencoder-based feature selection methods.

Sparse robust multiview feature selection via adaptive-weighting strategy

Sparse robust multiview feature selection via adaptive-weighting strategy

Prediction of Large Change in Lung Tumor Motion During Treatment Course: An Ensemble Learning Method With Enhanced Positive Predictive Values

During lung cancer radiotherapy, tumor motion magnitude may change, resulting in either inadequate margins or non-ideal dose conformality (margins are unnecessarily too big). To predict large motion change using only patient-specific clinical information and the simulation 4DCT images, we proposed an ensemble learning model (ELM) with robust feature selection (FS). For this specific study, we minimized the false negative prediction so that we will not miss patients who might have a subsequent large change in tumor motion. Thus, we employed positive predictive values (PPV) as the evaluation metric instead of commonly used accuracy scores.Seventeen features were collected including both clinical factors and imaging features. The motion change is defined by motion difference in SI direction between the simulation 4DCT and mid-course repeat 4DCT with the clinical threshold set as 3 mm. The dataset consisted of 140 patients, where 31 showed large tumor motion change. Four hundred seventy-two sets of 4DCT images were analyzed in this study. The ELM adopts 5 conventional predictive models (logistic regression (LR), decision tree (DT), support vector machine (SVM), random forest (RF), and XGBoost (XGB)) as base predictors. Then, the results are employed as inputs for a two-layer neural network to obtain a final prediction. To enhance the feature sensitivity and to reduce data noise, we developed a robust FS strategy with sequential backward selection. Each criterion score is an average from 5-fold cross-valuation, and the target feature of each round is based on 1000 repeated experiments. The peak of the criterion score after sequential removal of the least sensitive features was used to select the optimal feature set.By employing PPV as the criterion function, all the base predictors ranked initial SI tumor motion as the most important feature. ROC curve with selected sensitive features consistently outperformed the one with all features. For ELM, the AUCs (mean std. ± dev.) were 0.63 ± 0.04 when all features were included and increased to 0.72 ± 0.02 for PPV-based selected features only. The prediction comparisons were performed for all base predictors and ELM with 100 repetitions. The PPV results (Table 1) demonstrated the proposed ELM obtained superior performance than each base predictor.We proposed an ELM to identify patients with potential large tumor motion change using only information prior to the first treatment. The ELM showed superior performance with PPV-driven metrics. We also developed a feature selection strategy for each base predictor, and the consistent superiority was demonstrated by ROC results. In future work, additional information from on-treatment evaluation CTs may further improve the prediction.

Assessing Parkinson's Disease at Scale Using Telephone-Recorded Speech: Insights from the Parkinson's Voice Initiative.

Numerous studies have reported on the high accuracy of using voice tasks for the remote detection and monitoring of Parkinson’s Disease (PD). Most of these studies, however, report findings on a small number of voice recordings, often collected under acoustically controlled conditions, and therefore cannot scale at large without specialized equipment. In this study, we aimed to evaluate the potential of using voice as a population-based PD screening tool in resource-constrained settings. Using the standard telephone network, we processed 11,942 sustained vowel /a/ phonations from a US-English cohort comprising 1078 PD and 5453 control participants. We characterized each phonation using 304 dysphonia measures to quantify a range of vocal impairments. Given that this is a highly unbalanced problem, we used the following strategy: we selected a balanced subset (n = 3000 samples) for training and testing using 10-fold cross-validation (CV), and the remaining (unbalanced held-out dataset, n = 8942) samples for further model validation. Using robust feature selection methods we selected 27 dysphonia measures to present into a radial-basis-function support vector machine and demonstrated differentiation of PD participants from controls with 67.43% sensitivity and 67.25% specificity. These findings could help pave the way forward toward the development of an inexpensive, remote, and reliable diagnostic support tool for PD using voice as a digital biomarker.