Feature Selection Criteria Research Articles

Belief rule learning and reasoning for classification based on fuzzy belief decision tree

The belief rules which extend the classical fuzzy IF-THEN rules with belief consequent parts have been widely used for classifier design due to their capabilities of building linguistic models interpretable to users and addressing various types of uncertainty. However, in the rule learning process, a high number of features generally results in a belief rule base with large size, which degrades both the classification accuracy and the model interpretability. Motivated by this challenge, the decision tree building technique which implements feature selection and model construction jointly is introduced in this paper to learn a compact and accurate belief rule base. To this end, a new fuzzy belief decision tree (FBDT) with fuzzy feature partitions and belief leaf nodes is designed: a fuzzy information gain ratio is first defined as the feature selection criterion for node fuzzy splitting and then the belief distributions are introduced to the leaf nodes to characterize the class uncertainty. Based on the initial rules extracted from the constructed FBDT, a joint optimization objective considering both classification accuracy and model interpretability is then designed to further reduce the rule redundancy. Experimental results based on real datasets show that the proposed FBDT-based classification method has much smaller rule base and better interpretability than other rule-based methods on the premise of competitive accuracy.

Feature selection by replicate reproducibility and non-redundancy.

A fundamental step in many analyses of high-dimensional data is dimension reduction. Two basic approaches are introduction of new synthetic coordinates and selection of extant features. Advantages of the latter include interpretability, simplicity, transferability, and modularity. A common criterion for unsupervized feature selection is variance or dynamic range. However, in practice, it can occur that high-variance features are noisy, that important features have low variance, or that variances are simply not comparable across features because they are measured in unrelated numeric scales or physical units. Moreover, users may want to include measures of signal-to-noise ratio and non-redundancy into feature selection. Here, we introduce the RNR algorithm, which selects features based on (i) the reproducibility of their signal across replicates and (ii) their non-redundancy, measured by linear dependence. It takes as input a typically large set of features measured on a collection of objects with two or more replicates per object. It returns an ordered list of features, i1,i2,…,ik, where feature i1 is the one with the highest reproducibility across replicates, i2 that with the highest reproducibility across replicates after projecting out the dimension spanned by i1, and so on. Applications to microscopy-based imaging of cells and proteomics highlight benefits of the approach. The RNR method is available via Bioconductor (Huber W, Carey VJ, Gentleman R et al. (Orchestrating high-throughput genomic analysis with bioconductor. Nat Methods 2015;12:115-21.) in the R package FeatSeekR. Its source code is also available at https://github.com/tcapraz/FeatSeekR under the GPL-3 open source license.

Wireless communication channel modeling based on ML

Nowadays, wireless communication plays an important role in various fields of our lives, and in order to ensure good communication performance, reliable channel models are important. In this paper, we present a comprehensive survey of Machine Learning (ML) based channel modeling methods to help with the estimation and prediction accuracy on wireless channel modeling. To begin with, we start with a review of the traditional methods of wireless communication channel modeling, basically empirical method and deterministic method. Then, we introduce several ML methods to address limitations of traditional ways will be used for channel modeling, such as Support Vector Machine (SVM), Random Forests, autoencoder, Deep Learning, etc. Lastly, we demonstrate the application of ML methods including deep learning, SVM, and random forests on wireless channel modeling. Through learning the underlying channel distribution, Deep Learning methods have already demonstrated remarkable performance in channel modeling for different scenarios in some previous learning. Also, by training with suitable kernels and conducting sufficient training iterations, optimal hyperplanes can be identified, Support Vector Machines (SVM) can then be utilized to predict channel characteristics based on input features. Random Forests can identify the most relevant features influencing the channel and optimize system design by handling complex and high-dimensional data through iterative feature selection and splitting criteria. These methods achieve competitive accuracy with respect to traditional methods, however, there are still challenges and issues to be addressed in the application of machine learning methods for communication channel modeling.

Identifying unstable CNG repeat loci in the human genome: a heuristic approach and implications for neurological disorders

Tandem nucleotide repeat (TNR) expansions, particularly the CNG nucleotide configuration, are associated with a variety of neurodegenerative disorders. In this study, we aimed to identify novel unstable CNG repeat loci associated with the neurogenetic disorder spinocerebellar ataxia (SCA). Using a computational approach, 15,069 CNG repeat loci in the coding and noncoding regions of the human genome were identified. Based on the feature selection criteria (repeat length >10 and functional location of repeats), we selected 52 repeats for further analysis and evaluated the repeat length variability in 100 control subjects. A subset of 19 CNG loci observed to be highly variable in control subjects was selected for subsequent analysis in 100 individuals with SCA. The genes with these highly variable repeats also exhibited higher gene expression levels in the brain according to the tissue expression dataset (GTEx). No pathogenic expansion events were identified in patient samples, which is a limitation given the size of the patient group examined; however, these loci contain potential risk alleles for expandability. Recent studies have implicated GLS, RAI1, GIPC1, MED15, EP400, MEF2A, and CNKSR2 in neurological diseases, with GLS, GIPC1, MED15, RAI1, and MEF2A sharing the same repeat loci reported in this study. This finding validates the approach of evaluating repeat loci in different populations and their possible implications for human pathologies.

K‐Fuse: Credit card fraud detection based on a classification method with a priori class partitioning and a novel feature selection strategy

AbstractOnline transactions have become the dominant and most popular form of online payment in today's digital economy. Due to the growing popularity of e‐commerce and the convenience it offers, both consumers and businesses are rapidly adopting online transactions. Notably, credit cards have become one of the most popular and standard online payment methods. However, it should be noted that credit card transactions are not without challenges. In particular, detecting and preventing fraudulent transactions is a major concern of the online payment system. It is difficult to find an effective detection model that can detect the new patterns created by fraudsters, due to the constant evolution of their methods to exploit the vulnerability of current security protocols. These fraud patterns are evolving and may not correspond to existing documented models, leading to a reduction in their identification. In addition, the customer's behavior can affect the model detection as it is susceptible to change based on factors such as economic conditions, trends, and individual circumstances. When consumers deviate from their typical behavior, the model may generate false alerts, thereby reducing its ability to differentiate between legitimate and fraudulent transactions. This article presents a new supervised detection model, called K‐Fuse, which introduces an unsupervised phase in order to detect fraud patterns that may correspond to innovative models introduced by fraudsters. K‐Fuse is a supervised classification method that fuses three steps consisting of (i) unsupervised clustering to identify hidden patterns of transactions in a dataset, (ii) a novel feature selection criterion based on the unsupervised results, and (iii) supervised classification to exploit the results of clustering and feature selection to predict new transactions as fraudulent or legitimate.

Bi-LSTM/GRU-based anomaly diagnosis for virtual network function instance

In edge–cloud networks, Network Function Virtualization (NFV) technology provides users with high-performance Virtual Network Functions (VNFs) to replace dedicated hardware devices. In this paper, we focus on the VNF instance (VNFI) anomaly diagnosis (AD). In large-scale production data centers, VNFI anomalies are inevitable, causing degradation in network service performance, while accurate VNFI AD can minimize the losses. The AD task can be abstracted as a multi-classification problem, where the VNFI state is classified as normal or a specific fault type. Existing studies have not proposed mature VNFI AD systems and feature selection criteria, and the accuracy of AD tasks can still be improved. To address these gaps, we first design the Distributed Online Anomaly Diagnosis and Broadcasting (DOADB) system, providing online VNFI AD daemons, offline parameter training, and anomaly broadcasting. Then, we establish VNFI feature selection criteria and construct a compelling VNFI feature dataset containing simulated and real-time data for each VNFI. Based on the DOADB system and feature dataset, we adopt the Bi-LSTM and Bi-GRU networks to realize the AD multi-classification algorithm, leveraging the bidirectional network structure and long-term memory to enhance AD accuracy. Experimental results demonstrate that the proposed system and algorithm can provide effective AD daemons for VNFIs and outperform existing algorithms in both accuracy and macro F1-score metrics.

Fractal-Based Multi-Criteria Feature Selection to Enhance Predictive Capability of AI-Driven Mineral Prospectivity Mapping

AI-driven mineral prospectivity mapping (MPM) is a valid and increasingly accepted tool for delineating the targets of mineral exploration, but it suffers from noisy and unrepresentative input features. In this study, a set of fractal and multifractal methods, including box-counting calculation, concentration–area fractal modeling, and multifractal analyses, were employed to excavate the underlying nonlinear mineralization-related information from geological features. Based on these methods, multiple feature selection criteria, namely prediction–area plot, K-means clustering, information gain, chi-square, and the Pearson correlation coefficient, were jointly applied to rank the relative importance of ore-related features and their fractal representations, so as to choose the optimal input feature dataset readily used for training predictive AI models. The results indicate that fault density, the multifractal spectrum width (∆α) of the Yanshanian intrusions, information dimension (D1) of magnetic anomalies, correlation dimension (D2) of iron-oxide alteration, and the D2 of argillic alteration serve as the most effective predictor features representative of the corresponding ore-controlling elements. The comparative results of the model assessment suggest that all the AI models trained by the fractal datasets outperform their counterparts trained by raw datasets, demonstrating a significant improvement in the predictive capability of fractal-trained AI models in terms of both classification accuracy and predictive efficiency. A Shapley additive explanation was employed to trace the contributions of these features and to explain the modeling results, which imply that fractal representations provide more discriminative and definitive feature values that enhance the cognitive capability of AI models trained by these data, thereby improving their predictive performance, especially for those indirect predictor features that show subtle correlations with mineralization in the raw dataset. In addition, fractal-trained models can benefit practical mineral exploration by outputting low-risk exploration targets that achieve higher capturing efficiency and by providing new mineralization clues extracted from remote sensing data. This study demonstrates that the fractal representations of geological features filtered by multi-criteria feature selection can provide a feasible and promising means of improving the predictive capability of AI-driven MPM.

Enhancing land cover object classification in hyperspectral imagery through an efficient spectral-spatial feature learning approach.

The classification of land cover objects in hyperspectral imagery (HSI) has significantly advanced due to the development of convolutional neural networks (CNNs). However, challenges such as limited training data and high dimensionality negatively impact classification performance. Traditional CNN-based methods predominantly utilize 2D CNNs for feature extraction, which inadequately exploit the inter-band correlations in HSIs. While 3D CNNs can capture joint spectral-spatial information, they often encounter issues related to network depth and complexity. To address these issues, we propose an innovative land cover object classification approach in HSIs that integrates segmented principal component analysis (Seg-PCA) with hybrid 3D-2D CNNs. Our approach leverages Seg-PCA for effective feature extraction and employs the minimum-redundancy maximum relevance (mRMR) criterion for feature selection. By combining the strengths of both 3D and 2D CNNs, our method efficiently extracts spectral-spatial features. These features are then processed through fully connected dense layers and a softmax layer for classification. Extensive experiments on three widely used HSI datasets demonstrate that our method consistently outperforms existing state-of-the-art techniques in classification performance. These results highlight the efficacy of our approach and its potential to significantly enhance the classification of land cover objects in hyperspectral imagery.

Multi-ensemble machine learning framework for omics data integration: A case study using breast cancer samples

Integration of voluminous omics data aids to unravel biological complexities associated with different disease phenotypes. Machine learning (ML) approaches provide insightful techniques for systematic multi-omics data integration. In this study, survival prediction of breast cancer patients was undertaken using omics data of 302 female patients from The Cancer Genome Atlas (TCGA). The data included gene expression, miRNA expression, DNA methylation and copy number variation. Three computational multi-ensemble ML pipelines were tested using Support Vector Machine (SVM), Random Forest (RF) and Partial Least Squares-Discriminant Analysis (PLS-DA) algorithms. To overcome the limitations associated with univariate feature selection criteria, the ML pipelines were built along with latent factors obtained by multivariate dimension reduction method. This facilitated investigation of background genetic networks and identification of potential hub genes. Analysis of the results obtained revealed that SVM with PLS-DA method (integrated with gene expression, DNA methylation, and miRNA expression modalities) was the best-performing model with an Area Under Curve (AUC) of 89% and an accuracy of 83% for survival prediction. This study not only corroborated previously reported breast cancer-specific prognostic biomarkers but also predicted additional potential biomarkers. The work demonstrates the effective use of a multi-ensemble ML model with efficient feature selection methods as a robust protocol for cancer genotype to phenotype correlation.

Evaluation of Feature Selection and Pre-Processing Techniques for Ethylene Glycol-Water Ratio Classification in Process Thermostat

With the focus in the realm of automotive testing, process thermostats play a vital role in providing the required operating environment. These process thermostats, with the operating medium of an ethylene glycol-water ratio, play a crucial role in terms of controlling their thermal properties. With an emphasis on identifying the best preprocessing method for classifying these ratios, this study utilises a detailed comparison of statistical methods and the wrapper method-based Genetic Algorithm as a search method for the most relevant feature selections. Given the huge number of existing sensor parameters in the system, thereby emphasising the importance of feature selection criteria for effective analysis and model training. Furthermore, a random forest-based classifier is used with these parameters to predict the accurate ethylene glycol to water ratio.

On Exact Feature Screening in Ultrahigh-Dimensional Binary Classification

We propose a new model-free feature screening method based on energy distances for ultrahigh-dimensional binary classification problems. With a high probability, the proposed method retains only relevant features after discarding all the noise variables. The proposed screening method is also extended to identify pairs of variables that are marginally undetectable but have differences in their joint distributions. Finally, we build a classifier that maintains coherence between the proposed feature selection criteria and discrimination method, and also establish its risk consistency. An extensive numerical study with simulated and real benchmark datasets shows clear and convincing advantages of our proposed method over the state-of-the-art methods. Supplementary materials for this article are available online.

Enhancement of Backward Feature Elimination as a Pre-Processing Method for K Nearest Neighbor Algorithm Applied to Insurance Fraud Detection

This study presents novel enhancements to the Backward Feature Elimination (BFE) method for improved insurance fraud detection using the K-Nearest Neighbor (KNN) algorithm. The research addresses issues inherent in the baseline BFE process, such as the over-reliance on p-values, the potential for misleading results, and suboptimal feature selection leading to overfitting. To address these, the study integrates confidence intervals and feature importance into the BFE process, establishing a more robust and reliable criterion for feature selection. Moreover, feature engineering techniques are introduced during preprocessing to enhance model performance. The modified BFE method demonstrates superior performance over the baseline model regarding the recall, precision, and F1 score. Stratified K-Fold Cross-Validation, ROC-AUC Score, and Coefficient of Variation (CV) confirm the consistency and robustness of the enhanced model across varying data subsets. These innovations offer a comprehensive and reliable solution to feature selection in the BFE method, applied to the KNN model for effective insurance fraud detection. The study mitigates the issues related to p-value dependence and boosts model performance, paving the way for more accurate and robust fraud detection systems.

A Possibilistic Information Fusion-Based Unsupervised Feature Selection Method Using Information Quality Measures

The main goal of most information quality (IQ)-based measures is to combine data provided by multiple information sources to enhance the quality of information essential for decision makers to perform their tasks. However, there is few work to fuse multi-source information from the perspective of possibility distribution (PD) and use IQ as the evaluation criteria for feature selection. The PD is one of important concepts in possibility theory, which is a generally acknowledged method for describing a kind of uncertain knowledge. In this paper, we propose a novel representation model of possibility distributions based on fuzzy multisets, namely, a possibility distribution information system (PDIS). Then, several IQ measures are defined in the PDIS, including Gini entropy, compatibility, conflict, credibility and separability degrees. In view of this, a minimal-separability-minimal-uncertainty-based unsupervised feature selection algorithm (UmSMU) is designed. The proposed UmSMU can sufficiently fuse multiple possibilistic information. Meanwhile, the selected features maintain as much information as possible while minimizing the uncertainty of information. The experimental results show that the proposed algorithm performs well, especially when it comes to selecting fewer features and improving performance.

Distance Correlation-Based Feature Selection in Random Forest.

The Pearson correlation coefficient (ρ) is a commonly used measure of correlation, but it has limitations as it only measures the linear relationship between two numerical variables. The distance correlation measures all types of dependencies between random vectors X and Y in arbitrary dimensions, not just the linear ones. In this paper, we propose a filter method that utilizes distance correlation as a criterion for feature selection in Random Forest regression. We conduct extensive simulation studies to evaluate its performance compared to existing methods under various data settings, in terms of the prediction mean squared error. The results show that our proposed method is competitive with existing methods and outperforms all other methods in high-dimensional (p≥300) nonlinearly related data sets. The applicability of the proposed method is also illustrated by two real data applications.

Water Quality Prediction Based on Machine Learning and Comprehensive Weighting Methods.

In the context of escalating global environmental concerns, the importance of preserving water resources and upholding ecological equilibrium has become increasingly apparent. As a result, the monitoring and prediction of water quality have emerged as vital tasks in achieving these objectives. However, ensuring the accuracy and dependability of water quality prediction has proven to be a challenging endeavor. To address this issue, this study proposes a comprehensive weight-based approach that combines entropy weighting with the Pearson correlation coefficient to select crucial features in water quality prediction. This approach effectively considers both feature correlation and information content, avoiding excessive reliance on a single criterion for feature selection. Through the utilization of this comprehensive approach, a comprehensive evaluation of the contribution and importance of the features was achieved, thereby minimizing subjective bias and uncertainty. By striking a balance among various factors, features with stronger correlation and greater information content can be selected, leading to improved accuracy and robustness in the feature-selection process. Furthermore, this study explored several machine learning models for water quality prediction, including Support Vector Machines (SVMs), Multilayer Perceptron (MLP), Random Forest (RF), XGBoost, and Long Short-Term Memory (LSTM). SVM exhibited commendable performance in predicting Dissolved Oxygen (DO), showcasing excellent generalization capabilities and high prediction accuracy. MLP demonstrated its strength in nonlinear modeling and performed well in predicting multiple water quality parameters. Conversely, the RF and XGBoost models exhibited relatively inferior performance in water quality prediction. In contrast, the LSTM model, a recurrent neural network specialized in processing time series data, demonstrated exceptional abilities in water quality prediction. It effectively captured the dynamic patterns present in time series data, offering stable and accurate predictions for various water quality parameters.

Improving hyperspectral image classification through spectral-spatial feature reduction with a hybrid approach and deep learning

ABSTRACT The Hyperspectral Image (HSI) is a great source of information for observing the earth’s elements due to its numerous narrow and continuous spectral wavelength bands. There are some key difficulties, such as the fact that the image bands are spectrally and spatially highly correlated, and the ‘curse of dimensionality’ in using the original HSI for classification. Band (dimensionality/feature) reduction is necessary to improve classification performance through feature extraction and selection. As such, this paper proposes a hybrid HSI classification paradigm, termed Hybrid-2DNET. Specifically, our proposed Hybrid-2DNET incorporates factor analysis-based feature extraction along the Minimum-Redundancy-Maximum-Relevance (mRMR)-based feature selection criterion and a 2D-wavelet Convolutional Neural Network (CNN) to reduce both spectral and spatial dimensionalities. The experimental analyses performed on three remote sensings HSI datasets, namely Indian Pines, University of Pavia, and Salinas Scene, manifest that our proposed Hybrid-2DNET outperforms the handcrafted and deep learning methods.

Super ensemble based streamflow simulation using multi-source remote sensing and ground gauged rainfall data fusion

Traditional data-driven streamflow predictions usually apply a single model with inconsistent performance in different variability conditions. These days model ensembles or merging the benefits of different models without losing the general character of the data are becoming a trend in hydrology. This study compared three super ensemble learners with eight base models. Twelve years of monthly rolled daily time series data in three river catchments of Ethiopia (Borkena watershed: Awash river basin), (Gummera watershed: Abay river basin), and (Sore watershed: Baro Akobo river basin) is used for single-step daily streamflow simulation using previous thirty-day input timesteps. Five input scenarios are applied: three vegetation indices, three remote sensing-based precipitation products, ground-gauged rainfall, all fused inputs, and selected inputs with the Recursive Feature Elimination (RFE) algorithm. The time series is then divided into training and testing datasets with a ratio of 80:20. The performance of the proposed models was evaluated using the Root Mean Squared Error (RMSE), coefficient of determination (R2), Mean Absolute Error (MAE), and Median Absolute Error (MEDAE). Finally, the result is presented with the corresponding five input scenarios. The catchment's and input scenario's average performance indicated that the three super ensemble learners outperformed the eight base models with relatively stable performance. The top-ranked WASE model exceeded the linear regression baseline by 13.3%. XGB, CNN-GRU, and LSTM proved the highest performance of the base models. This study also revealed that LSTM's key downside is its performance drop in the absence of feature selection criteria. In comparison, XGB showed its superior performance after controlling redundant inputs internally. Moreover, this study uniquely highlights the potential of remote sensing-based vegetation indices in the science of data-driven streamflow modelling for non-gauged catchments with no meteorological time series.

A Novel Hybrid Approach for Classifying Osteosarcoma Using Deep Feature Extraction and Multilayer Perceptron.

Osteosarcoma is the most common type of bone cancer that tends to occur in teenagers and young adults. Due to crowded context, inter-class similarity, inter-class variation, and noise in H&E-stained (hematoxylin and eosin stain) histology tissue, pathologists frequently face difficulty in osteosarcoma tumor classification. In this paper, we introduced a hybrid framework for improving the efficiency of three types of osteosarcoma tumor (nontumor, necrosis, and viable tumor) classification by merging different types of CNN-based architectures with a multilayer perceptron (MLP) algorithm on the WSI (whole slide images) dataset. We performed various kinds of preprocessing on the WSI images. Then, five pre-trained CNN models were trained with multiple parameter settings to extract insightful features via transfer learning, where convolution combined with pooling was utilized as a feature extractor. For feature selection, a decision tree-based RFE was designed to recursively eliminate less significant features to improve the model generalization performance for accurate prediction. Here, a decision tree was used as an estimator to select the different features. Finally, a modified MLP classifier was employed to classify binary and multiclass types of osteosarcoma under the five-fold CV to assess the robustness of our proposed hybrid model. Moreover, the feature selection criteria were analyzed to select the optimal one based on their execution time and accuracy. The proposed model achieved an accuracy of 95.2% for multiclass classification and 99.4% for binary classification. Experimental findings indicate that our proposed model significantly outperforms existing methods; therefore, this model could be applicable to support doctors in osteosarcoma diagnosis in clinics. In addition, our proposed model is integrated into a web application using the FastAPI web framework to provide a real-time prediction.

New criteria for wrapper feature selection to enhance bearing fault classification

Classification is a critical task in many fields, including signal processing and data analysis. The accuracy and stability of classification results can be improved by selecting the most relevant features from the data. In this paper, a new criterion for feature selection using wrapper method is proposed, which is based on the evaluation of the classification results according to the accuracy and stability (standard deviation) of each class and the number of selected features. The proposed method is evaluated using Random Forest (RF) and Ant Colony Optimization (ACO) algorithms on a benchmark dataset. Results show that the proposed method outperforms classical feature selection methods in terms of accuracy and stability of classification results, especially for the difficult-to-classify combined damage class. This study demonstrates the effectiveness of the proposed new wrapper feature selection criterion to improve the performance of classification algorithms with higher stability (STD: C1 = 0.5, C2 = 0.8, C3 = 0.6, C4 = 1.8) and better accuracy (average C1 = 98.5%, C2 = 96.6%, C3 = 9.5%, C4 = 93) for the both; the statoric current and the vibration signal compared to other techniques. Machine learning methods had proven their efficiency in time-varying machines fault diagnosis when taking vibration signals and statoric currents extracted features as inputs. However, the use of the both demonstrated a higher robustness and a remarkable superiority.

Cost-constrained feature selection in multilabel classification using an information-theoretic approach

Feature selection is one of the key steps in building a predictive model in multi-label classification. However, most of the existing methods do not take into account information about the costs associated with considered features, such as the costs of performing diagnostic medical tests. We consider a problem of cost-constrained multilabel feature selection, which aims to select a feature subset relevant to multiple labels while satisfying a user-specific maximal admissible budget. This approach allows for building a model with high predictive power, for which the cost of making a prediction for a single instance does not exceed the user-specified budget. In this problem, the balance between the feature subset relevance and its cost should be considered concurrently, which is nontrivial in practice because their optimal balance is unknown. In this paper, we propose a novel criterion for cost-constrained multilabel feature selection that combines the relevance and cost of the candidate feature. The relevance measure is derived using the lower bound of the mutual information between the feature subset and label vector. Moreover, we propose an effective method for determining the cost-factor value that controls the trade-off between relevancy and cost. The experimental results on multilabel datasets with various characteristics demonstrate the superiority of the proposed method over conventional methods.