Feature Ranking Techniques Research Articles

Class-specific feature selection for enhancing explainability in ensemble classification models

AbstractFeature selection techniques aim at finding a relevant subset of features that perform equally or better than the original set of features at explaining the behavior of data. Typically, features are extracted from feature ranking or subset selection techniques, and the performance is measured by classification or regression tasks. However, while selected features may not have equal importance for the task, they do have equal importance for each class. The fundamental idea of the class-specific concept resides in the understanding that the significance of each feature can vary from one class to another. This contrasts with the traditional class-independent approach, which evaluates the importance of attributes collectively for all classes. For example, in tumor prediction scenarios, each type of tumor (class) may be associated with a distinct subset of relevant features. These features possess significant discriminatory power, enabling the differentiation of one tumor type from others (classes). This class-specific perspective offers a more effective approach to classification tasks by recognizing and leveraging the unique characteristics of each class. A novel deep one-versus-each strategy is introduced, which offers advantages from the point of view of explainability (feature selection) and decomposability (classification). In addition, the class-specific relevance matrix is presented, from which some more sophisticated classification schemes can be derived, such as the three-layer class-specific scheme. These schemes have the great advantage to combine independent classification units (e.g., neural networks) that use a reduced number of features to target each class. The potential for further advancements in this area is wide and will open new horizons for exploring novel research directions in interdisciplinary fields, particularly in complex, multiclass hyperdimensional contexts (e.g., in genomics).

HerbMet: Enhancing metabolomics data analysis for accurate identification of Chinese herbal medicines using deep learning.

Chinese herbal medicines have been utilized for thousands of years to prevent and treat diseases. Accurate identification is crucial since their medicinal effects vary between species and varieties. Metabolomics is a promising approach to distinguish herbs. However, current metabolomics data analysis and modeling in Chinese herbal medicines are limited by small sample sizes, high dimensionality, and overfitting. This study aims to use metabolomics data to develop HerbMet, a high-performance artificial intelligence system for accurately identifying Chinese herbal medicines, particularly those from different species of the same genus. We propose HerbMet, an AI-based system for accurately identifying Chinese herbal medicines. HerbMet employs a 1D-ResNet architecture to extract discriminative features from input samples and uses a multilayer perceptron for classification. Additionally, we design the double dropout regularization module to alleviate overfitting and improve model's performance. Compared to 10 commonly used machine learning and deep learning methods, HerbMet achieves superior accuracy and robustness, with an accuracy of 0.9571 and an F1-score of 0.9542 for distinguishing seven similar Panax ginseng species. After feature selection by 25 different feature ranking techniques in combination with prior knowledge, we obtained 100% accuracy and an F1-score for discriminating P. ginseng species. Furthermore, HerbMet exhibits acceptable inference speed and computational costs compared to existing approaches on both CPU and GPU. HerbMet surpasses existing solutions for identifying Chinese herbal medicines species. It is simple to use in real-world scenarios, eliminating the need for feature ranking and selection in classical machine learning-based methods.

Advancing feature ranking with hybrid feature ranking weighted majority model: a weighted majority voting strategy enhanced by the Harris hawks optimizer

Abstract Feature selection (FS) is vital in improving the performance of machine learning (ML) algorithms. Despite its importance, identifying the most important features remains challenging, highlighting the need for advanced optimization techniques. In this study, we propose a novel hybrid feature ranking technique called the Hybrid Feature Ranking Weighted Majority Model (HFRWM2). HFRWM2 combines ML models with the Harris Hawks Optimizer (HHO) metaheuristic. HHO is known for its versatility in addressing various optimization challenges, thanks to its ability to handle continuous, discrete, and combinatorial optimization problems. It achieves a balance between exploration and exploitation by mimicking the cooperative hunting behavior of Harris’s hawks, thus thoroughly exploring the search space and converging toward optimal solutions. Our approach operates in two phases. First, an odd number of ML models, in conjunction with HHO, generate feature encodings along with performance metrics. These encodings are then weighted based on their metrics and vertically aggregated. This process produces feature rankings, facilitating the extraction of the top-K features. The motivation behind our research is 2-fold: to enhance the precision of ML algorithms through optimized FS and to improve the overall efficiency of predictive models. To evaluate the effectiveness of HFRWM2, we conducted rigorous tests on two datasets: “Australian” and “Fertility.” Our findings demonstrate the effectiveness of HFRWM2 in navigating the search space and identifying optimal solutions. We compared HFRWM2 with 12 other feature ranking techniques and found it to outperform them. This superiority was particularly evident in the graphical comparison of the “Australian” dataset, where HFRWM2 showed significant advancements in feature ranking.

Towards Dimension Reduction: A Balanced Relative Discrimination Feature Ranking Technique for Efficient Text Classification (BRDC)

Towards Dimension Reduction: A Balanced Relative Discrimination Feature Ranking Technique for Efficient Text Classification (BRDC)

Integrating fNIRS and machine learning: shedding light on Parkinson's disease detection.

The purpose of this research is to introduce an approach to assist the diagnosis of Parkinson's disease (PD) by classifying functional near-infrared spectroscopy (fNIRS) studies as PD positive or negative. fNIRS is a non-invasive optical signal modality that conveys the brain's hemodynamic response, specifically changes in blood oxygenation in the cerebral cortex; and its potential as a tool to assist PD detection deserves to be explored since it is non-invasive and cost-effective as opposed to other neuroimaging modalities. Besides the integration of fNIRS and machine learning, a contribution of this work is that various approaches were implemented and tested to find the implementation that achieves the highest performance. All the implementations used a logistic regression model for classification. A set of 792 temporal and spectral features were extracted from each participant's fNIRS study. In the two best performing implementations, an ensemble of feature-ranking techniques was used to select a reduced feature subset, which was subsequently reduced with a genetic algorithm. Achieving optimal detection performance, our approach reached 100 % accuracy, precision, and recall, with an F1 score and area under the curve (AUC) of 1, using 14 features. This significantly advances PD diagnosis, highlighting the potential of integrating fNIRS and machine learning for non-invasive PD detection.

Utilizing deep learning, feature ranking, and selection strategies to classify diverse information technology ticketing data effectively

In today's internet world, information technology (IT) ticketing services are potentially increasing across many corporations. Therefore, the automatic classification of IT tickets becomes a significant challenge. Feature selection becomes most important, particularly in data sets with several variables and features. However, enhance classification's precision and performance by stopping insignificant variables. Through our earlier research, we have categorized the unsupervised ticket dataset. As a result, we have converted the dataset into a supervised dataset. In this article, the classification of different IT tickets on Machine learning algorithms, Feature ranking, and feature selection techniques are used to improve the efficiency of machine learning algorithms. However, compared to the machine learning (ML) algorithms, the convolutional neural network (CNN) algorithm provides a better classification of the token IDs and provide better accuracy.

Normalized effect size (NES): a novel feature selection model for Urdu fake news classification

Social media has become an essential source of news for everyday users. However, the rise of fake news on social media has made it more difficult for users to trust the information on these platforms. Most research studies focus on fake news detection in the English language, and only a limited number of studies deal with fake news in resource-poor languages such as Urdu. This article proposes a globally weighted term selection approach named normalized effect size (NES) to select highly discriminative features for Urdu fake news classification. The proposed model is based on the traditional inverse document frequency (TF-IDF) weighting measure. TF-IDF transforms the textual data into a weighted term-document matrix and is usually prone to the curse of dimensionality. Our novel statistical model filters the most discriminative terms to reduce the data’s dimensionality and improve classification accuracy. We compare the proposed approach with the seven well-known feature selection and ranking techniques, namely normalized difference measure (NDM), bi-normal separation (BNS), odds ratio (OR), GINI, distinguished feature selector (DFS), information gain (IG), and Chi square (Chi). Our ensemble-based approach achieves high performance on two benchmark datasets, BET and UFN, achieving an accuracy of 88% and 90%, respectively.

Autism Spectrum Disorder (ASD) Identification Using Feature-Based Machine Learning Classification Model

Autism Spectrum Disorder (ASD) is a developmental disorder that impairs the development of behaviors, communication, and learning abilities. Early detection of ASD helps patients to get beter training to communicate and interact with others. In this study, we identified ASD and non-ASD individuals using machine learning (ML) approaches. We used Gaussian naive Bayes (NB), k-nearest neighbors (KNN), random forest (RF), logistic regression (LR), Gaussian naive Bayes (NB), support vector machine (SVM) with linear basis function and decision tree (DT). We preprocessed the data using the imputation methods, namely linear regression, Mice forest, and Missforest. We selected the important features using the Simultaneous perturbation feature selection and ranking (SpFSR) technique from all 21 ASD features of three datasets combined (N=1,100 individuals) from University California Irvine (UCI) repository. We evaluated the performance of the method's discrimination, calibration, and clinical utility using a stratified 10-fold cross-validation method. We achieved the highest accuracy possible by using SVM with selected the most important 10 features. We observed the integration of imputation using linear regression, SpFSR and SVM as the most effective models, with an accuracy rate of 100% outperformed the previous studies in ASD prediciton

Majority voting based on different feature ranking techniques from gene expression

In bioinformatics studies, many modeling tasks are characterized by high dimensionality, leading to the widespread use of feature selection techniques to reduce dimensionality. There are a multitude of feature selection techniques that have been proposed in the literature, each relying on a single measurement method to select candidate features. This has an impact on the classification performance. To address this issue, we propose a majority voting method that uses five different feature ranking techniques: entropy score, Pearson’s correlation coefficient, Spearman correlation coefficient, Kendall correlation coefficient, and t-test. By using a majority voting approach, only the features that appear in all five ranking methods are selected. This selection process has three key advantages over traditional techniques. Firstly, it is independent of any particular feature ranking method. Secondly, the feature space dimension is significantly reduced compared to other ranking methods. Finally, the performance is improved as the most discriminatory and informative features are selected via the majority voting process. The performance of the proposed method was evaluated using an SVM, and the results were assessed using accuracy, sensitivity, specificity, and AUC on various biomedical datasets. The results demonstrate the superior effectiveness of the proposed method compared to state-of-the-art methods in the literature.

Two phase feature-ranking for new soil dataset for Coxiella burnetii persistence and classification using machine learning models

Coxiella burnetii (Cb) is a hardy, stealth bacterial pathogen lethal for humans and animals. Its tremendous resistance to the environment, ease of propagation, and incredibly low infectious dosage make it an attractive organism for biowarfare. Current research on the classification of Coxiella and features influencing its presence in the soil is generally confined to statistical techniques. Machine learning other than traditional approaches can help us better predict epidemiological modeling for this soil-based pathogen of public significance. We developed a two-phase feature-ranking technique for the pathogen on a new soil feature dataset. The feature ranking applies methods such as ReliefF (RLF), OneR (ONR), and correlation (CR) for the first phase and a combination of techniques utilizing weighted scores to determine the final soil attribute ranks in the second phase. Different classification methods such as Support Vector Machine (SVM), Linear Discriminant Analysis (LDA), Logistic Regression (LR), and Multi-Layer Perceptron (MLP) have been utilized for the classification of soil attribute dataset for Coxiella positive and negative soils. The feature-ranking methods established that potassium, chromium, cadmium, nitrogen, organic matter, and soluble salts are the most significant attributes. At the same time, manganese, clay, phosphorous, copper, and lead are the least contributing soil features for the prevalence of the bacteria. However, potassium is the most influential feature, and manganese is the least significant soil feature. The attribute ranking using RLF generates the most promising results among the ranking methods by generating an accuracy of 80.85% for MLP, 79.79% for LR, and 79.8% for LDA. Overall, SVM and MLP are the best-performing classifiers, where SVM yields an accuracy of 82.98% and 81.91% for attribute ranking by CR and RLF; and MLP generates an accuracy of 76.60% for ONR. Thus, machine models can help us better understand the environment, assisting in the prevalence of bacteria and decreasing the chances of false classification. Subsequently, this can assist in controlling epidemics and alleviating the devastating effect on the socio-economics of society.

Watershed prioritization and decision-making based on weighted sum analysis, feature ranking, and machine learning techniques

Prediction and validation of Compound factors for prioritization of watersheds are an essential application using machine learning (ML) techniques in water resource engineering. The current paper proposes a methodology to derive 14 morphometric and 3 topo-hydrological parameters using remote sensing (RS) and geographical information systems (GIS). Compound factor (CF) values are calculated using weighted sum analysis (WSA), ReliefF, and the Pearson correlation coefficient, and the important parameters are identified. Two machine learning models, multilayer perceptron (MLP) and support vector machine (SVM), are utilized to predict CF values. Predication accuracy of ML models is evaluated with three parameters, mean absolute error (MAE), Pearson correlation coefficient (PCC), and root mean square error (RMSE). It is observed that the maximum value of PCC equal to 1 is achieved through ReliefF and SVM, whereas minimum MAE and RMSE are observed with ReliefF and SVM when Tenfold cross-validation is applied. Since ReliefF shows better results, CF values are calculated and applied to create the watershed. The proposed methodology is helpful for accurately predicting CF values and advantageous to allocating the proper watershed, which will be useful for decision-making and implementation of conservation techniques for soil and water.

Feature Signature Discovery for Autism Detection: An Automated Machine Learning Based Feature Ranking Framework.

Autism spectrum disorder is the most used umbrella term for a myriad of neuro-degenerative/developmental conditions typified by inappropriate social behavior, lack of communication/comprehension skills, and restricted mental and emotional maturity. The intriguing factor of this disorder is attributed to the fact that it can be detected only by close monitoring of developmental milestones after childbirth. Moreover, the exact causes for the occurrence of this neurodevelopmental condition are still unknown. Besides, autism is prevalent across individuals irrespective of ethnicity, genetic/familial history, and economic/educational background. Although research suggests that autism is genetic in nature and early detection of this disorder can greatly enhance the independent lifestyle and societal adaptability of affected individuals, there is still a great dearth of information to support the statement of proven facts and figures. This research work places emphasis on the application of automated machine learning incorporated with feature ranking techniques to generate significant feature signatures for the early detection of autism. Publicly available datasets based on the Q-chat scores of individuals across diverse age groups-toddlers, children, adolescents, and adults have been employed in this study. A machine learning framework based on automated hyperparameter optimization is proposed in this work to rank the potential nonclinical markers for autism. Moreover, this study aimed at ranking the AutoML models based on Mathew's correlation coefficient and balanced accuracy via which nonclinical markers were identified from these datasets. Besides, the feature signatures and their significance in distinguishing between classes are being reported for the first time in autism detection. The proposed framework yielded ∼90% MCC and ∼95% balanced accuracy across all four age groups of autism datasets. Deep learning approaches have yielded a maximum of 92.7% accuracy on the same datasets but are limited in their ability to extract significant markers, have not reported on MCC for unbalanced data, and cannot adapt automatically to new data entries. However, AutoML approaches are more flexible, easier to implement, and provide automated optimization, thereby yielding the highest accuracy with minimal user intervention.

Screening of waterflooding using smart proxy model coupled with deep convolutional neural network

The objective of this work is to identify any efficiency and accuracy improvements in smart proxy modeling (SPM) compared to the existing SPMs in the literature. SPM is a novel methodology which include additional steps in the construction process compared to traditional proxy models (TPM). We discuss the advantages of SPM compared to TPM where SPM implements feature engineering techniques which involves generating new static and dynamic parameters. The new extracted parameters help the model to capture hidden patterns within the parameters, which eventually increase the accuracy of SPMs compared to TPMs.Based on our literature review, we target our investigation into techniques to improve efficiency and accuracy by focusing on sampling (sequential sampling), feature ranking, and underlying model construction (CNN instead of ANN). In existing SPM literature, only one technique is used during each construction step where there are opportunities to explore novel construction steps to improve overall SPM accuracy and efficiency. Sequential sampling helps to construct the SPM with the lowest number of high-fidelity model execution and it avoids resampling, thereby saving time and making the SPM workflow more efficient. The average feature ranking technique described in this work provides a more confident prioritization of input parameters which eventually helps the overall efficiency in the feature selection step. CNN model as the underlying model provides higher accuracy than implemented ANN models in literature. The SPM with ANN underlying model provides an accuracy of 89–92% compared to the 99% and 94% of the CNN technique for the pressure and oil saturation predictions, respectively.In this paper, we construct a grid-based SPM of a Norwegian offshore field undergoing waterflooding. The designed parameters for this case are the individual liquid rate of the producers, and the outputs are individual grid's oil saturation and pressure. It is shown that the final results for screening purposes generated by SPM can confidently be used to mimic the behavior of the numerical simulator. Also, the results of feature ranking illustrate that some of the extracted data used in the SPM construction steps influence the model's outputs, confirming SPM capability.

Decoding the neural signatures of valence and arousal from portable EEG headset.

Emotion classification using electroencephalography (EEG) data and machine learning techniques have been on the rise in the recent past. However, past studies use data from medical-grade EEG setups with long set-up times and environment constraints. This paper focuses on classifying emotions on the valence-arousal plane using various feature extraction, feature selection, and machine learning techniques. We evaluate different feature extraction and selection techniques and propose the optimal set of features and electrodes for emotion recognition. The images from the OASIS image dataset were used to elicit valence and arousal emotions, and the EEG data was recorded using the Emotiv Epoc X mobile EEG headset. The analysis is carried out on publicly available datasets: DEAP and DREAMER for benchmarking. We propose a novel feature ranking technique and incremental learning approach to analyze performance dependence on the number of participants. Leave-one-subject-out cross-validation was carried out to identify subject bias in emotion elicitation patterns. The importance of different electrode locations was calculated, which could be used for designing a headset for emotion recognition. The collected dataset and pipeline are also published. Our study achieved a root mean square score (RMSE) of 0.905 on DREAMER, 1.902 on DEAP, and 2.728 on our dataset for valence label and a score of 0.749 on DREAMER, 1.769 on DEAP, and 2.3 on our proposed dataset for arousal label.

Integration of Nonlinear Dynamics and Machine learning for Diagnostics of a Single-Stage Gear Box


 
 
 The current study concerns diagnostics of a one-stage gear- box based on the integration of physics and machine learn- ing. A physics-based model of this system is developed, then a nonlinear dynamic analysis is performed. The accuracy of the model is validated by comparing fundamental phenomena observed in synthetic and experimental data. To address the diagnostics problem synthetic data are generated for faulty and healthy conditions. Further, physics-informed features are extracted from the phase space of the dynamic system. It is shown that these features are highly informative about the health condition of the system. Also, their advantages over purely statistical features are demonstrated by a feature ranking technique. Subsequently, they are used as inputs in a machine learning model that is developed and optimized for fault diagnostics. The performance of the proposed method is investigated from different aspects, e.g., the accuracy of fault classification, robustness to noise, and generalization to unseen scenarios.
 
 

Machine learning and discriminant function analysis in the formulation of generic models for sex prediction using patella measurements

Sex prediction from bone measurements that display sexual dimorphism is one of the most important aspects of forensic anthropology. Some bones like the skull and pelvis display distinct morphological traits that are based on shape. These morphological traits which are sexually dimorphic across different population groups have been shown to provide an acceptably high degree of accuracy in the prediction of sex. A sample of 100 patella of Mixed Ancestry South Africans (MASA) was collected from the Dart collection. Six parameters: maximum height (maxh), maximum breadth (maxw), maximum thickness (maxt), the height of articular facet (haf), lateral articular facet breadth (lafb), and medial articular facet breath (mafb) were used in this study. Stepwise and direct discriminant function analyses were performed for measurements that exhibited significant differences between male and female mean measurements, and the “leave-one-out” approach was used for validation. Moreover, we have used eight classical machine learning techniques along with feature ranking techniques to identify the best feature combinations for sex prediction. A stacking machine learning technique was trained and validated to classify the sex of the subject. Here, we have used the top performing three ML classifiers as base learners and the predictions of these models were used as inputs to different machine learning classifiers as meta learners to make the final decision. The measurements of the patella of South Africans are sexually dimorphic and this observation is consistent with previous studies on the patella of different countries. The range of average accuracies obtained for pooled multivariate discriminant function equations is 81.9–84.2%, while the stacking ML technique provides 90.8% accuracy which compares well with those presented for previous studies in other parts of the world. In conclusion, the models proposed in this study from measurements of the patella of different population groups in South Africa are useful resent with reasonably high average accuracies.

K-best feature selection and ranking via stochastic approximation

This study presents SPFSR, a novel stochastic approximation approach for performing simultaneous k-best feature ranking (FR) and feature selection (FS) based on Simultaneous Perturbation Stochastic Approximation (SPSA) with Barzilai and Borwein (BB) non-monotone gains. SPFSR is a wrapper-based method which may be used in conjunction with any given classifier or regressor with respect to any suitable corresponding performance metric. Numerical experiments are performed on 47 public datasets which contain both classification and regression problems, with the mean accuracy and R2 reported from four different classifiers and four different regressors respectively. In over 80% of classification experiments and over 85% of regression experiments SPFSR provided a statistically significant improvement or equivalent performance compared to existing, well-known FR techniques. Furthermore, SPFSR obtained a better classification accuracy and R-squared on average compared to utilising the entire feature set.

A Model for Classifying Rice Varieties Grown in Turkey Using Image-Based Morphological Features and Machine Learning

Abstract: Rice is one of the important cereals which feeds more than half of the world's population. It is used frequently in a variety of flavourful recipes. In this work, a dataset of Cammeo and Osmancik species found in Turkey has been selected for the study. It have 3810 samples containing seven morphological features. The feature ranking methods like Fisher score, FSV, infFS, Laplacian, ReliefF, MCFS and MUTinfFS are selected and applied on the above datasets for the purpose of finding key features for proper feature selection. After selecting key feature values, feature vector have been prepared. Then, Support vector machine technique was applied for classification based on the results obtained from feature ranking techniques.

Impact of Healthcare on Stock Market Volatility and Its Predictive Solution Using Improved Neural Network.

The unprecedented Corona Virus Disease (COVID-19) pandemic has put the world in peril and shifted global landscape in unanticipated ways. The SARSCoV2 virus, which caused the COVID-19 outbreak, first appeared in Wuhan, Hubei Province, China, in December 2019 and quickly spread around the world. This pandemic is not only a global health crisis, but it has caused the major global economic depression. As soon as the virus spread, stock market prices plummeted and volatility increased. Predicting the market during this outbreak has been of substantial importance and is the primary motivation to carry out this work. Given the nonlinearity and dynamic nature of stock data, the prediction of stock market is a challenging task. The machine learning models have proven to be a good choice for the development of effective and efficient prediction systems. In recent years, the application of hyperparameter optimization techniques for the development of highly accurate models has increased significantly. In this study, a customized neural network model is proposed and the power of hyperparameter optimization in modelling stock index prices is explored. A novel dataset is generated using nine standard technical indicators and COVID-19 data. In addition, the primary focus is on the importance of selection of optimal features and their preprocessing. The utilization of multiple feature ranking techniques combined with extensive hyperparameter optimization procedures is comprehensive for the prediction of stock index prices. Moreover, the model is evaluated by comparing it with other models, and results indicate that the proposed model outperforms other models. Given the detailed design methodology, preprocessing, exploratory feature analysis, and hyperparameter optimization procedures, this work gives a significant contribution to stock analysis research community during this pandemic.

Proposal of a method to classify female smokers based on data mining techniques

Smoking is considered the main cause of preventable deaths in the world. The level of cigarette consumption can be determined from several attributes such as psychological stress, nicotine dependence, and behavioral characteristics. This paper seeks to contribute to this research field by applying data mining techniques using a methodology based on the Knowledge Discovery in Databases (KDD) process to classify female smokers as “light smokers” or “heavy smokers”. The tests were conducted on four datasets extracted from the National Survey on Drug Use and Health (NSDUH, USA) database. Feature selection and ranking techniques were applied to reduce dimensionality and the resulting dataset was used as input for eight data mining techniques. The best results were obtained with Artificial Neural Networks (ANN), Support Vector Machines (SVM) and Logistic Regression (LR), which had an accuracy of approximately 85% and recall of up to 77%. Precision values were approximately 81% and the Area Under the Receiver Operating Characteristic Curve (AUC-ROC) surpassed 91% in some tests. The features considered most relevant for both classes were identified and discussed. The CIGAGE variable, which is the age when a woman started smoking daily, for example, shows that each year that a woman lives without smoking on a daily basis reduces the chances of her being classified as a heavy smoker by a factor of 0.911. The findings in this paper enable an assessment of the impact of predictive features related to the consumption of industrialized cigarettes, indicating how public policies and financial resources should be applied to combat smoking among women.