Wrapper Feature Selection Method Research Articles

ADMET evaluation in drug discovery. 20. Prediction of breast cancer resistance protein inhibition through machine learning

Breast cancer resistance protein (BCRP/ABCG2), an ATP-binding cassette (ABC) efflux transporter, plays a critical role in multi-drug resistance (MDR) to anti-cancer drugs and drug–drug interactions. The prediction of BCRP inhibition can facilitate evaluating potential drug resistance and drug–drug interactions in early stage of drug discovery. Here we reported a structurally diverse dataset consisting of 1098 BCRP inhibitors and 1701 non-inhibitors. Analysis of various physicochemical properties illustrates that BCRP inhibitors are more hydrophobic and aromatic than non-inhibitors. We then developed a series of quantitative structure–activity relationship (QSAR) models to discriminate between BCRP inhibitors and non-inhibitors. The optimal feature subset was determined by a wrapper feature selection method named rfSA (simulated annealing algorithm coupled with random forest), and the classification models were established by using seven machine learning approaches based on the optimal feature subset, including a deep learning method, two ensemble learning methods, and four classical machine learning methods. The statistical results demonstrated that three methods, including support vector machine (SVM), deep neural networks (DNN) and extreme gradient boosting (XGBoost), outperformed the others, and the SVM classifier yielded the best predictions (MCC = 0.812 and AUC = 0.958 for the test set). Then, a perturbation-based model-agnostic method was used to interpret our models and analyze the representative features for different models. The application domain analysis demonstrated the prediction reliability of our models. Moreover, the important structural fragments related to BCRP inhibition were identified by the information gain (IG) method along with the frequency analysis. In conclusion, we believe that the classification models developed in this study can be regarded as simple and accurate tools to distinguish BCRP inhibitors from non-inhibitors in drug design and discovery pipelines.

Feature selection based multivariate time series forecasting: An application to antibiotic resistance outbreaks prediction.

Antimicrobial resistance has become one of the most important health problems and global action plans have been proposed globally. Prevention plays a key role in these actions plan and, in this context, we propose the use of Artificial Intelligence, specifically Time Series Forecasting techniques, for predicting future outbreaks of Methicillin-resistant Staphylococcus aureus (MRSA). Infection incidence forecasting is approached as a Feature Selection based Time Series Forecasting problem using multivariate time series composed of incidence of Staphylococcus aureus Methicillin-sensible and MRSA infections, influenza incidence and total days of therapy of both of Levofloxacin and Oseltamivir antimicrobials. Data were collected from the University Hospital of Getafe (Spain) from January 2009 to January 2018, using months as time granularity. The main contributions of the work are the following: the applications of wrapper feature selection methods where the search strategy is based on multi-objective evolutionary algorithms (MOEA) along with evaluators based on the most powerful state-of-the-art regression algorithms. The performance of the feature selection methods has been measured using the root mean square error (RMSE) and mean absolute error (MAE) performance metrics. A novel multi-criteria decision-making process is proposed in order to select the most satisfactory forecasting model, using the metrics previously mentioned, as well as the slopes of model prediction lines in the 1, 2 and 3 steps-ahead predictions. The multi-criteria decision-making process is applied to the best models resulting from a ranking of databases and regression algorithms obtained through multiple statistical tests. Finally, to the best of our knowledge, this is the first time that a feature selection based multivariate time series methodology is proposed for antibiotic resistance forecasting. Final results show that the best model according to the proposed multi-criteria decision making process provides a RMSE = (0.1349, 0.1304, 0.1325) and a MAE = (0.1003, 0.096, 0.0987) for 1, 2, and 3 steps-ahead predictions.

Using Machine-Learning Algorithms for Eutrophication Modeling: Case Study of Mar Menor Lagoon (Spain).

The Mar Menor is a hypersaline coastal lagoon with high environmental value and a characteristic example of a highly anthropized hydro-ecosystem located in the southeast of Spain. An unprecedented eutrophication crisis in 2016 and 2019 with abrupt changes in the quality of its waters caused a great social alarm. Understanding and modeling the level of a eutrophication indicator, such as chlorophyll-a (Chl-a), benefits the management of this complex system. In this study, we investigate the potential machine learning (ML) methods to predict the level of Chl-a. Particularly, Multilayer Neural Networks (MLNNs) and Support Vector Regressions (SVRs) are evaluated using as a target dataset information of up to nine different water quality parameters. The most relevant input combinations were extracted using wrapper feature selection methods which simplified the structure of the model, resulting in a more accurate and efficient procedure. Although the performance in the validation phase showed that SVR models obtained better results than MLNNs, experimental results indicated that both ML algorithms provide satisfactory results in the prediction of Chl-a concentration, reaching up to 0.7 R2CV (cross-validated coefficient of determination) for the best-fit models.

Unobtrusive monitoring of behavior and movement patterns to detect clinical depression severity level via smartphone.

The number of individuals with mental disorders is increasing and they are commonly found among individuals who avoid social interaction and like to live alone. Amongst such mental health disorders is depression which is both common and serious. The present paper introduces a method to assess the depression level of an individual using a smartphone by monitoring their daily activities. The time domain characteristics from a smartphone acceleration sensor were used alongside a vector machine algorithm to classify physical activities. Additionally, the geographical location information was clustered using a smartphone GPS sensor to simplify movement patterns. A total of 12 features were extracted from individuals' physical activity and movement patterns and were analyzed alongside their weekly depression scores using the nine-item Patient Health Questionnaire. Using a wrapper feature selection method, a subset of features was selected and applied to a linear regression model to estimate the depression score. The support vector machine algorithm was then used to classify the depression severity level among individuals (absence, moderate, severe) and had an accuracy of 87.2% in severe depression cases which outperformed other classification models including the k-nearest neighbor and artificial neural network. This method of identifying depression is a cost-effective solution for long-term use and can monitor individuals for depression without invading their personal space or creating other day-to-day disturbances.

A real-time botnet detection model based on an efficient wrapper feature selection method

Botnets are one of the most widespread and serious threats of cybersecurity that have infected millions of computers around the world over the past few years. Previous research has shown that machine learning methods can accurately detect botnet attacks. However, these methods often do not address the problem of real-time botnet detection, which is one of the main challenges in this area and is essential to prevent the damage caused by botnet attacks. This paper aims to present an efficient real-time model for botnet detection. In the proposed method, a subset of the effective features in detecting the bot traffic is initially selected using the world competitive contests algorithm. Then, based on the selected features, a support vector machine model is created offline to detect real-time bot traffic from the normal one. The test results show that the proposed method can detect botnets with 95% accuracy and outperforms other methods.

A real-time botnet detection model based on an efficient wrapper feature selection method

Botnets are one of the most widespread and serious threats of cybersecurity that have infected millions of computers around the world over the past few years. Previous research has shown that machine learning methods can accurately detect botnet attacks. However, these methods often do not address the problem of real-time botnet detection, which is one of the main challenges in this area and is essential to prevent the damage caused by botnet attacks. This paper aims to present an efficient real-time model for botnet detection. In the proposed method, a subset of the effective features in detecting the bot traffic is initially selected using the world competitive contests algorithm. Then, based on the selected features, a support vector machine model is created offline to detect real-time bot traffic from the normal one. The test results show that the proposed method can detect botnets with 95% accuracy and outperforms other methods.

ELMO: An Efficient Logistic Regression-Based Multi-Omic Integrated Analysis Method for Breast Cancer Intrinsic Subtypes

Breast cancer is one of the most frequently occurring female cancer types and represents a major cause of death among women worldwide. Breast cancer is heterogeneous in both molecular characteristics and clinical outcomes for its different molecular subtypes. High-throughput technologies facilitated the fast accumulations of the multiple Omic data for cancer patients. These data sources posed a computational challenge for the efficient integrated multi-Omic analysis. The existing studies usually investigated the differential representation or machine learning problems using a single type of Omic data. This study hypothesized that different Omic types contributed complementary information to each other, and their integrated analysis may improve the single-Omic models. An efficient logistic regression-based multi-Omic integrated analysis method (ELMO) was proposed to integrate the RNA-seq and DNA methylation data to detect the breast cancer intrinsic subtypes. ELMO achieved the highest accuracy with a smaller number of features compared with the existing filter and wrapper feature selection methods in this study. The experimental data supported our hypothesis that multi-Omic models outperformed the single-Omic ones.

Systematic Modeling of log D7.4 Based on Ensemble Machine Learning, Group Contribution, and Matched Molecular Pair Analysis.

Lipophilicity, as evaluated by the n-octanol/buffer solution distribution coefficient at pH = 7.4 (log D7.4), is a major determinant of various absorption, distribution, metabolism, elimination, and toxicology (ADMET) parameters of drug candidates. In this study, we developed several quantitative structure-property relationship (QSPR) models to predict log D7.4 based on a large and structurally diverse data set. Eight popular machine learning algorithms were employed to build the prediction models with 43 molecular descriptors selected by a wrapper feature selection method. The results demonstrated that XGBoost yielded better prediction performance than any other single model (RT2 = 0.906 and RMSET = 0.395). Moreover, the consensus model from the top three models could continue to improve the prediction performance (RT2 = 0.922 and RMSET = 0.359). The robustness, reliability, and generalization ability of the models were strictly evaluated by the Y-randomization test and applicability domain analysis. Moreover, the group contribution model based on 110 atom types and the local models for different ionization states were also established and compared to the global models. The results demonstrated that the descriptor-based consensus model is superior to the group contribution method, and the local models have no advantage over the global models. Finally, matched molecular pair (MMP) analysis and descriptor importance analysis were performed to extract transformation rules and give some explanations related to log D7.4. In conclusion, we believe that the consensus model developed in this study can be used as a reliable and promising tool to evaluate log D7.4 in drug discovery.

Comparison of Naïve Bayes Algorithm and Decision Tree C4.5 for Hospital Readmission Diabetes Patients using HbA1c Measurement

Diabetes is a metabolic disorder disease in which the pancreas does not produce enough insulin or the body cannot use insulin produced effectively. The HbA1c examination, which measures the average glucose level of patients during the last 2-3 months, has become an important step to determine the condition of diabetic patients. Knowledge of the patient's condition can help medical staff to predict the possibility of patient readmissions, namely the occurrence of a patient requiring hospitalization services back at the hospital. The ability to predict patient readmissions will ultimately help the hospital to calculate and manage the quality of patient care. This study compares the performance of the Naïve Bayes method and C4.5 Decision Tree in predicting readmissions of diabetic patients, especially patients who have undergone HbA1c examination. As part of this study we also compare the performance of the classification model from a number of scenarios involving a combination of preprocessing methods, namely Synthetic Minority Over-Sampling Technique (SMOTE) and Wrapper feature selection method, with both classification techniques. The scenario of C4.5 method combined with SMOTE and feature selection method produces the best performance in classifying readmissions of diabetic patients with an accuracy value of 82.74 %, precision value of 87.1 %, and recall value of 82.7 %.

A Systematic Evaluation of Profiling Through Focused Feature Selection

Profiled side-channel attacks consist of several steps one needs to take. An important, but sometimes ignored, step is a selection of the points of interest (features) within side-channel measurement traces. A large majority of the related works start the analyses with an assumption that the features are preselected. Contrary to this assumption, here, we concentrate on the feature selection step. We investigate how advanced feature selection techniques stemming from the machine learning domain can be used to improve the attack efficiency. To this end, we provide a systematic evaluation of the methods of interest. The experiments are performed on several real-world data sets containing software and hardware implementations of AES, including the random delay countermeasure. Our results show that wrapper and hybrid feature selection methods perform extremely well over a wide range of test scenarios and a number of features selected. We emphasize L1 regularization (wrapper approach) and linear support vector machine (SVM) with recursive feature elimination used after chi-square filter (Hybrid approach) that performs well in both accuracy and guessing entropy. Finally, we show that the use of appropriate feature selection techniques is more important for an attack on the high-noise data sets, including those with countermeasures, than on the low-noise ones.

A novel hybrid bat algorithm with a fast clustering-based hybridization

Bat algorithm (BA) is a new and promising metaheuristic search algorithm which could outperform existing algorithms. However, BA can be easily trapped in a local optimum regarded to low exploration ability. The present study proposed a new local-search-based hybrid heuristic to escape such scenario. The proposed hybrid BA (hBA) uses a clustering-based hybridization method which detects the early convergence of BA population by analyzing similarities among individuals. The main motivation for such an analysis is that when BA is continually converging, the similarity among individuals becomes higher. The proposed hBA is extensively evaluated on CEC2017 benchmark suite. The Experiments demonstrate that the algorithm achieves better results than continues variants of BA in every way. Moreover, as a case study, a binary version of the proposed hBA (hBBA) is applied to the well-known feature selection problem. The recorded results on 13 datasets demonstrate that hBBA would be considered as a new state-of-art in metaheuristic-based wrapper feature selection methods.

Online identification of functional regions in deep brain stimulation based on an unsupervised random forest with feature selection

Objective. The identification of functional regions, in particular the subthalamic nucleus, through microelectrode recording (MER) is the key step in deep brain stimulation (DBS). To eliminate variability in a neurosurgeon’s judgment, this study presents an online identification method for identifying functional regions along the electrode trajectory. Approach. Functional regions can be identified through offline clustering and online identification based on the unsupervised random forest (RF) algorithm. We took 106 features from MER and the estimated anatomical distance to target as the dataset to train the RF model. To improve the prediction performance and reduce the computation time, a wrapper feature selection (FS) method was added into the algorithm. The method contains feature ranking based on out-of-bag error or silhouette index and feature subset search based on the roulette selection algorithm. Main results. To evaluate the optimization effect of the FS method on the unsupervised RF algorithm, we compared the results of the algorithm with or without FS on the DBS dataset. In addition, the optimization effect of FS on the computation time is evaluated. The results show that for offline clustering, the accuracy obtained with the selected features is higher than that obtained with all features, and the running time decreased from 259.7 s to 60.8 s in the iteration of the FS. The accuracy in online identification improved from 76.19% to 92.08% through FS. In addition, the functional region online identification time is 41.5 ms, which can meet the requirements of DBS surgery. Significance. In conclusion, using the FS method can improve the accuracy and reduce the computation time of the online identification of functional regions. In addition, the online identification method can provide valuable assistance for both neurosurgeons and stereotactic surgery robots in guiding implantation of the electrode in real time.

An efficient methodology for discovering both of gene-environment interactions and gene-gene interactions causing genetic diseases

Genetic diseases are one of the most critical diseases facing human societies, their risk lies in the transmission of genetic characteristics from one generation to another, where the imbalance of these characteristics leads to an unhealthy offspring, which negatively affects the effort of this offspring and its services to society. Genetic disease is caused by a mutation in the Deoxyribonucleic Acid (DNA), these genetic mutations are generated by nonlinear interactions between two or more genes and / or environmental exposures. The aim of this paper is to discover both of gene-environment interactions and gene-gene interactions causing a genetic disease, the proposed methodology is based on both of the filter and wrapper feature selection methods, it uses the filter method using a Relief algorithm to detect the gene-environment interactions, wrapper method using genetic algorithm to discover gene-gene interactions, and classification decision tree algorithm to generate the conditional rules of gene-gene interactions. It has been evaluated using many different classifier models on four benchmark databases, and compared its performance with an Apriori algorithm for generating rules of gene-gene interactions, the proposed methodology achieved the highest performance and better classification accuracy on all databases containing patients affected by gene-environment interactions or gene-gene interactions or both of gene-environment and gene-gene interactions.

Time-varying hierarchical chains of salps with random weight networks for feature selection

Feature selection (FS) is considered as one of the most common and challenging tasks in Machine Learning. FS can be considered as an optimization problem that requires an efficient optimization algorithm to find its optimal set of features. This paper proposes a wrapper FS method that combines a time-varying number of leaders and followers binary Salp Swarm Algorithm (called TVBSSA) with Random Weight Network (RWN). In this approach, the TVBSSA is used as a search strategy, while RWN is utilized as an induction algorithm. The objective function is formulated in a manner to aggregate three objectives: maximizing the classification accuracy, maximizing the reduction rate of the selected features, and minimizing the complexity of generated RWN models. To assess the performance of the proposed approach, 20 well-known UCI datasets and a number of existing FS methods are employed. The comparative results show the ability of the proposed approach in outperforming similar algorithms in the literature and its merits to be used in systems that require FS.

Hybrid Filter–Wrapper Feature Selection Method for Sentiment Classification

The feature selection (FS) has been the latest challenge in the area of sentiment classification. The filter- and wrapper-based feature selection methods are applied in the domain to reduce feature set size and increase accuracy of the classifiers. In this paper, a hybrid of filter and wrapper method for selecting relevant features is proposed. The feature subset is first selected from the original feature set using computationally fast rank-based FS methods. The selected features are further refined using two wrapper approaches. In the first approach, recursive feature elimination is applied to select optimal feature set, and in the second approach, evolutionary method based on binary particle swarm optimization is applied for finalization of feature subset. The comparison between the two proposed techniques is conducted on five different domain datasets used in the area of sentiment analysis. We used simple and efficient ML algorithms (Naive Bayes, support vector machine and logistic regression) to evaluate the performance of the hybrid FS techniques. Finally, we assessed the performance of the proposed hybrid FS technique by comparing our results with the state-of-the-art methods. The results reveal that the proposed method is able to give better accuracy with fewer number of features.

An evolutionary gravitational search-based feature selection

With recent advancements in data collection tools and the widespread use of intelligent information systems, a huge amount of data streams with lots of redundant, irrelevant, and noisy features are collected and a large number of features (attributes) should be processed. Therefore, there is a growing demand for developing efficient Feature Selection (FS) techniques. Gravitational Search algorithm (GSA) is a successful population-based metaheuristic inspired by Newton’s law of gravity. In this research, a novel GSA-based algorithm with evolutionary crossover and mutation operators is proposed to deal with feature selection (FS) tasks. As an NP-hard problem, FS finds an optimal subset of features from a given set. For the proposed wrapper FS method, both K-Nearest Neighbors (KNN) and Decision Tree (DT) classifiers are used as evaluators. Eighteen well-known UCI datasets are utilized to assess the performance of the proposed approaches. In order to verify the efficiency of proposed algorithms, the results are compared with some popular nature-inspired algorithms (i.e. Genetic Algorithm (GA), Particle Swarm Optimizer (PSO), and Grey Wolf Optimizer (GWO)). The extensive results and comparisons demonstrate the superiority of the proposed algorithm in solving FS problems.

In silico markers: an evolutionary and statistical approach to select informative genes of human breast cancer subtypes.

Recent advancement in bioinformatics offers the ability to identify informative genes from high dimensional gene expression data. Selection of informative genes from these large datasets has emerged as an issue of major concern among researchers. Gene functionality and regulatory mechanisms can be understood through the analysis of these gene expression data. Here, we present a computational method to identify informative genes for breast cancer subtypes such as Basal, human epidermal growth factor receptor 2 (Her2), luminal A (LumA), and luminal B (LumB). The proposed In Silico Markers method is a wrapper feature selection method based on Least Absolute Shrinkage and Selection Operator (LASSO), Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and Support Vector Machine (SVM) as a classifier. Moreover, the composite measure consisting of relevance, redundancy, and rank score of frequently appeared genes are used to select informative genes. The informative genes are validated by statistical and biologically relevant criteria. For a comparative evaluation of the proposed approach, biological similarity score designed on semantic similarity measure of GO terms are investigated. Further, the proposed technique is evaluated with 7 existing gene selection techniques using two-class annotated breast cancer subtype datasets. The utilization of this method can bring about the discovery of informative genes. Furthermore, under multiple criteria decision-making set-up, informative genes selected by the In Silico Markers are found to be admirable than the compared methods selected genes.

A new multi-objective wrapper method for feature selection – Accuracy and stability analysis for BCI

Feature selection is an important step in building classifiers for high-dimensional data problems, such as EEG classification for BCI applications. This paper proposes a new wrapper method for feature selection, based on a multi-objective evolutionary algorithm, where the representation of the individuals or potential solutions, along with the breeding operators and objective functions, have been carefully designed to select a small subset of features that has good generalization capability, trying to avoid the over-fitting problems that wrapper methods usually suffer. A novel feature ranking procedure is also proposed in order to analyze the stability of the proposed wrapper method.Four different classification schemes have been applied within the proposed wrapper method in order to evaluate its accuracy and stability for feature selection on a real motor imagery dataset. Experimental results show that the wrapper method presented in this paper is able to obtain very small subsets of features, which are quite stable and also achieve high classification accuracy, regardless of the classifiers used.

Wrapper Framework for Test-Cost-Sensitive Feature Selection

Feature selection is an optional preprocessing procedure and is frequently used to improve the classification accuracy of a machine learning algorithm by removing irrelevant and/or redundant features. However, in many real-world applications, the test cost is also required for making optimal decisions, in addition to the classification accuracy. To the best of our knowledge, thus far, few studies have been conducted on test-cost-sensitive feature selection (TCSFS). In TCSFS, the objectives are twofold: 1) to improve the classification accuracy and 2) to decrease the test cost. Therefore, in fact, it constitutes a multiobjective optimization problem. In this paper, we transformed this multiobjective optimization problem into a single-objective optimization problem by utilizing a new evaluation function and in this paper, we propose a new general wrapper framework for TCSFS. Specifically, in our proposed framework, we add a new term to the evaluation function of a wrapper feature selection method so that the test cost of measuring features is taken into account. We experimentally tested our proposed framework, using 36 classification problems from the University of California at Irvine (UCI) repository, and compared it to some other state-of-the-art feature selection frameworks. The experimental results showed that our framework allows users to select an optimal feature subset with the minimal test cost, while simultaneously maintaining a high classification accuracy.

Improved Credit Scoring Model Based on Bagging Neural Network

The problem of nonperforming loans is one of the biggest problems in the banking sector. In order to mitigate this problem, it is necessary to improve the methods of credit risk assessment. One way to minimize credit risk is to improve the assessment of the creditworthiness of the applicant. In order to make a more accurate assessment, many models have been developed using classification techniques. This paper demonstrates the use of classification techniques in the form of a single classifier or in a classifier ensemble setting. We proposed bagging as a model ensemble using artificial neural networks. In the experiment conducted with the Bosnian commercial banks dataset, the proposed model showed promising results according to evaluation criteria, especially after the process of feature selection. Both individual and wrapper feature selection methods were used. Bagging with neural network (NNBag) outperforms commonly used techniques with accuracy improvement from 1% to 5%. The superiority of the proposed model (NNBag) is confirmed on two widely available datasets for assessing creditworthiness. Based on experimental results on three datasets, it is proven that NNBag is suitable for use in the assessment of the creditworthiness of applicants.