Ensemble Feature Selection Method Research Articles

Ensemble Classification Based on Feature Selection for Environmental Sound Recognition

Environmental sound recognition has been a hot topic in the domain of audio recognition. How to select the optimal feature subsets and enhance the performance of classification precisely is an urgent problem to be solved. Ensemble learning, a new kind of method presented recently, has been an effective way to improve the accuracy of classification in feature selection. In this paper, experiments were performed on environmental sound dataset. An improved method based on constraint score and multimodels ensemble feature selection methods (MmEnFs) were exploited in the experiments. The experimental results show that when enough attributes are selected, the improved method can get a better performance compared to other feature selection methods. And the ensemble feature selection method, which combines other methods, can obtain the optimal performance in most cases.

An Ensemble Feature Selection Method to Detect Web Spam

Feature selection is an important issue in data mining, and it is used to reduce dimensions of features set. Web spam detection is one of research fields of data mining. With regard to increasing available information in virtual space and the need of users to search, the role of search engines and used algorithms are important in terms of ranking. Web spam is an illegal method to increase mendacious rank of internet pages by deceiving the algorithms of search engines, so it is essential to use an efficient method. Up to now, many methods have been proposed to face with web spam. An ensemble feature selection method has been proposed in this paper to detect web spam. Content features of standard dataset of WEBSPAM-UK2007 are used for evaluation. Bayes network classifier is used along with 70-30% training-testing spilt of dataset. The presented results show that AUC of this method is higher than the other methods reported in this paper. Moreover, the best values of evaluation metrics in our proposed method are optimal in comparison to the other methods reported in this paper. In addition, it improves classification metrics in comparison to basic feature selection methods.

Predicting the Severity of Bug Reports Based on Feature Selection

In software maintenance process, it is a fairly important activity to predict the severity of bug reports. However, manually identifying the severity of bug reports is a tedious and time-consuming task. So developing automatic judgment methods for predicting the severity of bug reports has become an urgent demand. In general, a bug report contains a lot of descriptive natural language texts, thus resulting in a high-dimensional feature set which poses serious challenges to traditionally automatic methods. Therefore, we attempt to use automatic feature selection methods to improve the performance of the severity prediction of bug reports. In this paper, we introduce a ranking-based strategy to improve existing feature selection algorithms and propose an ensemble feature selection algorithm by combining existing ones. In order to verify the performance of our method, we run experiments over the bug reports of Eclipse and Mozilla and conduct comparisons with eight commonly used feature selection methods. The experiment results show that the ranking-based strategy can effectively improve the performance of the severity prediction of bug reports by up to 54.76% on average in terms of [Formula: see text]-measure, and it also can significantly reduce the dimension of the feature set. Meanwhile, the ensemble feature selection method can get better results than a single feature selection algorithm.

An Integrated Framework With Feature Selection for Dropout Prediction in Massive Open Online Courses

Massive open online courses (MOOCs) have flourished in recent years, which is conducive to the redistribution of high-quality educational resources globally. However, the high dropout rate in the course of operation has seriously affected its development. Therefore, in order to improve the degree of completion, it is an effective way to study how to effectively predict the dropout in MOOCs and intervene in advance. Traditional methods rely on manually extracted features, which is difficult to guarantee the final prediction effect. In order to solve this problem, this paper proposes an integrated framework with feature selection (FSPred) to predict the dropout in MOOCs, which includes feature generation, feature selection, and dropout prediction. Specifically, FSPred applies a fine-grained feature generation method in days to generate features and then uses an ensemble feature selection method to select valid features and feed them into a logistic regression model for prediction. Extensive experiments on a public data set have shown that FSPred can achieve the comparable results with other dropout prediction methods in terms of precision, recall, F1 score, and AUC score. Finally, through the analysis of the features of the final selection, the suggestions for the construction of the MOOCs are put forward.

Ensemble Feature Selection for Plant Phenotyping: A Journey From Hyperspectral to Multispectral Imaging

Hyperspectral imaging is becoming an increasingly popular tool for high-throughput plant phenotyping, because it provides remarkable insights about the health status of plants. Feature selection is a key component in a hyperspectral image analysis, largely because a significant portion of spectral features are redundant and/or irrelevant, depending on the desired application. This paper presents an ensemble feature selection method to identify the most informative spectral features for practical applications in plant phenotyping. The hyperspectral data set contained the images of four wheat lines, each with a control and a salt (NaCl) treatment. To rank spectral features, six feature selection methods were used as the base for the ensemble: correlation-based feature selection, ReliefF, sequential feature selection, support vector machine-recursive feature elimination (SVM-RFE), LASSO logistic regression, and random forest. The best results were achieved by the ensemble of ReliefF, SVM-RFE, and random forest, which drastically reduced the dimension of the hyperspectral data set from 215 to 15 features, while improving the accuracy in classifying the salt-treated vegetation pixels from the control pixels by 8.5%. To transform the hyperspectral data set into a multispectral data set, six wavelengths as the center of broad multispectral bands around the most prominent features were determined by a clustering algorithm. The result of salt tolerance assessment of the four wheat lines using the derived multispectral data set was similar to that of the hyperspectral data set. This demonstrates that the proposed feature selection pipeline can be utilized for determining the most informative features and can be a valuable tool in the development of tailored multispectral cameras.

An Ensemble Feature Selection Method for Biomarker Discovery.

Feature selection in Liquid Chromatography-Mass Spectrometry (LC-MS)-based metabolomics data (biomarker discovery) have become an important topic for machine learning researchers. High dimensionality and small sample size of LC-MS data make feature selection a challenging task. The goal of biomarker discovery is to select the few most discriminative features among a large number of irreverent ones. To improve the reliability of the discovered biomarkers, we use an ensemble-based approach. Ensemble learning can improve the accuracy of feature selection by combining multiple algorithms that have complementary information. In this paper, we propose an ensemble approach to combine the results of filter-based feature selection methods. To evaluate the proposed approach, we compared it to two commonly used methods, t-test and PLS-DA, using a real data set.

EFS-MI: an ensemble feature selection method for classification

Feature selection methods have been used in various applications of machine learning, bioinformatics, pattern recognition and network traffic analysis. In high dimensional datasets, due to redundant features and curse of dimensionality, a learning method takes significant amount of time and performance of the model decreases. To overcome these problems, we use feature selection technique to select a subset of relevant and non-redundant features. But, most feature selection methods are unstable in nature, i.e., for different training datasets, a feature selection method selects different subsets of features that yields different classification accuracy. In this paper, we provide an ensemble feature selection method using feature–class and feature-feature mutual information to select an optimal subset of features by combining multiple subsets of features. The method is validated using four classifiers viz., decision trees, random forests, KNN and SVM on fourteen UCI, five gene expression and two network datasets.

Ensemble feature selection using bi-objective genetic algorithm

Feature selection problem in data mining is addressed here by proposing a bi-objective genetic algorithm based feature selection method. Boundary region analysis of rough set theory and multivariate mutual information of information theory are used as two objective functions in the proposed work, to select only precise and informative data from the data set. Data set is sampled with replacement strategy and the method is applied to determine non-dominated feature subsets from each sampled data set. Finally, ensemble of such bi-objective genetic algorithm based feature selectors is developed with the help of parallel implementations to produce much generalized feature subset. In fact, individual feature selector outputs are aggregated using a novel dominance based principle to produce final feature subset. Proposed work is validated using repository especially for feature selection datasets as well as on UCI machine learning repository datasets and the experimental results are compared with related state of art feature selection methods to show effectiveness of the proposed ensemble feature selection method.

Classifier ensemble selection based on affinity propagation clustering

A small number of features are significantly correlated with classification in high-dimensional data. An ensemble feature selection method based on cluster grouping is proposed in this paper. Classification-related features are chosen using a ranking aggregation technique. These features are divided into unrelated groups by an affinity propagation clustering algorithm with a bicor correlation coefficient. Some diversity and distinguishing feature subsets are constructed by randomly selecting a feature from each group and are used to train base classifiers. Finally, some base classifiers that have better classification performance are selected using a kappa coefficient and integrated using a majority voting strategy. The experimental results based on five gene expression datasets show that the proposed method has low classification error rates, stable classification performance and strong scalability in terms of sensitivity, specificity, accuracy and G-Mean criteria.

A Review on Feature Selection Methods for Sentiment Analysis

Text documents are normally represented as a feature-document matrix in sentiment analysis. Features can be single words from the text document or more complex pairs extracted by different schemes that adds information in order to enrich the feature-document matrix representation. Having diverse feature types however creates a problem of high dimensionality due to the vast number of features and relations they hold. Thus, feature selection helps in ensuring that effective and efficient sentiment analysis applications can be developed by selecting features that are relevant and informative to assist classifiers to perform better and to reduce the processing load by narrowing down the feature set. This paper highlights methods used for feature selection, namely filter, wrapper and embedded. Prior to feature selection, preprocessing techniques are performed to reduce the amount of features first. This paper is concluded by summarizing this review and outlining the challenges faced and proposing the ensemble feature selection method for sentiment analysis data.

An ensemble regularization method for feature selection in mass spectral fingerprints

Successful identification of the significant features in complex mass spectral fingerprints is a crucial task in discriminating states or differences in natural systems (e.g., diseased vs. healthy, treated vs. untreated, and male vs. female) that are visualized using mass spectrometry technology. In this study, we present an ensemble regularization method that combines three regularization regression models to generate more robust results. Specifically, the coefficients from each of three regularization models were bootstrapped and the means and standard deviations of these coefficients were calculated. After obtaining these estimated statistics of the coefficients, we performed a hypothesis test for each feature. Finally, we determined the significant features that were simultaneously selected by the three hypothesis tests. Mass spectral data from six different extracts of mosquito cuticles were used to evaluate the performance of the proposed method. The purpose of this spectral analysis was to determine the major features needed to differentiate married-female mosquitoes having the potential to cause malaria infection from others. In addition, we compared the proposed ensemble feature selection method with random forest, a widely used feature selection algorithm. We found that the proposed method outperformed random forest in terms of feature selection efficiency.

Ensemble features selection method as tool for breast cancer classification

This work aims to gather experimental evidence of features relevance, as well as finding a breast cancer classification scheme that provides the high performance over the area under receiver operating characteristic curve (AUC). An ensemble feature selection method (named RMean) based on the mean criteria for indexing relevant features is presented. The proposed method provided better classification performances (statistically significant) than those who constitute the baseline, attaining AUC scores of 0.7775 with the support vector machine on microcalcifications dataset and 0.9440 with the feed-forward-backpropagation neural network classifier on masses dataset. The most relevant features for microcalcifications classification were: mammographic stroma distortion, density, right bottom quadrant, perimeter, standard deviation, entropy, and angular second moment. Meanwhile, to classify masses were: mammographic stroma distortion, mammographic calcification, mammographic nodules, density, circularity, roughness, and shape.

Penalized ensemble feature selection methods for hidden associations in time series environments case study: equities companies in Saudi Stock Exchange Market

The problem of optimal subset selection from a large number of time series is addressed in this work using machine learning of financial forecasting. This is a persistent problem in stock market that is largely due to the vast amount of daily-basis data points which require sensitive and robust intelligent data analysis techniques for capturing hidden associations between time series features. In this work, we are interested in generalizing the concept of capturing hidden associations between predictors from financial time series data points in the setting of penalized ensemble feature selection techniques. We have shown how recently developed penalized ensemble feature selection methods are capable of revealing hidden and informative dependencies between equity companies that appear in Saudi Stock Exchange Market in different daily time series datasets. The results have shown that our methods outperformed the well-known lasso regularization method particularly for small sample size.

Robust twin boosting for feature selection from high-dimensional omics data with label noise

Omics data such as microarray transcriptomic and mass spectrometry proteomic data are typically characterized by high dimensionality and relatively small sample sizes. In order to discover biomarkers for diagnosis and prognosis from omics data, feature selection has become an indispensable step to find a parsimonious set of informative features. However, many previous studies report considerable label noise in omics data, which will lead to unreliable inferences to select uninformative features. Yet, to the best of our knowledge, very few feature selection methods are proposed to address this problem. This paper proposes a novel ensemble feature selection algorithm, robust twin boosting feature selection (RTBFS), which is robust to label noise in omics data. The algorithm has been validated on an omics feature selection test bed and seven real-world heterogeneous omics datasets, of which some are known to have label noise. Compared with several state-of-the-art ensemble feature selection methods, RTBFS can select more informative features despite label noise and obtain better classification results. RTBFS is a general feature selection method and can be applied to other data with label noise. MATLAB implementation of RTBFS and sample datasets are available at: http://www.cs.bham.ac.uk/∼szh/TReBFSMatlab.zip.

On the Learning of Causal Relationships between Banks in Saudi Equities Market Using Ensemble Feature Selection Methods

Financial forecasting using machine learning techniques has received great efforts in the last decide . In this ongoing work, we show how machine learning of graphical models will be able to infer a visualized causal interactions between different banks in the Saudi equities market. One important discovery from such learned causal graphs is how companies influence each other and to what extend. In this work, a set of graphical models named Gaussian graphical models with developed ensemble penalized feature selection methods that combine ; filtering method, wrapper method and a regularizer will be shown. A comparison between these different developed ensemble combinations will also be shown. The best ensemble method will be used to infer the causal relationships between banks in Saudi equities market.

Penerapan Ensemble Feature Selection dan Klasterisasi Fitur pada Klasifikasi Dokumen Teks

An ensemble method is an approach where several classifiers are created from the training data which can be often more accurate than any of the single classifiers, especially if the base classifiers are accurate and different one each other. Menawhile, feature clustering can reduce feature space by joining similar words into one cluster. The objective of this research is to develop a text categorization system that employs feature clustering based on ensemble feature selection. The research methodology consists of text documents preprocessing, feature subspaces generation using the genetic algorithm-based iterative refinement, implementation of base classifiers by applying feature clustering, and classification result integration of each base classifier using both the static selection and majority voting methods. Experimental results show that the computational time consumed in classifying the dataset into 2 and 3 categories using the feature clustering method is 1.18 and 27.04 seconds faster in compared to those that do not employ the feature selection method, respectively. Also, using static selection method, the ensemble feature selection method with genetic algorithm-based iterative refinement produces 10% and 10.66% better accuracy in compared to those produced by the single classifier in classifying the dataset into 2 and 3 categories, respectively. Whilst, using the majority voting method for the same experiment, the similar ensemble method produces 10% and 12% better accuracy than those produced by the single classifier, respectively.

Combination of effective machine learning techniques and chemometric analysis for evaluation of Bupleuri Radix through high-performance thin-layer chromatography

Chaihu (Bupleuri Radix), the root of Bupleurum chinense and B. scorzonerifolium, is a traditional Chinese herbal medicine authenticated in the Chinese Pharmacopoeia. There are also several variations available from local herbal markets, for example, the roots of B. falcatum, B. bicaule, and B. marginatum var. stenophyllum. In the current study, we collected 64 Chaihu samples, including 33 authenticated samples and 31 commercial samples. Test solutions of all the examples were analysed by high-performance thin-layer chromatography (HPTLC) to assess the principal bio-active components (saikosaponins). The HPTLC fluorescent images acquired were analyzed by sophisticated image processing techniques for comprehensive quantification. High dimensional features for both gray-scale and true color images were constructed for the raw images. Classical classification algorithms, including naive Bayes, Support Vector Machine (SVM), K-nearest neighbors, neural network and logistic, were used to construct prediction models. To gain an insight into the principal components while evaluating the Chaihu sample, feature selection and ensemble feature selection methods were further combined with the classifiers to enhance the discrimination power. Ensemble feature selection was shown to achieve superior performance. Experimental results demonstrated that the roots of Chaihu from different species of the genus Bupleurum could be readily distinguished so that commercial samples could be easily classified.

Robust biomarker identification for cancer diagnosis with ensemble feature selection methods

Biomarker discovery is an important topic in biomedical applications of computational biology, including applications such as gene and SNP selection from high-dimensional data. Surprisingly, the stability with respect to sampling variation or robustness of such selection processes has received attention only recently. However, robustness of biomarkers is an important issue, as it may greatly influence subsequent biological validations. In addition, a more robust set of markers may strengthen the confidence of an expert in the results of a selection method. Our first contribution is a general framework for the analysis of the robustness of a biomarker selection algorithm. Secondly, we conducted a large-scale analysis of the recently introduced concept of ensemble feature selection, where multiple feature selections are combined in order to increase the robustness of the final set of selected features. We focus on selection methods that are embedded in the estimation of support vector machines (SVMs). SVMs are powerful classification models that have shown state-of-the-art performance on several diagnosis and prognosis tasks on biological data. Their feature selection extensions also offered good results for gene selection tasks. We show that the robustness of SVMs for biomarker discovery can be substantially increased by using ensemble feature selection techniques, while at the same time improving upon classification performances. The proposed methodology is evaluated on four microarray datasets showing increases of up to almost 30% in robustness of the selected biomarkers, along with an improvement of approximately 15% in classification performance. The stability improvement with ensemble methods is particularly noticeable for small signature sizes (a few tens of genes), which is most relevant for the design of a diagnosis or prognosis model from a gene signature. Supplementary data are available at Bioinformatics online.