Feature Importance Measure Research Articles

Accessing the temporal and spectral features in crop type mapping using multi-temporal Sentinel-2 imagery: A case study of Yi’an County, Heilongjiang province, China

Crop type mapping visualizes the spatial distribution patterns and proportions of the cultivated areas with different crop types, and is the basis for subsequent agricultural applications. Understanding the effectiveness of different temporal and spectral features in detailed crop classification can help users optimize temporal window selection and spectral feature space construction in crop type mapping applications. Therefore, in this study, we used time-series Sentinel-2 image data from Yi’an County, Heilongjiang province, China, to analyze the effectiveness of the temporal and spectral features used in three common machine learning classification methods: classification and regression tree (CART) decision tree, Support Vector Machine (SVM), and random forest (RF). For CART and SVM classifiers, the relative importance of the features was reflected by the order and frequency of attributes selected as the node and the square of the model weight. In RF, the change in prediction error as calculated by out of bag data is taken as the measure of feature importance. The standard deviation of the average value of all labeled pixels was used to evaluate the correctness of the unanimous conclusions drawn by these three methodologies. The quantitative evaluation results given by the confusion matrix show that random forest achieved the best overall accuracy, while support vector machine ranked second, and the decision tree algorithm yielded the least accurate classification results. From the perspective of feature importance, making full use of the discriminative information between different crops, and constructing a rational feature space, can help to improve classification accuracy significantly. In detail, the discriminative information between the different crop types is as follows: 1) images at the peak of the crop growth period are crucial in the classification of different crops; 2) the short-wave infrared bands are particularly suitable for fine crop classification; and 3) the red edge bands can effectively assist classification. Finally, our study achieved crop type mapping in the study area with an overall accuracy of 97.85% and a Kappa coefficient of 0.95.

A Novel Feature Extension Method for the Forest Disaster Monitoring Using Multispectral Data

Remote sensing images classification is the key technology for monitoring forest changes. Texture features have been demonstrated to have better effectiveness than spectral features in the improvement of the classification accuracy. The accuracy of extracting texture information by window-based method depends on the choice of the window size. Moreover, the size should ideally match the spatial scale of the object or class under consideration. However, most of the existing texture feature extraction methods are all based on a single window and do not adequately consider the scale of different objects. Our first proposition is to use a composite window for extracting texture features, which is a small window surrounded by a larger window. Our second proposition is to reinforce the performance of the trained ensemble classifier by training it using only the most important features. Considering the advantages of random forest classifier, such as fast training speed and few parameters, these features feed this classifier. Measures of feature importance are estimated along with the growth of the base classifiers, here decision trees. We aim to classify each pixel of the forest images disturbed by hurricanes and fires in three classes, damaged, not damaged, or unknown, as this could be used to compute time-dependent aggregates. In this study, two research areas—Nezer Forest in France and Blue Mountain Forest in Australia—are utilized to validating the effectiveness of the proposed method. Numerical simulations show increased performance and improved monitoring ability of forest disturbance when using these two propositions. When compared with the reference methods, the best increase of the overall accuracy obtained by the proposed algorithm is 4.77% and 2.96% on the Nezer forest data and Blue Mountain forest data, respectively.

Metropolized Knockoff Sampling

Model-X knockoffs is a wrapper that transforms essentially any feature importance measure into a variable selection algorithm, which discovers true effects while rigorously controlling the expected fraction of false positives. A frequently discussed challenge to apply this method is to construct knockoff variables, which are synthetic variables obeying a crucial exchangeability property with the explanatory variables under study. This article introduces techniques for knockoff generation in great generality: we provide a sequential characterization of all possible knockoff distributions, which leads to a Metropolis–Hastings formulation of an exact knockoff sampler. We further show how to use conditional independence structure to speed up computations. Combining these two threads, we introduce an explicit set of sequential algorithms and empirically demonstrate their effectiveness. Our theoretical analysis proves that our algorithms achieve near-optimal computational complexity in certain cases. The techniques we develop are sufficiently rich to enable knockoff sampling in challenging models including cases where the covariates are continuous and heavy-tailed, and follow a graphical model such as the Ising model. Supplementary materials for this article are available online.

Multi-objective feature selection using hybridization of a genetic algorithm and direct multisearch for key quality characteristic selection

A multi-objective feature selection approach for selecting key quality characteristics (KQCs) of unbalanced production data is proposed. We define KQC (feature) selection as a bi-objective problem of maximizing the quality characteristic (QC) subset importance and minimizing the QC subset size. Three candidate feature importance measures, the geometric mean (GM), F1 score and accuracy, are applied to construct three KQC selection models. To solve the models, a two-phase optimization method for selecting the candidate solutions (QC subsets) using a novel multi-objective optimization method (GADMS) and the final KQC set from the candidate solutions using the ideal point method (IPM) is proposed. GADMS is a hybrid method composed of a genetic algorithm (GA) and a local search strategy named direct multisearch (DMS). In GADMS, we combine binary encoding with real value encoding to utilize the advantages of GAs and DMS. The experimental results on four production datasets show that the proposed method with GM performs the best in handling the data imbalance problem and outperforms the benchmark methods. Moreover, GADMS obtains significantly better search performance than the benchmark multi-objective optimization methods, which include a modified nondominated sorting genetic algorithm II (NSGA-II), two multi-objective particle swarm optimization algorithms and an improved DMS method.

Specifying the precision of guiding features for visual search.

Visual search is the task of finding things with uncertain locations. Despite decades of research, the features that guide visual search remain poorly specified, especially in realistic contexts. This study tested the role of two features-shape and orientation-both in the presence and absence of hue information. We conducted five experiments to describe preview-target mismatch effects, decreases in performance caused by differences between the image of the target as it appears in the preview and as it appears in the actual search display. These mismatch effects provide direct measures of feature importance, with larger performance decrements expected for more important features. Contrary to previous conclusions, our data suggest that shape and orientation only guide visual search when color is not available. By varying the probability of mismatch in each feature dimension, we also show that these patterns of feature guidance do not change with the probability that the previewed feature will be invalid. We conclude that the target representations used to guide visual search are much less precise than previously believed, with participants encoding and using color and little else. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Teleconnections in spatial modelling

In pedology, spatial context is relevant to soil-landscape systems on at least three different scales: i) the scale of quasi-local processes, which are independent of influence from the direct or wider neighborhood, ii) the scale of short-range processes for example on the local hillslope or catena, and iii) the scale of long-range processes, or teleconnected systems. We can represent the effects of teleconnections using existing tools and covariates, but we cannot easily infer or identify their controls, landscape processes or landscape units. We consider that an ability to identify the relevant controls in teleconnected systems would greatly improve pedological interpretation and understanding. Such understanding relates to the interaction of environmental factors and processes in the spatial context, which is relevant for environmental mapping generally. Here we show that teleconnected systems can be disassembled and interpreted using contextual modelling in such a way that the controls, i.e. the cause, can be localized in space. We present examples of how teleconnected systems can be deciphered. The methodology is based on the previously described ConMap approach in combination with Random Forest's measures of local feature importance. ConMap uses elevation differences, computed along multiple rays radiating out from a center grid cell, as predictors instead of complex surface derivatives or decomposed scales of a digital elevation model (DEM) or terrain attribute. Using synthetic and real-world data sets, we show how to identify and interpret teleconnections in soil environmental systems. In the synthetic example, elevation peaks are shown to produce larger values of soil properties, while, in contrast, a valley-mountain system is the main control of soil texture in the real-world example. Our analyses of teleconnected soil environmental systems illustrate that the stochastic component of the universal model of spatial variation is an integral but typically unresolved part of the deterministic component.

A comparison of machine learning approaches for identifying high-poverty counties: robust features of DMSP/OLS night-time light imagery

ABSTRACTThe goal of the present study is to demonstrate that high-poverty counties and robust classification features can be identified by machine learning approaches using only DMSP/OLS night-time light imagery. To accomplish this goal, a total of 96 high-poverty and 96 non-poverty counties were classified using 15 statistical and spatial features extracted from night-time light imagery in China in 2010 formed a training set for identifying high-poverty counties. Seven machine learning approaches were adopted to classify high-poverty counties, and five feature importance measures were used to select robust features. The resulting metrics, including the user’s (>63%), producer’s (>66%) and overall (>82%) accuracies of the poor county identification (probability of poverty greater than 0.6), show that the seven machine learning approaches used in this paper exhibit good performance, although some differences exist among the approaches. The order of feature importance reveals that the relative importance of each feature differs among the models; however, the important features remain consistent. The nine most important features ranked in each approach are relatively robust for poverty identification at the county level. Both spatial feature and statistical features calculated in part from the central tendency, degree of dispersion, and the distribution of the night-time light data were identified as indispensable robust features in all the approaches, indicating that the complex social phenomenon of poverty requires analysis from different aspects. Previous studies that utilized primarily night-time light imagery applied single features related to the central tendency or the distribution features of the imagery; this study provides a new method and can act as a reference for feature selection and identification of high-poverty counties using night-time light imagery and has potential applications across several scientific domains.

An Empirical Comparison of Machine-Learning Methods on Bank Client Credit Assessments

Machine learning and artificial intelligence have achieved a human-level performance in many application domains, including image classification, speech recognition and machine translation. However, in the financial domain expert-based credit risk models have still been dominating. Establishing meaningful benchmark and comparisons on machine-learning approaches and human expert-based models is a prerequisite in further introducing novel methods. Therefore, our main goal in this study is to establish a new benchmark using real consumer data and to provide machine-learning approaches that can serve as a baseline on this benchmark. We performed an extensive comparison between the machine-learning approaches and a human expert-based model—FICO credit scoring system—by using a Survey of Consumer Finances (SCF) data. As the SCF data is non-synthetic and consists of a large number of real variables, we applied two variable-selection methods: the first method used hypothesis tests, correlation and random forest-based feature importance measures and the second method was only a random forest-based new approach (NAP), to select the best representative features for effective modelling and to compare them. We then built regression models based on various machine-learning algorithms ranging from logistic regression and support vector machines to an ensemble of gradient boosted trees and deep neural networks. Our results demonstrated that if lending institutions in the 2001s had used their own credit scoring model constructed by machine-learning methods explored in this study, their expected credit losses would have been lower, and they would be more sustainable. In addition, the deep neural networks and XGBoost algorithms trained on the subset selected by NAP achieve the highest area under the curve (AUC) and accuracy, respectively.

A Deep Learning-based Ranking Approach for Microblog Retrieval

Today, Twitter has become one of the most popular micro-blogging service with a large amount of information on various topics produced by millions of users every day. When searching for useful information in Twitter, users need to assess high quality content that meets their needs. However, the study of effective information retrieval in such microblog is still a challenge because there is a large difference in the quality level of relevant tweets returned in the search results for a given query. Therefore, looking for an effective microblog retrieval requires distinguishing the high-quality tweet content among thousands of results. Existing works are based on hand-crafted features (e.g., number of re-tweets, number of followers, etc.) using hard-hand-engineering to indicate the quality of tweets. In this paper, we focus on the problem of ranking tweets, and particularly on retrieving high quality content. We propose a ranking approach based on k-means clustering to distinguish high quality from low quality tweets. The clustering algorithm is based on learning features from deep learning autoencoder and hand-crafted features from tweets’ content and authors’ profiles. We used information gain as a feature importance measure to find the optimal set of features having stronger power in clustering the data. By conducting a pilot feature analysis study, we demonstrate the impact of the learned features to identify tweets’ quality in the clustering process. Our experimental results show that the integration of learned features has shown significant improvement in the quality of clustering and especially on the ranking performance compared to the use of hand-crafted features only.

On what to permute in test-based approaches for variable importance measures in Random Forests

In bioinformatics applications, it is currently customary to permute the outcome variable in order to produce inference on covariates to test novel methods or statistics whose distributions are poorly known. The seminal publication of Altmann et al. in Bioinformatics uses the same permutation scheme to obtain P-values that can be treated as corrected measure of feature importance to rectify the bias of the Gini variable importance in Random Forests. Since then, such method has been used in applied work to also draw statistical conclusions on variable importance measures from resulting P-values. In this paper, we show that permuting the outcome may produce unexpected results, including P-values with undesirable properties and illustrate how more refined permutation schemes can be appropriate to obtain desirable results, including high power in discovering relevant variables. Supplementary data are available at Bioinformatics online.

A new approach for interpreting Random Forest models and its application to the biology of ageing.

MotivationThis work uses the Random Forest (RF) classification algorithm to predict if a gene is over-expressed, under-expressed or has no change in expression with age in the brain. RFs have high predictive power, and RF models can be interpreted using a feature (variable) importance measure. However, current feature importance measures evaluate a feature as a whole (all feature values). We show that, for a popular type of biological data (Gene Ontology-based), usually only one value of a feature is particularly important for classification and the interpretation of the RF model. Hence, we propose a new algorithm for identifying the most important and most informative feature values in an RF model.ResultsThe new feature importance measure identified highly relevant Gene Ontology terms for the aforementioned gene classification task, producing a feature ranking that is much more informative to biologists than an alternative, state-of-the-art feature importance measure.Availability and implementationThe dataset and source codes used in this paper are available as ‘Supplementary Material’ and the description of the data can be found at: https://fabiofabris.github.io/bioinfo2018/web/.Supplementary information Supplementary data are available at Bioinformatics online.

Limb-Enhancer Genie: An accessible resource of accurate enhancer predictions in the developing limb.

Epigenomic mapping of enhancer-associated chromatin modifications facilitates the genome-wide discovery of tissue-specific enhancers in vivo. However, reliance on single chromatin marks leads to high rates of false-positive predictions. More sophisticated, integrative methods have been described, but commonly suffer from limited accessibility to the resulting predictions and reduced biological interpretability. Here we present the Limb-Enhancer Genie (LEG), a collection of highly accurate, genome-wide predictions of enhancers in the developing limb, available through a user-friendly online interface. We predict limb enhancers using a combination of >50 published limb-specific datasets and clusters of evolutionarily conserved transcription factor binding sites, taking advantage of the patterns observed at previously in vivo validated elements. By combining different statistical models, our approach outperforms current state-of-the-art methods and provides interpretable measures of feature importance. Our results indicate that including a previously unappreciated score that quantifies tissue-specific nuclease accessibility significantly improves prediction performance. We demonstrate the utility of our approach through in vivo validation of newly predicted elements. Moreover, we describe general features that can guide the type of datasets to include when predicting tissue-specific enhancers genome-wide, while providing an accessible resource to the general biological community and facilitating the functional interpretation of genetic studies of limb malformations.

An efficient feature selection technique for clustering based on a new measure of feature importance

An efficient feature selection technique for clustering based on a new measure of feature importance

ML2Motif-Reliable extraction of discriminative sequence motifs from learning machines.

High prediction accuracies are not the only objective to consider when solving problems using machine learning. Instead, particular scientific applications require some explanation of the learned prediction function. For computational biology, positional oligomer importance matrices (POIMs) have been successfully applied to explain the decision of support vector machines (SVMs) using weighted-degree (WD) kernels. To extract relevant biological motifs from POIMs, the motifPOIM method has been devised and showed promising results on real-world data. Our contribution in this paper is twofold: as an extension to POIMs, we propose gPOIM, a general measure of feature importance for arbitrary learning machines and feature sets (including, but not limited to, SVMs and CNNs) and devise a sampling strategy for efficient computation. As a second contribution, we derive a convex formulation of motifPOIMs that leads to more reliable motif extraction from gPOIMs. Empirical evaluations confirm the usefulness of our approach on artificially generated data as well as on real-world datasets.

Machine learning classifiers provide insight into the relationship between microbial communities and bacterial vaginosis.

BackgroundBacterial vaginosis (BV) is a disease associated with the vagina microbiome. It is highly prevalent and is characterized by symptoms including odor, discharge and irritation. No single microbe has been found to cause BV. In this paper we use random forests and logistic regression classifiers to model the relationship between the microbial community and BV. We use subsets of the microbial community features in order to determine which features are important to the classification models.ResultsWe find that models generated using logistic regression and random forests perform nearly identically and identify largely similar important features. Only a few features are necessary to obtain high BV classification accuracy. Additionally, there appears to be substantial redundancy between the microbial community features.ConclusionsThese results are in contrast to a previous study in which the important features identified by the classifiers were dissimilar. This difference appears to be the result of using different feature importance measures. It is not clear whether machine learning classifiers are capturing patterns different from simple correlations.Electronic supplementary materialThe online version of this article (doi:10.1186/s13040-015-0055-3) contains supplementary material, which is available to authorized users.

Feature selection in possibilistic modeling

Feature selection is becoming increasingly important for the reduction of computing complexity. In this context, conventional approaches have random performances, because They can succeed for some contexts and fail for others. Possibilistic modeling is a powerful paradigm being able to handle data imperfection or redundancy and is not affected by data variability. Therefore, in this paper, we propose a new feature selection strategy for possibilitic modeling. The proposed approach is based on two issues in order to extract relevant features: the measure of feature importance as well as the possibility distribution uncertainty degree. The importance of one feature can be considered under two aspects: The first one is related to the scattering within one class and the second one reflects the feature power for class discrimination. Therefore, we apply, here, Shapley index paradigm which selects features who minimize the intra-class distance and who maximize the inter-class distance. The previous process is refined using possibility distribution uncertainty degree in order to resolve some conflict problems between feature׳s importance values.

Temporal optimisation of image acquisition for land cover classification with Random Forest and MODIS time-series

The analysis and classification of land cover is one of the principal applications in terrestrial remote sensing. Due to the seasonal variability of different vegetation types and land surface characteristics, the ability to discriminate land cover types changes over time. Multi-temporal classification can help to improve the classification accuracies, but different constraints, such as financial restrictions or atmospheric conditions, may impede their application. The optimisation of image acquisition timing and frequencies can help to increase the effectiveness of the classification process. For this purpose, the Feature Importance (FI) measure of the state-of-the art machine learning method Random Forest was used to determine the optimal image acquisition periods for a general (Grassland, Forest, Water, Settlement, Peatland) and Grassland specific (Improved Grassland, Semi-Improved Grassland) land cover classification in central Ireland based on a 9-year time-series of MODIS Terra 16 day composite data (MOD13Q1). Feature Importances for each acquisition period of the Enhanced Vegetation Index (EVI) and Normalised Difference Vegetation Index (NDVI) were calculated for both classification scenarios. In the general land cover classification, the months December and January showed the highest, and July and August the lowest separability for both VIs over the entire nine-year period. This temporal separability was reflected in the classification accuracies, where the optimal choice of image dates outperformed the worst image date by 13% using NDVI and 5% using EVI on a mono-temporal analysis. With the addition of the next best image periods to the data input the classification accuracies converged quickly to their limit at around 8–10 images. The binary classification schemes, using two classes only, showed a stronger seasonal dependency with a higher intra-annual, but lower inter-annual variation. Nonetheless anomalous weather conditions, such as the cold winter of 2009/2010 can alter the temporal separability pattern significantly. Due to the extensive use of the NDVI for land cover discrimination, the findings of this study should be transferrable to data from other optical sensors with a higher spatial resolution. However, the high impact of outliers from the general climatic pattern highlights the limitation of spatial transferability to locations with different climatic and land cover conditions. The use of high-temporal, moderate resolution data such as MODIS in conjunction with machine-learning techniques proved to be a good base for the prediction of image acquisition timing for optimal land cover classification results.

Application of High-Dimensional Feature Selection in Near-Infrared Spectroscopy of Cigarettes' Qualitative Evaluation

ABSTRACT In order to increase the classification accuracy, a new feature selection method, RFFIM-PCA, based on the random forest feature importance measure (RFFIM) and principal component analysis (PCA) for analyzing the near-infrared (NIR) spectra of tobacco, is presented in this paper. We applied the method to the classification of cigarettes' qualitative evaluation and also compared it with other methods. The result showed that RFFIM-PCA discriminates the high-dimensional data effectively and can be used to identify the cigarettes' quality. The feature selection filters the noises, while PCA eliminates the redundant features and reduces the dimensionalities as well. The experimental results showed that RFFIM-PCA successfully eliminated the noises and redundant features in high-dimensional data, leading to a promising improvement on the feature selection and classification accuracy.

Random generalized linear model: a highly accurate and interpretable ensemble predictor

BackgroundEnsemble predictors such as the random forest are known to have superior accuracy but their black-box predictions are difficult to interpret. In contrast, a generalized linear model (GLM) is very interpretable especially when forward feature selection is used to construct the model. However, forward feature selection tends to overfit the data and leads to low predictive accuracy. Therefore, it remains an important research goal to combine the advantages of ensemble predictors (high accuracy) with the advantages of forward regression modeling (interpretability). To address this goal several articles have explored GLM based ensemble predictors. Since limited evaluations suggested that these ensemble predictors were less accurate than alternative predictors, they have found little attention in the literature.ResultsComprehensive evaluations involving hundreds of genomic data sets, the UCI machine learning benchmark data, and simulations are used to give GLM based ensemble predictors a new and careful look. A novel bootstrap aggregated (bagged) GLM predictor that incorporates several elements of randomness and instability (random subspace method, optional interaction terms, forward variable selection) often outperforms a host of alternative prediction methods including random forests and penalized regression models (ridge regression, elastic net, lasso). This random generalized linear model (RGLM) predictor provides variable importance measures that can be used to define a “thinned” ensemble predictor (involving few features) that retains excellent predictive accuracy.ConclusionRGLM is a state of the art predictor that shares the advantages of a random forest (excellent predictive accuracy, feature importance measures, out-of-bag estimates of accuracy) with those of a forward selected generalized linear model (interpretability). These methods are implemented in the freely available R software package randomGLM.

A semi-supervised feature ranking method with ensemble learning

We consider the problem of using a large amount of unlabeled data to improve the efficiency of feature selection in high-dimension when only a small amount of labeled examples is available. We propose a new method called semi-supervised ensemble learning guided feature ranking method (SEFR for short), that combines a bagged ensemble of standard semi-supervised approaches with a permutation-based out-of-bag feature importance measure that takes into account both labeled and unlabeled data. We provide empirical results on several benchmark data sets indicating that SEFR can lead to significant improvement over state-of-the-art supervised and semi-supervised algorithms.