Random Forest Variable Importance Measures Research Articles

Random Forest Variable Importance Measures for Spatial Dynamics: Case Studies from Urban Demography

Population growth in urban centres and the intensification of segregation phenomena associated with international mobility require improved urban planning and decision-making. More effective planning in turn requires better analysis and geospatial modelling of residential locations, along with a deeper understanding of the factors that drive the spatial distribution of various migrant groups. This study examines the factors that influence the distribution of migrants at the local level and evaluates their importance using machine learning, specifically the variable importance measures produced by the random forest algorithm. It is conducted on high spatial resolution (100×100 grid cells) register data in Amsterdam and Copenhagen, using demographic, housing and neighbourhood attributes for 2018. The results distinguish the ethnic and demographic composition of a location as an important factor in the residential distribution of migrants in both cities. We also examine whether certain migrant groups pay higher prices in the most attractive areas, using spatial statistics and mapping for 2008 and 2018. We find evidence of segregation in both cities, with Western migrants having higher purchasing power than non-Western migrants in both years. The method sheds light on the determinants of migrant distribution in destination cities and advances our understanding of the application of geospatial artificial intelligence to urban dynamics and population movements.

Independent prognostic value of method of detection in lung cancer: Analysis of the PLCO lung screening trial.

e22526 Background: Prognostic models in cancer use patient and tumor characteristics to predict survival and disease progression. Many tumors are detected as part of screening programs such as colonoscopy for colorectal cancer, mammograms for breast cancer, and pap tests for cervical cancer. Past work in breast cancer has shown that screen detected tumors exhibit better prognosis than symptom detected tumors. The prognostic significance of method of detection in other types of cancers is less well understood. Further, the importance of method of detection, relative to other prognostic variables such as sex, age, cancer stage at diagnosis, and histology, is largely unknown. Methods: The prognostic significance of method of detection in the PLCO Lung cancer screening trial was investigated. PLCO randomized patients to intervention, with four annual chest x-ray (CXR) screenings, or control, usual care. Patients were followed for a total of approximately 13 years. All patients diagnosed with non-small cell lung cancer within 4 years of randomization were considered in the analysis (N = 934). Cox proportional hazards model and Random Survival Forests were used to build prognostic models to predict overall survival. The importance of each variable in prognostic models was assessed using hazard ratios and Random Forest variable importance measures. Results: Screen detected tumors exhibit stage shift and have better prognosis than interval and control detected tumors. The improved prognosis remains significant after controlling for stage at diagnosis and other variables commonly used in prognostic models (e.g., age, histology, sex). Variable importance measures show that method of detection is a more important prognostic indicator than sex and histology. Results are robust to different methods for cohort selection. Conclusions: Method of cancer detection should be considered when developing prognostic models in lung cancer studies. Cancer registries should routinely collect method of cancer detection.

OA10 Factors associated with long-term opioid use in fibromyalgia patients newly started on an opioid: results from Clinical Practice Research Datalink

Abstract Background/Aims The impact of the opioid epidemic in North America has caused concern in the United Kingdom. Long-term opioid use is associated with several opioid-related harms and adverse outcomes. Fibromyalgia is frequently treated with opioids due to limited therapeutic options. Understanding what factors are associated with long-term opioid use in such patients is the first step in helping to develop targeted interventions for deprescribing. Therefore, the objective of this study was to (i) quantify the proportion of new opioid users with fibromyalgia who become long-term opioid users; and (ii) identify risk factors associated with long-term opioid use in fibromyalgia. Methods A retrospective cohort study was conducted using data from the Clinical Practice Research Datalink (CPRD). CPRD is a national database of electronic health records within primary care, representation of the general population. Fibromyalgia patients without prior cancer who were new opioid users between 01/Jan/2006 and 31/Aug/2021 were included. Long-term opioid use was defined as at least 3 opioid prescriptions issued within a 90-day period, or ≥ 1 opioid prescription lasting at least 90 days, in the first year of follow-up. Logistic regression analysis and a random forest model were used to identify risk factors related to long-term opioid use. Compared to logistic regression, random forest variable importance measures covered both the effects of individual predictor variables as well as multivariate interactions with other predictor variables. In the random forest model, the mean decrease accuracy (MDA) index to measure the variables’ importance was used; the larger the value of the mean decrease accuracy, the more important the variable is. Results Of the 28,552 fibromyalgia patients who were new opioids users, 7,375 (26%) became long-term opioid users in the first year. In the fully adjusted logistic regression model, suicide and self-harm (adjusted odds ratio [aOR]: 1.98, 95% confidence interval (CI): 1.72-2.27, p < 0.001), substance use disorder (aOR: 2.07, 95%CI: 1.62-2.66, p < 0.001), most deprived (Index of Multiple Deprivation (IMD) quintile 5/most deprived, aOR: 1.98, 95%CI: 1.69-2.33, p < 0.001, compared with quintile 1/least deprived), obesity (aOR: very severely obese, OR: 2.35, 95%CI: 1.76-3.18, p < 0.001) and mean daily morphine milligram equivalents (MME) at the initiation (High (120-199 MME/day): aOR: 34.81, 95%CI: 18.00-78.05, p < 0.001 compared to mean daily MME: Low (<50 MME/day)) were associated with higher risk of long-term opioid use. The three most important variables using the random forest model were mean MME/day at initiation (MDA 140), history of suicide and self-harm (MDA 40) and IMD (MDA 30). Conclusion Almost 1 in 4 patients with fibromyalgia starting opioids became a long-term opioid user in this nationally representative dataset within the first year. The dose of initial opioid prescription, high deprivation score, history of suicide and self-harm, substance use disorder and obesity were associated with long-term opioid use. Disclosure M. Feng: None. C. Ramirez Medina: None. Y. Huang: None. D. Jenkins: None. M. Jani: None.

Leveraging Important Covariate Groups for Corn Yield Prediction

Accurate yield information empowers farmers to adapt, their governments to adopt timely agricultural and food policy interventions, and the markets they supply to prepare for production shifts. Unfortunately, the most representative yield data in the US, provided by the US Department of Agriculture, National Agricultural Statistics Service (USDA-NASS) Surveys, are spatiotemporally patchy and inconsistent. This paper builds a more complete data product by examining the spatiotemporal efficacy of random forests (RF) in predicting county-level yields of corn—the most widely cultivated crop in the US. To meet our objective, we compare RF cross-validated prediction accuracy using several combinations of explanatory variables. We also utilize variable importance measures and partial dependence plots to compare and contextualize how key variables interact with corn yield. Results suggest that RF predicts US corn yields well using a relatively small subset of climate variables along with year and geographical location (RMSE = 17.1 bushels/acre (1.2 tons/hectare)). Of note is the insensitivity of RF prediction accuracy when removing variables traditionally thought to be predictive of yield or variables flagged as important by RF variable importance measures. Understanding what variables are needed to accurately predict corn yields provides a template for applying machine learning approaches to estimate county-level yields for other US crops.

Development of a casemix classification to predict costs of home care in the Netherlands: a study protocol

IntroductionCompared with fee-for-service systems, prospective payment based on casemix classification is thought to promote more efficient, needs-based care provision. We aim to develop a casemix classification to predict the costs...

Exploring Molecular Descriptors and Fingerprints to Predict mTOR Kinase Inhibitors using Machine Learning Techniques.

Mammalian Target of Rapamycin (mTOR) is a Ser/Thr protein kinase, and its role is integral to the autophagy pathway in cancer. Targeting mTOR for therapeutic interventions in cancer through autophagy pathway is challenging due to the dual roles of autophagy in tumor progression. The architecture of mTOR reveals two complexes - mTORC1 and mTORC2, each having multiple protein subunits. mTOR kinase inhibitors target the structurally and functionally similar catalytic subunits of both mTORC1 and mTORC2. In this paper, we have explored two different categories of molecular features - descriptors and fingerprints for developing predictive models using machine learning techniques. Random Forest variable importance measures and autoencoders are used to identify molecular descriptors and fingerprints, respectively. We have built various predictive models using identified features and their combination for predicting mTOR kinase inhibitors. Finally, the best model based on the Mathew correlation co-efficient value over the validation dataset is selected for screening kinase SARfari bioactivity dataset. In this study, we have identified twenty best performing descriptors for predicting mTOR kinase inhibitors. To the best of our knowledge, it is the first study on integrating traditional machine learning and deep learning-based approaches for feature extraction to predict mTOR kinase inhibitors.

Early Cerebral Vein After Endovascular Ischemic Stroke Treatment Predicts Symptomatic Reperfusion Hemorrhage.

Parenchymal hemorrhage (PH) after endovascular mechanical thrombectomy in acute ischemic stroke leads to worse outcomes. Better clinical and imaging biomarkers of symptomatic reperfusion PH are needed to identify patients at risk. We identified clinical and imaging predictors of reperfusion PH after endovascular mechanical thrombectomy with attention to early cerebral veins (ECVs) on postreperfusion digital subtraction angiography. We performed a retrospective cohort study of consecutive acute ischemic stroke patients undergoing endovascular mechanical thrombectomy at our neurovascular referral center. Clinical and imaging characteristics were collected from patient health records, and random forest variable importance measures were used to identify predictors of symptomatic PH. Predictors of secondary outcomes, including 90-day mortality, functional dependence (modified Rankin Scale score, >2), and National Institutes of Health Stroke Scale shift, were also determined. Diagnostic test characteristics of ECV for symptomatic PH were determined using a receiver operating characteristic analysis. Differences between patients with and without symptomatic PH were assessed with Fisher exact test and the Wilcoxon rank sum (Mann-Whitney U test) test at the 0.05 significance level. Of 64 patients with anterior circulation large-vessel occlusion identified, 6 (9.4%) developed symptomatic PH. ECV was the strongest predictor of symptomatic PH with more than twice the importance of the next best predictor, male sex. Although ECV was also predictive of 90-day mortality and functional dependence, other characteristics were more important than ECV for these outcomes. The sensitivity and specificity of ECV alone for subsequent hemorrhage were both 0.83, with an area under the curve of 0.83 and 95% confidence interval of 0.66 to 1.00. ECV on postendovascular mechanical thrombectomy digital subtraction angiography is highly diagnostic of subsequent symptomatic reperfusion hemorrhage in this data set. This finding has important implications for post-treatment management of blood pressure and anticoagulation.

A Systematic Approach for Variable Selection With Random Forests: Achieving Stable Variable Importance Values

Random Forests variable importance measures are often used to rank variables by their relevance to a classification problem and subsequently reduce the number of model inputs in high-dimensional data sets, thus increasing computational efficiency. However, as a result of the way that training data and predictor variables are randomly selected for use in constructing each tree and splitting each node, it is also well known that if too few trees are generated, variable importance rankings tend to differ between model runs. In this letter, we characterize the effect of the number of trees (ntree) and class separability on the stability of variable importance rankings and develop a systematic approach to define the number of model runs and/or trees required to achieve stability in variable importance measures. Results demonstrate that both a large ntree for a single model run, or averaged values across multiple model runs with fewer trees, are sufficient for achieving stable mean importance values. While the latter is far more computationally efficient, both the methods tend to lead to the same ranking of variables. Moreover, the optimal number of model runs differs depending on the separability of classes. Recommendations are made to users regarding how to determine the number of model runs and/or trees that are required to achieve stable variable importance rankings.

Object-based crop classification using multi-temporal SPOT-5 imagery and textural features with a Random Forest classifier

In this study, an object-based image analysis (OBIA) approach was developed to classify field crops using multi-temporal SPOT-5 images with a random forest (RF) classifier. A wide range of features, including the spectral reflectance, vegetation indices (VIs), textural features based on the grey-level co-occurrence matrix (GLCM) and textural features based on geostatistical semivariogram (GST) were extracted for classification, and their performance was evaluated with the RF variable importance measures. Results showed that the best segmentation quality was achieved using the SPOT image acquired in September, with a scale parameter of 40. The spectral reflectance and the GST had a stronger contribution to crop classification than the VIs and GLCM textures. A subset of 60 features was selected using the RF-based feature selection (FS) method, and in this subset, the near-infrared reflectance and the image acquired in August (jointing and heading stages) were found to be the best for crop classification.

Do little interactions get lost in dark random forests?

BackgroundRandom forests have often been claimed to uncover interaction effects. However, if and how interaction effects can be differentiated from marginal effects remains unclear. In extensive simulation studies, we investigate whether random forest variable importance measures capture or detect gene-gene interactions. With capturing interactions, we define the ability to identify a variable that acts through an interaction with another one, while detection is the ability to identify an interaction effect as such.ResultsOf the single importance measures, the Gini importance captured interaction effects in most of the simulated scenarios, however, they were masked by marginal effects in other variables. With the permutation importance, the proportion of captured interactions was lower in all cases. Pairwise importance measures performed about equal, with a slight advantage for the joint variable importance method. However, the overall fraction of detected interactions was low. In almost all scenarios the detection fraction in a model with only marginal effects was larger than in a model with an interaction effect only.ConclusionsRandom forests are generally capable of capturing gene-gene interactions, but current variable importance measures are unable to detect them as interactions. In most of the cases, interactions are masked by marginal effects and interactions cannot be differentiated from marginal effects. Consequently, caution is warranted when claiming that random forests uncover interactions.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-016-0995-8) contains supplementary material, which is available to authorized users.

An experimental study of the intrinsic stability of random forest variable importance measures.

BackgroundThe stability of Variable Importance Measures (VIMs) based on random forest has recently received increased attention. Despite the extensive attention on traditional stability of data perturbations or parameter variations, few studies include influences coming from the intrinsic randomness in generating VIMs, i.e. bagging, randomization and permutation. To address these influences, in this paper we introduce a new concept of intrinsic stability of VIMs, which is defined as the self-consistence among feature rankings in repeated runs of VIMs without data perturbations and parameter variations. Two widely used VIMs, i.e., Mean Decrease Accuracy (MDA) and Mean Decrease Gini (MDG) are comprehensively investigated. The motivation of this study is two-fold. First, we empirically verify the prevalence of intrinsic stability of VIMs over many real-world datasets to highlight that the instability of VIMs does not originate exclusively from data perturbations or parameter variations, but also stems from the intrinsic randomness of VIMs. Second, through Spearman and Pearson tests we comprehensively investigate how different factors influence the intrinsic stability.ResultsThe experiments are carried out on 19 benchmark datasets with diverse characteristics, including 10 high-dimensional and small-sample gene expression datasets. Experimental results demonstrate the prevalence of intrinsic stability of VIMs. Spearman and Pearson tests on the correlations between intrinsic stability and different factors show that #feature (number of features) and #sample (size of sample) have a coupling effect on the intrinsic stability. The synthetic indictor, #feature/#sample, shows both negative monotonic correlation and negative linear correlation with the intrinsic stability, while OOB accuracy has monotonic correlations with intrinsic stability. This indicates that high-dimensional, small-sample and high complexity datasets may suffer more from intrinsic instability of VIMs. Furthermore, with respect to parameter settings of random forest, a large number of trees is preferred. No significant correlations can be seen between intrinsic stability and other factors. Finally, the magnitude of intrinsic stability is always smaller than that of traditional stability.ConclusionFirst, the prevalence of intrinsic stability of VIMs demonstrates that the instability of VIMs not only comes from data perturbations or parameter variations, but also stems from the intrinsic randomness of VIMs. This finding gives a better understanding of VIM stability, and may help reduce the instability of VIMs. Second, by investigating the potential factors of intrinsic stability, users would be more aware of the risks and hence more careful when using VIMs, especially on high-dimensional, small-sample and high complexity datasets.

An AUC-based permutation variable importance measure for random forests

BackgroundThe random forest (RF) method is a commonly used tool for classification with high dimensional data as well as for ranking candidate predictors based on the so-called random forest variable importance measures (VIMs). However the classification performance of RF is known to be suboptimal in case of strongly unbalanced data, i.e. data where response class sizes differ considerably. Suggestions were made to obtain better classification performance based either on sampling procedures or on cost sensitivity analyses. However to our knowledge the performance of the VIMs has not yet been examined in the case of unbalanced response classes. In this paper we explore the performance of the permutation VIM for unbalanced data settings and introduce an alternative permutation VIM based on the area under the curve (AUC) that is expected to be more robust towards class imbalance.ResultsWe investigated the performance of the standard permutation VIM and of our novel AUC-based permutation VIM for different class imbalance levels using simulated data and real data. The results suggest that the new AUC-based permutation VIM outperforms the standard permutation VIM for unbalanced data settings while both permutation VIMs have equal performance for balanced data settings.ConclusionsThe standard permutation VIM loses its ability to discriminate between associated predictors and predictors not associated with the response for increasing class imbalance. It is outperformed by our new AUC-based permutation VIM for unbalanced data settings, while the performance of both VIMs is very similar in the case of balanced classes. The new AUC-based VIM is implemented in the R package party for the unbiased RF variant based on conditional inference trees. The codes implementing our study are available from the companion website: http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/070_drittmittel/janitza/index.html.

SNP interaction detection with Random Forests in high-dimensional genetic data

BackgroundIdentifying variants associated with complex human traits in high-dimensional data is a central goal of genome-wide association studies. However, complicated etiologies such as gene-gene interactions are ignored by the univariate analysis usually applied in these studies. Random Forests (RF) are a popular data-mining technique that can accommodate a large number of predictor variables and allow for complex models with interactions. RF analysis produces measures of variable importance that can be used to rank the predictor variables. Thus, single nucleotide polymorphism (SNP) analysis using RFs is gaining popularity as a potential filter approach that considers interactions in high-dimensional data. However, the impact of data dimensionality on the power of RF to identify interactions has not been thoroughly explored. We investigate the ability of rankings from variable importance measures to detect gene-gene interaction effects and their potential effectiveness as filters compared to p-values from univariate logistic regression, particularly as the data becomes increasingly high-dimensional.ResultsRF effectively identifies interactions in low dimensional data. As the total number of predictor variables increases, probability of detection declines more rapidly for interacting SNPs than for non-interacting SNPs, indicating that in high-dimensional data the RF variable importance measures are capturing marginal effects rather than capturing the effects of interactions.ConclusionsWhile RF remains a promising data-mining technique that extends univariate methods to condition on multiple variables simultaneously, RF variable importance measures fail to detect interaction effects in high-dimensional data in the absence of a strong marginal component, and therefore may not be useful as a filter technique that allows for interaction effects in genome-wide data.

The impact of chemical and sensorial characteristics on the market price of Italian red wines

The hedonic price analysis presented in this paper is carried out on a dataset containing observable, sensory and chemical characteristics of a sample of Italian red wines. The study starts from the commonly observed evidence that in general the market price can be explained by the objective characteristics appearing on the label of the bottle and not by the wine's quality. The aim of the analysis is to discover how quality matters. This objective is pursued by means of the construction of latent sensory and chemical factors, whose implicit value, quantified using Random Forest variable importance measures, turns out to be appreciably high.

I*: Optimizing Logistic Regression to Match Ensemble Performance Using Random Forest Variable Importance

An automated directed search procedure called interaction miner or I* is outlined as an entity which allows logistic regression models to be built automatically based on theory suggested by random forest variable importance measures of predictive value of attributes. The fact that interaction effects can be added to regression models using intelligent directed information show that predictive models can be built without art and with science. It is unclear how important this is, but it appears ensemble methods derive their power by extracting information about interaction effects in data. Once this is accounted for regression models can match or outperform random forests. Tuning regression to outperform ensemble methods is the goal of this algorithm. It is shown to work on 3 credit data sets. This is an automated heuristic approach based on the observations in various credit and behavioral data sets that out of the box random forest outperforms logistic regression but after tuning based on random forest variable importance logistic regression can be tuned to match or outperform random forest models by adding interaction terms.

Letter to the Editor: On the stability and ranking of predictors from random forest variable importance measures

A recent study examined the stability of rankings from random forests using two variable importance measures (mean decrease accuracy (MDA) and mean decrease Gini (MDG)) and concluded that rankings based on the MDG were more robust than MDA. However, studies examining data-specific characteristics on ranking stability have been few. Rankings based on the MDG measure showed sensitivity to within-predictor correlation and differences in category frequencies, even when the number of categories was held constant, and thus may produce spurious results. The MDA measure was robust to these data characteristics. Further, under strong within-predictor correlation, MDG rankings were less stable than those using MDA.

Mining data with random forests: A survey and results of new tests

Random forests (RF) has become a popular technique for classification, prediction, studying variable importance, variable selection, and outlier detection. There are numerous application examples of RF in a variety of fields. Several large scale comparisons including RF have been performed. There are numerous articles, where variable importance evaluations based on the variable importance measures available from RF are used for data exploration and understanding. Apart from the literature survey in RF area, this paper also presents results of new tests regarding variable rankings based on RF variable importance measures. We studied experimentally the consistency and generality of such rankings. Results of the studies indicate that there is no evidence supporting the belief in generality of such rankings. A high variance of variable importance evaluations was observed in the case of small number of trees and small data sets.

Fault detection and diagnosis with random forest feature extraction and variable importance methods

The ever-present drive to safer, more cost-effective and cleaner processes motivates the exploration of a variety of process monitoring methods. In the domain of data-driven approaches, random forest models present a nonlinear framework. Random forest models consist of ensembles of classification and regression trees in which the model response is determined by voting committees of independent binary decision trees. Data-driven approaches to fault diagnosis often involve summarizing potentially large numbers of process variables in lower dimensional diagnostic sequences. Random forest feature extraction allows for the monitoring of process in feature and residual spaces, while random forest variable importance measures can potentially be used to identify process variables contribution to fault conditions. In this study, a framework for diagnosing steady state faults with random forests is proposed and demonstrated with a simple nonlinear system and the benchmark Tennessee Eastman process.

Predictor correlation impacts machine learning algorithms: implications for genomic studies

The advent of high-throughput genomics has produced studies with large numbers of predictors (e.g. genome-wide association, microarray studies). Machine learning algorithms (MLAs) are a computationally efficient way to identify phenotype-associated variables in high-dimensional data. There are important results from mathematical theory and numerous practical results documenting their value. One attractive feature of MLAs is that many operate in a fully multivariate environment, allowing for small-importance variables to be included when they act cooperatively. However, certain properties of MLAs under conditions common in genomic-related data have not been well-studied--in particular, correlations among predictors pose a problem. Using extensive simulation, we showed considering correlation within predictors is crucial in making valid inferences using variable importance measures (VIMs) from three MLAs: random forest (RF), conditional inference forest (CIF) and Monte Carlo logic regression (MCLR). Using a case-control illustration, we showed that the RF VIMs--even permutation-based--were less able to detect association than other algorithms at effect sizes encountered in complex disease studies. This reduction occurred when 'causal' predictors were correlated with other predictors, and was sharpest when RF tree building used the Gini index. Indeed, RF Gini VIMs are biased under correlation, dependent on predictor correlation strength/number and over-trained to random fluctuations in data when tree terminal node size was small. Permutation-based VIM distributions were less variable for correlated predictors and are unbiased, thus may be preferred when predictors are correlated. MLAs are a powerful tool for high-dimensional data analysis, but well-considered use of algorithms is necessary to draw valid conclusions. Supplementary data are available at Bioinformatics online.

Empirical characterization of random forest variable importance measures

Microarray studies yield data sets consisting of a large number of candidate predictors (genes) on a small number of observations (samples). When interest lies in predicting phenotypic class using gene expression data, often the goals are both to produce an accurate classifier and to uncover the predictive structure of the problem. Most machine learning methods, such as k -nearest neighbors, support vector machines, and neural networks, are useful for classification. However, these methods provide no insight regarding the covariates that best contribute to the predictive structure. Other methods, such as linear discriminant analysis, require the predictor space be substantially reduced prior to deriving the classifier. A recently developed method, random forests (RF), does not require reduction of the predictor space prior to classification. Additionally, RF yield variable importance measures for each candidate predictor. This study examined the effectiveness of RF variable importance measures in identifying the true predictor among a large number of candidate predictors. An extensive simulation study was conducted using 20 levels of correlation among the predictor variables and 7 levels of association between the true predictor and the dichotomous response. We conclude that the RF methodology is attractive for use in classification problems when the goals of the study are to produce an accurate classifier and to provide insight regarding the discriminative ability of individual predictor variables. Such goals are common among microarray studies, and therefore application of the RF methodology for the purpose of obtaining variable importance measures is demonstrated on a microarray data set.