Tree Ensemble Methods Research Articles

A novel key-frame selection-based sign language recognition framework for the video data

ABSTRACT Sign language is a medium of communication for people with hearing disabilities. Static and dynamic gestures are identified in a video-based sign language recognition and translated them into humanly understandable phrases to achieve the communication objective. However, videos contain redundant Key-frames which require additional processing. Number of such Key-frames can be reduced. The selection of particular Key-frames without losing the required information is a challenging task. The Key-frame extraction algorithm is used which helps to speed-up the sign language recognition process by extracting essential key-frames. The proposed framework eliminates the computation overhead by picking up the distinct Key-frames for the recognition process. Discrete Wavelet Transform (DWT), Discrete Cosine Transform (DCT), and Histograms of Oriented Gradient (HOG) are used for unique features extraction. We used the bagged tree, boosted tree ensemble method, Fine KNN, and SVM for classification. We tested methodology on video-based datasets of Pakistani Sign Language. It achieved an overall 97.5% accuracy on 37 Urdu alphabets and 95.6% accuracy on 100 common words.

Feature Ranking for Hierarchical Multi-Label Classification with Tree Ensemble Methods

Feature Ranking for Hierarchical Multi-Label Classification with Tree Ensemble Methods

Machine-learning adsorption on binary alloy surfaces for catalyst screening

Over the last few years, machine learning is gradually becoming an essential approach for the investigation of heterogeneous catalysis. As one of the important catalysts, binary alloys have attracted extensive attention for the screening of bifunctional catalysts. Here we present a holistic framework for machine learning approach to rapidly predict adsorption energies on the surfaces of metals and binary alloys. We evaluate different machine-learning methods to understand their applicability to the problem and combine a tree-ensemble method with a compressed-sensing method to construct decision trees for about 60000 adsorption data. Compared to linear scaling relations, our approach enables to make more accurate predictions lowering predictive root-mean-square error by a factor of two and more general to predict adsorption energies of various adsorbates on thousands of binary alloys surfaces, thus paving the way for the discovery of novel bimetallic catalysts.

VEF: a variant filtering tool based on ensemble methods.

Variants identified by current genomic analysis pipelines contain many incorrectly called variants. These can be potentially eliminated by applying state-of-the-art filtering tools, such as Variant Quality Score Recalibration (VQSR) or Hard Filtering (HF). However, these methods are very user-dependent and fail to run in some cases. We propose VEF, a variant filtering tool based on decision tree ensemble methods that overcomes the main drawbacks of VQSR and HF. Contrary to these methods, we treat filtering as a supervised learning problem, using variant call data with known 'true' variants, i.e. gold standard, for training. Once trained, VEF can be directly applied to filter the variants contained in a given Variants Call Format (VCF) file (we consider training and testing VCF files generated with the same tools, as we assume they will share feature characteristics). For the analysis, we used whole genome sequencing (WGS) Human datasets for which the gold standards are available. We show on these data that the proposed filtering tool VEF consistently outperforms VQSR and HF. In addition, we show that VEF generalizes well even when some features have missing values, when the training and testing datasets differ in coverage, and when sequencing pipelines other than GATK are used. Finally, since the training needs to be performed only once, there is a significant saving in running time when compared with VQSR (4 versus 50 min approximately for filtering the single nucleotide polymorphisms of a WGS Human sample). Code and scripts available at: github.com/ChuanyiZ/vef. Supplementary data are available at Bioinformatics online.

Network inference with ensembles of bi-clustering trees

BackgroundNetwork inference is crucial for biomedicine and systems biology. Biological entities and their associations are often modeled as interaction networks. Examples include drug protein interaction or gene regulatory networks. Studying and elucidating such networks can lead to the comprehension of complex biological processes. However, usually we have only partial knowledge of those networks and the experimental identification of all the existing associations between biological entities is very time consuming and particularly expensive. Many computational approaches have been proposed over the years for network inference, nonetheless, efficiency and accuracy are still persisting open problems. Here, we propose bi-clustering tree ensembles as a new machine learning method for network inference, extending the traditional tree-ensemble models to the global network setting. The proposed approach addresses the network inference problem as a multi-label classification task. More specifically, the nodes of a network (e.g., drugs or proteins in a drug-protein interaction network) are modelled as samples described by features (e.g., chemical structure similarities or protein sequence similarities). The labels in our setting represent the presence or absence of links connecting the nodes of the interaction network (e.g., drug-protein interactions in a drug-protein interaction network).ResultsWe extended traditional tree-ensemble methods, such as extremely randomized trees (ERT) and random forests (RF) to ensembles of bi-clustering trees, integrating background information from both node sets of a heterogeneous network into the same learning framework. We performed an empirical evaluation, comparing the proposed approach to currently used tree-ensemble based approaches as well as other approaches from the literature. We demonstrated the effectiveness of our approach in different interaction prediction (network inference) settings. For evaluation purposes, we used several benchmark datasets that represent drug-protein and gene regulatory networks. We also applied our proposed method to two versions of a chemical-protein association network extracted from the STITCH database, demonstrating the potential of our model in predicting non-reported interactions.ConclusionsBi-clustering trees outperform existing tree-based strategies as well as machine learning methods based on other algorithms. Since our approach is based on tree-ensembles it inherits the advantages of tree-ensemble learning, such as handling of missing values, scalability and interpretability.

Evaluating the performance of a predictive modeling approach to identifying members at high-risk of hospitalization

Aims: To evaluate the risk-of-hospitalization (ROH) models developed at Blue Cross Blue Shield of Louisiana (BCBSLA) and compare this approach to the DxCG risk-score algorithms utilized by many health plans.Materials and Methods: Time zero for this study was December 31, 2016. BCBSLA members were eligible for study inclusion if they were fully insured; aged 80 years or younger; and had continuous enrollment starting on or before June 1, 2016, through time zero. Up to 2 years of historical claims data from time zero per patient was included for model development. Members were excluded if they had cancer, renal failure, or were admitted for hospice. The Blue Cross ROH models were developed using (1) regularized logistic regression and (2) random decision forests (a tree ensemble learning classification method). All models were generated using Scikit-learn: Machine Learning in Python. Prognostic capabilities of DxCG risk-score algorithms were compared to those of the Blue Cross models.Results: When stratifying by the top 0.1% of members with the highest ROH, the Blue Cross logistic regression model had the highest area under the receiving operator characteristics curve (0.862) based on the result of 10-fold cross-validation. The Blue Cross random decision forests model had the highest positive predictive value (49.0%) and positive likelihood ratio (61.4), but sensitivity, specificity, negative predictive values, and negative likelihood ratios were similar across all four models.Limitations: The Blue Cross ROH models were developed and evaluated using BCBSLA data, and predictive power may fluctuate if applied to other databases.Conclusions: The predictability of the Blue Cross models show how member-specific, regional data can be used to accurately identify patients with a high ROH, which may allow healthcare workers to intervene earlier and subsequently reduce the healthcare burden for patients and providers.

Travel time prediction in transport and logistics

PurposeMany transport and logistics companies nowadays use raw vehicle GPS data for travel time prediction. However, they face difficult challenges in terms of the costs of information storage, as well as the quality of the prediction. This paper aims to systematically investigate various meta-data (features) that require significantly less storage space but provide sufficient information for high-quality travel time predictions.Design/methodology/approachThe paper systematically studied the combinatorial effects of features and different model fitting strategies with two popular decision tree ensemble methods for travel time prediction, namely, random forests and gradient boosting regression trees. First, the investigation was conducted using pseudo travel time data that were generated using a pseudo travel time sampling algorithm, which allows generating travel time data using different noise processes so that the prediction performance under different travel conditions and noise characteristics can be studied systematically. The results and findings were then further compared and evaluated through a real-life case.FindingsThe paper provides empirical insights and guidelines about how raw GPS data can be reduced into a small-sized feature vector for the purposes of vehicle travel time prediction. It suggests that, add travel time observations from the previous departure time intervals are beneficial to the prediction, particularly when there is no other types of real-time information (e.g. traffic flow, speed) are available. It was also found that modular model fitting does not improve the quality of the prediction in all experimental settings used in this paper.Research limitations/implicationsThe findings are primarily based on empirical studies on limited real-life data instances, and the results may lack generalisabilities. Therefore, the researchers are encouraged to test them further in more real-life data instances.Practical implicationsThe paper includes implications and guidelines for the development of efficient GPS data storage and high-quality travel time prediction under different types of travel conditions.Originality/valueThis paper systematically studies the combinatorial feature effects for tree-ensemble-based travel time prediction approaches.

Ensemble based biomarker identification on pancreatic ductal adenocarcinoma gene expressions

Malignancy order is an imperative advance in biomarker distinguishing proof. Creating machine learning techniques that effectively anticipate tumor subtypes can help in distinguishing potential disease biomarkers. In this critique, we have introduced ensemble classification approach and contrasted its execution with other characterization approaches. PDAC microarray-based gene expression given in Gene Expression Omnibus (GEO) datasets are analyzed. After preprocessing of data, we have classified using Bagged tree Ensemble method and compared with other classifiers. The general achievement rate hence acquired is average of 96.64% for five testing datasets. Such a rate is 6–15% higher than the comparing rates obtained by different existing DT (decision tree), DA (discriminant analysis) and SVM (support vector machines) and NN (nearest neighbor) approaches, inferring that the gathering classifier is exceptionally encouraging and may turn into a significant test for biomarker identification. Finally, the biological analysis has been done to detect the common biomarkers for PDAC.

Decision Tree Ensemble Method for Analyzing Traffic Accidents of Novice Drivers in Urban Areas.

Presently, there is a critical need to analyze traffic accidents in order to mitigate their terrible economic and human impact. Most accidents occur in urban areas. Furthermore, driving experience has an important effect on accident analysis, since inexperienced drivers are more likely to suffer fatal injuries. This work studies the injury severity produced by accidents that involve inexperienced drivers in urban areas. The analysis was based on data provided by the Spanish General Traffic Directorate. The information root node variation (IRNV) method (based on decision trees) was used to get a rule set that provides useful information about the most probable causes of fatalities in accidents involving inexperienced drivers in urban areas. This may prove useful knowledge in preventing this kind of accidents and/or mitigating their consequences.

Optimizing predictive performance of criminal recidivism models using registration data with binary and survival outcomes.

In a recidivism prediction context, there is no consensus on which modeling strategy should be followed for obtaining an optimal prediction model. In previous papers, a range of statistical and machine learning techniques were benchmarked on recidivism data with a binary outcome. However, two important tree ensemble methods, namely gradient boosting and random forests were not extensively evaluated. In this paper, we further explore the modeling potential of these techniques in the binary outcome criminal prediction context. Additionally, we explore the predictive potential of classical statistical and machine learning methods for censored time-to-event data. A range of statistical manually specified statistical and (semi-)automatic machine learning models is fitted on Dutch recidivism data, both for the binary outcome case and censored outcome case. To enhance generalizability of results, the same models are applied to two historical American data sets, the North Carolina prison data. For all datasets, (semi-) automatic modeling in the binary case seems to provide no improvement over an appropriately manually specified traditional statistical model. There is however evidence of slightly improved performance of gradient boosting in survival data. Results on the reconviction data from two sources suggest that both statistical and machine learning should be tried out for obtaining an optimal model. Even if a flexible black-box model does not improve upon the predictions of a manually specified model, it can serve as a test whether important interactions are missing or other misspecification of the model are present and can thus provide more security in the modeling process.

Mapping global forest biomass and its changes over the first decade of the 21st century

Forests played an important role in carbon sequestration during the past two decades. Using a model tree ensemble method (MTE) to regress the seven reflectance bands of EOS-Terra-MODIS satellite data against country level forest biomass carbon density (BCD) of 2001–2005 provided by United Nations’s Forest Resource Assessment (FRA), we developed a global map of forest BCD at 1 km ×1 km resolution for both 2001–2005 and 2006–2010. For 2006–2010, the total global forest biomass carbon stock is estimated as 279.6±7.1 Pg C, and the tropical forest biomass carbon stock is estimated as 174.4±5.4 Pg C. During the first decade of the 21st century, we estimated an increase of global forest biomass of 0.28±0.75 Pg C yr−1. Tropical forest biomass carbon stock slightly decreased (−0.31±0.60 Pg C yr−1); by contrast, temperate and boreal forest biomass increased (0.58±0.28 Pg C yr−1) during the same period. Our estimation of the global forest biomass carbon stock and its changes is subject to uncertainties due to lack of extensive ground measurements in the tropics, spatial heterogeneity in large countries, and different definitions of forest. The continuously monitoring of forest biomass carbon stock with MODIS satellite data will provide useful information for detecting forest changes.

Decision Tree Ensemble-Based Wafer Map Failure Pattern Recognition Based on Radon Transform-Based Features

Wafer maps contain information about defects and clustered defects that form failure patterns. Failure patterns exhibit the information related to defect generation mechanisms. The accurate classification of failure patterns in wafer maps can provide crucial information for engineers to recognize the causes of the fabrication problems. In this paper, we proposed a decision tree ensemble learning-based wafer map failure pattern recognition method based on radon transform-based features. Radon transform is applied on raw wafer map data to generate the new features which are exhibiting the geometric information of failure patterns in wafer map. Decision tree algorithm is applied to build decision tree ensemble and the final decision is made by aggregating the prediction results of decision trees. The effectiveness of the proposed method has been verified by using the real world wafer map data set (WM-811K).

Bayesian Regression Trees for High-Dimensional Prediction and Variable Selection

ABSTRACTDecision tree ensembles are an extremely popular tool for obtaining high-quality predictions in nonparametric regression problems. Unmodified, however, many commonly used decision tree ensemble methods do not adapt to sparsity in the regime in which the number of predictors is larger than the number of observations. A recent stream of research concerns the construction of decision tree ensembles that are motivated by a generative probabilistic model, the most influential method being the Bayesian additive regression trees (BART) framework. In this article, we take a Bayesian point of view on this problem and show how to construct priors on decision tree ensembles that are capable of adapting to sparsity in the predictors by placing a sparsity-inducing Dirichlet hyperprior on the splitting proportions of the regression tree prior. We characterize the asymptotic distribution of the number of predictors included in the model and show how this prior can be easily incorporated into existing Markov chain Monte Carlo schemes. We demonstrate that our approach yields useful posterior inclusion probabilities for each predictor and illustrate the usefulness of our approach relative to other decision tree ensemble approaches on both simulated and real datasets. Supplementary materials for this article are available online.

Bayesian Additive Regression Trees using Bayesian Model Averaging.

Bayesian Additive Regression Trees (BART) is a statistical sum of trees model. It can be considered a Bayesian version of machine learning tree ensemble methods where the individual trees are the base learners. However for datasets where the number of variables p is large the algorithm can become inefficient and computationally expensive. Another method which is popular for high dimensional data is random forests, a machine learning algorithm which grows trees using a greedy search for the best split points. However its default implementation does not produce probabilistic estimates or predictions. We propose an alternative fitting algorithm for BART called BART-BMA, which uses Bayesian Model Averaging and a greedy search algorithm to obtain a posterior distribution more efficiently than BART for datasets with large p. BART-BMA incorporates elements of both BART and random forests to offer a model-based algorithm which can deal with high-dimensional data. We have found that BART-BMA can be run in a reasonable time on a standard laptop for the "small n large p" scenario which is common in many areas of bioinformatics. We showcase this method using simulated data and data from two real proteomic experiments, one to distinguish between patients with cardiovascular disease and controls and another to classify aggressive from non-aggressive prostate cancer. We compare our results to their main competitors. Open source code written in R and Rcpp to run BART-BMA can be found at: https://github.com/BelindaHernandez/BART-BMA.git.

Hyperspectral Image Classification With Canonical Correlation Forests

Multiple classifier systems or ensemble learning is an effective tool for providing accurate classification results of hyperspectral remote sensing images. Two well-known ensemble learning classifiers for hyperspectral data are random forest (RF) and rotation forest (RoF). In this paper, we proposed to use a novel decision tree (DT) ensemble method, namely, canonical correlation forest (CCF). More specifically, several individual canonical correlation trees (CCTs) that are binary DTs, which use canonical correlation components for the hyperplane splitting, are used to construct the CCF. Additionally, we adopt the projection bootstrap technique in CCF, in which the full spectral bands are retained for split selection in the projected space. The techniques aforementioned allow the CCF to improve the accuracy of member classifiers and diversity within the ensemble. Furthermore, the CCF is extended to the spectral-spatial frameworks that incorporate Markov random fields, extended multiattribute profiles (EMAPs), and the ensemble of independent component analysis and rolling guidance filter (E-ICA-RGF). Experimental results on six hyperspectral data sets are used to indicate the comparative effectiveness of the proposed method, in terms of accuracy and computational complexity, compared with RF and RoF, and it turns out that CCF is a promising approach for hyperspectral image classification not only with spectral information but also in the spectral-spatial frameworks.

Cross-Validated Variable Selection in Tree-Based Methods Improves Predictive Performance.

Recursive partitioning methods producing tree-like models are a long standing staple of predictive modeling. However, a fundamental flaw in the partitioning (or splitting) rule of commonly used tree building methods precludes them from treating different types of variables equally. This most clearly manifests in these methods' inability to properly utilize categorical variables with a large number of categories, which are ubiquitous in the new age of big data. We propose a framework to splitting using leave-one-out (LOO) cross validation (CV) for selecting the splitting variable, then performing a regular split (in our case, following CART's approach) for the selected variable. The most important consequence of our approach is that categorical variables with many categories can be safely used in tree building and are only chosen if they contribute to predictive power. We demonstrate in extensive simulation and real data analysis that our splitting approach significantly improves the performance of both single tree models and ensemble methods that utilize trees. Importantly, we design an algorithm for LOO splitting variable selection which under reasonable assumptions does not substantially increase the overall computational complexity compared to CART for two-class classification.

Finding structure in data using multivariate tree boosting.

Technology and collaboration enable dramatic increases in the size of psychological and psychiatric data collections, but finding structure in these large data sets with many collected variables is challenging. Decision tree ensembles such as random forests (Strobl, Malley, & Tutz, 2009) are a useful tool for finding structure, but are difficult to interpret with multiple outcome variables which are often of interest in psychology. To find and interpret structure in data sets with multiple outcomes and many predictors (possibly exceeding the sample size), we introduce a multivariate extension to a decision tree ensemble method called gradient boosted regression trees (Friedman, 2001). Our extension, multivariate tree boosting, is a method for nonparametric regression that is useful for identifying important predictors, detecting predictors with nonlinear effects and interactions without specification of such effects, and for identifying predictors that cause 2 or more outcome variables to covary. We provide the R package "mvtboost" to estimate, tune, and interpret the resulting model, which extends the implementation of univariate boosting in the R package "gbm" (Ridgeway, 2015) to continuous, multivariate outcomes. To illustrate the approach, we analyze predictors of psychological well-being (Ryff & Keyes, 1995). Simulations verify that our approach identifies predictors with nonlinear effects and achieves high prediction accuracy, exceeding or matching the performance of (penalized) multivariate multiple regression and multivariate decision trees over a wide range of conditions. (PsycINFO Database Record

Calibrating aquatic microfossil proxies with regression-tree ensembles: Cross-validation with modern chironomid and diatom data

We examine the ability of four different regression-tree ensemble techniques (bagging, random forest, rotation forest and boosted tree) in calibration of aquatic microfossil proxies. The methods are tested with six chironomid and diatom datasets, using a variety of cross-validation schemes. We find random forest, rotation forest and the boosted tree to have a similar performance, while bagging performs less well and in several cases has trouble producing continuous predictions. In comparison with commonly used parametric transfer-function approaches (PLS, WA, WA-PLS), we find that in some cases tree-ensemble methods outperform the best-performing transfer-function technique, especially with large datasets characterized by complex taxon responses and abundant noise. However, parametric transfer functions remain competitive with datasets characterized by low number of samples or linear taxon responses. We present an implementation of the rotation forest algorithm in R.

DIAGNOSIS OF MULTICLASS TACHYCARDIA BEATS USING RECURRENCE QUANTIFICATION ANALYSIS AND ENSEMBLE CLASSIFIERS

Atrial Fibrillation (A-Fib), Atrial Flutter (AFL) and Ventricular Fibrillation (V-Fib) are fatal cardiac abnormalities commonly affecting people in advanced age and have indication of life-threatening condition. To detect these abnormal rhythms, Electrocardiogram (ECG) signal is most commonly visualized as a significant clinical tool. Concealed non-linearities in the ECG signal can be clearly unraveled using Recurrence Quantification Analysis (RQA) technique. In this paper, RQA features are applied for classifying four classes of ECG beats namely Normal Sinus Rhythm (NSR), A-Fib, AFL and V-Fib using ensemble classifiers. The clinically significant ([Formula: see text]) features are ranked and fed independently to three classifiers viz. Decision Tree (DT), Random Forest (RAF) and Rotation Forest (ROF) ensemble methods to select the best classifier. The training and testing of the feature set is accomplished using 10-fold cross-validation strategy. The RQA coefficients using ROF provided an overall accuracy of 98.37% against 96.29% and 94.14% for the RAF and DT, respectively. The results achieved evidently ratify the superiority of ROF ensemble classifier in the diagnosis of A-Fib, AFL and V-Fib. Precision of four classes is measured using class-specific accuracy (%) and reliability of the performance is assessed using Cohen’s kappa statistic ([Formula: see text]). The developed approach can be used in therapeutic devices and help the physicians in automatic monitoring of fatal tachycardia rhythms.

Macroscale intraspecific variation and environmental heterogeneity: analysis of cold and warm zone abundance, mortality, and regeneration distributions of four eastern US tree species.

I test for macroscale intraspecific variation of abundance, mortality, and regeneration of four eastern US tree species (Tsuga canadensis,Betula lenta,Liriodendron tulipifera, and Quercus prinus) by splitting them into three climatic zones based on plant hardiness zones (PHZs). The primary goals of the analysis are to assess the differences in environmental heterogeneity and demographic responses among climatic zones, map regional species groups based on decision tree rules, and evaluate univariate and multivariate patterns of species demography with respect to environmental variables. I use the Forest Inventory Analysis (FIA) data to derive abundance, mortality, and regeneration indices and split the range into three climatic zones based on USDA PHZs: (1) cold adapted, leading region; (2) middle, well-adapted region; and (3) warm adapted, trailing region. I employ decision tree ensemble methods to assess the importance of environmental predictors on the abundance of the species between the cold and warm zones and map zonal variations in species groups. Multivariate regression trees are used to simultaneously explore abundance, mortality, and regeneration in tandem to assess species vulnerability. Analyses point to the relative importance of climate in the warm adapted, trailing zone (especially moisture) compared to the cold adapted, leading zone. Higher mortality and lower regeneration patterns in the warm trailing zone point to its vulnerability to growing season temperature and precipitation changes that could figure more prominently in the future. This study highlights the need to account for intraspecific variation of demography in order to understand environmental heterogeneity and differential adaptation. It provides a methodology for assessing the vulnerability of tree species by delineating climatic zones based on easily available PHZ data, and FIA derived abundance, mortality, and regeneration indices as a proxy for overall growth and fitness. Based on decision tree rules, ecologically meaningful variations in species abundance among the climatic zones can be related to environmental variability and mapped.