Ensemble Of Random Trees Research Articles

Ensembles of random trees with coalitions - a classification model for dispersed data

The paper delves into the challenge of classification using dispersed data gathered from independent sources. The examined approach involves local models as ensembles of random trees constructed based on local data and randomly selected attributes. In the proposed model, a conflict analysis is used to identify the coalitions of local models. Finally, four different strategies for generating final decisions are explored: utilizing coalitions with and without weights and allowing one or two of the strongest coalitions to make decisions. The paper demonstrates that, regardless of the chosen method for making the final decision, the proposed model with conflict analysis and coalitions obtained better results, particularly in terms of the F1 measure and accuracy, compared to approaches from the literature, such as random forest or a single tree generated based on each local table separately with majority voting.

Mode choice analysis of school trips using random forest technique

Mode choice analysis of school trips becomes important due to the fact that these trips contribute to the second largest share of peak hour traffic. This scenario is more relevant in India, which has almost 265 million students enrolled in different accredited urban and rural schools of India, from Class I to XII as per the UDISE report of 2019-20. Thus, it becomes necessary to understand what mode of transport will be mostly used for school trips in order to design an efficient transportation system. Modal attributes and socio-economic characteristics are mostly considered as explana-tory variables in travel mode choice models. Multinomial Logit (MNL) model is one of the classic models used in the development of mode choice models. These logistic regression models predict outcomes based on a set of independent variables. With the recent advances in machine learning, transportation problems are getting a wide arena of methods and solutions. Among them the method of ensemble learning is finding a prominent place in contemporary modelling. This study explores the potential of using ensembles of random decision trees in mode choice analysis by Random Forest Technique with a comparative analysis on conventional method. It was observed that Random Forest method outperforms MNL method in predicting the mode choice preference of students. The high accuracy of machine learning models is mainly due to its ability to consider complex nonlinear relationship between socio-economic attributes and travel mode choice. These models can learn and identify pattern characteristics extracted from sample data and form adaptive structures through computational process thereby offering insights into the relationships between variables that random utility models cannot recognize. This study considered activity -travel information, personal data and household characteristics of students as attributes for model development and observed that the age of the student and distance of school from home plays a significant role in deciding the mode choice of school trips.

A General Iterative Clustering Algorithm.

The quality of a cluster analysis of unlabeled units depends on the quality of the between units dissimilarity measures. Data‐dependent dissimilarity is more objective than data independent geometric measures such as Euclidean distance. As suggested by Breiman, many data driven approaches are based on decision tree ensembles, such as a random forest (RF), that produce a proximity matrix that can easily be transformed into a dissimilarity matrix. An RF can be obtained using labels that distinguish units with real data from units with synthetic data. The resulting dissimilarity matrix is input to a clustering program and units are assigned labels corresponding to cluster membership. We introduce a general iterative cluster (GIC) algorithm that improves the proximity matrix and clusters of the base RF. The cluster labels are used to grow a new RF yielding an updated proximity matrix, which is entered into the clustering program. The process is repeated until convergence. The same procedure can be used with many base procedures such as the extremely randomized tree ensemble. We evaluate the performance of the GIC algorithm using benchmark and simulated data sets. The properties measured by the Silhouette score are substantially superior to the base clustering algorithm. The GIC package has been released in R: https://cran.r‐project.org/web/packages/GIC/index.html.

An ensemble of random decision trees with local differential privacy in edge computing

Edge computing is an emerging computing paradigm, which offers a great opportunity to implement data mining-based services and applications for a large number of devices and sensors in Internet of Things. However, the new paradigm is faced with security and privacy challenges due to the diversity and the limited capability of edge components. In particular, data privacy is one of the most concerned problems for all the participants. In this paper, we propose a framework of privacy-preserving data mining based on private random decision trees in edge computing, which not only gives the strong privacy guarantee, but also provides a certain amount of data utility. Firstly, we design a preservation framework to implement private random decision trees satisfying local differential privacy. Secondly, we present the concrete implementations of algorithms and the corresponding task that each participant needs to undertake. Thirdly, we analyze the key factors to influence privacy and utility, including the allocation of data and privacy budget. Fourthly, we give the improved algorithms to further increase the utility with strong privacy preservation. Finally, extensive experiments demonstrate the good performance of our designed framework.

Identifying Drone Operator by Deep Learning and Ensemble Learning of IMU and Control Data

Drone flight controls and ground stations are known to be vulnerable to attacks. Besides posing a threat to integrity and confidentiality of drone data, their vulnerabilities endanger safety. Onboard continuous authentication is a vital countermeasure to hijacking attempts. Motivated by the success of Machine Learning (ML) techniques in the field of behavioral biometrics, this paper investigates the use of sensor readings generated onboard drones and of control data reaching them from the ground to feed an onboard ML model continuously authenticating pilots. We analyze fifteen inertial measurement units (IMU) and four radio control signals obtained from the drone's onboard sensors or coming from its remote controller, to identify the controlling pilot. We investigate three sequence classification schemes. In the first scheme, raw sensor sequences are directly fed to a deep Long/Short-Term Memory (LSTM) learner. In the second scheme, frequency-domain features are extracted from the data sequences and interpreted by an ensemble of random trees. In the third scheme, instantaneous sensor readings are classified using the same ensemble learning technique as in the second scheme, yet a final decision fusion method is adopted to provide a sequence-based decision. We compare the three schemes in terms of accuracy, complexity, and delay. The winning scheme is validated and tested against an unseen intruder scenario. Our tests show that an LSTM model trained with data from 19 users is able to identify the operating user at a 97% accuracy, while it can identify an unknown intruder at an average accuracy of 73%.

Models for Heart Failure Admissions and Admission Rates, 2016 through 2018.

Background: Approximately 6.5 to 6.9 million individuals in the United States have heart failure, and the disease costs approximately $43.6 billion in 2020. This research provides geographical incidence and cost models of this disease in the U.S. and explanatory models to account for hospitals’ number of heart failure DRGs using technical, workload, financial, geographical, and time-related variables. Methods: The number of diagnoses is forecast using regression (constrained and unconstrained) and ensemble (random forests, extra trees regressor, gradient boosting, and bagging) techniques at the hospital unit of analysis. Descriptive maps of heart failure diagnostic-related groups (DRGs) depict areas of high incidence. State- and county-level spatial and non-spatial regression models of heart failure admission rates are performed. Expenditure forecasts are estimated. Results: The incidence of heart failure has increased over time with the highest intensities in the East and center of the country; however, several Northern states have seen large increases since 2016. The best predictive model for the number of diagnoses (hospital unit of analysis) was an extremely randomized tree ensemble (predictive R2 = 0.86). The important variables in this model included workload metrics and hospital type. State-level spatial lag models using first-order Queen criteria were best at estimating heart failure admission rates (R2 = 0.816). At the county level, OLS was preferred over any GIS model based on Moran’s I and resultant R2; however, none of the traditional models performed well (R2 = 0.169 for the OLS). Gradient-boosted tree models predicted 36% of the total sum of squares; the most important factors were facility workload, mean cash on hand of the hospitals in the county, and mean equity of those hospitals. Online interactive maps at the state and county levels are provided. Conclusions. Heart failure and associated expenditures are increasing. Costs of DRGs in the study increased $61 billion from 2016 through 2018. The increase in the more expensive DRG 291 outpaced others with an associated increase of $92 billion. With the increase in demand and steady-state supply of cardiologists, the costs are likely to balloon over the next decade. Models such as the ones presented here are needed to inform healthcare leaders.

Isolation Forests to Evaluate Class Separability and the Representativeness of Training and Validation Areas in Land Cover Classification

Supervised land cover classification from remote sensing imagery is based on gathering a set of training areas to characterise each of the classes and to train a predictive model that is then used to predict land cover in the rest of the image. This procedure relies mainly on the assumptions of statistical separability of the classes and the representativeness of the training areas. This paper uses isolation forests, a type of random tree ensembles, to analyse both assumptions and to easily correct lack of representativeness by digitising new training areas where needed to improve the classification of a Landsat-8 set of images with Random Forest. The results show that the improved set of training areas after the isolation forest analysis is more representative of the whole image and increases classification accuracy. Besides, the distribution of isolation values can be useful to estimate class separability. A class separability parameter that summarises such distributions is proposed. This parameter is more correlated to omission and commission errors than other separability measures such as the Jeffries–Matusita distance.

Applying a random forest method approach to model travel mode choice behavior

The analysis of travel mode choice is important in transportation planning and policy-making in order to understand and forecast travel demands. Research in the field of machine learning has been exploring the use of random forest as a framework within which many traffic and transport problems can be investigated. The random forest (RF) is a powerful method for constructing an ensemble of random decision trees. It de-correlates the decision trees in the ensemble via randomization that leads to an improvement of forecasting and reduces the variance when averaged over the trees. However, the usefulness of RF for travel mode choice behavior remains largely unexplored. This paper proposes a robust random forest method to analyze travel mode choices for examining the prediction capability and model interpretability. Using the travel diary data from Nanjing, China in 2013, enriched with variables on the built environment, the effects of different model parameters on the prediction performance are investigated. The comparison results show that the random forest method performs significantly better in travel mode choice prediction for higher accuracy and less computation cost. In addition, the proposed method estimates the relative importance of explanatory variables and how they relate to mode choices. This is fundamental for a better understanding and effective modeling of people’s travel behavior.

Effective mammogram classification based on center symmetric-LBP features in wavelet domain using random forests.

Mammogram classification is a crucial and challenging problem, because it helps in early diagnosis of breast cancer and supports radiologists in their decision to analyze similar mammograms out of a database by recognizing the classes of current mammograms. This paper proposes an effective method for classifying mammograms using random forests with wavelet based center-symmetric local binary pattern (WCS-LBP). To classify mammograms, multi-resolution CS-LBP texture characteristics from non-overlapping regions of the mammograms are captured. Further, we examine most relevant features using support vector machine-recursive feature elimination (SVM-RFE). Finally, we feed the selected features to decision trees and construct random forests which are an ensemble of random decision trees. Using wavelet based local CS-LBP features with random forest, we classify the test images into different categories having the maximum posterior probability. The proposed method shows the improved performance as compared with other variant features and state-of-art methods. The obtained performance measures are 97.3% accuracy, 97.3% precision, 97.2% recall, 97.2% F-measure and 94.1% Matthews correlation coefficient (MCC).

Differentially private random decision forests using smooth sensitivity

We propose a new differentially-private decision forest algorithm that minimizes both the number of queries required, and the sensitivity of those queries. To do so, we build an ensemble of random decision trees that avoids querying the private data except to find the majority class label in the leaf nodes. Rather than using a count query to return the class counts like the current state-of-the-art, we use the Exponential Mechanism to only output the class label itself. This drastically reduces the sensitivity of the query – often by several orders of magnitude – which in turn reduces the amount of noise that must be added to preserve privacy. Our improved sensitivity is achieved by using “smooth sensitivity”, which takes into account the specific data used in the query rather than assuming the worst-case scenario. We also extend work done on the optimal depth of random decision trees to handle continuous features, not just discrete features. This, along with several other improvements, allows us to create a differentially private decision forest with substantially higher predictive power than the current state-of-the-art.

Predicting ENR Construction Cost Index Using Machine-Learning Algorithms

ABSTRACTConstruction Cost Index (CCI) is calculated monthly and published by Engineering News-Record (ENR). CCI is utilized for capital project budgeting and construction cost estimation, especially in cases where mid- and long-term forecasts are needed. Accurate prediction of CCI helps avoid underestimating and overestimating project costs but the current prevailing time series prediction models do not show promising results, especially in mid- and long-term forecasting. The capability of two machine-learning algorithms, k nearest neighbor (k-NN) and perfect random tree ensembles (PERT), are utilized to enhance CCI forecasting, especially in the mid- and long-term. The proposed machine-learning algorithms are able to significantly enhance the predictability of forecasting CCI in all the scenarios, short-, mid-, and long-term. Data from January 1985 to December 2014 is collected from ENR and bureau of labor statistics to conduct empirical studies and quantitatively measure the performance of the proposed methods. As the outcomes show, the prediction accuracies of both proposed methods are better than those of current prevailing time series models under all the tested scenarios. It is anticipated that cost estimators can benefit from CCI forecasting by incorporating predicted price variations in their estimates and preparing more-precise bids for contractors and developing more-accurate budgets for owners.

Random Composite Forests

We introduce a broad family of decision trees, Composite Trees, whose leaf classifiers are selected out of a hypothesis set composed of p subfamilies with different complexities. We prove new data-dependent learning guarantees for this family in the multi-class setting. These learning bounds provide a quantitative guidance for the choice of the hypotheses at each leaf. Remarkably, they depend on the Rademacher complexities of the sub-families of the predictors and the fraction of sample points correctly classified at each leaf. We further introduce random composite trees and derive learning guarantees for random composite trees which also apply to Random Forests. Using our theoretical analysis, we devise a new algorithm, RANDOMCOMPOSITEFORESTS (RCF), that is based on forming an ensemble of random composite trees. We report the results of experiments demonstrating that RCF yields significant performance improvements over both Random Forests and a variant of RCF in several tasks.

Bounds on the moments for an ensemble of random decision trees

An ensemble of random decision trees is a popular classification technique, especially known for its ability to scale to large domains. In this paper, we provide an efficient strategy to compute bounds on the moments of the generalization error computed over all datasets of a particular size drawn from an underlying distribution, for this classification technique. Being able to estimate these moments can help us gain insights into the performance of this model. As we will see in the experimental section, these bounds tend to be significantly tighter than the state-of-the-art Breiman's bounds based on strength and correlation and hence more useful in practice.

Confidence in predictions from random tree ensembles

Obtaining an indication of confidence of predictions is desirable for many data mining applications. Predictions complemented with confidence levels can inform on the certainty or extent of reliability that may be associated with the prediction. This can be useful in varied application contexts where model outputs form the basis for potentially costly decisions, and in general across risk sensitive applications. The conformal prediction framework presents a novel approach for obtaining valid confidence measures associated with predictions from machine learning algorithms. Confidence levels are obtained from the underlying algorithm, using a non-conformity measure which indicates how ‘atypical’ a given example set is. The non-conformity measure is a key to determining the usefulness and efficiency of the approach. This paper considers inductive conformal prediction in the context of random tree ensembles like random forests, which have been noted to perform favorably across problems. Focusing on classification tasks, and considering realistic data contexts including class imbalance, we develop non-conformity measures for assessing the confidence of predicted class labels from random forests. We examine the performance of these measures on multiple data sets. Results demonstrate the usefulness and validity of the measures, their relative differences, and highlight the effectiveness of conformal prediction random forests for obtaining predictions with associated confidence.

Spectra of Large Random Trees

We analyze the eigenvalues of the adjacency matrices of a wide variety of random trees. Using general, broadly applicable arguments based on the interlacing inequalities for the eigenvalues of a principal submatrix of a Hermitian matrix and a suitable notion of local weak convergence for an ensemble of random trees that we call probability fringe convergence, we show that the empirical spectral distributions for many random tree models converge to a deterministic (model-dependent) limit as the number of vertices goes to infinity.Moreover, the masses assigned by the empirical spectral distributions to individual points also converge in distribution to constants. We conclude for ensembles such as the linear preferential attachment models, random recursive trees, and the uniform random trees that the limiting spectral distribution has a set of atoms that is dense in the real line. We obtain lower bounds on the mass assigned to zero by the empirical spectral measures via the connection between the number of zero eigenvalues of the adjacency matrix of a tree and the cardinality of a maximal matching on the tree. In particular, we employ a simplified version of an algorithm due to Karp and Sipser to construct maximal matchings and understand their properties. Moreover, we show that the total weight of a weighted matching is asymptotically equivalent to a constant multiple of the number of vertices when the edge weights are independent, identically distributed, nonnegative random variables with finite expected value, thereby significantly extending a result obtained by Aldous and Steele in the special case of uniform random trees.We greatly generalize a celebrated result obtained by Schwenk for the uniform random trees by showing that if any ensemble converges in the probability fringe sense and a very mild further condition holds, then, with probability converging to one, the spectrum of a realization is shared by at least one other (nonisomorphic) tree.For the linear preferential attachment model with parameter a>−1, we show that for any fixed k, the k largest eigenvalues jointly converge in distribution to a nontrivial limit when rescaled by \(n^{1/2\gamma_{a}}\), where γ a =a+2 is the Malthusian rate of growth parameter for an associated continuous-time branching process.

X-ray Image Classification Using Random Forests with Local Wavelet-Based CS-Local Binary Patterns

This paper presents a fast and efficient method for classifying X-ray images using random forests with proposed local wavelet-based local binary pattern (LBP) to improve image classification performance and reduce training and testing time. Most studies on local binary patterns and its modifications, including centre symmetric LBP (CS-LBP), focus on using image pixels as descriptors. To classify X-ray images, we first extract local wavelet-based CS-LBP (WCS-LBP) descriptors from local parts of the images to describe the wavelet-based texture characteristic. Then we apply the extracted feature vector to decision trees to construct random forests, which are an ensemble of random decision trees. Using the random forests with local WCS-LBP, we classified one test image into the category having the maximum posterior probability. Compared with other feature descriptors and classifiers, the proposed method shows both improved performance and faster processing time.

Learning Random Model Trees for Regression

Regression is one of the most important tasks in real-world data mining applications. Among a large number of regression models, model tree is an excellent regression model. In this paper, we single out an improved model tree algorithm via introducing randomness into the process of building model trees. We call our improved algorithm random model trees, simply RMT. RMT firstly builds an ensemble of random model trees and then averages the predictions of these random trees to predict the target value of an unseen instance. In building each random model tree, the split is selected at random from the best k splits at each non-terminal node. We experimentally test its accuracy on the 36 benchmark datasets, and compared it with some interrelated regression models. The experimental results show that RMT significantly outperforms all the other algorithms used to compare. Our work provides an effective data mining algorithm for applications especially when high-accuracy regression is required.

Learning random forests for ranking

The random forests (RF) algorithm, which combines the predictions from an ensemble of random trees, has achieved significant improvements in terms of classification accuracy. In many real-world applications, however, ranking is often required in order to make optimal decisions. Thus, we focus our attention on the ranking performance of RF in this paper. Our experimental results based on the entire 36 UC Irvine Machine Learning Repository (UCI) data sets published on the main website of Weka platform show that RF doesn't perform well in ranking, and is even about the same as a single C4.4 tree. This fact raises the question of whether several improvements to RF can scale up its ranking performance. To answer this question, we single out an improved random forests (IRF) algorithm. Instead of the information gain measure and the maximum-likelihood estimate, the average gain measure and the similarity-weighted estimate are used in IRF. Our experiments show that IRF significantly outperforms all the other algorithms used to compare in terms of ranking while maintains the high classification accuracy characterizing RF.

Long-range disassortative correlations in generic random trees

We explicitly calculate the distance dependent correlation functions in a maximal entropy ensemble of random trees. We show that correlations remain disassortative at all distances and vanish only as a second inverse power of the distance. We discuss in detail the example of scale-free trees where the diverging second moment of the degree distribution leads to some interesting phenomena.

The method of ensembles of decision trees for the analysis of the electrical activity produced by the human brain

In this paper, we develop a method for localization of the active domains of the brain by electroencephalography signals on the basis of ensembles of random decision trees. We suggest a method for reducing the problem of localization to the problem of classification of the quality of the dipole sources. We present a localization algorithm, which consists in constructing an ensemble of decision trees for the parameters of the dipole sources (which are responsible for the approximation of the potential detected at various time instants) and finding the most probable source regions on the basis of a special voting procedure. It is demonstrated that the approach that employs the decision trees technique allows a stable determination of the parameters of the transient function between the source and the signal detected, which is essential for the construction of the brain-computer interface. The method suggested is demonstrated to converge both on the exact solutions and on signals of real experiments by the analysis of evoked potentials.