Ensemble Of Regression Trees Research Articles

Regression Tree Ensemble Rainfall–Runoff Forecasting Model and Its Application to Xiangxi River, China

The development of an efficient and accurate hydrological forecasting model is essential for water management and flood control. In this study, the ensemble model was applied to predict the daily discharge; it not only could enhance the algorithm and improve the learning accuracy, but it was also the most effective representative model among various combinations of learning parameters. Using the survey data of Xingshan station in Xiangxi River, China, the suitability of the model was proven. The performance of the ensemble model was compared with the multiple linear regression model and the artificial neural network models. Furthermore, the length of the training samples and the peak value predictions were analyzed. The results showed that, firstly, the best effect of the discharge simulation model appeared in the ensemble model, while the simulation accuracy of the multiple linear regression model was lower than that of the artificial neural network model in some cases. Secondly, the prediction effect of the ensemble model for discharge was better than that of the single model to some extent, whereby the maximum absolute value of relative error was 8.11% using the ensemble model. A comprehensive analysis showed that the ensemble model was optimal. Furthermore, the ensemble model performed outstandingly in terms of hydrological forecasting. The ensemble model also provided theoretical support for hydrological forecasting and could be considered as an alternative to multiple linear regression models and artificial neural networks.

A Stepwise Clustered Hydrological Model for Addressing the Temporal Autocorrelation of Daily Streamflows in Irrigated Watersheds

AbstractStreamflow simulations at daily time steps are vital to water resources management, especially in arid regions. Previously, data‐driven models have been used as an effective tool for daily streamflow simulation. However, the accuracy of conventional data‐driven approaches is affected by the temporal autocorrelation of daily streamflow, especially in irrigated watersheds where the persistence of saturated flows dominates irrigation seasons. This study presents a Stepwise Clustered Regression Tree Ensemble (SCRTE) to address the streamflow autocorrelation. With the provision of a state‐of‐the‐art data‐driven model Stepwise Cluster Analysis (SCA), the SCRTE enables both single‐ and multi‐output settings (i.e., model predictand can be either a scalar or a vector), which can thus address interactions among streamflow values over multiple consecutive days. The autocorrelation effect of daily streamflow is evaluated based on single‐ and multi‐output SCA ensembles, which can then be aggregated according to their performance for various streamflow quantile ranges. To facilitate the irrigation scheduling decision‐making under rigorous transboundary water regulations, the SCRTE is applied to three interconnected watersheds with mixed land use, located in a floodplain of the Yellow River basin in China. The results show that the SCRTE outperforms seven well‐known benchmark models across seven evaluation metrics. Our findings reveal that the SCRTE can reflect the varying effects of autocorrelation over different streamflow quantile ranges, thereby improving the streamflow simulation. The multi‐output SCA ensembles are more capable of addressing the medium flows, while the single‐output one can better simulate the low and high flows.

Distilled Neural Networks for Efficient Learning to Rank

Recent studies in Learning to Rank have shown the possibility to effectively distill a neural network from an ensemble of regression trees. This result leads neural networks to become a natural competitor of tree-based ensembles on the ranking task. Nevertheless, ensembles of regression trees outperform neural models both in terms of efficiency and effectiveness, particularly when scoring on CPU. In this paper, we propose an approach for speeding up neural scoring time by applying a combination of Distillation, Pruning and Fast Matrix multiplication. We employ knowledge distillation to learn shallow neural networks from an ensemble of regression trees. Then, we exploit an efficiency-oriented pruning technique that performs a sparsification of the most computationally-intensive layers of the neural network that is then scored with optimized sparse matrix multiplication. Moreover, by studying both dense and sparse high performance matrix multiplication, we develop a scoring time prediction model which helps in devising neural network architectures that match the desired efficiency requirements. Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off when compared with tree-based ensembles, providing up to 4x scoring time speed-up without affecting the ranking quality.

Hybrid Machine Learning Approaches and a Systematic Model Selection Process for Predicting Soot Emissions in Compression Ignition Engines

The standards for emissions from diesel engines are becoming more stringent and accurate emission modeling is crucial in order to control the engine to meet these standards. Soot emissions are formed through a complex process and are challenging to model. A comprehensive analysis of diesel engine soot emissions modeling for control applications is presented in this paper. Physical, black-box, and gray-box models are developed for soot emissions prediction. Additionally, different feature sets based on the least absolute shrinkage and selection operator (LASSO) feature selection method and physical knowledge are examined to develop computationally efficient soot models with good precision. The physical model is a virtual engine modeled in GT-Power software that is parameterized using a portion of experimental data. Different machine learning methods, including Regression Tree (RT), Ensemble of Regression Trees (ERT), Support Vector Machines (SVM), Gaussian Process Regression (GPR), Artificial Neural Network (ANN), and Bayesian Neural Network (BNN) are used to develop the black-box models. The gray-box models include a combination of the physical and black-box models. A total of five feature sets and eight different machine learning methods are tested. An analysis of the accuracy, training time and test time of the models is performed using the K-means clustering algorithm. It provides a systematic way for categorizing the feature sets and methods based on their performance and selecting the best method for a specific application. According to the analysis, the black-box model consisting of GPR and feature selection by LASSO shows the best performance with test R2 of 0.96. The best gray-box model consists of SVM-based method with physical insight feature set along with LASSO for feature selection with test R2 of 0.97.

Modelling species presence‐only data with random forests

The random forest (RF) algorithm is an ensemble of classification or regression trees and is widely used, including for species distribution modelling (SDM). Many researchers use implementations of RF in the R programming language with default parameters to analyse species presence‐only data together with ‘background' samples. However, there is good evidence that RF with default parameters does not perform well for such ‘presence‐background' modelling. This is often attributed to the disparity between the number of presence and background samples, also known as 'class imbalance', and several solutions have been proposed. Here, we first set the context: the background sample should be large enough to represent all environments in the region. We then aim to understand the drivers of poor performance of RF when models are fitted to presence‐only species data alongside background samples. We show that 'class overlap' (where both classes occur in the same environment) is an important driver of poor performance, alongside class imbalance. Class overlap can even degrade performance for presence–absence data. We explain, test and evaluate suggested solutions. Using simulated and real presence‐background data, we compare performance of default RF with other weighting and sampling approaches. Our results demonstrate clear evidence of improvement in the performance of RFs when techniques that explicitly manage imbalance are used. We show that these either limit or enforce tree depth. Without compromising the environmental representativeness of the sampled background, we identify approaches to fitting RF that ameliorate the effects of imbalance and overlap and allow excellent predictive performance. Understanding the problems of RF in presence‐background modelling allows new insights into how best to fit models, and should guide future efforts to best deal with such data.

Gradient boosted trees for spatial data and its application to medical imaging data

Boosting Trees are one of the most successful statistical learning approaches that involve sequentially growing an ensemble of simple regression trees (“weak learners”). This paper proposes a gradient Boosted Trees algorithm for Spatial Data (Boost-S) with covariate information. Boost-S integrates the spatial correlation into the classical framework of eXtreme Gradient Boosting. Each tree is constructed by solving a regularized optimization problem, where the objective function takes into account the underlying spatial correlation and involves two penalty terms on tree complexity. A computationally-efficient greedy heuristic algorithm is proposed to obtain an ensemble of trees. The proposed Boost-S is applied to the spatially-correlated FDG-PET (fluorodeoxyglucose-positron emission tomography) imaging data collected from clinical trials of cancer chemoradiotherapy. Our numerical investigations successfully demonstrate the advantages of the proposed Boost-S over existing approaches for this particular application.

Modeling the response of negative air ions to environmental factors using multiple linear regression and random forest

Negative air ion (NAI) plays a vital role in promoting the psychological and physiological functions of the human body and is an essential indicator for measuring the air cleanliness of a given area. In this paper, we presented and compared the results of two methods for identifying the main environmental factors affecting changes of NAI in a warm-temperate region of China. NAI concentration was estimated based on measured data during the main growing season in a warm-temperate forest and was used as the dependent variable in the traditional multiple linear regression and random forest models. Air pollutants and certain weather, radiation, and soil factors were selected as predictors based on their potential influence on NAI. Two methods were applied for the analysis, and the latter was a non-parametric alternative based on an ensemble of classification and regression trees. We compared the precision of the two models, and the variables of each method on the basis of their levels of importance; Independent samples was used in model validation, then we discussed the important environmental factors affecting changes of NAI concentration for both linear and nonlinear perspectives, along with the potential implications of environmental factors on NAI. The random forest model showed a higher accuracy comparison with the multiple linear regression model. Furthermore, the analysis also indicated its better performance by using independent test data for 10-fold cross-validation of the random forest model, and showing that this method has potential for broad-scale application in the assessment of environmental-factor influence on NAI. Certain selected variables that were common to both models (particulate matter 2.5, soil moisture, and relative humidity) appeared to influence NAI to a relatively large extent, demonstrating the decidedly influential role of these parameters on NAI concentration.

Active learning of potential-energy surfaces of weakly bound complexes with regression-tree ensembles

Several pool-based active learning (AL) algorithms were employed to model potential-energy surfaces (PESs) with a minimum number of electronic structure calculations. Theoretical and empirical results suggest that superior strategies can be obtained by sampling molecular structures corresponding to large uncertainties in their predictions while at the same time not deviating much from the true distribution of the data. To model PESs in an AL framework, we propose to use a regression version of stochastic query by forest, a hybrid method that samples points corresponding to large uncertainties while avoiding collecting too many points from sparse regions of space. The algorithm is implemented with decision trees that come with relatively small computational costs. We empirically show that this algorithm requires around half the data to converge to the same accuracy in comparison to the uncertainty-based query-by-committee algorithm. Moreover, the algorithm is fully automatic and does not require any prior knowledge of the PES. Simulations on a 6D PES of pyrrole(H2O) show that <15 000 configurations are enough to build a PES with a generalization error of 16 cm-1, whereas the final model with around 50 000 configurations has a generalization error of 11 cm-1.

Machine Learning to Estimate Surface Roughness from Satellite Images

We apply the Support Vector Regression (SVR) machine learning model to estimate surface roughness on a large alluvial fan of the Kosi River in the Himalayan Foreland from satellite images. To train the model, we used input features such as radar backscatter values in Vertical–Vertical (VV) and Vertical–Horizontal (VH) polarisation, incidence angle from Sentinel-1, Normalised Difference Vegetation Index (NDVI) from Sentinel-2, and surface elevation from Shuttle Radar Topographic Mission (SRTM). We generated additional features (VH/VV and VH–VV) through a linear data fusion of the existing features. For the training and validation of our model, we conducted a field campaign during 11–20 December 2019. We measured surface roughness at 78 different locations over the entire fan surface using an in-house-developed mechanical pin-profiler. We used the regression tree ensemble approach to assess the relative importance of individual input feature to predict the surface soil roughness from SVR model. We eliminated the irrelevant input features using an iterative backward elimination approach. We then performed feature sensitivity to evaluate the riskiness of the selected features. Finally, we applied the dimension reduction and scaling to minimise the data redundancy and bring them to a similar level. Based on these, we proposed five SVR methods (PCA-NS-SVR, PCA-CM-SVR, PCA-ZM-SVR, PCA-MM-SVR, and PCA-S-SVR). We trained and evaluated the performance of all variants of SVR with a 60:40 ratio using the input features and the in-situ surface roughness. We compared the performance of SVR models with six different benchmark machine learning models (i.e., Gaussian Process Regression (GPR), Generalised Regression Neural Network (GRNN), Binary Decision Tree (BDT), Bragging Ensemble Learning, Boosting Ensemble Learning, and Automated Machine Learning (AutoML)). We observed that the PCA-MM-SVR perform better with a coefficient of correlation (R = 0.74), Root Mean Square Error (RMSE = 0.16 cm), and Mean Square Error (MSE = 0.025 cm2). To ensure a fair selection of the machine learning model, we evaluated the Akaike’s Information Criterion (AIC), corrected AIC (AICc), and Bayesian Information Criterion (BIC). We observed that SVR exhibits the lowest values of AIC, corrected AIC, and BIC of all the other methods; this indicates the best goodness-of-fit. Eventually, we also compared the result of PCA-MM-SVR with the surface roughness estimated from different empirical and semi-empirical radar backscatter models. The accuracy of the PCA-MM-SVR model is better than the backscatter models. This study provides a robust approach to measure surface roughness at high spatial and temporal resolutions solely from the satellite data.

Driver Fatigue Detection Based On Facial Feature Analysis

Fatigue driving is one of the main causes of traffic accidents. In recent years, considerable attention has been paid to fatigue detection systems, which is an important solution for preventing fatigue driving. In order to prevent and reduce fatigue driving, a driver fatigue detection system based on computer vision is proposed. In this system, an improved face detection method is used to detect the driver’s face from the image obtained by a charge coupled device (CCD) camera. Then, the feature points of the eyes and mouth are located by an ensemble of regression trees. Next, fatigue characteristic parameters are calculated by the improved percentage of eyelid closure over the pupil over time algorithm. Finally, the state of drivers is evaluated by using a fuzzy neural network. The system can effectively monitor and remind the state of drivers so as to significantly avoid or decrease the occurrence of traffic accidents. The experimental results show that the system is of wonderful real-time performance and accurate recognition rate, so it meets the requirements of practicality in driver fatigue detection greatly.

Online transient stability margin prediction of power systems with wind farms using ensemble regression trees

Online transient stability margin prediction of power systems with wind farms using ensemble regression trees

UAV Parameter Estimation Through Machine Learning

Parameter identification of Unmanned Aerial Vehicles (UAV) is very helpful for understanding cause-effect relationships of physical phenomenon, investigating system performance and characteristics, fault diagnostics, control development/tuning, and more. Traditional ways of performing parameter identification involve establishing a mathematical model that describes the system’s behavior. This process requires knowledge of the physics involved, careful aircraft instrumentation, and special flight maneuvers for thorough excitation of the flight dynamics involved. The purpose of this paper is to show the application of an equation-less identification of a UAV model method using machine learning. The machine learning algorithm is trained with a set of simulation flight data that incorporates variations in the parameters to be identified. To achieve autonomous and consistent flights, a Software-In-the-Loop (SIL) simulation is constructed between X-Plane and Mission Planner. Several machine learning regression models are explored including linear regression, regression trees, gaussian process regression, support vector machine and ensembles of regression trees.

Modeling of Compressive Strength of Self-Compacting Rubberized Concrete Using Machine Learning.

This paper gives a comprehensive overview of the state-of-the-art machine learning methods that can be used for estimating self-compacting rubberized concrete (SCRC) compressive strength, including multilayered perceptron artificial neural network (MLP-ANN), ensembles of MLP-ANNs, regression tree ensembles (random forests, boosted and bagged regression trees), support vector regression (SVR) and Gaussian process regression (GPR). As a basis for the development of the forecast model, a database was obtained from an experimental study containing a total of 166 samples of SCRC. Ensembles of MLP-ANNs showed the best performance in forecasting with a mean absolute error (MAE) of 2.81 MPa and Pearson’s linear correlation coefficient (R) of 0.96. The significantly simpler GPR model had almost the same accuracy criterion values as the most accurate model; furthermore, feature reduction is easy to combine with GPR using automatic relevance determination (ARD), leading to models with better performance and lower complexity.

Two-Level Regression Method Using Ensembles of Trees with Optimal Divergence

The article discusses a new two-level regression analysis method in which a corrective procedure is applied to optimal ensembles of regression trees. Optimization is carried out based on the simultaneous achievement of the divergence of the algorithms in the forecast space and a good approximation of the data by individual algorithms of the ensemble. Simple averaging, random regression forest, and gradient boosting are used as corrective procedures. Experiments are presented comparing the proposed method with the standard decision forest and the standard gradient boosting method for decision trees.

Prognosis of stroke upper limb recovery with physiological variables using regression tree ensembles

Objective. This study assesses upper limb recovery prognosis after stroke with solely physiological information, which can provide an objective estimation of recovery. Approach. Clinical recovery was forecasted using EEG-derived Event-Related Desynchronization/Synchronization and coherence, in addition to Transcranial Magnetic Stimulation elicited motor-evoked potentials and upper limb grip and pinch strength. A Regression Tree Ensemble predicted clinical recovery of a stroke database (n = 10) measured after a two-month intervention with the Fugl-Meyer Assessment for the Upper Extremity (FMA-UE) and the Action Research Arm Test (ARAT). Main results. There were no significant differences between predicted and actual outcomes with FMA-UE (p = 0.29) and ARAT (p = 0.5). Median prediction error for FMA-UE and ARAT were of 0.3 (IQR = 6.2) and 3.4 (IQR = 9.4) points, respectively. Predictions with the most pronounced errors were due to an underestimation of high upper limb recovery. The best features for FMA-UE prediction included mostly beta activity over the sensorimotor cortex. Best ARAT prediction features were cortical beta activity, corticospinal tract integrity of the unaffected hemisphere, and upper limb strength. Significance. Results highlighted the importance of measuring cortical activity related to motor control processes, the unaffected hemisphere’s integrity, and upper limb strength for prognosis. It was also implied that stroke upper limb recovery prediction is feasible using solely physiological variables with a Regression Tree Ensemble, which can also be used to analyze physiological relationships with recovery.

Financial time series analysis and forecasting with Hilbert–Huang transform feature generation and machine learning

AbstractWe present the method of complementary ensemble empirical mode decomposition and Hilbert–Huang transform (HHT) for analyzing nonstationary financial time series. This noise‐assisted approach decomposes any time series into a number of intrinsic mode functions, along with the corresponding instantaneous amplitudes and instantaneous frequencies. Different combinations of modes allow us to reconstruct the time series using components of different timescales. We then apply Hilbert spectral analysis to define and compute the associated instantaneous energy‐frequency spectrum to illustrate the properties of various timescales embedded in the original time series. Using HHT, we generate a collection of new features and integrate them into machine learning models, such as regression tree ensemble, support vector machine, and long short‐term memory neural network. Using empirical financial data, we compare several HHT‐enhanced machine learning models in terms of forecasting performance.

Sustainability Performance Assessment Using Self-Organizing Maps (SOM) and Classification and Ensembles of Regression Trees (CART)

This study aims to develop a new approach based on machine learning techniques to assess sustainability performance. Two main dimensions of sustainability, ecological sustainability, and human sustainability, were considered in this study. A set of sustainability indicators was used, and the research method in this study was developed using cluster analysis and prediction learning techniques. A Self-Organizing Map (SOM) was applied for data clustering, while Classification and Regression Trees (CART) were applied to assess sustainability performance. The proposed method was evaluated through Sustainability Assessment by Fuzzy Evaluation (SAFE) dataset, which comprises various indicators of sustainability performance in 128 countries. Eight clusters from the data were found through the SOM clustering technique. A prediction model was found in each cluster through the CART technique. In addition, an ensemble of CART was constructed in each cluster of SOM to increase the prediction accuracy of CART. All prediction models were assessed through the adjusted coefficient of determination approach. The results demonstrated that the prediction accuracy values were high in all CART models. The results indicated that the method developed by ensembles of CART and clustering provide higher prediction accuracy than individual CART models. The main advantage of integrating the proposed method is its ability to automate decision rules from big data for prediction models. The method proposed in this study could be implemented as an effective tool for sustainability performance assessment.

Tree-based models for variable annuity valuation: parameter tuning and empirical analysis

AbstractVariable annuities have become popular retirement and investment vehicles due to their attractive guarantee features. Nonetheless, managing the financial risks associated with the guarantees poses great challenges for insurers. One challenge is risk quantification, which involves frequent valuation of the guarantees. Insurers rely on the use of Monte Carlo simulation for valuation as the guarantees are too complicated to be valued by closed-form formulas. However, Monte Carlo simulation is computationally intensive. In this paper, we empirically explore the use of tree-based models for constructing metamodels for the valuation of the guarantees. In particular, we consider traditional regression trees, tree ensembles, and trees based on unbiased recursive partitioning. We compare the performance of tree-based models to that of existing models such as ordinary kriging and generalised beta of the second kind (GB2) regression. Our results show that tree-based models are efficient in producing accurate predictions and the gradient boosting method is considered the most superior in terms of prediction accuracy.

Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive

Objectives Face swapping technology has important application value in entertainment, virtual reality, film and so on. However, existing methods are limited by face pose consistency and can not overcome the influence of occlusion. Methods We proposes a method of face swapping using convolutional neural network and tiny facet primitive. Firstly, detect the face using the cascade convolutional neural network and segment the face to determine the replacement region using fully convolutional network.Then, the Wallis transform is applied to adjust the skin color of the source image to make it consistent with the skin color of the face in the target image.After that, using facial key points detection method based on an ensemble of regression trees and Delaunay triangulation to construct the face triangulation network, then replacing faces based on tiny facet primitive. Finally, applying Poisson fusion to eliminate splicing traces between different images. Results We evaluate the performance of the proposed method compared with existing method through qualitative and quantitative experiments. Experimental results show that face segment can well solve the problem of occlusion such as cap, glasses, and hair.Moreover, when source image and target image have different face poses, replacing face area using tiny facet primitive separately performs better than using the whole face area. Conclusions Our method can well solve the problem of face pose consistency limitation and occlusion, which has certain practical application value.

Indirect Estimation of Vertical Ground Reaction Force from a Body-Mounted INS/GPS Using Machine Learning.

Vertical ground reaction force (vGRF) can be measured by force plates or instrumented treadmills, but their application is limited to indoor environments. Insoles remove this restriction but suffer from low durability (several hundred hours). Therefore, interest in the indirect estimation of vGRF using inertial measurement units and machine learning techniques has increased. This paper presents a methodology for indirectly estimating vGRF and other features used in gait analysis from measurements of a wearable GPS-aided inertial navigation system (INS/GPS) device. A set of 27 features was extracted from the INS/GPS data. Feature analysis showed that six of these features suffice to provide precise estimates of 11 different gait parameters. Bagged ensembles of regression trees were then trained and used for predicting gait parameters for a dataset from the test subject from whom the training data were collected and for a dataset from a subject for whom no training data were available. The prediction accuracies for the latter were significantly worse than for the first subject but still sufficiently good. K-nearest neighbor (KNN) and long short-term memory (LSTM) neural networks were then used for predicting vGRF and ground contact times. The KNN yielded a lower normalized root mean square error than the neural network for vGRF predictions but cannot detect new patterns in force curves.