Boosted Tree Method Research Articles

Gradient boosting trees method in the task of software identification

Subject of Research.The paper proposes an approach to the use of gradient boosted decision trees algorithm. For this purpose, CatBoost algorithm developed by Yandex is proposed. Its implementation is aimed at the problem solution of OS Linux software identification in order to reduce the number of system vulnerabilities, which occur due to the installation of unauthorized software by automated system users. We consider an approach to the program signatures formation and further training of CatBoostClassifier classifier model. The subsequent recognition task is set for the identified programs that were not previously involved in the model training process. Method. Free CatBoost software was used for implementation of the gradient boosted decision trees algorithm. CatBoostClassifier multi-classification model was created on its basis. The use of this model allows identifying test sample elf-files.Main Results. The training parameters of the classification model are selected. An experiment is carried out to identify elf-files with the use of ten different featuresof emerging signature programs. The results obtained in the new approach are compared with the results of the previously developed method of identification based on the application of the statistical criterion of Chi-square homogeneity at the significance level p = 0.01. Practical Relevance. The results of the study can be recommended to information security specialists for data media audit. The developed approach gives the possibility to identify violations of the established security policy in the processing of confidential information.

Cyanotoxin level prediction in a reservoir using gradient boosted regression trees: a case study.

Cyanotoxins are a type of cyanobacteria that is poisonous and poses a health threat in waters that could be used for drinking or recreational purposes. Thus, it is necessary to predict their presence to avoid risks. This paper presents a nonparametric machine learning approach using a gradient boosted regression tree model (GBRT) for prediction of cyanotoxin contents from cyanobacterial concentrations determined experimentally in a reservoir located in the north of Spain. GBRT models seek and obtain good predictions in highly nonlinear problems, like the one treated here, where the studied variable presents low concentrations of cyanotoxins mixed with high concentration peaks. Two types of results have been obtained: firstly, the model allows the ranking or the dependent variables according to its importance in the model. Finally, the high performance and the simplicity of the model make the gradient boosted tree method attractive compared to conventional forecasting techniques.

Theoretical validation of potential habitability via analytical and boosted tree methods: An optimistic study on recently discovered exoplanets

Abstract Seven Earth-sized planets, known as the TRAPPIST-1 system, was discovered with great fanfare in the last week of February 2017. Three of these planets are in the habitable zone of their star, making them potentially habitable planets (PHPs) a mere 40 light years away. The discovery of the closest potentially habitable planet to us just a year before — Proxima b and a realization that Earth-type planets in circumstellar habitable zones are a common occurrence provides the impetus to the existing pursuit for life outside the Solar System. The search for life has two goals essentially: looking for planets with Earth-like conditions (Earth similarity) and looking for the possibility of life in some form (habitability). An index was recently developed, the Cobb–Douglas Habitability Score (CDHS), based on Cobb–Douglas habitability production function (CD-HPF), which computes the habitability score by using measured and estimated planetary parameters. As an initial set, radius, density, escape velocity and surface temperature of a planet were used. The proposed metric, with exponents accounting for metric elasticity, is endowed with analytical properties that ensure global optima and can be scaled to accommodate a finite number of input parameters. We show here that the model is elastic, and the conditions on elasticity to ensure global maxima can scale as the number of predictor parameters increase. K-NN (K-Nearest Neighbor) classification algorithm, embellished with probabilistic herding and thresholding restriction, utilizes CDHS scores and labels exoplanets into appropriate classes via feature-learning methods yielding granular clusters of habitability. The algorithm works on top of a decision-theoretical model using the power of convex optimization and machine learning. The goal is to characterize the recently discovered exoplanets into an “Earth League” and several other classes based on their CDHS values. A second approach, based on a novel feature-learning and tree-building method classifies the same planets without computing the CDHS of the planets and produces a similar outcome. For this, we use XGBoosted trees. The convergence of the outcome of the two different approaches indicates the strength of the proposed solution scheme and the likelihood of the potential habitability of the recently announced discoveries.

Connecting Social Media to E-Commerce: Cold-Start Product Recommendation using Micro blogging Information

In recent years, the edge between e-commerce and social networking have become increasingly blurred. Many e-commerce websites support the mechanism of social login where users can sign on the websites using their social network identities such as their Facebook or Twitter accounts. Users can also post their newly purchased products on microblogs with links to the e-commerce product web pages. In this paper we represent a novel solution for cross-site cold-start product recommendation, which aims to recommend products from e-commerce websites to users at social networking sites in “coldstart” situations, a problem which has rarely been explored before. A major threat is how to leverage knowledge extracted from social networking sites for cross-site cold-start product recommendation. We propose to use the linked users across social networking sites and e-commerce websites (users who have social networking accounts and have made purchases on e-commerce websites) as a bridge to map users’ social networking features to another feature representation for product recommendation. In specific, we propose learning both users’ and products’ feature representations (called user embeddings and product embeddings, respectively) from data collected from e-commerce websites using recurrent neural networks and then apply a modified gradient boosting trees method to transform users’ social networking features into user embeddings. We then develop a feature-based matrix factorization approach which can leverage the learnt user embeddings for cold-start product recommendation. Experimental calculation on a large dataset build from the largest Chinese micro blogging service SINA WEIBO and the largest Chinese B2C e-commerce website JINGDONG have given the effectiveness of our proposed framework.

Guiding multimodal registration with learned optimization updates.

In this paper, we address the multimodal registration problem from a novel perspective, aiming to predict the transformation aligning images directly from their visual appearance. We formulate the prediction as a supervised regression task, with joint image descriptors as input and the output are the parameters of the transformation that guide the moving image towards alignment. We model the joint local appearance with context aware descriptors that capture both local and global cues simultaneously in the two modalities, while the regression function is based on the gradient boosted trees method capable of handling the very large contextual feature space. The good properties of our predictions allow us to couple them with a simple gradient-based optimization for the final registration. Our approach can be applied to any transformation parametrization as well as a broad range of modality pairs. Our method learns the relationship between the intensity distributions of a pair of modalities by using prior knowledge in the form of a small training set of aligned image pairs (in the order of 1-5 in our experiments). We demonstrate the flexibility and generality of our method by evaluating its performance on a variety of multimodal imaging pairs obtained from two publicly available datasets, RIRE (brain MR, CT and PET) and IXI (brain MR). We also show results for the very challenging deformable registration of Intravascular Ultrasound and Histology images. In these experiments, our approach has a larger capture range when compared to other state-of-the-art methods, while improving registration accuracy in complex cases.

Pattern recognition of rigid hoisting guides based on vibration characteristics

A test rig is built to simulate the typical fault patterns of rigid hoisting guides and to collect vibration and inclination signals. In this work, we use these signals to perform data mining for fault-pattern recognition. Parameters are initially defined by analyzing collected signals. Then, the importance of each parameter is calculated using the boosting-tree method. Some valuable parameters are retained. To establish a data-mining algorithm that works remarkably for the fault recognition of rigid hoisting guides, six different algorithms including the boosting tree, K-nearest neighbor, MARSpline, neural network, random forest, and support vector machine are compared. Results show that the best performance is that of the boosting-tree algorithm, whose mechanism is then presented in detail.

Assessment of the possibility of using data mining methods to predict sorption isotherms of selected organic compounds on activated carbon

The paper analyses the use of four data mining methods (Support Vector Machines. Cascade Neural Networks. Random Forests and Boosted Trees) to predict sorption on activated carbons. The input data for statistical models included the activated carbon parameters, organic substances and equilibrium concentrations in the solution. The assessment of the predictive abilities of the developed models was made with the use of mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean squared error (RMSE). The computations proved that methods of data mining considered in the study can be applied to predict sorption of selected organic compounds 011 activated carbon. The lowest values of sorption prediction errors were obtained with the Cascade Neural Networks method (MAE = 1.23 g/g; MAPE = 7.90% and RMSE = 1.81 g/g), while the highest error values were produced by the Boosted Trees method (MAE=14.31 g/g; MAPE = 39.43% and RMSE = 27.76 g/g).

Connecting Social Media to E-Commerce: Cold-Start Product Recommendation Using Microblogging Information

In recent years, the boundaries between e-commerce and social networking have become increasingly blurred. Many e-commerce websites support the mechanism of social login where users can sign on the websites using their social network identities such as their Facebook or Twitter accounts. Users can also post their newly purchased products on microblogs with links to the e-commerce product web pages. In this paper, we propose a novel solution for cross-site cold-start product recommendation , which aims to recommend products from e-commerce websites to users at social networking sites in “cold-start” situations, a problem which has rarely been explored before. A major challenge is how to leverage knowledge extracted from social networking sites for cross-site cold-start product recommendation. We propose to use the linked users across social networking sites and e-commerce websites (users who have social networking accounts and have made purchases on e-commerce websites) as a bridge to map users’ social networking features to another feature representation for product recommendation. In specific, we propose learning both users’ and products’ feature representations (called user embeddings and product embeddings, respectively) from data collected from e-commerce websites using recurrent neural networks and then apply a modified gradient boosting trees method to transform users’ social networking features into user embeddings. We then develop a feature-based matrix factorization approach which can leverage the learnt user embeddings for cold-start product recommendation. Experimental results on a large dataset constructed from the largest Chinese microblogging service Sina Weibo and the largest Chinese B2C e-commerce website JingDong have shown the effectiveness of our proposed framework.

Comparison of classification methods with “n-class” receiver operating characteristic curves: A case study of energy drinks

Four classification methods were compared using receiver operating characteristic (ROC) curves to identify the best one: two common ones (linear discriminant analysis, LDA and partial least squares, PLS) and another two (random forest, boosted tree) that are not applied as frequently as LDA or PLS yet. A dataset with 90 commercially available (in Hungary) energy drink samples were studied. Near-infrared (NIR) spectra were utilized for the classification of the energy drinks into three “natural” groups based on their sugar content. Another dataset, which contained the first ten principal components (PCs) was also used because of the limitation in the number of variables for LDA. The models were validated using n-fold cross-validation and randomization test. A new practice was elaborated to compare the pattern recognition methods with ROC curves. This new methodology was designed to provide an easy and straightforward way for the calculation of ROC curves for multi-class classification problems.In each case the energy drink samples could be classified to the appropriate groups very accurately. The best ROC curve belonged to the boosted tree method, but all of the studied methods were able to classify the samples to a great extent of correctness. The use of AUC values instead of correct classification rates can be a viable option for method comparison and also as a classification parameter.

Antimüllerian hormone levels and antral follicle count as prognostic indicators in a personalized prediction model of live birth

To compare antimüllerian hormone (AMH) and antral follicle count (AFC) separately and in combination with clinical characteristics for the prediction of live birth after controlled ovarian stimulation. Retrospective development and temporal external validation of prediction model. Outpatient IVF clinic. We applied the boosted tree method to develop three prediction models incorporating clinical characteristics plus AMH or AFC or the combination on 2,124 linked IVF cycles from 2006 to 2010 and temporally externally validated predicted live-birth probabilities with an independent data set comprising 1,121 cycles from 2011 to 2012. None. Predictive power (posterior log of odds ratio compared to age, or PLORA), reclassification, receiver operator characteristic analysis, calibration, dynamic range. Predictive power, was highest for the AMH model (PLORA = 29.1), followed by the AMH-AFC model (PLORA = 28.3) and AFC model (PLORA = 22.5). The prediction errors were 1% to <5% in each prognostic tier for all three models, except for the predicted live-birth probabilities of <10% in the AFC model, where the prediction error was 8%. The improvement in predictive power was highest for the AMH model: 76.2% improvement over age alone relative to 59% improvement for AFC and 73.3% for the combined model. Receiver operating characteristic analysis demonstrated that the AMH and the combined model had comparable discrimination (area under the curve = 0.716) and similar prediction error for high and low strata of live-birth prediction, with an improvement of 6.3% over age alone. The validated prediction model confirmed that AMH when combined with clinical characteristics can accurately identify the likelihood of live birth with a low prediction error. AFC provided no added predictive value beyond AMH.

A gradient boosting method to improve travel time prediction

Tree based ensemble methods have reached a celebrity status in prediction field. By combining simple regression trees with ‘poor’ performance, they usually produce high prediction accuracy. In contrast to other machine learning methods that have been treated as black-boxes, tree based ensemble methods provide interpretable results, while requiring little data preprocessing, are able to handle different types of predictor variables, and can fit complex nonlinear relationship. These properties make the tree based ensemble methods good candidates for solving travel time prediction problems. However, applications of tree-based ensemble algorithms in traffic prediction area are limited. In this paper, we employ a gradient boosting regression tree method (GBM) to analyze and model freeway travel time to improve the prediction accuracy and model interpretability. The gradient boosting tree method strategically combines additional trees by correcting mistakes made by its previous base models, therefore, potentially improves prediction accuracy. Different parameters’ effect on model performance and correlations of input–output variables are discussed in details by using travel time data provided by INRIX along two freeway sections in Maryland. The proposed method is, then, compared with another popular ensemble method and a bench mark model. Study results indicate that the GBM model has its considerable advantages in freeway travel time prediction.

How Macroecomic Factors Influence the Commodity Market in the Financialization Period: The Case of S &amp; P GSCI Commodity Index

In connection to the process of financialization of commodity markets which is caused by the sharp increase of money flowing into the commodity markets, the question of which factors affect commodity and commodity indices prices is discussed. In this article, the importance of chosen macroeconomic determinants to the price variability of one of the most important commodity indexes S &amp; P GSCI by using the Boosted Trees method is quantified. The results obtained in the research show that changes in the monthly values of macroeconomic determinants reflect and can, according to the model used, explain the volatility of the monthly average index S &amp; P GSCI Total Return to more than 75%. The most important macroeconomic determinants proved to be Nominal Effective Exchange Rate of USD or US – Short-term interest rates.

Personalized prediction of first-cycle in vitro fertilization success

To test whether the probability of having a live birth (LB) with the first IVF cycle (C1) can be predicted and personalized for patients in diverse environments. Retrospective validation of multicenter prediction model. Three university-affiliated outpatient IVF clinics located in different countries. Using primary models aggregated from >13,000 C1s, we applied the boosted tree method to train a preIVF-diversity model (PreIVF-D) with 1,061 C1s from 2008 to 2009, and validated predicted LB probabilities with an independent dataset comprising 1,058 C1s from 2008 to2009. None. Predictive power, reclassification, receiver operator characteristic analysis, calibration, dynamic range. Overall, with PreIVF-D, 86% of cases had significantly different LB probabilities compared with age control, and more than one-half had higher LB probabilities. Specifically, 42% of patients could have been identified by PreIVF-D to have a personalized predicted success rate >45%, whereas an age-control model could not differentiate them from others. Furthermore, PreIVF-D showed improved predictive power, with 36% improved log-likelihood (or 9.0-fold by log-scale; >1,000-fold linear scale), and prediction errors for subgroups ranged from 0.9% to 3.7%. Validated prediction of personalized LB probabilities from diverse multiple sources identify excellent prognoses in more than one-half of patients.

Predicting personalized multiple birth risks after in vitro fertilization–double embryo transfer

To report and evaluate the performance and utility of an approach to predicting IVF-double embryo transfer (DET) multiple birth risks that is evidence-based, clinic-specific, and considers each patient's clinical profile. Retrospective prediction modeling. An outpatient university-affiliated IVF clinic. We used boosted tree methods to analyze 2,413 independent IVF-DET treatment cycles that resulted in live births. The IVF cycles were retrieved from a database that comprised more than 33,000 IVF cycles. None. The performance of this prediction model, MBP-BIVF, was validated by an independent data set, to evaluate predictive power, discrimination, dynamic range, and reclassification. Multiple birth probabilities ranging from 11.8% to 54.8% were predicted by the model and were significantly different from control predictions in more than half of the patients. The prediction model showed an improvement of 146% in predictive power and 16.0% in discrimination over control. The population standard error was 1.8%. We showed that IVF patients have inherently different risks of multiple birth, even when DET is specified, and this risk can be predicted before ET. The use of clinic-specific prediction models provides an evidence-based and personalized method to counsel patients.

Ranking function adaptation with boosting trees

Machine-learned ranking functions have shown successes in Web search engines. With the increasing demands on developing effective ranking functions for different search domains, we have seen a big bottleneck, that is, the problem of insufficient labeled training data, which has significantly slowed the development and deployment of machine-learned ranking functions for different domains. There are two possible approaches to address this problem: (1) combining labeled training data from similar domains with the small target-domain labeled data for training or (2) using pairwise preference data extracted from user clickthrough log for the target domain for training. In this article, we propose a new approach called tree-based ranking function adaptation (Trada) to effectively utilize these data sources for training cross-domain ranking functions. Tree adaptation assumes that ranking functions are trained with the Stochastic Gradient Boosting Trees method—a gradient boosting method on regression trees. It takes such a ranking function from one domain and tunes its tree-based structure with a small amount of training data from the target domain. The unique features include (1) automatic identification of the part of the model that needs adjustment for the new domain and (2) appropriate weighing of training examples considering both local and global distributions. Based on a novel pairwise loss function that we developed for pairwise learning, the basic tree adaptation algorithm is also extended (Pairwise Trada) to utilize the pairwise preference data from the target domain to further improve the effectiveness of adaptation. Experiments are performed on real datasets to show that tree adaptation can provide better-quality ranking functions for a new domain than other methods.

Statistical downscaling of river flows

summary An extensive statistical ‘downscaling’ study is done to relate large-scale climate information from a general circulation model (GCM) to local-scale river flows in SW France for 51 gauging stations ranging from nival (snow-dominated) to pluvial (rainfall-dominated) river-systems. This study helps to select the appropriate statistical method at a given spatial and temporal scale to downscale hydrology for future climate change impact assessment of hydrological resources. The four proposed statistical downscaling models use large-scale predictors (derived from climate model outputs or reanalysis data) that characterize precipitation and evaporation processes in the hydrological cycle to estimate summary flow statistics. The four statistical models used are generalized linear (GLM) and additive (GAM) models, aggregated boosted trees (ABT) and multi-layer perceptron neural networks (ANN). These four models were each applied at two different spatial scales, namely at that of a single flow-gauging station (local downscaling) and that of a group of flow-gauging stations having the same hydrological behaviour (regional downscaling). For each statistical model and each spatial resolution, three temporal resolutions were considered, namely the daily mean flows, the summary statistics of fortnightly flows and a daily ‘integrated approach’. The results show that flow sensitivity to atmospheric factors is significantly different between nival and pluvial hydrological systems which are mainly influenced, respectively, by shortwave solar radiations and atmospheric temperature. The non-linear models (i.e. GAM, ABT and ANN) performed better than the linear GLM when simulating fortnightly flow percentiles. The aggregated boosted trees method showed higher and less variable R 2 values to downscale the hydrological variability in both nival and pluvial regimes. Based on GCM cnrm-cm3 and scenarios A2 and A1B, future relative changes of fortnightly median flows were projected based on the regional downscaling approach. The results suggest a global decrease of flow in both pluvial and nival regimes, especially in spring, summer and autumn, whatever the considered scenario. The discussion considers the performance of each statistical method for downscaling flow at different spatial and temporal scales as well as the relationship between atmospheric processes and flow variability.