Tree-based Machine Learning Algorithms Research Articles

Pilot-Study to Explore Metabolic Signature of Type 2 Diabetes: A Pipeline of Tree-Based Machine Learning and Bioinformatics Techniques for Biomarkers Discovery.

This study aims to identify unique metabolomics biomarkers associated with Type 2 Diabetes (T2D) and develop an accurate diagnostics model using tree-based machine learning (ML) algorithms integrated with bioinformatics techniques. Univariate and multivariate analyses such as fold change, a receiver operating characteristic curve (ROC), and Partial Least-Squares Discriminant Analysis (PLS-DA) were used to identify biomarker metabolites that showed significant concentration in T2D patients. Three tree-based algorithms [eXtreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), and Adaptive Boosting (AdaBoost)] that demonstrated robustness in high-dimensional data analysis were used to create a diagnostic model for T2D. As a result of the biomarker discovery process validated with three different approaches, Pyruvate, D-Rhamnose, AMP, pipecolate, Tetradecenoic acid, Tetradecanoic acid, Dodecanediothioic acid, Prostaglandin E3/D3 (isobars), ADP and Hexadecenoic acid were determined as potential biomarkers for T2D. Our results showed that the XGBoost model [accuracy = 0.831, F1-score = 0.845, sensitivity = 0.882, specificity = 0.774, positive predictive value (PPV) = 0.811, negative-PV (NPV) = 0.857 and Area under the ROC curve (AUC) = 0.887] had the slight highest performance measures. ML integrated with bioinformatics techniques offers accurate and positive T2D candidate biomarker discovery. The XGBoost model can successfully distinguish T2D based on metabolites.

Vibration behavior prediction of submerged nanobeams with axially traveling supports: numerical, analytical, and machine learning approaches

This work surveyed the vibration behavior and stability analysis of an underwater moving flexible nanosize beam, implementing numerical and analytical methods as well as tree-based machine learning (ML) algorithms. Dynamic modeling is performed based on the nonlocal stress-strain gradient theory (NSSGT), incorporating variable environmental conditions, surface energy, and rotational inertia effects. The natural frequencies and instability threshold are computed numerically. The exact closed-form mathematical expression for the critical towing velocity of the nanosize beam is determined analytically. Also, decision tree regression and least-squares boosting tree (LSBT) regression algorithms are exploited to predict stability boundaries and the fundamental vibration frequency, and their efficiency is surveyed accordingly. Comparative studies and parametric investigations are conducted. The impressions of geometry, added mass coefficient, scale ratio parameter, surface layer characteristics, fluid mass ratio, humidity, temperature rise, and external magnetic field intensity on the nanosystem dynamics are examined and illuminated. It is detected that the results of the analytical method and ML-based approaches are consistent with those reported by the numerical technique and literature. The results asserted that the proposed ML algorithms have acceptable performance and excellent effectiveness in computational time and cost. It is comprehended that the dynamic features of submerged traveling nanobeams drastically depend on the rotational inertia effects and thickness-dependent scale effects. The stability of the immersed movable nanoscale beams is perceived to improve by increasing the scale ratio parameter and the added mass coefficient. From the perspective of the optimal design of engineering instrumentation tools, the outcomes of the current article can be applied as an inclusive benchmark.

Precipitation data merging via machine learning: Revisiting conceptual and technical aspects

The development of accurate precipitation products with wide spatio-temporal coverage is crucial for a wide range of applications. In this context, precipitation data merging (PDM) that entails the blending of satellite-based estimates with ground-based measurements holds a prominent position, while currently there is an increasing trend in the deployment of machine learning (ML) algorithms in such endeavors. In the light of recent advances in the field, this work discusses key aspects of the PDM problem associated with: a) the conceptual formulation of the problem, that is closely related to the training of ML models and their predictive capacity, b) the selection of products fused, that is associated with the latency of final product and operational applicability of the method, c) the efficiency of single-step and two-step merging approaches, with the former one treating the problem via only regression algorithms and the latter one via the combined use of classification and regression algorithms. By formulating PDM as a spatio-temporal prediction problem, we define and assess two different training strategies for the ML models, termed as full and per time step strategy, which entail the building of a single or several ML models, respectively. Furthermore, the performance of the full training strategy, which allows the development of predictions in both spatial and temporal dimensions, is assessed in the context of single-step and two-step merging. In each of the three scenarios, three popular ensemble tree-based ML algorithms, i.e., the random forest, gradient boosting and extreme gradient boosting algorithm, are employed resulting in nine merged products. To provide empirical evidence, we employ a datacube composed by ground-based daily precipitation observations, satellite-based and reanalysis estimates, as well as auxiliary covariates, from 1009 uniformly distributed cells (representative of a sampling area of 25 × 25 km), over four countries around the world (Australia, USA, India and Italy). The large-scale experiment indicates that: (i) full training strategy is a competitive alternative to the per time step strategy, since it enables the development of methods with improved accuracy, with respect to performance metrics and reproduction of statistics, but also with higher predictive capability and operational applicability, (ii) two-step merging enables a much better reproduction of precipitation occurrence characteristics, as reflected in the improvement of relevant categorical metrics, the reproduction of probability and autocorrelation coefficient, (iii) no significant difference was noticed in the performance of different ML algorithms.

Predicting maximum pitting corrosion depth in buried transmission pipelines: Insights from tree-based machine learning and identification of influential factors

Pitting corrosion is a primary type of external corrosion that poses a critical challenge in the oil and gas industry, potentially leading to severe environmental, health, and economic consequences if left unaddressed. Researchers have attempted to comprehend and forecast it through diverse methods, including machine learning and artificial intelligence. However, achieving a precise estimation of pitting growth while considering the influence of various environmental and operational factors remains a concern. This study aims to develop a model by employing tree-based machine learning algorithms, including Decision Tree, Random Forest, LightGBM, CatBoost, and XGBoost, to enhance the prediction accuracy of the maximum depth of pitting corrosion in buried transmission pipelines while also facilitating interpretability. The Random Forest model is found to be the best-performing model with a mean absolute error (MAE) of 0.588. Beyond prediction accuracy, the present analysis contributes practical insights for industry application. First, the model identifies pipes at high risk of severe pitting corrosion under challenging environmental conditions, enabling targeted preventive measures. Secondly, the model's interpretation, utilising game-theoretic Shapely values and permutation importance, informs strategic decisions on the burial of new pipes, ultimately extending their operational lifespan. These applications emphasise the potential of our findings to significantly improve the safety, reliability, and risk mitigation strategies in oil and gas pipeline operations.

Identifying key products to trigger new exports: an explainable machine learning approach

Tree-based machine learning algorithms provide the most precise assessment of the feasibility for a country to export a target product given its export basket. However, the high number of parameters involved prevents a straightforward interpretation of the results and, in turn, the explainability of policy indications. In this paper, we propose a procedure to statistically validate the importance of the products used in the feasibility assessment. In this way, we are able to identify which products, called explainers, significantly increase the probability to export a target product in the near future. The explainers naturally identify a low dimensional representation, the Feature Importance Product Space, that enhances the interpretability of the recommendations and provides out-of-sample forecasts of the export baskets of countries. Interestingly, we detect a positive correlation between the complexity of a product and the complexity of its explainers.

5-EPIFAT trial protocol: a multi-center, randomized, placebo-controlled trial of the efficacy of pharmacotherapy for fatigue using methylphenidate, bupropion, ginseng, and amantadine in advanced cancer patients on active treatment

BackgroundCancer-related fatigue (CRF) is still undertreated in most patients, as evidence for pharmacological treatments is limited and conflicting. Also, the efficacy of the pharmacological agents relative to each other is still unclear. Therefore, medications that may potentially contribute to improving CRF will be investigated in this head-to-head trial. Our main objective is to compare the efficacy of methylphenidate vs. bupropion vs. ginseng vs. amantadine vs. placebo in patients with advanced cancer.MethodsThe 5-EPIFAT study is a 5-arm, randomized, multi-blind, placebo-controlled, multicenter trial that will use a parallel-group design with an equal allocation ratio comparing the efficacy and safety of four medications (Methylphenidate vs. Bupropion vs. Ginseng vs. Amantadine) versus placebo for management of CRF. We will recruit 255 adult patients with advanced cancer who experience fatigue intensity ≥ 4 based on a 0–10 scale. The study period includes a 4-week intervention and a 4-week follow-up with repeated measurements over time. The primary outcome is the cancer-related fatigue level over time, which will be measured by the functional assessment of chronic illness therapy-fatigue (FACIT-F) scale. To evaluate safety, the secondary outcome is the symptomatic adverse events, which will be assessed using the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events in cancer clinical trials (PRO-CTCAE). Also, a subgroup analysis based on a decision tree-based machine learning algorithm will be employed for the clinical prediction of different agents in homogeneous subgroups.DiscussionThe findings of the 5-EPIFAT trial could be helpful to guide clinical decision-making, personalization treatment approach, design of future trials, as well as the development of CRF management guidelines.Trial registrationIRCT.ir IRCT20150302021307N6. Registered on 13 May 2023.

Data correlation matrix-based spam URL detection using machine learning algorithms

In recent years, the widespread availability of internet access has brought both advantages and disadvantages. Users now enjoy numerous benefits, including unlimited access to vast amounts of information and seamless communication with others. However, this accessibility also exposes users to various threats, including malicious software and deceptive practices, leading to victimization of many individuals. Common issues encountered include spam emails, fake websites, and phishing attempts. Given the essential nature of internet usage in contemporary society, the development of systems to protect users from such malicious activities has become imperative. Accordingly, this study utilized eight prominent machine learning algorithms to identify spam URLs using a large dataset. Since the dataset only contained URL information and spam classification, additional feature extractions such as URL length and the number of digits were necessary. The inclusion of such features enhances decision-making processes within the framework of machine learning, resulting in more efficient detection. As the effectiveness of feature extraction significantly impacts the results of the methods, the study initially conducted feature extraction and trained models based on the weight of features. This paper proposes a data correlated matrix approach for spam URL detection using machine learning algorithms. The distinctive aspect of this study lies in the feature extraction process applied to the dataset, aimed at discerning the most impactful features, and subsequently training models while considering the weighting of these features. The entire dataset was used without any reduction in data. Experimental findings indicate that tree-based machine learning algorithms yield superior results. Among all applied methods, the Random Forest approach achieved the highest success rate, with a detection rate of 96.33% for the non-spam class. Additionally, a combined and weighted calculation method yielded an accuracy of 94.16% for both spam and non-spam data.

Forecasting East and West Coast Gasoline Prices with Tree-Based Machine Learning Algorithms

This study aims to forecast New York and Los Angeles gasoline spot prices on a daily frequency. The dataset includes gasoline prices and a big set of 128 other relevant variables spanning the period from 17 February 2004 to 26 March 2022. These variables were fed to three tree-based machine learning algorithms: decision trees, random forest, and XGBoost. Furthermore, a variable importance measure (VIM) technique was applied to identify and rank the most important explanatory variables. The optimal model, a trained random forest, achieves a mean absolute percent error (MAPE) in the out-of-sample of 3.23% for the New York and 3.78% for the Los Angeles gasoline spot prices. The first lag, AR (1), of gasoline is the most important variable in both markets; the top five variables are all energy-related. This paper can strengthen the understanding of price determinants and has the potential to inform strategic decisions and policy directions within the energy sector, making it a valuable asset for both industry practitioners and policymakers.

Fourier transform infrared spectroscopy and machine learning for Porphyromonas gingivalis detection in oral bacteria.

Porphyromonas gingivalis, a Gram-negative anaerobic bacillus, is the primary pathogen in periodontitis. Herein, we cultivated strains of oral bacteria, including P. gingivalis and the oral commensal bacteria Actinomyces viscosus and Streptococcus mutans, and recorded the infrared absorption spectra of the gases released by the cultured bacteria at a resolution of 0.5cm-1 within the wavenumber range of 500-7500cm-1. From these spectra, we identified the infrared wavenumbers associated with characteristic absorptions in the gases released by P. gingivalis using a decision tree-based machine learning algorithm. Finally, we compared the obtained absorbance spectra of ammonia (NH3) and carbon monoxide (CO) using the HITRAN database. We observed peaks at similar positions in the P. gingivalis gases, NH3, and CO spectra. Our results suggest that P. gingivalis releases higher amounts of NH3 and CO than A. viscosus and S. mutans. Thus, combining Fourier transform infrared spectroscopy with machine learning enabled us to extract the specific wavenumber range that differentiates P. gingivalis from a vast dataset of peak intensity ratios. Our method distinguishes the gases from P. gingivalis from those of other oral bacteria and provides an effective strategy for identifying P. gingivalis in oral bacteria. Our proposed methodology could be valuable in clinical settings as a simple, noninvasive pathogen diagnosis technique.

Application of regression decision tree and machine learning algorithms to examine students’ online learning preferences during COVID-19 pandemic

The emergence of the novel coronavirus (COVID-19) profoundly disrupted the field of education, ushering in an era of widespread online learning adoption. This research paper seeks to investigate the multifaceted factors influencing students' preferences for online learning. Employing data exploration techniques and machine learning algorithms, the study aimed to identify the pivotal variables affecting students' willingness and performance in online educational environments. The research encompassed data collection through designated questionnaires and the application of decision tree-based machine learning algorithms to analyze these diverse factors. Through this approach, seven specific prerequisites were derived, employing multiple linear regression analysis within the decision tree framework, to illuminate the relationships between these factors. Key aspects considered in these prerequisites included factors such as "internet connectivity issues," "COVID-19 pandemic-induced stress," "COVID-19 vaccination status," and "close relatives' COVID-19 infections". Foremost among the reasons for students' reluctance to embrace online learning was the presence of "internet difficulties," including issues like slow connections and frequent disruptions. From the results of this research, it can be concluded that basic computer and internet courses can be beneficial for encouraging online education. Findings of this study underscore the potential benefits of offering basic computer and internet courses as a means to encourage and facilitate effective online education, particularly in the context of the COVID-19 pandemic.

A comparative analysis of tree-based machine learning algorithms for predicting the mechanical properties of fibre-reinforced GGBS geopolymer concrete

A comparative analysis of tree-based machine learning algorithms for predicting the mechanical properties of fibre-reinforced GGBS geopolymer concrete

Dataset meta-level and statistical features affect machine learning performance

What dataset features affect machine learning (ML) performance has primarily been unknown in the current literature. This study examines the impact of tabular datasets' different meta-level and statistical features on the performance of various ML algorithms. The three meta-level features this study considered are the dataset size, the number of attributes and the ratio between the positive (class 1) and negative (class 0) class instances. It considered four statistical features for each dataset: mean, standard deviation, skewness and kurtosis. After applying the required scaling, this study averaged (uniform and weighted) each dataset’s different attributes to quantify its four statistical features. We analysed 200 open-access tabular datasets from the Kaggle (147) and UCI Machine Learning Repository (53) and developed ML classification models (through classification implementation and hyperparameter tuning) for each dataset. Then, this study developed multiple regression models to explore the impact of dataset features on ML performance. We found that kurtosis has a statistically significant negative effect on the accuracy of the three non-tree-based ML algorithms of the Support vector machine (SVM), Logistic regression (LR) and K-nearest neighbour (KNN) for their classical implementation with both uniform and weighted aggregations. This study observed similar findings in most cases for ML implementations through hyperparameter tuning, except for SVM with weighted aggregation. Meta-level and statistical features barely show any statistically significant impact on the accuracy of the two tree-based ML algorithms (Decision tree and Random forest), except for implementation through hyperparameter tuning for the weighted aggregation. When we excluded some datasets based on the imbalanced statistics and a significantly higher contribution of one attribute compared to others to the classification performance, we found a significant effect of the meta-level ratio feature and statistical mean and standard deviation features on SVM, LR and KNN accuracy in many cases. Our findings open a new research direction in understanding how dataset characteristics affect ML performance and will help researchers select appropriate ML algorithms for a possible optimal accuracy outcome.

Assessing the predictive value of morphological traits on primary lifestyle of birds through the extreme gradient boosting algorithm.

The relationship between morphological traits and ecological performance in birds is an important area of research, as it can help us to understand how birds are able to adapt and how they are affected by changes in their environment. Many studies have investigated the relationship between morphological traits and certain aspects of the performance and ecological niche of these animals. However, the relationship between morphological traits and the primary lifestyle of birds has not previously been explored. This paper aims to evaluate the predictive potential of morphological data to determine the primary lifestyle of birds through a tree-based machine learning algorithm. By doing this, it is also possible to evaluate these artificial categories that we used to split up birds and know whether they are suitable for dividing them in function of shared morphological characteristics or need a redefinition under more discriminant criteria. Supplementary dataset 1 of the AVONET project was used, which comprises the 11 morphological predictors used in this work and the classification according to the primary lifestyle for more than 95% of the existing bird species. For all morphological traits used, statistically significant univariate differences were found between primary lifestyles. The three fitted machine learning models showed high accuracy, in all cases above 78% and superior to the ones achieved through traditional approaches used as contrasts. The results obtained provide evidence that primary lifestyle can be predicted in birds based on morphological traits, as well as more insights about the relevance of functional traits for ecological modeling. This is another step forward in our mechanistic understanding of bird ecology, while exploring how birds have adapted to their environments and how they interact with their surroundings.

Confirming the statistically significant superiority of tree-based machine learning algorithms over their counterparts for tabular data.

Many individual studies in the literature observed the superiority of tree-based machine learning (ML) algorithms. However, the current body of literature lacks statistical validation of this superiority. This study addresses this gap by employing five ML algorithms on 200 open-access datasets from a wide range of research contexts to statistically confirm the superiority of tree-based ML algorithms over their counterparts. Specifically, it examines two tree-based ML (Decision tree and Random forest) and three non-tree-based ML (Support vector machine, Logistic regression and k-nearest neighbour) algorithms. Results from paired-sample t-tests show that both tree-based ML algorithms reveal better performance than each non-tree-based ML algorithm for the four ML performance measures (accuracy, precision, recall and F1 score) considered in this study, each at p<0.001 significance level. This performance superiority is consistent across both the model development and test phases. This study also used paired-sample t-tests for the subsets of the research datasets from disease prediction (66) and university-ranking (50) research contexts for further validation. The observed superiority of the tree-based ML algorithms remains valid for these subsets. Tree-based ML algorithms significantly outperformed non-tree-based algorithms for these two research contexts for all four performance measures. We discuss the research implications of these findings in detail in this article.

Predicting hydrogen production from co-gasification of biomass and plastics using tree based machine learning algorithms

Hydrogen production from co-gasification of biomass and plastics are predicted using Machine Learning Algorithms, e.g., Decision tree and Ensemble methods. Independent variables are particle sizes of biomass and plastics, feedstock ratio and temperatures. The dependent variable is Hydrogen production. Model and prediction performances were evaluated/validated using model parameters. The relative importance scores for independent variables are RSS particle size > HDPE particle size > Temperature > Percent plastics. Size dependence of Hydrogen production indicated a surface-controlled reaction. Temperatures between 500 °C and 900 °C have less impact on H2 production compared to the size. Predictions were carried out using Train-test split, Cross-validation, and GridsearchCV model on the data unseen. Gradient Boosting performed the best.

Accurate prediction of pressure losses using machine learning for the pipeline transportation of emulsions

One of the significant challenges to designing an emulsion transportation system is predicting frictional pressure losses with confidence. The state-of-the-art method for enhancing reliability in prediction is to employ artificial intelligence (AI) based on various machine learning (ML) tools. Six traditional and tree-based ML algorithms were analyzed for the prediction in the current study. A rigorous feature importance study using RFECV method and relevant statistical analysis was conducted to identify the parameters that significantly contributed to the prediction. Among 16 input variables, the fluid velocity, mass flow rate, and pipe diameter were evaluated as the top predictors to estimate the frictional pressure losses. The significance of the contributing parameters was further validated by estimation error trend analyses. A comprehensive assessment of the regression models demonstrated an ensemble of the top three regressors to excel over all other ML and theoretical models. The ensemble regressor showcased exceptional performance, as evidenced by its high R2 value of 99.7 % and an AUC-ROC score of 98 %. These results were statistically significant, as there was a noticeable difference (within a 95 % confidence interval) compared to the estimations of the three base models. In terms of estimation error, the ensemble model outperformed the top base regressor by demonstrating improvements of 6.6 %, 11.1 %, and 12.75 % for the RMSE, MAE, and CV_MSE evaluation metrics, respectively. The precise and robust estimations achieved by the best regression model in this study further highlight the effectiveness of AI in the field of pipeline engineering.

MACHINE LEARNING MODEL FOR THE PREDICTION OF CONDITION OF MUSEUM OBJECTS

An accurate prediction of the future condition of museum objects is crucial for developing appropriate proactive maintenance and preservation strategies. Despite this, there are very few such damage models that can be used in practice. The main reasons, for this lack of deterioration models, include complexity of deterioration problem and lack of understanding of the degradation mechanisms affecting various materials and objects, and lack of reliable quantitative approaches. In the article, we discuss the machine learning model, which predicts the future condition of museum objects. For this purpose, the model uses the data of MuIS (Estonian Museum Information System). To predict deterioration, we experimented primarily with various tree-based machine learning algorithms, such as the decision tree, the random forest, and XGBoost. The best results were obtained using the decision forest algorithm, which was able to identify 92% of deteriorating museum objects with 50% accuracy. The machine learning model provides a way to model ageing processes of museum objects over the course of time and thus better plan the preservation work of museums.

Estimation of sugarcane evapotranspiration from remote sensing and limited meteorological variables using machine learning models

Rapid and accurate crop evapotranspiration (ETc) estimation is essential for water resource management of irrigated lands. Nevertheless, due to the significant uncertainties in the inputs of conventional ETc equations and the inaccessibility of their proper spatial and temporal distribution values, precision agriculture requires a more accurate evaluation of ETc using efficient methods. Combining remote sensing (RS) data and machine learning (ML) techniques has provided a considerable capacity for estimating ETc, which can address these challenges. This study aims to develop a framework for ETc estimation based on the four most popular tree-based ML algorithms, namely M5-pruned (M5P), random forest regression (RFR), gradient-boosted regression trees (GBRT), and extreme gradient boosting (XGBoost). For this purpose, multisource RS data and meteorological and ground measurements (Meteo-GM) were used during three growing seasons (2018–2021) in irrigated croplands of the Sugarcane & By-Products Development Company in Khuzestan, southwest Iran. Since the unavailability of optical sensors, e.g., Landsat-8 data, in cloudy conditions restricts their enduring application, an alternative approach employing the Sentinel-1 Synthetic Aperture Radar (SAR) data for ETc estimation was also proposed. A set of 23 scenarios of various unique combinations of the available parameters was developed to ensure that a proper decision could be made for ETc estimation under any conditions. Eventually, the contribution of the input variables to estimate ETc was evaluated using the Sobol sensitivity analysis. The results showed that the RFR algorithm provided the most accurate ETc estimation in all scenarios, followed by the XGBoost, GBRT, and M5P algorithms (R2 = 0.92–0.99, RMSE = 2.02–0.32 mm d-1). The Meteo-GM models’ performances were improved by incorporating optical and thermal infrared (TIR) (R2 = 0.99, RMSE = 0.32 mm d-1) and SAR (R2 = 0.98, RMSE = 0.65 mm d-1) RS data. In addition, Sobol’s sensitivity analysis revealed that maximum temperature, wind speed, pan evaporation, leaf area index, moisture stress index, and inverse dual-pol diagonal distance were the most influential input variables in the ETc estimation. This study can be used as a reference to simplify the ETc calculations using a novel application of ML as a helpful basis for precision irrigation management from the point to the regional scale in any weather conditions.

A comparative analysis of converters of tabular data into image for the classification of Arboviruses using Convolutional Neural Networks

Tabular data is commonly used in business and literature and can be analyzed using tree-based Machine Learning (ML) algorithms to extract meaningful information. Deep Learning (DL) excels in data such as image, sound, and text, but it is less frequently utilized with tabular data. However, it is possible to use tools to convert tabular data into images for use with Convolutional Neural Networks (CNNs) which are powerful DL models for image classification. The goal of this work is to compare the performance of converters for tabular data into images, select the best one, optimize a CNN using random search, and compare it with an optimized ML algorithm, the XGBoost. Results show that even a basic CNN, with only 1 convolutional layer, can reach comparable metrics to the XGBoost, which was trained on the original tabular data and optimized with grid search and feature selection. However, further optimization of the CNN with random search did not significantly improve its performance.

Predicting the Customer Behaviour Utilizing Tree Based Machine Learning Algorithms

Predicting the Customer Behaviour Utilizing Tree Based Machine Learning Algorithms