Tree-based Models Research Articles

Uncovering phishing attacks using principles of persuasion analysis

With the rising of Internet in early ’90s, many fraudulent activities have migrated from physical to digital: one of them is phishing. Phishing is a deceptive practice focused on exploiting the human factor, which is the most vulnerable aspect of any security process. In this scam, social engineering techniques are extensively utilized, specifically focusing on the principles of persuasion, to deceive individuals into disclosing sensitive information or engaging in malicious actions. This research explores the use of message subjectivity for detecting phishing attacks. It does so by assessing the impact of various data representations and classifiers on automatically identifying principles of persuasion. Furthermore, it investigates how these detected principles of persuasion can be leveraged for identifying phishing attacks. The experiments conducted revealed that there is no universal solution for data representation and classifier selection to effectively detect all principles of persuasion. Instead, a tailored combination of data representation and classifiers is required for detecting each principle. The Machine Learning models created automatically detect principles of persuasion with confidence levels ranging from 0.7306 to 0.8191 for AUC-ROC. Next, principles of persuasion detected are used for phishing detection. This study also emphasizes the need for user-friendly and comprehensible models. To validate the proposal presented, several families of classifiers were tested, but among all of them, tree-based models (and Random Forest in particular) stand out as preferred option. These models achieve similar level of effectiveness as alternative methods while offering improved clarity and user-friendliness, with an AUC-ROC of 0.859842.

A Comparison of Variable Input Strategies used for Risk-adjustment Models of Antimicrobial Use

Background: External comparisons of antimicrobial use (AU) may be more informative if adjusted for encounter characteristics. Optimal methods to define input variables for encounter-level risk-adjustment models of AU are not established. Methods: This retrospective analysis of electronic health record data included 50 US hospitals in 2020-2021. We used NHSN definitions for all antibacterials days of therapy (DOT), including adult and pediatric encounters with at least 1 day present in inpatient locations. We assessed 4 methods to define input variables: 1) diagnosis-related group (DRG) categories by Yu et al., 2) adjudicated Elixhauser comorbidity categories by Goodman et al., 3) all Clinical Classification Software Refined (CCSR) diagnosis and procedure categories, and 4) adjudicated CCSR categories where codes not appropriate for AU risk-adjustment were excluded by expert consensus, requiring review of 867 codes over 4 months to attain consensus. Data were split randomly, stratified by bed size as follows: 1) training dataset including two-thirds of encounters among two-thirds of hospitals; 2) internal testing set including one-third of encounters within training hospitals, and 3) external testing set including the remaining one-third of hospitals. We used a gradient-boosted machine (GBM) tree-based model and two-staged approach to first identify encounters with zero DOT, then estimate DOT among those with >0.5 probability of receiving antibiotics. Accuracy was assessed using mean absolute error (MAE) in testing datasets. Correlation plots compared model estimates and observed DOT among testing datasets. The top 20 most influential variables were defined using modeled variable importance. Results: Our datasets included 629,445 training, 314,971 internal testing, and 419,109 external testing encounters. Demographic data included 41% male, 59% non-Hispanic White, 25% non-Hispanic Black, 9% Hispanic, and 5% pediatric encounters. DRG was missing in 29% of encounters. MAE was lower in pediatrics as compared to adults, and lowest for models incorporating CCSR inputs (Figure 1). Performance in internal and external testing was similar, though Goodman/Elixhauser variable strategies were less accurate in external testing and underestimated long DOT outliers (Figure 2). Agnostic and adjudicated CCSR model estimates were highly correlated; their influential variables lists were similar (Figure 3). Conclusion: Larger numbers of CCSR diagnosis and procedure inputs improved risk-adjustment model accuracy compared with prior strategies. Variable importance and accuracy were similar for agnostic and adjudicated approaches. However, maintaining adjudications by experts would require significant time and potentially introduce personal bias. If findings are confirmed, the need for expert adjudication of input variables should be reconsidered.Disclosure: Elizabeth Dodds Ashley: Advisor- HealthTrackRx. David J Weber: Consultant on vaccines: Pfizer; DSMB chair: GSK; Consultant on disinfection: BD, GAMA, PDI, Germitec

Gradient Boosting Decision Tree-Based PMM Model Integrated Into FDTD Method for Solving Subsurface Sensing Problems

Gradient Boosting Decision Tree-Based PMM Model Integrated Into FDTD Method for Solving Subsurface Sensing Problems

A Tree-Based World Model for Reducing System Complexity in Autonomous Mobile Manipulation

A Tree-Based World Model for Reducing System Complexity in Autonomous Mobile Manipulation

Efficacy of Tree-Based Models for Pipe Failure Prediction and Condition Assessment: A Comprehensive Review

Efficacy of Tree-Based Models for Pipe Failure Prediction and Condition Assessment: A Comprehensive Review

Probabilistic and explainable tree-based models for rotational reactionary flight delay prediction

Probabilistic and explainable tree-based models for rotational reactionary flight delay prediction

A new integrated framework to fault detection and diagnosis of air handling unit: Emphasizing the impact of symptoms

This study proposes a comprehensive framework for Fault Detection and Diagnosis (FDD) in Air Handling Units (AHU), emphasizing the impact of symptoms associated with various faults. The aim is to address an important limitation in FDD model development where detection is based simply on fault intensity without considering symptom impact or subjective severity criteria across faults. Instead, a new approach is introduced that factors in fault impact, utilizing impact analysis to enhance FDD accuracy and efficiency. The methodology involves three interconnected components: First, a fault impact analysis categorizes 18 fault types into stages per intensity and assesses resulting symptom impacts. Second, symptom-based intensity thresholds are established to categorize fault severity into three levels based on symptom severity. Finally, the goal is to integrate these findings into the FDD process for optimized fault detection and diagnosis. The study successfully categorized 18 faults into three severity levels, with thresholds identified for each. Furthermore, Tree-based models demonstrated effective performance when conducting FDD based on these levels. The approach provides a clear framework through three linked processes. This research combines comprehensive impact analysis with sophisticated classification methods, presenting a more defined, systematic, and efficient FDD approach. This novel methodology substantially advances fault detection and diagnosis, especially for building energy systems.

Molecular delimitation of cryptic Australian squid species of the genus Uroteuthis Rehder, 1945 (Cephalopoda: Loliginidae), provides a baseline of diversity to resolve classification challenges throughout the Indo-Pacific

ABSTRACT This study provides a comprehensive molecular phylogenetic analysis of Uroteuthis squid from the Indo-Pacific region. The main aim was to increase sample coverage from northern and eastern Australian waters to resolve the identity and distribution of Uroteuthis species taken by local fisheries. Two mitochondrial regions, cytochrome c oxidase subunit I (COI) and 16S ribosomal RNA (16S rRNA), were sequenced from 220 new specimens and analysed with a further 51 sequences from GenBank to create a combined phylogeny for the genus. Three nuclear regions, 18S ribosomal DNA (18S rDNA), 28S ribosomal DNA (28S rDNA) and rhodopsin, were also sequenced from representatives of each species. Based on the mitochondrial phylogeny plus distance and tree-based delimitation models, a COI species barcode gap of 4–5% is proposed for discriminating Uroteuthis species. Applying this gap partitioned many described species into species complexes; for example, U. duvaucelii, U. noctiluca and U. edulis resolved into 10 species. Although more conserved, mitochondrial 16S rRNA sequences differentiated all new species clades, whereas none of the nuclear markers resolved the closest species. Results confirm that neither U. chinensis nor U. edulis occurs in Australian waters. Five undescribed species are identified from northern and eastern Australia, of which four are consistent with earlier allozyme studies (and two align with existing DNA sequences). One is a new southeastern, deeper shelf species differentiated in this study, along with a sixth undescribed species from Indonesian waters. Results of the molecular analysis are now being used to inform complementary morphometric analyses for new species descriptions, and genetic stock structure assessments of these important fisheries resources.

Data analysis and machine learning aided integrated catalyst activity and process modelling for selective H2 production from biomass gasification

Hydrogen energy derived through biomass gasification is considered as one of the most sorted sustainable sources of renewable energy. This process enhances the H2 production from biomass in the presence of specific catalysts. Among different kinds of models that have been employed for this process, ML models adept at approximating non-linear functions and facilitate outcome prediction without detailed mathematical descriptions. Thus, the current work focuses on understanding structural-composition-operating-target property relationships, and integrated catalyst and process modelling using ML framework for thermo-catalytic biomass gasification to H2 production, and demonstrates outliers handling, data normalization for efficient handling of data-driven modelling with non-linear database. Linear, tree-based, kernel-based, and ANN models were developed with 589 datapoints screened from the 59 relevant papers with 24 inputs and 4 outputs (H2, CO, CO2, and CH4 as vol. %). Performance of these models are evaluated through 5-fold cross-validation and test data with the help of statistical measures. ANN with Basian-regularization learning algorithm using tan-sigmoid activation function in both layers, resulted superior performance in prediction of H2 production (RMSE = 6.85 & R2 = 0.80) and other output gases with high accuracy (i.e., minimum deviation from experimental data) compared to other ML models. Further, using the best ML model, input contribution and PDP analysis were performed to interpret the significance of predominate input parameters affecting on the product composition. Feature contribution analysis reveals that temperature, S/B ratio, catalyst support type, and sulphur content in biomass are significant parameters for enhancing H2 production from catalytic-biomass gasification, and PDP analysis discloses their optimal operating region.

Machine learning analysis of thermophysical and thermohydraulic properties in ethylene glycol- and glycerol-based SiO2 nanofluids

The study investigates the heat transfer and friction factor properties of ethylene glycol and glycerol-based silicon dioxide nanofluids flowing in a circular tube under continuous heat flux circumstances. This study tackles the important requirement for effective thermal management in areas such as electronics cooling, the automobile industry, and renewable energy systems. Previous research has encountered difficulties in enhancing thermal performance while handling the increased friction factor associated with nanofluids. This study conducted experiments in the Reynolds number range of 1300 to 21,000 with particle volume concentrations of up to 1.0%. Nanofluids exhibited superior heat transfer coefficients and friction factor values than the base liquid values. The highest enhancement in heat transfer was 5.4% and 8.3% for glycerol and ethylene glycol -based silicon dioxide Nanofluid with a relative friction factor penalty of ∼30% and 75%, respectively. To model and predict the complicated, nonlinear experimental data, five machine learning approaches were used: linear regression, random forest, extreme gradient boosting, adaptive boosting, and decision tree. Among them, the decision tree-based model performed well with few errors, while the random forest and extreme gradient boosting models were also highly accurate. The findings indicate that these advanced machine learning models can accurately anticipate the thermal performance of nanofluids, providing a dependable tool for improving their use in a variety of thermal systems. This study's findings help to design more effective cooling solutions and improve the sustainability of energy systems.

Temporal heterogeneity in the performance of machine learning models for PM2.5 concentration estimation

Machine learning (ML) methods have been applied extensively to simulate air pollutant concentrations and assess individual exposure in epidemiological studies. However, there is still a paucity of research on the temporal heterogeneity of ML model performance and the impact of dataset size. To explore the temporal heterogeneity in model performance when estimating daily concentrations of fine particulate matter (PM2.5) across China in 2021, we compared five decision tree-based ML models (Random Forest (RF), Categorical Boosting (CatBoost), Gradient Boost Regression Tree (GBRT), eXtreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LightGBM)) across daily scales within three distinct timeframes. The performance of all models was evaluated using cross-validation. We observed that the performance of ML models varied with time, which showed a significant correlation with PM2.5 concentration. Among the 365 days in 2021, RF model performed best, the annual mean R2 was 0.86, with a minimum of 0.84 and a maximum of up to 0.95. For RF, we chose a cubic polynomial curve to fit the relationship between model performance and PM2.5 concentrations, and based on this, we devised a model selection strategy for different time scales, achieving an accuracy rate of up to 79.45 %, with the selected models having an average R2 of 0.85, and a maximum of up to 0.95. Additionally, we found that increasing the dataset size did not significantly improve model performance. Instead, it resulted in considerably longer runtime and increased memory usage. The methodology and findings of this study hold significant value for advancing the development of more efficient and precise modeling approaches for air pollutant concentrations. Furthermore, this research provides a foundation for regional air pollutant governance and future health-related research.

A data quality management framework for equipment failure risk estimation: Application to the oil and gas industry

In engineering asset management, accurate failure risk estimation is essential for averting equipment breakdowns and optimizing risk-based maintenance strategies. Data quality and model fitting are two critical sources of prediction uncertainty in risk estimation. While much attention has been devoted to model fitting for risk estimation, the critical role of data quality has often been overshadowed. To bridge this research gap, this paper presents a novel data quality management framework tailored for industrial equipment failure risk estimation. The framework covers the steps from data to model. It consists of the following main phases: data development, data quality assessment, data quality requirement decision-making, data quality improvement, and risk estimation model development. The framework provides detailed guidelines that can facilitate data practitioners to build individualized data quality requirement decision-making models for failure risk estimation of their equipment. The decision-making model can measure the adequacy of existing data to build a risk estimation model that meets the specified requirements and further determines the best risk estimation model given the available data. A case study using actual data collected during oil well drilling operations from multiple oil fields demonstrates the practicality and effectiveness of the framework. In this case study, four risk estimation models are compared, including two baseline models (mean time to failure and median time to failure) and two machine-learning models (quantile regression and hidden Markov model). In addition, a decision tree-based decision model is developed to determine whether the data quality meets the requirements and the best risk estimation model in case the data quality meets the requirements.

A tree-based explainable AI model for early detection of Covid-19 using physiological data

With the outbreak of COVID-19 in 2020, countries worldwide faced significant concerns and challenges. Various studies have emerged utilizing Artificial Intelligence (AI) and Data Science techniques for disease detection. Although COVID-19 cases have declined, there are still cases and deaths around the world. Therefore, early detection of COVID-19 before the onset of symptoms has become crucial in reducing its extensive impact. Fortunately, wearable devices such as smartwatches have proven to be valuable sources of physiological data, including Heart Rate (HR) and sleep quality, enabling the detection of inflammatory diseases. In this study, we utilize an already-existing dataset that includes individual step counts and heart rate data to predict the probability of COVID-19 infection before the onset of symptoms. We train three main model architectures: the Gradient Boosting classifier (GB), CatBoost trees, and TabNet classifier to analyze the physiological data and compare their respective performances. We also add an interpretability layer to our best-performing model, which clarifies prediction results and allows a detailed assessment of effectiveness. Moreover, we created a private dataset by gathering physiological data from Fitbit devices to guarantee reliability and avoid bias.The identical set of models was then applied to this private dataset using the same pre-trained models, and the results were documented. Using the CatBoost tree-based method, our best-performing model outperformed previous studies with an accuracy rate of 85% on the publicly available dataset. Furthermore, this identical pre-trained CatBoost model produced an accuracy of 81% when applied to the private dataset. You will find the source code in the link: https://github.com/OpenUAE-LAB/Covid-19-detection-using-Wearable-data.git.

PnT: Born-again tree-based model via fused decision path encoding

Decision forests, such as random forest (RF) are widely used for tabular data, mainly due to their predictive performance and ease of usage. However, given that the forest’s trees may produce contradictory predictions for a certain sample, the usage of decision forests in applications that involve decision-making necessitates further reliability assessment of the predictions for generating a trustworthy combined prediction. Model distillation using a born-again tree is a common approach for converting a decision forest into a single decision tree (DT) while preserving the forest’s predictive performance and supporting decision-makers. In this paper, we introduce PnT (Path in Tree), a novel approach that learns a path-based encoding from a decision forest. PnT applies an iterative algorithm, where in each iteration, a batch of trees is trained and then used to identify informative paths. These paths are then encoded and utilized for PnT-DT, an approach for producing a contradiction-free born-again DT. We also show that PnT can be leveraged for PnT-RF, a born-again forest approach, capable of improving the predictive performance of a plain decision forest. We evaluate PnT-DT and PnT-RF on 40 classification datasets and demonstrate that both PnT-DT and PnT-RF significantly outperform existing state-of-the-art (SOTA) born-again DT and decision forest methods in terms of predictive performance.

Compact machine learning model for the accurate prediction of first 24-hour survival of mechanically ventilated patients.

The field of machine learning has been evolving and applied in medical applications. We utilised a public dataset, MIMIC-III, to develop compact models that can accurately predict the outcome of mechanically ventilated patients in the first 24 h of first-time hospital admission. 67 predictive features, grouped into 6 categories, were selected for the classification and prediction task. 4 tree-based algorithms (Decision Tree, Bagging, eXtreme Gradient Boosting and Random Forest), and 5 non-tree-based algorithms (Logistic Regression, K-Nearest Neighbour, Linear Discriminant Analysis, Support Vector Machine and Naïve Bayes), were employed to predict the outcome of 18,883 mechanically ventilated patients. 5 scenarios were crafted to mirror the target population as per existing literature. S1.1 reflected an imbalanced situation, with significantly fewer mortality cases than survival ones, and both the training and test sets played similar target class distributions. S1.2 and S2.2 featured balanced classes; however, instances from the majority class were removed from the test set and/or the training set. S1.3 and S 2.3 generated additional instances of the minority class via the Synthetic Minority Over-sampling Technique. Standard evaluation metrics were used to determine the best-performing models for each scenario. With the best performers, Autofeat, an automated feature engineering library, was used to eliminate less important features per scenario. Tree-based models generally outperformed the non-tree-based ones. Moreover, XGB consistently yielded the highest AUC score (between 0.91 and 0.97), while exhibiting relatively high Sensitivity (between 0.58 and 0.88) on 4 scenarios (1.2, 2.2, 1.3, and 2.3). After reducing a significant number of predictors, the selected calibrated ML models were still able to achieve similar AUC and MCC scores across those scenarios. The calibration curves of the XGB and BG models, both prior to and post dimension reduction in Scenario 2.2, showed better alignment to the perfect calibration line than curves produced from other algorithms. This study demonstrated that dimension-reduced models can perform well and are able to retain the important features for the classification tasks. Deploying a compact machine learning model into production helps reduce costs in terms of computational resources and monitoring changes in input data over time.

Deriving PM2.5 from satellite observations with spatiotemporally weighted tree-based algorithms: enhancing modeling accuracy and interpretability

Tree-based machine learning algorithms, such as random forest, have emerged as effective tools for estimating fine particulate matter (PM2.5) from satellite observations. However, they typically have unchanged model structures and configurations over time and space, and thus may not fully capture the spatiotemporal variations in the relationship between PM2.5 and predictors, resulting in limited accuracy. Here, we propose geographically and temporally weighted tree-based models (GTW-Tree) for remote sensing of surface PM2.5. Unlike traditional tree-based models, GTW-Tree models vary by time and space to simulate the variability in PM2.5 estimation, and they can output variable importance for every location for the deeper understanding of PM2.5 determinants. Experiments in China demonstrate that GTW-Tree models significantly outperform the conventional tree-based models with predictive error reduced by >21%. The GTW-Tree-derived time-location-specific variable importance reveals spatiotemporally varying impacts of predictors on PM2.5. Aerosol optical depth (AOD) contributes largely to PM2.5 estimation, particularly in central China. The proposed models are valuable for spatiotemporal modeling and interpretation of PM2.5 and other various fields of environmental remote sensing.

SpChar: Characterizing the sparse puzzle via decision trees

Sparse matrix computation is crucial in various modern applications, including large-scale graph analytics, deep learning, and recommender systems. The performance of sparse kernels varies greatly depending on the structure of the input matrix, making it difficult to gain a comprehensive understanding of sparse computation and its relationship to inputs, algorithms, and target machine architecture. Despite extensive research on certain sparse kernels, such as Sparse Matrix-Vector Multiplication (SpMV), the overall family of sparse algorithms has yet to be investigated as a whole. This paper introduces SpChar, a workload characterization methodology for general sparse computation. SpChar employs tree-based models to identify the most relevant hardware and input characteristics, starting from hardware and input-related metrics gathered from Performance Monitoring Counters (PMCs) and matrices. Our analysis enables the creation of a characterization loop that facilitates the optimization of sparse computation by mapping the impact of architectural features to inputs and algorithmic choices. We apply SpChar to more than 600 matrices from the SuiteSparse Matrix collection and three state-of-the-art Arm Central Processing Units (CPUs) to determine the critical hardware and software characteristics that affect sparse computation. In our analysis, we determine that the biggest limiting factors for high-performance sparse computation are (1) the latency of the memory system, (2) the pipeline flush overhead resulting from branch misprediction, and (3) the poor reuse of cached elements. Additionally, we propose software and hardware optimizations that designers can implement to create a platform suitable for sparse computation. We then investigate these optimizations using the gem5 simulator to achieve a significant speedup of up to 2.63× compared to a CPU where the optimizations are not applied.

Predicting the PCM-incorporated building's performance using optimized linear kernel and tree-based machine learning methods

The devising of a precise machine learning-based energy consumption prediction model holds significant importance for taking effective decisions during the early stages of phase change material (PCM) incorporated building design to reduce energy consumption. Few researchers have developed prediction models for estimating the consumption of energy in PCM incorporated buildings. However, there is a need to develop the best-performing machine learning-based prediction model, which is less complex, involve a lower number of hyperparameters, and has a greater ability to generalize hidden patterns among the dependent and independent variables. To address the above shortcomings, the linear kernel and tree model-based prediction models were formulated for PCM-incorporated buildings located in eight cities of subtropical highland climate, considering the variations in their hyperparameters. The evaluation and validation of the formulated models for prediction were performed utilizing several statistical metrics. Test results substantiated that the ensemble-boosted tree-based prediction model (EBT20) exhibited superior efficacy in estimating the energy consumption for PCM-integrated building, having an average R2 > 0.93, NMBE <4 %, and CV-RMSE <8 %. The developed model showed less complexity and better generalization ability at the optimized values of ensemble boosted tree hyperparameters, which are as follows: minimum leaf size = 32, number of learners = 99, and learning rate = 0.1. From comprehensive evaluation conducted to evaluate the interpretability of the prediction model, it was found that the building orientation and PCMs melting temperature are the most influential parameters for achieving energy savings of up to 33 % in the selected climate zone.

Light curve classification with DistClassiPy: A new distance-based classifier

The rise of synoptic sky surveys has ushered in an era of big data in time-domain astronomy, making data science and machine learning essential tools for studying celestial objects. While tree-based models (e.g. Random Forests) and deep learning models dominate the field, we explore the use of different distance metrics to aid in the classification of astrophysical objects. We developed DistClassiPy, a new distance metric based classifier. The direct use of distance metrics is unexplored in time-domain astronomy, but distance-based methods can help make classification more interpretable and decrease computational costs. In particular, we applied DistClassiPy to classify light curves of variable stars, comparing the distances between objects of different classes. Using 18 distance metrics on a catalog of 6,000 variable stars across 10 classes, we demonstrate classification and dimensionality reduction. Our classifier meets state-of-the-art performance but has lower computational requirements and improved interpretability. Additionally, DistClassiPy can be tailored to specific objects by identifying the most effective distance metric for that classification. To facilitate broader applications within and beyond astronomy, we have made DistClassiPy open-source and available at https://pypi.org/project/distclassipy/.

Assessing and analysing energy system balance: A decision tree approach

Due to the impacts of climate change, the global energy system must undergo a green transformation. However, this transition process may entail both positive and negative impacts. Therefore, governments must formulate balanced energy policies to establish a secure, accessible, affordable, and environmentally sustainable energy system. To assist decision-makers in evaluating the status of energy systems and formulating rational energy policies to address the complex and dynamic challenges, this study focuses on proposing a method to assess and analyze the equilibrium of a nation's energy system. To achieve these objectives, we propose the Energy Balance Index as a comprehensive method for assessing the equilibrium of a country's energy system. Subsequently, we employ a decision tree-based model to analyze the impact of different factors on the Energy Balance Index. The findings reveal that environmental sustainability holds the most significant influence, followed by energy equity, while energy security has a minor impact. Moreover, the relative influence of these factors varies over time, with energy equity exerting a greater influence prior to 2015, and environmental sustainability emerging as the foremost determinant since then. Through development status analysis of China, this study provides valuable references and recommendations for decision-makers in designing energy policies.