Optimal Decision Tree Research Articles

A Polyhedral Study of Multivariate Decision Trees

Decision trees are a widely used tool for interpretable machine learning. Multivariate decision trees employ hyperplanes at the branch nodes to route datapoints throughout the tree and yield more compact models than univariate trees. Recently, mixed-integer programming (MIP) has been applied to formulate the optimal decision tree problem. A key component of most MIP formulations is a specification of how to route datapoints in the tree from the root to the leaves. Our goal is to provide polyhedral characterizations of realizable routings, that is, routings that can be realized using multivariate branching rules. We first focus on shattering inequalities, a class of valid inequalities that can be used to strengthen almost any MIP formulation and that have been shown to be computationally effective. We prove that if all the feature vectors are in general position, then the shattering inequalities defined for the root of the tree are facet defining for the convex hull of the realizable routings. We then show that every facet-defining inequality of a depth one tree involving at least two variables is also facet defining for trees of arbitrary depth. Finally, we show that facet-defining inequalities characterizing realizable routings are also facet-defining for a complete MIP formulation. We validate our theoretical results by performing numerical experiments that specifically exploit shattering inequalities defined at the root node and we observe an improvement in performance for both numerical and categorical data sets, especially for decision trees of intermediate size. Funding: C. Michini gratefully acknowledges funding from the DoD, Air Force [Award FA9550-23-1-0487].

Advanced CEEMD hybrid model for VIX forecasting: optimized decision trees and ARIMA integration

Advanced CEEMD hybrid model for VIX forecasting: optimized decision trees and ARIMA integration

Simplex-lattice design and decision tree optimization of endophytic Trichoderma-multi-walled carbon nanotube composite for enhanced methylene blue removal

This study investigates a novel approach for enhancing methylene blue (MB) removal from water using a composite of endophytic Trichoderma mate and multi-walled carbon nanotubes (MWCNTs). For the first time, a unique combination of simplex-lattice design and decision tree learning algorithm was employed to optimize MB removal. This innovative approach effectively identified the optimal composite ratio of hyphal mate (0.5354 g/L) and MWCNTs (0.4646 g/L) for maximizing MB removal, which achieved remarkable removal efficiency ranging from 63.50 to 95.78 % depending on the combination used. The DT model further demonstrated promising potential for predicting MB removal efficiency. SEM revealed a unique hybrid material formed by the intertwining or entrapment of MWCNTs within the hyphal network of Trichoderma mate. FT-IR analysis confirmed the presence of novel functional groups on the MWCNTs' surface at 2438.79 and 528.25 cm−1, likely due to interactions with the endophytic fungi's biomolecules. These functional groups presumably act as reducing and stabilizing agents, promoting efficient MB adsorption. This research paves the way for utilizing the combined biological and chemical approach (fungal biomass and MWCNTs) in bioremediation applications. The findings suggest significant potential for practical applications in wastewater treatment, providing an eco-friendly and cost-effective method for dye removal. Furthermore, the proposed method shows promise for scaling up to industrial wastewater treatment and applicability in resource-limited settings, offering a sustainable solution for global water pollution challenges. Further investigations with larger datasets incorporating additional influencing factors are necessary to refine the predictive power of the DT model for practical applications.

Improved survival prediction for kidney transplant outcomes using artificial intelligence-based models: development of the UK Deceased Donor Kidney Transplant Outcome Prediction (UK-DTOP) Tool

The UK Deceased Donor Kidney Transplant Outcome Prediction (UK-DTOP) Tool, developed using advanced artificial intelligence (AI), significantly enhances the prediction of outcomes for deceased-donor kidney transplants in the UK. This study analyzed data from the UK Transplant Registry (UKTR), including 29,713 transplant cases between 2008 and 2022, to assess the predictive performance of three machine learning models: XGBoost, Random Survival Forest, and Optimal Decision Tree. Among these, XGBoost demonstrated exceptional performance with the highest concordance index of 0.74 and an area under the curve (AUC) consistently above 0.73, indicating robust discriminative ability and calibration. In comparison to the traditional Kidney Donor Risk Index (KDRI), which achieved a lower concordance index of 0.57, the UK-DTOP model marked a significant improvement, underscoring its superior predictive accuracy. The advanced capabilities of the XGBoost model were further highlighted through calibration assessments using the Integrated Brier Score (IBS), showing a score of 0.14, indicative of precise survival probability predictions. Additionally, unsupervised learning via k-means clustering was employed to identify five distinct clusters based on donor and transplant characteristics, uncovering nuanced insights into graft survival outcomes. These clusters were further analyzed using Bayesian Cox regression, which confirmed significant survival outcome variations across the clusters, thereby validating the model’s effectiveness in enhancing risk stratification. The UK-DTOP tool offers a comprehensive decision-support system that significantly refines pre-transplant decision-making. The study’s findings advocate for the adoption of AI-enhanced tools in healthcare systems to optimize organ matching and transplant success, potentially guiding future developments in global transplant practices.

Approximating decision trees with priority hypotheses

This paper addresses the problem of creating decision trees for identifying hypotheses, also known as entities, in a setting where the cost of an action is dependent on the true hypothesis. Specifically, we consider the scenario where n hypotheses are divided into m groups based on their priority levels. Taking an action on a higher priority hypothesis incurs a higher cost. This is relevant to many real-world applications where cost-sensitive decisions need to be made. For example, in a medical diagnosis task, the goal is to take a series of actions (such as medical tests) to identify a cause. Each action in this process requires conducting a test on the patient and observing the outcome, which can take anywhere from a few minutes to several weeks depending on the test. In this case, the cost (the result of waiting for the outcome) is higher if the true hypothesis is more time-sensitive. For example, if the true hypothesis is toxic chemical exposure (as opposed to a chronic disease such as diabetes), a delay of a few minutes could significantly increase the patient's risk of mortality. We propose a group greedy algorithm to solve this problem. We demonstrate that under worst-case scenarios, our algorithm has an approximation ratio of O(mlog⁡n). Importantly, when m=1, meaning there is only one group of hypotheses, our result is consistent with the logarithmic approximation bound for the traditional optimal decision tree problem.

Policy Learning for Many Outcomes of Interest: Combining Optimal Policy Trees with Multi-objective Bayesian Optimisation

AbstractMethods for learning optimal policies use causal machine learning models to create human-interpretable rules for making choices around the allocation of different policy interventions. However, in realistic policy-making contexts, decision-makers often care about trade-offs between outcomes, not just single-mindedly maximising utility for one outcome. This paper proposes an approach termed Multi-Objective Policy Learning (MOPoL) which combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach to explore the trade-off between multiple outcomes. It does this by building a Pareto frontier of non-dominated models for different hyperparameter settings which govern outcome weighting. The method is applied to a real-world case-study of pricing targetting subsididies for anti-malarial medication in Kenya.

Prediction of Dielectric Constant in Series of Polymers by Quantitative Structure-Property Relationship (QSPR)

This work is devoted to the investigation of dielectric permittivity which is influenced by electronic, ionic, and dipolar polarization mechanisms, contributing to the material’s capacity to store electrical energy. In this study, an extended dataset of 86 polymers was analyzed, and two quantitative structure–property relationship (QSPR) models were developed to predict dielectric permittivity. From an initial set of 1273 descriptors, the most relevant ones were selected using a genetic algorithm, and machine learning models were built using the Gradient Boosting Regressor (GBR). In contrast to Multiple Linear Regression (MLR)- and Partial Least Squares (PLS)-based models, the gradient boosting models excel in handling nonlinear relationships and multicollinearity, iteratively optimizing decision trees to improve accuracy without overfitting. The developed GBR models showed high R2 coefficients of 0.938 and 0.822, for the training and test sets, respectively. An Accumulated Local Effect (ALE) technique was applied to assess the relationship between the selected descriptors—eight for the GB_A model and six for the GB_B model, and their impact on target property. ALE analysis revealed that descriptors such as TDB09m had a strong positive effect on permittivity, while MLOGP2 showed a negative effect. These results highlight the effectiveness of the GBR approach in predicting the dielectric properties of polymers, offering improved accuracy and interpretability.

Biosorption of methylene blue by bone meal: experimental and modeling with machine learning and full factorial design

Sorption technologies have been proposed for the treatment of water containing methylene blue (MB), a toxic and persistent pollutant. Despite its environmental risks, the role of process variables in MB removal has not been fully explored through experimental design. The objective of this study is to assess the potential of bone meal powder (BMP), an underexplored agricultural byproduct, as an affordable adsorbent for the removal of MB from water. BMP was subjected to a series of analytical characterization techniques, and its adsorption capacity was evaluated through a comprehensive factorial design, which investigated the effects of biosorbent dosage, solution pH, and initial MB concentration. The study revealed that the highest adsorption level was 14.49 mg g−1, attained under the following conditions: 1 g L−1 BMP, pH 11, and 100 mg L−1 MB. The adsorption equilibrium was reached within 60 min, with a measured capacity (qexp) of 18 mg g−1. Theoretical adsorption isotherms indicated a capacity of 63 mg g−1, which aligned well with the Langmuir model. To predict adsorption outcomes, machine learning models were applied, with multiple linear regression performing best. Optimization of decision trees and neural networks improved accuracy but risked overfitting. FT-IR, XRD, and ICP analyses indicated ion exchange as a significant mechanism of adsorption. In desorption studies, H2SO4 was the most effective agent, achieving 68.72% desorption efficiency. BMP exhibited optimal recyclability for up to four cycles before efficiency declined.

TDBAMLA: Temporal and dynamic behavior analysis in Android malware using LSTM and attention mechanisms

The increasing ubiquity of Android devices has precipitated a concomitant surge in sophisticated malware attacks, posing critical challenges to cybersecurity infrastructures worldwide. Existing models have achieved significant strides in malware detection but often suffer from high false-positive rates, lower recall, and computational delays, thus demanding a more efficient and accurate system. Current techniques primarily rely on static features and simplistic learning models, leading to inadequate handling of temporal aspects and dynamic behaviors exhibited by advanced malware. These limitations compromise the detection of modern, evasive malware, and impede real-time analysis. This paper introduces a novel framework for Android malware detection that incorporates Temporal and Dynamic Behavior Analysis using Long Short-Term Memory (LSTM) networks and Attention Mechanisms. We further propose development of an efficient Grey Wolf Optimized (GWO) Decision Trees to find the most salient API call patterns associated with malwares. An Iterative Fuzzy Logic (IFL) layer is also deployed before classification to assess the "trustworthiness" of app metadata samples. For Ongoing Learning, we propose use of Deep Q-Networks (DQNs), which helps the reinforcement learning model to adapt more quickly to changes in the threat landscapes. By focusing on crucial system calls and behavioral characteristics in real-time, our model captures the nuanced temporal patterns often exhibited by advanced malwares. Empirical evaluations demonstrate remarkable improvements across multiple performance metrics. Compared to existing models, our approach enhances the precision of malware identification by 8.5 %, accuracy by 5.5 %, and recall by 4.9 %, while also achieving an 8.3 % improvement in the Area Under the Receiver Operating Characteristic Curve (AUC), with higher specificity and a 4.5 % reduction in identification delay. In malware pre-emption tasks, our model outperforms by improving precision by 4.3 %, accuracy by 3.9 %, recall by 4.9 %, AUC by 3.5 %, and increasing specificity by 2.9 %. These gains make our framework highly applicable for real-time detection systems, cloud-based security solutions, and threat intelligence services, thereby contributing to a safer Android ecosystem.

Research on Time Series Steel Leakage Prediction Model Based on TS-GA Optimization Decision Tree

Research on Time Series Steel Leakage Prediction Model Based on TS-GA Optimization Decision Tree

An automated OpenMP mutation testing framework for performance optimization

Performance optimization continues to be a challenge in modern HPC software. Existing performance optimization techniques, including profiling-based and auto-tuning techniques, fail to indicate program modifications at the source level thus preventing their portability across compilers. This paper describes Muppet, a new approach that identifies program modifications called mutations aimed at improving program performance. Muppet’s mutations help developers reason about performance defects and missed opportunities to improve performance at the source code level. In contrast to compiler techniques that optimize code at intermediate representations (IR), Muppet uses the idea of source-level mutation testing to relax correctness constraints and automatically discover optimization opportunities that otherwise are not feasible using the IR. We demonstrate the Muppet’s concept in the OpenMP programming model. Muppet generates a list of OpenMP mutations that alter the program parallelism in various ways, and is capable of running a variety of optimization algorithms such as delta debugging, Bayesian Optimization and decision tree optimization to find a subset of mutations which, when applied to the original program, cause the most speedup while maintaining program correctness. When Muppet is evaluated against a diverse set of benchmark programs and proxy applications, it is capable of finding sets of mutations that induce speedup in 75.9% of the evaluated programs.

Estimating optimal decision trees for treatment assignment: The case of K > 2 treatment alternatives.

For many problems in clinical practice, multiple treatment alternatives are available. Given data from a randomized controlled trial or an observational study, an important challenge is to estimate an optimal decision rule that specifies for each client the most effective treatment alternative, given his or her pattern of pretreatment characteristics. In the present paper we will look for such a rule within the insightful family of classification trees. Unfortunately, however, there is dearth of readily accessible software tools for optimal decision tree estimation in the case of more than two treatment alternatives. Moreover, this primary tree estimation problem is also cursed with two secondary problems: a structural missingness in typical studies on treatment evaluation (because every individual is assigned to a single treatment alternative only), and a major issue of replicability. In this paper we propose solutions for both the primary and the secondary problems at stake. We evaluate the proposed solution in a simulation study, and illustrate with an application on the search for an optimal tree-based treatment regime in a randomized controlled trial on K = 3 different types of aftercare for younger women with early-stage breast cancer. We conclude by arguing that the proposed solutions may have relevance for several other classification problems inside and outside the domain of optimal treatment assignment.

Diagnose Breast Cancer Using Two Machine Learning Methods andComparing and Analyzing Them

The breast cancer is currently the number one cancer in the world and has a high early treatment rate. Therefore, early screening and diagnosis of breast cancer is very important. Machine learning is very strong in analyzing and processing massive data. In this paper, we train and optimize decision tree and SVM models to identify malignant breast cancer and compare and analyze the performance of the two models. The accuracy of the earliest optimized models reaches about 98%, and it is found that the two models focus on different classification effects for the two samples, which can be further optimized by stacking and other ways. The dataset used to train the models in this paper is from the Wisconsin breast cancer database (WBCD), a benchmark dataset commonly used to compare different algorithms, which is of some significance for exploring the use of machine learning in the field of medical diagnosis.

Feature selection in peer-to-peer lending based on hybrid modified grey wolf optimization with optimized decision tree for credit risk assessment

ABSTRACT Lending platforms operating on a peer-to-peer (P2P) basis encounter the intricate challenge of assessing borrower creditworthiness to minimize the risk of defaults. This study addresses this challenge by proposing an advanced approach to feature selection that leverages the Grey Wolf Optimizer (GWO) in conjunction with a finely tuned Decision Tree (DT) model. The main objective is to enhance the precision and efficiency of feature selection processes within P2P lending datasets. The study begins by fine-tuning DT hyperparameters using Genetic Algorithms (GA), yielding an optimal configuration: ‘max_depth’ = 40, ‘min_samples_leaf’ = 20, and ‘criterion’ = ‘entropy’. Subsequent phases involve the application of GWO and modified GWO (nGWO, cGWO, and lGWO) to conduct feature selection under distinct Search Agent (SA) setups (SA = 5, SA = 20, SA = 50). Particularly impressive is the performance of the lGWO model with the SA = 50 setup, achieving a remarkable 91% accuracy while selecting 80.55% of the total 36 features. This study significantly improves how lenders manage risks in P2P lending by identifying high-risk borrowers more effectively, helping lenders reduce financial risks and benefiting all parties involved.

Integrating linear and nonlinear models to assess the built environment’s impact on older adults’ behaviors in China

China is experiencing rapid population aging. The one contributing factor affecting senior citizens’ lives is the disconnect between the built environment in urban and rural areas and the behavioral preferences of older adults. However, research on the relation between the built environment and the behavior of older individuals has been limited. Thus, this paper uses the most recent health tracking data on factors influencing aging in China released in 2020 (China Senior Health Survey Tracking Survey). Applying traditional regression, least absolute shrinkage and selection operator regression, and two decision tree optimization models from machine learning, a comprehensive comparative study is carried out to investigate the correlation between the built environment and the physical activity, dietary habits, and social interactions of older age groups. The findings reveal that built environment variables most significantly impact physical activity, accounting for 52.525%, followed by social interaction behaviors at 50.202% and dietary intake at 47.991%. Furthermore, the authors identify population density and greenness rate as the built environment factors having considerable effects on the behavior of older adults. Thus, this study establishes a theoretical foundation for developing age-friendly community environments for older adults.

Balancing Interpretability and Performance: Optimizing Random Forest Algorithm Based on Point-to-Point Federated Learning

Federated learning is extensively applied in collaborative data scenarios involving multiple data owners. While the majority of state-of-the-art federated learning algorithms are currently black-box models, making it challenging for users to comprehend how decisions are made. Random forest models are extensively utilized in medical contexts owing to their exceptional interpretability. However, when faced with multicenter data, the heterogeneity of data from each center often leads to its predictive performance falling short of expectations. To mitigate this challenge, the present study introduces DFLRF (Decentralized Federated Learning Random Forest), a federated learning algorithm based on random forests. Expanding on conventional random forests, DFLRF employs federated learning to disseminate decision tree models. It assesses and consolidates tree models from all client sites, thereby comprehensively addressing data disparities across various centers. The algorithm selects the optimal decision tree model based on the magnitude of model loss to guarantee the predictive performance of the final federated random forest model. The algorithm undergoes testing on a public dataset. Experimental results demonstrate that, compared to baseline algorithms, DFLRF enhances the AUC by 1.5% and the recall rate by 6%, while also ensuring superior interpretability.

Learning optimal biomarker-guided treatment policy for chronic disorders.

Electroencephalogram (EEG) provides noninvasive measures of brain activity and is found to be valuable for the diagnosis of some chronic disorders. Specifically, pre-treatment EEG signals in the alpha and theta frequency bands have demonstrated some association with antidepressant response, which is well-known to have a low response rate. We aim to design an integrated pipeline that improves the response rate of patients with major depressive disorder by developing a treatment policy guided by the resting state pre-treatment EEG recordings and other treatment effects modifiers. First, we design an innovative automatic site-specific EEG preprocessing pipeline to extract features with stronger signals than raw data. We then estimate the conditional average treatment effect (CATE) using causal forests and use a doubly robust technique to improve efficiency in the estimation of the average treatment effect. We present evidence of heterogeneity in the treatment effect and the modifying power of the EEG features, as well as a significant average treatment effect, a result that cannot be obtained with conventional methods. Finally, we employ an efficient policy learning algorithm to learn an optimal depth-2 treatment assignment decision tree and compare its performance with Q-Learning and outcome-weighted learning via simulation studies and an application to a large multi-site, double-blind, randomized controlled clinical trial, EMBARC.

Development of clinical screening tool for exocrine pancreatic insufficiency in patients with definite chronic pancreatitis

Background/ObjectivesNo simple, accurate diagnostic tests exist for exocrine pancreatic insufficiency (EPI), and EPI remains underdiagnosed in chronic pancreatitis (CP). We sought to develop a digital screening tool to assist clinicians to predict EPI in patients with definite CP. MethodsThis was a retrospective case-control study of patients with definite CP with/without EPI. Overall, 49 candidate predictor variables were utilized to train a Classification and Regression Tree (CART) model to rank all predictors and select a parsimonious set of predictors for EPI status. Five-fold cross-validation was used to assess generalizability, and the full CART model was compared with 4 additional predictive models. EPI misclassification rate (mRate) served as primary endpoint metric. Results274 patients with definite CP from 6 pancreatitis centers across the United States were included, of which 58 % had EPI based on predetermined criteria. The optimal CART decision tree included 10 variables. The mRate without/with 5-fold cross-validation of the CART was 0.153 (training error) and 0.314 (prediction error), and the area under the receiver operating characteristic curve was 0.889 and 0.682, respectively. Sensitivity and specificity without/with 5-fold cross-validation was 0.888/0.789 and 0.794/0.535, respectively. A trained second CART without pancreas imaging variables (n = 6), yielded 8 variables. Training error/prediction error was 0.190/0.351; sensitivity was 0.869/0.650, and specificity was 0.728/0.649, each without/with 5-fold cross-validation. ConclusionWe developed two CART models that were integrated into one digital screening tool to assess for EPI in patients with definite CP and with two to six input variables needed for predicting EPI status.

Heart Disease Prediction: A Machine Learning Approach with Higher Accuracy

Heart disease, a prevalent cardiovascular condition, poses significant health risks and affects millions worldwide. The alarming rise in heart disease cases in recent years demands proactive measures, making early prediction of these conditions crucial and concerning. By employing machine learning techniques, this study aims to identify patients who are more susceptible to heart disease based on diverse medical attributes. The Heart Disease Dataset from Kaggle, consisting of 1025 samples and 14 features, was incorporated into this investigation. And after preprocessing the dataset by removing duplicate and null values and implementing statistical imputation and several data graphs, like a scatter plot, box plot, histogram, etc., we split it into training and testing datasets and apply SMOTE technique on the training one. Various machinelearning approaches were used in this study, out of which the optimized decision tree gave the best accuracy of 98.96%.

Generative Model for Decision Trees

Decision trees are among the most popular supervised models due to their interpretability and knowledge representation resembling human reasoning. Commonly-used decision tree induction algorithms are based on greedy top-down strategies. Although these approaches are known to be an efficient heuristic, the resulting trees are only locally optimal and tend to have overly complex structures. On the other hand, optimal decision tree algorithms attempt to create an entire decision tree at once to achieve global optimality. We place our proposal between these approaches by designing a generative model for decision trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. Then, it adopts a genetic procedure to explore such latent space to find a compact decision tree with good predictive performance. We compare our proposal against classical tree induction methods, optimal approaches, and ensemble models. The results show that our proposal can generate accurate and shallow, i.e., interpretable, decision trees.