Tree Ensemble Methods Research Articles

A Study on the Layoff Problem After COVID-19 Based on Decision Tree Ensemble Methods

The COVID-19 pandemic, a cataclysmic event in modern history, has upended the global economy, precipitating extensive layoffs across diverse sectors. This study delves into the determinants of these workforce reductions through the lens of decision tree ensemble methods. By scrutinizing a comprehensive dataset, we harnessed statistical and machine learning algorithms to assess the significance of key variablesincluding industry type, geographic location, company development stage, and capital acquisitionon the scale of layoffs. This analysis further entailed predicting the total number of employees laid off and examining their distribution. The use of decision tree ensemble methods, such as random forests and gradient boosting, provided robust insights into the complex interplay of factors influencing layoff decisions. We discovered that industry type and company development stage were particularly critical in predicting layoff patterns, while geographic location and capital acquisition also played notable roles. This research offers a data-driven perspective on the layoff phenomena, shedding light on the multifaceted influences at play and offering a foundation for further inquiry and strategic policy development in response to economic downturns. By understanding these dynamics, policymakers and business leaders can better navigate future economic crises, potentially mitigating the adverse impacts on the workforce and promoting more resilient economic structures.

Enhancing long-term vegetation monitoring in Australia: a new approach for harmonising the Advanced Very High Resolution Radiometer normalised-difference vegetation (NVDI) with MODIS NDVI

Abstract. Long-term, reliable datasets of satellite-based vegetation condition are essential for understanding terrestrial ecosystem responses to global environmental change, particularly in Australia, which is characterised by diverse ecosystems and strong interannual climate variability. We comprehensively evaluate several existing global Advanced Very High Resolution Radiometer (AVHRR) normalised-difference vegetation index (NDVI) products for their suitability for long-term vegetation monitoring in Australia. Comparisons with the MODIS NDVI highlight significant deficiencies, particularly over densely vegetated regions. Moreover, all the assessed products failed to adequately reproduce the interannual variability in the pre-MODIS era as indicated by Landsat NDVI anomalies. To address these limitations, we propose a new approach to calibrating and harmonising NOAA's Climate Data Record of AVHRR NDVI to the MODIS MCD43A4 NDVI for Australia using a gradient-boosting decision tree ensemble method. Two versions of the datasets are developed, one incorporating climate data in the predictors (“AusENDVI-clim”: Australian Empirical NDVI-climate) and another that is independent of climate data (“AusENDVI-noclim”). These datasets, spanning 1982–2013 at a spatial resolution of 0.05° and with a monthly time step, exhibit strong correlations (r2=0.89–0.94) and low mean errors compared with MODIS MCD43A4 NDVI (mean absolute error (MAE) = 0.014–0.028, RMSE = 0.021–0.046), accurately reproducing seasonal cycles over densely vegetated regions. Furthermore, they closely replicate the interannual variability in vegetation condition in the pre-MODIS era. A reliable method for gap-filling the AusENDVI record is also developed that leverages climate, atmospheric CO2 concentration, and woody-cover fraction predictors. The resulting synthetic NDVI dataset shows excellent agreement with the MODIS MCD43A4 NDVI and the recalibrated AVHRR NDVI time series (r2=0.82–0.95, MAE = 0.016–0.029, RMSE = 0.039–0.041). Finally, we provide a complete 41-year dataset where the gap-filled AusENDVI-clim from January 1982 to February 2000 is joined with the MODIS MCD43A4 NDVI from March 2000 to December 2022. Analysing 40-year per-pixel trends in Australia's annual maximum NDVI revealed increasing values, and shifts in the timing, of the annual peak NDVI across most of the continent, underscoring the dataset's potential to address crucial questions regarding the changing vegetation phenology and its drivers. The AusENDVI dataset can be used for studying Australia's changing vegetation dynamics and downstream impacts on the terrestrial carbon and water cycles, and it provides a reliable foundation for further research into the drivers of vegetation change. AusENDVI is open access and available at https://doi.org/10.5281/zenodo.10802703 (Burton et al., 2024).

Boosting the Accuracy and Chemical Space Coverage of the Detection of Small Colloidal Aggregating Molecules Using the BAD Molecule Filter.

The ability to conduct effective high throughput screening (HTS) campaigns in drug discovery is often hampered by the detection of false positives in these assays due to small colloidally aggregating molecules (SCAMs). SCAMs can produce artifactual hits in HTS by nonspecific inhibition of the protein target. In this work, we present a new computational prediction tool for detecting SCAMs based on their 2D chemical structure. The tool, called the boosted aggregation detection (BAD) molecule filter, employs decision tree ensemble methods, namely, the CatBoost classifier and the light gradient-boosting machine, to significantly improve the detection of SCAMs. In developing the filter, we explore models trained on individual data sets, a consensus approach using these models, and, third, a merged data set approach, each tailored for specific drug discovery needs. The individual data set method emerged as most effective, achieving 93% sensitivity and 90% specificity, outperforming existing state-of-the-art models by 20 and 5%, respectively. The consensus models offer broader chemical space coverage, exceeding 90% for all testing sets. This feature is an important aspect particularly for early stage medicinal chemistry projects, and provides information on applicability domain. Meanwhile, the merged data set models demonstrated robust performance, with a notable sensitivity of 79% in the comprehensive 10-fold cross-validation test set. A SHAP analysis of model features indicates the importance of hydrophobicity and molecular complexity as primary factors influencing the aggregation propensity. The BAD molecule filter is readily accessible for the public usage on https://molmodlab-aau.com/Tools.html. This filter provides a new, more robust tool for aggregate prediction in the early stages of drug discovery to optimize hit rates and reduce associated testing and validation overheads.

Unboxing Tree ensembles for interpretability: A hierarchical visualization tool and a multivariate optimal re-built tree

The interpretability of models has become a crucial issue in Machine Learning because of algorithmic decisions' growing impact on real-world applications. Tree ensemble methods, such as Random Forests or XgBoost, are powerful learning tools for classification tasks. However, while combining multiple trees may provide higher prediction quality than a single one, it sacrifices the interpretability property resulting in “black-box” models. In light of this, we aim to develop an interpretable representation of a tree-ensemble model that can provide valuable insights into its behavior. First, given a target tree-ensemble model, we develop a hierarchical visualization tool based on a heatmap representation of the forest's feature use, considering the frequency of a feature and the level at which it is selected as an indicator of importance. Next, we propose a mixed-integer linear programming (MILP) formulation for constructing a single optimal multivariate tree that accurately mimics the target model predictions. The goal is to provide an interpretable surrogate model based on oblique hyperplane splits, which uses only the most relevant features according to the defined forest's importance indicators. The MILP model includes a penalty on feature selection based on their frequency in the forest to further induce sparsity of the splits. The natural formulation has been strengthened to improve the computational performance of mixed-integer software. Computational experience is carried out on benchmark datasets from the UCI repository using a state-of-the-art off-the-shelf solver. Results show that the proposed model is effective in yielding a shallow interpretable tree approximating the tree-ensemble decision function.

Machine Learning Approach for Improved Longitudinal Prediction of Progression from Mild Cognitive Impairment to Alzheimer's Disease.

Mild cognitive impairment (MCI) is cognitive decline that can indicate future risk of Alzheimer's disease (AD). We developed and validated a machine learning algorithm (MLA), based on a gradient-boosted tree ensemble method, to analyze phenotypic data for individuals 55-88 years old (n = 493) diagnosed with MCI. Data were analyzed within multiple prediction windows and averaged to predict progression to AD within 24-48 months. The MLA outperformed the mini-mental state examination (MMSE) and three comparison models at all prediction windows on most metrics. Exceptions include sensitivity at 18 months (MLA and MMSE each achieved 0.600); and sensitivity at 30 and 42 months (MMSE marginally better). For all prediction windows, the MLA achieved AUROC ≥ 0.857 and NPV ≥ 0.800. With averaged data for the 24-48-month lookahead timeframe, the MLA outperformed MMSE on all metrics. This study demonstrates that machine learning may provide a more accurate risk assessment than the standard of care. This may facilitate care coordination, decrease healthcare expenditures, and maintain quality of life for patients at risk of progressing from MCI to AD.

Z-number-valued rule-based decision trees

As a novel architecture of a fuzzy decision tree constructed on fuzzy rules, the fuzzy rule-based decision tree (FRDT) achieved better performance in terms of both classification accuracy and the size of the resulted decision tree than other classical decision trees such as C4.5, LADtree, BFtree, SimpleCart and NBTree. The concept of Z-number extends the classical fuzzy number to model both uncertain and partial reliable information. Z-numbers have significant potential in rule-based systems due to their strong representation capability. This paper designs a Z-number-valued rule-based decision tree (ZRDT) and provides the learning algorithm. Firstly, the information gain is used to replace the fuzzy confidence in FRDT to select features in each rule. Additionally, we use the negative samples to generate the second fuzzy numbers that adjust the first fuzzy numbers and improve the model's fit to the training data. The proposed ZRDT is compared with the FRDT with three different parameter values and two classical decision trees, PUBLIC and C4.5, and a decision tree ensemble method, AdaBoost.NC, in terms of classification effect and size of decision trees. Based on statistical tests, the proposed ZRDT has the highest classification performance with the smallest size for the produced decision tree.

Hybrid Renewable Energy System Design: A Machine Learning Approach for Optimal Sizing with Net-Metering Costs

Hybrid renewable energy systems with photovoltaic and energy storage systems have gained popularity due to their cost-effectiveness, reduced dependence on fossil fuels and lower CO2 emissions. However, their techno-economic advantages are crucially dependent on the optimal sizing of the system. Most of the commercially available optimization programs adopt an algorithm that assumes repeated weather conditions, which is becoming more unrealistic considering the recent erratic behavior of weather patterns. To address this issue, a data-driven framework is proposed that combines machine learning and hybrid metaheuristics to predict weather patterns over the lifespan of a hybrid renewable energy system in optimizing its size. The framework uses machine learning tree ensemble methods such as the cat boost regressor, light gradient boosting machine and extreme gradient boosting to predict the hourly solar radiation and load demand. Nine different hybrid metaheuristics are used to optimize the hybrid renewable energy system using forecasted data over 15 years, and the optimal sizing results are compared with those obtained from 1-year data simulation. The proposed approach leads to a more realistic hybrid renewable energy system capacity that satisfies all system constraints while being more reliable and environmentally friendly. The proposed framework provides a robust approach to optimizing hybrid renewable energy system sizing and performance evaluation that accounts for changing weather conditions over the lifespan of the system.

Patient-specific quality assurance failure prediction with deep tabular models

Purpose. Patient-specific quality assurance (PSQA) failures in radiotherapy can cause a delay in patient care and increase the workload and stress of staff. We developed a tabular transformer model based directly on the multi-leaf collimator (MLC) leaf positions (without any feature engineering) to predict IMRT PSQA failure in advance. This neural model provides an end-to-end differentiable map from MLC leaf positions to the probability of PSQA plan failure, which could be useful for regularizing gradient-based leaf sequencing optimization algorithms and generating a plan that is more likely to pass PSQA. Method. We retrospectively collected DICOM RT PLAN files of 968 patient plans treated with volumetric arc therapy. We constructed a beam-level tabular dataset with 1873 beams as samples and MLC leaf positions as features. We trained an attention-based neural network FT-Transformer to predict the ArcCheck-based PSQA gamma pass rates. In addition to the regression task, we evaluated the model in the binary classification context predicting the pass or fail of PSQA. The performance was compared to the results of the two leading tree ensemble methods (CatBoost and XGBoost) and a non-learned method based on mean-MLC-gap. Results. The FT-Transformer model achieves 1.44% Mean Absolute Error (MAE) in the regression task of the gamma pass rate prediction and performs on par with XGBoost (1.53 % MAE) and CatBoost (1.40 % MAE). In the binary classification task of PSQA failure prediction, FT-Transformer achieves 0.85 ROC AUC (compared to the mean-MLC-gap complexity metric achieving 0.72 ROC AUC). Moreover, FT-Transformer, CatBoost, and XGBoost all achieve 80% true positive rate while keeping the false positive rate under 20%. Conclusions. We demonstrated that reliable PSQA failure predictors can be successfully developed based solely on MLC leaf positions. FT-Transformer offers an unprecedented benefit of providing an end-to-end differentiable map from MLC leaf positions to the probability of PSQA failure.

Inferring feature importance with uncertainties with application to large genotype data

Estimating feature importance, which is the contribution of a prediction or several predictions due to a feature, is an essential aspect of explaining data-based models. Besides explaining the model itself, an equally relevant question is which features are important in the underlying data generating process. We present a Shapley-value-based framework for inferring the importance of individual features, including uncertainty in the estimator. We build upon the recently published model-agnostic feature importance score of SAGE (Shapley additive global importance) and introduce Sub-SAGE. For tree-based models, it has the advantage that it can be estimated without computationally expensive resampling. We argue that for all model types the uncertainties in our Sub-SAGE estimator can be estimated using bootstrapping and demonstrate the approach for tree ensemble methods. The framework is exemplified on synthetic data as well as large genotype data for predicting feature importance with respect to obesity.

QuantumTox: Utilizing quantum chemistry with ensemble learning for molecular toxicity prediction

Molecular toxicity prediction plays an important role in drug discovery, which is directly related to human health and drug fate. Accurately determining the toxicity of molecules can help weed out low-quality molecules in the early stage of drug discovery process and avoid depletion later in the drug development process. Nowadays, more and more researchers are starting to use machine learning methods to predict the toxicity of molecules, but these models do not fully exploit the 3D information of molecules. Quantum chemical information, which provides stereo structural information of molecules, can influence their toxicity. To this end, we propose QuantumTox, the first application of quantum chemistry in the field of drug molecule toxicity prediction compared to existing work. We extract the quantum chemical information of molecules as their 3D features. In the downstream prediction phase, we use Gradient Boosting Decision Tree and Bagging ensemble learning methods together to improve the accuracy and generalization of the model. A series of experiments on various tasks show that our model consistently outperforms the baseline model and that the model still performs well on small datasets of less than 300.

Machine learning determination of applied behavioral analysis treatment plan type

BackgroundApplied behavioral analysis (ABA) is regarded as the gold standard treatment for autism spectrum disorder (ASD) and has the potential to improve outcomes for patients with ASD. It can be delivered at different intensities, which are classified as comprehensive or focused treatment approaches. Comprehensive ABA targets multiple developmental domains and involves 20–40 h/week of treatment. Focused ABA targets individual behaviors and typically involves 10–20 h/week of treatment. Determining the appropriate treatment intensity involves patient assessment by trained therapists, however, the final determination is highly subjective and lacks a standardized approach. In our study, we examined the ability of a machine learning (ML) prediction model to classify which treatment intensity would be most suited individually for patients with ASD who are undergoing ABA treatment.MethodsRetrospective data from 359 patients diagnosed with ASD were analyzed and included in the training and testing of an ML model for predicting comprehensive or focused treatment for individuals undergoing ABA treatment. Data inputs included demographics, schooling, behavior, skills, and patient goals. A gradient-boosted tree ensemble method, XGBoost, was used to develop the prediction model, which was then compared against a standard of care comparator encompassing features specified by the Behavior Analyst Certification Board treatment guidelines. Prediction model performance was assessed via area under the receiver-operating characteristic curve (AUROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).ResultsThe prediction model achieved excellent performance for classifying patients in the comprehensive versus focused treatment groups (AUROC: 0.895; 95% CI 0.811–0.962) and outperformed the standard of care comparator (AUROC 0.767; 95% CI 0.629–0.891). The prediction model also achieved sensitivity of 0.789, specificity of 0.808, PPV of 0.6, and NPV of 0.913. Out of 71 patients whose data were employed to test the prediction model, only 14 misclassifications occurred. A majority of misclassifications (n = 10) indicated comprehensive ABA treatment for patients that had focused ABA treatment as the ground truth, therefore still providing a therapeutic benefit. The three most important features contributing to the model’s predictions were bathing ability, age, and hours per week of past ABA treatment.ConclusionThis research demonstrates that the ML prediction model performs well to classify appropriate ABA treatment plan intensity using readily available patient data. This may aid with standardizing the process for determining appropriate ABA treatments, which can facilitate initiation of the most appropriate treatment intensity for patients with ASD and improve resource allocation.

Deep Network Architectures as Feature Extractors for Multi-Label Classification of Remote Sensing Images

Data in the form of images are now generated at an unprecedented rate. A case in point is remote sensing images (RSI), now available in large-scale RSI archives, which have attracted a considerable amount of research on image classification within the remote sensing community. The basic task of single-target multi-class image classification considers the case where each image is assigned exactly one label from a predefined finite set of class labels. Recently, however, image annotations have become increasingly complex, with images labeled with several labels (instead of just one). In other words, the goal is to assign multiple semantic categories to an image, based on its high-level context. The corresponding machine learning tasks is called multi-label classification (MLC). The classification of RSI is currently predominantly addressed by deep neural network (DNN) approaches, especially convolutional neural networks (CNNs), which can be utilized as feature extractors as well as end-to-end methods. After only considering single-target classification for a long period, DNNs have recently emerged that address the task of MLC. On the other hand, trees and tree ensembles for MLC have a long tradition and are the best-performing class of MLC methods, but need predefined feature representations to operate on. In this work, we explore different strategies for model training based on the transfer learning paradigm, where we utilize different families of (pre-trained) CNN architectures, such as VGG, EfficientNet, and ResNet. The architectures are trained in an end-to-end manner and used in two different modes of operation, namely, as standalone models that directly perform the MLC task, and as feature extractors. In the latter case, the learned representations are used with tree ensemble methods for MLC, such as random forests and extremely randomized trees. We conduct an extensive experimental analysis of methods over several publicly available RSI datasets and evaluate their effectiveness in terms of standard MLC measures. Of these, ranking-based evaluation measures are most relevant, especially ranking loss. The results show that, for addressing the RSI-MLC task, it is favorable to use lightweight network architectures, such as EfficientNet-B2, which is the best performing end-to-end approach, as well as a feature extractor. Furthermore, in the datasets with a limited number of images, using traditional tree ensembles for MLC can yield better performance compared to end-to-end deep approaches.

Analysis of Parkinson's Disease Using an Imbalanced-Speech Dataset by Employing Decision Tree Ensemble Methods.

Parkinson's disease (PD) currently affects approximately 10 million people worldwide. The detection of PD positive subjects is vital in terms of disease prognostics, diagnostics, management and treatment. Different types of early symptoms, such as speech impairment and changes in writing, are associated with Parkinson disease. To classify potential patients of PD, many researchers used machine learning algorithms in various datasets related to this disease. In our research, we study the dataset of the PD vocal impairment feature, which is an imbalanced dataset. We propose comparative performance evaluation using various decision tree ensemble methods, with or without oversampling techniques. In addition, we compare the performance of classifiers with different sizes of ensembles and various ratios of the minority class and the majority class with oversampling and undersampling. Finally, we combine feature selection with best-performing ensemble classifiers. The result shows that AdaBoost, random forest, and decision tree developed for the RUSBoost imbalanced dataset perform well in performance metrics such as precision, recall, F1-score, area under the receiver operating characteristic curve (AUROC) and the geometric mean. Further, feature selection methods, namely lasso and information gain, were used to screen the 10 best features using the best ensemble classifiers. AdaBoost with information gain feature selection method is the best performing ensemble method with an F1-score of 0.903.

Evaluation of machine learning models in the classification of pool boiling regimes up to critical heat flux based on boiling acoustics

The present study focuses on the performance of the machine learning methods in classifying the boiling regimes of water up to critical heat flux conditions based on the acoustic characteristics of boiling. The data set is generated by conducting a pool boiling experiment on a wire heater at various heat fluxes varying from 54.95 kW/m2 to 2898.67 kW/m2. A Kanthal D wire of standard wire gauge 36 is used. The data set is divided into three classes: no boiling, nucleate boiling, and critical heat flux to identify the boiling incipience and critical heat flux. Much focus is insisted on identifying critical heat flux as it carries more practical importance in the safety of the cooling systems. Data set size optimization is performed to find the lowest number of records required for each method. Three machine-learning methods are employed to predict the boiling regime, namely, binary decision tree method, decision tree ensemble method and naive Bayes method. Out of these, the decision tree ensemble outperformed the binary decision tree and naive Bayes classifiers. The decision tree ensemble classified the regimes in the given data with the lowest classification error and inference time. The accurate classification of boiling regimes based on boiling acoustics strengthens the safety measures in real-time monitoring of cooling systems.

Enhancing Reliability Analysis with Multisource Data: Mitigating Adverse Selection Problems in Bridge Monitoring and Management

Data collected using sensors plays an essential role in active bridge health monitoring. When analyzing a large number of bridges in the U.S., the National Bridge Inventory data as been widely used. Yet, the database does not provide information about live loads, one of the most indeterminate variables for monitoring bridges. Such asymmetric information can lead to an adverse selection problem in making maintenance, rehabilitation, and repair decisions. This study proposes a data-driven reliability analysis to assess probabilities of bridge failure by synthesizing NBI data and Weigh-In-Motion (WIM) data for a large number of bridges in Georgia. On the resistance side, tree ensemble methods are employed to support the hypothesis that the NBI operating load rating represents the distribution of bridge resistance capacities which change over time. On the loading side, the live load distribution is derived from field data collected using WIM sensors. Our results show that the proposed WIM data-enabled reliability analysis substantially enhances information symmetry and provides a reliability index that supports monitoring of bridge conditions, depending on live loads and load-carrying capacities.

Data-Driven Direct Diagnosis of PV Connected Batteries

Photovoltaic systems are providing a growing share of power to the electric grid worldwide. As more and more of these systems are tied with battery energy storage to balance their intermittency, being able to ensure long and safe operation of these batteries will become critical. Because of their sporadic usage, far from the typical constant current or constant power, diagnosability without having to perform lengthy maintenance cycles will be a key factor to accelerate deployment.This work proposes a new methodology to enable the training of machine learning algorithms so they could be applied directly to photovoltaic (PV) battery charging data for opportunistic diagnosis. This is done by training the algorithms on synthetic voltage data under different degradations calculated from clear-sky model irradiance data. Validation was performed on synthetic voltage responses calculated from plane of array irradiance observations for a PV system located in Maui, HI, USA.Herein, we will present the result of our proof-of-concept study where we investigated the diagnosability of the voltage response of PV charged batteries under 18 different cloud coverages scenarios and for more than 10,000 different degradations. We also evaluated the effect of training conditions on clear-sky irradiance data by analyzing the resulting differences from training on 700,000 different voltage curves for a single day vs. 45,000 different voltage curves for of each of the 18 days used for cloud coverage estimation. In addition, we considered and quantified the effect of cell-to-cell variations. Training was performed on five state-of-the-art machine learning algorithms for battery diagnosis which included decision tree ensemble methods and neural networks. Finally, because the diagnosis was done outside of constant current, the capacity- and time-based information were decorrelated and compared. While time-base diagnosis is the most interesting for deployed systems, it was shown to be less accurate than its capacity counterpart for the tested algorithms. Yet, the accuracy of the diagnosis is still satisfactory for days where cloud effect is limited (average RMSE of 2.5% compared to 1.3% for capacity based if 50% or more clear skies).Summarizing, we will show that the degradation of PV-connected batteries could be diagnosed opportunistically without the need for maintenance cycles using well-trained state-of-art machine learning algorithms. Calculated RMSEs were below 2% for days with 50% or more clear skies considering more than 10,000 different degradation paths with 25% or less of the three thermodynamic degradation modes. In addition to the results themselves, and before real PV battery degradation data becomes available, this work is showing the significant benefits of using synthetic data to understand the expected variations and to start preparing adapted tools for when data become available.

Quantitative Comparison of Tree Ensemble Learning Methods for Perfume Identification Using a Portable Electronic Nose

Perfume identification (PI) based on an electronic nose (EN) can be used for exposing counterfeit perfumes more time-efficiently and cost-effectively than using gas chromatography and mass spectrometry instruments. During the past five years, decision-tree-based ensemble learning methods, also called tree ensemble learning methods, have demonstrated excellent performance when solving multi-class classification problems. However, the performance of tree ensemble learning methods for the EN-based PI problem remains uncertain. In this paper, four well-known tree ensemble learning classification methods, random forest (RF), stagewise additive modeling using a multi-class exponential loss function (SAMME), gradient-boosting decision tree (GBDT), and extreme gradient boosting (XGBoost), were implemented for PI using our self-designed EN. For fair comparison, all the tested classification methods used as input the same feature data extracted using principal component analysis. Moreover, two benchmark methods, neural network and support vector machine, were also tested with the same experimental setup. The quantitative results of experiments undertaken demonstrated that the mean PI accuracy achieved by XGBoost was up to 97.5%, and that XGBoost outperformed other tested methods in terms of accuracy mean and variance based on our self-designed EN.

Minimum relevant features to obtain AI explainable system for predicting breast cancer in WDBC

The potential to explain why a machine learning model produces a certain prediction in incomprehensible terms is becoming increasingly crucial, as it provides accountability and confidence in the algorithm's decision-making process. The interpretation of complex models is difficult. Various approaches to dealing with this issue are being offered. These problems are typically handled in tree ensemble methods by assigning priority levels to input features globally or for a specific prediction. We show that current feature attribution approaches are inconclusive, and develop solutions using SHAP (SHapley Additive Explanation) values, LIME (Local Interpretable Model-Agnostic Explanations), and the Skope Rules package. We employ feature selection methods from SHAP and LIME in this work, which uses the Breast cancer Wisconsin data sets. In the suggested method, features are chosen at the first level of feature selection using Decision tree entropy values. Based on the SHAP and LIME reports, level 2 features are chosen from fewer options. The features are tested on a Decision Tree (DT) model and a DT and Support Vector Machine (SVM) ensemble. Experiments suggest that the ensemble works better as compared to DT. We have also used the Skope Rules package to generate global rules for generalization.

DeepTLF: robust deep neural networks for heterogeneous tabular data

Although deep neural networks (DNNs) constitute the state of the art in many tasks based on visual, audio, or text data, their performance on heterogeneous, tabular data is typically inferior to that of decision tree ensembles. To bridge the gap between the difficulty of DNNs to handle tabular data and leverage the flexibility of deep learning under input heterogeneity, we propose DeepTLF, a framework for deep tabular learning. The core idea of our method is to transform the heterogeneous input data into homogeneous data to boost the performance of DNNs considerably. For the transformation step, we develop a novel knowledge distillations approach, TreeDrivenEncoder, which exploits the structure of decision trees trained on the available heterogeneous data to map the original input vectors onto homogeneous vectors that a DNN can use to improve the predictive performance. Within the proposed framework, we also address the issue of the multimodal learning, since it is challenging to apply decision tree ensemble methods when other data modalities are present. Through extensive and challenging experiments on various real-world datasets, we demonstrate that the DeepTLF pipeline leads to higher predictive performance. On average, our framework shows 19.6% performance improvement in comparison to DNNs. The DeepTLF code is publicly available.

A novel bagged tree ensemble regression method with multiple correlation coefficients to predict the train body vibrations using rail inspection data

Prediction of the train body vibrations induced by the train running is desirable and useful to ensure comfortable service, reliable, safe, and secure operation of railway systems. By using daily measurement data from GJ-5 rail detection vehicle, this paper presents a novel prediction algorithm, which is based on bagged tree ensemble regression with multiple correlation coefficients. To obtain the valuable data sets from a large amount of inspection data, an approach of multiple correlation coefficients is used for the data pre-processing. Then the prediction model of train body vibrations is established by combining regression tree algorithm and bagged ensemble algorithm. By training the valuable data sets, the prediction results are calculated by the bagged tree ensemble regression method. Finally, the proposed method is evaluated with experimental data and the traditional method. The experimental results show that the proposed method not only has higher accuracy but also can effectively reduce the number of the data sets, the accuracy is up to 98% and the number of valuable training data sets is reduced by 78.3%. The new method proposed in the paper can accurately predict the vibration status of the train body without installing any new sensors and other monitoring equipment on the train, which can reduce maintenance costs and prevent potential safety risks.