Gradient Boosting Method Research Articles

Application of the extreme gradient boosting method to quantitatively analyze the mechanism of radon anomalous change in Banglazhang hot spring before the Lijiang Mw 7.0 earthquake

Radon concentration in groundwater is affected by meteorological and seismo-tectonic factors and is sensitive indicators of crustal stress. While previous studies have focused on the mechanisms of periodic and precursory radon anomalies, few have quantitatively assessed their influences, and compared their rates of change under normal and earthquake conditions. This study constructs four models to analyze the mechanisms of radon variation under natural and seismic conditions using the Extreme Gradient Boosting method: 1) the multi-year dynamic variation model, 2) the maximal radon concentration model, 3) the minimum radon concentration model, and 4) the precursory anomaly model. The feature relative importance of external factors influencing radon concentration in groundwater are estimated. The optimized extreme gradient boosting model estimated the feature importance of spring discharge (SD), water temperature (WT), precipitation (P), barometric pressure (BP), and antecedent radon (AR) to be 23.79%, 22.16%, 9.81%, 18.8% and 25.44%, respectively. Thus, SD is the most important influence on radon variation under normal conditions. For radon variation during the earthquake preparation period, the feature relative importance indicates significant changes in WT (increasing from 22.16% to 27.70%) and SD (decreasing from 23.79% to 17.99%). Although WT was the most important predictor in the precursory anomaly model, its effect on radon solubility is insufficient to explain the radon anomalies prior to the Lijiang Mw 7.0 earthquake. Analysis of the precursory mechanisms of these radon anomalies in terms of SD, WT, P, BP, and radon emanation found that radon anomalies are most likely caused by increases in radon emanation due to the earthquake-induced formation of microfractures in rock.

EXtreme Gradient Boosting-based method to classify patients with COVID-19

Different demographic, clinical and laboratory variables have been related to the severity and mortality following SARS-CoV-2 infection. Most studies applied traditional statistical methods and in some cases combined with a...

Phenotypes based Classification of Blood-Brain-Barrier Drugs using Feature Selection Methods and Extreme Gradient Boosting.

In this work, an attempt has been made to discriminate drug with blood brain barrier (BBB) permeability using clinical phenotypes and extreme gradient boosting (XGBoost) methods. For this, the drug name and their clinical phenotypes namely side effects and indications are obtained from public available database. Prominent clinical phenotypes are selected using genetic algorithm (GA) and binary particle swarm optimization (BPSO). Four machine algorithms namely k-Nearest Neighbours, support vector machines, rotation forest and XGBoost are used for classification of BBB drugs. The result show that the proposed clinical phenotypes based features are able to distinguish drugs with BBB permeability. The maximum number of clinical phenotypes (69%) is reduced by BPSO and GA for classification. The XGBoost method is found to be most accurate [Formula: see text] is discriminating drugs with BBB permeability. The proposed approach are found to be capable of handling multi-parametric characteristics of the drugs. Particularly, the combination of XGBoost with combination of side effects and indications could be used for precision medicine applications. Clinical relevance- This establishes XGBoost approach for improved BBB permeability based drug classification with F1 =98.7% using exclusively clinical phenotypes.

How do machines predict energy use? Comparing machine learning approaches for modeling household energy demand in the United States

This paper illustrates the use of different machine learning techniques to estimate household energy demand. To demonstrate the performance of the techniques, we discuss how the different machine learning algorithms select a model or models of energy usage and we explore how well the models predict usage. Our study employs a high-dimensional dataset of housing, socioeconomic, and behavioral characteristics, provided by the U.S. Energy Information Administration's ongoing Residential Energy Consumption Survey. In addition to discussing the machine learning models, we estimate energy price elasticities, which are important indicators of how sensitive households are to changes in residential energy prices. Given the broad set of data in the survey, we compare and contrast various machine learning techniques to see which model provides the best overall fit to the data. We find that a random forest algorithm performs better than the other machine learning approaches, which include a step-wise Akaike Information Criterion, partial least squares, ordinary least squares, k-nearest neighbors, penalized regression, and gradient boosting methods. Finally, we discuss how machine learning can be used to inform residential energy policies and predict household energy consumption.

QSAR analysis and experimental evaluation of new quinazoline-containing hydroxamic acids as histone deacetylase 6 inhibitors

ABSTRACT Histone deacetylase inhibitors represent the most important class of drugs for the treatment of human cancer and other diseases due to their influence on cell growth, differentiation, and apoptosis. Among the well-known eighteen histone deacetylases, histone deacetylase 6 (HDAC6), which is involved in oncogenesis, cell survival, and cancer cell metastasis, is of great importance. Using the CDK and alvaDesc molecular descriptors and the Random Forest and EXtreme Gradient Boosting methods, we propose a number of adequate QSAR classification models, which are integrated into a consensus model and are freely available on the OCHEM web platform (https://ochem.eu). The consensus QSAR model is used for virtual screening of a series of seven new compounds, the derivatives of N-((hydroxyamino)-oxoalkyl)-2-(quinazoline-4-ilamino)-benzamides, the synthesis schemes of which are also presented in this work. In vitro evaluation of the inhibitory activity (IC50) of this series of compounds against HDAC6 allowed us to confirm the results of virtual screening and to reveal promising compounds V-2 and V-4, IC50 of which is 3.25 nM and 0.04 nM, respectively. The subsequent in silico evaluation of the main ADMET properties of active compounds V-2 and V-4 allowed us to find that they have acceptable pharmacokinetic parameters and level of acute toxicity.

XGB-Hybrid Fingerprint Classification Model for Virtual Screening of Meningitis Drug Compounds Candidate

Meningitis is an infection of the lining of the brain caused by diffuse inflammation, and this condition is caused by viruses or bacteria that cause Meningitis. Prevention for this disease is still in the form of strengthening antibodies with vaccines. There is no significant compound to relieve or treat Meningitis patients. In previous studies, they got seven proteins vital to Meningitis. We continued to investigate the compounds associated with the seven proteins. We chose the in-silico process by utilizing data in an open database. We use several databases for the data collection process. After that, the compound data were extracted for bonding features and chemical elements using molecular fingerprints. We use two fingerprint methods, where both we combine with three types of combinations. The combined results produce three types of datasets with different matrix sizes. We establish the Extreme Gradient Boosting (XGB) method to form the classification model for the three datasets, select the best classification model, and compare it with other classification algorithms. The XGB model has better quality than the classification model of other algorithms. We used this model to predict and quantify compounds that strongly bind to seven vital meningitis proteins. The compound with the highest predictive score (we found more than 0.99) became a drug candidate to inhibit or neutralize Meningitis.

Strain prediction for historical timber buildings with a hybrid Prophet-XGBoost model

The accurate characterization of environmental effects, especially temperature and humidity, on historical timber buildings is essential for the timely detection of deterioration and structural performance damage. This research aims to effectively exclude environmental impacts in future structural state assessments by constructing an integrated model based on combining the Prophet and extreme gradient boosting (XGBoost) methods. This hybrid model integrates the two single algorithms based on their specialties. Prophet is applied to time series analysis with the addition of humidity effects to characterize their impact on series periodicity, and XGBoost based on improved Bayesian optimization (BayesOpt) is used to describe the effects of ambient temperature and humidity on the structural responses. The hybrid model is constructed by a new optimized weighting method. The performance of the proposed model is compared with that of single models, singular spectrum analysis and polynomial regression (SSA-PR) model and two traditional weighting-based hybrid models on data collected from the structural health monitoring system of the Feiyun Wood Pavilion (FWP). The results show that the inclusion of humidity effects enables the model to more accurately characterize the relationships between environmental factors and strain, and the predictive performance of the proposed hybrid model is better than that of other models.

Hippocampal Metastasis Rate Based on Non-Small Lung Cancer TNM Stage and Molecular Markers.

Hippocampal-avoidance whole-brain radiation therapy (HA-WBRT) is justified because of low hippocampal brain metastases (BM) rate and its prevention of cognitive decline. However, we hypothesize that the risk of developing BM in the hippocampal-avoidance region (HAR) may differ depending on the lung-cancer stage and molecular status. We retrospectively reviewed 123 patients with non-small cell lung cancer (NSCLC) at the initial diagnosis of BM. The number of BMs within the HAR (5 mm expansion) was counted. The cohort was divided into patients with and without BMs in the HAR, and their clinical variables, TNM stage, and epidermal growth factor receptor (EGFR) status were compared. The most influential variable predicting BMs in the HAR was determined using multi-variable logistic regression, classification and regression tree (CART) analyses, and gradient boosting method (GBM). The feasibility of HAR expansion was tested using generalized estimating equation marginal model. Patients with BMs in the HAR were more frequently non-smokers, and more likely to have extra-cranial metastases and EGFR mutations (p<0.05). Multi-variable analysis revealed that extra-cranial metastases were independently associated with the presence of BM in the HAR (odds ratio=8.75, p=0.04). CART analysis and GBM revealed that the existence of extra-cranial metastasis was the most influential variable predicting BM occurrence in the HAR (variable importance: 23% and relative influence: 37.38). The estmated BM incidence of patients without extra-cranial metastases in th extended HAR (7.5-mm and 10-mm expansion) did not differ significantly from that in the conventional HAR. In conclusion, NSCLC patients with extra-cranial metastases were more likely to have BMs in the HAR than those without extra-cranial metastases.

Towards Machine Learning Interpretability for Tabular Data with Mixed Data Types

Gradient Boosting (GB) algorithms have been proposed for a variety of automated predictions and classification tasks with applications in many domains. These methods work faster and provide superior performance compared to deep learning methods when applied to tabular datasets. Another advantage is their interpretability. There are many machine learning methods that can train tabular data successfully, however, the inner workings are usually hidden from the user. In this context, SHAP values combined with GB methods, increase model transparency and provide not only consistent feature rankings but also show the contributions of the predictors for individual instances. In this work, we train multiple GB models using several tabular datasets and compare the result in terms of speed, performance, and the global and local models' interpretability.

Clinical Prediction of Heart Failure in Hemodialysis Patients: Based on the Extreme Gradient Boosting Method.

Background: Heart failure (HF) is the main cause of mortality in hemodialysis (HD) patients. However, it is still a challenge for the prediction of HF in HD patients. Therefore, we aimed to establish and validate a prediction model to predict HF events in HD patients. Methods: A total of 355 maintenance HD patients from two hospitals were included in this retrospective study. A total of 21 variables, including traditional demographic characteristics, medical history, and blood biochemical indicators, were used. Two classification models were established based on the extreme gradient boosting (XGBoost) algorithm and traditional linear logistic regression. The performance of the two models was evaluated based on calibration curves and area under the receiver operating characteristic curves (AUCs). Feature importance and SHapley Additive exPlanation (SHAP) were used to recognize risk factors from the variables. The Kaplan–Meier curve of each risk factor was constructed and compared with the log-rank test. Results: Compared with the traditional linear logistic regression, the XGBoost model had better performance in accuracy (78.5 vs. 74.8%), sensitivity (79.6 vs. 75.6%), specificity (78.1 vs. 74.4%), and AUC (0.814 vs. 0.722). The feature importance and SHAP value of XGBoost indicated that age, hypertension, platelet count (PLT), C-reactive protein (CRP), and white blood cell count (WBC) were risk factors of HF. These results were further confirmed by Kaplan–Meier curves. Conclusions: The HF prediction model based on XGBoost had a satisfactory performance in predicting HF events, which could prove to be a useful tool for the early prediction of HF in HD.

Development of New Performance Measures Based on Data Mining Weights for Hotspot Identification

In this study, new performance measures are proposed for hotspot identification in urban intersections. These measures reflect severity factor weights, which are determined based on data mining. To estimate the severity factor weights of crashes at urban intersections, the study utilizes tree-based random forest (RF) and extreme gradient boosting (XGB) methods. The importance of variables in the severity classification model is standardized and utilized for calculating the score of each crash, which is aggregated into intersections. The aggregated score is used as a dependent variable for the safety performance functions (SPFs) in the network screening process. To illustrate the under-dispersed severity score aggregation data, SPFs that follow the COM-Poisson distribution as well as the negative binomial (NB) are developed. Independent variables in SPFs set up intersection geometry elements that can be collected from online GIS services. Four additional performance measures are proposed, each reflecting a severity weight. Data about a total of 42,513 intersection crashes from 2017 to 2018 in South Korea were collected for crash injury severity analysis. Hotspot identification was performed on 81 intersections, and three consistency tests were conducted to validate the four measures. Tests show that the RF-based weighted [Formula: see text] and [Formula: see text] have the best consistency. Since the severity factor weights of each crash are reflected, intersections vulnerable to dangerous crashes can be analyzed in more detail. Using this method, it is expected that effective safety improvement project plans can be established with the input of safety managers in the future.

Utilization of machine learning methods for predicting surgical outcomes after total knee arthroplasty.

BackgroundPredictive models could help clinicians identify risk factors that cause adverse events after total knee arthroplasty (TKA), allowing for appropriate preoperative preventive interventions and allocation of resources.MethodsThe National Inpatient Sample datasets from 2010–2014 were used to build Logistic Regression (LR), Gradient Boosting Method (GBM), Random Forest (RF), and Artificial Neural Network (ANN) predictive models for three clinically relevant outcomes after TKA—disposition at discharge, any post-surgical complications, and blood transfusion. Model performance was evaluated using the Brier scores as calibration measures, and area under the ROC curve (AUC) and F1 scores as discrimination measures.ResultsGBM-based predictive models were observed to have better calibration and discrimination than the other models; thus, indicating comparatively better overall performance. The Brier scores for GBM models predicting the outcomes under investigation ranged from 0.09–0.14, AUCs ranged from 79–87%, and F1-scores ranged from 41–73%. Variable importance analysis for GBM models revealed that admission month, patient location, and patient’s income level were significant predictors for all the outcomes. Additionally, any post-surgical complications and blood transfusions were significantly predicted by deficiency anemias, and discharge disposition by length of stay and age groups. Notably, any post-surgical complications were also significantly predicted by the patient undergoing blood transfusion.ConclusionsThe predictive abilities of the ML models were successfully demonstrated using data from the National Inpatient Sample (NIS), indicating a wide range of clinical applications for obtaining accurate prognoses of complications following orthopedic surgical procedures.

Detection of Target Genes for Drug Repurposing to Treat Skeletal Muscle Atrophy in Mice Flown in Spaceflight.

Skeletal muscle atrophy is a common condition in aging, diabetes, and in long duration spaceflights due to microgravity. This article investigates multi-modal gene disease and disease drug networks via link prediction algorithms to select drugs for repurposing to treat skeletal muscle atrophy. Key target genes that cause muscle atrophy in the left and right extensor digitorum longus muscle tissue, gastrocnemius, quadriceps, and the left and right soleus muscles are detected using graph theoretic network analysis, by mining the transcriptomic datasets collected from mice flown in spaceflight made available by GeneLab. We identified the top muscle atrophy gene regulators by the Pearson correlation and Bayesian Markov blanket method. The gene disease knowledge graph was constructed using the scalable precision medicine knowledge engine. We computed node embeddings, random walk measures from the networks. Graph convolutional networks, graph neural networks, random forest, and gradient boosting methods were trained using the embeddings, network features for predicting links and ranking top gene-disease associations for skeletal muscle atrophy. Drugs were selected and a disease drug knowledge graph was constructed. Link prediction methods were applied to the disease drug networks to identify top ranked drugs for therapeutic treatment of skeletal muscle atrophy. The graph convolution network performs best in link prediction based on receiver operating characteristic curves and prediction accuracies. The key genes involved in skeletal muscle atrophy are associated with metabolic and neurodegenerative diseases. The drugs selected for repurposing using the graph convolution network method were nutrients, corticosteroids, anti-inflammatory medications, and others related to insulin.

High waist circumference is a risk factor of new-onset hypertension: Evidence from the China Health and Retirement Longitudinal Study.

This study aims to investigate the association between waist circumference and the development of hypertension based on a nationwide cohort Chinese population. A total of 5330 individuals free of hypertension at baseline were collected from the China Health and Retirement Longitudinal Study. The association between waist circumference and the development of hypertension was analyzed by an adjusted cox regression model and visualized by restricted cubic splines. Further, we applied the supervised machine learning methods to evaluate the importance of multiple variates for new‐onset hypertension. Additionally, the robustness of the association was assessed by a subgroup analysis. A total of 1490 individuals (28.0%) developed hypertension during a mean follow‐up of 3.32 years. The new‐onset hypertension was more observed in those with increased waist circumference (P for trend < .001). In the fully adjusted Cox regression, each 10 cm increase of waist circumference would result in an 18% elevated risk of hypertension. The random forest method and the Extreme Gradient Boosting method revealed waist circumference as an important feature to predict the development of hypertension. The sensitivity analysis indicated a consistent trend between waist circumference and new‐onset hypertension in all BMI categories. This study suggested high waist circumference as an independent risk factor for new‐onset hypertension based on a nationwide cohort of Chinese adults aged ≥45 years old. Our results supported that waist circumference should be routinely measured.

An Effective Fault Detection Method for HVAC Systems Using the LSTM-SVDD Algorithm

Fault detection in heating, ventilation and air-conditioning (HVAC) systems can effectively prevent equipment damage and system energy loss, and enhance the stability and reliability of system operation. However, existing fault detection strategies have not realized high effectiveness, mainly due to the time-delay characteristics of HVAC system faults and the lack of system-fault operation data. Therefore, aiming at the time delay of system faults and the lack of actual system-fault operation data, this paper proposes a fault detection method that combines a system simulation model and an intelligent detection algorithm. The method first uses the Modelica modeling language to build a scalable simulation model of the system to obtain fault data that are not easily accessible in practice. The long short-term memory-support vector data description (LSTM-SVDD) algorithm is then applied to detect faults in real time by dynamically adjusting the fault residuals according to the absolute difference between the predicted and actual values. The experimental results show that the LSTM-SVDD method improves the average detection accuracy by 9.675% and 9.85% over the classical LSTM network and the extreme gradient boosting (XGBoost) method, respectively, under different fault levels.

Effects of maintenance, traffic and climate condition on International Roughness Index of flexible pavement

ABSTRACT This paper focuses on developing the predictive models for International Roughness Index (IRI) of flexible pavement using several soft computing systems. About 564 sections with 4229 observations of the flexible pavement from the database of the long-term pavement performance (LTPP) program were used for developing and validating the prediction models. Equivalent single axle load (ESAL), structural number (SN), thickness, resilient modulus (MR) of the base layer, age, annual truck volume trend, total annual precipitation, freezing index, and maintenance treatments were the predictive attributes used in models. Various statistical criteria were used for evaluating the performance of the developed models, comprehensively. The results showed that the models developed in the current study are fairly promising tools for the prediction of IRI and capable of representing the complex relationship between the IRI and the predictive attributes selected. The models created through the Gradient Boosting Method (GBM) and Random Forest (RF), which were markedly outperformed the results of other methods, produced a correlation coefficient (R) greater than 0.90 and mean absolute error lower than 0.2. Furthermore, the SN was found to be the most important parameter for the IRI according to the variable importance score of the best performing prediction model.

Prediction of Mohr-Coulomb Constants of Selected Korean Rocks Based on Extreme Gradient Boosting Method and Its Evaluation

An accurate understanding of the physical and mechanical properties of rocks is of great importance in several rock mechanical engineering projects. Direct measurement is the most reliable method for obtaining the properties, however, it is sometimes restricted. In such cases, applying prediction models is a practical alternative. The Mohr-Coulomb failure criterion is one of the most frequently adopted models in rock mechanics and is represented by the cohesion and internal friction angle. However, despite their importance, prediction models for the two properties are relatively fewer than those of other properties. In this study, prediction models for the two properties were constructed based on a database collected from southern part of the Korean Peninsula. The extreme gradient boosting method was adopted to construct the models, and their performances were evaluated by comparing them with conventional regression models and artificial neural networks. Consequently, the extreme gradient boosting model exhibited the lowest error and highest prediction performance for both the Mohr-Coulomb constants. In addition, sensitivity analyses were performed to investigate the relative importance of the input variables. This paper aims to provide a database of several properties and to suggest prediction models for the Mohr-Coulomb constants so that it can be utilized as a reference in several rock engineering applications.

Filling gaps of black-sky surface albedo of the Arctic sea ice using gradient boosting and brightness temperature data

Surface albedo is a necessary parameter for climate studies and modeling. There is a need for a full spatial coverage of albedo data, but clouds and high solar zenith angle cause missing values to the optical satellite products, especially around the polar areas. Therefore, our motivation is to develop gap filling models. For that purpose, we will apply monthly gradient boosting (GB) based models to the Arctic sea ice area of the 34 years long albedo time series of the Satellite Application Facility on Climate Monitoring (CM SAF) project. We demonstrate the ability of the GB models to accurately fill missing data using albedo monthly mean, brightness temperature, and sea ice concentration as model inputs. Monthly GB models produce the most unbiased, precise, and robust estimates when compared to alternative estimates presented, such as monthly mean albedo values or estimates from monthly linear regression (LR) models. The mean relative differences between GB based estimates and original pentad values vary from −20% to 20% with RMSE being 0.048, compared to relative differences varying from −20% to over 60% with RMSE varying from 0.054 to 0.074 between other estimates and original pentad values. Pixelwise mean differences and standard deviations (std) over the whole Arctic sea ice area show that GB based estimates are more accurate (mean differences from −0.02 to 0.02) and more precise (std from 0.02 to 0.08) than other estimates (mean differences varying between −0.05 to over 0.05, and std varying from around 0.03 to over 0.1). Also, albedo of the melting sea ice is predicted better by the GB model, with negligible mean differences, compared to the LR model. Based on these results, we show that GB method is a useful technique to fill missing data, and the brightness temperature and sea ice concentration are useful additional model input data sources.

Multivariate monthly water demand prediction using ensemble and gradient boosting machine learning techniques

Water management planning requires reliable and accurate water demand forecasting. Water demand prediction is affected by variables, such as climate, socio-economic, and demographic data. This paper investigates urban monthly average water demand prediction, using classical, ensemble, and gradient boosting-based machine learning models, using the available monthly water demand, climatic, economic, and demographic data. Three train-test data split schemes on water demand timeseries were considered to determine the effect of data size on water demand prediction. Sensitivity analysis was employed to reduce input feature dimensionality while maintaining model accuracy. A univariate timeseries (water demand only) produced R2 scores up to 0.91, which increased to 0.94 with the addition of calendar and climatic features. Increasing the training data size from 70% to 90% improved the RMSE and MAE scores by ensemble and gradient boosting methods, with the random forest and the AdaBoost models showing improvements of up to 69%. The sensitivity analysis revealed a successful input reduction scheme from a potential 17 input attributes to seven inputs. Gradient boosting models showed robust and faster execution time, especially with the increase in training data, which is attractive for medium-term urban water demand forecasting.

Multiclassification Prediction of Clay Sensitivity Using Extreme Gradient Boosting Based on Imbalanced Dataset

Predicting clay sensitivity is important to geotechnical engineering design related to clay. Classification charts and field tests have been used to predict clay sensitivity. However, the imbalanced distribution of clay sensitivity is often neglected, and the predictive performance could be more accurate. The purpose of this study was to investigate the performance that extreme gradient boosting (XGboost) method had in predicting multiclass of clay sensitivity, and the ability that synthetic minority over-sampling technique (SMOTE) had in addressing imbalanced categories of clay sensitivity. Six clay parameters were used as the input parameters of XGBoost, and SMOTE was used to deal with imbalanced classes. Then, the dataset was divided using the cross-validation (CV) method. Finally, XGBoost, artificial neural network (ANN), and Naive Bayes (NB) were used to classify clay sensitivity. The F1 score, receiver operating characteristic (ROC), and area under the ROC curve (AUC) were considered as the performance indicators. The results revealed that XGBoost showed the best performance in the multiclassification prediction of clay sensitivity. The F1 score and mean AUC of XGBoost were 0.72 and 0.89, respectively. SMOTE was useful in addressing imbalanced issues, and XGBoost was an effective and reliable method of classifying clay sensitivity.