Ensemble Machine Learning Model Research Articles

Enhanced Bearing Fault Diagnosis Through Trees Ensemble Method and Feature Importance Analysis

Abstract Purpose This research introduces a groundbreaking method for bearing defect detection. It leverages ensemble machine learning (ML) models and conducts comprehensive feature importance analysis. The key innovation is the training and benchmarking of three tree ensemble models—Decision Tree (DT), Random Forest (RF), and Extreme Gradient Boosting (XGBoost)—on an extensive experimental dataset (QU-DMBF) collected from bearing tests with seeded defects of varying sizes on the inner and outer raceways under different operating conditions. Method The dataset was meticulously prepared with categorical variable encoding and Min–Max data normalization to ensure consistent class distribution and model accuracy. Implementing the ML models involved a grid search method for hyperparameter tuning, focusing on reporting the models’ accuracy. The study also explores applying ensemble methods and using supervised and unsupervised learning algorithms for bearing fault detection. It underscores the value of feature importance analysis in understanding the contributions of specific inputs to the model’s performance. The research compares the ML models to traditional methods and discusses their potential for advanced fault diagnosis in bearing systems. Results and Conclusions The XGBoost model, trained on data from actual bearing tests, outperformed the others, achieving 92% accuracy in detecting bearing health and fault location. However, a deeper analysis of feature importance reveals that the models weigh certain experimental conditions differently—such as sensor location and motor speed. This research’s primary novelties and contributions are comparative evaluation, experimental validation, accuracy benchmarking, and interpretable feature importance analysis. This comprehensive methodology advances the bearing health monitoring field and has significant practical implications for condition-based maintenance, potentially leading to substantial cost savings and improved operational efficiency.

An ensemble machine learning model generates a focused screening library for the identification of CDK8 inhibitors.

The identification of an effective inhibitor is an important starting step in drug development. Unfortunately, many issues such as the characterization of protein binding sites, the screening library, materials for assays, etc., make drug screening a difficult proposition. As the size of screening libraries increases, more resources will be inefficiently consumed. Thus, new strategies are needed to preprocess and focus a screening library towards a targeted protein. Herein, we report an ensemble machine learning (ML) model to generate a CDK8-focused screening library. The ensemble model consists of six different algorithms optimized for CDK8 inhibitor classification. The models were trained using a CDK8-specific fragment library along with molecules containing CDK8 activity. The optimized ensemble model processed a commercial library containing 1.6 million molecules. This resulted in a CDK8-focused screening library containing 1,672 molecules, a reduction of more than 99.90%. The CDK8-focused library was then subjected to molecular docking, and 25 candidate compounds were selected. Enzymatic assays confirmed six CDK8 inhibitors, with one compound producing an IC50 value of ≤100 nM. Analysis of the ensemble ML model reveals the role of the CDK8 fragment library during training. Structural analysis of molecules reveals the hit compounds to be structurally novel CDK8 inhibitors. Together, the results highlight a pipeline for curating a focused library for a specific protein target, such as CDK8.

Reconstructing long-term natural flows by ensemble machine learning

Large-scale human activities have caused significant disruption to the natural flow regime of rivers worldwide, leading to substantial deviations from river flows’ natural state. The natural hydrological regime of rivers holds great significance for the conservation and management of riverine ecosystems, which relies on the accurate reconstruction of natural river flows. However, the application of existing high spatiotemporal resolution large-scale natural flow datasets at the regional scale has not been adequately assessed, thereby limiting their regional-scale utility. In this study, utilizing global reanalysis datasets and observation data, in combination with ensemble machine learning models, we constructed long-term natural flow time series for inflows into the Pearl River Delta and evaluated the performance of these reconstructed inflows based on observed flow records during undisturbed periods. The reconstruction shows that the ensemble model, the AutoGluon model, indicates the best performance when compared with other machine learning models and outperforms the existing publicly available large-scale datasets in terms of evaluation metrics for the corresponding natural flows. The results of Indicator of Hydrologic Alteration and Range of Variability Analysis (IHA-RVA) indicate that climate variability, reservoir regulation and human activities have significantly altered the natural flow regime of the Pearl River Delta. Specifically, human activities generally tend to smooth natural flow variability in the Pearl River Delta. This research is expected to provide an efficient and reliable method for reconstructing natural flows and offers guidance for water resource management and ecosystem conservation.

Cooling load prediction of a double-story terrace house using ensemble learning techniques and genetic programming with SHAP approach

Since the cooling systems used in buildings in hot climates account for a significant portion of the energy consumption, it is very important for both economy and environment to accurately predict the cooling load and consider it in building designs. This study aimed to maximize energy efficiency by appropriately selecting the features of a building that affect its cooling load. To this end, data-driven, accurate, and accessible tools were developed that enable the prediction of the cooling load of a building by practitioners. The study involves simulating the energy consumption of a mid-rise, double-story terrace house in Malaysia using building information modeling (BIM) and estimating the cooling load using ensemble machine learning models and genetic programming. Categorical Boosting (CatBoost), eXtreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), and Random Forest (RF) models have been developed and made available as an online interactive graphical user interface on the Streamlit platform. Furthermore, the symbolic regression technique has been utilized to obtain a closed-form equation that predicts the cooling load. The dataset used for training the predictive models comprised 94,310 data points with 10 input variables and the cooling load as the output variable. Performance metrics such as the coefficient of determination (R2), root mean squared error (RMSE), and mean absolute error (MAE) were used to measure the predictive model performances. The results of the machine learning models indicated successful prediction, with the CatBoost model achieving the highest score (R2 = 0.9990) among the four ensemble models and the predictive equation. The SHAP analysis determined the aspect ratio of the building as the most impactful feature of the building.

Urban flood susceptibility mapping using remote sensing, social sensing and an ensemble machine learning model

Flood susceptibility mapping is crucial for urban disaster management. However, the heterogeneity of urban land use and the complexity of terrain pose challenges to the accuracy and generalizability of flood models. In this paper, we propose a framework named the EMF model for urban flood mapping. Specifically, the social sensing and remote sensing were used for flooding information collection. The XGBoost, Support Vector Classifier (SVC), Multilayer Perceptron (MLP), and Multimodal Deep Learning (MDL), were used as the predictive models, and their results were ensembled using a Random Forest model to produce the final outcome. Results show that the EMF model outperforms the standalone models, with an accuracy of 0.942 on the training set and 0.940 on the testing set. The accuracy of the five models is ranked as EMF > MDL > XGBoost > SVC > MLP. The flood map indicates that flooding has a greater impact on the outskirts and suburbs of the city compared to its central urban areas. Farmland is the most affected land type, making up 54.8 % of the flooded area. Overall, the proposed framework enables rapid and accurate identification of flood-prone areas, providing technical support for managers in formulating effective flood prevention strategies.

A hybrid quantum ensemble learning model for malicious code detection

Quantum computing as a new computing model with parallel computing capability and high information carrying capacity, has attracted a lot of attention from researchers. Ensemble learning is an effective strategy often used in machine learning to improve the performance of weak classifiers. Currently, the classification performance of quantum classifiers is not satisfactory enough due to factors such as the depth of quantum circuit, quantum noise, and quantum coding method, etc. For this reason, this paper combined the ensemble learning idea and quantum classifiers to design a novel hybrid quantum machine learning model. Firstly, we run the Stacking method in classical machine learning to realize the dimensionality reduction of high-latitude data while ensuring the validity of data features. Secondly, we used the Bagging method and Bayesian hyperparameter optimization method applied to quantum support vector machine (QSVM), quantum K nearest neighbors (QKNN), variational quantum classifier (VQC). Thirdly, the voting method is used to ensemble the predict results of QSVM, QKNN, VQC as the final result. We applied the hybrid quantum ensemble machine learning model to malicious code detection. The experimental results show that the classification precision (accuracy, F1-score) of this model has been improved to 98.9% (94.5%, 94.24%). Combined with the acceleration of quantum computing and the higher precision rate, it can effectively deal with the growing trend of malicious codes, which is of great significance to cyberspace security.

Ensemble Learning Approach for Developing Performance Models of Flexible Pavement

This research utilizes the Long-Term Pavement Performance database, focusing on devel-oping a predictive model for flexible pavement performance in the Southern United States. Analyzing 367 pavement sections, this study investigates crucial factors influencing asphaltic concrete (AC) pavement deterioration, such as structural and material components, air voids, compaction density, temperature at laydown, traffic load, precipitation, and freeze–thaw cycles. The objective of this study is to develop a predictive machine learning model for AC pavement wheel path cracking (WpCrAr) and the age at which cracking initiates (WpCrAr) as performance indicators. This study thoroughly investigated three ensemble machine learning models, including random forest, extremely randomized trees (ETR), and extreme gradient boosting (XGBoost). It was observed that XGBoost, optimized using Bayesian methods, emerged as the most effective among the evaluated models, demonstrating good predictive accuracy, with an R2 of 0.79 for WpCrAr and 0.92 for AgeCrack and mean absolute errors of 1.07 and 0.74, respectively. The most important features influencing crack initiation and progression were identified, including equivalent single axle load (ESAL), pavement age, number of layers, precipitation, and freeze–thaw cycles. This paper also showed the impact of pavement material combinations for base and subgrade layers on the delay of crack initiation.

Ensemble machine learning model for forecasting wind farm generation

Ensemble machine learning model for forecasting wind farm generation

NOx Emission Predictions in Gas Turbines Through Integrated Data-Driven Machine Learning Approaches

Abstract The reduction of NOx emissions is a paramount endeavor in contemporary engineering and energy production, as these emissions are closely linked to adverse environmental and health impacts. The prediction of NOx emission from gas turbines through several integrated data-driven machine learning methods has been evaluated in study. The study compares the performance of ensemble and conventional machine learning models, demonstrating superior accuracy achieved by the ensemble models. Specifically, the Random Forest model achieved an accuracy rate of 91.68%, XGBoost yielded an accuracy of 91.54%, and CATBoost exhibited the highest accuracy at 92.76%. These findings highlight the capability of data-driven machine learning techniques in enhancing NOx emission predictions in gas turbines. The improved prediction by ensembles can be utilized in the development and implementation of more effective control and mitigation strategies in practical applications. Through the application of these advanced machine learning approaches, the gas turbine industry can play a pivotal role in minimizing its environmental impact while optimizing operational efficiency. This study also provides valuable insights into the effectiveness of ensemble machine learning models, advancing our understanding of their capabilities in addressing the critical issue of NOx emissions from gas turbines.

Requirement Dependency Extraction Based on Improved Stacking Ensemble Machine Learning

To address the cost and efficiency issues of manually analysing requirement dependency in requirements engineering, a requirement dependency extraction method based on part-of-speech features and an improved stacking ensemble learning model (P-Stacking) is proposed. Firstly, to overcome the problem of singularity in the feature extraction process, this paper integrates part-of-speech features, TF-IDF features, and Word2Vec features during the feature selection stage. The particle swarm optimization algorithm is used to allocate weights to part-of-speech tags, which enhances the significance of crucial information in requirement texts. Secondly, to overcome the performance limitations of standalone machine learning models, an improved stacking model is proposed. The Low Correlation Algorithm and Grid Search Algorithms are utilized in P-stacking to automatically select the optimal combination of the base models, which reduces manual intervention and improves prediction performance. The experimental results show that compared with the method based on TF-IDF features, the highest F1 scores of a standalone machine learning model in the three datasets were improved by 3.89%, 10.68%, and 21.4%, respectively, after integrating part-of-speech features and Word2Vec features. Compared with the method based on a standalone machine learning model, the improved stacking ensemble machine learning model improved F1 scores by 2.29%, 5.18%, and 7.47% in the testing and evaluation of three datasets, respectively.

Performance investigation of osmotically assisted reverse osmosis using explainable machine learning models: A comparative study

Osmotically assisted reverse osmosis (OARO) can treat highly concentrated water and achieve higher recovery rates than reverse osmosis (RO). However, existing mathematical models cannot satisfactorily explain OARO systems due to certain limitations. To explore OARO more dynamically, we aimed to analyze OARO performance based on the laboratory-scale data acquired under different operating conditions, by implementing six machine learning (ML) models with the OARO data, evaluating their predictive capabilities, and investigating the effects of input variables on output variables using Shapley Additive Explanation (SHAP) analysis. In predicting the OARO outputs, the ensemble ML models outperformed the conventional ML models. Incorporating FO/RO membrane datasets significantly enhanced the ML model accuracy and mitigated overfitting issues. Regarding the prediction of water flux, the adjusted determination coefficient values increased by 0.478 for Random Forest and 0.569 for Extreme Gradient Boost in the test step. The SHAP analysis consistently ranked the importance of the input variables for OARO outputs. It revealed the presence of potentially influential but previously unrecognized input variables, such as temperature, for the draw concentration. The results of this study are expected to provide a better understanding of OARO.

Reliable arrival time picking of acoustic emission using ensemble machine learning models

This study presents an innovative method for accurately picking the first-wave arrival time in acoustic emission (AE) localization, particularly effective in environments with low or variable signal-to-noise ratios (SNR). Utilizing an ensemble learning model, it synergizes multiple automatic arrival time estimation algorithms to enhance both consistency and robustness. The model, rooted in decision tree methodologies, integrates a variety of techniques, including Akaike information criterion (AIC), improved AIC, Hinkley, energy ratios, and wavelet transformation-based binary map. Its efficacy is demonstrated through testing on 549 manually annotated AE datasets, where the model’s predictions were benchmarked against manual picks, showcasing superior accuracy and robustness in AE event source localization. Evaluations of the model’s performance across various ensemble machine learning models highlighted its ability to significantly diminish localization errors, achieving an average absolute error of less than 1.5 mm. The study also delved into the impact of base pickers and the size of the training dataset on the model’s predictions. Findings indicated consistent performance across different decision tree models, with the accuracy of base pickers and training set size playing a significant role in outcomes. This research culminates in a decision tree-based ensemble machine learning solution that effectively estimates AE first-wave arrival times with high accuracy and robustness, even amidst fluctuating SNR conditions. Its adaptability and interpretability greatly reduce source localization errors in practical AE monitoring, effectively overcoming challenges associated with imprecise arrival time picking.

Enhancing Breast Cancer Detection with Ensemble Machine Learning Models

Breast cancer remains a significant public health issue worldwide, underlining the need for accurate and efficient diagnostic methods. In this paper, we propose a new technique to enhance breast cancer diagnosis through the integration of multiple machine-learning models. Our strategy employs a combination of the Naive Bayes classifier, Stochastic Gradient Descent (SGD), Bagging, and the ZeroR classifier, alongside Bayes Network learning. The cornerstone of our approach is Bayes Network learning, a probabilistic graphical model designed to map out the intricate interconnections among various diagnostic factors. This is significant in the way that it can help to uncover complex relationships in the data for the sake of leading to more accurate predictions. Added to the above, we use the Naïve Bayes classifier, a classifier showing good validity in classification tasks and based on probabilistic reasoning, for the screening of breast cancer. Further, a refined model's parameter is included using the SGD and leads to enhancement of the generalization and overall performance of the model. In addition, as part of controlling overfitting, one can also use Bagging. It is an ensemble method in the sense that it considers several models. ZeroR classifier is a very basic classifier and is just used to compare its performance with our composite approach. We are comparing complex ensemble results to its simplicity. We will validate the ability of our proposed methodology to compare the performance of our integrated models against ZeroR.

Prediction of soil thermal conductivity using individual and ensemble machine learning models

Prediction of soil thermal conductivity using individual and ensemble machine learning models

A comparative study of ensemble machine learning models for compressive strength prediction in recycled aggregate concrete and parametric analysis

A comparative study of ensemble machine learning models for compressive strength prediction in recycled aggregate concrete and parametric analysis

An ultrasound-based ensemble machine learning model for the preoperative classification of pleomorphic adenoma and Warthin tumor in the parotid gland.

The preoperative classification of pleomorphic adenomas (PMA) and Warthin tumors (WT) in the parotid gland plays an essential role in determining therapeutic strategies. This study aims to develop and validate an ultrasound-based ensemble machine learning (USEML) model, employing nonradiative and noninvasive features to differentiate PMA from WT. A total of 203 patients with histologically confirmed PMA or WT who underwent parotidectomy from two centers were enrolled. Clinical factors, ultrasound (US) features, and radiomic features were extracted to develop three types of machine learning model: clinical models, US models, and USEML models. The diagnostic performance of the USEML model, as well as that of physicians based on experience, was evaluated and validated using receiver operating characteristic (ROC) curves in internal and external validation cohorts. DeLong's test was used for comparisons of AUCs. SHAP values were also utilized to explain the classification model. The USEML model achieved the highest AUC of 0.891 (95% CI, 0.774-0.961), surpassing the AUCs of both the US (0.847; 95% CI, 0.720-0.932) and clinical (0.814; 95% CI, 0.682-0.908) models. The USEML model also outperformed physicians in both internal and external validation datasets (both p < 0.05). The sensitivity, specificity, negative predictive value, and positive predictive value of the USEML model and physician experience were 89.3%/75.0%, 87.5%/54.2%, 87.5%/65.6%, and 89.3%/65.0%, respectively. The USEML model, incorporating clinical factors, ultrasound factors, and radiomic features, demonstrated efficient performance in distinguishing PMA from WT in the parotid gland. This study developed a machine learning model for preoperative diagnosis of pleomorphic adenoma and Warthin tumor in the parotid gland based on clinical, ultrasound, and radiomic features. Furthermore, it outperformed physicians in an external validation dataset, indicating its potential for clinical application. • Differentiating pleomorphic adenoma (PMA) and Warthin tumor (WT) affects management decisions and is currently done by invasive biopsy. • Integration of US-radiomic, clinical, and ultrasound findings in a machine learning model results in improved diagnostic accuracy. • The ultrasound-based ensemble machine learning (USEML) model consistently outperforms physicians, suggesting its potential applicability in clinical settings.

Comparative study of machine learning models for predicting the compressive strength of concrete using Non-Destructive Testing methods

This study proposes and compares the conventional machine learning (CML) models as artificial neural networks (ANN), support vector machines (SVM) and ensemble machine learning (EML) models namely, random forest (RF), and gradient boosting (GB). These four-machine learning (ML) models were used to predict the compressive strength (CS) of concrete based on two non-destructive testing (NDT) methods, namely ultrasonic pulse velocity (UPV) and rebound hammer number (RHN). The article uses a large and reliable database of experimental data, collected from the literature to train and test the models, and evaluates their performance using various metrics, scatter plots and a Taylor diagram. It was found that gradient boosting (GB) is the most accurate and robust model for this study, followed by RF model, while ANN and SVM have lower performance. For this study, it suggests that EML models are more suitable and reliable than CML models for predicting the CS of concrete based on NDT methods.

Comparative analysis of conventional and ensemble machine learning models for predicting split tensile strength in thermal stressed SCM-blended lightweight concrete

This study aimedto predict the split tensile strength (STS) of lightweight concrete having silica fume (SF) as supplementary cementitious material (SCM) under thermal stresses. Machine learning (ML) models including a conventional machine learning (CML) model as Artificial Neural Networks (ANN) and ensemble machine learning (EML) models, particularly Random Forest (RF) and Gradient Boosting (GB) were developed to predict the STS. Dataset comprising of 96 instances was splitted into 80% for training and 20% for testing phase respectively. With the help of the 80% training data and subsequent hyperparameter tuning, various CML and EML models were developed. ML models were analysed statistically for both the training and testing phases, under the model performanceparameters, line diagrams, and Taylor diagrams. After the thorough statistical analysis, it was concluded that the GB as an EML model outperformed all other ML models. This was demonstrated by the fact that it had constantly higher model performance parameter values of RMSE, MAE and R2 as 0.151, 0.115 and 0.977 for training and 0.23, 0.196 and 0.908 for testing respectively. Also, from the line diagrams which shows a good correlation with the actual and predicted STS values and from the Taylor diagram of the various ML models for both the phases, it was concluded that the GB model is found superior model for this specific study.

Ensemble Machine Learning for Predicting 90-Day Outcomes and Analyzing Risk Factors in Acute Kidney Injury Requiring Dialysis.

Our objectives were to (1) employ ensemble machine learning algorithms utilizing real-world clinical data to predict 90-day prognosis, including dialysis dependence and mortality, following the first hospitalized dialysis and (2) identify the significant factors associated with overall outcomes. We identified hospitalized patients with Acute kidney injury requiring dialysis (AKI-D) from a dataset of the Taipei Medical University Clinical Research Database (TMUCRD) from January 2008 to December 2020. The extracted data comprise demographics, comorbidities, medications, and laboratory parameters. Ensemble machine learning models were developed utilizing real-world clinical data through the Google Cloud Platform. The Study Analyzed 1080 Patients in the Dialysis-Dependent Module, Out of Which 616 Received Regular Dialysis After 90 Days. Our Ensemble Model, Consisting of 25 Feedforward Neural Network Models, Demonstrated the Best Performance with an Auroc of 0.846. We Identified the Baseline Creatinine Value, Assessed at Least 90 Days Before the Initial Dialysis, as the Most Crucial Factor. We selected 2358 patients, 984 of whom were deceased after 90 days, for the survival module. The ensemble model, comprising 15 feedforward neural network models and 10 gradient-boosted decision tree models, achieved superior performance with an AUROC of 0.865. The pre-dialysis creatinine value, tested within 90 days prior to the initial dialysis, was identified as the most significant factor. Ensemble machine learning models outperform logistic regression models in predicting outcomes of AKI-D, compared to existing literature. Our study, which includes a large sample size from three different hospitals, supports the significance of the creatinine value tested before the first hospitalized dialysis in determining overall prognosis. Healthcare providers could benefit from utilizing our validated prediction model to improve clinical decision-making and enhance patient care for the high-risk population.

Assessing the Efficiency of Foreign Investment in a Certification Procedure Using an Ensemble Machine Learning Model

Many methods exist for solving the problem of evaluating efficiency in different processes. They are divided into two basic groups, parametric and non-parametric methods, which can have significant differences in the results. In this study, the authors consider the process of assessing the business climate depending on realized foreign investments. Due to the expected difference in efficiency assessment using different approaches, the goal of this paper is to create an optimization model of an ensemble for efficiency assessment that uses both types of methods with the aim of creating a symmetrical approach that achieves better results than each type of method individually. The proposed solution simultaneously analyzes the impact of different factors on foreign investments in order to determine the most important factors and thus enable each local government to ensure the best possible efficiency in this process. The innovative idea of this study is in the inclusion of classification and feature selection methods of machine learning to fulfill the set goal. Our research, focused on a specific case study in various cities across the Republic of Serbia, evaluated the effectiveness of that process. This study extends previous research and confirms the published results, highlighting the advantages of the newly proposed model.