Ensemble Machine Learning Model Research Articles

Dust detection and susceptibility mapping by aiding satellite imagery time series and integration of ensemble machine learning with evolutionary algorithms

To mitigate the impact of dust on human health and the environment, it is crucial to create a model and map that identifies the areas susceptible to dust. The present study focused on identifying dust occurrences in the Bushehr province of Iran between 2002 and 2022 using moderate-resolution imaging spectroradiometer (MODIS) imagery. Subsequently, an ensemble machine learning model was improved to prepare a dust susceptibility map (DSM). The study employed differential evolution (DE), genetic algorithm (GA), and flower pollination algorithm (FPA) - three evolutionary algorithms - to enhance the random forest (RF) ensemble model. A spatial database was created for modeling, including 519 dust occurrence points (extracted from MODIS imagery) and 15 factors affecting dust (Slope, bulk density, aspect, clay, altitude, sand, rainfall, lithology, soil order, distance to river, soil texture, normalized difference vegetation index (NDVI), soil water content, land cover, and wind speed). By utilizing the differential evolution (DE) algorithm, we determined the significance of these factors in impacting dust occurrences. The results indicated that altitude, wind speed, and land cover were the most influential factors, while the distance to the river, bulk density, and soil texture had less impact on dust occurrence. Data were preprocessed using multicollinearity analysis and the frequency ratio (FR) approach. For this research, three RF-based meta-heuristic optimization algorithms, namely RF-FPA, RF-GA, and RF-DE, were created for DSM. The effectiveness prediction of the constructed models by indexes of root-mean-square-error (RMSE), the area under the receiver operating characteristic (AUC-ROC), and coefficient of determination (R2) from best to worst were RF-DE (RMSE = 0.131, AUC-ROC = 0.988, and R2 = 0.93), RF-GA (RMSE = 0.141, AUC-ROC = 0.986, and R2 = 0.919), RF-FPA (RMSE = 0.157, AUC-ROC = 0.981, and R2 = 0.9), and RF (RMSE = 0.173, AUC-ROC = 0.964, and R2 = 0.878). The results showed that combining evolutionary algorithms with an RF model improves the accuracy of dust susceptibility modeling.

Improving spatial predictions of Eucalypt plantation growth by combining interpretable machine learning with the 3-PG model

Accurate predictions of forest plantation growth provide forest managers with improved forest inventory estimates, forest valuation, and timely harvest schedules. Forest process-based models are increasingly used for quantifying current and potential productivity, yield gaps, and factors limiting growth, such as climate variability, soil characteristics, and water deficit. Improvements in the availability and resolution of spatial and temporal data combined with advancements in machine learning algorithms provide new opportunities to improve model predictions. This study shows how interpretable machine learning (ML) can be used to independently predict site soil fertility rating (FR) and incorporate these results into the 3-PG forest process-based model to accurately predict plantation growth. Four ensemble decision tree machine learning models—random forest trees, extremely randomized trees, gradient boost, and XG boost—were trained and compared using spatial cross-validation across the study area. FR predictions were estimated in relation to the influencing soil type and terrain characteristics, and interpretable ML methods were used to show how input feature permutations may relate to the soil fertility predictions. The results show the explanatory variables are similar across the selected ML models, with the strongest influencing variables being water leaching index, site aspect, and the silt and sand soil texture properties. The extremely randomized tree models showed the overall best performance, with only a small variation in performance across the four ML models. The method was applied to Eucalyptus nitens plantations covering over 63,000 ha in north-west Tasmania, Australia. The results using the predicted FR spatial grid with 3-PG show a strong correlation with observed growth for tree diameter and stand volume (R2 tree diameter at breast height = 0.97, RMSE = 0.85 m; R2 stand volume = 0.96, RMSE = 23.1 m3 ha−1) obtained from 161 permanent sample inventory plots ranging from 3 to 31 years old. This method has practical utility for other study sites to calibrate forest plantation soil fertility rating, in both the spatial and point-scale 3-PG model, where spatial data of soil characteristics are available. The derived soil fertility grid can provide valuable insights into the spatial variability of soil fertility in unknown areas.

A hybrid ensemble machine learning model for detecting APT attacks based on network behavior anomaly detection

SummaryA persistent, targeted cyber attack is called an advanced persistent threat (APT) attack. The attack is mainly launched to gain sensitive information, take over the system, and for financial gain, which creates nowadays more hurdles and challenges for the organization in preventing, detecting, and recovering from such attacks. Due to the nature of APT attacks, it is difficult to detect them quickly. Therefore machine learning techniques come into these research areas. This study uses deep and machine learning models such as random forest, decision tree, convolutional neural network, multilayer perceptron and so forth to categorize and effectively detect APT attacks by utilizing publicly accessible datasets. The datasets used in this study are CSE‐CIC‐IDS2018, CIC‐IDS2017, NSL‐KDD, and UNSW‐NB15. This study proposes the hybrid ensemble machine learning model, a mixed approach of random forest and XGBoost classifiers. It has obtained the maximum prediction accuracy of 98.92%, 99.91%, 99.24%, and 97.11% for datasets CSE‐CIC‐IDS2018, CIC‐IDS2017, NSL‐KDD, and UNSW‐NB15, with a false positive rate of 0.52%, 0.12%, 0.62%, and 5.29% respectively. These results are compared to other closely related recent studies in the literature. Our experiment's findings show that our model has performed significantly better for all datasets.

A hybrid of ensemble machine learning models with RFE and Boruta wrapper-based algorithms for flash flood susceptibility assessment

Flash floods are among the world most destructive natural disasters, and developing optimum hybrid Machine Learning (ML) models for flash flood susceptibility (FFS) modeling remains a challenge. This study proposed novel intelligence algorithms based on a hybrid of several ensemble ML models (i.e., Bagged Flexible Discriminant Analysis (BAFDA), Extreme Gradient Boosting (XBG), Rotation Forest (ROF) and Boosted Generalized Additive Model (BGAM)) and wrapper-based factor optimization algorithms (i.e., Recursive Feature Elimination (RFE) and Boruta) to improve the accuracy of FFS mapping at Neka-Haraz watershed in Iran. In addition, the Random Search (RS) method is proposed for meta-optimization of the developed hybrid models hyper-parameters. This study considers 20 flash flood conditioning factors (CgFs) and 380 flood and non-flood locations to create a geospatial database. The performance of each hybrid model was evaluated by area under the receiver operating characteristic (ROC) curve (AUC) and several validation methods, such as efficiency. The developed hybrid models demonstrated good performance, with BGAM-Boruta achieving the highest performance (AUC = 0.953, and Efficiency = 0.910), followed by ROF-Boruta (AUC = 0.952), ROF-RFE (AUC = 0.951), BAFDA-Boruta (AUC = 0.950), BGAM-RFE (AUC = 0.950), ROF (AUC = 0.949), BGAM (AUC = 0.948), BAFDA-RFE (AUC = 0.943), XGB-Boruta (AUC = 0.943), BAFDA (AUC = 0.939), XGB-RFE (AUC = 0.938) and XGB (AUC = 0.911). In the BGAM-Boruta model, the regional coverage was about 46% for high to very high FFS areas. Moreover, the study revealed that distance to river, slope, rainfall, altitude, and distance to road CgFs are the most significant for FFS modeling in this region.

An ensemble machine learning model to uncover potential sites of hazardous waste illegal dumping based on limited supervision experience

With the soaring generation of hazardous waste (HW) during industrialization and urbanization, HW illegal dumping continues to be an intractable global issue. Particularly in developing regions with lax regulations, it has become a major source of soil and groundwater contamination. One dominant challenge for HW illegal dumping supervision is the invisibility of dumping sites, which makes HW illegal dumping difficult to be found, thereby causing a long-term adverse impact on the environment. How to utilize the limited historic supervision records to screen the potential dumping sites in the whole region is a key challenge to be addressed. In this study, a novel machine learning model based on the positive-unlabeled (PU) learning algorithm was proposed to resolve this problem through the ensemble method which could iteratively mine the features of limited historic cases. Validation of the random forest-based PU model showed that the predicted top 30% of high-risk areas could cover 68.1% of newly reported cases in the studied region, indicating the reliability of the model prediction. This novel framework will also be promising in other environmental management scenarios to deal with numerous unknown samples based on limited prior experience.

Numerical and ensemble machine learning-based investigation of the energy and exergy yields of a concentrating photovoltaic thermal device equipped with a perforated twisted tube turbulator

The current research was carried out with the aim of numerically investigating the effect of employing a perforated twisted tube turbulator on the energy and exergy yields of a concentrating photovoltaic thermal (PVT) device. The simulations were performed in 4 different Reynolds numbers (Re) (i.e. 500, 1000, 1500 and 2000) and considering 4 different twist distance (TD) (i.e. L/25, L/50, L/70, and L/100) for the non-perforated turbulator and three different perforated turbulators (with 1, 2, and 3 holes) with TD = L/100. Among the examined cases, the best and worst performance belonged to the PVT device with perforated turbulator and without a turbulator, respectively. For the PVT device with non-perforated turbulator, the lowest PV panel temperature, the highest water outlet temperature, and the highest energy and exergy efficiencies occurred at the highest Re (i.e. 2000) and the lowest TD (i.e. L/100). Also, it was revealed that among the examined perforated turbulators, the best performance belongs to the turbulator with 3 holes in each pitch. In this case, the temperature of the PV panel, the overall energy efficiency and the overall exergy efficiency of the PVT device are respectively 3 ºC lower, 7.43% higher and 3.21% higher than the case without turbulator. As another novelty, a new ensemble machine learning model, namely boosted regression tree (BRT) was developed to simulation of the overall energy and exergy efficiencies based on the Reynolds number and volume fraction. The outcomes revealed the promising accuracy for both targets in terms various statistical metrics.

Interpretable ensemble machine learning framework to predict wear rate of modified ZA-27 alloy

This study investigates the impact of adding manganese (Mn) to ZA-27 alloy on microstructure and tribological properties. The Mn content varied from 0.2% to 1%. Volumetric wear rates were measured under different operating conditions. XRD and SEM were employed for phase identification and surface analysis. Ensemble Machine Learning (EML) regression models, including bagging, decision trees, random forest, ada boost, gradient boosting, and extreme gradient boost, were used to predict wear properties. Results indicate that the lowest wear rate occurred at 0.5% Mn content. Different wear mechanisms were observed for varying Mn contents. Among the EML models, extreme gradient boost showed superior performance with R2 values of 0.999 and 0.985 in training and testing, respectively.

Ensemble Learning for Breast Cancer Lesion Classification: A Pilot Validation Using Correlated Spectroscopic Imaging and Diffusion-Weighted Imaging.

The main objective of this work was to evaluate the application of individual and ensemble machine learning models to classify malignant and benign breast masses using features from two-dimensional (2D) correlated spectroscopy spectra extracted from five-dimensional echo-planar correlated spectroscopic imaging (5D EP-COSI) and diffusion-weighted imaging (DWI). Twenty-four different metabolite and lipid ratios with respect to diagonal fat peaks (1.4 ppm, 5.4 ppm) from 2D spectra, and water and fat peaks (4.7 ppm, 1.4 ppm) from one-dimensional non-water-suppressed (NWS) spectra were used as the features. Additionally, water fraction, fat fraction and water-to-fat ratios from NWS spectra and apparent diffusion coefficients (ADC) from DWI were included. The nine most important features were identified using recursive feature elimination, sequential forward selection and correlation analysis. XGBoost (AUC: 93.0%, Accuracy: 85.7%, F1-score: 88.9%, Precision: 88.2%, Sensitivity: 90.4%, Specificity: 84.6%) and GradientBoost (AUC: 94.3%, Accuracy: 89.3%, F1-score: 90.7%, Precision: 87.9%, Sensitivity: 94.2%, Specificity: 83.4%) were the best-performing models. Conventional biomarkers like choline, myo-Inositol, and glycine were statistically significant predictors. Key features contributing to the classification were ADC, 2D diagonal peaks at 0.9 ppm, 2.1 ppm, 3.5 ppm, and 5.4 ppm, cross peaks between 1.4 and 0.9 ppm, 4.3 and 4.1 ppm, 2.3 and 1.6 ppm, and the triglyceryl-fat cross peak. The results highlight the contribution of the 2D spectral peaks to the model, and they demonstrate the potential of 5D EP-COSI for early breast cancer detection.

Evaluating the accuracy and effectiveness of machine learning methods for rapidly determining the safety factor of road embankments

PurposeSlope stability analysis is essential for ensuring the safe design of road embankments. While various conventional methods, such as the finite element approach, are used to determine the safety factor of road embankments, there is ongoing interest in exploring the potential of machine learning techniques for this purpose.Design/methodology/approachWithin the study context, the outcomes of the ensemble machine learning models will be compared and benchmarked against the conventional techniques used to predict this parameter.FindingsGenerally, the study results have shown that the proposed machine learning models provide rapid and accurate estimates of the safety factor of road embankments and are, therefore, promising alternatives to traditional methods.Originality/valueAlthough machine learning algorithms hold promise for rapidly and accurately estimating the safety factor of road embankments, few studies have systematically compared their performance with traditional methods. To address this gap, this study introduces a novel approach using advanced ensemble machine learning techniques for efficient and precise estimation of the road embankment safety factor. Besides, the study comprehensively assesses the performance of these ensemble techniques, in contrast with established methods such as the finite element approach and empirical models, demonstrating their potential as robust and reliable alternatives in the realm of slope stability assessment.

Early prediction of medical students' performance in high-stakes examinations using machine learning approaches

IntroductionSince the advent of medical education systems, managing high-stakes exams has been a top priority and challenge for all policymakers. However, considering machine learning (ML) techniques as a replacement for medical licensing examinations, particularly during crises such as the COVID-19 outbreak, could be an effective solution. This study uses ML models to develop a framework for predicting medical students' performance on high-stakes exams, such as the Comprehensive Medical Basic Sciences Examination (CMBSE). Material and methodsPrediction of students' status and score on high-stakes examinations faces several challenges, including an imbalanced number of failing and passing students, a large number of heterogeneous and complex features, and the need to identify at-risk and top-performing students. In this study, two major categories of ML approaches are compared: first, classic models (logistic regression (LR), support vector machine (SVM), and k-nearest neighbors (KNN)), and second, ensemble models (voting, bagging (BG), random forests (RF), adaptive boosting (ADA), extreme gradient boosting (XGB), and stacking). ResultsTo evaluate the models' discrimination ability, they are assessed using a real dataset containing information on medical students over a five-year period (n = 1005). The findings indicate that ensemble ML models demonstrate optimal performance in predicting CMBSE status (RF and stacking). Similarly, among the classic regressors, LR exhibited the highest root-mean-square deviation (RMSD) (0.134) and coefficient of determination (R2) (0.62), whereas the RF model had the highest RMSD (0.077) and R2 (0.80) overall. Furthermore, Anatomical Sciences, Biochemistry, Parasitology, and Entomology grade point average (GPA) and grades demonstrated the strongest positive correlation with the outcomes. ConclusionComparing classic and ensemble ML models revealed that ensemble models are superior to classic models. Therefore, the presented framework could be considered a suitable alternative for the CMBSE and other comparable medical licensing examinations.

A practical framework for early detection of diabetes using ensemble machine learning models

A practical framework for early detection of diabetes using ensemble machine learning models

A Reinforcement Learning Approach for Ensemble Machine Learning Models in Peak Electricity Forecasting

Electricity peak load forecasting plays an important role in electricity generation capacity planning to ensure reliable power supplies. To achieve high forecast accuracy, multiple machine learning models have been implemented to forecast the monthly peak load in Thailand over the past few years, yielding promising results. One approach to further improve forecast accuracy is to effectively select the most accurate forecast value for each period from among the forecast values generated by these models. This article presents a novel reinforcement learning approach using the double deep Q-network (Double DQN), which acts as a model selector from a pool of available models. The monthly electricity peak load data of Thailand from 2004 to 2017 are used to demonstrate the effectiveness of the proposed method. A hyperparameter tuning methodology using a fractional factorial design is implemented to significantly reduce the number of required experimental runs. The results indicate that the proposed selection model using Double DQN outperforms all tested individual machine learning models in terms of mean square error.

Implications of rapid population growth on survey design and HIV estimates in the Rakai Community Cohort Study (RCCS), Uganda

ObjectiveSince rapid population growth challenges longitudinal population-based HIV cohorts in Africa to maintain coverage of their target populations, this study evaluated whether the exclusion of some residents due to growing...

Performance Comparison of Ensemble Machine Learning Models for Estimating Basketball Win-Loss Determinants: Focusing on the 2022 FIBA Men’s Basketball Asia Cup

Performance Comparison of Ensemble Machine Learning Models for Estimating Basketball Win-Loss Determinants: Focusing on the 2022 FIBA Men’s Basketball Asia Cup

Reducing False Negative Intrusions Rates of Ensemble Machine Learning Model based on Imbalanced Multiclass Datasets

In spite of the efforts to improve the efficiency of intrusion detection systems based on machine learning algorithms, these systems still need more. The false negative (FN) prediction outcome is of a major priority among other outcomes, when attacks are considered as normal by classifiers. FN outputs are highly a concern issue, especially for multiclass classification, where minor classes have less instances in imbalanced datasets. In this work, three types of well-known imbalanced multiclass classification datasets are used with ensemble machine learning classifiers. The datasets: KDD99, UNSW_NB15, and CICIDS2017 are balanced using different combination of oversampling and under-sampling techniques to improve false negative rates. Suitable performance metrics have been used to obtain significant outputs improvements in all three datasets types using Random Forest classifier. Achieved accuracies are 99.9852% for KDD99, 83.5451% for UNSW_NB15 and 99.8613% for CICIDS2017. The outcomes of the work using the mentioned datasets have been compared with state-of-the-art related works and the results show a clear improvement in false negative rates.

A stacking ensemble classifier-based machine learning model for classifying pollution sources on photovoltaic panels

Solar energy is a very efficient alternative for generating clean electric energy. However, pollution on the surface of solar panels reduces solar radiation, increases surface transmittance, and raises the surface temperature. All these factors cause photovoltaic (PV) panels to be less efficient. To address this problem, a stacking ensemble classifier-based machine learning model is proposed. In this study, different sources of pollution on each solar panel are used, and their power generation is recorded. The proposed model includes gradient boost, extra tree, and random forest classifiers, with the extra tree classifier serving as a meta-learner. The model takes into account various weather features during the training process, including irradiance and temperature, aiming to increase its accuracy and robustness in identifying pollution sources on the PV panel. Moreover, the proposed model is evaluated using various methods in order to examine performance metrics such as accuracy, F1 score, and precision. Results show that the model can achieve an accuracy score of 97.37%. The model’s performance is also compared to state-of-the-art machine learning models, demonstrating its superiority in accurately classifying pollution sources on PV panels. By utilizing different sources of pollution and weather features during training, the model can accurately classify different pollution sources, resulting in increased power generation efficiency and the longevity of PV panels. The main results of this study can be used to manage and maintain PV panels since the model can identify PV modules that need to be cleaned to keep producing the most power. Furthermore, the efficiency, reliability, and sustainability of PV panels can be further enhanced by the proposed model.

Biomes of the world under climate change scenarios: increasing aridity and higher temperatures lead to significant shifts in natural vegetation.

The global potential distribution of biomes (natural vegetation) was modelled using 8,959 training points from the BIOME 6000 dataset and a stack of 72 environmental covariates representing terrain and the current climatic conditions based on historical long term averages (1979-2013). An ensemble machine learning model based on stacked regularization was used, with multinomial logistic regression as the meta-learner and spatial blocking (100 km) to deal with spatial autocorrelation of the training points. Results of spatial cross-validation for the BIOME 6000 classes show an overall accuracy of 0.67 and R2logloss of 0.61, with "tropical evergreen broadleaf forest" being the class with highest gain in predictive performances (R2logloss = 0.74) and "prostrate dwarf shrub tundra" the class with the lowest (R2logloss = -0.09) compared to the baseline. Temperature-related covariates were the most important predictors, with the mean diurnal range (BIO2) being shared by all the base-learners (i.e.,random forest, gradient boosted trees and generalized linear models). The model was next used to predict the distribution of future biomes for the periods 2040-2060 and 2061-2080 under three climate change scenarios (RCP 2.6, 4.5 and 8.5). Comparisons of predictions for the three epochs (present, 2040-2060 and 2061-2080) show that increasing aridity and higher temperatures will likely result in significant shifts in natural vegetation in the tropical area (shifts from tropical forests to savannas up to 1.7 ×105 km2 by 2080) and around the Arctic Circle (shifts from tundra to boreal forests up to 2.4 ×105 km2 by 2080). Projected global maps at 1 km spatial resolution are provided as probability and hard classes maps for BIOME 6000 classes and as hard classes maps for the IUCN classes (six aggregated classes). Uncertainty maps (prediction error) are also provided and should be used for careful interpretation of the future projections.

Enhancing Hairfall Prediction: A Comparative Analysis of Individual Algorithms and An Ensemble Method

Hair fall, a prevalent issue affecting many individuals globally, necessitates early detection for preventive measures and hair health maintenance. Machine learning algorithms have gained attention in predicting hair fall by analysing genetic predisposition, lifestyle habits, and environmental factors. However, the performance of individual algorithms can be improved through ensemble models that combine their strengths. This research paper proposes an ensemble machine learning approach tailored for hair fall prediction. Comparative evaluations with individual algorithms reveal the ensemble models consistently outperform in accuracy, precision, and recall. Leveraging diverse algorithms, the ensemble approach captures a wider range of hair fall patterns, enhancing prediction accuracy. The ensemble models also exhibit higher precision and recall rates, correctly identifying both hair fall and non-hair fall instances. The ensemble models' superiority stems from mitigating the limitations of individual algorithms, resulting in a comprehensive and robust prediction framework. Overall, this research showcases the efficacy of ensemble machine learning models in hair fall prediction, enabling early detection and intervention for hair loss prevention. These findings provide valuable insights for researchers, practitioners, and individuals concerned about hair health.

Securing AMI-IoT networks against multiple RPL attacks using ensemble learning IDS and light-chain based prediction detection and mitigation mechanisms

ABSTRACT Advanced Metering Infrastructure (AMI) is one of the Internet of Things (IoT) enabled smart applications of smart grids. The Routing Protocol for Low Power and Lossy network (RPL) has been accepted to facilitate effective routing services for the AMI. However, numerous RPL attacks appear in AMI due to resource scarcity and dubious wireless medium, which significantly impedes the successful deployment of AMI-RPL. To enable secure and reliable AMI-RPL, this work proposes a novel Intrusion Detection System (IDS) named AMI Attack-aware Intelligent Machine learning IDS (AIMS). The primary objectives of AIMS are to predict, detect, and mitigate different types of RPL security attacks in the AMI environment. To predict the RPL attacks using the Stacked Ensemble (SE) machine learning model, a novel AMI-RPL Attack Dataset (ARAD) is generated by the Cooja simulator with the suitable pre-processing and the Spider Monkey Optimization (SMO) based feature selection. The advanced prediction of attack nodes improves the performance and significantly diminishes the future damages of AMI. The attack detection is based on immutable blocks of a light-chain model, and the cryptocurrency-based mitigation model effectively isolates the attackers. AIMS mechanism amplifies RPL security with high reliability and maximizes the AMI network lifetime by delivering superior results.

Brain Tumor Classification based on Improved Stacked Ensemble Deep Learning Methods.

Brain Tumor diagnostic prediction is essential for assisting radiologists and other healthcare professionals in identifying and classifying brain tumors. For the diagnosis and treatment of cancer diseases, prediction and classification accuracy are crucial. The aim of this study was to improve ensemble deep learning models for classifing brain tumor and increase the performance of structure models by combining different model of deep learning to develop a model with more accurate predictions than the individual models. Convolutional neural networks (CNNs), which are made up of a single algorithm called CNN model, are the foundation of most current methods for classifying cancer illness images. The model CNN is combined with other models to create other methods of classification called ensemble method. However, compared to a single machine learning algorithm, ensemble machine learning models are more accurate. This study used stacked ensemble deep learning technology. The data set used in this study was obtained from Kaggle and included two categories: abnormal & normal brains. The data set was trained with three models: VGG19, Inception v3, and Resnet 10. The 96.6% accuracy for binary classification (0,1) have been achieved by stacked ensemble deep learning model with Loss binary cross entropy, and Adam optimizer take into consideration with stacking models. The stacked ensemble deep learning model can be improved over a single framework.