Heterogeneous Ensemble Learning Research Articles

Research on User Default Prediction Algorithm Based on Adjusted Homogenous and Heterogeneous Ensemble Learning

In the field of risk assessment, the traditional econometric models are generally used to assess credit risk. And with the introduction of the “dual-carbon” goals to promote the development of a low-carbon economy, the scale of green credit in China has rapidly expanded. But with the advent of the big data era, due to the poor interpretability of a traditional single machine learning model, it is difficult to capture nonlinear relationships, and there are shortcomings in prediction accuracy and robustness. This paper selects the adjusted ensemble learning model based on the homogeneous and heterogeneous factors for user default prediction, which can efficiently process large quantities of high-dimensional data. This article adjusts each model to adapt to the task and innovatively compares various models. In this paper, the missing value filling method, feature selection, and ensemble model are studied and discussed, and the optimal ensemble model is obtained. When comparing the predictions of single models and ensemble models, the accuracy, sensitivity, specificity, F1-Score, Kappa, and MCC of Categorical Features Gradient Boosting (CatBoost) and Random undersampling Boosting (RUSBoost) all reach 100%. The experimental results prove that the algorithm based on adjusted homogeneous and heterogeneous ensemble learning can predict the user default efficiently and accurately. This paper also provides some references for establishing a risk assessment index system.

Feature group partitioning: an approach for depression severity prediction with class balancing using machine learning algorithms

In contemporary society, depression has emerged as a prominent mental disorder that exhibits exponential growth and exerts a substantial influence on premature mortality. Although numerous research applied machine learning methods to forecast signs of depression. Nevertheless, only a limited number of research have taken into account the severity level as a multiclass variable. Besides, maintaining the equality of data distribution among all the classes rarely happens in practical communities. So, the inevitable class imbalance for multiple variables is considered a substantial challenge in this domain. Furthermore, this research emphasizes the significance of addressing class imbalance issues in the context of multiple classes. We introduced a new approach Feature group partitioning (FGP) in the data preprocessing phase which effectively reduces the dimensionality of features to a minimum. This study utilized synthetic oversampling techniques, specifically Synthetic Minority Over-sampling Technique (SMOTE) and Adaptive Synthetic (ADASYN), for class balancing. The dataset used in this research was collected from university students by administering the Burn Depression Checklist (BDC). For methodological modifications, we implemented heterogeneous ensemble learning stacking, homogeneous ensemble bagging, and five distinct supervised machine learning algorithms. The issue of overfitting was mitigated by evaluating the accuracy of the training, validation, and testing datasets. To justify the effectiveness of the prediction models, balanced accuracy, sensitivity, specificity, precision, and f1-score indices are used. Overall, comprehensive analysis demonstrates the discrimination between the Conventional Depression Screening (CDS) and FGP approach. In summary, the results show that the stacking classifier for FGP with SMOTE approach yields the highest balanced accuracy, with a rate of 92.81%. The empirical evidence has demonstrated that the FGP approach, when combined with the SMOTE, able to produce better performance in predicting the severity of depression. Most importantly the optimization of the training time of the FGP approach for all of the classifiers is a significant achievement of this research.

An ensemble spatial prediction method considering geospatial heterogeneity

Ensemble learning synthesizes the advantages of different models and has been widely applied in the field of spatial prediction. However, the nonlinear constraints of spatial heterogeneity on the model ensemble process make it difficult to adaptively determine the ensemble weights, greatly limiting the predictive ability of the ensemble learning model. This paper therefore proposes a novel geographical spatial heterogeneous ensemble learning method (GSH-EL). Firstly, the geographically weighted regression model, geographically optimal similarity model, and random forest model are used as three base learners to express local spatial heterogeneity, global feature correlation, and nonlinear relationship of geographic elements, respectively. Then, a spatially weighted ensemble neural network module (SWENN) of GSH-EL is proposed to express spatial heterogeneity by exploring the complex nonlinear relationship between the spatial proximity and ensemble weights. Finally, the outputs of the three base learners are combined with the spatial heterogeneous ensemble weights from SWENN to obtain the spatial prediction results. The proposed method is validated on the PM2.5 air quality and landslide dataset in China, both of which obtain more accurate prediction results than the existing ensemble learning strategies. The results confirm the need to accurately express spatial heterogeneity in the model ensemble process.

A fault diagnosis framework based on heterogeneous ensemble learning for air conditioning chiller with unbalanced samples

Big data-based air conditioning fault diagnosis research has developed rapidly in recent years, but in actual engineering, the fault sample size of air conditioning systems is much smaller than the normal sample size, and the resulting sample imbalance problem makes conventional data-driven diagnostic methods based on low accuracy and poor stability. In order to solve the problem of unbalanced fault diagnosis of air-conditioning chillers, this paper proposes an integrated learning-based diagnostic model, which achieves diagnosis by combining multiple base models and by majority voting. The method uses four classification models, namely, random forest model, decision tree model, k nearest neighbor model, and isomorphic integration model, as base classifiers, and synthesizes the four base classifiers into a heterogeneous integration algorithmic model (IMV) through integrated learning, and performs diagnostic detection of seven types of typical faults of chiller units using the majority voting method of integrated learning. The effectiveness of the proposed algorithm is verified on the RP-1043 dataset, and the experimental results show that the accuracy of the heterogeneous integrated algorithm model (IMV) can reach 96.87%, which is a significant improvement compared with the accuracy of the other four base classifier models (81.04%–96.25%). Therefore, the integrated learning model has some application prospects in fault diagnosis when targeting unbalanced datasets.

An Adaptive Heterogeneous Ensemble Learning Model for Credit Card Fraud Detection

An Adaptive Heterogeneous Ensemble Learning Model for Credit Card Fraud Detection

Landslide susceptibility assessment based on frequency ratio and semi-supervised heterogeneous ensemble learning model.

Epistemic uncertainty in data-driven landslide susceptibility assessment often tends to be increased by the limited accuracy of an individual model, as well as uncertainties associated with the selection of non-landslide samples. To address these issues, this paper centers on the landslide disaster in Ji'an City, China, and proposes a heterogeneous ensemble learning method incorporating frequency ratio (FR) and semi-supervised sample expansion. Based on the superimposed results of 12 environmental factor frequency ratios (FFR), non-landslide samples were selected and input into light gradient boosting machine (LightGBM), random forest (RF), and convolutional neural network (CNN) models for prediction along with historical landslide samples. The predicted probability values are integrated by four heterogeneous ensemble strategies to expand samples from high-confidence results. The model's performance is evaluated using the area under the receiver operating characteristic curve (AUC), partition frequency ratio (PFR), and other verification methods. The results demonstrate that the negative sample based on FFR sampling is more accurate than the random sampling method, and the FR-SSELR model based on frequency ratio sampling and semi-supervised ensemble strategy exhibits the highest performance (AUC = 0.971, ACC = 0.941). A more reasonable landslide susceptibility map was drawn based on this model, with the lowest percentage of landslides in the low and very low susceptibility zones (sum of PFR = 0.194), as well as the highest percentage of landslides in the high and very high susceptibility zones (sum of PFR = 6.800). Furthermore, the FR-SSELR model improved economic benefits by 3.82-14.2%, offering valuable guidance for decision-making regarding landslide management and the sustainability of Ji'an City.

Prediction of rhinitis with class imbalance based on heterogeneous ensemble learning

Common clinical rhinitis is characterized by different types of cases and class imbalance. Its prediction belongs to multiple output classification. Low recognition rate and poor generalization performance often occur for minority class. Therefore, we propose a novel integrated classification model, ARF-OOBEE, which transforms the multi-output classification to multi-label classification and multi-class classification. The multi-label classifier automatically adjusts the number and depth of integrated forest learners according to the imbalance ratio of single class label in a subset. It can effectively reduce the impact of class imbalance on classification and improve prediction performance of both majority or minority class concurrently. Also, we build a multi-class classification based on out-of-bag Extra-Tree to accomplish finer classification for the predicted labels. In addition, we calculate the feature importance for rhinitis on the grounds of the purity of nodes in decision-making tree inside Random Forest and study the correlation between rhinitis features. We conduct 12 folds cross-validation experiments on 461 cases of clinical rhinitis. The outcomes show that the evaluation indicators of ARF-OOBEE, such as Sensitivity, Specificity, Accuracy, F1-Score, AUC, and G-Mean are 74.9%,86.5%,92.0%,78.3%,95.3%, and 79.9%, respectively. In comparison to the other methods, ARF-OOBEE has better evaluation indicator and is more effective for the early clinical diagnosis of rhinitis.

A machine learning method based on stacking heterogeneous ensemble learning for prediction of indoor humidity of greenhouse

Efficient production management, high productivity, and improved product quality are essential for the success of greenhouse production in producing sustainable agricultural products. Several environmental factors, including air temperature, humidity, CO2 levels, and light levels, have a major influence on this. Managing internal humidity is critical to preventing climate variation, disease, and pests in glasshouses that can cause significant damage if not properly controlled. This article assesses the performance of machine learning models in predicting indoor humidity levels in a greenhouse using a dataset from Guilan University's greenhouse located in Rasht City, Iran. Seven regression models were used to make predictions: multiple linear regression (MR), polynomial regression (PR), decision tree regression (DT), k-nearest neighbors regression (KNN), support vector regression (SVR), random forest regression (RF), and extreme gradient boosting regression (XGBoost). Evaluation criteria including coefficient of determination (R2), mean absolute error (MAE), mean square error (MSE), and root mean square error (RMSE) were used to evaluate each model. The best machine learning models were selected based on these criteria values (R2 > 0.94) and combined using the stacking method, a popular ensemble learning technique, to create a metamodel for accurately predicting internal humidity within the greenhouse. The metamodel showed exceptional performance, with significantly improved evaluation criteria on the test dataset, specifically R2 of 0.96515, MAE of 0.01395, MSE of 0.03205, and RMSE of 0.00102.

Prediction of Ship Painting Man-Hours Based on Selective Ensemble Learning

The precise prediction of painting man-hours is significant to ensure the efficient scheduling of shipyard production and maintain a stable production pace, which directly impacts shipbuilding cycles and costs. However, traditional forecasting methods suffer from issues such as low efficiency and poor accuracy. To solve this problem, this paper proposes a selective integrated learning model (ISA-SE) based on an improved simulated annealing algorithm to predict ship painting man-hours. Firstly, the improved particle swarm optimization (MPSO) algorithm and data grouping techniques are employed to achieve the optimal selection and hyperparameter optimization of base learners, constructing a candidate set of base learners. Subsequently, the simulated annealing algorithm is improved by adding random perturbations and using a parallel perturbation search mechanism to enhance the algorithm’s global search capability. Finally, an optimal set of base learners is composed of the candidate set utilizing the ISA-SE model, and a heterogeneous ensemble learning model is constructed with the optimal set of base learners to achieve the precise prediction of ship painting man-hours. The results indicate that the proposed ISA-SE model demonstrates improvements in accuracy, mean absolute error, and root mean square error compared to other models, validating the effectiveness and robustness of ISA-SE in predicting ship painting man-hours.

Application of Heterogeneous Ensemble Learning for CO2–Brine Interfacial Tension Prediction: Implications for CO2 Storage

Application of Heterogeneous Ensemble Learning for CO₂–Brine Interfacial Tension Prediction: Implications for CO₂ Storage

Multi-Task Heterogeneous Ensemble Learning-Based Cross-Subject EEG Classification Under Stroke Patients.

Robot-assisted motor training is applied for neurorehabilitation in stroke patients, using motor imagery (MI) as a representative paradigm of brain-computer interfaces to offer real-life assistance to individuals facing movement challenges. However, the effectiveness of training with MI may vary depending on the location of the stroke lesion, which should be considered. This paper introduces a multi-task electroencephalogram-based heterogeneous ensemble learning (MEEG-HEL) specifically designed for cross-subject training. In the proposed framework, common spatial patterns were used for feature extraction, and the features according to stroke lesions are shared and selected through sequential forward floating selection. The heterogeneous ensembles were used as classifiers. Nine patients with chronic ischemic stroke participated, engaging in MI and motor execution (ME) paradigms involving finger tapping. The classification criteria for the multi-task were established in two ways, taking into account the characteristics of stroke patients. In the cross-subject session, the first involved a direction recognition task for two-handed classification, achieving a performance of 0.7419 (±0.0811) in MI and 0.7061 (±0.1270) in ME. The second task focused on motor assessment for lesion location, resulting in a performance of 0.7457 (±0.1317) in MI and 0.6791 (±0.1253) in ME. Comparing the specific-subject session, except for ME on the motor assessment task, performance on both tasks was significantly higher than the cross-subject session. Furthermore, classification performance was similar to or statistically higher in cross-subject sessions compared to baseline models. The proposed MEEG-HEL holds promise in improving the practicality of neurorehabilitation in clinical settings and facilitating the detection of lesions.

SR-Forest: A Genetic Programming based Heterogeneous Ensemble Learning Method

Ensemble learning methods have been widely used in machine learning in recent years due to their high predictive performance. With the development of genetic programming-based symbolic regression methods, many papers begin to choose a popular ensemble learning method, random forests, as the baseline competitor. Instead of considering them as competitors, an alternative idea might be to consider symbolic regression as an enhancement technique for random forest. Genetic programming-based symbolic regression methods which fit a smooth function are complementary to the piecewise nature of decision trees, as the smooth variation is common in regression problems. In this article, we propose to form an ensemble model with symbolic regression-based decision trees to address this issue. Furthermore, we design a guided mutation operator to speed up the search on high-dimensional problems, a multi-fidelity evaluation strategy to reduce the computational cost and an ensemble selection mechanism to improve predictive performance. Finally, experimental results on a regression benchmark with 120 datasets show that the proposed ensemble model outperforms 25 existing symbolic regression and ensemble learning methods. Moreover, the proposed method can provide notable insights on an XGBoost hyperparameter performance prediction task, which is an important application area of ensemble learning methods.

A Loss Differentiation Method Based on Heterogeneous Ensemble Learning Model for Low Earth Orbit Satellite Networks.

In light of the high bit error rate in satellite network links, the traditional transmission control protocol (TCP) fails to distinguish between congestion and wireless losses, and existing loss differentiation methods lack heterogeneous ensemble learning models, especially feature selection for loss differentiation, individual classifier selection methods, effective ensemble strategies, etc. A loss differentiation method based on heterogeneous ensemble learning (LDM-HEL) for low-Earth-orbit (LEO) satellite networks is proposed. This method utilizes the Relief and mutual information algorithms for selecting loss differentiation features and employs the least-squares support vector machine, decision tree, logistic regression, and K-nearest neighbor as individual learners. An ensemble strategy is designed using the stochastic gradient descent method to optimize the weights of individual learners. Simulation results demonstrate that the proposed LDM-HEL achieves higher accuracy rate, recall rate, and F1-score in the simulation scenario, and significantly improves throughput performance when applied to TCP. Compared with the integrated model LDM-satellite, the above indexes can be improved by 4.37%, 4.55%, 4.87%, and 9.28%, respectively.

PM2.5 Prediction using Heterogeneous Ensemble Learning

Air pollution is a great concern to mankind and is causing too many adverse effects on every living organism on earth by increasing lung diseases, skin diseases, and many other problems caused by it. This research presents a comprehensive study on the application of heterogenous ensemble learning techniques for PM2.5 concentration prediction, aiming to enhance prediction accuracy and provide insights into the driving factors behind pollution levels. The primary objective is to conduct a comparative analysis of heterogenous ensemble method, namely, blending and stacking in conjunction with individual base models, such as multiple linear regression (LR), decision trees (DT), support vector regression (SVR) and artificial neural networks (ANN). In total 28 models were created using blending and 28 models were created using stacking. Hyperparameter tuning is done to optimize the models.

A heterogeneous ensemble architecture coupling model selection sorting and residual error iterative correction for crude oil price forecasting

Although crude oil price forecasts can provide policymakers, investors, consumers, and producers with scientific and reliable guidance, the nonlinearity and complexity of crude oil markets still pose serious challenges to this work. To this end, we propose a crude oil price ensemble prediction model that combines model selection sorting, multi-objective heterogeneous ensemble learning, and residual error iterative correction. First, a selection strategy based on the weighted information criterion is used to screen benchmark models in the model pool. Subsequently, candidate models are heterogeneously integrated using the multi-objective multi-verse optimization algorithm. VIseKriterijumska Optimizacija I Kompromisno Resenje, Technique for Order Preference by Similarity to Ideal Solution, and Grey Relational Analysis are utilized to select the optimal non-dominated solution of the Pareto surface. Finally, correction technology is designed to adaptively correct the residual error. Optimum error shrinkage and the number of corrections ensure maximum predictable component development. West Texas Intermediate and Europe Brent spot price data are utilized for experimental verification. Results show that the model can accurately describe crude oil price fluctuations, and has excellent robustness and stability. The forecasting method can be used as a reliable and promising energy economic decision algorithm.

Detection of Cache Pollution Attack Based on Ensemble Learning in ICN-Based VANET

Content Centric Network (CCN) can be extended to efficiently and reliably support content delivery and solve the network performance degradation caused by dynamic topology and intermittent connectivity of Vehicle Ad hoc NETwork (VANET). However, the in-network caching mechanism of Vehicular Content Centric Network (VCCN) is vulnerable against Cache Pollution Attack (CPA), where attackers aim to fill the buffer space with non-popular contents by releasing fake requests. Unavoidably, the cache hit ratio of content requests from legal users is degraded and the content retrieval latency is increased under CPA. Hence, it is critical to detect and mitigate CPA. The current solutions for static CCN cannot be directly applied into dynamic VCCN. In this paper, we propose a detection scheme based on hybrid heterogeneous multi-classifier ensemble learning, where CPA is determined by the cooperation of multiple vehicles. In our scheme, each vehicle can build or join a cluster whose head possesses more common moving attributes of position, speed and direction with other members. Besides, the cluster head as a base learner is responsible for training its own classifier by making some relevant statistics on requests and hit ratio. Specifically, the problem of ensemble classifier making from the individual classifiers is formulated as a linear optimization problem, with the goal of minimizing the false ratio of detecting CPA. The generalization ability of ensemble learning can make very accurate predictions on CPA. By comparison, our detection scheme outperforms the existing schemes in terms of detection ratio, hit ratio, retrieval delay. Besides, simulations have proved that the overfitting problem of adopting a singe base learning algorithm can be alleviated in our scheme.

Stacked Heterogeneous Ensemble Learning Model in Mixed Convection Heat Transfer from a Vertically Oscillating Flat Plate

In this study, the effects of mixed convection heat transfer from a moving vertical flat plate with an experimental and stacked heterogeneous ensemble learning approach are analyzed. In the experimental work, the effects on both natural and forced convection of dimensionless oscillation amplitude (Ao), dimensionless oscillation frequency (Wo) and Rayleigh number (Ra) are investigated. In the experiments, the vertical movement of the plate is provided by a flywheel-motor assembly. The average Nusselt numbers (Nu) on the fixed plate and the moving plate surface were obtained. Additionally, this study is focused on the prediction of heat transfer of a moving flat plate using single-based algorithms (Gradient Boosting, AdaBoost, Multilayer Per-ceptron) and a stacked heterogeneous ensemble learning model. The statistical per-formance of the single-based algorithms and the stacked ensemble model is meas-ured in the prediction of mixed convection heat transfer. The results show that the stacked-based ensemble learning model yielded the MSE = 2.01, RMSE = 1.42, MAE = 1.1 and R2 = 0.99 values. Overall, this study reveals that the proposed stacked en-semble machine learning model can be used successfully for modeling convection heat transfer of a moving plate.

Ensemble Deep Learning Ultimate Tensile Strength Classification Model for Weld Seam of Asymmetric Friction Stir Welding

Friction stir welding is a material processing technique used to combine dissimilar and similar materials. Ultimate tensile strength (UTS) is one of the most common objectives of welding, especially friction stir welding (FSW). Typically, destructive testing is utilized to measure the UTS of a welded seam. Testing for the UTS of a weld seam typically involves cutting the specimen and utilizing a machine capable of testing for UTS. In this study, an ensemble deep learning model was developed to classify the UTS of the FSW weld seam. Consequently, the model could classify the quality of the weld seam in relation to its UTS using only an image of the weld seam. Five distinct convolutional neural networks (CNNs) were employed to form the heterogeneous ensemble deep learning model in the proposed model. In addition, image segmentation, image augmentation, and an efficient decision fusion approach were implemented in the proposed model. To test the model, 1664 pictures of weld seams were created and tested using the model. The weld seam UTS quality was divided into three categories: below 70% (low quality), 70–85% (moderate quality), and above 85% (high quality) of the base material. AA5083 and AA5061 were the base materials used for this study. The computational results demonstrate that the accuracy of the suggested model is 96.23%, which is 0.35% to 8.91% greater than the accuracy of the literature’s most advanced CNN model.

National student loans default risk prediction: A heterogeneous ensemble learning approach and the SHAP method

National student loans are crucial for ensuring that economically disadvantaged students are able to complete their education successfully, however, the high default rate and excessive demand associated with these loans pose significant risks to various stakeholders. Students' repayment behavior can have adverse impacts on the state, banks, universities, and themselves. Despite the importance of this issue, there has been a lack of empirical research on national student loan default data using machine learning methods, and it has ignored the impact that student growth processes may have on default behavior. In this study, we addressed this research gap by integrating multiple heterogeneous machine learning models through performance analysis to improve prediction accuracy. Furthermore, we utilized the SHAP interpretable method to examine the relationship between students' growth process and loan default behavior in greater depth. The results of this study confirmed that the total amount of scholarships received by students during their school years has a significant impact on the occurrence of loan default. Specifically, the more scholarships students receive, the less likely they are to default. Additionally, we found that students' performance during school, as reflected by their GPA, is a significant predictor of default behavior. Surprisingly, we also found that college entrance examination scores can influence the risk of loan default to a certain extent. This study provides valuable insights for schools regarding students who apply for national student loans and establishes a bridge for college administrators to predict students' default behavior based on campus “big data”. Based on the study results, college administrators can design “precise” education programs during students' study period to effectively reduce the risk of national student loan default.

Identifying the optimal heterogeneous ensemble learning model for building energy prediction using the exhaustive search method

Heterogeneous ensemble learning has received increasing attention in building energy prediction research because it provides more stable and accurate predictions than the conventional single model-based methods. Although existing studies have proved the feasibility of adopting heterogeneous ensemble learning in building energy prediction, they have not fully explored the method of constructing an optimal heterogeneous ensemble learning model based on limited base model resources. This paper attempts to identify the optimal heterogeneous ensemble learning model for building energy prediction by using the exhaustive search method. Six machine learning methods are used as the alternative base model for constructing heterogeneous ensemble learning models. All 57 heterogeneous ensemble learning models that can be built from the six base models are compared and the optimal model is identified. The hourly building energy prediction experiment is performed on an institutional building located at the University of Florida to verify the proposed exhaustive search method. Three modules representing different semesters of the test building are developed and tested to examine the predictive performance of the proposed heterogeneous ensemble learning models. The research findings demonstrate the feasibility of the proposed exhaustive search method in identifying the optimal heterogeneous ensemble learning model. The optimal heterogeneous ensemble learning model found in this study can reduce the MAPE by 4.9% − 6.6% compared with its most accurate base model in the model testing stage. The experimental results and research findings provide critical guidance for applying heterogeneous ensemble learning to data-driven building energy prediction.