Ensemble Learning Strategy Research Articles

Airborne LiDAR Point Cloud Classification Using Ensemble Learning for DEM Generation

Airborne laser scanning (ALS) point clouds have emerged as a predominant data source for the generation of digital elevation models (DEM) in recent years. Traditionally, the generation of DEM using ALS point clouds involves the steps of point cloud classification or ground point filtering to extract ground points and labor-intensive post-processing to correct the misclassified ground points. The current deep learning techniques leverage the ability of geometric recognition for ground point classification. However, the deep learning classifiers are generally trained using 3D point clouds with simple geometric terrains, which decrease the performance of model inferencing. In this study, a point-based deep learning model with boosting ensemble learning and a set of geometric features as the model inputs is proposed. With the ensemble learning strategy, this study integrates specialized ground point classifiers designed for different terrains to boost classification robustness and accuracy. In experiments, ALS point clouds containing various terrains were used to evaluate the feasibility of the proposed method. The results demonstrated that the proposed method can improve the point cloud classification and the quality of generated DEMs. The classification accuracy and F1 score are improved from 80.9% to 92.2%, and 82.2% to 94.2%, respectively, by using the proposed methods. In addition, the DEM generation error, in terms of mean squared error (RMSE), is reduced from 0.318–1.362 m to 0.273–1.032 m by using the proposed ensemble learning.

Optimizing and benchmarking polygenic risk scores with GWAS summary statistics

BackgroundPolygenic risk score (PRS) is a major research topic in human genetics. However, a significant gap exists between PRS methodology and applications in practice due to often unavailable individual-level data for various PRS tasks including model fine-tuning, benchmarking, and ensemble learning.ResultsWe introduce an innovative statistical framework to optimize and benchmark PRS models using summary statistics of genome-wide association studies. This framework builds upon our previous work and can fine-tune virtually all existing PRS models while accounting for linkage disequilibrium. In addition, we provide an ensemble learning strategy named PUMAS-ensemble to combine multiple PRS models into an ensemble score without requiring external data for model fitting. Through extensive simulations and analysis of many complex traits in the UK Biobank, we demonstrate that this approach closely approximates gold-standard analytical strategies based on external validation, and substantially outperforms state-of-the-art PRS methods.ConclusionsOur method is a powerful and general modeling technique that can continue to combine the best-performing PRS methods out there through ensemble learning and could become an integral component for all future PRS applications.

Stacked generalization ensemble learning strategy for multivariate prediction of delamination and maximum thrust force in composite drilling

The complexity of drilling carbon fiber reinforced polymers (CFRP) requires accurate predictive models. This study addresses the challenge using an ensemble machine learning (ML) approach with stacked generalization. The model captures the relationships between key input variables—such as graphene nanoplatelet (GNP) content, ultrasonic assistance, tool type, stacking sequence, and feed rate—and output parameters, specifically thrust force and delamination. A nested feature scoring (NFS) method was employed for importance analysis, revealing tooling type and feed rate as key features for minimizing delamination and reducing thrust force, respectively. The machinability results revealed that ultrasonic drilling lowered thrust force by improving chip evacuation and reducing fiber breakage. HSS tools with cobalt content, alongside symmetrical stacking sequence, helped to further minimize both thrust force and delamination. However, the inclusion of GNPs led to an increase in thrust force and delamination, attributed to the increased strength of the CFRP/GNP composite. The process involved meticulous training, resulting in four optimal-fit models serving as inputs for the stacked meta-model. Iterative enhancements fortified the ensemble robustness, with fine-tuning of hyperparameters through Bayesian optimization. The ensemble superiority over individual models manifested in a remarkable reduction of mean absolute error (MAE) and root mean squared error (RMSE) by up to 97% and 124% for delamination, and 205% and 154% for thrust force, compared to the best base learner. Visual and statistical assessments effectively illuminated the intricate interactions between variables in the drilling process. The methodology resulted in a highly adaptable predictive model with applications across diverse manufacturing contexts.

An ensemble optimizer with a stacking ensemble surrogate model for identification of groundwater contamination source

The application of the simulation-optimization method for groundwater contamination source identification (GCSI) encounters two main challenges: the substantial time cost of calling the simulation model, and the limitations on the accuracy of identification results due to the complexity, nonlinearity, and ill-posed nature of the inverse problem. To address these issues, we have innovatively developed an inversion framework based on ensemble learning strategies. This framework comprises a stacking ensemble model (SEM), which integrates three distinct machine learning models (Extremely Randomized Trees, Adaptive Boosting, and Bidirectional Gated Recurrent Unit), and an ensemble optimizer (E-GKSEEFO), which combines two newly proposed swarm intelligence optimizers (Genghis Khan Shark Optimizer and Electric Eel Foraging Optimizer). Specifically, the SEM serves as a surrogate model for the groundwater numerical simulation model. Compared to the original simulation model, it significantly reduces time cost while maintaining accuracy. The E-GKSEEFO, functioning as the search strategy for the optimization model, greatly enhances the accuracy of the optimization results. We have verified the performance of the SEM-E-GKSEEFO ensemble inversion framework through two hypothetical scenarios derived from an actual coal gangue pile. The results are as follows. (1) The SEM exhibits improved fitting performance compared to single machine learning models when dealing with high-dimensional nonlinear data from GCSI. (2) The E-GKSEEFO achieves significantly higher accuracy in the identification results of GCSI than individual optimizers. These findings affirm the effectiveness and superiority of the proposed SEM-E-GKSEEFO ensemble inversion framework.

Spatio-Temporal Heterogeneous Ensemble Learning Method for Predicting Land Subsidence

The prediction of land subsidence is of significant value for the early warning and prevention of geological disasters. Although numerous land subsidence prediction methods are currently available, two obstacles still exist: (i) spatio-temporal heterogeneity of land subsidence is not well considered, and (ii) the prediction performance of individual models is unsatisfactory when the data do not meet their assumptions. To address these issues, we developed a spatio-temporal heterogeneous ensemble learning method for predicting land subsidence. Firstly, a two-stage hybrid spatio-temporal clustering method was proposed to divide the dataset into internally homogeneous spatio-temporal clusters. Secondly, within each spatio-temporal cluster, an ensemble learning strategy was employed to combine one time series prediction model and three spatio-temporal prediction models to reduce the prediction uncertainty of an individual model. Experiments on a land subsidence dataset from Cangzhou, China, show that the prediction accuracy of the proposed method is significantly higher than that of four individual prediction models.

Machine-learning-based prediction by stacking ensemble strategy for surgical outcomes in patients with degenerative cervical myelopathy

BackgroundMachine learning (ML) is extensively employed for forecasting the outcome of various illnesses. The objective of the study was to develop ML based classifiers using a stacking ensemble strategy to predict the Japanese Orthopedic Association (JOA) recovery rate for patients with degenerative cervical myelopathy (DCM).MethodsA total of 672 patients with DCM were included in the study and labeled with JOA recovery rate by 1-year follow-up. All data were collected during 2012–2023 and were randomly divided into training and testing (8:2) sub-datasets. A total of 91 initial ML classifiers were developed, and the top 3 initial classifiers with the best performance were further stacked into an ensemble classifier with a supported vector machine (SVM) classifier. The area under the curve (AUC) was the main indicator to assess the prediction performance of all classifiers. The primary predicted outcome was the JOA recovery rate.ResultsBy applying an ensemble learning strategy (e.g., stacking), the accuracy of the ML classifier improved following combining three widely used ML models (e.g., RFE-SVM, EmbeddingLR-LR, and RFE-AdaBoost). Decision curve analysis showed the merits of the ensemble classifiers, as the curves of the top 3 initial classifiers varied a lot in predicting JOA recovery rate in DCM patients.ConclusionsThe ensemble classifiers successfully predict the JOA recovery rate in DCM patients, which showed a high potential for assisting physicians in managing DCM patients and making full use of medical resources.

House Price Prediction Taking Account of the Influence of COVID-19

Machine learning algorithms are utilized in various practical applications and academic research. Due to the advent of digital technology, there is now a greater availability of massive, organized, and georeferenced datasets. This has simplified the use of algorithms for trend analysis and detection as well as forecasting, which helps users make decisions. The purpose of this study is to determine which Machine Learning (ML) algorithms are most effective in projecting housing price and to assess how the COVID-19 has affected these prices. The method includes the stages of data preprocessing, feature manipulation, hyperparameter tuning, evaluation, selection, and, in the end, interpretation of the model. Another objective is to assess and compare several ensemble learning strategies, comprising bagging techniques like random forest, and boosting techniques like Gradient Boosting Regressor, using a linear regression model. The results of this study confirm that the real estate sector is very resistant to pandemics, as the decline in prices was not as severe as initially anticipated. Among the several models that were examined, extreme gradient boosting demonstrated the highest level of accuracy, albeit with a minimal difference.

Fed-MStacking: Heterogeneous Federated Learning With Stacking Misaligned Labels for Abnormal Heart Sound Detection.

Ubiquitous sensing has been widely applied in smart healthcare, providing an opportunity for intelligent heart sound auscultation. However, smart devices contain sensitive information, raising user privacy concerns. To this end, federated learning (FL) has been adopted as an effective solution, enabling decentralised learning without data sharing, thus preserving data privacy in the Internet of Health Things (IoHT). Nevertheless, traditional FL requires the same architectural models to be trained across local clients and global servers, leading to a lack of model heterogeneity and client personalisation. For medical institutions with private data clients, this study proposes Fed-MStacking, a heterogeneous FL framework that incorporates a stacking ensemble learning strategy to support clients in building their own models. The secondary objective of this study is to address scenarios involving local clients with data characterised by inconsistent labelling. Specifically, the local client contains only one case type, and the data cannot be shared within or outside the institution. To train a global multi-class classifier, we aggregate missing class information from all clients at each institution and build meta-data, which then participates in FL training via a meta-learner. We apply the proposed framework to a multi-institutional heart sound database. The experiments utilise random forests (RFs), feedforward neural networks (FNNs), and convolutional neural networks (CNNs) as base classifiers. The results show that the heterogeneous stacking of local models performs better compared to homogeneous stacking.

Identification of millet origin using terahertz spectroscopy combined with ensemble learning

It’s crucial for both producers and consumers to accurately trace the origin of millet, given the significant differences in price and taste that exist between millets from various origins. The traditional method of identifying the origin of millet is time-consuming, laborious, complex, and destructive. In this study, a new method for fast and non-destructive differentiation of millet origins is developed by combining terahertz time domain spectroscopy with ensemble learning. Firstly, three machine learning algorithms, namely support vector machine (SVM), random forest (RF), and kernel extreme learning machine (KELM), were used to build different discriminative models, and then the impact of six different preprocessing methods on the models’ classification performance was compared. It was observed that models employing Savitzky-Golay preprocessing exhibited pronounced superiority in accurately determining the millet’s geographical origins. Building upon these findings, the research introduces an innovative ensemble learning strategy, leveraging both topsis and stacking techniques, to harness the collective strengths of the three algorithms. The outcomes of this approach reveal its remarkable capacity to distinguish millets originating from five distinct locations without the necessity for any parameter fine-tuning. The accuracy, F1 score, and Kappa on the prediction set are all 100 %, which significantly outperforms the single model, traditional voting method, and stacking method. The culmination of this study suggests that the integration of terahertz time-domain spectroscopy and TOPSIS-Stacking ensemble learning emerges as a promising method for the swift and non-intrusive discrimination of millet geographical origins with remarkable precision.

Accurate MRI-Based Brain Tumor Diagnosis: Integrating Segmentation and Deep Learning Approaches

Magnetic Resonance Imaging (MRI) is vital in diagnosing brain tumours, offering crucial insights into tumour morphology and precise localisation. Despite its pivotal role, accurately classifying brain tumours from MRI scans is inherently complex due to their heterogeneous characteristics. This study presents a novel integration of advanced segmentation methods with deep learning ensemble algorithms to enhance the classification accuracy of MRI-based brain tumour diagnosis. We conduct a thorough review of both traditional segmentation approaches and contemporary advancements in region-based and machine learning-driven segmentation techniques. This paper explores the utility of deep learning ensemble algorithms, capitalising on the diversity of model architectures to augment tumour classification accuracy and robustness. Through the synergistic amalgamation of sophisticated segmentation techniques and ensemble learning strategies, this research addresses the shortcomings of traditional methodologies, thereby facilitating more precise and efficient brain tumour classification.

Large-scale IoT attack detection scheme based on LightGBM and feature selection using an improved salp swarm algorithm

Due to the swift advancement of the Internet of Things (IoT), there has been a significant surge in the quantity of interconnected IoT devices that send and exchange vital data across the network. Nevertheless, the frequency of attacks on the Internet of Things is steadily rising, posing a persistent risk to the security and privacy of IoT data. Therefore, it is crucial to develop a highly efficient method for detecting cyber threats on the Internet of Things. Nevertheless, several current network attack detection schemes encounter issues such as insufficient detection accuracy, the curse of dimensionality due to excessively high data dimensions, and the sluggish efficiency of complex models. Employing metaheuristic algorithms for feature selection in network data represents an effective strategy among the myriad of solutions. This study introduces a more comprehensive metaheuristic algorithm called GQBWSSA, which is an enhanced version of the Salp Swarm Algorithm with several strategy improvements. Utilizing this algorithm, a threshold voting-based feature selection framework is designed to obtain an optimized set of features. This procedure efficiently decreases the number of dimensions in the data, hence preventing the negative effects of having a high number of dimensions and effectively extracting the most significant and crucial information. Subsequently, the extracted feature data is combined with the LightGBM algorithm to form a lightweight and efficient ensemble learning scheme for IoT attack detection. The proposed enhanced metaheuristic algorithm has superior performance in feature selection compared to the recent metaheuristic algorithms, as evidenced by the experimental evaluation conducted using the NSLKDD and CICIoT2023 datasets. Compared to current popular ensemble learning solutions, the proposed overall solution exhibits excellent performance on multiple key indicators, including accuracy, precision, as well as training and detection time. Especially on the large-scale dataset CICIoT2023, the proposed scheme achieves an accuracy rate of 99.70% in binary classification and 99.41% in multi classification.

Ensemble learning of decomposition-based machine learning models for multistep-ahead daily streamflow forecasting in northwest China

ABSTRACT Accurate daily streamflow forecasts remain challenging in arid regions. A Bayesian model averaging (BMA) ensemble learning strategy was proposed to forecast 1-, 2-, and 3-day-ahead streamflow in Dunhuang Oasis, northwest China. The efficiency of BMA was compared with four decomposition-based machine learning and deep learning models. Satisfactory forecasts were achieved with all proposed models at all lead times; however, based on Nash-Sutcliffe efficiency values of 0.976, 0.967, and 0.957, BMA achieved the greatest accuracy for 1-, 2-, and 3-day-ahead streamflow forecasts, respectively. Uncertainty analysis confirmed the reliability of BMA in yielding consistently accurate streamflow forecasts. Thus, BMA could provide an efficient alternative approach to multistep-ahead daily streamflow forecasting. The incorporation of data decomposition techniques (e.g. variational mode decomposition) and deep learning algorithms (e.g. deep belief network) into BMA may provide worthy technical references for supervised learning of streamflow systems in data-scarce regions.

Interpolation-split: a data-centric deep learning approach with big interpolated data to boost airway segmentation performance

The morphology and distribution of airway tree abnormalities enable diagnosis and disease characterisation across a variety of chronic respiratory conditions. In this regard, airway segmentation plays a critical role in the production of the outline of the entire airway tree to enable estimation of disease extent and severity. Furthermore, the segmentation of a complete airway tree is challenging as the intensity, scale/size and shape of airway segments and their walls change across generations. The existing classical techniques either provide an undersegmented or oversegmented airway tree, and manual intervention is required for optimal airway tree segmentation. The recent development of deep learning methods provides a fully automatic way of segmenting airway trees; however, these methods usually require high GPU memory usage and are difficult to implement in low computational resource environments. Therefore, in this study, we propose a data-centric deep learning technique with big interpolated data, Interpolation-Split, to boost the segmentation performance of the airway tree. The proposed technique utilises interpolation and image split to improve data usefulness and quality. Then, an ensemble learning strategy is implemented to aggregate the segmented airway segments at different scales. In terms of average segmentation performance (dice similarity coefficient, DSC), our method (A) achieves 90.55%, 89.52%, and 85.80%; (B) outperforms the baseline models by 2.89%, 3.86%, and 3.87% on average; and (C) produces maximum segmentation performance gain by 14.11%, 9.28%, and 12.70% for individual cases when (1) nnU-Net with instant normalisation and leaky ReLU; (2) nnU-Net with batch normalisation and ReLU; and (3) modified dilated U-Net are used respectively. Our proposed method outperformed the state-of-the-art airway segmentation approaches. Furthermore, our proposed technique has low RAM and GPU memory usage, and it is GPU memory-efficient and highly flexible, enabling it to be deployed on any 2D deep learning model.

Displacement prediction for landslide with step-like behavior based on stacking ensemble learning strategy

Displacement prediction for landslide with step-like behavior based on stacking ensemble learning strategy

Using ensemble learning and hierarchical strategy to predict the outcomes of ESWL for upper ureteral stone treatment

Urinary tract stones are a common and frequently recurring medical issue. Accurately predicting the success rate after surgery can help avoid ineffective medical procedures and reduce unnecessary healthcare costs. This study collected data from patients with upper ureter stones who underwent extracorporeal shock wave lithotripsy, including cases of successful as well as unsuccessful stone removal after the first and second lithotripsy procedures, and constructed prediction systems for the outcomes of the first and second lithotripsy procedures. Features were extracted from three categories of information: patient characteristics, stone characteristics, and extracorporeal shock wave lithotripsy machine data, and additional features were created using Feature Creation. Finally, the impact of features on the models was analyzed using six methods to calculate feature importance. Our prediction model for the first lithotripsy, selected from among 43 methods and seven ensemble learning techniques, achieves an AUC of 0.91. For the second lithotripsy, the AUC reaches 0.76. The results indicate that the detailed and binary information provided by patients regarding their history of stone experiences contributes differently to the predictive accuracy of the first and second lithotripsy procedures. The prediction tool is available at https://predictor.isu.edu.tw/ks.

The robust SmartEnsembleNet: A game changer in finger knuckle biometrics

SmartEnsembleNet represents a major leap forward in the domain of biometric identification, focusing on the relatively underexplored area of finger knuckle biometrics. This research enhances the effectiveness and uniqueness of biometric systems by leveraging the distinct and secure features of finger knuckle patterns. As a complement to existing biometric techniques like fingerprints and facial recognition, it adds an additional layer of security and reliability. Addressing the challenges faced by finger knuckle identification, such as image acquisition, environmental effects, and aging, SmartEnsembleNet introduces a pioneering multi-layered ensemble learning strategy that combines eight diverse machine learning models. These models, trained via a 5-fold cross-validation on datasets from IIT Delhi and PolyU, demonstrate significant improvements in accuracy, precision, recall, and F1-score. A distinctive feature of SmartEnsembleNet is its innovative meta-model, streamlining the creation of a meta-dataset. This dataset is created by applying bilinear interpolation to the predicted probabilities from the top-performing model, and it is further refined by subtracting these interpolated probabilities from the validation data. This iterative process generates ‘smart data’, which serves as the basis for training the ultimate ‘smart model’. Logistic regression is chosen for its outstanding performance. This method, applied uniformly to both training and testing phases, considerably enhances classification accuracy in finger knuckle biometrics, establishing new benchmarks. SmartEnsembleNet’s performance undergoes a detailed evaluation using the Friedman test and Nemenyi post-hoc analysis, guaranteeing a selection process grounded in statistical rigor. With its resilience to noise and adaptability under various conditions, SmartEnsembleNet not only addresses existing gaps in finger knuckle biometric research but also pioneers future advancements, providing a versatile and secure solution impactful for both academia and high-security scenarios.

An AI-Based Classification and Recommendation System for Digital Libraries

The immense volume of online content linked to digital libraries has given emergence to the advancement of screening and recommendation systems. A recommendation system is vitally important in both academic institutions and elibraries to assist professors, instructors, students, and researchers in finding appropriate sources of information. Distributed or collaborative screening is the most common method used in current recommendation systems. However, collaborative approaches cannot promote library repositories, including unrated or unpurchased electronic information. Thus, this paper deals with the automated classification and recommendation of a multiclass corpus found in virtual repositories (cloud databases). In various stages, Neuro-Fuzzy (NF) and Support Vector Machine (SVM) techniques are used as the base classifiers for the categorization of the essential subjects (contents). Later, a high-level ensemble learning strategy is utilized to recommend appropriate subjects from the available multiclass corpus. The methods use a CoC (Coherence of Content)-based inference mechanism to extract and filter the critical components before beginning the recommendation process. Experiments demonstrated that a recommended approach based on detailed conceptual descriptions instead of a handful of phrases/words might help academic and research communities to find relevant sources. Observing the results over a period of months shows that the suggested method increases user comfort, proving the system’s acceptability to users in this way. In addition, compared to previous models, the accuracy in categorizing therequisite subjects is more than 97.16 per cent.

Long-short-view aware multi-agent reinforcement learning for signal snippet distillation in delirium movement detection

Automatic movement analysis utilizing surveillance video is believed to be an important and convenient way for timely delirium detection in an Intensive Care Unit (ICU). However, video-based delirium movement detection (DMD) faces inherent challenges: 1) Irregular movements with large differences in the four limbs of a patient; 2) similar movements in delirium and normal situations, and large movement variations between patients experiencing delirium. To address the challenges 1, this paper proposes a Long-Short-View Aware Multi-Agent Reinforcement Learning (LS-MARL) method to identify the most representative movement snippet for DMD, considering that the global state provided by the long-view is important for guiding the agent's decision-making, but ignored in existing MARL methods. The proposed LS-MARL has two novel designs. First, a novel Teacher Auxiliary Policy (TAP) is developed for direction preperception of the representative movement snippet. Second, a new reward mechanism, Team Intrinsic Reward (TIR), is introduced to quantify the contribution of each agent. Experiments demonstrate that the proposed LS-MARL method outperforms state-of-the-art methods. Furthermore, to handle the second challenge, a new Self-Adjusting Ensemble Learning (SAEL) strategy is built to adaptively integrate an optimal classifier combination from multidomain features, which further improves the performance of classification tasks in the proposed LS-MARL method.

An ensemble spatial prediction method considering geospatial heterogeneity

Ensemble learning synthesizes the advantages of different models and has been widely applied in the field of spatial prediction. However, the nonlinear constraints of spatial heterogeneity on the model ensemble process make it difficult to adaptively determine the ensemble weights, greatly limiting the predictive ability of the ensemble learning model. This paper therefore proposes a novel geographical spatial heterogeneous ensemble learning method (GSH-EL). Firstly, the geographically weighted regression model, geographically optimal similarity model, and random forest model are used as three base learners to express local spatial heterogeneity, global feature correlation, and nonlinear relationship of geographic elements, respectively. Then, a spatially weighted ensemble neural network module (SWENN) of GSH-EL is proposed to express spatial heterogeneity by exploring the complex nonlinear relationship between the spatial proximity and ensemble weights. Finally, the outputs of the three base learners are combined with the spatial heterogeneous ensemble weights from SWENN to obtain the spatial prediction results. The proposed method is validated on the PM2.5 air quality and landslide dataset in China, both of which obtain more accurate prediction results than the existing ensemble learning strategies. The results confirm the need to accurately express spatial heterogeneity in the model ensemble process.

Improving Social Relations Between Students Through Learning Dayak Music Ensembles Using the Peer Tutor Method

Preserving Dayak culture and developing musical abilities among the younger generation are important aspects in the revitalization of society. Learning Dayak music ensembles can help students to improve their social skills more effectively, so that they can be more calm, focused and productive in making music. This research used a qualitative descriptive method involving first semester students of the Performing Arts Education Study Program. Dayak music ensemble learning is carried out once a week. This learning includes theoretical and practical material for playing Dayak musical instruments, such as dau, gedobokng and agukng. These three instruments are often found in accompanying music for Dayak traditional ritual ceremonies. Learning activities take the form of exercises in ensemble format so that in the process students need to interact and socialize in order to be able to play music correctly. The development of students' social skills is obtained through observation, interviews and documents. Data analysis shows that the peer tutoring method makes a significant contribution to the development of social skills among students. This technique is used to determine aspects of social abilities, such as the ability to communicate, collaborate, leadership and responsibility effectively. The findings of this research contribute to the development of structured Dayak music ensemble learning strategies to improve students' social abilities which are reflected in aspects of social sensitivity, the emergence of a sense of empathy and self-confidence. Additionally, these findings highlight the transformative power of peer mentoring in fostering social-emotional development and musical skills.