Ensemble Learning Methodology Research Articles

Mapping high-resolution XCO2 concentrations in China from 2015 to 2020 based on spatiotemporal ensemble learning model

High-resolution column-averaged dry air mole fraction of CO2 (XCO2) data is crucial for understanding the spatiotemporal patterns of XCO2 and for mitigating carbon emissions. Due to the limited scanning range of sensors and strict inversion conditions, satellite-retrieved XCO2 data are often significantly incomplete. Machine learning models are widely used to fill these gaps in satellite XCO2 data. However, the limitations of individual machine learning models and the complexity of the spatial distribution of XCO₂ mean that the accuracy of XCO2 predictions still needs improvement. In this study, a new spatiotemporal stacked ensemble learning model (STEL) was developed by combining random forest (RF), extremely randomized trees (ERT), extreme gradient boosting (XGBoost), optical gradient boosting (LightGBM), and categorical boosting (CatBoost) using the stacking ensemble learning methodology. Considering the spatiotemporal heterogeneity of XCO2, a novel spatiotemporal weighting feature was constructed as part of the model's input parameters. Finally, the XCO2 observed by Orbiting Carbon Observatory 2 (OCO-2) was reconstructed using STEL, and a monthly mean XCO2 dataset covering China from 2015 to 2020 was generated at a spatial resolution of 0.1°. The results show that STEL exhibits superior performance and generalization capabilities compared to individual machine-learning models. R2 RMSE and MAPE were 0.9624, 1.0023 ppm, and 0.1583 % on the test set, and 0.8970, 1.4213 ppm, and 0.2475 % for R2, RMSE, and MAPE in ground validation, respectively. In 10-fold cross-validation, STEL's RMSE was reduced by 9.52 % compared to the best-performing single model (RF). The spatiotemporal trend of CO2 in China from 2015 to 2020 was analyzed using STEL XCO2 data. The results indicate that this dataset accurately reflects the spatiotemporal heterogeneity of XCO2 distribution at a fine scale. Overall, XCO2 exhibited a spatiotemporal pattern of “high in the east and low in the west” and “high in spring and low in summer.” Except in summer, high XCO₂ values were mainly distributed in the North China Plain. XCO2 trends and hotspots showed considerable spatial variation. The Pearl River Delta and Yangtze River Delta urban agglomerations have the fastest XCO2 growth rates, and the distribution of XCO2 hotspots is consistent with the distribution of population and economic centers. In the sparsely populated northwest of China, XCO2 is growing rapidly due to increased thermal power generation and coal mining. XCO2 hotspots in Northwest China are mainly located in Xinjiang, Ningxia, and Inner Mongolia. The methodology and data presented are useful for further research on carbon emissions, carbon sinks, and climate change.

Electrical Power Generation Forecasting from Renewable Energy Systems Using Artificial Intelligence Techniques

Renewable energy (RE) sources, such as wind, geothermal, bioenergy, and solar, have gained interest in developed regions. The rapid expansion of the economies in the Middle East requires massive increases in electricity production capacity, and currently fossil fuel reserves meet most of the power station demand. There is a considerable measure of unpredictability surrounding the locations of the concerned regions where RE can be used to generate electricity. This makes forecasting difficult for the investor to estimate future electricity production that could be generated in each area over the course of a specific period. Energy production forecasting with complex time-series data is a challenge. However, artificial neural networks (ANNs) are well suited for handling nonlinearity effectively. This research aims to investigate the various ANN models capable of providing reliable predictions for sustainable sources of power such as wind and solar. In addition to the ANN models, a state-of-the-art ensemble learning approach is used to improve the accuracy of predictions further. The proposed strategies can forecast RE generation accurately over short and long time frames, relying on historical data for precise predictions. This work proposes a new hybrid ensemble framework that strategically combines multiple complementary machine learning (ML) models to improve RE forecasting accuracy. The ensemble learning (EL) methodology outperforms long short-term memory (LSTM), light gradient boosting machine (LightGBM), and sequenced-GRU in predicting wind power (MAE: 0.782, MAPE: 0.702, RMSE: 0.833) and solar power (MAE: 1.082, MAPE: 0.921, RMSE: 1.055). It achieved an impressive R2 value of 0.9821, indicating its superior accuracy.

Comparative performance of ensemble machine learning for Arabic cyberbullying and offensive language detection

Since cyberbullying impacts both individual victims and entire society, research on abusive language and its detection has attracted attention in recent years. Because social media sites like Facebook, Instagram, Twitter, and others are so widely accessible, hate speech, bullying, sexism, racism, aggressive material, harassment, poisonous comments, and other types of abuse have all substantially increased. Due to the critical requirement to detect, regulate, and limit the spread of harmful content on social networking sites, we conducted this study to automate the detection of offensive language or cyberbullying. We created a new Arabic balanced data set to be used in the offensive detection process because having a balanced data set for a model would result in improved accuracy models. Recently, the performance of single classifiers has been improved using ensemble machine learning. The purpose of this study is to examine the effectiveness of several single and ensemble machine learning algorithms in identifying Arabic text that contains foul language and cyberbullying. Applying them to three Arabic datasets, we have selected three machine learning classifiers and three ensemble models for this aim. Two of them are offensive datasets that are readily accessible in the public, while the third one was created. The results showed that the single learner machine learning strategy is inferior to the ensemble machine learning methodology. Voting performs is the best performing trained ensemble machine learning classifier, outperforming the best single learner classifier (65.1%, 76.2%, and 98%) for the same datasets with accuracy scores of (71.1%, 76.7%, and 98.5%) for each of the three datasets used. Finally, we improve the voting technique’s performance through hyperparameter tuning on the Arabic cyberbullying data set.

Ensemble machine learning approach to identify excitation failure in synchronous generators

Loss of Excitation (LOE) is a significant failure that occurs in synchronous generators, affecting both generator and power grid. The conventional LOE protection relays work based on the impedance measurement at generator terminal but they tend to mal-operate other than excitation failure like Stable Power Swing (SPoS). To achieve more reliable and accurate protection, a proper speedy discrimination of LOE failure from SPoS is indeed. This paper deals an ensemble machine learning methodology to identify LOE.In order to analyse LOE failure, generator dynamic model is considered and simulation is performed on IEEE-9 bus system. In this, various observations are formed for LOE, SPoS and other events under various loading scenarios. Discrete Wavelet Transform (DWT) analysis is employed on generator terminal voltage signal and Time-Frequency Domain (TFD) features are extracted to define unique characteristics of LOE failure. The ensemble Adaboost binary classifier is trained with feature matrix and tested. Finally, the ensemble model is established and identified LOE failure from SPoS in 1.517 s. Comprehensive simulation analysis confirmed the supremacy of the proposed method than other compared methods.

Ensemble Learning Methods for Dirty Data: A Keynote at CIKM 2022

Neural network ensemble is a collaborative learning paradigm that utilizes multiple neural networks to solve a complex learning problem. Constructing predictive models with high generalization performance is an important and yet most challenging goal for robust AI systems in the presence of dirty data. Given a target learning task, popular approaches have been dedicated to designing and finding the top performing model. However, it is difficult in general to estimate the best model when available data is finite, possibly dirty, or insufficient for the problem. The problem of dirty data in machine learning (ML) can be characterized by the out of distribution data and the digital or physical deception of data. Such dirty data may cause unintended or harmful behavior for well trained ML models. In this paper, a curated version of my keynote at ACM CIKM 2022, I will first give a brief overview of ensemble learning methodology. Then I will review different types of dirty data that could deceive well-trained ML models. Finally, I will describe a focal diversity optimized ensemble learning framework, developed at Georgia Tech, for measuring, enforcing, and combining multiple neural networks, delivering high generalization performance of ensemble learner, while maximizing ensemble utility and resilience to dirty data. Date: 17 October 2022.

Predicting protein-membrane interfaces using molecular simulations and ensemble machine learning.

Abnormal protein-membrane attachment is involved in deregulated cellular pathways and in disease. The possibility to modulate protein-membrane interactions represents a new promising therapeutic strategy for membrane proteins. In this talk, we explore the free energy landscape of membrane protein dimerization using parallel tempering metadynamics simulations in the well-tempered ensemble and coarse-grained force fields and reproduce the structure and energetics of the dimerization process of membrane proteins and proteins in an aqueous solution in reasonable accuracy and throughput. We propose that the use of enhanced sampling simulations with a refined coarse-grained force field and appropriately defined collective variables is a robust approach for studying the protein dimerization process, although one should be cautious of the energy minima ranking. Moreover, we study oncogenes, including the H1047R and E545K hotspot mutants of PI3Kα, and KRAS-4B to understand the basis of protein overactivation. We calculate their allosteric pathways and show residues important in delivering communication signals between functional domains of each protein. Finally, we describe an ensemble machine learning methodology to predict protein-membrane interfaces of peripheral membrane proteins and present a drug design pipeline for drugging protein-membrane interfaces using the DREAMM web-server https://dreamm.ni4os.eu. Taking into account these results, we investigate opportunities for allosteric drug design.

Improving Drug Combination Repositioning using Positive Unlabelled Learning and Ensemble Learning

Drug repositioning is a cost-effective and time-effective concept that enables the use of existing drugs/drug combinations for therapeutic effects. The number of drug combinations used for therapeutic effects is smaller than all possible drug combinations in the present drug databases. These databases consist of a smaller set of labelled positives and a majority of unlabelled drug combinations. Therefore, there is a need for determining both reliable positive and reliable negative samples to develop binary classification models. Since, we only have labelled positives, the unlabelled data has to be separated into positives and negatives by a reliable technique. This study proposes and demonstrates the significance of using Positive Unlabelled Learning, for determining reliable positive and negative drug combinations for drug repositioning. In the proposed approach, the dataset with known positives and unlabelled samples was clustered by a Deep Learning based Self Organizing Map. Then, an ensemble learning methodology was followed by employing three classification models. The proposed PUL model was compared with the frequently used approach that randomly selects negative drug pairs from unlabelled samples. A significant improvement of 19.15%, 20.56% and 20.23% in the Precision, Recall and F-measure, respectively, was observed for the proposed PUL-based ensemble learning approach. Moreover, 128 drug repositioning candidates were predicted by the proposed methodology. Further, we found literature-based evidence to support five drug combinations that may be able to be repositioned. These discoveries show our proposed PUL approach as a promising strategy that is applicable in drug combination prediction for repositioning.

Detecting diseases in medical prescriptions using data mining methods

Every year, the health of millions of people around the world is compromised by misdiagnosis, which sometimes could even lead to death. In addition, it entails huge financial costs for patients, insurance companies, and governments. Furthermore, many physicians’ professional life is adversely affected by unintended errors in prescribing medication or misdiagnosing a disease. Our aim in this paper is to use data mining methods to find knowledge in a dataset of medical prescriptions that can be effective in improving the diagnostic process. In this study, using 4 single classification algorithms including decision tree, random forest, simple Bayes, and K-nearest neighbors, the disease and its category were predicted. Then, in order to improve the performance of these algorithms, we used an Ensemble Learning methodology to present our proposed model. In the final step, a number of experiments were performed to compare the performance of different data mining techniques. The final model proposed in this study has an accuracy and kappa score of 62.86% and 0.620 for disease prediction and 74.39% and 0.720 for prediction of the disease category, respectively, which has better performance than other studies in this field.In general, the results of this study can be used to help maintain the health of patients, and prevent the wastage of the financial resources of patients, insurance companies, and governments. In addition, it can aid physicians and help their careers by providing timely information on diagnostic errors. Finally, these results can be used as a basis for future research in this field.

AI-Empowered Attack Detection and Prevention Scheme for Smart Grid System

The existing grid infrastructure has already begun transforming into the next-generation cyber-physical smart grid (SG) system. This transformation has improved the grid’s reliability and efficiency but has exposed severe vulnerabilities due to growing cyberattacks and threats. For example, malicious actors may be able to tamper with system readings, parameters, and energy prices and penetrate to get direct access to the data. Several works exist to handle the aforementioned issues, but they have not been fully explored. Consequently, this paper proposes an AI-ADP scheme for the SG system, which is an artificial intelligence (AI)-based attack-detection and prevention (ADP) mechanism by using a cryptography-driven recommender system to ensure data security and integrity. The proposed AI-ADP scheme is divided into two phases: (i) attack detection and (ii) attack prevention. We employed the extreme gradient-boosting (XGBoost) mechanism for attack detection and classification. It is a new ensemble learning methodology that offers many advantages over similar methods, including built-in features, etc. Then, SHA-512 is used to secure the communication that employs faster performance, allowing the transmission of more data with the same security level. The performance of the proposed AI-ADP scheme is evaluated based on various parameters, such as attack-detection accuracy, cycles used per byte, and total cycles used. The proposed AI-ADP scheme outperformed the existing approaches and obtained 99.12% accuracy, which is relatively high compared to the pre-existing methods.

PREDICTING INDONESIA SOVEREIGN DEBT MANAGEMENT’S RISK OCCURRENCE USING MACHINE LEARNING ALGORITHMS

The purpose of developing this predictive model is to predict the potential occurrence of risks in the implementation of projects financed through government loans. It uses data on Indonesian sovereign loans from the MoF DMFAS in the period between 1998 and 2019, consisting of 1930 government loans. Government loan performance is represented by two qualities: timeliness of loan disbursement and rate of loan realization. In the development of the model, the ensemble learning methodology was applied with the voting classifier algorithm. The algorithms used to make predictions for voting classifier include k-NN Classifier, Random Forest, and Logistic Regression. The predictive model developed can accurately predict 73.14% of observations on the rate of disbursement of government loans and is able to predict numerous amendments to drawing limit correctly as much as 89.69% of observations.

Predicting protein-membrane interfaces of peripheral membrane proteins using ensemble machine learning.

Abnormal protein–membrane attachment is involved in deregulated cellular pathways and in disease. Therefore, the possibility to modulate protein–membrane interactions represents a new promising therapeutic strategy for peripheral membrane proteins that have been considered so far undruggable. A major obstacle in this drug design strategy is that the membrane-binding domains of peripheral membrane proteins are usually unknown. The development of fast and efficient algorithms predicting the protein–membrane interface would shed light into the accessibility of membrane–protein interfaces by drug-like molecules. Herein, we describe an ensemble machine learning methodology and algorithm for predicting membrane-penetrating amino acids. We utilize available experimental data from the literature for training 21 machine learning classifiers and meta-classifiers. Evaluation of the best ensemble classifier model accuracy yields a macro-averaged F1 score = 0.92 and a Matthews correlation coefficient = 0.84 for predicting correctly membrane-penetrating amino acids on unknown proteins of a validation set. The python code for predicting protein–membrane interfaces of peripheral membrane proteins is available at https://github.com/zoecournia/DREAMM.

A Mixed Ensemble Learning and Time-Series Methodology for Category-Specific Vehicular Energy and Emissions Modeling

The serially-correlated nature of engine operation is overlooked in the vehicular fuel and emission modeling literature. Furthermore, enabling the calibration and use of time-series models for instrument-independent eco-driving applications requires reliable forecast aggregation procedures. To this end, an ensemble time-series machine-learning methodology is developed using data collected through extensive field experiments on a fleet of 35 vehicles. Among other results, it is found that Long Short-Term Memory (LSTM) architecture is the best fit for capturing the dynamic and lagged effects of speed, acceleration, and grade on fuel and emission rates. The developed vehicle-specific ensembles outperformed state-of-the-practice benchmark models by a significant margin and the category-specific models outscored the vehicle-specific sub-models by an average margin of 6%. The results qualify the developed ensembles to work as representatives for vehicle categories and allows them to be utilized in both eco-driving services as well as environmental assessment modules.

Car-Following Model with Automatic Reaction Delay Estimation: An Attention-Based Ensemble Learning Methodology

Car-following behavior is a vital traffic phenomenon in the process of vehicle driving. For modeling the car-following behavior, it is crucial to capture the reaction delay for balancing with safety and comfort, but it is generally ignored in existing works. This work proposes a car-following model based on attention-based ensemble learning to automatically capture the reaction delay from driving data and better depict the traffic flow characteristics. The model integrates a data-driven model and a theory-driven model, and a weight computation method is proposed to combine the advantage of these two different models. In detail, an encoder-decoder model and attention mechanism are employed to capture the reaction delay from driving data. Extensive experiments show that the proposed model could balance safety with comfort and help avoid unsafe driving behavior.

Анализ метагеосистем полигона «Инерка» посредством ансамблей моделей машинного обучения

The article describes a geoinformation algorithm for interpreting Earth remote sensing data based on the Ensemble Learning methodology. The proposed solution can be used to assess the stability of geosystems and predict natural (including exogeodynamic) processes. The difference of the created approach is determined by a fundamentally new organization scheme of the metaclassifier as a decision-making unit, as well as the use of a geosystem approach to preparing data for automated analysis using deep neural network models. The article shows that the use of ensembles, built according to the proposed method, makes it possible to carry out an operational automated analysis of spatial data for solving the problem of thematic mapping of metageosystems and natural processes. At the same time, combining models into an ensemble based on the proposed architecture of the metaclassifier makes it possible to increase the stability of the analyzing system: the accuracy of decisions made by the ensemble tends to tend to the accuracy of the most efficient monoclassifier of the system. The integration of individual classifiers into ensembles makes it possible to approach the solution of the scientific problem of finding classifier hyperparameters through the combined use of models of the same type with different configurations. The formation of a metaclassifier according to the proposed algorithm is an opportunity to add an element of predictability and control to the use of neural network models, which are traditionally a “black box”. Mapping of the geosystems of the Inerka test site shows their weak resistance to recreational development. The main limiting factors are the composition of Quaternary deposits, the nature of the relief, the mechanical composition of soils, soil moisture, the thickness of the humus horizon of the soil, the genesis and composition of vegetation.

A neural network ensemble approach for GDP forecasting

We propose an ensemble learning methodology to forecast the future US GDP growth release. Our approach combines a Recurrent Neural Network (RNN) and a Dynamic Factor model accounting for time-variation in the mean with a Generalized Autoregressive Score (DFM-GAS). We show how our approach improves forecasts in the aftermath of the 2008-09 global financial crisis by reducing the forecast error for the one-quarter horizon. An exercise on the COVID-19 recession shows a good performance during the economic rebound. Eventually, we provide an interpretable machine learning routine based on integrated gradients to evaluate how the features of the model reflect the evolution of the business cycle.

Marketable value estimation of patents using ensemble learning methodology: Focusing on U.S. patents for the electricity sector.

Patent valuation is required to revitalize patent transactions, but calculating a reasonable value that consumers and suppliers could satisfy is difficult. When machine learning is used, a quantitative evaluation based on a large volume of data is possible, and evaluation can be conducted quickly and inexpensively, contributing to the activation of patent transactions. However, due to patent characteristics, securing the necessary training data is challenging because most patents are traded privately to prevent technical information leaks. In this study, the derived marketable value of a patent through event study is used for patent value evaluation, matching it with the semantic information from the patent calculated using latent Dirichlet allocation (LDA)-based topic modeling. In addition, an ensemble learning methodology that combines the predicted values of multiple predictive models was used to determine the prediction stability. Base learners with high predictive power for each fold were different, but the ensemble model that was trained on the base learners’ predicted values exceeded the predictive power of the individual models. The Wilcoxon rank-sum test indicated that the superiority of the accuracy of the ensemble model was statistically significant at the 95% significance level.

Leveraging colour-based pseudo-labels to supervise saliency detection in hyperspectral image datasets

Saliency detection mimics the natural visual attention mechanism that identifies an imagery region to be salient when it attracts visual attention more than the background. This image analysis task covers many important applications in several fields such as military science, ocean research, resources exploration, disaster and land-use monitoring tasks. Despite hundreds of models have been proposed for saliency detection in colour images, there is still a large room for improving saliency detection performances in hyperspectral imaging analysis. In the present study, an ensemble learning methodology for saliency detection in hyperspectral imagery datasets is presented. It enhances saliency assignments yielded through a robust colour-based technique with new saliency information extracted by taking advantage of the abundance of spectral information on multiple hyperspectral images. The experiments performed with the proposed methodology provide encouraging results, also compared to several competitors.

A Neural Network Ensemble Approach for GDP Forecasting

We propose an ensemble learning methodology to forecast the future US GDP growth release. Our approach combines a Recurrent Neural Network (RNN) with a Dynamic Factor model accounting for time-variation in the mean with a Generalized Autoregressive Score (DFM-GAS). We show how this combination improves forecasts in the aftermath of the 2008-09 global financial crisis, as well as in the latest COVID-19 recession, by reducing the root mean squared error for the short-term forecast horizon. Thus, we evaluate the predictive ability of each component of the ensemble by considering variations in the mean, as the latter are potentially caused by recessions affecting the economy. For our scope, we employ a set of predictors encompassing a wide range of variables measured at different time frequencies. Thus, we provide dynamic coefficients for predictors after an interpretable machine learning routine to assess how the model reflects the evolution of the business cycle.

Docker-Based Intelligent Fall Detection Using Edge-Fog Cloud Infrastructure

Health sector is a life critical domain, which requires fast and intelligent decisions. Artificial intelligence-based monitoring systems can help the elderly people in situations like fall. In e-health, systems are equipped with wearable devices that aid in remote monitoring with the help of Internet of Things (IoT). Our proposed work ensures fall detection using a three-layer architecture (Edge-Fog-Cloud) which takes advantage of the available smart devices. The edge detection involves vision-based detection using a compressed neural network running on a smart device constructed using transfer learning. Decision making in fog involves ensemble learning methodology using sensor-based data and decision from the edge. The cloud is used for permanent storage and model building. The work also takes advantage of image augmentation for data set building to improve the performance of the model. The model is evaluated based on accuracy, and the advantage of the fog layer is evaluated based on the latency. The proposed model gives an accuracy of 98.5% which is compared with the existing state-of-the-art algorithms available for detection.

A Combined Deep Learning and Ensemble Learning Methodology to Avoid Electricity Theft in Smart Grids

Electricity is widely used around 80% of the world. Electricity theft has dangerous effects on utilities in terms of power efficiency and costs billions of dollars per annum. The enhancement of the traditional grids gave rise to smart grids that enable one to resolve the dilemma of electricity theft detection (ETD) using an extensive amount of data formulated by smart meters. This data are used by power utilities to examine the consumption behaviors of consumers and to decide whether the consumer is an electricity thief or benign. However, the traditional data-driven methods for ETD have poor detection performances due to the high-dimensional imbalanced data and their limited ETD capability. In this paper, we present a new class balancing mechanism based on the interquartile minority oversampling technique and a combined ETD model to overcome the shortcomings of conventional approaches. The combined ETD model is composed of long short-term memory (LSTM), UNet and adaptive boosting (Adaboost), and termed LSTM–UNet–Adaboost. In this regard, LSTM–UNet–Adaboost combines the advantages of deep learning (LSTM-UNet) along with ensemble learning (Adaboost) for ETD. Moreover, the performance of the proposed LSTM–UNet–Adaboost scheme was simulated and evaluated over the real-time smart meter dataset given by the State Grid Corporation of China. The simulations were conducted using the most appropriate performance indicators, such as area under the curve, precision, recall and F1 measure. The proposed solution obtained the highest results as compared to the existing benchmark schemes in terms of selected performance measures. More specifically, it achieved the detection rate of 0.92, which was the highest among existing benchmark schemes, such as logistic regression, support vector machine and random under-sampling boosting technique. Therefore, the simulation outcomes validate that the proposed LSTM–UNet–Adaboost model surpasses other traditional methods in terms of ETD and is more acceptable for real-time practices.