Neural Network Ensemble Research Articles

Can eXplainable AI Offer a New Perspective for Groundwater Recharge Estimation?—Global‐Scale Modeling Using Neural Network

AbstractDue to the difficulties in estimating groundwater recharge and cross‐boundary nature of many aquifers, estimating groundwater recharge at large scale has been called upon. Process‐based models as well as data‐driven models have been established to meet this need. Meanwhile, with the advent of explainable artificial intelligence (XAI) methods, data‐driven machine learning models can take advantage of enhanced explainability while keeping the strength of high flexibility. In this study, an ensemble neural network model was built to check the suitability of the model to predict groundwater recharge and the possibility to gain new insights from large data set. Recent large inputs of groundwater recharge data and additional input for the Arabian Peninsula collated in this study were fed to the model with multiple predictors related to climatology considering seasonality, soil and plant characteristics, topography, and hydrogeology. The model showed higher performance (adjusted R2: 0.702, RMSE: 193.35 mm yr−1) than a recent global process‐based model in predicting groundwater recharge. Using XAI methods as individual conditional expectations and Shapley Additive Explanation interaction values, the model behavior was analyzed and possible linear and non‐linear relationships between the predictors and the groundwater recharge rate were found. Long‐term averaged precipitation and enhanced vegetation index showed non‐linear relationships with groundwater recharge rate, while slope, compound topographic index, and water table depth showed low importance to the model results. Most model behaviors followed the domain knowledge, while multi‐correlation between predictors and data skewness hindered the model from learning.

Enhancing Pneumonia Diagnosis Through Chest Imaging and Machine Learning A Comprehensive Ensemble Approach with Diverse CNN Architectures

Abstract: This research presents a novel approach for pneumonia diagnosis in chest X-ray images utilizing an ensemble of convolutional neural networks (CNNs). The proposed system integrates state-of-the-art architectures such as ResNet, DenseNet, InceptionV3, MobileNet, and AlexNet, helping transfer learning to fine-tune these models on a curated chest X-ray dataset obtained from Kaggle. The dataset comprises two classes: normal and pneumonia. The ensemble methodology combines the predictive strengths of individual CNN models, harnessing their diverse feature extraction capabilities. A key innovation lies in the incorporation of the AlexNet architecture into the ensemble, aiming to further enhance the ensemble's discriminative power. The system undergoes a comprehensive training, validation, and testing pipeline, culminating in real-time predictions on new chest X-ray images. The experimental results showcase the effectiveness of the ensemble approach, demonstrating improved accuracy and robustness in pneumonia detection compared to individual models. The incorporation of AlexNet contributes unique features to the ensemble, ighlighting the potential of diverse model architectures in enhancing diagnostic performance

Using an Ensemble of Neural Networks for Determining the Diagnostic Parameters of the Vertebrae

Artificial intelligence opens up great prospects in many areas of human activity, primarily in medicine. One of the priority directions of using artificial intelligence in this field is the segmentation of medical images for the purpose of automatic diagnosis of common diseases. The application of neural network approaches to image analysis of medical images is becoming an increasingly promising direction in the field of medical diagnostics. In particular, this paper investigates the possibility of using an ensemble of neural networks for diagnosing osteoporosis. To achieve this goal, a study was conducted on the possibility of using machine learning methods to segment and determine the shape and size of certain vertebrae: Th8, Th9, Th10, Th11 of a human vertebra on X-ray images obtained in real conditions. Each network is configured and tested on different sets of medical images. Then, the two best networks were selected according to the accuracy and efficiency of the segmentation. One of the main results of the study was the selection of the two best neural networks that provide the most accurate segmentation of vertebrae. Next, the ensemble method was applied, based on the averaging of the predictions of the selected networks. This approach made it possible to improve the overall accuracy of determining the diagnostic parameters of the spine. The obtained results emphasize the effectiveness of using an ensemble of neural networks in the context of medical segmentation. Ensembles provide more stable and accurate predictions by reducing the impact of random errors of individual networks. Ensemble predictions of these networks lead to a statistically significant improvement in results compared to individual approaches. This is an important step in the direction of creating reliable systems of automated diagnostics capable of helping doctors in conducting more accurate and operative analyses.

Deep Learning for Real-time Affective Hand Gesture Recognition in EMASPEL

This research marks a transformative leap in personalized learning through real-time affective hand gesture recognition in EMASPEL (Emotional Multi-Agents System for Peer-to-peer E-Learning), an educational platform. Our deep learning model, a meticulously crafted ensemble of convolutional and recurrent neural networks, deciphers the unspoken language of emotions embedded within student gestures, accurately capturing both spatial and temporal patterns. This detailed emotional map empowers EMASPEL to tailor its interactions with exquisite precision, addressing frustration, nurturing curiosity, and maximizing student engagement. The impact is profound: students flourish in personalized learning environments, experiencing enhanced outcomes and a newfound connection to their educational journey. Teachers, equipped with real-time emotional insights, provide targeted support and cultivate a more inclusive, responsive classroom. Beyond gestures, we envision a future enriched by multimodal data integration, encompassing facial expressions, voice analysis, and potentially physiological sensors, to paint even richer portraits of student emotions and cognitive states. Continuous refinement through rigorous longitudinal studies will pave the way for deeper understanding and ensure responsible implementation. Ultimately, this research reimagines education as a dynamic ensemble of personalized learning, where technology serves as a bridge between teacher and student, unlocking not just academic success but a lifelong love of knowledge.

A Comparison Study of Deep Learning Methodologies for Music Emotion Recognition.

Classical machine learning techniques have dominated Music Emotion Recognition. However, improvements have slowed down due to the complex and time-consuming task of handcrafting new emotionally relevant audio features. Deep learning methods have recently gained popularity in the field because of their ability to automatically learn relevant features from spectral representations of songs, eliminating such necessity. Nonetheless, there are limitations, such as the need for large amounts of quality labeled data, a common problem in MER research. To understand the effectiveness of these techniques, a comparison study using various classical machine learning and deep learning methods was conducted. The results showed that using an ensemble of a Dense Neural Network and a Convolutional Neural Network architecture resulted in a state-of-the-art 80.20% F1 score, an improvement of around 5% considering the best baseline results, concluding that future research should take advantage of both paradigms, that is, combining handcrafted features with feature learning.

A Hybrid Approach for the Ensembles of Neural Networks for Solar Power Forecasting

In this study, a unique hybrid method to solar power forecasting is presented. This strategy makes use of ensemble approaches, which include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory networks (LSTMs). A more accurate prediction of solar power production may be achieved using the hybrid ensemble, which takes advantage of the specific characteristics of each neural network design. Extensive trials conducted using datasets taken from the actual world demonstrate that the suggested technique is superior to both individual neural network models and conventional methods of predicting. Therefore, the ensemble is a feasible option for practical solar power forecasting applications, helping to the effective utilisation of renewable energy resources. Its flexibility to changing weather conditions, interpretability, and superior accuracy position it as a promising solution.

Prediction of multi-stage froth flotation efficiency of complex lead–zinc sulfide ore using an integrated ensemble neural network–random forest model

Despite the long history of flotation in the mineral processing industry, its prediction and understanding remains a great challenge owing to its many variables acting in a complex manner. In this study, we introduced machine learning (ML) models to predict the grade and yield of a multi-stage flotation process of a complex lead–zinc sulfide ore. Over 100 batch flotation tests were conducted in a stepwise manner to characterize different rougher-cleaner-scavenger configurations. Performing a pre-flotation of talc prior to sulfide flotation remarkably improved the grade of lead concentrate. The experimental data were divided into four subsets for the ML models: rougher without pre-flotation, rougher with pre-flotation, cleaner, and cleaner-scavenger datasets. Different ML models were evaluated to determine whether they could predict the lead grade, zinc grade, and yield related to key flotation parameters, including particle size, reagent dosage, and pulp pH. The integrated ensemble neural network and random forest model yielded the best prediction results with R2 values of 0.924, 0.902, 0.973, and 0.894 for the rougher without pre-flotation, rougher with pre-flotation, cleaner, and cleaner-scavenger subsets, respectively. The developed ML model, with the connection of subset models, effectively predicted the flotation outcome of the rougher-cleaner-scavenger circuit, demonstrating a better prediction performance than previous methods. This indicates that the developed ML model can potentially predict flotation process performance and evaluate the efficiency of newly designed multi-stage froth flotation processes.

A Survey on Lung Cancer Detection Using Ensemble Neural Networks

A Survey on Lung Cancer Detection Using Ensemble Neural Networks

Computer-Aided Diagnosis of Maxillary Sinus Anomalies: Validation and Clinical Correlation.

Computer aided diagnostics (CAD) systems can automate the differentiation of maxillary sinus (MS) with and without opacification, simplifying the typically laborious process and aiding in clinical insight discovery within large cohorts. This study uses Hamburg City Health Study (HCHS) a large, prospective, long-term, population-based cohort study of participants between 45 and 74 years of age. We develop a CAD system using an ensemble of 3D Convolutional Neural Network (CNN) to analyze cranial MRIs, distinguishing MS with opacifications (polyps, cysts, mucosal thickening) from MS without opacifications. The system is used to find correlations of participants with and without MS opacifications with clinical data (smoking, alcohol, BMI, asthma, bronchitis, sex, age, leukocyte count, C-reactive protein, allergies). The evaluation metrics of CAD system (Area Under Receiver Operator Characteristic: 0.95, sensitivity: 0.85, specificity: 0.90) demonstrated the effectiveness of our approach. MS with opacification group exhibited higher alcohol consumption, higher BMI, higher incidence of intrinsic asthma and extrinsic asthma. Male sex had higher prevalence of MS opacifications. Participants with MS opacifications had higher incidence of hay fever and house dust allergy but lower incidence of bee/wasp venom allergy. The study demonstrates a 3D CNN's ability to distinguish MS with and without opacifications, improving automated diagnosis and aiding in correlating clinical data in population studies. 3 Laryngoscope, 134:3927-3934, 2024.

DiagSet: a dataset for prostate cancer histopathological image classification

Cancer diseases constitute one of the most significant societal challenges. In this paper, we introduce a novel histopathological dataset for prostate cancer detection. The proposed dataset, consisting of over 2.6 million tissue patches extracted from 430 fully annotated scans, 4675 scans with assigned binary diagnoses, and 46 scans with diagnoses independently provided by a group of histopathologists can be found at https://github.com/michalkoziarski/DiagSet. Furthermore, we propose a machine learning framework for detection of cancerous tissue regions and prediction of scan-level diagnosis, utilizing thresholding to abstain from the decision in uncertain cases. The proposed approach, composed of ensembles of deep neural networks operating on the histopathological scans at different scales, achieves 94.6% accuracy in patch-level recognition and is compared in a scan-level diagnosis with 9 human histopathologists showing high statistical agreement.

Ensemble of artificial neural networks weighted by operational characteristics for prognosis of acute heart failure

Ensemble of artificial neural networks weighted by operational characteristics for prognosis of acute heart failure

Quantifying predictive uncertainty in damage classification for nondestructive evaluation using Bayesian approximation and deep learning

Magnetic flux leakage (MFL), a widely used nondestructive evaluation (NDE) method, for inspecting pipelines to prevent potential long-term failures. However, during field testing, uncertainties can affect the accuracy of the inspection and the decision-making process regarding damage conditions. Therefore, it is essential to identify and quantify these uncertainties to ensure the reliability of the inspection. This study focuses on the uncertainties that arise during the inverse NDE process due to the dynamic magnetization process, which is affected by the relative motion of the MFL sensor and the material being tested. Specifically, the study investigates the uncertainties caused by sensing liftoff, which can affect the output signal of the sensing system. Due to the complexity of describing the forward uncertainty propagation process, this study compared two typical machine learning (ML)-based approximate Bayesian inference methods, convolutional neural network and deep ensemble, to address the input uncertainty from the MFL response data. Besides, an autoencoder method is applied to tackle the lack of experimental data for the training model by augmenting the dataset, which is constructed with the pre-trained model based on transfer learning. Prior knowledge learned from large simulated MFL signals can fine-tune the autoencoder model which enhances the subsequent learning process on experimental MFL data with faster generalization. The augmented data from the fine-tuned autoencoder is further applied for ML-based defect size classification. This study conducted prediction accuracy and uncertainty analysis with calibration, which can evaluate the prediction performance and reveal the relation between the liftoff uncertainty and prediction accuracy. Further, to strengthen the trustworthiness of the prediction results, the decision-making process guided by uncertainty is applied to provide valuable insights into the reliability of the final prediction results. Overall, the proposed framework for uncertainty quantification offers valuable insights into the assessment of reliability in MFL-based decision-making and inverse problems.

Space–time landslide hazard modeling via Ensemble Neural Networks

Abstract. Until now, a full numerical description of the spatio-temporal dynamics of a landslide could be achieved only via physically based models. The part of the geoscientific community in developing data-driven models has instead focused on predicting where landslides may occur via susceptibility models. Moreover, they have estimate when landslides may occur via models that belong to the early-warning system or to the rainfall-threshold classes. In this context, few published research works have explored a joint spatio-temporal model structure. Furthermore, the third element completing the hazard definition, i.e., the landslide size (i.e., areas or volumes), has hardly ever been modeled over space and time. However, technological advancements in data-driven models have reached a level of maturity that allows all three components to be modeled (Location, Frequency, and Size). This work takes this direction and proposes for the first time a solution to the assessment of landslide hazard in a given area by jointly modeling landslide occurrences and their associated areal density per mapping unit, in space and time. To achieve this, we used a spatio-temporal landslide database generated for the Nepalese region affected by the Gorkha earthquake. The model relies on a deep-learning architecture trained using an Ensemble Neural Network, where the landslide occurrences and densities are aggregated over a squared mapping unit of 1 km × 1 km and classified or regressed against a nested 30 m lattice. At the nested level, we have expressed predisposing and triggering factors. As for the temporal units, we have used an approximately 6 month resolution. The results are promising as our model performs satisfactorily both in the susceptibility (AUC = 0.93) and density prediction (Pearson r = 0.93) tasks over the entire spatio-temporal domain. This model takes a significant distance from the common landslide susceptibility modeling literature, proposing an integrated framework for hazard modeling in a data-driven context.

Visual acuity prediction on real-life patient data using a machine learning based multistage system

In ophthalmology, intravitreal operative medication therapy (IVOM) is a widespread treatment for diseases related to the age-related macular degeneration (AMD), the diabetic macular edema, as well as the retinal vein occlusion. However, in real-world settings, patients often suffer from loss of vision on time scales of years despite therapy, whereas the prediction of the visual acuity (VA) and the earliest possible detection of deterioration under real-life conditions is challenging due to heterogeneous and incomplete data. In this contribution, we present a workflow for the development of a research-compatible data corpus fusing different IT systems of the department of ophthalmology of a German maximum care hospital. The extensive data corpus allows predictive statements of the expected progression of a patient and his or her VA in each of the three diseases. For the disease AMD, we found out a significant deterioration of the visual acuity over time. Within our proposed multistage system, we subsequently classify the VA progression into the three groups of therapy “winners”, “stabilizers”, and “losers” (WSL classification scheme). Our OCT biomarker classification using an ensemble of deep neural networks results in a classification accuracy (F1-score) of over 98%, enabling us to complete incomplete OCT documentations while allowing us to exploit them for a more precise VA modelling process. Our VA prediction requires at least four VA examinations and optionally OCT biomarkers from the same time period to predict the VA progression within a forecasted time frame, whereas our prediction is currently restricted to IVOM/no therapy. We achieve a final prediction accuracy of 69% in macro average F1-score, while being in the same range as the ophthalmologists with 57.8 and 50±10.7\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$50 \\pm 10.7$$\\end{document}% F1-score.

Comparison of individual and ensemble machine learning models for prediction of sulphate levels in untreated and treated Acid Mine Drainage

Machine learning was used to provide data for further evaluation of potential extraction of octathiocane (S8), a commercially useful by-product, from Acid Mine Drainage (AMD) by predicting sulphate levels in an AMD water quality dataset. Individual ML regressor models, namely: Linear Regression (LR), Least Absolute Shrinkage and Selection Operator (LASSO), Ridge (RD), Elastic Net (EN), K-Nearest Neighbours (KNN), Support Vector Regression (SVR), Decision Tree (DT), Extreme Gradient Boosting (XGBoost), Random Forest (RF), Multi-Layer Perceptron Artificial Neural Network (MLP) and Stacking Ensemble (SE-ML) combinations of these models were successfully used to predict sulphate levels. A SE-ML regressor trained on untreated AMD which stacked seven of the best-performing individual models and fed them to a LR meta-learner model was found to be the best-performing model with a Mean Squared Error (MSE) of 0.000011, Mean Absolute Error (MAE) of 0.002617 and R2 of 0.9997. Temperature (°C), Total Dissolved Solids (mg/L) and, importantly, iron (mg/L) were highly correlated to sulphate (mg/L) with iron showing a strong positive linear correlation that indicated dissolved products from pyrite oxidation. Ensemble learning (bagging, boosting and stacking) outperformed individual methods due to their combined predictive accuracies. Surprisingly, when comparing SE-ML that combined all models with SE-ML that combined only the best-performing models, there was only a slight difference in model accuracies which indicated that including bad-performing models in the stack had no adverse effect on its predictive performance.

A Novel Explainable AI Model for Medical Data Analysis

Abstract This research focuses on the development of an explainable artificial intelligence (Explainable AI or XAI) system aimed at the analysis of medical data. Medical imaging and related datasets present inherent complexities due to their high-dimensional nature and the intricate biological patterns they represent. These complexities necessitate sophisticated computational models to decode and interpret, often leading to the employment of deep neural networks. However, while these models have achieved remarkable accuracy, their ”black-box” nature raises legitimate concerns regarding their interpretability and reliability in the clinical context. To address this challenge, we can consider the following approaches: traditional statistical methods, a singular complex neural network, or an ensemble of simpler neural networks. Traditional statistical methods, though transparent, often lack the nuanced sensitivity required for the intricate patterns within medical images. On the other hand, a singular complex neural network, while powerful, can sometimes be too generalized, making specific interpretations challenging. Hence, our proposed strategy employs a hybrid system, combining multiple neural networks with distinct architectures, each tailored to address specific facets of the medical data interpretation challenges. The key components of this proposed technology include a module for anomaly detection within medical images, a module for categorizing detected anomalies into specific medical conditions and a module for generating user-friendly, clinically-relevant interpretations.

E2ETCA: End-to-end training of CNN and attention ensembles for rice disease diagnosis1

Rice is one of the most important crops worldwide. Diseases of the rice plant can drastically reduce crop yield and even lead to complete loss of production. Early diagnosis can reduce the severity and help efforts to establish effective treatment and reduce the usage of pesticides. Traditional machine learning approaches have already been employed for automatic diagnosis. However, they heavily rely on manual preprocessing of images and handcrafted features, which is challenging, time-consuming, and may require domain expertise. Recently, a single end-to-end deep learning (DL)-based approach was employed to diagnose rice diseases. However, it is not highly robust, nor is it generalizable to every dataset. Hence, we propose a novel end-to-end training of convolutional neural network (CNN) and attention (E2ETCA) ensemble framework that fuses the features of two CNN-based state-of-the-art (SOTA) models along with those of an attention-based vision transformer model. These fused features are utilized for diagnosis by the addition of an extra fully connected layer with softmax. The whole procedure is performed end-to-end, which is very important for real-world applications. Additionally, we feed the extracted features into a traditional machine learning approach support vector machine for classification and further analysis. To verify the effectiveness of our proposed E2ETCA framework, we demonstrate it on three publicly available datasets: the Mendeley Rice Leaf Disease Image Samples dataset, the Kaggle Rice Diseases Image dataset, the Bangladesh Rice Research Institute dataset, and a combination of these three datasets. On the basis of various evaluation metrics (accuracy, precision, recall, and F1-score), our proposed E2ETCA framework exhibits superior performance to existing SOTA approaches for rice disease diagnosis, which can also be generalizable in similar other domains.

Vulnerable JavaScript functions detection using stacking of convolutional neural networks.

System security for web-based applications is paramount, and for the avoidance of possible cyberattacks it is important to detect vulnerable JavaScript functions. Developers and security analysts have long relied upon static analysis to investigate vulnerabilities and faults within programs. Static analysis tools are used for analyzing a program's source code and identifying sections of code that need to be further examined by a human analyst. This article suggests a new approach for identifying vulnerable code in JavaScript programs by using ensemble of convolutional neural networks (CNNs) models. These models use vulnerable information and code features to detect related vulnerable code. For identifying different vulnerabilities in JavaScript functions, an approach has been tested which involves the stacking of CNNs with misbalancing, random under sampler, and random over sampler. Our approach uses these CNNs to detect vulnerable code and improve upon current techniques' limitations. Previous research has introduced several approaches to identify vulnerable code in JavaScript programs, but often have their own limitations such as low accuracy rates and high false-positive or false-negative results. Our approach addresses this by using the power of convolutional neural networks and is proven to be highly effective in the detection of vulnerable functions that could be used by cybercriminals. The stacked CNN approach has an approximately 98% accuracy, proving its robustness and usability in real-world scenarios. To evaluate its efficacy, the proposed method is trained using publicly available JavaScript blocks, and the results are assessed using various performance metrics. The research offers a valuable insight into better ways to protect web-based applications and systems from potential threats, leading to a safer online environment for all.

Accurate, Uncertainty-Aware Classification of Molecular Chemical Motifs from Multimodal X-ray Absorption Spectroscopy.

Accurate classification of molecular chemical motifs from experimental measurement is an important problem in molecular physics, chemistry, and biology. In this work, we present neural network ensemble classifiers for predicting the presence (or lack thereof) of 41 different chemical motifs on small molecules from simulated C, N, and O K-edge X-ray absorption near-edge structure (XANES) spectra. Our classifiers not only achieve class-balanced accuracies of more than 0.95 but also accurately quantify uncertainty. We also show that including multiple XANES modalities improves predictions notably on average, demonstrating a "multimodal advantage" over any single modality. In addition to structure refinement, our approach can be generalized to broad applications with molecular design pipelines.

Dynamic ensemble of regression neural networks based on predictive uncertainty

In a data-decentralized environment where data are distributed across individual local nodes, inter-node data sharing is often restricted due to security concerns and communication costs. Under these restrictions, using entire data to train a global predictive model is challenging. A viable strategy is to leverage local predictive models trained independently on local nodes. However, this adversely affect generalization performance, particularly when the data distribution varies between local nodes. To address this issue in regression problems, we propose a dynamic ensemble method that combines the predictions of locally trained neural networks without the need to access data from local nodes. For a query instance, the prediction and its uncertainty from each neural network are computed by applying Monte Carlo dropout. Subsequently, smaller/larger weights are assigned to neural networks with higher/lower predictive uncertainty. The final prediction result is derived using the weighted average of the individual predictions. The proposed method is applicable to a data-decentralized environment where data are inaccessible and thus cannot be shared across local nodes. Transmitting query instances to local nodes and receiving predictions and weights from the local nodes are the only requirements, thereby ensuring low communication costs. Experimental results on simulated data-decentralized environments using benchmark regression datasets validated the superior generalization performance of the proposed method compared with baseline methods. The proposed method was found to be more effective for regression neural networks that demonstrate higher performance at their respective local nodes.