Deep Ensemble Research Articles

An Approach for Egg Parasite Classification Based on Ensemble Deep Learning

Opisthorchis viverrini and minute intestinal fluke (MIF) infections are heavily epidemic in northeastern Thailand. Their primary cause is eating raw or undercooked cyprinid fishes, and they cause health problems in the human digestive system. In cases of liver fluke, these parasites can go through the bile duct system, which may cause cholangiocarcinoma (bile duct cancer). When a medical doctor suspects that a patient is infected with parasites, they typically request a stool analysis to determine the type of egg parasites using microscopy. Both parasites have similar characteristics, thus, it is necessary for a specialist to identify the specific type of egg parasites present. Many automatic systems have been developed using deep learning to assist doctors in diagnosing the type of egg parasite. In this study, we proposed three models of deep learning architectures and created voting ensembles to analyze egg parasite images. Images of similar liver fluke eggs and MIF eggs were taken from the Parasitology Laboratory, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand. Image data augmentation is used to expand images from different perspectives and assist the system in acquiring a greater variety of images. Three models performed effectively, by employing the hard voting ensemble, the accuracy increased to 86.67%, while for the second group, the accuracies reached 68.00%, 76.00%, and 77.33%, respectively. Using the soft voting ensemble, the accuracy improved to 79.33%. These outcomes highlight the potential of ensemble deep learning in image classification. Furthermore, these results align closely with those achieved by several experts in image classification. Hence, a promising ensemble approach can aid doctors in accurately classifying images of egg parasites.

Ensemble Deep Learning Approach for Turbidity Prediction of Dooskal Lake Using Remote Sensing Data

Ensemble Deep Learning Approach for Turbidity Prediction of Dooskal Lake Using Remote Sensing Data

Looking at the posterior: accuracy and uncertainty of neural-network predictions

Bayesian inference can quantify uncertainty in the predictions of neural networks using posterior distributions for model parameters and network output. By looking at these posterior distributions, one can separate the origin of uncertainty into aleatoric and epistemic contributions. One goal of uncertainty quantification is to inform on prediction accuracy. Here we show that prediction accuracy depends on both epistemic and aleatoric uncertainty in an intricate fashion that cannot be understood in terms of marginalized uncertainty distributions alone. How the accuracy relates to epistemic and aleatoric uncertainties depends not only on the model architecture, but also on the properties of the dataset. We discuss the significance of these results for active learning and introduce a novel acquisition function that outperforms common uncertainty-based methods. To arrive at our results, we approximated the posteriors using deep ensembles, for fully-connected, convolutional and attention-based neural networks.

A novel hybrid model in the diagnosis and classification of Alzheimer's disease using EEG signals: Deep ensemble learning (DEL) approach

Recent years have witnessed a surge of sophisticated computer-aided diagnosis techniques involving Artificial Intelligence (AI) to accurately diagnose and classify Alzheimer’s disease (AD) and other forms of Dementia. Despite these advancements, there is still a lack of reliable and accurate methods for distinguishing between (AD) and Healthy Controls (HC) using Electroencephalography signals (EEG). The main challenge is finding the right features from the intricate spectral-temporal EEG data, which can provide information sufficient for diagnosis. This study proposes a new approach integrating Deep Ensemble Learning (DEL) and 2-dimensional Convolutional Neural Networks (2D-CNN) to address these issues. Combining state-of-the-art supervised deep learning algorithms within an ensemble model architecture aims to accurately diagnose and classify EEG signals of AD and HC subjects. Public EEG-based Alzheimer's datasets have been classified in the DEL model without applying any feature extraction after cleaning from noise and artifacts.Furthermore, the proposed DEL model used 5 different 2D-CNN models as internal classifiers. As a result, the EEG-based DEL model proposed for the first time provided high accuracy in AD classification. The proposed DEL model reached an average accuracy of 97.9% in AD classification due to 5 cross-fold training.In conclusion, this work renders that incorporating ensemble learning techniques into automotive health applications create extensible and stable AI models needed for computer-aided diagnostic. However, although the reported results and evaluation are promising,further efforts will need to be made to improve the accuracy of our proposed model. In addition, a fine-grid evaluation will be necessary to accurately understand potential impacts in clinical applications, such as earlier diagnosis or treatment decisions.

Smart Parking System: Optimized Ensemble Deep Learning Model with Internet of Things for Smart Cities

In the recent era of smart city ecosystems and the Internet of Things (IoT), innovative, intelligent parking systems must make cities more sustainable. Every year, the increasing number of city vehicles requires more time to search for parking slots. In large cities, 10\% of the traffic congestion occurs because of cruising; drivers spend almost 20 minutes searching for free space to park their vehicles. The passing time of waiting for parking in the traffic leads the issues such as energy, pollution, and stress. There needs to be more than the developed solutions. Therefore, the necessary to create a parking slot availability detection system that informs the drivers in advance about the free parking slot based on location. This paper introduces an enhanced ensemble Deep Learning (DL) model designed to forecast parking slot availability through the integration of IoT, cloud technology, and sensor networks. The devised model, known as Ensemble CNN-Boosted Graph LSTM (ECNN-BGLSTM), is optimized using a Genetic Algorithm (GA) framework. The model's performance is rigorously evaluated using a dataset from Europe, and various metrics, including Root Mean Square Error (RMSE), Mean Square Error (MSE), and Mean Absolute Error (MAE), are employed for assessment. The experimental findings demonstrate the superior performance of the proposed model compared to existing state-of-the-art approaches.

Quantifying uncertainty in soil moisture retrieval using a Bayesian neural network framework

Soil moisture (SM) is an important parameter for precision agriculture and water cycle. Recent studies of using Global Navigation Satellite System-Reflectometry (GNSS-R) to retrieve SM have shown that the joint use of data and models is an effective method. However, data and models are often accompanied by two types of uncertainties (i.e., data uncertainty and model uncertainty), which can lead to retrieved results with high uncertainty resulting in false reliability. In this study, we propose a new Bayesian neural network (BNN) framework composed of two techniques: Monte-Carlo (MC)-dropout and Deep Ensembles. It quantifies the uncertainty and feeds it back to the framework to ultimately reduce the uncertainty in the retrieved results and improve accuracy. To verify the proposed framework, we conduct experiments using in situ data sets and Global Land Data Assimilation System (GLDAS) data sets respectively. The results show that MC-dropout can improve the correlation coefficient (R) and root-mean-square error (RMSE) by 8%∼21% and 7%∼20%, compared to basic multilayer perceptron (MLP). Deep Ensembles can improve the R and RMSE by 9%∼36% and 11%∼25%. Concerning uncertainty, MC-dropout and Deep Ensembles can decrease the uncertainty by 37%∼60% and 82%∼84% relative to maximum a posteriori (MAP), respectively. This study demonstrates that BNN framework can be used to quantify the uncertainty in retrieving SM and reduce the uncertainty as well as improve the accuracy.

Cloud-Based Quad Deep Ensemble Framework for the Detection of COVID-19 Omicron and Delta Variants.

The mortality rates of patients contracting the Omicron and Delta variants of COVID-19 are very high, and COVID-19 is the worst variant of COVID. Hence, our objective is to detect COVID-19 Omicron and Delta variants from lung CT-scan images. We designed a unique ensemble model that combines the CNN architecture of a deep neural network-Capsule Network (CapsNet)-and pre-trained architectures, i.e., VGG-16, DenseNet-121, and Inception-v3, to produce a reliable and robust model for diagnosing Omicron and Delta variant data. Despite the solo model's remarkable accuracy, it can often be difficult to accept its results. The ensemble model, on the other hand, operates according to the scientific tenet of combining the majority votes of various models. The adoption of the transfer learning model in our work is to benefit from previously learned parameters and lower data-hunger architecture. Likewise, CapsNet performs consistently regardless of positional changes, size changes, and changes in the orientation of the input image. The proposed ensemble model produced an accuracy of 99.93%, an AUC of 0.999 and a precision of 99.9%. Finally, the framework is deployed in a local cloud web application so that the diagnosis of these particular variants can be accomplished remotely.

Deep ensemble geophysics-informed neural networks for the prediction of celestial pole offsets

SUMMARYCelestial Pole Offsets (CPO), denoted by dX and dY, describe the differences in the observed position of the pole in the celestial frame with respect to a certain precession-nutation model. Precession and nutation components are part of the transformation matrix between terrestrial and celestial systems. Therefore, various applications in geodetic science such as high-precision spacecraft navigation require information regrading precession and nutation. For this purpose, CPO can be added to the precession-nutation model to precisely describe the motion of the celestial pole. However, as Very Long Baseline Interferometry (VLBI)—currently the only technique providing CPO—requires long data processing times resulting in several weeks of latency, predictions of CPO become necessary. Here we present a new methodology named Deep Ensemble Geophysics-Informed Neural Networks (DEGINNs) to provide accurate CPO predictions. The methodology has three main elements: (1) deep ensemble learning to provide the prediction uncertainty; (2) broad-band Liouville equation as a geophysical constraint connecting the rotational dynamics of CPO to the atmospheric and oceanic Effective Angular Momentum (EAM) functions and (3) coupled oscillatory recurrent neural networks to model the sequential characteristics of CPO time-series, also capable of handling irregularly sampled time-series. To test the methodology, we use the newest version of the final CPO time-series of International Earth Rotation and Reference Systems Service (IERS), namely IERS 20 C04. We focus on a forecasting horizon of 90 days, the practical forecasting horizon needed in space-geodetic applications. Furthermore, for validation purposes we generate an independent global VLBI solution for CPO since 1984 up to the end of 2022 and analyse the series. We draw the following conclusions. First, the prediction performance of DEGINNs demonstrates up to 25 and 33 percent improvement, respectively, for dX and dY, with respect to the rapid data provided by IERS. Secondly, predictions made with the help of EAM are more accurate compared to those without EAM, thus providing a clue to the role of atmosphere and ocean on the excitation of CPO. Finally, free core nutation period shows temporal variations with a dominant periodicity of around one year, partially excited by EAM.

Deep Ensemble Classification with Self Improved Optimization for Attack Detection Towards Secured Virtualization in Cloud

Due to the extraordinary advances in tech companies and development in cloud computing, it has turned out to be ever more familiar for companies to integrate virtualization in data centers to entirely exploit their hardware sources. Accordingly, virtualization and security have gone under varied changes recently. Virtualization and its exceptional design have numerous advantages and characteristics over conventional non-virtualized technologies. Nevertheless, these novel characters form novel vulnerabilities and probable attacks in virtualized systems. Many recent studies have developed to detect those attacks, but they still have challenges like, need to invest more, if it must be done fast, data security could be at danger and scalability could be difficult. As a result, the primary goal of this study is to detect attack for getting secured virtualization in cloud. Therefore, the purpose of this research is to suggest a new, two-stage, protected virtualization paradigm for the cloud. At first, the “higher-order statistical features, flow-based features, mutual information (MI) based features and improved correlation features” are derived. Deep convolutional neural network (DCNN) 1 and 2 are then used to classify the derived features. To detect network assaults, LSTM 1 and 2 are applied to the outputs from DCNN 1 and 2. Here, the Self Customized Aquila Optimizer is used to adjust the DCNN weights to their optimum level (SC-AO). Additionally, analysis is performed using a range of metrics. Particularly, the proposed HC + SC-AO scheme attained improved accuracy of 0.938108 and 0.958249 for both the datasets respectively.

A unified benchmark for the unknown detection capability of deep neural networks

Deep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: Over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out these unknown samples, but they only considered narrow and specific tasks, referred to as misclassification detection, open-set recognition, or out-of-distribution detection. In this work, we argue that these tasks should be treated as fundamentally an identical problem because an ideal model should possess detection capability for all those tasks. Therefore, we introduce the unknown detection task, an integration of previous individual tasks, for a rigorous examination of the detection capability of deep neural networks on a wide spectrum of unknown samples. To this end, unified benchmark datasets on different scales were constructed and the unknown detection capabilities of existing popular methods were subject to comparison. We found that Deep Ensemble consistently outperforms the other approaches in detecting unknowns; however, all methods are only successful for a specific type of unknown. The reproducible code and benchmark datasets are available at https://github.com/daintlab/unknown-detection-benchmarks.

Enhanced Classification of Diabetic Retinopathy via Vessel Segmentation: A Deep Ensemble Learning Approach

Enhanced Classification of Diabetic Retinopathy via Vessel Segmentation: A Deep Ensemble Learning Approach

Real-world post-deployment performance of a novel machine learning-based digital health technology for skin lesion assessment and suggestions for post-market surveillance.

Deep Ensemble for Recognition of Malignancy (DERM) is an artificial intelligence as a medical device (AIaMD) tool for skin lesion assessment. We report prospective real-world performance from its deployment within skin cancer pathways at two National Health Service hospitals (UK) between July 2021 and October 2022. A total of 14,500 cases were seen, including patients 18-100 years old with Fitzpatrick skin types I-VI represented. Based on 8,571 lesions assessed by DERM with confirmed outcomes, versions A and B demonstrated very high sensitivity for detecting melanoma (95.0-100.0%) or malignancy (96.0-100.0%). Benign lesion specificity was 40.7-49.4% (DERM-vA) and 70.1-73.4% (DERM-vB). DERM identified 15.0-31.0% of cases as eligible for discharge. We show DERM performance in-line with sensitivity targets and pre-marketing authorisation research, and it reduced the caseload for hospital specialists in two pathways. Based on our experience we offer suggestions on key elements of post-market surveillance for AIaMDs.

Comparison of Different Methods for Building Ensembles of Convolutional Neural Networks

In computer vision and image analysis, Convolutional Neural Networks (CNNs) and other deep-learning models are at the forefront of research and development. These advanced models have proven to be highly effective in tasks related to computer vision. One technique that has gained prominence in recent years is the construction of ensembles using deep CNNs. These ensembles typically involve combining multiple pretrained CNNs to create a more powerful and robust network. The purpose of this study is to evaluate the effectiveness of building CNN ensembles by combining several advanced techniques. Tested here are CNN ensembles constructed by replacing ReLU layers with different activation functions, employing various data-augmentation techniques, and utilizing several algorithms, including some novel ones, that perturb network weights. Experimental results performed across many datasets representing different tasks demonstrate that our proposed methods for building deep ensembles produces superior results.

Malware Attack Detection in Large Scale Networks using the Ensemble Deep Restricted Boltzmann Machine

Today, cyber attackers use Artificial Intelligence (AI) to boost the sophistication and scope of their attacks. On the defense side, AI is used to improve defense plans, robustness, flexibility, and efficiency of defense systems by adapting to environmental changes. With the developments in information and communication technologies, various exploits that are changing rapidly constitute a danger sign for cyber security. Cybercriminals use new and sophisticated tactics to boost their attack speed and size. Consequently, there is a need for more flexible, adaptable, and strong cyber defense systems that can identify a wide range of threats in real time. In recent years, the adoption of AI approaches has increased and maintained a vital role in the detection and prevention of cyber threats. This paper presents an Ensemble Deep Restricted Boltzmann Machine (EDRBM) to classify cybersecurity threats in large-scale network environments. EDRBM acts as a classification model that enables the classification of malicious flowsets in a large-scale network. Simulations were carried out to evaluate the efficacy of the proposed EDRBM model under various malware attacks. The results showed that the proposed method achieved a promising malware classification rate in malicious flowsets.

Adapted Deep Ensemble Learning-Based Voting Classifier for Osteosarcoma Cancer Classification.

The study utilizes osteosarcoma hematoxylin and the Eosin-stained image dataset, which is unevenly dispersed, and it raises concerns about the potential impact on the overall performance and reliability of any analyses or models derived from the dataset. In this study, a deep-learning-based convolution neural network (CNN) and adapted heterogeneous ensemble-learning-based voting classifier have been proposed to classify osteosarcoma. The proposed methods can also resolve the issue and develop unbiased learning models by introducing an evenly distributed training dataset. Data augmentation is employed to boost the generalization abilities. Six different pre-trained CNN models, namely MobileNetV1, Mo-bileNetV2, ResNetV250, InceptionV2, EfficientNetV2B0, and NasNetMobile, are applied and evaluated in frozen and fine-tuned-based phases. In addition, a novel CNN model and adapted heterogeneous ensemble-learning-based voting classifier developed from the proposed CNN model, fine-tuned NasNetMobile model, and fine-tuned Efficient-NetV2B0 model are also introduced to classify osteosarcoma. The proposed CNN model outperforms other pre-trained models. The Kappa score obtained from the proposed CNN model is 93.09%. Notably, the proposed voting classifier attains the highest Kappa score of 96.50% and outperforms all other models. The findings of this study have practical implications in telemedicine, mobile healthcare systems, and as a supportive tool for medical professionals.

Stacked ensemble deep learning for pancreas cancer classification using extreme gradient boosting.

Ensemble learning aims to improve prediction performance by combining several models or forecasts. However, how much and which ensemble learning techniques are useful in deep learning-based pipelines for pancreas computed tomography (CT) image classification is a challenge. Ensemble approaches are the most advanced solution to many machine learning problems. These techniques entail training multiple models and combining their predictions to improve the predictive performance of a single model. This article introduces the idea of Stacked Ensemble Deep Learning (SEDL), a pipeline for classifying pancreas CT medical images. The weak learners are Inception V3, VGG16, and ResNet34, and we employed a stacking ensemble. By combining the first-level predictions, an input train set for XGBoost, the ensemble model at the second level of prediction, is created. Extreme Gradient Boosting (XGBoost), employed as a strong learner, will make the final classification. Our findings showed that SEDL performed better, with a 98.8% ensemble accuracy, after some adjustments to the hyperparameters. The Cancer Imaging Archive (TCIA) public access dataset consists of 80 pancreas CT scans with a resolution of 512 * 512 pixels, from 53 male and 27 female subjects. A sample of two hundred and twenty-two images was used for training and testing data. We concluded that implementing the SEDL technique is an effective way to strengthen the robustness and increase the performance of the pipeline for classifying pancreas CT medical images. Interestingly, grouping like-minded or talented learners does not make a difference.

Physics-informed deep learning for lithium-ion battery diagnostics using electrochemical impedance spectroscopy

Lithium-ion batteries (LIBs) are prevalent energy storage devices in industrial fields and modern life, but are subjected to capacity degradation during operation due to the varying internal states. To ensure the efficiency, safety and reliability of LIBs, LIBs diagnostics by analyzing the internal states and estimating the capacity is crucial. However, model-based methods and deep learning models for LIBs diagnostics are mainly based on charging and discharging curves that contain limited information on battery internal states, and suffer from either model inaccuracy or limited interpretability. To achieve accurate and interpretable LIBs diagnostics, a novel physics-informed deep learning (PIDL) framework is proposed in this study using the electrochemical impedance spectroscopy (EIS), which is a powerful, non-invasive and information-rich technique. The EIS together with three physically informative parameters, which are extracted using model-based methods to investigate the varying internal states, are taken as inputs for capacity estimation, and two data fusion manners are proposed and investigated. Besides, to extract physically interpretable and conducive latent features, physical regularization and multi-task learning are incorporated, which make effective use of three information sources: physical knowledge, measurements and domain knowledge. Moreover, to evaluate the predictive uncertainty of capacity estimation, the deep ensemble strategy is adopted. The effectiveness and superiority of the PIDL framework are demonstrated using EIS datasets of eight commercially LIBs collected from the University of Cambridge.

How Does Traffic Environment Quantitatively Affect the Autonomous Driving Prediction?

Accurate trajectory prediction is essential for safe and efficient autonomous driving in complex traffic environments. While artificial intelligence has shown great promise in improving prediction accuracy, its inherent uncertainty and lack of explainability may lead to unpredictable failures, creating challenges for safety-critical decision-making. This study aims to address these challenges by exploring the impact of traffic environment on prediction algorithms. The study proposes a trajectory prediction framework with epistemic uncertainty estimation ability to output high uncertainty when facing unforeseeable or unknown scenarios. The framework analyzes the environmental effect on the trajectory prediction by considering scenario features and shifts. Features are divided into kinematic features of a target agent, features of surrounding traffic participants, and other scenario features. Feature correlation and importance analyses are performed to study their influence on prediction error and epistemic uncertainty. The impact of unavoidable distributional shifts in the real world on trajectory predictions is investigated using multiple intersection datasets. The results indicate that deep ensemble-based methods have advantages in improving robustness while estimating epistemic uncertainty. Consistent conclusions were obtained from the correlation and importance analyses, indicating that kinematic features of the target agent have relatively strong effects on both prediction error and epistemic uncertainty. Finally, the study analyzes the accuracy deterioration caused by distributional shifts and the potential of the deep ensemble-based method. Through deep ensemble, the errors of the prediction methods based on GRIP++ and Trajectron++ have been improved by 6.4% and 10.8% in the same-dataset test, and 6.3% and 10.8% in the cross-dataset test.

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise.

In the past years, deep learning has seen an increase in usage in the domain of histopathological applications. However, while these approaches have shown great potential, in high-risk environments deep learning models need to be able to judge their uncertainty and be able to reject inputs when there is a significant chance of misclassification. In this work, we conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images, with a focus on the task of selective classification, where the model should reject the classification in situations in which it is uncertain. We conduct our experiments on tile-level under the aspects of domain shift and label noise, as well as on slide-level. In our experiments, we compare Deep Ensembles, Monte-Carlo Dropout, Stochastic Variational Inference, Test-Time Data Augmentation as well as ensembles of the latter approaches. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise, while contrary to results from classical computer vision benchmarks no systematic gain of the other methods can be shown. Across methods, a rejection of the most uncertain samples reliably leads to a significant increase in classification accuracy on both in-distribution as well as out-of-distribution data. Furthermore, we conduct experiments comparing these methods under varying conditions of label noise. Lastly, we publish our code framework to facilitate further research on uncertainty estimation on histopathological data.

OA1‐AM23‐TU‐12 | Deep Ensemble Learning Enables Highly Accurate Classification of Stored Red Blood Cell Morphology

OA1‐AM23‐TU‐12 | Deep Ensemble Learning Enables Highly Accurate Classification of Stored Red Blood Cell Morphology