Ensemble Of Deep Neural Networks Research Articles

Pneumonia Detection on Chest X-ray Images Using Ensemble of Deep Convolutional Neural Networks

Pneumonia is a life-threatening lung infection resulting from several different viral infections. Identifying and treating pneumonia on chest X-ray images can be difficult due to its similarity to other pulmonary diseases. Thus, the existing methods for predicting pneumonia cannot attain substantial levels of accuracy. This paper presents a computer-aided classification of pneumonia, coined Ensemble Learning (EL), to simplify the diagnosis process on chest X-ray images. Our proposal is based on Convolutional Neural Network (CNN) models, which are pretrained CNN models that have been recently employed to enhance the performance of many medical tasks instead of training CNN models from scratch. We propose to use three well-known CNNs (DenseNet169, MobileNetV2, and Vision Transformer) pretrained using the ImageNet database. These models are trained on the chest X-ray data set using fine-tuning. Finally, the results are obtained by combining the extracted features from these three models during the experimental phase. The proposed EL approach outperforms other existing state-of-the-art methods and obtains an accuracy of 93.91% and a F1-score of 93.88% on the testing phase.

Carbon Emissions Estimation and Spatiotemporal Analysis of China at City Level Based on Multi-Dimensional Data and Machine Learning

Carbon emissions caused by the massive consumption of energy have brought enormous pressure on the Chinese government. Accurately and rapidly characterizing the spatiotemporal characteristics of Chinese city-level carbon emissions is crucial for policy decision making. Based on multi-dimensional data, including nighttime light (NTL) data, land use (LU) data, land surface temperature (LST) data, and added-value secondary industry (AVSI) data, a deep neural network ensemble (DNNE) model was built to analyze the nonlinear relationship between multi-dimensional data and province-level carbon emission statistics (CES) data. The city-level carbon emissions data were estimated, and the spatiotemporal characteristics were analyzed. As compared to the energy statistics released by partial cities, the results showed that the DNNE model based on multi-dimensional data could well estimate city-level carbon emissions data. In addition, according to a linear trend analysis and standard deviational ellipse (SDE) analysis of China from 2001 to 2019, we concluded that the spatiotemporal changes in carbon emissions at the city level were in accordance with the development of China’s economy. Furthermore, the results can provide a useful reference for the scientific formulation, implementation, and evaluation of carbon emissions reduction policies.

Machine Learning for heat radiation modeling of bi- and polydisperse particle systems including walls

We investigated the ability of four popular Machine Learning methods i.e., Deep Neural Networks (DNNs), Random Forest-based regressors (RFRs), Extreme Gradient Boosting-based regressors (XGBs), and stacked ensembles of DNNs, to model the radiative heat transfer based on view factors in bi- and polydisperse particle beds including walls. Before training and analyzing the predictive capability of each method, an adjustment of markers used in monodisperse systems, as well as an evaluation of new markers was performed. On the basis of our dataset that considers a wide range of particle radii ratios, system sizes, particle volume fractions, as well as different particle-species volume fractions, we found that (i) the addition of particle size information allows the transition from monodisperse to bi- and polydisperse beds, and (ii) the addition of particle volume fraction information as the fourth marker leads to very accurate predictions. In terms of the overall performance, DNNs and RFRs should be preferred compared to the other two options. For particle–particle view factors, DNN and RFR are on par, while for particle–wall the RFR is superior. We demonstrate that DNNs and RFRs can be built to meet or even exceed the prediction quality standards achieved in a monodisperse system.

Deep neural network ensembles for detecting self-admitted technical debt

Effectively identifying self-admitted technical debt (SATD) from project source code comments helps developers quickly find and repay these debts, thereby reducing its negative impact. Previous studies used techniques based on patterns, text mining, natural language processing, and neural networks to detect SATD. Compared with these above, Convolutional Neural Networks (CNN) have the strong feature extraction ability. Deep network ensembles are demonstrated great potential for the task of sentences classification. In order to boost the performance of CNN-based SATD detecting, we propose a deep neural network ensemble contribute to ensemble learning in a simple yet effective way. Specifically, CNN, CNN-LSTM (convolutional neural network and long short-term memory), and DPCNN (Deep Pyramid Convolutional Neural Networks) are used as individual classifiers to diversify the deep network ensembles. In order to improve the explainability, we introduce attention to measure the contribution of feature words to SATD classification. 62,285 source code comments from 10 projects were used in our experiments. The results show that our approach can effectively reduce misjudgment and detect more SATD, especially for cross-project, so as to greatly improve the detection accuracy.

Failure Mode Detection of Reinforced Concrete Shear Walls Using Ensemble Deep Neural Networks

Reinforced concrete structural walls (RCSWs) are one of the most efficient lateral force-resisting systems used in buildings, providing sufficient strength, stiffness, and deformation capacities to withstand the forces generated during earthquake ground motions. Identifying the failure mode of the RCSWs is a critical task that can assist engineers and designers in choosing appropriate retrofitting solutions. This study evaluates the efficiency of three ensemble deep neural network models, including the model averaging ensemble, weighted average ensemble, and integrated stacking ensemble for predicting the failure mode of the RCSWs. The ensemble deep neural network models are compared against previous studies that used traditional well-known ensemble models (AdaBoost, XGBoost, LightGBM, CatBoost) and traditional machine learning methods (Naïve Bayes, K-Nearest Neighbors, Decision Tree, and Random Forest). The weighted average ensemble model is proposed as the best-suited prediction model for identifying the failure mode since it has the highest accuracy, precision, and recall among the alternative models. In addition, since complex and advanced machine learning-based models are commonly referred to as black-box, the SHapley Additive exPlanation method is also used to interpret the model workflow and illustrate the importance and contribution of the components that impact determining the failure mode of the RCSWs.

En‐ConvNet: A novel approach for glaucoma detection from color fundus images using ensemble of deep convolutional neural networks

AbstractGlaucomatous optic neuropathy is the preeminent cause of incurable vision impairment and blindness across the world. Manual interpretation of the pathological structures in fundus images is time‐consuming and requires the expertise of a competent specialist. With the development of deep learning approaches, automated glaucoma diagnosis is easy and effective for larger screening. Convolutional neural networks, in particular, have emerged as a promising choice for glaucoma detection from fundus images due to their remarkable success in image classification. Transferring the optimized weights from a pre‐trained model expedites and simplifies the training process of deep neural network. In this paper, a deep ensemble model using the stacking ensemble learning technique is developed to attain the optimum performance for the classification of glaucomatous and normal images. Thirteen pre‐trained models such as Alexnet, Googlenet, VGG‐16, VGG‐19, Squeezenet, Resnet‐18, Resnet‐50, Resnet‐101, Efficientnet‐b0, Mobilenet‐v2, Densenet‐201, Inception‐v3, and Xception are implemented. Their performance is compared in 65 different configurations, comprising 13 CNN architectures and five various classification approaches. A two‐stage ensemble selection technique is proposed to select the optimal configurations. Selected configurations are pooled using a probability averaging technique. The final classification is performed using an SVM classifier. In this work, publicly available databases are modified (such as: DRISHTI‐GS1‐R, ORIGA‐R, RIM‐ONE2‐R, LAG‐R, and ACRIMA‐R) based on oversampling data‐level technique for validating the performance of deep ensemble model. Ensembling the best configurations reports an overall classification accuracy of 93.4%, 79.6%, 91.3%, 99.5%, and 99.6% in DRISHTI‐GS1‐R, ORIGA‐R, RIM‐ONE2‐R, ACRIMA‐R, and LAG‐R databases, respectively.

Breast Cancer Semantic Segmentation for Accurate Breast Cancer Detection with an Ensemble Deep Neural Network

Breast Cancer Semantic Segmentation for Accurate Breast Cancer Detection with an Ensemble Deep Neural Network

Automatic Cancer Cell Taxonomy Using an Ensemble of Deep Neural Networks.

Simple SummaryAs of recently, cancer is considered a major cause of death in developed and developing countries. Therefore, there is an urgent need for improvements in existing diagnostic methods for effective early diagnosis. However, cross-contamination of cancer cell lines results in the development of inappropriate treatments that cannot be administered to patients. To address this issue, we propose an automatic cancer cell taxonomy with high accuracy using optical images of cells obtained through low-scale benchtop optical microscopy. Specifically, we built a deep-learning-based framework to classify cervical, hepatocellular, breast, and lung cancer cells. The experimental results demonstrated that the proposed deep-learning-based approach facilitates the automatic identification of cancer cells. Moreover, our findings provide important insights into the design of convolutional neural networks for various clinical tasks that utilize microscopic images.Microscopic image-based analysis has been intensively performed for pathological studies and diagnosis of diseases. However, mis-authentication of cell lines due to misjudgments by pathologists has been recognized as a serious problem. To address this problem, we propose a deep-learning-based approach for the automatic taxonomy of cancer cell types. A total of 889 bright-field microscopic images of four cancer cell lines were acquired using a benchtop microscope. Individual cells were further segmented and augmented to increase the image dataset. Afterward, deep transfer learning was adopted to accelerate the classification of cancer types. Experiments revealed that the deep-learning-based methods outperformed traditional machine-learning-based methods. Moreover, the Wilcoxon signed-rank test showed that deep ensemble approaches outperformed individual deep-learning-based models (p < 0.001) and were in effect to achieve the classification accuracy up to 97.735%. Additional investigation with the Wilcoxon signed-rank test was conducted to consider various network design choices, such as the type of optimizer, type of learning rate scheduler, degree of fine-tuning, and use of data augmentation. Finally, it was found that the using data augmentation and updating all the weights of a network during fine-tuning improve the overall performance of individual convolutional neural network models.

Ensemble of deep convolutional neural networks for real-time gravitational wave signal recognition

With the rapid development of deep learning technology, more and more researchers apply it to gravitational wave (GW) data analysis. Previous studies focused on a single deep learning model. In this paper we design an ensemble algorithm combining a set of convolutional neural networks (CNN) for GW signal recognition. The whole ensemble model consists of two sub-ensemble models. Each sub-ensemble model is also an ensemble model of deep learning. The two sub-ensemble models treat data of Hanford and Livinston detectors respectively. Proper voting scheme is adopted to combine the two sub-ensemble models to form the whole ensemble model. We apply this ensemble model to all reported GW events in the first observation and second observation runs (O1/O2) by LIGO-VIRGO Scientific Collaboration. We find that the ensemble algorithm can clearly identify all binary black hole merger events except GW170818. We also apply the ensemble model to one month (August 2017) data of O2. There is no false trigger happens although only O1 data are used for training. Our test results indicate that the ensemble learning algorithms can be used in real-time GW data analysis.

A classification of MRI brain tumor based on two stage feature level ensemble of deep CNN models

The brain tumor is one of the deadliest cancerous diseases and its severity has turned it to the leading cause of cancer related mortality. The treatment procedure of the brain tumor depends on the type, location and size of the tumor. Relying solely on human inspection for precise categorization can lead to inevitably dangerous situation. This manual diagnosis process can be improved and accelerated through an automated Computer Aided Diagnosis (CADx) system. In this article, a novel approach using two-stage feature ensemble of deep Convolutional Neural Networks (CNN) is proposed for precise and automatic classification of brain tumors. Three unique Magnetic Resonance Imaging (MRI) datasets and a dataset merging all the unique datasets are considered. The datasets contain three types of brain tumor (meningioma, glioma, pituitary) and normal brain images. From five pre-trained models and a proposed CNN model, the best models are chosen and concatenated in two stages for feature extraction. The best classifier is also chosen among five different classifiers based on accuracy. From the extracted features, most substantial features are selected using Principal Component Analysis (PCA) and fed into the classifier. The robustness of the proposed two stage ensemble model is analyzed using several performance metrics and three different experiments. Through the prominent performance, the proposed model is able to outperform other existing models attaining an average accuracy of 99.13% by optimization of the developed algorithms. Here, the individual accuracy for Dataset 1, Dataset 2, Dataset 3, and Merged Dataset is 99.67%, 98.16%, 99.76%, and 98.96% respectively. Finally a User Interface (UI) is created using the proposed model for real time validation.

ORBoost: An Orthogonal AdaBoost

Ensemble learners and deep neural networks are state-of-the-art schemes for classification applications. However, deep networks suffer from complex structure, need large amount of samples and also require plenty of time to be converged. In contrast, ensemble learners (especially AdaBoost) are fast to be trained, can work with small and large datasets and also benefit strong mathematical background. In this paper, we have developed a new orthogonal version of AdaBoost, termed as ORBoost, in order to desensitize its performance against noisy samples as well as exploiting low number of weak learners. In ORBoost, after reweighting the distribution of each learner, the Gram-Schmidt rule updates those weights to make a new samples’ distribution to be orthogonal to the former distributions. In contrast in AdaBoost, there is no orthogonality constraint even between two successive weak learners while there is a similarity between the distributions of samples in different learners. To assess the performance of ORBoost, 16 UCI-Repository datasets along with six big datasets are deployed. The performance of ORBoost is compared to the standard AdaBoost, LogitBoost and AveBoost-II over the selected datasets. The achieved results support the significant superiority of ORBoost to the counterparts in terms of accuracy, robustness, number of exploited weak learners and generalization on most of the datasets.

ICA-Evolution Based Data Augmentation with Ensemble Deep Neural Networks Using Time and Frequency Kernels for Emotion Recognition from EEG-Data

The aim of this study is to recognize human emotions from electroencephalographic (EEG) signals using deep neural networks. Large training data is an important prerequisite for successful implementation of deep neural networks. In this view, we propose an independent component analysis (ICA) - evolution based data augmentation method. This method performs ICA to extract and accumulate clean independent components (ICs) of each class. The new ICs are generated by selection which uses a fitness function such as mutual information (MI) and crossover in component space. Data augmentation is done by performing mutation, and crossover on generated data in sensor space. Since EEG signals are non-stationary, with time-varying frequency contents, emotional patterns associated with EEG are detected in the time-frequency (TF) domain using a spectrogram. To extract emotion related features from a spectrogram, we train an ensemble convolutional neural networks (CNNs) with convolutional kernels in time and frequency axes. The information integrated over both the axes is concatenated and fed to long short-term memory (LSTM). We used the benchmark DEAP dataset for emotion classification to evaluate our approach. The results highlight the potential of proposed ICA-evolution based data augmentation and an ensemble CNNs with LSTM model for emotion recognition.

Clinical validation of saliency maps for understanding deep neural networks in ophthalmology

Deep neural networks (DNNs) have achieved physician-level accuracy on many imaging-based medical diagnostic tasks, for example classification of retinal images in ophthalmology. However, their decision mechanisms are often considered impenetrable leading to a lack of trust by clinicians and patients. To alleviate this issue, a range of explanation methods have been proposed to expose the inner workings of DNNs leading to their decisions. For imaging-based tasks, this is often achieved via saliency maps. The quality of these maps are typically evaluated via perturbation analysis without experts involved. To facilitate the adoption and success of such automated systems, however, it is crucial to validate saliency maps against clinicians. In this study, we used three different network architectures and developed ensembles of DNNs to detect diabetic retinopathy and neovascular age-related macular degeneration from retinal fundus images and optical coherence tomography scans, respectively. We used a variety of explanation methods and obtained a comprehensive set of saliency maps for explaining the ensemble-based diagnostic decisions. Then, we systematically validated saliency maps against clinicians through two main analyses - a direct comparison of saliency maps with the expert annotations of disease-specific pathologies and perturbation analyses using also expert annotations as saliency maps. We found the choice of DNN architecture and explanation method to significantly influence the quality of saliency maps. Guided Backprop showed consistently good performance across disease scenarios and DNN architectures, suggesting that it provides a suitable starting point for explaining the decisions of DNNs on retinal images.

Analysis of an adaptive lead weighted ResNet for multiclass classification of 12-lead ECGs

Background. Twelve lead ECGs are a core diagnostic tool for cardiovascular diseases. Here, we describe and analyse an ensemble deep neural network architecture to classify 24 cardiac abnormalities from 12 lead ECGs. Method. We proposed a squeeze and excite ResNet to automatically learn deep features from 12-lead ECGs, in order to identify 24 cardiac conditions. The deep features were augmented with age and gender features in the final fully connected layers. Output thresholds for each class were set using a constrained grid search. To determine why the model made incorrect predictions, two expert clinicians independently interpreted a random set of 100 misclassified ECGs concerning left axis deviation. Results. Using the bespoke weighted accuracy metric, we achieved a 5-fold cross-validation score of 0.684, and sensitivity and specificity of 0.758 and 0.969, respectively. We scored 0.520 on the full test data, and ranked 2nd out of 41 in the official challenge rankings. On a random set of misclassified ECGs, agreement between two clinicians and training labels was poor (clinician 1: κ = −0.057, clinician 2: κ = −0.159). In contrast, agreement between the clinicians was very high (κ = 0.92). Discussion. The proposed prediction model performed well on the validation and hidden test data in comparison to models trained on the same data. We also discovered considerable inconsistency in training labels, which is likely to hinder development of more accurate models.

The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition

The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition

EDNC: Ensemble Deep Neural Network for COVID-19 Recognition.

The automatic recognition of COVID-19 diseases is critical in the present pandemic since it relieves healthcare staff of the burden of screening for infection with COVID-19. Previous studies have proven that deep learning algorithms can be utilized to aid in the diagnosis of patients with potential COVID-19 infection. However, the accuracy of current COVID-19 recognition models is relatively low. Motivated by this fact, we propose three deep learning architectures, F-EDNC, FC-EDNC, and O-EDNC, to quickly and accurately detect COVID-19 infections from chest computed tomography (CT) images. Sixteen deep learning neural networks have been modified and trained to recognize COVID-19 patients using transfer learning and 2458 CT chest images. The proposed EDNC has then been developed using three of sixteen modified pre-trained models to improve the performance of COVID-19 recognition. The results suggested that the F-EDNC method significantly enhanced the recognition of COVID-19 infections with 97.75% accuracy, followed by FC-EDNC and O-EDNC (97.55% and 96.12%, respectively), which is superior to most of the current COVID-19 recognition models. Furthermore, a localhost web application has been built that enables users to easily upload their chest CT scans and obtain their COVID-19 results automatically. This accurate, fast, and automatic COVID-19 recognition system will relieve the stress of medical professionals for screening COVID-19 infections.

Dementia diagnosis by ensemble deep neural networks using FDG-PET scans

Dementia diagnosis by ensemble deep neural networks using FDG-PET scans

Automated Ensemble for Deep Learning Inference on Edge Computing Platforms

Advances in deep learning (DL) have triggered an explosion of mobile intelligence, posing a soaring demand for computing resources that cannot be satisfied by mobile devices. In this article, we employ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">edge computing</i> to deliver better DL inference services to end users. The key is to leverage deep neural network (DNN) ensemble techniques that provide state-of-the-art performance for many machine learning applications in terms of inference accuracy and robustness. Compared to end devices, the edge computing platform is endowed with more powerful computing resources, making it feasible to implement DNN ensembles for DL inferences. However, due to the constrained computing capacity of edge servers and the possible service response deadline, an edge server can only use a limited number of DNNs to construct DNN ensembles. This poses a unique problem, namely, DNN ensemble selection, for identifying the best-fit DNN ensembles. We propose a novel algorithm called automated DNN ensemble selection (AES) algorithm to solve this problem. Because DNNs exhibit performance variations over different distributions of input data, AES adaptively determines a DNN ensemble according to the features of admitted inference tasks. AES is an online learning algorithm that learns DNNs’ in-use performance over time. An ensemble selection rule is further designed as a subroutine of AES to recruit members into the DNN ensemble based on the accuracy and diversity of DNNs. In particular, we theoretically prove that AES can achieve asymptotic optimality. We carry out experiments on real-world data sets. The results show that using the DNN ensemble technique on edge computing platforms dramatically improves the DL inference quality, and AES outperforms other benchmark schemes.

Preliminary Classification of Selected Farmland Habitats in Ireland Using Deep Neural Networks.

Ireland has a wide variety of farmlands that includes arable fields, grassland, hedgerows, streams, lakes, rivers, and native woodlands. Traditional methods of habitat identification rely on field surveys, which are resource intensive, therefore there is a strong need for digital methods to improve the speed and efficiency of identification and differentiation of farmland habitats. This is challenging because of the large number of subcategories having nearly indistinguishable features within the habitat classes. Heterogeneity among sites within the same habitat class is another problem. Therefore, this research work presents a preliminary technique for accurate farmland classification using stacked ensemble deep convolutional neural networks (DNNs). The proposed approach has been validated on a high-resolution dataset collected using drones. The image samples were manually labelled by the experts in the area before providing them to the DNNs for training purposes. Three pre-trained DNNs customized using the transfer learning approach are used as the base learners. The predicted features derived from the base learners were then used to train a DNN based meta-learner to achieve high classification rates. We analyse the obtained results in terms of convergence rate, confusion matrices, and ROC curves. This is a preliminary work and further research is needed to establish a standard technique.

An ensemble deep neural network approach for predicting TOC concentration in lakes along the middle-lower reaches of Yangtze River

In recent years, lakes pollution has become increasingly serious, so water quality monitoring is becoming increasingly important. The concentration of total organic carbon (TOC) in lakes is an important indicator for monitoring the emission of organic pollutants. Therefore, it is of great significance to determine the TOC concentration in lakes. In this paper, the water quality dataset of the middle and lower reaches of the Yangtze River is obtained, and then the temperature, transparency, pH value, dissolved oxygen, conductivity, chlorophyll and ammonia nitrogen content are taken as the impact factors, and the stacking of different epochs’ deep neural networks (SDE-DNN) model is constructed to predict the TOC concentration in water. Five deep neural networks and linear regression are integrated into a strong prediction model by the stacking ensemble method. The experimental results show the prediction performance, the Nash-Sutcliffe efficiency coefficient (NSE) is 0.5312, the mean absolute error (MAE) is 0.2108 mg/L, the symmetric mean absolute percentage error (SMAPE) is 43.92%, and the root mean squared error (RMSE) is 0.3064 mg/L. The model has good prediction performance for the TOC concentration in water. Compared with the common machine learning models, traditional ensemble learning models and existing TOC prediction methods, the prediction error of this model is lower, and it is more suitable for predicting the TOC concentration. The model can use a wireless sensor network to obtain water quality data, thus predicting the TOC concentration of lakes in real time, reducing the cost of manual testing, and improving the detection efficiency.