Ensemble Of Deep Neural Networks Research Articles

Steel Surface Defect Detection Using an Ensemble of Deep Residual Neural Networks

Abstract Steel defect diagnostics is important for industry task as it is tied to the product quality and production efficiency. The aim of this paper is evaluating the application of residual neural networks for recognition of industrial steel defects of three classes. Developed and investigated models based on deep residual neural networks for the recognition and classification of surface defects of rolled steel. Investigated the influence of various loss functions, optimizers and hyperparameters on the obtained result and selected optimal model parameters. Based on an ensemble of two deep residual neural networks ResNet50 and ResNet152, a classifier was constructed to detect defects of three classes on flat metal surfaces. The proposed technique allows classifying images with high accuracy. The average binary accuracy of classifying the test data is 96.7% for all images (including defect-free ones). The fields of neuron activation in the convolutional layers of the model were investigated. Feature maps formed in the process were found to reflect the position, size, and shape of the objects of interest very well. The proposed ensemble model has proven to be robust and able to accurately recognize steel surface defects. Erroneous recognition cases of the classifier application are investigated. It was shown that errors most often occur in ambiguous situations, where surface artifacts of different types are similar.

Применение ансамбля глубоких нейронных сетей в задачах краткосрочного прогнозирования почасового электропотребления гарантирующего поставщика электроэнергии

Применение ансамбля глубоких нейронных сетей в задачах краткосрочного прогнозирования почасового электропотребления гарантирующего поставщика электроэнергии

Acoustic-Based UAV Detection Using Late Fusion of Deep Neural Networks

Multirotor UAVs have become ubiquitous in commercial and public use. As they become more affordable and more available, the associated security risks further increase, especially in relation to airspace breaches and the danger of drone-to-aircraft collisions. Thus, robust systems must be set in place to detect and deal with hostile drones. This paper investigates the use of deep learning methods to detect UAVs using acoustic signals. Deep neural network models are trained with mel-spectrograms as inputs. In this case, Convolutional Neural Networks (CNNs) are shown to be the better performing network, compared with Recurrent Neural Networks (RNNs) and Convolutional Recurrent Neural Networks (CRNNs). Furthermore, late fusion methods have been evaluated using an ensemble of deep neural networks, where the weighted soft voting mechanism has achieved the highest average accuracy of 94.7%, which has outperformed the solo models. In future work, the developed late fusion technique could be utilized with radar and visual methods to further improve the UAV detection performance.

IoT-enabled stacked ensemble of deep neural networks for the diagnosis of COVID-19 using chest CT scans

The ongoing COVID-19 (novel coronavirus disease 2019) pandemic has triggered a global emergency, resulting in significant casualties and a negative effect on socioeconomic and healthcare systems around the world. Hence, automatic and fast screening of COVID-19 infections has become an urgent need of this pandemic. Real-time reverse transcription polymerase chain reaction (RT-PCR), a commonly used primary clinical method, is expensive and time-consuming for skilled health professionals. With the aid of various AI functionalities and advanced technologies, chest CT scans may thus be a viable alternative for quick and automatic screening of COVID-19. At the moment, significant advances in 5G cellular and internet of things (IoT) technology are finding use in various applications in the healthcare sector. This study presents an IoT-enabled deep learning-based stacking model to analyze chest CT scans for effective diagnosis of COVID-19 encounters. At first, patient data will be obtained using IoT devices and sent to a cloud server during the data procurement stage. Then we use different fine-tuned CNN sub-models, which are stacked together using a meta-learner to detect COVID-19 infection from input CT scans. The proposed model is evaluated using an open access dataset containing both COVID-19 infected and non-COVID CT images. Evaluation results show the efficacy of the proposed stacked model containing fine-tuned CNNs and a meta-learner in detecting coronavirus infections using CT scans.

A Spatio-Temporal Ensemble Deep Learning Architecture for Real-Time Defect Detection during Laser Welding on Low Power Embedded Computing Boards

In modern production environments, advanced and intelligent process monitoring strategies are required to enable an unambiguous diagnosis of the process situation and thus of the final component quality. In addition, the ability to recognize the current state of product quality in real-time is an important prerequisite for autonomous and self-improving manufacturing systems. To address these needs, this study investigates a novel ensemble deep learning architecture based on convolutional neural networks (CNN), gated recurrent units (GRU) combined with high-performance classification algorithms such as k-nearest neighbors (kNN) and support vector machines (SVM). The architecture uses spatio-temporal features extracted from infrared image sequences to locate critical welding defects including lack of fusion (false friends), sagging, lack of penetration, and geometric deviations of the weld seam. In order to evaluate the proposed architecture, this study investigates a comprehensive scheme based on classical machine learning methods using manual feature extraction and state-of-the-art deep learning algorithms. Optimal hyperparameters for each algorithm are determined by an extensive grid search. Additional work is conducted to investigate the significance of various geometrical, statistical and spatio-temporal features extracted from the keyhole and weld pool regions. The proposed method is finally validated on previously unknown welding trials, achieving the highest detection rates and the most robust weld defect recognition among all classification methods investigated in this work. Ultimately, the ensemble deep neural network is implemented and optimized to operate on low-power embedded computing devices with low latency (1.1 ms), demonstrating sufficient performance for real-time applications.

A three-stage, deep learning, ensemble approach for prognosis in patients with Parkinson\u2019s disease

BackgroundDiagnosis of Parkinson’s disease (PD) is informed by the presence of progressive motor and non-motor symptoms and by imaging dopamine transporter with [123I]ioflupane (DaTscan). Deep learning and ensemble methods have recently shown promise in medical image analysis. Therefore, this study aimed to develop a three-stage, deep learning, ensemble approach for prognosis in patients with PD.MethodsRetrospective data of 198 patients with PD were retrieved from the Parkinson’s Progression Markers Initiative database and randomly partitioned into the training, validation, and test sets with 118, 40, and 40 patients, respectively. The first and second stages of the approach extracted features from DaTscan and clinical measures of motor symptoms, respectively. The third stage trained an ensemble of deep neural networks on different subsets of the extracted features to predict patient outcome 4 years after initial baseline screening. The approach was evaluated by assessing mean absolute percentage error (MAPE), mean absolute error (MAE), Pearson’s correlation coefficient, and bias between the predicted and observed motor outcome scores. The approach was compared to individual networks given different data subsets as inputs.ResultsThe ensemble approach yielded a MAPE of 18.36%, MAE of 4.70, a Pearson’s correlation coefficient of 0.84, and had no significant bias indicating accurate outcome prediction. The approach outperformed individual networks not given DaTscan imaging or clinical measures of motor symptoms as inputs, respectively.ConclusionThe approach showed promise for longitudinal prognostication in PD and demonstrated the synergy of imaging and non-imaging information for the prediction task.

COVID-19 classification of X-ray images using deep neural networks.

ObjectivesIn the midst of the coronavirus disease 2019 (COVID-19) outbreak, chest X-ray (CXR) imaging is playing an important role in diagnosis and monitoring of patients with COVID-19. We propose a deep learning model for detection of COVID-19 from CXRs, as well as a tool for retrieving similar patients according to the model’s results on their CXRs. For training and evaluating our model, we collected CXRs from inpatients hospitalized in four different hospitals.MethodsIn this retrospective study, 1384 frontal CXRs, of COVID-19 confirmed patients imaged between March and August 2020, and 1024 matching CXRs of non-COVID patients imaged before the pandemic, were collected and used to build a deep learning classifier for detecting patients positive for COVID-19. The classifier consists of an ensemble of pre-trained deep neural networks (DNNS), specifically, ReNet34, ReNet50¸ ReNet152, and vgg16, and is enhanced by data augmentation and lung segmentation. We further implemented a nearest-neighbors algorithm that uses DNN-based image embeddings to retrieve the images most similar to a given image.ResultsOur model achieved accuracy of 90.3%, (95% CI: 86.3–93.7%) specificity of 90% (95% CI: 84.3–94%), and sensitivity of 90.5% (95% CI: 85–94%) on a test dataset comprising 15% (350/2326) of the original images. The AUC of the ROC curve is 0.96 (95% CI: 0.93–0.97).ConclusionWe provide deep learning models, trained and evaluated on CXRs that can assist medical efforts and reduce medical staff workload in handling COVID-19.Key Points• A machine learning model was able to detect chest X-ray (CXR) images of patients tested positive for COVID-19 with accuracy and detection rate above 90%.• A tool was created for finding existing CXR images with imaging characteristics most similar to a given CXR, according to the model’s image embeddings.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00330-021-08050-1.

Classification of Alzheimer's Disease Using Ensemble of Deep Neural Networks Trained Through Transfer Learning.

Alzheimer's disease (AD) is one of the deadliest neurodegenerative diseases ailing the elderly population all over the world. An ensemble of Deep learning (DL) models can learn highly complicated patterns from MRI scans for the detection of AD by utilizing diverse solutions. In this work, we propose a computationally efficient, DL-architecture agnostic, ensemble of deep neural networks, named 'Deep Transfer Ensemble (DTE)' trained using transfer learning for the classification of AD. DTE leverages the complementary feature views and diversity introduced by many different locally optimum solutions reached by individual networks through the randomization of hyper-parameters. DTE achieves an accuracy of 99.05% and 85.27% on two independent splits of the large dataset for cognitively normal (NC) vs AD classification task. For the task of mild cognitive impairment (MCI) vs AD classification, DTE achieves 98.71% and 83.11% respectively on the two independent splits. It also performs reasonable on a small dataset consisting of only 50 samples per class. It achieved a maximum accuracy of 85% for NC vs AD on the small dataset. It also outperformed snapshot ensembles along with several other existing deep models from similar kind of previous works by other researchers.

Ensemble of multi-stage deep convolutional neural networks for automated grading of diabetic retinopathy using image patches

Diabetic retinopathy (DR) is one of the most common retinal diseases that cause preventable blindness in diabetic patients. The timely screening and grading of retinal images minimize the possibility of vision loss. However, manual screening of retinal images, for detecting micro lesions in the early stages of DR, is time-consuming. This paper proposes an ensemble of deep convolutional neural network (CNN) models for accurate detection and grading of DR using fundus images. Each input image is divided into four patches at the first stage and passed on to pre-trained CNN models (InceptionV3, Xception) for training. The relevant features in the shallow-dense layers of CNN models are utilized as prior knowledge. The integration of shallow and dense layer features helps the model learn the significant information of DR images. At the second stage, a classifier based on artificial neural network is trained using the fused probability vectors of four patches. The results of individual CNN models are combined to generate the final decision in the third stage. This ensemble approach of multi-stage deep learning model improves the overall classification accuracy of diabetic retinopathy grading. Among the five different classification schemes presented in this paper, multistage patch-based deep CNN (MPDCNN), in which local patch-based and holistic details of fundus image are concatenated, provides the best classification accuracy. This ensemble classifier exhibits 96.2 % classification accuracy with fivefold cross-validation.

COVID-19 dynamics across the US: A deep learning study of human mobility and social behavior

This paper presents a deep learning framework for epidemiology system identification from noisy and sparse observations with quantified uncertainty. The proposed approach employs an ensemble of deep neural networks to infer the time-dependent reproduction number of an infectious disease by formulating a tensor-based multi-step loss function that allows us to efficiently calibrate the model on multiple observed trajectories. The method is applied to a mobility and social behavior-based SEIR model of COVID-19 spread. The model is trained on Google and Unacast mobility data spanning a period of 66 days, and is able to yield accurate future forecasts of COVID-19 spread in 203 US counties within a time-window of 15 days. Interestingly, a sensitivity analysis that assesses the importance of different mobility and social behavior parameters reveals that attendance of close places, including workplaces, residential, and retail and recreational locations, has the largest impact on the effective reproduction number. The model enables us to rapidly probe and quantify the effects of government interventions, such as lock-down and re-opening strategies. Taken together, the proposed framework provides a robust workflow for data-driven epidemiology model discovery under uncertainty and produces probabilistic forecasts for the evolution of a pandemic that can judiciously provide information for policy and decision making. All codes and data accompanying this manuscript are available at https://github.com/PredictiveIntelligenceLab/DeepCOVID19.

A Robust Prediction Model for Species Distribution Using Bagging Ensembles with Deep Neural Networks

Species distribution models have been used for various purposes, such as conserving species, discovering potential habitats, and obtaining evolutionary insights by predicting species occurrence. Many statistical and machine-learning-based approaches have been proposed to construct effective species distribution models, but with limited success due to spatial biases in presences and imbalanced presence-absences. We propose a novel species distribution model to address these problems based on bootstrap aggregating (bagging) ensembles of deep neural networks (DNNs). We first generate bootstraps considering presence-absence data on spatial balance to alleviate the bias problem. Then we construct DNNs using environmental data from presence and absence locations, and finally combine these into an ensemble model using three voting methods to improve prediction accuracy. Extensive experiments verified the proposed model’s effectiveness for species in South Korea using crowdsourced observations that have spatial biases. The proposed model achieved more accurate and robust prediction results than the current best practice models.

Toward Detection and Attribution of Cyber-Attacks in IoT-Enabled Cyber–Physical Systems

Securing Internet-of-Things (IoT)-enabled cyber-physical systems (CPS) can be challenging, as security solutions developed for general information/operational technology (IT/OT) systems may not be as effective in a CPS setting. Thus, this article presents a two-level ensemble attack detection and attribution framework designed for CPS, and more specifically in an industrial control system (ICS). At the first level, a decision tree combined with a novel ensemble deep representation-learning model is developed for detecting attacks imbalanced ICS environments. At the second level, an ensemble deep neural network is designed to facilitate attack attribution. The proposed model is evaluated using real-world data sets in gas pipeline and water treatment system. Findings demonstrate that the proposed model outperforms other competing approaches with similar computational complexity.

Abstract P05: EpistoNet: An ensemble of deep convolutional neural networks using mixture of discriminative experts for detecting COVID-19 on chest X-ray images

Abstract Introduction: The Coronavirus has spread across the globe and infected millions of people, having devastating effect on the global public health and economies. A fast diagnostic system should be implemented to mitigate the impact of the virus and save lives. In this study, we propose a decision tree-based ensemble model using two mixtures of discriminative experts (MoE) to classify COVID-19 and non-COVID-19 lung infections on chest X-ray images. The Epistocracy algorithm, a hyper-heuristic evolutionary method, is employed to optimize the neural networks used in this work. Using this approach can help detect COVID-19 cases and accelerate treatment of those who need it the most. Data: we collected 2,500 chest X-ray images from Henry Ford Health System consisting of 1,250 Covid images and 1,250 non-Covid images. The input images have been cropped and resized to 224 by 224 pixels. Out of 2,500 images, we left out 500 images containing 250 Covid and 250 non-Covid for testing. The rest, 2,000 images, were used 80% for training and 20% for validation. Methods and Results: To improve the accuracy of the proposed model, first we divided our 2,000 images into 5 different clusters using K-Means clustering algorithm with VGG16 feature extractor to help build strong discriminative expert models to be used in our proposed classifier. We trained VGG16, VGG19, InceptionV3, InceptionResNetV2, MobileNetV2, EfficientNetB7, Xception, and DenseNet201 to classify each cluster into Covid and non-Covid cases. The best result was obtained from VGG16 as a base model with a deep neural network as a head model optimized by Epistocracy algorithm. Then we built a mixture of transfer learning-based experts consisting of 5 different VGG16 models supervised by InceptionV3 as a gating network. Finally, we built a decision tree-based ensemble model to determine the classification of the data using two different MoEs with highest accuracies. As a result, for initial clusters c1, c2, c3, c4, and c5 we obtained validation accuracy of 92.50%, 86.30%, 86.51%, 85.34%, and 93.62% respectively. The first MoE had 93.75% accuracy on validation, and the second MoE had 94.25%. The final ensemble model on average obtained 94% accuracy on the testing dataset. More specifically, we got 96% accuracy on Covid images and 92% accuracy on non-Covid. Conclusion: we showed that an ensemble model consisting of two mixtures of cluster-based discriminative convolutional neural network experts can be used to detect Covid from non-Covid with high accuracy, and Epistocracy algorithm can be effectively used to optimize the hyper-parameters of the proposed models. Citation Format: Seyed Ziae Mousavi Mojab, Seyedmohammad Shams, Farshad Fotouhi, Hamid Soltanian-Zadeh. EpistoNet: An ensemble of deep convolutional neural networks using mixture of discriminative experts for detecting COVID-19 on chest X-ray images [abstract]. In: Proceedings of the AACR Virtual Meeting: COVID-19 and Cancer; 2021 Feb 3-5. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(6_Suppl):Abstract nr P05.

Deep neural network ensemble for on-the-fly quality control-driven segmentation of cardiac MRI T1 mapping.

Recent developments in artificial intelligence have generated increasing interest to deploy automated image analysis for diagnostic imaging and large-scale clinical applications. However, inaccuracy from automated methods could lead to incorrect conclusions, diagnoses or even harm to patients. Manual inspection for potential inaccuracies is labor-intensive and time-consuming, hampering progress towards fast and accurate clinical reporting in high volumes. To promote reliable fully-automated image analysis, we propose a quality control-driven (QCD) segmentation framework. It is an ensemble of neural networks that integrate image analysis and quality control. The novelty of this framework is the selection of the most optimal segmentation based on predicted segmentation accuracy, on-the-fly. Additionally, this framework visualizes segmentation agreement to provide traceability of the quality control process. In this work, we demonstrated the utility of the framework in cardiovascular magnetic resonance T1-mapping - a quantitative technique for myocardial tissue characterization. The framework achieved near-perfect agreement with expert image analysts in estimating myocardial T1 value (r=0.987,p<.0005; mean absolute error (MAE)=11.3ms), with accurate segmentation quality prediction (Dice coefficient prediction MAE=0.0339) and classification (accuracy=0.99), and a fast average processing time of 0.39 second/image. In summary, the QCD framework can generate high-throughput automated image analysis with speed and accuracy that is highly desirable for large-scale clinical applications.

Genetic Algorithm-based Optimization of Deep Neural Network Ensemble

Personal identification is the task of authenticating a person using individual biological features. Deep neural networks (DNNs) have demonstrated an impressive performance in this field. Since no general algorithm is available for the design of network structures and the parameters adopted in DNNs for every application problem, DNNs should be determined according to the programmers’ experiments and know-how. For a new application task, it is very time-consuming for non-experts to design network structure, hyperparameters and an ensemble of base models adequately and effectively. In this paper, we present a genetic algorithm (GA)-based approach to construct network structures, tune their hyperparameters, and generate base models for the ensemble algorithm. The ensemble is constructed from base models with different network structures according to the voting ensemble algorithm. Our original personal identification dataset is employed as the numerical example to illustrate the performance of the proposed method. The results show that the prediction accuracy of the ensemble model is better than that of the base models and that the prediction of walking behavior toward the Kinect at 90 degrees and 225 degrees is more difficult than other walking behaviors.

Wood species identification based on an ensemble of deep convolution neural networks.

Wood species identification based on an ensemble of deep convolution neural networks.

Wood species identification based on an ensemble of deep convolution neural networks

Wood species identification based on an ensemble of deep convolution neural networks

Colorectal Histology Tumor Detection Using Ensemble Deep Neural Network

With a mortality rate of approximately 33.33%, Colorectal cancer serves as the second most prevalent malignant tumor type in the world. AI-guided clinical care/tool can help in reducing health disparities, specifically in resource-constrained regions. In this paper, using multi-class tissue features, we proposed an Ensemble Deep Neural Network to Tumor in Colorectal Histology images. On two different publicly available datasets: NCT-CRC-HE-100K (107,180 images) and Colorectal Histology (5000 images), we achieved accuracies of 96.16% and 92.83%, respectively. When datasets are combined, it provided a benchmark accuracy of 99.13%. We efficiently used resourced data, thereby achieving results that outperformed the state-of-the-art works.

On learning effective ensembles of deep neural networks for intrusion detection

Classification-oriented Machine Learning methods are a precious tool, in modern Intrusion Detection Systems (IDSs), for discriminating between suspected intrusion attacks and normal behaviors. Many recent proposals in this field leveraged Deep Neural Network (DNN) methods, capable of learning effective hierarchical data representations automatically. However, many of these solutions were validated on data featuring stationary distributions and/or large amounts of training examples. By contrast, in real IDS applications different kinds of attack tend to occur over time, and only a small fraction of the data instances is labeled (usually with far fewer examples of attacks than of normal behavior). A novel ensemble-based Deep Learning framework is proposed here that tries to face the challenging issues above. Basically, the non-stationary nature of IDS log data is faced by maintaining an ensemble consisting of a number of specialized base DNN classifiers, trained on disjoint chunks of the data instances’ stream, plus a combiner model (reasoning on both the base classifiers predictions and original instance features). In order to learn deep base classifiers effectively from small training samples, an ad-hoc shared DNN architecture is adopted, featuring a combination of dropout capabilities, skip-connections, along with a cost-sensitive loss (for dealing with unbalanced data). Tests results, conducted on two benchmark IDS datasets and involving several competitors, confirmed the effectiveness of our proposal (in terms of both classification accuracy and robustness to data scarcity), and allowed us to evaluate different ensemble combination schemes.

An Ensemble Deep Neural Network Approach for Oral Cancer Screening

One of the ways to reduce oral cancer mortality rate is diagnosing oral lesions at initial stages to classify them as precancerous or normal lesions. During routine oral examination, oral lesions are normally screened manually. In a low resource setting area where there is lack of medical facilities and also medical expertise, an automated mechanism for oral cancer screening is required. The present work is an attempt towards developing an automated system for diagnosing oral lesions using deep learning techniques. An ensemble deep learning model that combines the benefits of Resnet-50 and VGG-16 has been developed. This model has been trained with an augmented dataset of oral lesion images. The model outperforms other popularly used deep learning models in performing the classification of oral images. An accuracy of 96.2%, 98.14% sensitivity and 94.23% specificity was achieved with the ensemble deep learning model.