Ensemble Of Deep Neural Networks Research Articles

Enhancing breast cancer segmentation and classification: An Ensemble Deep Convolutional Neural Network and U-net approach on ultrasound images

Breast cancer is a condition where the irregular growth of breast cells occurs uncontrollably, leading to the formation of tumors. It poses a significant threat to women’s lives globally, emphasizing the need for enhanced methods of detecting and categorizing the disease. In this work, we propose an Ensemble Deep Convolutional Neural Network (EDCNN) model that exhibits superior accuracy compared to several transfer learning models and the Vision Transformer model. Our EDCNN model integrates the strengths of the MobileNet and Xception models to improve its performance in breast cancer detection and classification. We employ various preprocessing techniques, including image resizing, data normalization, and data augmentation, to prepare the data for analysis. By following these measures, the formatting is optimized, and the model’s capacity to make generalizations is improved. We trained and evaluated our proposed EDCNN model using ultrasound images, a widely available modality for breast cancer imaging. The outcomes of our experiments illustrate that the EDCNN model attains an exceptional accuracy of 87.82% on Dataset 1 and 85.69% on Dataset 2, surpassing the performance of several well-known transfer learning models and the Vision Transformer model. Furthermore, an AUC value of 0.91 on Dataset 1 highlights the robustness and effectiveness of our proposed model. Moreover, we highlight the incorporation of the Grad-CAM Explainable Artificial Intelligence (XAI) technique to improve the interpretability and transparency of our proposed model. Additionally, we performed image segmentation using the U-Net segmentation technique on the input ultrasound images. This segmentation process allowed for the identification and isolation of specific regions of interest, facilitating a more comprehensive analysis of breast cancer characteristics. In conclusion, the study presents a creative approach to detecting and categorizing breast cancer, demonstrating the superior performance of the EDCNN model compared to well-established transfer learning models. Through advanced deep learning techniques and image segmentation, this study contributes to improving diagnosis and treatment outcomes in breast cancer.

Interpretable detection of epiretinal membrane from optical coherence tomography with deep neural networks

This study aimed to automatically detect epiretinal membranes (ERM) in various OCT-scans of the central and paracentral macula region and classify them by size using deep-neural-networks (DNNs). To this end, 11,061 OCT-images were included and graded according to the presence of an ERM and its size (small 100–1000 µm, large > 1000 µm). The data set was divided into training, validation and test sets (75%, 10%, 15% of the data, respectively). An ensemble of DNNs was trained and saliency maps were generated using Guided-Backprob. OCT-scans were also transformed into a one-dimensional-value using t-SNE analysis. The DNNs’ receiver-operating-characteristics on the test set showed a high performance for no-ERM, small-ERM and large-ERM cases (AUC: 0.99, 0.92, 0.99, respectively; 3-way accuracy: 89%), with small-ERMs being the most difficult ones to detect. t-SNE analysis sorted cases by size and, in particular, revealed increased classification uncertainty at the transitions between groups. Saliency maps reliably highlighted ERM, regardless of the presence of other OCT features (i.e. retinal-thickening, intraretinal pseudo-cysts, epiretinal-proliferation) and entities such as ERM-retinoschisis, macular-pseudohole and lamellar-macular-hole. This study showed therefore that DNNs can reliably detect and grade ERMs according to their size not only in the fovea but also in the paracentral region. This is also achieved in cases of hard-to-detect, small-ERMs. In addition, the generated saliency maps can be used to highlight small-ERMs that might otherwise be missed. The proposed model could be used for screening-programs or decision-support-systems in the future.

Weight Averaging and re-adjustment ensemble for QRCD

Question Answering (QA) is a prominent task in the field of Natural Language Processing (NLP) with extensive applications. Recently, there has been a notable surge in research interest concerning the development of QA systems for the Holy Qur’an, an Islamic religious text. The Qur’an Reading Comprehension Dataset (QRCD) Malhas and Elsayed (2020) is a highly commendable effort in this respect. It stands as the first benchmark dataset specifically designed for a set of directly answerable questions from the Qur’an. Each question in the dataset is meticulously labeled with all potential answers sourced from the Holy Qur’an. From our perspective, the main challenge in QRCD stems from the limited volume of training data it offers. As a solution we propose an innovative approach to build a Deep Neural Network (DNN) ensemble, centered around Ara-Electra model (Antoun et al., 2021), that we called Weight Averaging and Re-adjustment (WAR) model. The model is constructed by computing running averages of all model states that evolve during a single training session and ensuring that model weights are readjusted prior to each training epoch, in order to hold it back from over fitting the training data. The scheme results in a single standalone model that exhibits the benefits of multi-model ensembles. It is distinguished from other ensembles proposed for QRCD that accumulates outputs from multiple expert models and employs classic techniques like hard voting or score averaging on output probabilities to build unified results. Each expert model costs individual training time and compute resources. The WAR model outperforms existing systems with improved generalization over unseen data. It achieves F1, partial Reciprocal Rank (pRR), and exact-match (EM) scores of 0.567, 0.60 and 0.29 respectively, exceeding best reported QRCD scores by 3%, 1.5% and 0.69% respectively. Notably, we are comparing our results with the top scores from different models, highlighting our model’s consistent performance across all three metrics.

Improving migration forecasting for transitory foreign tourists using an Ensemble DNN-LSTM model

The tourism industry is a key component in many nations’ economies, and precise Forecasting of the number of arrivals of transient international visitors is necessary for efficient resource allocation and strategic planning. Traditional approaches for predicting time series need to catch up when capturing the intricate patterns and interdependencies in data about visitor movement. To overcome this obstacle, we have developed a model that combines Ensemble Deep Neural Networks (EDNN) with Long Short-Term Memories (LSTM) to enhance migration predictions. EDNNs learn complex patterns from historical data, whereas LSTMs model sequential relationships. We start with data from the study-based Migration Forecasting for Transitory Foreign Tourists. The data was preprocessed using Adaptive mean filter (AMF) methods. We used Linear Discriminant Analysis (LDA) to look at different configurations for extracting features. The EDNN-LSTM method had better results than its comparison in terms of a wide range of performance measures having an accuracy of 97%, an MAE of 40.75%, an MSE of 33.10%, and an RMSE of 30.10%. According to the results, the EDNN-LSTM model can represent the intricate dynamics of transitory foreign visitor movement. Consequently, this model is a useful resource for tourism stakeholders, policymakers, and enterprises.

DiagSet: a dataset for prostate cancer histopathological image classification

Cancer diseases constitute one of the most significant societal challenges. In this paper, we introduce a novel histopathological dataset for prostate cancer detection. The proposed dataset, consisting of over 2.6 million tissue patches extracted from 430 fully annotated scans, 4675 scans with assigned binary diagnoses, and 46 scans with diagnoses independently provided by a group of histopathologists can be found at https://github.com/michalkoziarski/DiagSet. Furthermore, we propose a machine learning framework for detection of cancerous tissue regions and prediction of scan-level diagnosis, utilizing thresholding to abstain from the decision in uncertain cases. The proposed approach, composed of ensembles of deep neural networks operating on the histopathological scans at different scales, achieves 94.6% accuracy in patch-level recognition and is compared in a scan-level diagnosis with 9 human histopathologists showing high statistical agreement.

Visual acuity prediction on real-life patient data using a machine learning based multistage system

In ophthalmology, intravitreal operative medication therapy (IVOM) is a widespread treatment for diseases related to the age-related macular degeneration (AMD), the diabetic macular edema, as well as the retinal vein occlusion. However, in real-world settings, patients often suffer from loss of vision on time scales of years despite therapy, whereas the prediction of the visual acuity (VA) and the earliest possible detection of deterioration under real-life conditions is challenging due to heterogeneous and incomplete data. In this contribution, we present a workflow for the development of a research-compatible data corpus fusing different IT systems of the department of ophthalmology of a German maximum care hospital. The extensive data corpus allows predictive statements of the expected progression of a patient and his or her VA in each of the three diseases. For the disease AMD, we found out a significant deterioration of the visual acuity over time. Within our proposed multistage system, we subsequently classify the VA progression into the three groups of therapy “winners”, “stabilizers”, and “losers” (WSL classification scheme). Our OCT biomarker classification using an ensemble of deep neural networks results in a classification accuracy (F1-score) of over 98%, enabling us to complete incomplete OCT documentations while allowing us to exploit them for a more precise VA modelling process. Our VA prediction requires at least four VA examinations and optionally OCT biomarkers from the same time period to predict the VA progression within a forecasted time frame, whereas our prediction is currently restricted to IVOM/no therapy. We achieve a final prediction accuracy of 69% in macro average F1-score, while being in the same range as the ophthalmologists with 57.8 and 50±10.7\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$50 \\pm 10.7$$\\end{document}% F1-score.

An Empirical Evaluation of a Novel Ensemble Deep Neural Network Model and Explainable AI for Accurate Segmentation and Classification of Ovarian Tumors Using CT Images.

Ovarian cancer is one of the leading causes of death worldwide among the female population. Early diagnosis is crucial for patient treatment. In this work, our main objective is to accurately detect and classify ovarian cancer. To achieve this, two datasets are considered: CT scan images of patients with cancer and those without, and biomarker (clinical parameters) data from all patients. We propose an ensemble deep neural network model and an ensemble machine learning model for the automatic binary classification of ovarian CT scan images and biomarker data. The proposed model incorporates four convolutional neural network models: VGG16, ResNet 152, Inception V3, and DenseNet 101, with transformers applied for feature extraction. These extracted features are fed into our proposed ensemble multi-layer perceptron model for classification. Preprocessing and CNN tuning techniques such as hyperparameter optimization, data augmentation, and fine-tuning are utilized during model training. Our ensemble model outperforms single classifiers and machine learning algorithms, achieving a mean accuracy of 98.96%, a precision of 97.44%, and an F1-score of 98.7%. We compared these results with those obtained using features extracted by the UNet model, followed by classification with our ensemble model. The transformer demonstrated superior performance in feature extraction over the UNet, with a mean Dice score and mean Jaccard score of 0.98 and 0.97, respectively, and standard deviations of 0.04 and 0.06 for benign tumors and 0.99 and 0.98 with standard deviations of 0.01 for malignant tumors. For the biomarker data, the combination of five machine learning models-KNN, logistic regression, SVM, decision tree, and random forest-resulted in an improved accuracy of 92.8% compared to single classifiers.

Towards Detection and Attribution of Cyber Attacks in IoT Enabled Cyber-Physical Systems

Internet of Things (IoT) enabled cyber physical systems such as Industrial equipment’s and operational IT to send and receive data over internet. This equipment’s will have sensors to sense equipment condition and report to centralized server using internet connection. Sometime some malicious users may attack or hack such sensors and then alter their data and this false data will be report to centralized server and false action will be taken. Due to false data many countries equipment and production system got failed and many algorithms was developed to detect attack, but all these algorithms suffer from data imbalance (one class my contains huge records (for example NORMAL records and other class like attack may contains few records which lead to imbalance problem and detection algorithms may failed to predict accurately). To deal with data imbalance, existing algorithms were using OVER and UNDER sampling which will generate new records for FEWER class only. Securing Internet of Things (IoT)-enabled cyberphysical systems (CPS) can be challenging, as security solutions developed for general information / operational technology (IT / OT) systems may not be as effective in a CPS setting. Thus, this paper presents a two-level ensemble attack detection and attribution framework designed for CPS, and more specifically in an industrial control system (ICS). At the first level, a decision tree combined with a novel ensemble deep representation learning model is developed for detecting attacks imbalanced ICS environments. At the second level, an ensemble deep neural network is designed for attack attribution.The connection between ICS or IIoT-based systems with public networks, however, increases their attack surfaces and risks of being targeted by cyber criminals.To overcome from this issue, we are introducing novel technique without using any under or oversampling algorithms. The proposed technique consists of 2 parts. Auto encoder: It will get trained on imbalanced dataset and then extract features from it and these extracted features will get trained with DECISION TREE algorithm to predict label for known or unknown attacks. Decision tree get trained on reduced number of features obtained from PCA (principal component analysis) algorithm. Deep Neural Network (DNN): In this level, DNN algorithm get trained on known and unknown attacks. Key Word: Cyber Physical Systems (CPS), Industrial Control System (ICS).

Multiple microgrid sustainable energy management employing ensemble deep neural network to enhance QoS in stealthy false data

Multiple microgrid sustainable energy management employing ensemble deep neural network to enhance QoS in stealthy false data

Hierarchical Pruning of Deep Ensembles with Focal Diversity

Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study and apply deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This article presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal ensemble diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble team, which can guide ensemble pruning. Second, we design a focal ensemble diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high ensemble execution efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better classification generalizability while being more time and space efficient in ensemble decision making. We have released the source codes on GitHub at https://github.com/git-disl/HQ-Ensemble .

Ensemble of Deep Features for Breast Cancer Histopathological Image Classification

Abstract Analysis of histopathological images (HIs) is crucial for detecting breast cancer (BR). However, because they vary, it is still very difficult to extract well-designed elements. Deep learning (DL) is a recent development that is used to extract high-level features. However, DL techniques continue to confront several difficult problems, such as the need for sufficient training data for DL models, which reduces the classification findings. In this study, an ensemble deep transfer convolutional neural network is presented to address this problem. The pre-trained models (ResNet50 and MobileNet) are employed to extract high-level features by freezing the front layer parameters while fine-tuning the last layers. In the proposed ensemble framework, KNN, SVM, logistic regression and neural networks are used as base classifiers. The majority vote and product approaches are used to integrate the predictions of each separate classifier. In the benchmark BreaKHis dataset, the suggested ensemble model is compared to some current approaches. It demonstrates that while the ensemble model obtains a considerable accuracy of 97.72% for the multiclass classification test, it achieves an accuracy of 99.2% for the binary task. The suggested ensemble model’s effectiveness in extracting useful features for BR images is demonstrated by comparison with existing cutting-edge models.

Interpretable Wind Power Short-Term Power Prediction Model Using Deep Graph Attention Network

High-precision spatial-temporal wind power prediction technology is of great significance for ensuring the safe and stable operation of power grids. The development of artificial intelligence technology provides a new scheme for modeling with strong spatial-temporal correlation. In addition, the existing prediction models are mostly ‘black box’ models, lacking interpretability, which may lead to a lack of trust in the model by power grid dispatchers. Therefore, improving the model to obtain interpretability has become an important challenge. In this paper, an interpretable short-term wind power prediction model based on ensemble deep graph neural network is designed. Firstly, the graph network model (GNN) with an attention mechanism is applied to the aggregate and the spatial-temporal features of wind power data are extracted, and the interpretable ability is obtained. Then, the long short-term memory (LSTM) method is used to process the extracted features and establish a wind power prediction model. Finally, the random sampling algorithm is used to optimize the hyperparameters to improve the learning rate and performance of the model. Through multiple comparative experiments and a case analysis, the results show that the proposed model has a higher prediction accuracy than other traditional models and obtains reasonable interpretability in time and space dimensions.

A deep transfer learning approach to construct the allowable load space of notched composite laminates

Allowable load is one of the core parameters for the design and service of composite structures (CSs). However, due to the large design parameters, cumbersome design procedure and complicated simulation model of CSs, considerable amounts of experiments and simulations are needed to determine the allowable load of CSs, which is costly and inefficient, hindering the wide application of CSs in industry. In this work, we present a general deep transfer learning approach to construct the allowable load space (ALS) of notched laminates by considering various design parameters such as geometries, materials, stacking angles, ply numbers, laminate types and load types. Initially, an ensemble deep neural network (Ensemble) is trained, which can predict the allowable loads of notched laminate with known design parameters well. Here the Ensemble guided by ensemble learning is trained to reduce the errors generated by manually setting. Then, the pre-trained Ensemble is transferred to new design parameters by fine-tuning it with scarce new samples. Finally, the ALS of notch laminates is constructed by integrating the pre-trained and fine-tuned models, covering both known and diverse new design parameters. Our approach can easily be extended to other CSs.

Enhancing the Odia Handwritten Character and Numeral Recognition System's Performance with an Ensemble of Deep Neural Networks

Enhancing the Odia Handwritten Character and Numeral Recognition System's Performance with an Ensemble of Deep Neural Networks

Segmentation and Classification of Diabetic Retinopathy using Ensemble Deep Neural Network

Diabetic Retinopathy (DR) is an eye ailment in diabetics, often causing vision loss. Timely retinal screenings can help prevent blindness. While DR cannot be cured, early detection can curb vision decline. Ophthalmologists diagnose DR using Optical Coherence Tomography (OCT) or Fundus Images. Evaluating via fundus images can be complex and potentially imprecise. The study offers a computer-supported diagnosis to aid ophthalmologists. It uses Multi-level Otsu thresholding to segment Fundus Images, with segmented outputs processed further. Advanced feature extraction techniques like the Gray-Level Co-occurrence Matrix (GLCM) and dynamic Flemingo optimization enhance DR feature identification. Additionally, a novel cascaded voting ensemble deep neural network model is introduced, merging multiple algorithmic insights to improve classification. The paper concludes by aligning classifications with a standard grading system, giving clinicians a clear DR severity gauge for better treatment planning.

An Efficient and Robust Approach Using Inductive Transfer-Based Ensemble Deep Neural Networks for Kidney Stone Detection

An Efficient and Robust Approach Using Inductive Transfer-Based Ensemble Deep Neural Networks for Kidney Stone Detection

Graph Learning and Deep Neural Network Ensemble for Supporting Cognitive Decline Assessment

Cognitive decline represents a significant public health concern due to its severe implications on memory and general health. Early detection is crucial to initiate timely interventions and improve patient outcomes. However, traditional diagnosis methods often rely on personal interpretations or biases, may not detect the early stages of cognitive decline, or involve invasive screening procedures; thus, there is a growing interest in developing non-invasive methods benefiting also from the technological advances. Wearable devices and Internet of Things sensors can monitor various aspects of daily life together with health parameters and can provide valuable data regarding people’s behavior. In this paper, we propose a technical solution that can be useful for potentially supporting cognitive decline assessment in early stages, by employing advanced machine learning techniques for detecting higher activity fragmentation based on daily activity monitoring using wearable devices. Our approach also considers data coming from wellbeing assessment questionnaires that can offer other important insights about a monitored person. We use deep neural network models to capture complex, non-linear relationships in the daily activities data and graph learning for the structural wellbeing information in the questionnaire answers. The proposed solution is evaluated in a simulated environment on a large synthetic dataset, the results showing that our approach can offer an alternative as a support for early detection of cognitive decline during patient-assessment processes.

Deep learning predicts cervical lymph node metastasis in clinically node-negative papillary thyroid carcinoma

ObjectivesPrecise determination of cervical lymph node metastasis (CLNM) involvement in patients with early-stage thyroid cancer is fairly significant for identifying appropriate cervical treatment options. However, it is almost impossible to directly judge lymph node metastasis based on the imaging information of early-stage thyroid cancer patients with clinically negative lymph nodes.MethodsPreoperative US images (BMUS and CDFI) of 1031 clinically node negative PTC patients definitively diagnosed on pathology from two independent hospitals were divided into training set, validation set, internal test set, and external test set. An ensemble deep learning model based on ResNet-50 was built integrating clinical variables, BMUS, and CDFI images using a bagging classifier to predict metastasis of CLN. The final ensemble model performance was compared with expert interpretation.ResultsThe ensemble deep convolutional neural network (DCNN) achieved high performance in predicting CLNM in the test sets examined, with area under the curve values of 0.86 (95% CI 0.78–0.94) for the internal test set and 0.77 (95% CI 0.68–0.87) for the external test set. Compared to all radiologists averaged, the ensemble DCNN model also exhibited improved performance in making predictions. For the external validation set, accuracy was 0.72 versus 0.59 (p = 0.074), sensitivity was 0.75 versus 0.58 (p = 0.039), and specificity was 0.69 versus 0.60 (p = 0.078).ConclusionsDeep learning can non-invasive predict CLNM for clinically node-negative PTC using conventional US imaging of thyroid cancer nodules and clinical variables in a multi-institutional dataset with superior accuracy, sensitivity, and specificity comparable to experts.Critical relevance statementDeep learning efficiently predicts CLNM for clinically node-negative PTC based on US images and clinical variables in an advantageous manner.Key points• A deep learning-based ensemble algorithm for predicting CLNM in PTC was developed.• Ultrasound AI analysis combined with clinical data has advantages in predicting CLNM.• Compared to all experts averaged, the DCNN model achieved higher test performance.Graphical

Deep neural network ensembles for THz-TDS refractive index extraction exhibiting resilience to experimental and analytical errors.

Terahertz time-domain spectroscopy (THz-TDS) achieves excellent signal-to-noise ratios by measuring the amplitude of the electric field in the time-domain, resulting in the full, complex, frequency-domain information of materials' optical parameters, such as the refractive index. However the data extraction process is non-trivial and standardization of practices are still yet to be cemented in the field leading to significant variation in sample measurements. One such contribution is low frequency noise offsetting the phase reconstruction of the Fourier transformed signal. Additionally, experimental errors such as fluctuations in the power of the laser driving the spectrometer (laser drift) can heavily contribute to erroneous measurements if not accounted for. We show that ensembles of deep neural networks trained with synthetic data extract the frequency-dependent complex refractive index, whereby required fitting steps are automated and show resilience to phase unwrapping variations and laser drift. We show that training with synthetic data allows for flexibility in the functionality of networks yet the produced ensemble supersedes current extraction techniques.

Composing Diverse Ensembles of Convolutional Neural Networks by Penalization

Ensemble-based systems are well known to have the capacity to outperform individual approaches if the ensemble members are sufficiently accurate and diverse. This paper investigates how an efficient ensemble of deep convolutional neural networks (CNNs) can be created by forcing them to adjust their parameters during the training process to increase diversity in their decisions. As a new theoretical approach to reach this aim, we join the member neural architectures via a fully connected layer and insert a new correlation penalty term in the loss function to obstruct their similar operation. With this complementary term, we implement the standard guideline of ensemble creation to increase the members’ diversity for CNNs in a more detailed and flexible way than similar existing techniques. As for applicability, we show that our approach can be efficiently used in various classification tasks. More specifically, we demonstrate its performance in challenging medical image analysis and natural image classification problems. Besides the theoretical considerations and foundations, our experimental findings suggest that the proposed technique is competitive. Namely, on the one hand, the classification rate of the ensemble trained in this way outperformed all the individual accuracies of the state-of-the-art member CNNs according to the standard error functions of these application domains. On the other hand, it is also validated that the ensemble members get more diverse and their accuracies are raised by adding the penalization term. Moreover, we performed a full comparative analysis, including other state-of-the-art ensemble-based approaches recommended for the same classification tasks. This comparative study also confirmed the superiority of our method, as it overcame the current solutions.