Convolutional Neural Network Training Research Articles

Deep learning with uncertainty estimation for automatic tumor segmentation in PET/CT of head and neck cancers: impact of model complexity, image processing and augmentation

Objective. Target volumes for radiotherapy are usually contoured manually, which can be time-consuming and prone to inter- and intra-observer variability. Automatic contouring by convolutional neural networks (CNN) can be fast and consistent but may produce unrealistic contours or miss relevant structures. We evaluate approaches for increasing the quality and assessing the uncertainty of CNN-generated contours of head and neck cancers with PET/CT as input. Approach. Two patient cohorts with head and neck squamous cell carcinoma and baseline 18F-fluorodeoxyglucose positron emission tomography and computed tomography images (FDG-PET/CT) were collected retrospectively from two centers. The union of manual contours of the gross primary tumor and involved nodes was used to train CNN models for generating automatic contours. The impact of image preprocessing, image augmentation, transfer learning and CNN complexity, architecture, and dimension (2D or 3D) on model performance and generalizability across centers was evaluated. A Monte Carlo dropout technique was used to quantify and visualize the uncertainty of the automatic contours. Main results. CNN models provided contours with good overlap with the manually contoured ground truth (median Dice Similarity Coefficient: 0.75–0.77), consistent with reported inter-observer variations and previous auto-contouring studies. Image augmentation and model dimension, rather than model complexity, architecture, or advanced image preprocessing, had the largest impact on model performance and cross-center generalizability. Transfer learning on a limited number of patients from a separate center increased model generalizability without decreasing model performance on the original training cohort. High model uncertainty was associated with false positive and false negative voxels as well as low Dice coefficients. Significance. High quality automatic contours can be obtained using deep learning architectures that are not overly complex. Uncertainty estimation of the predicted contours shows potential for highlighting regions of the contour requiring manual revision or flagging segmentations requiring manual inspection and intervention.

Automated Assessment of Wheat Leaf Disease Spore Concentration Using a Smart Microscopy Scanning System

An advanced approach to the automated assessment of a microscopic slide containing spores is presented. The objective is to develop an intelligent system for the rapid and precise estimation of phytopathogenic spore concentration on microscopic slides, thereby enabling automated processing. The smart microscopy scanning system comprises an electronic microscope, a coordinate table, and software for the control of the coordinate table and image processing. The developed smart microscopy scanning system processes the entire microscope slide with multiple exposed strips, which are automatically determined based on the novel two-stage algorithm. The analysis of trained convolutional neural networks employed for the detection of spore phytopathogens demonstrates high precision and recall metrics. The system is capable of identifying and counting the number of spores of phytopathogenic fungi species Blumeria graminis, Puccinia striiformis, and Pyrenophora tritici-repentis on each exposed strip. A methodology for estimating the spore distribution on a microscopic slide is proposed, which involves calculating the average spore concentration density.

Beyond signatures and thresholds: Intelligent DDoS detection using Convolutional Neural Networks

Distributed Denial of Service (DDoS) attacks have evolved into sophisticated threats against digital infrastructure. This research investigates Convolutional Neural Networks (CNNs) for DDoS detection using the CIC-IDS2017 dataset. A CNN model was developed, leveraging convolutional and pooling layers for hierarchical feature extraction. Preprocessing ensured dataset consistency and generalization. The model achieved 99.978% test accuracy, demonstrating proficiency in recognizing DDoS patterns. Comparatively, a Deep Neural Network (DNN) benchmark obtained 99.9512% accuracy, indicating CNNs superiority. However, CNN training time was longer. While results highlight deep learning's potential against DDoS attacks, optimizations for real-time deployment warrant exploration. This empirical evaluation and comparative analysis enriches the discourse on utilizing machine learning for robust cybersecurity.

Regression convolutional neural network models implicate peripheral immune regulatory variants in the predisposition to Alzheimer's disease.

Alzheimer's disease (AD) involves aggregation of amyloid β and tau, neuron loss, cognitive decline, and neuroinflammatory responses. Both resident microglia and peripheral immune cells have been associated with the immune component of AD. However, the relative contribution of resident and peripheral immune cell types to AD predisposition has not been thoroughly explored due to their similarity in gene expression and function. To study the effects of AD-associated variants on cis-regulatory elements, we train convolutional neural network (CNN) regression models that link genome sequence to cell type-specific levels of open chromatin, a proxy for regulatory element activity. We then use in silico mutagenesis of regulatory sequences to predict the relative impact of candidate variants across these cell types. We develop and apply criteria for evaluating our models and refine our models using massively parallel reporter assay (MPRA) data. Our models identify multiple AD-associated variants with a greater predicted impact in peripheral cells relative to microglia or neurons. Our results support their use as models to study the effects of AD-associated variants and even suggest that peripheral immune cells themselves may mediate a component of AD predisposition. We make our library of CNN models and predictions available as a resource for the community to study immune and neurological disorders.

CNN-Based Damage Identification of Submerged Structure-Foundation System Using Vibration Data

This study presents a convolutional neural network (CNN) deep learning approach for identifying damage in submerged structure-foundation systems using vibration data. Firstly, foundation damage in a lab-scale caisson-foundation system is simulated to measure time-history responses. Singular value decomposition (SVD) responses are derived from the time-history responses. Secondly, the 1-D CNN deep learning model is trained using both the time-history responses and SVD responses. Finally, the trained CNN models are implemented to evaluate the foundation damage under conditions of noise contamination and partially untrained data. The experimental results demonstrate the effectiveness of CNN models for damage identification and highlight the comparative strengths of time-history and SVD data. The CNN model trained using SVD data outperforms the other model when under noise contamination conditions, while the CNN model trained using time-history data maintains better accuracy in partially untrained data conditions. Integrating both types of data enhances the accuracy of damage classification.

Integrated deep learning model for automatic detection and classification of stenosis in coronary angiography

Coronary artery disease poses a significant threat to human health. In clinical settings, coronary angiography remains the gold standard for diagnosing coronary heart disease. A crucial aspect of this diagnosis involves detecting arterial narrowings. Categorizing these narrowings can provide insight into whether patients should receive vascular revascularization treatment. The majority of current deep learning methods for analyzing coronary angiography are mostly confined to the theoretical research domain, with limited studies offering direct practical support to clinical practitioners. This paper proposes an integrated deep-learning model for the localization and classification of narrowings in coronary angiography images. The experimentation employed 1606 coronary angiography images obtained from 132 patients, resulting in an accuracy of 88.9 %, a recall rate of 85.4 %, an F1 score of 0.871, and a MAP value of 0.875 for vascular stenosis detection. Furthermore, we developed the "Hemadostenosis" web platform (http://bioinfor.imu.edu.cn/hemadostenosis) using Django, a highly mature HTTP framework. Users are able to submit coronary angiography image data for assessment via a visual interface. Subsequently, the system sends the images to a trained convolutional neural network model to localize and categorize the narrowings. Finally, the visualized outcomes are displayed to users and are downloadable. Our proposed approach pioneers the recognition and categorization of arterial narrowings in vascular angiography, offering practical support to clinical practitioners in their learning and diagnostic processes.

Efficient model updating of shaft-raft-hull system using multi-stage convolutional neural network combined with sensitivity analysis

Model updating for marine shaft-raft-hull systems presents significant challenges due to the numerous components and the resulting inaccessible parameters. This study introduces an advanced framework for updating the model parameters of these complex coupling systems using convolutional neural networks (CNNs). Rapid analysis technique based on the substructure synthesis method is employed to enhance the efficiency of generating the CNN training dataset. Global sensitivity analysis is then utilized to identify critical parameters across various frequency bands. Informed by parameter sensitivity, the frequency bands are segmented, and a multi-stage CNN model updating strategy is proposed. The effectiveness of this method is validated through numerical simulations and experimental studies on a scaled shaft-raft-hull model. The findings demonstrate that segmenting the dataset based on global sensitivity analysis results markedly improves the convergence of CNNs, providing a robust solution for model updating in complex marine systems.

Hand Gesture Recognition Using Deep Learning

Hand gesture recognition (HGR) has gained significant attention due to its potential for various applications. This paper explores the use of deep learning, specifically Convolutional Neural Networks (CNNs), for HGR using the TensorFlow library. We investigate existing research on CNN-based HGR, focusing on image classification tasks. We then provide a brief overview of CNNs and their suitability for image recognition. Subsequently, we describe the typical workflow of a deep learning-based HGR system, including data preprocessing, hand detection, feature extraction with CNNs, and classification. We highlight the advantages of using TensorFlow to build and train CNN models for HGR. Finally, we conclude by summarizing the key findings from related work and mentioning the specific dataset and number of gestures classified in our research. This work contributes to the growing body of research on CNN-based HGR using TensorFlow and emphasizes its potential for developing accurate and efficient HGR systems.

An analytic formulation of convolutional neural network learning for pattern recognition

Training convolutional neural networks (CNNs) using back-propagation (BP) is a time-consuming and resource-intensive process, primarily due to the need to iterate over the dataset multiple times. In contrast, analytic learning aims to train neural networks in a single epoch, offering a potential solution to these challenges. However, existing studies of analytic learning have been limited to multilayer perceptrons (MLPs). In this article, we propose an analytic formulation for convolutional neural network learning (ACnnL), which represents a significant advancement towards non-iterative learning paradigms for CNNs. Our formulation demonstrates that ACnnL extends the principles of MLP regularization constraints. From the implicit regularization and network interpretability viewpoints, we provide insights into why CNNs often exhibit superior generalization capabilities. The ACnnL is validated by conducting classification tasks on benchmark datasets such as MNIST, FashionMNIST, CIFAR10, CIFAR100 and Tiny-ImageNet. It is encouraging that the ACnnL trains CNNs in a significantly fast manner with reasonably close prediction accuracies to those using BP. In particular, a 5-layer vanilla CNN trained by ACnnL gave an accuracy of 0.9931, 0.9155, 0.7049 and 0.4628 for these datasets. The ACnnL achieves training speeds that are approximately 17 times faster than BP on GPU and 113 times faster than BP on CPU, while maintaining competitive prediction accuracies. Moreover, our experiments disclose a unique advantage of ACnnL under the small-sample scenario when training data are scarce or expensive. In a nutshell, an analytic method which deals well with small-sample size data has been put forward for the first time for fast CNN training with inherent network interpretability.

Energy-efficient neural network training through runtime layer freezing, model quantization, and early stopping

Background:In the last years, neural networks have been massively adopted by industry and research in a wide variety of contexts. Neural network milestones are generally reached by scaling up computation, completely disregarding the carbon footprint required for the associated computations. This trend has become unsustainable given the ever-growing use of deep learning, and could cause irreversible damage to the environment of our planet if it is not addressed soon. Objective:In this study, we aim to analyze not only the effects of different energy saving methods for neural networks but also the effects of the moment of intervention, and what makes certain moments optimal. Method:We developed a novel dataset by training convolutional neural networks in 12 different computer vision datasets and applying runtime decisions regarding layer freezing, model quantization and early stopping at different epochs in each run. We then fit an auto-regressive prediction model on the data collected capable to predict the accuracy and energy consumption achieved on future epochs for different methods. The predictions on accuracy and energy are used to estimate the optimal training path. Results:Following the predictions of the model can save 56.5% of energy consumed while also increasing validation accuracy by 2.38% by avoiding overfitting.The prediction model developed can predict the validation accuracy with a 8.4% of error, the energy consumed with a 14.3% of error and the trade-off between both with a 8.9% of error. Conclusions:This prediction model could potentially be used by the training algorithm to decide which methods apply to the model and at what moment in order to maximize the accuracy-energy trade-off.

CINNAMON-GUI: Revolutionizing Pap Smear Analysis with CNN-Based Digital Pathology Image Classification

Background Medical imaging has seen significant advancements through machine learning, particularly convolutional neural networks (CNNs). These technologies have transformed the analysis of pathological images, enhancing the accuracy of diagnosing and classifying cellular anomalies. Digital pathology methodologies, including image analysis, have improved cervical cancer diagnostics. However, existing commercial platforms are often costly and restrictive, limiting customization and scalability. Methods CINNAMON-GUI is an open-source digital pathology tool based on CNNs for classifying Pap smear images. Transitioning to a Shiny app in Python, it offers enhanced user interface and interactivity. The application supports dynamic web interactions, advanced features for image analysis, and state-of-the-art CNN models tailored for digital pathology. Key features include intuitive UI components, real-time image and plot generation, memory-efficient data handling, and robust training capabilities with customizable CNN architectures. The tool also integrates with Labelme for defining regions of interest and allows testing on external biospecimens. Results Model A (seed 42, 100 epochs) and Model B (same architecture with adjusted augmentation parameters) were compared. Model A stabilized with training accuracy around 0.88 and validation accuracy around 0.913. Model B showed improved performance with training accuracy around 0.91 and validation accuracy around 0.95. Feature mapping highlighted critical morphological aspects, improving classification accuracy. Model B reduced misclassification errors significantly compared to Model A. Conclusions CINNAMON-GUI demonstrates the potential of an open-source platform in digital pathology, providing transparency and collaborative opportunities. The tool enhances diagnostic accuracy through feature map analysis and optimized CNN training. Future development aims to extend its application to other cancer types, leveraging its dynamic and user-friendly interface for broader use in diagnostics.

Measurement of Centroid Distance in Determining Stunting Clusters

This study aims to develop a vegetable freshness classification model using Convolutional Neural Networks (CNN). The methodology used includes collecting vegetable image datasets, image preprocessing, training CNN models, and evaluating classification accuracy. The dataset used consists of various types of vegetables with varying levels of freshness. The results of the study show that the developed CNN model is able to classify vegetable freshness with high accuracy, so it can assist in the automation process of determining vegetable freshness in the food industry. Model evaluation is carried out using accuracy metrics and mean squared error (MSE).

A hybrid approach using convolutional neural networks and genetic algorithm to improve of sensing brain tumor prediction

Brain tumor is the most constantly diagnosed cancer, and the opinion of the brain is veritably sensitive and complex, which is the subject of numerous studies and inquiries. In computer vision, deep literacy ways, similar as the convolutional neural network (CNN), are employed due to their bracket capabilities using learned point styles and their capability to work with complex images. still, their performance is largely dependent on the network structure and the named optimization system for tuning the network parameters. In this paper, we present new yet effective styles for training convolutional neural networks. The maturity of current state-of-the-art literacy styles for convolutional neural networks are grounded on grade descent. In discrepancy to traditional convolutional neural network training styles, we propose an enhancement by incorporating the inheritable algorithm for brain tumor prediction. Our work involves designing a convolutional neural network model to grease the bracket process, training the model using different optimizers (Adam and the inheritable algorithm), and assessing the model through colorful trials on the brain magnetic resonance imaging (MRI) dataset. We demonstrate that the convolutional neural network model trained using the inheritable algorithm performs as well as the Adam optimizer, achieving a bracket delicacy of 99.5.

Image-based quantification of pool boiling heat flux on varied heating surfaces: Enhancing prediction performance with automated machine learning

The present study applied image-trained Convolutional Neural Networks (CNN) to quantify the heat flux dissipated in a pool boiling experiment. In total, four heating surfaces were used to generate over 200 thousand nucleate and film boiling images in two visualization modes: direct and indirect. In the former, the flow images included the heating surface; in the latter, the image was cropped and the heating surface was omitted. Prior to training and testing the CNN, the images were preprocessed, by grayscaling, downscaling, size uniformization and standardization. The main goals were to assess the CNN’s capability to generalize to multiple operating conditions, and to quantify the performance of a CNN architecture optimized with Automated Machine Learning (AutoML) in comparison with a reference architecture. The results suggest that multi-dataset training is required to improve the CNN generalization. In other words, the CNN must be trained, even if partially, with images of the specific heating surface for which it is trying to infer the heat flux. However, the results indicate a significant performance drop for the results of multi-surface trained CNNs when compared with single-surface CNNs, suggesting limited generalization capability. Furthermore, the results obtained showed that AutoML was capable of increasing the performance of CNN models when compared with parametrically determined architectures. Also, optimized architectures tend to present a larger number of convolutional layers associated with dense blocks and, at the same time, a reduced number of trainable variables.

Machine learning-based non-destructive terahertz detection of seed quality in peanut

Rapid identification of peanut seed quality is crucial for public health. In this study, we present a terahertz wave imaging system using a convolutional neural network (CNN) machine learning approach. Terahertz waves are capable of penetrating the seed shell to identify the quality of peanuts without causing any damage to the seeds. The specificity of seed quality on terahertz wave images is investigated, and the image characteristics of five different qualities are summarized. Terahertz wave images are digitized and used for training and testing of convolutional neural networks, resulting in a high model accuracy of 98.7% in quality identification. The trained THz-CNNs system can accurately identify standard, mildewed, defective, dried and germinated seeds, with an average detection time of 2.2 s. This process does not require any sample preparation steps such as concentration or culture. Our method swiftly and accurately assesses shelled seed quality non-destructively.

Data‐driven target localization using adaptive radar processing and convolutional neural networks

AbstractLeveraging the advanced functionalities of modern radio frequency (RF) modeling and simulation tools, specifically designed for adaptive radar processing applications, this paper presents a data‐driven approach to improve accuracy in radar target localization post adaptive radar detection. To this end, we generate a large number of radar returns by randomly placing targets of variable strengths in a predefined area, using RFView®, a high‐fidelity, site‐specific, RF modeling & simulation tool. We produce heatmap tensors from the radar returns, in range, azimuth [and Doppler], of the normalized adaptive matched filter (NAMF) test statistic. We then train a regression convolutional neural network (CNN) to estimate target locations from these heatmap tensors, and we compare the target localization accuracy of this approach with that of peak‐finding and local search methods. This empirical study shows that our regression CNN achieves a considerable improvement in target location estimation accuracy. The regression CNN offers significant gains and reasonable accuracy even at signal‐to‐clutter‐plus‐noise ratio (SCNR) regimes that are close to the breakdown threshold SCNR of the NAMF. We also study the robustness of our trained CNN to mismatches in the radar data, where the CNN is tested on heatmap tensors collected from areas that it was not trained on. We show that our CNN can be made robust to mismatches in the radar data through few‐shot learning, using a relatively small number of new training samples.

Evaluating CNN Models for Gait Recognition: A Study on the CASIA-B Dataset

Gait recognition, a form of biometric identification, has garnered considerable interest for its ability to identify individuals from a distance. Yet, its effectiveness depends significantly on the precision of the underlying categorization models. This study thoroughly evaluates many Convolutional Neural Network (CNN) models to determine their effectiveness in properly identifying and categorizing gait patterns. We train and test several CNN architectures using advanced deep learning techniques on the widely known CASIA-B dataset, which is a benchmark dataset for gait recognition research. The analyses involve models like as VGG16, VGG19, NASNetLarge, NASNetMobile, EfficientNetB0, and Xception, each well-known for their effectiveness in picture categorization tasks. We evaluate the performance of these models by conducting thorough experiments and detailed analysis, focusing on accuracy, validation loss, and other pertinent metrics. The Xception model demonstrated the highest accuracy of 97.17% of the models assessed. This model frequently surpasses similar models, demonstrating its strength and effectiveness in precisely recognizing gait patterns. On the other hand, the accuracy of other models varied from 9.78% to 93.85%, demonstrating the variable levels of efficacy among various designs. The findings have significant implications for the creation and implementation of gait recognition systems. The high precision demonstrated by the Xception model highlights the promise of CNN-based methods in progressing gait identification technology. Our work highlights the significance of choosing suitable model architectures for achieving the best performance in gait detection tasks. In the future, research may focus on using larger datasets and exploring other CNN architectures to improve the accuracy and reliability of gait detection systems. Our goal is to enhance gait recognition technology to improve biometric identification for security, surveillance, and healthcare applications. GUB JOURNAL OF SCIENCE AND ENGINEERING, Vol 10(1), 2023 P 17-26

CNNs for remote extraction of urban features: A survey-driven benchmarking

Accurate extraction of urban features such as buildings and roads lays the foundation for the current trends of digital twins of urban systems to support planning, monitoring, navigation, and decision processes. The process of such extraction involves training convolutional neural networks (CNNs) on high-resolution earth observation (EO) images. The spatial resolution of images has increased to a centimetre level and the CNNs are fast evolving in computer vision. The last 10 years of this development have resulted in both high-performance and computationally efficient CNNs, but they are merely benchmarked under a uniform setting. We present a survey-driven benchmark of CNNs starting with a systematic survey of 165 research articles to understand the state-of-the-art of urban feature extraction. The survey looks for the most prominent urban feature, EO source, benchmark dataset, CNN-based deep learning configuration, and hyperparameters. Further, more CNNs are searched in the computer vision domain. Identified from the survey and search, 65 CNNs are trained and evaluated in an encoder–decoder configuration using a benchmark dataset under uniform settings. Extensive hyperparameter tuning of the best-performing CNN is performed with six optimisers and nine loss functions. The tuned CNN is then tested as an encoder in other state-of-the-art encoder–decoder networks. The CNNs and network configurations with the highest scores are further benchmarked on the Massachusetts Building and WHU Building datasets. The findings from this survey-driven benchmark of CNNs will be useful for both academia and industry involved in the science of earth observation and computer vision.

Quantum-enhanced forecasting: Leveraging quantum gramian angular field and CNNs for stock return predictions

This paper presents the Quantum Gramian Angular Field (QGAF) method, which integrates quantum computing with deep learning to enhance forecasting in time series analysis. This method effectively converts stock return time series data into a format compatible with Convolutional Neural Network (CNN) training. Our empirical tests, conducted on stock market data from China, Hong Kong, and the United States, demonstrate that QGAF significantly outperforms the traditional GAF approach in accuracy. These findings underscore the potential of combining quantum computing and deep learning in financial time series forecasting.

Gasoline Engine Misfire Fault Diagnosis Method Based on Improved YOLOv8

In order to realize the online diagnosis and prediction of gasoline engine fire faults, this paper proposes an improved misfire fault detection algorithm model based on YOLOv8 for sound signals of gasoline engines. The improvement involves substituting a C2f module in the YOLOv8 backbone network by a BiFormer attention module and another C2f module substituted by a CBAM module that combines channel and spatial attention mechanisms which enhance the neural network’s capacity to extract the complex features. The normal and misfire sound signals of a gasoline engine are processed by wavelet transformation and converted to time–frequency images for the training, verification, and testing of convolutional neural network. The experimental results show that the precision of the improved YOLOv8 algorithm model is 99.71% for gasoline engine fire fault tests, which is 2 percentage points higher than for the YOLOv8 network model. The diagnosis time of each sound is less than 100 ms, making it suitable for developing IoT devices for gasoline engine misfire fault diagnosis and driverless vehicles.