Training Deep Neural Networks Research Articles

Activated biochar production from young coconut waste (Cocos nucifera) as bioadsorbent: a pathway through Artificial Neural Network (ANN) optimization.

This pioneering work explores the immense potential of young coconut waste, a continuously marginalized residue of the food and beverage industry, to serve as an indispensable feedstock in the production of biochar. Through an examination of the key carbonization factors that include time, temperature, and concentrations of the activating agent, KOH, the outcomes offer relevant insights that could be leveraged to maximize biochar production for tailored applications. This study stands out for its innovative use of Artificial Neural Network (ANN) approaches for predictive modeling. Fifty datasets, supplemented with secondary data obtained from the literature and experiments, were utilized for the purposes of training, testing, and validating the neural network model. Here, the datasets were processed utilizing the Deep Neural Network (DNN) framework, which was designed and implemented with the minimal loss function framework feasible. The architectural configuration comprises the following; an input layer, four hidden layers (128-neuron dense layer, batch normalization, and 64-neuron dense layer, batch normalization), a dropout layer, and an output layer. With an R2 of 0.8238 for biochar yield and 0.7324 for iodine number, the trained DNN model showed a relatively high degree of accuracy in making predictions.

Deep-neural-network model for predicting ground motion parameters using earthquake horizontal-to-vertical spectral ratios

This study proposed a deep-neural-network (DNN) model for seismic ground motion prediction by utilizing a unified strong motion database by the National Research Institute for Earth Science and Disaster Resilience, and earthquake horizontal-to-vertical spectral ratio (EHVR) database in Japan. The model aims to enhance the accuracy of predictions by incorporating the EHVRs for complementing site effects, and utilizing existing ground motion prediction equations (GMPE) as the base model for source and propagation path effects. The hybrid approach enables the prediction of peak ground accelerations (PGAs), peak ground velocities (PGVs), and 5% damped absolute acceleration response spectra (SAs). After classifying the training and test sets from the database, the trained DNN models were applied on the test set to evaluate the performance of the predicted results. The accuracy assessment by the residuals, R-squared ( R2), and root mean square error (RMSE) between the predicted and observed values in the test set revealed the superior performance of the proposed model compared with the traditional GMPE with proxy-based site effects such as V S30s especially in predicting both the spectral amplitude and shape of SAs.

Tuning parameters of deep neural network training algorithms pays off: a computational study

AbstractThe paper aims to investigate the impact of the optimization algorithms on the training of deep neural networks with an eye to the interaction between the optimizer and the generalization performance. In particular, we aim to analyze the behavior of state-of-the-art optimization algorithms in relationship to their hyperparameters setting to detect robustness with respect to the choice of a certain starting point in ending on different local solutions. We conduct extensive computational experiments using nine open-source optimization algorithms to train deep Convolutional Neural Network architectures on an image multi-class classification task. Precisely, we consider several architectures by changing the number of layers and neurons per layer, to evaluate the impact of different width and depth structures on the computational optimization performance. We show that the optimizers often return different local solutions and highlight the strong correlation between the quality of the solution found and the generalization capability of the trained network. We also discuss the role of hyperparameters tuning and show how a tuned hyperparameters setting can be re-used for the same task on different problems achieving better efficiency and generalization performance than a default setting.

Image deduplication using efficient visual indexing and retrieval: optimizing storage, time and energy for deep neural network training

Image deduplication using efficient visual indexing and retrieval: optimizing storage, time and energy for deep neural network training

Nondeterministic Features in Deep neural network design, training and inference

Neural networks are increasingly integral to scientific modeling and a wide range of real-world applications. However, standard neural networks often lack reliable certainty and confidence, and can be poorly calibrated, leading to nondeterministic results. To address these challenges, researchers have focused on understanding and quantifying uncertainty in neural network predictions. This work offers a comprehensive discussion of uncertainty estimation in neural networks, discussing recent advances, current challenges, and future research opportunities. This work explains key sources of uncertainty, challenges from model uncertainty and irreducible data uncertainty. We explore various methods for modeling these uncertainties, such as deterministic neural networks, Bayesian neural networks (BNNs), ensembles, and test-time data augmentation approaches. Additionally, the paper provides practical examples from fields like medical data analysis, robotics, and autonomous driving, illustrating the challenges and requirements associated with uncertainty in real-world applications. We also examine the limitations of current uncertainty quantification methods in safety-critical applications, and provide an outlook on future developments aimed at broader adoption of these methods in diverse domains. This paper serves as a valuable resource for both newcomers and those experienced in the field of uncertainty estimation in neural networks.

An approach of 2D convolutional neural network–based seismic data fault interpretation with linear annotation and pixel thinking

AbstractThis article introduces an improved method for geological fault interpretation utilizing a 2D convolutional neural network approach with a focus on coalbed horizons, dealing with the problem as an image classification task. At the beginning, linear annotations, reflecting geological fault features, are applied to multiple seismic sections. By considering texture differences between fault and non‐fault areas, we construct samples that represent these distinct zones for training deep neural networks. Initially, fault annotations are transformed into single dots to facilitate pixel‐based processing. To depict a specific dot's geological structure, we employ a matrix clipped around the point, determined by a combination of range and step parameters. Convolutional layers generate filters equivalent to seismic data transformation, streamlining the need for analysis and selection of seismic attributes. The article discusses enhancing the efficiency of 2D convolutional neural network–based fault interpretation by optimizing sample selection, data extraction and model construction procedures. Through the incorporation of data from two mining areas (totalling 27.09 km2) in sample creation, the overall accuracy exceeds 0.99. Recognition extends seamlessly to unlabelled sections, showcasing the innovative technical route and methodology of fault interpretation with linear annotation and pixel‐based thinking. This study presents a method that integrates planar and raster thinking, transitioning from vision‐oriented geological structure annotation to algorithm‐oriented pixel location. The proposed 2D convolutional neural network–based matrix‐oriented fault/non‐fault binary classification demonstrates feasibility and reproducibility, offering a new automated approach for fault detection in coalbeds through convolutional neural network algorithms.

Enhancing early Parkinson’s disease detection through multimodal deep learning and explainable AI: insights from the PPMI database

Parkinson’s is the second most common neurodegenerative disease, affecting nearly 8.5M people and steadily increasing. In this research, Multimodal Deep Learning is investigated for the Prodromal stage detection of Parkinson's Disease (PD), combining different 3D architectures with the novel Excitation Network (EN) and supported by Explainable Artificial Intelligence (XAI) techniques. Utilizing data from the Parkinson’s Progression Markers Initiative, this study introduces a joint co-learning approach for multimodal fusion, enabling end-to-end training of deep neural networks and facilitating the learning of complementary information from both imaging and clinical modalities. DenseNet with EN outperformed other models, showing a substantial increase in accuracy when supplemented with clinical data. XAI methods, such as Integrated Gradients for ResNet and DenseNet, and Attention Heatmaps for Vision Transformer (ViT), revealed that DenseNet focused on brain regions believed to be critical to prodromal pathophysiology, including the right temporal and left pre-frontal areas. Similarly, ViT highlighted the lateral ventricles associated with cognitive decline, indicating their potential in the Prodromal stage. These findings underscore the potential of these regions as early-stage PD biomarkers and showcase the proposed framework's efficacy in predicting subtypes of PD and aiding in early diagnosis, paving the way for innovative diagnostic tools and precision medicine.

MixTrain: accelerating DNN training via input mixing.

Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. An important factor contributing to the long training times is the increasing dataset complexity required to reach state-of-the-art performance in real-world applications. To address this challenge, we explore the use of input mixing, where multiple inputs are combined into a single composite input with an associated composite label for training. The goal is for training on the mixed input to achieve a similar effect as training separately on each the constituent inputs that it represents. This results in a lower number of inputs (or mini-batches) to be processed in each epoch, proportionally reducing training time. We find that naive input mixing leads to a considerable drop in learning performance and model accuracy due to interference between the forward/backward propagation of the mixed inputs. We propose two strategies to address this challenge and realize training speedups from input mixing with minimal impact on accuracy. First, we reduce the impact of inter-input interference by exploiting the spatial separation between the features of the constituent inputs in the network's intermediate representations. We also adaptively vary the mixing ratio of constituent inputs based on their loss in previous epochs. Second, we propose heuristics to automatically identify the subset of the training dataset that is subject to mixing in each epoch. Across ResNets of varying depth, MobileNetV2 and two Vision Transformer networks, we obtain upto 1.6 × and 1.8 × speedups in training for the ImageNet and Cifar10 datasets, respectively, on an Nvidia RTX 2080Ti GPU, with negligible loss in classification accuracy.

Improved region proposal network for enhanced few-shot object detection

Despite significant success of deep learning in object detection tasks, the standard training of deep neural networks requires access to a substantial quantity of annotated images across all classes. Data annotation is an arduous and time-consuming endeavor, particularly when dealing with infrequent objects. Few-shot object detection (FSOD) methods have emerged as a solution to the limitations of classic object detection approaches based on deep learning. FSOD methods demonstrate remarkable performance by achieving robust object detection using a significantly smaller amount of training data. A challenge for FSOD is that instances from novel classes that do not belong to the fixed set of training classes appear in the background and the base model may pick them up as potential objects. These objects behave similarly to label noise because they are classified as one of the training dataset classes, leading to FSOD performance degradation. We develop a semi-supervised algorithm to detect and then utilize these unlabeled novel objects as positive samples during the FSOD training stage to improve FSOD performance. Specifically, we develop a hierarchical ternary classification region proposal network (HTRPN) to localize the potential unlabeled novel objects and assign them new objectness labels to distinguish these objects from the base training dataset classes. Our improved hierarchical sampling strategy for the region proposal network (RPN) also boosts the perception ability of the object detection model for large objects. We test our approach and COCO and PASCAL VOC baselines that are commonly used in FSOD literature. Our experimental results indicate that our method is effective and outperforms the existing state-of-the-art (SOTA) FSOD methods. Our implementation is provided as a supplement to support reproducibility of the results https://github.com/zshanggu/HTRPN.11Early partial results of this work is presented in the 2023 ICCV Workshop on Visual Continual Learning (Shangguan and Rostami, 2023).

Synthetic ground motions in heterogeneous geologies from various sources: the HEMEWS-3D database

Abstract. The ever-improving performances of physics-based simulations and the rapid developments of deep learning are offering new perspectives to study earthquake-induced ground motion. Due to the large amount of data required to train deep neural networks, applications have so far been limited to recorded data or two-dimensional (2D) simulations. To bridge the gap between deep learning and high-fidelity numerical simulations, this work introduces a new database of physics-based earthquake simulations. The HEterogeneous Materials and Elastic Waves with Source variability in 3D (HEMEWS-3D) database comprises 30 000 simulations of elastic wave propagation in 3D geological domains. Each domain is parametrized by a different geological model built from a random arrangement of layers augmented by random fields that represent heterogeneities. Elastic waves originate from a randomly located pointwise source parametrized by a random moment tensor. For each simulation, ground motion is synthesized at the surface by a grid of virtual sensors. The high frequency of waveforms (fmax⁡=5 Hz) allows for extensive analyses of surface ground motion. Existing and foreseen applications range from statistical analyses of the ground motion variability and machine learning methods on geological models to deep-learning-based predictions of ground motion that depend on 3D heterogeneous geologies and source properties. Data are available at https://doi.org/10.57745/LAI6YU (Lehmann, 2023).

Feature balanced re-enhanced network with multi-factor margin loss for long-tailed visual recognition

Real-world data often exhibits a long-tailed distribution, where the number of training samples for head classes far exceeds that of tail classes. This class imbalance phenomenon poses significant challenges for training deep neural networks. Existing class-aware loss methods typically focus only on the numerical relationship between class samples, blindly favoring the optimization of tail classes during the process, while neglecting the difficulty of samples and the similarity between the current class and other classes. To this end, relying only on the number relationship can easily lead to over-fitting of tail classes, thereby failing to fully utilize the potential information in the data. Therefore, we propose the Multi-Factor Margin Loss (MFMLoss), which consists of positive margin loss and negative margin loss. MFMLoss comprehensively considers three factors at three levels: overall, class, and sample: (1) quantitative relationships, (2) inter-class similarity relationships, and (3) sample recognition difficulty. The combined consideration of these three factors enables the model to pay more attention to confusing classes and difficult samples during the training process, rather than solely on tail classes, thus achieving optimization from coarse-grained to fine-grained. To further mitigate the negative impact of the imbalance between head and tail classes on feature learning, we design a new network architecture, called F-BREN. F-BREN consists of two components: the feature balancing network and the feature re-enhancement network. The former is trained with negative margin loss, which reduces the recognizability of easy samples. The latter is trained with positive margin loss, using positive margin to give more attention to hard samples, thus balancing the model’s attention to all samples. We conducted extensive experiments on four long-tailed benchmark datasets: CIFAR10-LT, CIFAR100-LT, ImageNet-LT and iNaturalist 2018, comparing the recognition accuracy of our method with eight state-of-the-art methods. The experimental results demonstrate that our proposed method outperforms the eight compared methods.

Recognizing Digital Ink Chinese Characters Written by International Students Using a Residual Network with 1-Dimensional Dilated Convolution

Due to the complex nature of Chinese characters, junior international students often encounter writing problems related to strokes, components, and their combinations when writing Chinese characters. Digital ink Chinese characters (DICCs) are obtained by sampling the writing trajectory of Chinese characters with a pen input device. DICCs contain rich information, such as the time and space of strokes and sampling points. Recognizing DICCs is crucial for evaluating and correcting writing errors and enhancing the quality of Chinese character teaching for international students. Here, the paper first employs a one-dimensional dilated convolution to digital ink Chinese character recognition (DICCR) and proposes a novel residual network with one-dimensional dilated convolution (1-D ResNetDC). The 1-D ResNetDC not only utilizes multi-scale convolution kernels, but also employs different dilation rates on a single-scale convolution kernel to obtain information from various ranges. Additionally, residual connections facilitate the training of deep one-dimensional convolutional neural networks. Moreover, the paper proposes a more expressive ten-dimensional feature representation that includes spatial, temporal, and writing direction information for each sampling point, thereby improving classification accuracy. Because the DICC dataset of international students is small and unbalanced, the 1-D ResNetDC is pre-trained on the published available dataset. The experiments demonstrate that our approach is effective and superior. This model features a compact architecture, a reduced number of parameters, and excellent scalability.

Improving Deep Neural Networks' Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum.

Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

Hierarchical Weight Averaging for Deep Neural Networks.

Despite simplicity, stochastic gradient descent (SGD)-like algorithms are successful in training deep neural networks (DNNs). Among various attempts to improve SGD, weight averaging (WA), which averages the weights of multiple models, has recently received much attention in the literature. Broadly, WA falls into two categories: 1) online WA, which averages the weights of multiple models trained in parallel, is designed for reducing the gradient communication overhead of parallel mini-batch SGD and 2) offline WA, which averages the weights of one model at different checkpoints, is typically used to improve the generalization ability of DNNs. Though online and offline WA are similar in form, they are seldom associated with each other. Besides, these methods typically perform either offline parameter averaging or online parameter averaging, but not both. In this work, we first attempt to incorporate online and offline WA into a general training framework termed hierarchical WA (HWA). By leveraging both the online and offline averaging manners, HWA is able to achieve both faster convergence speed and superior generalization performance without any fancy learning rate adjustment. Besides, we also analyze the issues faced by the existing WA methods, and how our HWA addresses them, empirically. Finally, extensive experiments verify that HWA outperforms the state-of-the-art methods significantly.

Sigmoidal learning rate optimizer for deep neural network training using a two-phase adaptation approach

The optimization of machine learning models is an open research question since an optimal procedure to change the learning rate throughout training is still unknown. Manually defining a learning rate schedule involves troublesome, time-consuming try-and-error procedures to determine hyperparameters, such as learning rate decay epochs and learning rate decay rates, in order to navigate the intricate landscape of loss functions. Although adaptive learning rate optimizers automatize this process, recent studies suggest they may produce overfitting and reduce performance compared to fine-tuned learning rate schedules. Considering that the new machine learning loss function approaches present landscapes with much more saddle points than local minima, we proposed the Training Aware Sigmoidal Optimizer (TASO). This automated two-phase learning rate adaptation mechanism significantly reduces the need for manual hyperparameter tuning. The first phase uses a high learning rate to quickly traverse the numerous saddle points in the error surface, while the second phase uses a low learning rate to gradually approach the center of the local minimum previously found. We compared the proposed approach with commonly used adaptive learning rate schedules such as Adam, RMSProp, and Adagrad. The validation experiments were performed in image and text datasets and showed that TASO outperformed all competing methods in optimal (i.e., performing hyperparameter validation) and suboptimal (i.e., using default hyperparameters) scenarios. In our benchmark tests, TASO demonstrated promising performance, achieving an 8.32% increase in accuracy and a significant 46.62% decrease in training loss on average across various datasets and models. This remarkable performance positions TASO ahead of well-established adaptive optimizers, suggesting higher effectiveness and consistency in performance.

Driven traffic flow prediction in smart cities using hunter-prey optimization with hybrid deep learning models

Accurate and timely flow prediction is the most significant element for intelligent traffic management systems. However, developing a robust and potential prediction method is a challenge because of the nonlinear characteristics and inherent randomness of the traffic flow in smart cities. Deep learning can analyze historical traffic data and predict future traffic patterns in traffic flow prediction. This can be done by training deep neural networks on large datasets, such as traffic speed and volume data, to learn the underlying relationships between various factors influencing traffic flow. The resulting models can then be used to predict future traffic conditions, helping optimize traffic management, reduce congestion, and improve safety. This study introduces a Hunter Prey Optimization with Hybrid Deep Learning-Driven Traffic Flow in smart cities Prediction (HPOHDL-TFPM). The HPOHDL-TFPM approach's primary goal is to accurately and rapidly forecast traffic flow. The HPOHDL-TFPM technique uses Z-score normalization to normalize the traffic data to achieve this. In addition, the CBLSTM-AE model, which combines convolutional bidirectional long short-term memory and autoencoder, is utilized in the prediction of traffic flow in smart cities. Moreover, the HPO technique is applied as a hyperparameter optimizer to select the hyperparameter values properly. The experimental validation of the HPOHDL-TFPM approach is tested in several contexts. Numerous comparative studies demonstrated the improved performance of the HPOHDL-TFPM approach over other existing methods.

Permeability prediction using logging data from tight reservoirs based on deep neural networks

Permeability is a critical parameter for evaluating reservoir properties, and accurate prediction is an important basis for identifying high-quality reservoirs and geological modeling. However, the strong heterogeneity, complex lithology and diagenesis in the reservoirs of this region pose a major challenge for the accurate assessment of reservoir permeability. In recent years, the use of machine learning (ML) to solve problems in geophysical well logging and related fields has gained much attention thanks to advances in data science and artificial intelligence. ML is any predictive algorithm or combination of algorithms that learns from data and makes predictions without being explicitly coded with a deterministic model. The most immediate example is deep neural networks (DNN) that are trained with data to minimize a cost function and make predictions. The tight reservoirs in the Chang 7 Member of the Ordos Basin host significant oil and gas resources and have recently emerged as the main focus of unconventional oil and gas exploration and development. In this work, we performed DNN-based permeability prediction for the tight reservoirs in the Ordos Basin area. From 19 well logs, we selected effective data points from 17 wells for DNN training after preprocessing and used the remaining two wells for testing. First, we trained the DNN with all collected parameters as inputs, resulting in permeability prediction R2 values of 0.64 and 0.72 for the two wells, indicating a good fit. We then optimized the input parameters by performing a crossplot analysis between these parameters and the permeability. Using the same network structure (with all hyperparameters set the same), we trained the DNN again to obtain a new DNN-based model. The prediction results showed that removing input parameters with poor correlation to permeability improved the prediction accuracy with R2 values of 0.70 and 0.87 for the two wells.

One-Shot Learning from Prototype Stock Keeping Unit Images

This paper highlights the importance of one-shot learning from prototype Stock Keeping Unit (SKU) images for efficient product recognition in retail and inventory management. Traditional methods require large supervised datasets to train deep neural networks, which can be costly and impractical. One-shot learning techniques mitigate this issue by enabling classification from a single prototype image per product class, thus reducing data annotation efforts. We introduce the Variational Prototyping Encoder (VPE), a novel deep neural network for one-shot classification. Utilizing a support set of prototype SKU images, VPE learns to classify query images by capturing image similarity and prototypical concepts. Unlike metric learning-based approaches, VPE pre-learns image translation from real-world object images to prototype images as a meta-task, facilitating efficient one-shot classification with minimal supervision. Our research demonstrates that VPE effectively reduces the need for extensive datasets by utilizing a single image per class while accurately classifying query images into their respective categories, thus providing a practical solution for product classification tasks.

Hybrid two-level protection system for preserving pre-trained DNN models ownership

AbstractRecent advancements in deep neural networks (DNNs) have made them indispensable for numerous commercial applications. These include healthcare systems and self-driving cars. Training DNN models typically demands substantial time, vast datasets and high computational costs. However, these valuable models face significant risks. Attackers can steal and sell pre-trained DNN models for profit. Unauthorised sharing of these models poses a serious threat. Once sold, they can be easily copied and redistributed. Therefore, a well-built pre-trained DNN model is a valuable asset that requires protection. This paper introduces a robust hybrid two-level protection system for safeguarding the ownership of pre-trained DNN models. The first-level employs zero-bit watermarking. The second-level incorporates an adversarial attack as a watermark by using a perturbation technique to embed the watermark. The robustness of the proposed system is evaluated against seven types of attacks. These are Fast Gradient Method Attack, Auto Projected Gradient Descent Attack, Auto Conjugate Gradient Attack, Basic Iterative Method Attack, Momentum Iterative Method Attack, Square Attack and Auto Attack. The proposed two-level protection system withstands all seven attack types. It maintains accuracy and surpasses current state-of-the-art methods.

XECG-Beats: an explainable deep transfer learning approach for ECG-based heartbeat classification

Early detection of abnormal heartbeats is of great importance for cardiologists for early diagnosis of cardiac diseases. This will help patients to receive in time diagnosis and prevention. Conventionally, physicians provide cardiac diagnoses by visual examination of electrocardiograms (ECGs). However, this can be a very time consuming and demanding task and, in some cases, may lead to overlooking and wrong diagnosis of life-threatening heart diseases. Therefore, an intelligent model can help to automatically analyze these huge amount of ECGs captured by different devices in clinical practice. A deep transfer learning approach is used to utilize the capability of different trained deep neural networks and to test them on new unseen datasets without the need to fully re-train the model. Two deep neural networks, namely, Visual Geometry Group (VGG) and Residual Network (ResNet) are utilized for classification of ECGs heartbeats. The models are evaluated using two unseen ECG datasets (i.e., SVDB and INCARTDB) by only optimizing their last classification layers. The overall area under curve for receiver operating characteristic (AUCROC) of two VGG and ResNet models are 0.961 and 0.966 on the SVDB dataset, respectively, and both models achieve 0.981 on the INCARTDB. This paper proposes an accurate and explainable model to classify ECG heartbeats into five categories recommended by the ANSI/AAMI standard. The proposed method paves the way to use pre-trained deep neural networks in real-time monitoring of heart patients using ECG data and to help clinicians understand the decision made by the models on each case using an explainable approach.