Training Deep Neural Networks Research Articles

Deep-neural-network model for predicting ground motion parameters using earthquake horizontal-to-vertical spectral ratios

This study proposed a deep-neural-network (DNN) model for seismic ground motion prediction by utilizing a unified strong motion database by the National Research Institute for Earth Science and Disaster Resilience, and earthquake horizontal-to-vertical spectral ratio (EHVR) database in Japan. The model aims to enhance the accuracy of predictions by incorporating the EHVRs for complementing site effects, and utilizing existing ground motion prediction equations (GMPE) as the base model for source and propagation path effects. The hybrid approach enables the prediction of peak ground accelerations (PGAs), peak ground velocities (PGVs), and 5% damped absolute acceleration response spectra (SAs). After classifying the training and test sets from the database, the trained DNN models were applied on the test set to evaluate the performance of the predicted results. The accuracy assessment by the residuals, R-squared ( R2), and root mean square error (RMSE) between the predicted and observed values in the test set revealed the superior performance of the proposed model compared with the traditional GMPE with proxy-based site effects such as V S30s especially in predicting both the spectral amplitude and shape of SAs.

Tuning parameters of deep neural network training algorithms pays off: a computational study

AbstractThe paper aims to investigate the impact of the optimization algorithms on the training of deep neural networks with an eye to the interaction between the optimizer and the generalization performance. In particular, we aim to analyze the behavior of state-of-the-art optimization algorithms in relationship to their hyperparameters setting to detect robustness with respect to the choice of a certain starting point in ending on different local solutions. We conduct extensive computational experiments using nine open-source optimization algorithms to train deep Convolutional Neural Network architectures on an image multi-class classification task. Precisely, we consider several architectures by changing the number of layers and neurons per layer, to evaluate the impact of different width and depth structures on the computational optimization performance. We show that the optimizers often return different local solutions and highlight the strong correlation between the quality of the solution found and the generalization capability of the trained network. We also discuss the role of hyperparameters tuning and show how a tuned hyperparameters setting can be re-used for the same task on different problems achieving better efficiency and generalization performance than a default setting.

Nondeterministic Features in Deep neural network design, training and inference

Neural networks are increasingly integral to scientific modeling and a wide range of real-world applications. However, standard neural networks often lack reliable certainty and confidence, and can be poorly calibrated, leading to nondeterministic results. To address these challenges, researchers have focused on understanding and quantifying uncertainty in neural network predictions. This work offers a comprehensive discussion of uncertainty estimation in neural networks, discussing recent advances, current challenges, and future research opportunities. This work explains key sources of uncertainty, challenges from model uncertainty and irreducible data uncertainty. We explore various methods for modeling these uncertainties, such as deterministic neural networks, Bayesian neural networks (BNNs), ensembles, and test-time data augmentation approaches. Additionally, the paper provides practical examples from fields like medical data analysis, robotics, and autonomous driving, illustrating the challenges and requirements associated with uncertainty in real-world applications. We also examine the limitations of current uncertainty quantification methods in safety-critical applications, and provide an outlook on future developments aimed at broader adoption of these methods in diverse domains. This paper serves as a valuable resource for both newcomers and those experienced in the field of uncertainty estimation in neural networks.

Enhancing early Parkinson’s disease detection through multimodal deep learning and explainable AI: insights from the PPMI database

Parkinson’s is the second most common neurodegenerative disease, affecting nearly 8.5M people and steadily increasing. In this research, Multimodal Deep Learning is investigated for the Prodromal stage detection of Parkinson's Disease (PD), combining different 3D architectures with the novel Excitation Network (EN) and supported by Explainable Artificial Intelligence (XAI) techniques. Utilizing data from the Parkinson’s Progression Markers Initiative, this study introduces a joint co-learning approach for multimodal fusion, enabling end-to-end training of deep neural networks and facilitating the learning of complementary information from both imaging and clinical modalities. DenseNet with EN outperformed other models, showing a substantial increase in accuracy when supplemented with clinical data. XAI methods, such as Integrated Gradients for ResNet and DenseNet, and Attention Heatmaps for Vision Transformer (ViT), revealed that DenseNet focused on brain regions believed to be critical to prodromal pathophysiology, including the right temporal and left pre-frontal areas. Similarly, ViT highlighted the lateral ventricles associated with cognitive decline, indicating their potential in the Prodromal stage. These findings underscore the potential of these regions as early-stage PD biomarkers and showcase the proposed framework's efficacy in predicting subtypes of PD and aiding in early diagnosis, paving the way for innovative diagnostic tools and precision medicine.

Improved region proposal network for enhanced few-shot object detection

Despite significant success of deep learning in object detection tasks, the standard training of deep neural networks requires access to a substantial quantity of annotated images across all classes. Data annotation is an arduous and time-consuming endeavor, particularly when dealing with infrequent objects. Few-shot object detection (FSOD) methods have emerged as a solution to the limitations of classic object detection approaches based on deep learning. FSOD methods demonstrate remarkable performance by achieving robust object detection using a significantly smaller amount of training data. A challenge for FSOD is that instances from novel classes that do not belong to the fixed set of training classes appear in the background and the base model may pick them up as potential objects. These objects behave similarly to label noise because they are classified as one of the training dataset classes, leading to FSOD performance degradation. We develop a semi-supervised algorithm to detect and then utilize these unlabeled novel objects as positive samples during the FSOD training stage to improve FSOD performance. Specifically, we develop a hierarchical ternary classification region proposal network (HTRPN) to localize the potential unlabeled novel objects and assign them new objectness labels to distinguish these objects from the base training dataset classes. Our improved hierarchical sampling strategy for the region proposal network (RPN) also boosts the perception ability of the object detection model for large objects. We test our approach and COCO and PASCAL VOC baselines that are commonly used in FSOD literature. Our experimental results indicate that our method is effective and outperforms the existing state-of-the-art (SOTA) FSOD methods. Our implementation is provided as a supplement to support reproducibility of the results https://github.com/zshanggu/HTRPN.11Early partial results of this work is presented in the 2023 ICCV Workshop on Visual Continual Learning (Shangguan and Rostami, 2023).

Synthetic ground motions in heterogeneous geologies from various sources: the HEMEWS-3D database

Abstract. The ever-improving performances of physics-based simulations and the rapid developments of deep learning are offering new perspectives to study earthquake-induced ground motion. Due to the large amount of data required to train deep neural networks, applications have so far been limited to recorded data or two-dimensional (2D) simulations. To bridge the gap between deep learning and high-fidelity numerical simulations, this work introduces a new database of physics-based earthquake simulations. The HEterogeneous Materials and Elastic Waves with Source variability in 3D (HEMEWS-3D) database comprises 30 000 simulations of elastic wave propagation in 3D geological domains. Each domain is parametrized by a different geological model built from a random arrangement of layers augmented by random fields that represent heterogeneities. Elastic waves originate from a randomly located pointwise source parametrized by a random moment tensor. For each simulation, ground motion is synthesized at the surface by a grid of virtual sensors. The high frequency of waveforms (fmax⁡=5 Hz) allows for extensive analyses of surface ground motion. Existing and foreseen applications range from statistical analyses of the ground motion variability and machine learning methods on geological models to deep-learning-based predictions of ground motion that depend on 3D heterogeneous geologies and source properties. Data are available at https://doi.org/10.57745/LAI6YU (Lehmann, 2023).

Feature balanced re-enhanced network with multi-factor margin loss for long-tailed visual recognition

Real-world data often exhibits a long-tailed distribution, where the number of training samples for head classes far exceeds that of tail classes. This class imbalance phenomenon poses significant challenges for training deep neural networks. Existing class-aware loss methods typically focus only on the numerical relationship between class samples, blindly favoring the optimization of tail classes during the process, while neglecting the difficulty of samples and the similarity between the current class and other classes. To this end, relying only on the number relationship can easily lead to over-fitting of tail classes, thereby failing to fully utilize the potential information in the data. Therefore, we propose the Multi-Factor Margin Loss (MFMLoss), which consists of positive margin loss and negative margin loss. MFMLoss comprehensively considers three factors at three levels: overall, class, and sample: (1) quantitative relationships, (2) inter-class similarity relationships, and (3) sample recognition difficulty. The combined consideration of these three factors enables the model to pay more attention to confusing classes and difficult samples during the training process, rather than solely on tail classes, thus achieving optimization from coarse-grained to fine-grained. To further mitigate the negative impact of the imbalance between head and tail classes on feature learning, we design a new network architecture, called F-BREN. F-BREN consists of two components: the feature balancing network and the feature re-enhancement network. The former is trained with negative margin loss, which reduces the recognizability of easy samples. The latter is trained with positive margin loss, using positive margin to give more attention to hard samples, thus balancing the model’s attention to all samples. We conducted extensive experiments on four long-tailed benchmark datasets: CIFAR10-LT, CIFAR100-LT, ImageNet-LT and iNaturalist 2018, comparing the recognition accuracy of our method with eight state-of-the-art methods. The experimental results demonstrate that our proposed method outperforms the eight compared methods.

Recognizing Digital Ink Chinese Characters Written by International Students Using a Residual Network with 1-Dimensional Dilated Convolution

Due to the complex nature of Chinese characters, junior international students often encounter writing problems related to strokes, components, and their combinations when writing Chinese characters. Digital ink Chinese characters (DICCs) are obtained by sampling the writing trajectory of Chinese characters with a pen input device. DICCs contain rich information, such as the time and space of strokes and sampling points. Recognizing DICCs is crucial for evaluating and correcting writing errors and enhancing the quality of Chinese character teaching for international students. Here, the paper first employs a one-dimensional dilated convolution to digital ink Chinese character recognition (DICCR) and proposes a novel residual network with one-dimensional dilated convolution (1-D ResNetDC). The 1-D ResNetDC not only utilizes multi-scale convolution kernels, but also employs different dilation rates on a single-scale convolution kernel to obtain information from various ranges. Additionally, residual connections facilitate the training of deep one-dimensional convolutional neural networks. Moreover, the paper proposes a more expressive ten-dimensional feature representation that includes spatial, temporal, and writing direction information for each sampling point, thereby improving classification accuracy. Because the DICC dataset of international students is small and unbalanced, the 1-D ResNetDC is pre-trained on the published available dataset. The experiments demonstrate that our approach is effective and superior. This model features a compact architecture, a reduced number of parameters, and excellent scalability.

Improving Deep Neural Networks' Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum.

Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

Hierarchical Weight Averaging for Deep Neural Networks.

Despite simplicity, stochastic gradient descent (SGD)-like algorithms are successful in training deep neural networks (DNNs). Among various attempts to improve SGD, weight averaging (WA), which averages the weights of multiple models, has recently received much attention in the literature. Broadly, WA falls into two categories: 1) online WA, which averages the weights of multiple models trained in parallel, is designed for reducing the gradient communication overhead of parallel mini-batch SGD and 2) offline WA, which averages the weights of one model at different checkpoints, is typically used to improve the generalization ability of DNNs. Though online and offline WA are similar in form, they are seldom associated with each other. Besides, these methods typically perform either offline parameter averaging or online parameter averaging, but not both. In this work, we first attempt to incorporate online and offline WA into a general training framework termed hierarchical WA (HWA). By leveraging both the online and offline averaging manners, HWA is able to achieve both faster convergence speed and superior generalization performance without any fancy learning rate adjustment. Besides, we also analyze the issues faced by the existing WA methods, and how our HWA addresses them, empirically. Finally, extensive experiments verify that HWA outperforms the state-of-the-art methods significantly.

Sigmoidal learning rate optimizer for deep neural network training using a two-phase adaptation approach

Sigmoidal learning rate optimizer for deep neural network training using a two-phase adaptation approach

Driven traffic flow prediction in smart cities using hunter-prey optimization with hybrid deep learning models

Accurate and timely flow prediction is the most significant element for intelligent traffic management systems. However, developing a robust and potential prediction method is a challenge because of the nonlinear characteristics and inherent randomness of the traffic flow in smart cities. Deep learning can analyze historical traffic data and predict future traffic patterns in traffic flow prediction. This can be done by training deep neural networks on large datasets, such as traffic speed and volume data, to learn the underlying relationships between various factors influencing traffic flow. The resulting models can then be used to predict future traffic conditions, helping optimize traffic management, reduce congestion, and improve safety. This study introduces a Hunter Prey Optimization with Hybrid Deep Learning-Driven Traffic Flow in smart cities Prediction (HPOHDL-TFPM). The HPOHDL-TFPM approach's primary goal is to accurately and rapidly forecast traffic flow. The HPOHDL-TFPM technique uses Z-score normalization to normalize the traffic data to achieve this. In addition, the CBLSTM-AE model, which combines convolutional bidirectional long short-term memory and autoencoder, is utilized in the prediction of traffic flow in smart cities. Moreover, the HPO technique is applied as a hyperparameter optimizer to select the hyperparameter values properly. The experimental validation of the HPOHDL-TFPM approach is tested in several contexts. Numerous comparative studies demonstrated the improved performance of the HPOHDL-TFPM approach over other existing methods.

Permeability prediction using logging data from tight reservoirs based on deep neural networks

Permeability is a critical parameter for evaluating reservoir properties, and accurate prediction is an important basis for identifying high-quality reservoirs and geological modeling. However, the strong heterogeneity, complex lithology and diagenesis in the reservoirs of this region pose a major challenge for the accurate assessment of reservoir permeability. In recent years, the use of machine learning (ML) to solve problems in geophysical well logging and related fields has gained much attention thanks to advances in data science and artificial intelligence. ML is any predictive algorithm or combination of algorithms that learns from data and makes predictions without being explicitly coded with a deterministic model. The most immediate example is deep neural networks (DNN) that are trained with data to minimize a cost function and make predictions. The tight reservoirs in the Chang 7 Member of the Ordos Basin host significant oil and gas resources and have recently emerged as the main focus of unconventional oil and gas exploration and development. In this work, we performed DNN-based permeability prediction for the tight reservoirs in the Ordos Basin area. From 19 well logs, we selected effective data points from 17 wells for DNN training after preprocessing and used the remaining two wells for testing. First, we trained the DNN with all collected parameters as inputs, resulting in permeability prediction R2 values of 0.64 and 0.72 for the two wells, indicating a good fit. We then optimized the input parameters by performing a crossplot analysis between these parameters and the permeability. Using the same network structure (with all hyperparameters set the same), we trained the DNN again to obtain a new DNN-based model. The prediction results showed that removing input parameters with poor correlation to permeability improved the prediction accuracy with R2 values of 0.70 and 0.87 for the two wells.

One-Shot Learning from Prototype Stock Keeping Unit Images

This paper highlights the importance of one-shot learning from prototype Stock Keeping Unit (SKU) images for efficient product recognition in retail and inventory management. Traditional methods require large supervised datasets to train deep neural networks, which can be costly and impractical. One-shot learning techniques mitigate this issue by enabling classification from a single prototype image per product class, thus reducing data annotation efforts. We introduce the Variational Prototyping Encoder (VPE), a novel deep neural network for one-shot classification. Utilizing a support set of prototype SKU images, VPE learns to classify query images by capturing image similarity and prototypical concepts. Unlike metric learning-based approaches, VPE pre-learns image translation from real-world object images to prototype images as a meta-task, facilitating efficient one-shot classification with minimal supervision. Our research demonstrates that VPE effectively reduces the need for extensive datasets by utilizing a single image per class while accurately classifying query images into their respective categories, thus providing a practical solution for product classification tasks.

Hybrid two-level protection system for preserving pre-trained DNN models ownership

AbstractRecent advancements in deep neural networks (DNNs) have made them indispensable for numerous commercial applications. These include healthcare systems and self-driving cars. Training DNN models typically demands substantial time, vast datasets and high computational costs. However, these valuable models face significant risks. Attackers can steal and sell pre-trained DNN models for profit. Unauthorised sharing of these models poses a serious threat. Once sold, they can be easily copied and redistributed. Therefore, a well-built pre-trained DNN model is a valuable asset that requires protection. This paper introduces a robust hybrid two-level protection system for safeguarding the ownership of pre-trained DNN models. The first-level employs zero-bit watermarking. The second-level incorporates an adversarial attack as a watermark by using a perturbation technique to embed the watermark. The robustness of the proposed system is evaluated against seven types of attacks. These are Fast Gradient Method Attack, Auto Projected Gradient Descent Attack, Auto Conjugate Gradient Attack, Basic Iterative Method Attack, Momentum Iterative Method Attack, Square Attack and Auto Attack. The proposed two-level protection system withstands all seven attack types. It maintains accuracy and surpasses current state-of-the-art methods.

XECG-Beats: an explainable deep transfer learning approach for ECG-based heartbeat classification

Early detection of abnormal heartbeats is of great importance for cardiologists for early diagnosis of cardiac diseases. This will help patients to receive in time diagnosis and prevention. Conventionally, physicians provide cardiac diagnoses by visual examination of electrocardiograms (ECGs). However, this can be a very time consuming and demanding task and, in some cases, may lead to overlooking and wrong diagnosis of life-threatening heart diseases. Therefore, an intelligent model can help to automatically analyze these huge amount of ECGs captured by different devices in clinical practice. A deep transfer learning approach is used to utilize the capability of different trained deep neural networks and to test them on new unseen datasets without the need to fully re-train the model. Two deep neural networks, namely, Visual Geometry Group (VGG) and Residual Network (ResNet) are utilized for classification of ECGs heartbeats. The models are evaluated using two unseen ECG datasets (i.e., SVDB and INCARTDB) by only optimizing their last classification layers. The overall area under curve for receiver operating characteristic (AUCROC) of two VGG and ResNet models are 0.961 and 0.966 on the SVDB dataset, respectively, and both models achieve 0.981 on the INCARTDB. This paper proposes an accurate and explainable model to classify ECG heartbeats into five categories recommended by the ANSI/AAMI standard. The proposed method paves the way to use pre-trained deep neural networks in real-time monitoring of heart patients using ECG data and to help clinicians understand the decision made by the models on each case using an explainable approach.

Improved Hybrid Bagging Resampling Framework for Deep Learning-Based Side-Channel Analysis

As cryptographic implementations leak secret information through side-channel emissions, the Hamming weight (HW) leakage model is widely used in deep learning profiling side-channel analysis (SCA) attacks to expose the leaked model. However, imbalanced datasets often arise from the HW leakage model, increasing the attack complexity and limiting the performance of deep learning-based SCA attacks. Effective management of class imbalance is vital for training deep neural network models to achieve optimized and improved performance results. Recent works focus on either improved deep-learning methodologies or data augmentation techniques. In this work, we propose the hybrid bagging resampling framework, a two-pronged strategy for tackling class imbalance in side-channel datasets, consisting of data augmentation and ensemble learning. We show that adopting this framework can boost attack performance results in a practical setup. From our experimental results, the SMOTEENN ensemble achieved the best performance in the ASCAD dataset, and the basic ensemble performed the best in the CHES dataset, with both contributing over 70% practical improvements in performance compared to the original imbalanced dataset, and accelerating practical attack space in comparison to the classical setup of the attack.

Fast and robust analog in-memory deep neural network training

Analog in-memory computing is a promising future technology for efficiently accelerating deep learning networks. While using in-memory computing to accelerate the inference phase has been studied extensively, accelerating the training phase has received less attention, despite its arguably much larger compute demand to accelerate. While some analog in-memory training algorithms have been suggested, they either invoke significant amount of auxiliary digital compute—accumulating the gradient in digital floating point precision, limiting the potential speed-up—or suffer from the need for near perfectly programming reference conductance values to establish an algorithmic zero point. Here, we propose two improved algorithms for in-memory training, that retain the same fast runtime complexity while resolving the requirement of a precise zero point. We further investigate the limits of the algorithms in terms of conductance noise, symmetry, retention, and endurance which narrow down possible device material choices adequate for fast and robust in-memory deep neural network training.

Hebbian Descent: A Unified View on Log-Likelihood Learning.

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

A brief review of hypernetworks in deep learning

Hypernetworks, or hypernets for short, are neural networks that generate weights for another neural network, known as the target network. They have emerged as a powerful deep learning technique that allows for greater flexibility, adaptability, dynamism, faster training, information sharing, and model compression. Hypernets have shown promising results in a variety of deep learning problems, including continual learning, causal inference, transfer learning, weight pruning, uncertainty quantification, zero-shot learning, natural language processing, and reinforcement learning. Despite their success across different problem settings, there is currently no comprehensive review available to inform researchers about the latest developments and to assist in utilizing hypernets. To fill this gap, we review the progress in hypernets. We present an illustrative example of training deep neural networks using hypernets and propose categorizing hypernets based on five design criteria: inputs, outputs, variability of inputs and outputs, and the architecture of hypernets. We also review applications of hypernets across different deep learning problem settings, followed by a discussion of general scenarios where hypernets can be effectively employed. Finally, we discuss the challenges and future directions that remain underexplored in the field of hypernets. We believe that hypernetworks have the potential to revolutionize the field of deep learning. They offer a new way to design and train neural networks, and they have the potential to improve the performance of deep learning models on a variety of tasks. Through this review, we aim to inspire further advancements in deep learning through hypernetworks.