Training Deep Neural Networks Research Articles

Minimizing Communication Overhead in Decentralized Deep Neural Networks

Minimizing communication overhead in decentralized deep neural network (DNN) training has become a critical challenge, particularly with the increasing adoption of distributed systems for large-scale machine learning tasks. This paper introduces advanced techniques that leverage gradient compression, adaptive sparsification, and hybrid aggregation to optimize communication efficiency while maintaining model accuracy and convergence rates. Experimental results on benchmark datasets such as CIFAR-10 and ImageNet show that the proposed methods reduce communication costs by up to 70% compared to standard approaches while achieving comparable or superior model accuracy. Additionally, scalability tests on diverse neural network architectures highlight the robustness of the approach, demonstrating efficient performance across varying network sizes and computational setups. These findings underscore the potential of the proposed strategies to enable faster, cost-effective, and sustainable decentralized deep learning systems.

Synthetic Image Generation for Military Vehicle Detection

This research paper investigates the effectiveness of using computer graphics(CG) based synthetic data for deep learning in military vehicle detection. In particular, we explore the use of synthetic image generation techniques to train deep neural networks for object detection tasks. Our approach involves the generation of a large dataset of synthetic images of military vehicles, which is then used to train a deep learning model. The resulting model is then evaluated on real-world images to measure its effectiveness. Our experimental results show that synthetic training data alone can achieve effective results in object detection. Our findings demonstrate the potential of CG-based synthetic data for deep learning and suggest its value as a tool for training models in a variety of applications, including military vehicle detection.

Label flipping attacks in hierarchical federated learning for intrusion detection in IoT

ABSTRACT Federated learning (FL) is a promising approach for distributed training of deep neural networks within Internet of Things (IoT) environments, where the data generated by IoT devices stays local, and only model updates are communicated to a central server. This methodology is particularly relevant for intrusion detection systems in IoT networks, where security is paramount. However, the decentralized nature of FL introduces vulnerabilities, such as the risk of data poisoning by malicious participants. In this paper, we propose a hierarchical federated learning to reduce communication overhead and improve privacy by limiting data spread. We further explore the impact of label-flipping attacks on hierarchical FL systems used in IoT-based intrusion detection. We focus on scenarios where a subset of malicious participants attempts to degrade the global model’s performance by submitting corrupted model updates based on intentionally mislabeled data. Our findings reveal significant decreases in classification accuracy by 10.53% and recall rates, even with a minimal number of compromised participants, primarily affecting the specific classes targeted by the attackers. We further examine how the availability of these malicious nodes influences the attack’s success. To counteract these threats, we introduce a defense mechanism that successfully identifies all malicious clients and mitigates their impact. As a result, the global model’s accuracy is maintained at the original 95% level found during training without the presence of malicious clients, thereby enhancing the resilience of federated learning models in IoT security applications.

ENHANCING DEEP LEARNING PERFORMANCE THROUGH A GENETIC ALGORITHM-ENHANCED APPROACH: FOCUSING ON LSTM

Deep learning has shown remarkable success in various applications, such as image classification, natural language processing, and speech recognition. However, training deep neural networks is challenging due to their complex architecture and the number of parameters required. Genetic algorithms have been proposed as an alternative optimization technique for deep learning, offering an efficient alternative way to find an optimal set of network parameters that minimize the objective function. In this paper, we propose a novel approach integrating genetic algorithms with deep learning, specifically LSTM models, to enhance performance. Our method optimizes crucial hyper-parameters including learning rate, batch size, neuron count per layer, and layer depth through genetic algorithms. Additionally, we conduct a comprehensive analysis of how genetic algorithm parameters influence the optimization process and illustrate their significant impact on improving LSTM model performance. Overall, the presented method provides a powerful mechanism for improving the performance of deep neural networks, and; thus, we believe that it has significant potential for future applications in the artificial intelligence discipline.

On the turnpike to design of deep neural networks: Explicit depth bounds

It is well-known that the training of Deep Neural Networks (DNN) can be formalized in the language of optimal control. In this context, this paper leverages classical turnpike properties of optimal control problems to attempt a quantifiable answer to the question of how many layers should be considered in a DNN. The underlying assumption is that the number of neurons per layer—i.e., the width of the DNN—is kept constant. Pursuing a different route than the classical analysis of approximation properties of sigmoidal functions, we prove explicit bounds on the required depths of DNNs based on asymptotic reachability assumptions and a dissipativity-inducing choice of the regularization terms in the training problem. Numerical results obtained for the two spiral task data set for classification indicate that the proposed constructive estimates can provide non-conservative depth bounds.

Model Parameter Prediction Method for Accelerating Distributed DNN Training

As the size of deep neural network (DNN) models and datasets increases, distributed training becomes popular to reduce the training time. However, a severe communication bottleneck in distributed training limits its scalability. Many methods aim to solve this communication bottleneck by reducing communication traffic, such as gradient sparsification and quantification. However, these methods either are at the expense of losing model accuracy or introducing lots of computing overhead. We observe that the data distribution between layers of neural network models is similar. Thus, we propose a model parameter prediction method (MP2) to accelerate distributed DNN training under parameter server (PS) framework, where workers push only a subset of model parameters to PS, and residual model parameters are locally predicted by an already-trained deep neural network model on PS. Some key challenges are addressed. First, we build a hierarchical parameters dataset by randomly sampling a subset of model from normal distributed trainings. Second, we build a neural network model with the structure of “convolution + channel attention + Max pooling” for predicting model parameters by using a prediction result-based evaluation method. For VGGNet, ResNet, and AlexNet models on CIFAR10 and CIFAR100 datasets, compared with Baseline, Top-k, deep gradient compression (DGC), and weight nowcaster network (WNN), MP2 can reduce traffic by up to 88.98%; and accelerates the training by up to 47.32% while not losing the model accuracy. MP2 has shown good generalization.

Margin-enhanced average precision optimization for visible-infrared person re-identification

Average precision-based loss functions are effective for training deep neural networks in ranking tasks and single-modality person re-identification. However, they have difficulty in uniformly ranking cross-modality samples due to high intra-modality similarity. To address this limitation, this paper proposes a Margin-Enhanced Average Precision (MEAP) optimization approach for cross-modality person re-identification. MEAP integrates average precision optimization with inter-class and cross-modality margin parameters. The inter-class margin improves performance by increasing class separation, while the cross-modality margin enhances performance by prioritizing cross-modality positive samples. Additionally, we propose an innovative algorithm to augment visible and infrared person images using horizontal stripes, aiming to bridge the gap between the two modalities by creating diverse and enriched training data. Experiments on two public datasets demonstrate the effectiveness of our approach in comparison to state-of-the-art methods. MEAP and horizontal stripe augmentation significantly improve accuracy and robustness in matching individuals across different modalities. The code is available at: https://github.com/NihatTekeli/meapnet

Sigmoidal learning rate optimizer for deep neural network training using a two-phase adaptation approach

The optimization of machine learning models is an open research question since an optimal procedure to change the learning rate throughout training is still unknown. Manually defining a learning rate schedule involves troublesome, time-consuming try-and-error procedures to determine hyperparameters, such as learning rate decay epochs and learning rate decay rates, in order to navigate the intricate landscape of loss functions. Although adaptive learning rate optimizers automatize this process, recent studies suggest they may produce overfitting and reduce performance compared to fine-tuned learning rate schedules. Considering that the new machine learning loss function approaches present landscapes with much more saddle points than local minima, we proposed the Training Aware Sigmoidal Optimizer (TASO). This automated two-phase learning rate adaptation mechanism significantly reduces the need for manual hyperparameter tuning. The first phase uses a high learning rate to quickly traverse the numerous saddle points in the error surface, while the second phase uses a low learning rate to gradually approach the center of the local minimum previously found. We compared the proposed approach with commonly used adaptive learning rate schedules such as Adam, RMSProp, and Adagrad. The validation experiments were performed in image and text datasets and showed that TASO outperformed all competing methods in optimal (i.e., performing hyperparameter validation) and suboptimal (i.e., using default hyperparameters) scenarios. In our benchmark tests, TASO demonstrated promising performance, achieving an 8.32% increase in accuracy and a significant 46.62% decrease in training loss on average across various datasets and models. This remarkable performance positions TASO ahead of well-established adaptive optimizers, suggesting higher effectiveness and consistency in performance.

Automated Shoreline Segmentation in Satellite Imagery Using USV Measurements

Generating aerial shoreline segmentation masks can be a daunting task, often requiring manual labeling or correction. This is further problematic because neural segmentation models require decent and abundant data for training, requiring even more manpower to automate the process. In this paper, we propose utilizing Unmanned Surface Vehicles (USVs) in an automated shoreline segmentation system on satellite imagery. The remotely controlled vessel first collects above- and underwater shoreline information using light detection and ranging (LiDAR) and multibeam echosounder (MBES) measuring instruments, resulting in a geo-referenced 3D point cloud. After cleaning and processing these data, the system integrates the projected map with an aerial image of the region. Based on the height values of the mapped points, the image is segmented. Finally, post-processing methods and the k-NN algorithm are introduced, resulting in a complete binary shoreline segmentation mask. The obtained data were used for training U-Net-type segmentation models with pre-trained backbones. The InceptionV3-based model achieved an accuracy of 96% and a dice coefficient score of 93%, demonstrating the effectiveness of the proposed system as a source of data acquisition for training deep neural networks.

Accelerated optimization in deep learning with a proportional-integral-derivative controller

High-performance optimization algorithms are essential in deep learning. However, understanding the behavior of optimization (i.e., learning process) remains challenging due to the instability and weak interpretability of the algorithms. Since gradient-based optimizations can be interpreted as continuous-time dynamical systems, applying feedback control to the dynamical systems that model the optimizers may provide another perspective for exploring more robust, accurate and explainable optimization algorithms. In this study, we present a framework for optimization called controlled heavy-ball optimizer. By employing the proportional-integral-derivative (PID) controller in the optimizer, we develop a deterministic continuous-time optimizer called Proportional-Integral-Derivative Accelerated Optimizer (PIDAO), and provide theoretical convergence analysis of PIDAO in unconstrained (non-)convex optimizations. As a byproduct, we derive PIDAO-family schemes for training deep neural networks by using specific discretization methods. Compared to classical optimizers, PIDAO can be empirically proven a more aggressive capacity to explore the loss landscape with lower computational costs due to the property of PID controller. Experimental evaluations demonstrate that PIDAO can accelerate the convergence and enhance the accuracy of deep learning, achieving state-of-the-art performance compared with advanced algorithms.

Fuel-optimal guidance using costate supervised learning with local refinement

This work presents a novel method to compute continuous fuel-optimal guidance updates on board a spacecraft by leveraging a combination of deep learning and a local refinement using differential algebraic techniques. In order to create the large datasets necessary to train the artificial neural network, we also propose a new method using polynomial maps to rapidly and efficiently generate arbitrarily high numbers of optimal trajectories. Constructed at the initial time, such maps readily provide entire fuel-optimal trajectories for any given state deviation from the nominal, by means of the simple and efficient evaluation of polynomials. Whilst a trained deep neural network is then capable of providing continuous guidance updates, we observe that it is not capable of learning the optimal guidance policy to an accuracy required for immediate application. Further differential algebraic techniques are therefore employed to locally refine the output of the neural network into highly accurate fuel-optimal guidance updates via a lightweight and iterative-less mapping procedure, suitable for onboard implementation. Performance assessment in both an interplanetary transfer from Earth to Psyche as well as a landing scenario on Psyche demonstrates the accuracy and robustness of the proposed guidance scheme.

Multi-objective optimization approach for permanent magnet machine via improved soft actor–critic based on deep reinforcement learning

As awareness of environmental protection grows, the development of electric vehicles has emerged as a prominent area of research. Naturally, optimizing Permanent Magnet (PM) machines is critical for enhancing the power performance of electric vehicles, as they are essential components of these vehicles. Benefitting from the advantage of Deep Neural Network (DNN), optimizing the PM machine by leveraging DNN has drawn great attention from industry and academia. To increase the torque of the PM machine and reduce torque ripple from a structural design perspective, this paper proposes a Multi-objective Optimization approach for PM machine via improved Soft Actor Critic (called MOPM-SAC) based on Deep Reinforcement Learning (DRL). MOPM-SAC consists of two core components: a surrogate model based on DNN and a SAC algorithm. Specifically, the SAC algorithm based on DNN is used to optimize the PM machine. The surrogate model is established by training the DNN to use Finite Element (FE) simulation samples, which possesses higher accuracy than other methods, such as Support Vector Machine (SVM), Random Forest (RF), and so on. Moreover, the SAC algorithm integrates a trained DNN as the state transition function within the DRL environment, which enhances the optimization algorithm’s generalization ability in the context of PM machines. In addition, the reward function is designed based on optimization requirements to guide the agent in the SAC algorithm toward learning the optimal strategy. Finally, a 75 kW prototype is manufactured and tested. The effectiveness of the proposed method is validated through FE simulations and prototype experiments.

Subword Dictionary Learning and Segmentation for Expanding the Vocabulary of Automatic Speech Recognition in Tamil and Kannada

We present automatic speech recognition (ASR) systems for Tamil and Kannada based on subword modeling to effectively handle unlimited vocabulary arising due to the highly agglutinative nature of the languages. We propose a variant of the byte pair encoding (BPE) algorithm named extended-BPE, and Morfessor tool to segment each word into its subwords. We have effectively incorporated maximum likelihood and Viterbi estimation techniques with weighted finite state transducers framework in these algorithms to learn the subword dictionary from a large text corpus. Using the learned subword dictionary, the words in the transcriptions of the training data are segmented into subwords. We train deep neural network ASR systems that recognize the subword sequence for any given test speech utterance. The output subword sequence is then post-processed using deterministic rules to get the final word sequence. Because of this subword design, the number of words that can be recognized is much larger than the number of words in the training corpus. For Tamil ASR, we use 152 hours of data for training and 65 hours for testing, whereas for Kannada ASR, we use 275 hours for training and 72 hours for testing. Upon experimenting with different combinations of segmentation and estimation techniques, we find that the word error rate (WER) reduces drastically when compared to the baseline word-level ASR, achieving a maximum absolute WER reduction of 6.24% and 6.63% for Tamil and Kannada, respectively. Further, comparing our results with that of an end-to-end ASR model available on Github, we have seen that our subword language models can perform comparable to or exceed that of recent end-to-end ASR models for Kannada and Tamil languages.

Predicting intermediate wavelength depth using a deep neural network combined with gravity data in the Sea of Japan

ABSTRACT Influenced by various geophysical factors, gravity and bathymetry exhibit significant correlations only at certain intermediate wavelengths. In this study, a deep neural network (DNN) combined with multi-source gravity data was used to recovery the intermediate wavelength (15–160 km) depth in the Sea of Japan (37–42°N, 130–138°E), to analyze the contributions of different gravity data to the bathymetric model. Subsequently, the gravity anomaly (GA), vertical gravity gradient (VGG), and vertical deflection were used as training data with a suitable DNN framework and designed training strategy. Evaluation results show that: the precision of the final DNN training models at test points surpasses that of the topo_25.1 model by 13 to 14 m; the TC1 model, with GA, VGG, and vertical deflection, is optimal, and the accuracy is 7.2 m higher than that of the (gravity geology method) GGM model on check cruises. The three types of gravity data can complement and constrain each other. In addition, predicting depths at intermediate wavelengths with significant coherence, as opposed to directly predicting absolute depths, results in higher accuracy by 20–30 m at test points, and an improvement of approximately 10 m in check accuracy.

A Data-Loader Tunable Knob to Shorten GPU Idleness for Distributed Deep Learning

Deep Neural Networks (DNNs) have been applied as an effective machine learning algorithm to tackle problems in different domains. However, the endeavor to train sophisticated DNN models can stretch from days into weeks, presenting substantial obstacles in the realm of research focused on large-scale DNN architectures. Distributed Deep Learning (DDL) contributes to accelerating DNN training by distributing training workloads across multiple computation accelerators, for example, graphics processing units (GPUs). Despite the considerable amount of research directed toward enhancing DDL training, the influence of data loading on GPU utilization and overall training efficacy remains relatively overlooked. It is non-trivial to optimize data-loading in DDL applications that need intensive central processing unit (CPU) and input/output (I/O) resources to process enormous training data. When multiple DDL applications are deployed on a system (e.g., Cloud and High-Performance Computing (HPC) system), the lack of a practical and efficient technique for data-loader allocation incurs GPU idleness and degrades the training throughput. Therefore, our work first focuses on investigating the impact of data-loading on the global training throughput. We then propose a throughput prediction model to predict the maximum throughput for an individual DDL training application. By leveraging the predicted results, A-Dloader is designed to dynamically allocate CPU and I/O resources to concurrently running DDL applications and use the data-loader allocation as a knob to reduce GPU idle intervals and thus improve the overall training throughput. We implement and evaluate A-Dloader in a DDL framework for a series of DDL applications arriving and completing across the runtime. Our experimental results show that A-Dloader can achieve a 28.9% throughput improvement and a 10% makespan improvement compared with allocating resources evenly across applications.

Adadelta-CSA: Adadelta-Chameleon Swarm Algorithm for EEG-Based Epileptic Seizure Detection

Epilepsy is referred to as a neurological disorder, which is detected via examination and manual comprehension of Electroencephalogram (EEG) signals. In deep learning schemes, various enhancements have emerged to efficiently address complex issues by end-to-end learning. The major objective of this research is to propose a new seizure detection approach from EEG signals using a deep learning-based classification technique. The pre-processing is the initial stage, where denoising is performed using a Short-Time Fourier Transform (STFT). Subsequently, the statistical features, time-domain features and spectral features are extracted from the pre-processed signal. Finally, an efficient optimization approach, named Adadelta-Chameleon Swarm Algorithm (Adadelta-CSA), is proposed and employed to train Deep Neural Network (DNN) to carry out the precise seizure prediction. Here, the integration of the Adadelta concept in the Chameleon Swarm Algorithm (CSA) has resulted in Adadelta-CSA. At last, the performance of the Adadelta-CSA scheme-based DNN is compared with the existing techniques by considering accuracy, sensitivity and specificity, and it is found to produce better values of 0.951, 0.966, and 0.935, respectively.

Few-shot learning for inference in medical imaging with subspace feature representations.

Unlike in the field of visual scene recognition, where tremendous advances have taken place due to the availability of very large datasets to train deep neural networks, inference from medical images is often hampered by the fact that only small amounts of data may be available. When working with very small dataset problems, of the order of a few hundred items of data, the power of deep learning may still be exploited by using a pre-trained model as a feature extractor and carrying out classic pattern recognition techniques in this feature space, the so-called few-shot learning problem. However, medical images are highly complex and variable, making it difficult for few-shot learning to fully capture and model these features. To address these issues, we focus on the intrinsic characteristics of the data. We find that, in regimes where the dimension of the feature space is comparable to or even larger than the number of images in the data, dimensionality reduction is a necessity and is often achieved by principal component analysis or singular value decomposition (PCA/SVD). In this paper, noting the inappropriateness of using SVD for this setting we explore two alternatives based on discriminant analysis (DA) and non-negative matrix factorization (NMF). Using 14 different datasets spanning 11 distinct disease types we demonstrate that at low dimensions, discriminant subspaces achieve significant improvements over SVD-based subspaces and the original feature space. We also show that at modest dimensions, NMF is a competitive alternative to SVD in this setting. The implementation of the proposed method is accessible via the following link.

Pole structure of Pψ N(4312)+ via machine learning and uniformized S-matrix

Abstract We probed the pole structure of the Pψ N(4312)+ using a trained deep neural network. The training dataset was generated using uniformized independent S-matrix poles to ensure that the resulting interpretation is as model-independent as possible. To prevent possible ambiguity in the interpretation of the pole structure, we included the contribution from the off-diagonal elements of the S-matrix. Our trained model favors a possible three-pole structure of Pψ N(4312)+ signal, with one pole on each of the unphysical sheets. The two poles on two different Riemann sheets can be associated with a pole-shadow pair which is a characteristic of a true resonance. On the other hand, the last isolated pole is most likely associated with the coupled-channel effect in the meson-baryon system. The combined effect of these poles produced a peak below the ΣcD threshold which mimics the line shape of a hadronic molecule.

Systematic Analysis of Low-Precision Training in Deep Neural Networks: Factors Influencing Matrix Computations

As Deep Neural Networks (DNNs) continue to increase in complexity, the computational demands of their training have become a significant bottleneck. Low-precision training has emerged as a crucial strategy, wherein full-precision values are quantized to lower precisions, reducing computational overhead while aiming to maintain model accuracy. While prior research has primarily focused on minimizing quantization noise and optimizing performance for specific models and tasks, a comprehensive understanding of the general principles governing low-precision computations across diverse DNN architectures has been lacking. In this paper, we address this gap by systematically analyzing the factors that influence low-precision matrix computations, which are fundamental to DNN training. We investigate three critical factors—accumulation in matrix calculations, the frequency of element usage, and the depth of matrices within the model—and their impact on low-precision training. Through controlled experiments on standard models, as well as customized experiments designed to isolate individual factors, we derive several key insights: layers with higher accumulation and matrices with lower usage frequencies demonstrate greater tolerance to low-precision noise, without significantly compromising the stability of model training. Additionally, while the depth of matrices influences the stability of matrix operations to some extent, it does not have a noticeable effect on the overall training outcomes. Our findings contribute to the development of generalizable principles for low-precision training, offering a systematic framework applicable across various DNN architectures. We provide empirical evidence supporting the strategic allocation of training bit-widths based on the analyzed factors, thereby enhancing the efficiency and effectiveness of DNN training in resource-constrained environments.

Locally linearized physics-informed neural networks for Riemann problems of hyperbolic conservation laws

In this work, we demonstrate that physics-informed neural networks (PINNs) tend to propagate predicted shock wave information bidirectionally in time, which does not align with the actual evolution direction of solutions to hyperbolic conservation laws. This mismatch results in instability and hinders the reduction of the loss of governing equations, as well as the initial condition loss by the deep neural network. In order to tackle this problem, we simplify the complexity of the problem by constructing equivalent linear transport equations in the region of shock wave generation. The speeds of these linearized waves are governed by the Rankine–Hugoniot relations of conservation laws. This approach is termed the Locally Linearized PINNs method. Specifically, an appropriate shock wave detector is initially designed to identify domains where shock waves occur. Near shock waves, the original nonlinear equations are transformed into their linearized forms, thereby modifying the residual terms of the partial differential equations. Additionally, an equilibrium factor is introduced in fluid compression regions to reduce prediction errors and stabilize the training of deep neural networks. Numerical examples illustrate that Locally Linearized PINNs effectively address the challenge of predicting global solutions in PINNs and significantly improve shock-capturing performance for hyperbolic conservation laws.