Training Deep Neural Networks Research Articles

Adadelta-CSA: Adadelta-Chameleon Swarm Algorithm for EEG-Based Epileptic Seizure Detection

Epilepsy is referred to as a neurological disorder, which is detected via examination and manual comprehension of Electroencephalogram (EEG) signals. In deep learning schemes, various enhancements have emerged to efficiently address complex issues by end-to-end learning. The major objective of this research is to propose a new seizure detection approach from EEG signals using a deep learning-based classification technique. The pre-processing is the initial stage, where denoising is performed using a Short-Time Fourier Transform (STFT). Subsequently, the statistical features, time-domain features and spectral features are extracted from the pre-processed signal. Finally, an efficient optimization approach, named Adadelta-Chameleon Swarm Algorithm (Adadelta-CSA), is proposed and employed to train Deep Neural Network (DNN) to carry out the precise seizure prediction. Here, the integration of the Adadelta concept in the Chameleon Swarm Algorithm (CSA) has resulted in Adadelta-CSA. At last, the performance of the Adadelta-CSA scheme-based DNN is compared with the existing techniques by considering accuracy, sensitivity and specificity, and it is found to produce better values of 0.951, 0.966, and 0.935, respectively.

Anatomic Interpretability in Neuroimage Deep Learning: Saliency Approaches for Typical Aging and Traumatic Brain Injury.

The black box nature of deep neural networks (DNNs) makes researchers and clinicians hesitant to rely on their findings. Saliency maps can enhance DNN explainability by suggesting the anatomic localization of relevant brain features. This study compares seven popular attribution-based saliency approaches to assign neuroanatomic interpretability to DNNs that estimate biological brain age (BA) from magnetic resonance imaging (MRI). Cognitively normal (CN) adults (N = 13,394, 5,900 males; mean age: 65.82 ± 8.89years) are included for DNN training, testing, validation, and saliency map generation to estimate BA. To study saliency robustness to the presence of anatomic deviations from normality, saliency maps are also generated for adults with mild traumatic brain injury (mTBI, = 214, 135 males; mean age: 55.3 ± 9.9years). We assess saliency methods' capacities to capture known anatomic features of brain aging and compare them to a surrogate ground truth whose anatomic saliency is known a priori. Anatomic aging features are identified most reliably by the integrated gradients method, which outperforms all others through its ability to localize relevant anatomic features. Gradient Shapley additive explanations, input × gradient, and masked gradient perform less consistently but still highlight ubiquitous neuroanatomic features of aging (ventricle dilation, hippocampal atrophy, sulcal widening). Saliency methods involving gradient saliency, guided backpropagation, and guided gradient-weight class attribution mapping localize saliency outside the brain, which is undesirable. Our research suggests the relative tradeoffs of saliency methods to interpret DNN findings during BA estimation in typical aging and after mTBI.

Pole structure of Pψ N(4312)+ via machine learning and uniformized S-matrix

Abstract We probed the pole structure of the Pψ N(4312)+ using a trained deep neural network. The training dataset was generated using uniformized independent S-matrix poles to ensure that the resulting interpretation is as model-independent as possible. To prevent possible ambiguity in the interpretation of the pole structure, we included the contribution from the off-diagonal elements of the S-matrix. Our trained model favors a possible three-pole structure of Pψ N(4312)+ signal, with one pole on each of the unphysical sheets. The two poles on two different Riemann sheets can be associated with a pole-shadow pair which is a characteristic of a true resonance. On the other hand, the last isolated pole is most likely associated with the coupled-channel effect in the meson-baryon system. The combined effect of these poles produced a peak below the ΣcD threshold which mimics the line shape of a hadronic molecule.

Systematic Analysis of Low-Precision Training in Deep Neural Networks: Factors Influencing Matrix Computations

As Deep Neural Networks (DNNs) continue to increase in complexity, the computational demands of their training have become a significant bottleneck. Low-precision training has emerged as a crucial strategy, wherein full-precision values are quantized to lower precisions, reducing computational overhead while aiming to maintain model accuracy. While prior research has primarily focused on minimizing quantization noise and optimizing performance for specific models and tasks, a comprehensive understanding of the general principles governing low-precision computations across diverse DNN architectures has been lacking. In this paper, we address this gap by systematically analyzing the factors that influence low-precision matrix computations, which are fundamental to DNN training. We investigate three critical factors—accumulation in matrix calculations, the frequency of element usage, and the depth of matrices within the model—and their impact on low-precision training. Through controlled experiments on standard models, as well as customized experiments designed to isolate individual factors, we derive several key insights: layers with higher accumulation and matrices with lower usage frequencies demonstrate greater tolerance to low-precision noise, without significantly compromising the stability of model training. Additionally, while the depth of matrices influences the stability of matrix operations to some extent, it does not have a noticeable effect on the overall training outcomes. Our findings contribute to the development of generalizable principles for low-precision training, offering a systematic framework applicable across various DNN architectures. We provide empirical evidence supporting the strategic allocation of training bit-widths based on the analyzed factors, thereby enhancing the efficiency and effectiveness of DNN training in resource-constrained environments.

Locally linearized physics-informed neural networks for Riemann problems of hyperbolic conservation laws

In this work, we demonstrate that physics-informed neural networks (PINNs) tend to propagate predicted shock wave information bidirectionally in time, which does not align with the actual evolution direction of solutions to hyperbolic conservation laws. This mismatch results in instability and hinders the reduction of the loss of governing equations, as well as the initial condition loss by the deep neural network. In order to tackle this problem, we simplify the complexity of the problem by constructing equivalent linear transport equations in the region of shock wave generation. The speeds of these linearized waves are governed by the Rankine–Hugoniot relations of conservation laws. This approach is termed the Locally Linearized PINNs method. Specifically, an appropriate shock wave detector is initially designed to identify domains where shock waves occur. Near shock waves, the original nonlinear equations are transformed into their linearized forms, thereby modifying the residual terms of the partial differential equations. Additionally, an equilibrium factor is introduced in fluid compression regions to reduce prediction errors and stabilize the training of deep neural networks. Numerical examples illustrate that Locally Linearized PINNs effectively address the challenge of predicting global solutions in PINNs and significantly improve shock-capturing performance for hyperbolic conservation laws.

Model Parameter Prediction Method for Accelerating Distributed DNN Training

As the size of deep neural network (DNN) models and datasets increases, distributed training becomes popular to reduce the training time. However, a severe communication bottleneck in distributed training limits its scalability. Many methods aim to solve this communication bottleneck by reducing communication traffic, such as gradient sparsification and quantification. However, these methods either are at the expense of losing model accuracy or introducing lots of computing overhead. We observe that the data distribution between layers of neural network models is similar. Thus, we propose a model parameter prediction method (MP2) to accelerate distributed DNN training under parameter server (PS) framework, where workers push only a subset of model parameters to PS, and residual model parameters are locally predicted by an already-trained deep neural network model on PS. Some key challenges are addressed. First, we build a hierarchical parameters dataset by randomly sampling a subset of model from normal distributed trainings. Second, we build a neural network model with the structure of “convolution + channel attention + Max pooling” for predicting model parameters by using a prediction result-based evaluation method. For VGGNet, ResNet, and AlexNet models on CIFAR10 and CIFAR100 datasets, compared with Baseline, Top-k, deep gradient compression (DGC), and weight nowcaster network (WNN), MP2 can reduce traffic by up to 88.98%; and accelerates the training by up to 47.32% while not losing the model accuracy. MP2 has shown good generalization.

Quijote-PNG: Optimizing the Summary Statistics to Measure Primordial Non-Gaussianity

We apply a suite of different estimators to the Quijote-png halo catalogs to find the best approach to constrain Primordial non-Gaussianity (PNG) at nonlinear cosmological scales, up to kmax=0.5hMpc−1 . The set of summary statistics considered in our analysis includes the power spectrum, bispectrum, halo mass function, marked power spectrum, and marked modal bispectrum. Marked statistics are used here for the first time in the context of the PNG study. We perform a Fisher analysis to estimate their cosmological information content, showing substantial improvements when marked observables are added to the analysis. Starting from these summaries, we train deep neural networks to perform likelihood-free inference of cosmological and PNG parameters. We assess the performance of different subsets of summary statistics; in the case of fNLequil , we find that a combination of the power spectrum and a suitable marked power spectrum outperforms the combination of power spectrum and bispectrum, the baseline statistics usually employed in PNG analysis. A minimal pipeline to analyze the statistics we identified can be implemented either with our ML algorithm or via more traditional estimators, if these are deemed more reliable.

Les Houches lectures on deep learning at large and infinite width*

Abstract These lectures, presented at the 2022 Les Houches Summer School on Statistical Physics and Machine Learning, focus on the infinite-width limit and large-width regime of deep neural networks. Topics covered include the various statistical and dynamical properties of these networks. In particular, the lecturers discuss properties of random deep neural networks, connections between trained deep neural networks, linear models, kernels and Gaussian processes that arise in the infinite-width limit, and perturbative and non-perturbative treatments of large but finite-width networks, at initialization and after training.

Stochastic collapse: how gradient noise attracts SGD dynamics towards simpler subnetworks*

Abstract In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify invariant sets, or subsets of parameter space that remain unmodified by SGD. We focus on two classes of invariant sets that correspond to simpler (sparse or low-rank) subnetworks and commonly appear in modern architectures. Our analysis uncovers that SGD exhibits a property of stochastic attractivity towards these simpler invariant sets. We establish a sufficient condition for stochastic attractivity based on a competition between the loss landscape’s curvature around the invariant set and the noise introduced by stochastic gradients. Remarkably, we find that an increased level of noise strengthens attractivity, leading to the emergence of attractive invariant sets associated with saddle-points or local maxima of the train loss. We observe empirically the existence of attractive invariant sets in trained deep neural networks, implying that SGD dynamics often collapses to simple subnetworks with either vanishing or redundant neurons. We further demonstrate how this simplifying process of stochastic collapse benefits generalization in a linear teacher-student framework. Finally, through this analysis, we mechanistically explain why early training with large learning rates for extended periods benefits subsequent generalization.

Exploiting Compress Sensing in Training of Deep Neural Network for Self-Noise Cancellation in Underwater Acoustics

Exploiting Compress Sensing in Training of Deep Neural Network for Self-Noise Cancellation in Underwater Acoustics

Anatomic Interpretability in Neuroimage Deep Learning: Saliency Approaches for Typical Aging and Traumatic Brain Injury.

The black box nature of deep neural networks (DNNs) makes researchers and clinicians hesitant to rely on their findings. Saliency maps can enhance DNN explainability by suggesting the anatomic localization of relevant brain features. This study compares seven popular attribution-based saliency approaches to assign neuroanatomic interpretability to DNNs that estimate biological brain age (BA) from magnetic resonance imaging (MRI). Cognitively normal (CN) adults ( N = 13,394, 5,900 males; mean age: 65.82 ± 8.89 years) are included for DNN training, testing, validation, and saliency map generation to estimate BA. To study saliency robustness to the presence of anatomic deviations from normality, saliency maps are also generated for adults with mild traumatic brain injury (mTBI, \(\:N\) = 214, 135 males; mean age: 55.3 ± 9.9 years). We assess saliency methods' capacities to capture known anatomic features of brain aging and compare them to a surrogate ground truth whose anatomic saliency is known a priori . Anatomic aging features are identified most reliably by the integrated gradients method, which outperforms all others through its ability to localize relevant anatomic features. Gradient Shapley additive explanations, input × gradient, and masked gradient perform less consistently but still highlight ubiquitous neuroanatomic features of aging (ventricle dilation, hippocampal atrophy, sulcal widening). Saliency methods involving gradient saliency, guided backpropagation, and guided gradient-weight class attribution mapping localize saliency outside the brain, which is undesirable. Our research suggests the relative tradeoffs of saliency methods to interpret DNN findings during BA estimation in typical aging and after mTBI.

Impact of Mask Type as Training Target for Speech Intelligibility and Quality in Cochlear-Implant Noise Reduction

The selection of a target when training deep neural networks for speech enhancement is an important consideration. Different masks have been shown to exhibit different performance characteristics depending on the application and the conditions. This paper presents a comprehensive comparison of several different masks for noise reduction in cochlear implants. The study incorporated three well-known masks, namely the Ideal Binary Mask (IBM), Ideal Ratio Mask (IRM) and the Fast Fourier Transform Mask (FFTM), as well as two newly proposed masks, based on existing masks, called the Quantized Mask (QM) and the Phase-Sensitive plus Ideal Ratio Mask (PSM+). These five masks are used to train networks to estimate masks for the purpose of separating speech from noisy mixtures. A vocoder was used to simulate the behavior of a cochlear implant. Short-time Objective Intelligibility (STOI) and Perceptual Evaluation of Speech Quality (PESQ) scores indicate that the two new masks proposed in this study (QM and PSM+) perform best for normal speech intelligibility and quality in the presence of stationary and non-stationary noise over a range of signal-to-noise ratios (SNRs). The Normalized Covariance Measure (NCM) and similarity scores indicate that they also perform best for speech intelligibility/gauging the similarity of vocoded speech. The Quantized Mask performs better than the Ideal Binary Mask due to its better resolution as it approximates the Wiener Gain Function. The PSM+ performs better than the three existing benchmark masks (IBM, IRM, and FFTM) as it incorporates both magnitude and phase information.

Deep learning of Sentinel-1 SAR for burnt peatland detection in Ireland

Peatlands represent vital carbon reserves; however, once ignited, they release stored carbon, inflicting lasting environmental harm and necessitating prolonged recovery periods. An innovative method merging Sentinel-1 satellite imagery and deep learning (DL) is proposed to monitor burnt peat across diverse regions of Ireland, regardless of weather conditions or time of day. Sentinel-2 images and field measurements were used to train deep neural networks (DNN) and the accuracy in detecting burnt peat areas reached 80 %. This was achieved by combining the VV (vertical transmit, vertical receive) and VH (vertical transmit, horizontal receive) from Sentinel-1. Time-series analysis of Sentinel-1 VV backscatter change for Wicklow Mountains in 2018 highlights the Sentinel-1's capacity to detect various phenomena, including snowfall and burnt peat, evident prior to the peat fire event. Furthermore, an examination of peat fire occurrences in Wicklow Mountains from 2018 to 2023 through time series and mapping shows a significant escalation, with the largest burnt areas detected in 2023 spanning over 40 km².

Simple integrated circuit reverse-engineering with deep learning: A proof of concept for automating die-polygon-capturing

The purpose of this research is to demonstrate the feasibility of automating ‘die-polygon-capturing’, an economical yet still labor-intensive technique for circuit extraction during the reverse-engineering of simple integrated circuits. As microchip designs become increasingly diverse with the ongoing trend of using application-specific integrated circuits, and considering the importance of reverse-engineering these components, die-polygon-capturing is set to play a more critical role in the future. Due to the apparent absence of prior scientific publications and limited automation efforts in this area, this paper presents a proof of concept for automating the die-polygon-capturing technique, thereby addressing a notable gap in the existing literature, with an overarching goal of reducing the labor-intensity of die-polygon-capturing. Our method consists of training deep neural networks on variations of a dataset and evaluating their segmentation capabilities of various layers and connections in an integrated circuit. Given the limited accessibility to high-quality labeled datasets, the dataset used for this research consists of two images of a single microchip, the AMD 9085D. We implemented a data augmentation process that expanded this dataset to as many as 4872 images. The experiment’s results proof the automation’s feasibility, demonstrating high scores on Intersection over Union and F-beta evaluation metrics. However the primary conclusion drawn is the need for focus on generalizability towards other types of microchips in order to effectively automate this technique for circuit extraction.

Deep learning as Ricci flow.

Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data. It is known that data passing through a trained DNN classifier undergoes a series of geometric and topological simplifications. While some progress has been made toward understanding these transformations in neural networks with smooth activation functions, an understanding in the more general setting of non-smooth activation functions, such as the rectified linear unit (ReLU), which tend to perform better, is required. Here we propose that the geometric transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow-a tool from differential geometry that evolves a manifold by smoothing its curvature, in order to identify its topology. To illustrate this idea, we present a computational framework to quantify the geometric changes that occur as data passes through successive layers of a DNN, and use this framework to motivate a notion of 'global Ricci network flow' that can be used to assess a DNN's ability to disentangle complex data geometries to solve classification problems. By training more than 1500 DNN classifiers of different widths and depths on synthetic and real-world data, we show that the strength of global Ricci network flow-like behaviour correlates with accuracy for well-trained DNNs, independently of depth, width and data set. Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.

Multi-fault diagnosis of industrial rotating machines using advanced sliding window and WSST-CNN

Accurate fault diagnosis of rotating machines is the most important part of industries, as it helps manage the workforce and inventory within the stipulated time. In industries, when the fault occurs at one component, that may propagate to other components due to excessive speed and load, leading to the occurrence of a simultaneous fault condition. Identification of multi-component faults is relatively more difficult than single-component faults due to the possibility of characteristic frequency overlapping that is observed during signal processing. This work considered various single and complex multi-component fault conditions of a rotating machine, and data is acquired at different speed and load conditions using a standard acoustic array setup. The signals are then converted into signal segments using a continuously moving advanced sliding window. From the extracted segments time-frequency spectrums are generated by using Wavelet Synchrosqueezed Transform (WSST) followed by the training of deep neural networks. The performance analysis of the model is carried out, and it is found that the proposed methodology helped diagnose multi-faults at different speed and load conditions. It mitigates the complexity of high-end signal processing techniques for distinguishing the fault characteristics. This study will be beneficial for industries to diagnose multi-faults in incipiently.

Identification of perceived sentences using deep neural networks in EEG

Objetive. Decoding speech from brain activity can enable communication for individuals with speech disorders. Deep neural networks (DNNs) have shown great potential for speech decoding applications. However, the limited availability of large datasets containing neural recordings from speech-impaired subjects poses a challenge. Leveraging data from healthy participants can mitigate this limitation and expedite the development of speech neuroprostheses while minimizing the need for patient-specific training data.Approach. In this study, we collected a substantial dataset consisting of recordings from 56 healthy participants using 64 EEG channels. Multiple neural networks were trained to classify perceived sentences in the Spanish language using subject-independent, mixed-subjects, and fine-tuning approaches. The dataset has been made publicly available to foster further research in this area.Main results. Our results demonstrate a remarkable level of accuracy in distinguishing sentence identity across 30 classes, showcasing the feasibility of training DNNs to decode sentence identity from perceived speech using EEG. Notably, the subject-independent approach rendered accuracy comparable to the mixed-subjects approach, although with higher variability among subjects. Additionally, our fine-tuning approach yielded even higher accuracy, indicating an improved capability to adapt to individual subject characteristics, which enhances performance. This suggests that DNNs have effectively learned to decode universal features of brain activity across individuals while also being adaptable to specific participant data. Furthermore, our analyses indicate that EEGNet and DeepConvNet exhibit comparable performance, outperforming ShallowConvNet for sentence identity decoding. Finally, our Grad-CAM visualization analysis identifies key areas influencing the network's predictions, offering valuable insights into the neural processes underlying language perception and comprehension.Significance. These findings advance our understanding of EEG-based speech perception decoding and hold promise for the development of speech neuroprostheses, particularly in scenarios where subjects cannot provide their own training data.

Soft sensor for melting flow rate prediction based on data-enhanced classification method

Abstract The melt flow rate (MFR) is a crucial quality indicator for polyolefin products. However, the offline laboratory measurement method suffers from significant time delays between sampling and obtaining test values, making it inconvenient for real-time product quality control. Additionally, installing an online analyzer can be expensive. In this study, we propose a soft sensor method that combines mechanism analysis and data-driven approaches to predict the MFR in the extruder process. Firstly, we analyze the physical mechanism of the extruder process, and the time series data are processed into independent enhanced data sets by reasonably selecting data enhancement parameters. Secondly, we statistically model the distribution of particle MFR and classify the data into four sections. We train Deep Neural Network (DNN) and Gradient Boosting Decision Tree (GBDT) regression models separately for low and non-low MFR sections while also developing a global classification model to invoke corresponding regression models when needed. The results demonstrate that our classification model achieves 100% accuracy in distinguishing between low and non-low MFR categories; furthermore, our regression models achieve an RMSE of 0.600, and an R2 value of 0.998, outperforming existing models on various indicators. Finally, considering viscosity variations in polyolefin products' influence on prediction performance, we recommend setting data enhancement parameters contrary to particle MFR trends.

Development of AI-assisted microscopy frameworks through realistic simulation with pySTED

The integration of artificial intelligence into microscopy systems significantly enhances performance, optimizing both image acquisition and analysis phases. Development of artificial intelligence-assisted super-resolution microscopy is often limited by access to large biological datasets, as well as by difficulties to benchmark and compare approaches on heterogeneous samples. We demonstrate the benefits of a realistic stimulated emission depletion microscopy simulation platform, pySTED, for the development and deployment of artificial intelligence strategies for super-resolution microscopy. pySTED integrates theoretically and empirically validated models for photobleaching and point spread function generation in stimulated emission depletion microscopy, as well as simulating realistic point-scanning dynamics and using a deep learning model to replicate the underlying structures of real images. This simulation environment can be used for data augmentation to train deep neural networks, for the development of online optimization strategies and to train reinforcement learning models. Using pySTED as a training environment allows the reinforcement learning models to bridge the gap between simulation and reality, as showcased by its successful deployment on a real microscope system without fine tuning.

Constructed encoded data based coded distributed DNN training for edge computing scenario

Deep learning in unmanned aerial vehicle (UAV) deployment encounters two problems: 1, One UAV may fail to store and execute large Deep Neural Network (DNN) model. 2, One UAV fails to accomplish real time services. One possible solution is the group of UAVs (nodes) collectively generate swarm intelligence in the form of edge computing scenario, namely in the distributed computing (DC) mode with clever arrangement, say Coded Distributed Computing (CDC). In CDC systems, the redundant computation introduced by linear coding can compensate stragglers. However, since linear property cannot pass the nonlinear activation function in Deep Neural Network (DNN) training, coding/decoding for CDC need to be applied layer by layer, which slows down the training. To avoid layer-by-layer coding/decoding, we propose a novel DNN training scheme based on constructing encoded data. This construction process lies before the training process (can be done before training without any impact on training efficiency). Based on both the original data and the newly constructed encoded data, the training phase can take advantage of the (n,k) property (Wait for the first k returned data) and hence improve the training speed. The training process does not require encoding/decoding operations, and hence significantly improves the training speed. Experimental results show that the training scheme based on constructed encoded data can achieve prediction accuracy approximating that of the centralized one and significantly reduce the latency compared to the layer-by-layer linear encoding and decoding scheme.