Training Deep Neural Networks Research Articles

Stochastic collapse: how gradient noise attracts SGD dynamics towards simpler subnetworks*

Abstract In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify invariant sets, or subsets of parameter space that remain unmodified by SGD. We focus on two classes of invariant sets that correspond to simpler (sparse or low-rank) subnetworks and commonly appear in modern architectures. Our analysis uncovers that SGD exhibits a property of stochastic attractivity towards these simpler invariant sets. We establish a sufficient condition for stochastic attractivity based on a competition between the loss landscape’s curvature around the invariant set and the noise introduced by stochastic gradients. Remarkably, we find that an increased level of noise strengthens attractivity, leading to the emergence of attractive invariant sets associated with saddle-points or local maxima of the train loss. We observe empirically the existence of attractive invariant sets in trained deep neural networks, implying that SGD dynamics often collapses to simple subnetworks with either vanishing or redundant neurons. We further demonstrate how this simplifying process of stochastic collapse benefits generalization in a linear teacher-student framework. Finally, through this analysis, we mechanistically explain why early training with large learning rates for extended periods benefits subsequent generalization.

Identification of perceived sentences using deep neural networks in EEG.

Decoding speech from brain activity can enable communication for individuals with speech disorders. Deep neural networks have shown great potential for speech decoding applications. However, the limited availability of large datasets containing neural recordings from speech-impaired subjects poses a challenge. Leveraging data from healthy participants can mitigate this limitation and expedite the development of speech neuroprostheses while minimizing the need for patient-specific training data. 
Approach. In this study, we collected a substantial dataset consisting of recordings from 56 healthy participants using 64 EEG channels. Multiple neural networks were trained to classify perceived sentences in the Spanish language using subject-independent, mixed-subjects, and fine-tuning approaches. The dataset has been made publicly available to foster further research in this area.
Main results. Our results demonstrate a remarkable level of accuracy in distinguishing sentence identity across 30 classes, showcasing the feasibility of training Deep Neural Networks (DNNs) to decode sentence identity from perceived speech using EEG. Notably, the subject-independent approach rendered accuracy comparable to the mixed-subjects approach, although with higher variability among subjects. Additionally, our fine-tuning approach yielded even higher accuracy, indicating an improved capability to adapt to individual subject characteristics, which enhances performance. This suggests that DNNs have effectively learned to decode universal features of brain activity across individuals while also being adaptable to specific participant data. Furthermore, our analyses indicate that EEGNet and DeepConvNet exhibit comparable performance, outperforming ShallowConvNet for sentence identity decoding. Finally, our Grad-CAM visualization analysis identifies key areas influencing the network's predictions, offering valuable insights into the neural processes underlying language perception and comprehension.
Significance. These findings advance our understanding of EEG-based speech perception decoding and hold promise for the development of speech neuroprostheses, particularly in scenarios where subjects cannot provide their own training data.

Exploiting Compress Sensing in Training of Deep Neural Network for Self-Noise Cancellation in Underwater Acoustics

Exploiting Compress Sensing in Training of Deep Neural Network for Self-Noise Cancellation in Underwater Acoustics

Impact of Mask Type as Training Target for Speech Intelligibility and Quality in Cochlear-Implant Noise Reduction

The selection of a target when training deep neural networks for speech enhancement is an important consideration. Different masks have been shown to exhibit different performance characteristics depending on the application and the conditions. This paper presents a comprehensive comparison of several different masks for noise reduction in cochlear implants. The study incorporated three well-known masks, namely the Ideal Binary Mask (IBM), Ideal Ratio Mask (IRM) and the Fast Fourier Transform Mask (FFTM), as well as two newly proposed masks, based on existing masks, called the Quantized Mask (QM) and the Phase-Sensitive plus Ideal Ratio Mask (PSM+). These five masks are used to train networks to estimate masks for the purpose of separating speech from noisy mixtures. A vocoder was used to simulate the behavior of a cochlear implant. Short-time Objective Intelligibility (STOI) and Perceptual Evaluation of Speech Quality (PESQ) scores indicate that the two new masks proposed in this study (QM and PSM+) perform best for normal speech intelligibility and quality in the presence of stationary and non-stationary noise over a range of signal-to-noise ratios (SNRs). The Normalized Covariance Measure (NCM) and similarity scores indicate that they also perform best for speech intelligibility/gauging the similarity of vocoded speech. The Quantized Mask performs better than the Ideal Binary Mask due to its better resolution as it approximates the Wiener Gain Function. The PSM+ performs better than the three existing benchmark masks (IBM, IRM, and FFTM) as it incorporates both magnitude and phase information.

Simple integrated circuit reverse-engineering with deep learning: A proof of concept for automating die-polygon-capturing

The purpose of this research is to demonstrate the feasibility of automating ‘die-polygon-capturing’, an economical yet still labor-intensive technique for circuit extraction during the reverse-engineering of simple integrated circuits. As microchip designs become increasingly diverse with the ongoing trend of using application-specific integrated circuits, and considering the importance of reverse-engineering these components, die-polygon-capturing is set to play a more critical role in the future. Due to the apparent absence of prior scientific publications and limited automation efforts in this area, this paper presents a proof of concept for automating the die-polygon-capturing technique, thereby addressing a notable gap in the existing literature, with an overarching goal of reducing the labor-intensity of die-polygon-capturing. Our method consists of training deep neural networks on variations of a dataset and evaluating their segmentation capabilities of various layers and connections in an integrated circuit. Given the limited accessibility to high-quality labeled datasets, the dataset used for this research consists of two images of a single microchip, the AMD 9085D. We implemented a data augmentation process that expanded this dataset to as many as 4872 images. The experiment’s results proof the automation’s feasibility, demonstrating high scores on Intersection over Union and F-beta evaluation metrics. However the primary conclusion drawn is the need for focus on generalizability towards other types of microchips in order to effectively automate this technique for circuit extraction.

Deep learning as Ricci flow.

Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data. It is known that data passing through a trained DNN classifier undergoes a series of geometric and topological simplifications. While some progress has been made toward understanding these transformations in neural networks with smooth activation functions, an understanding in the more general setting of non-smooth activation functions, such as the rectified linear unit (ReLU), which tend to perform better, is required. Here we propose that the geometric transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow-a tool from differential geometry that evolves a manifold by smoothing its curvature, in order to identify its topology. To illustrate this idea, we present a computational framework to quantify the geometric changes that occur as data passes through successive layers of a DNN, and use this framework to motivate a notion of 'global Ricci network flow' that can be used to assess a DNN's ability to disentangle complex data geometries to solve classification problems. By training more than 1500 DNN classifiers of different widths and depths on synthetic and real-world data, we show that the strength of global Ricci network flow-like behaviour correlates with accuracy for well-trained DNNs, independently of depth, width and data set. Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.

Multi-fault diagnosis of industrial rotating machines using advanced sliding window and WSST-CNN

Accurate fault diagnosis of rotating machines is the most important part of industries, as it helps manage the workforce and inventory within the stipulated time. In industries, when the fault occurs at one component, that may propagate to other components due to excessive speed and load, leading to the occurrence of a simultaneous fault condition. Identification of multi-component faults is relatively more difficult than single-component faults due to the possibility of characteristic frequency overlapping that is observed during signal processing. This work considered various single and complex multi-component fault conditions of a rotating machine, and data is acquired at different speed and load conditions using a standard acoustic array setup. The signals are then converted into signal segments using a continuously moving advanced sliding window. From the extracted segments time-frequency spectrums are generated by using Wavelet Synchrosqueezed Transform (WSST) followed by the training of deep neural networks. The performance analysis of the model is carried out, and it is found that the proposed methodology helped diagnose multi-faults at different speed and load conditions. It mitigates the complexity of high-end signal processing techniques for distinguishing the fault characteristics. This study will be beneficial for industries to diagnose multi-faults in incipiently.

Sentinel-1 SAR and deep learning for peatland fire detection in Ireland

Peatlands represent vital carbon reserves; however, once ignited, they release stored carbon, inflicting lasting environmental harm and necessitating prolonged recovery periods. An innovative method merging Sentinel-1 satellite imagery and deep learning (DL) is proposed to monitor peat fires across diverse regions of Ireland, regardless of weather conditions or time of day. A comprehensive dataset was assembled to train deep neural networks (DNN) using Sentinel-2 thermal band data and field measurement, resulting in 80% accuracy in detecting peat fire areas. This was achieved by incorporating inputs of both VV (vertical transmit, vertical receive) and VH (vertical transmit, horizontal receive) from Sentinel-1. Analysis of Sentinel-1 VV backscatter change time series for Wicklow Mountain in 2018 underscores the satellite's capacity to detect various phenomena, including snowfall and peat fire, evident prior to the peat fire event. Furthermore, scrutiny of peat fire occurrences in Wicklow Mountain from 2018 to 2023 via time series and mapping reveals a significant escalation, with the largest fire recorded in 2023 spanning over 40 km².

Soft sensor for melting flow rate prediction based on data-enhanced classification method

Abstract The melt flow rate (MFR) is a crucial quality indicator for polyolefin products. However, the offline laboratory measurement method suffers from significant time delays between sampling and obtaining test values, making it inconvenient for real-time product quality control. Additionally, installing an online analyzer can be expensive. In this study, we propose a soft sensor method that combines mechanism analysis and data-driven approaches to predict the MFR in the extruder process. Firstly, we analyze the physical mechanism of the extruder process, and the time series data are processed into independent enhanced data sets by reasonably selecting data enhancement parameters. Secondly, we statistically model the distribution of particle MFR and classify the data into four sections. We train Deep Neural Network (DNN) and Gradient Boosting Decision Tree (GBDT) regression models separately for low and non-low MFR sections while also developing a global classification model to invoke corresponding regression models when needed. The results demonstrate that our classification model achieves 100% accuracy in distinguishing between low and non-low MFR categories; furthermore, our regression models achieve an RMSE of 0.600, and an R2 value of 0.998, outperforming existing models on various indicators. Finally, considering viscosity variations in polyolefin products' influence on prediction performance, we recommend setting data enhancement parameters contrary to particle MFR trends.

Development of AI-assisted microscopy frameworks through realistic simulation with pySTED

The integration of artificial intelligence into microscopy systems significantly enhances performance, optimizing both image acquisition and analysis phases. Development of artificial intelligence-assisted super-resolution microscopy is often limited by access to large biological datasets, as well as by difficulties to benchmark and compare approaches on heterogeneous samples. We demonstrate the benefits of a realistic stimulated emission depletion microscopy simulation platform, pySTED, for the development and deployment of artificial intelligence strategies for super-resolution microscopy. pySTED integrates theoretically and empirically validated models for photobleaching and point spread function generation in stimulated emission depletion microscopy, as well as simulating realistic point-scanning dynamics and using a deep learning model to replicate the underlying structures of real images. This simulation environment can be used for data augmentation to train deep neural networks, for the development of online optimization strategies and to train reinforcement learning models. Using pySTED as a training environment allows the reinforcement learning models to bridge the gap between simulation and reality, as showcased by its successful deployment on a real microscope system without fine tuning.

Constructed encoded data based coded distributed DNN training for edge computing scenario

Deep learning in unmanned aerial vehicle (UAV) deployment encounters two problems: 1, One UAV may fail to store and execute large Deep Neural Network (DNN) model. 2, One UAV fails to accomplish real time services. One possible solution is the group of UAVs (nodes) collectively generate swarm intelligence in the form of edge computing scenario, namely in the distributed computing (DC) mode with clever arrangement, say Coded Distributed Computing (CDC). In CDC systems, the redundant computation introduced by linear coding can compensate stragglers. However, since linear property cannot pass the nonlinear activation function in Deep Neural Network (DNN) training, coding/decoding for CDC need to be applied layer by layer, which slows down the training. To avoid layer-by-layer coding/decoding, we propose a novel DNN training scheme based on constructing encoded data. This construction process lies before the training process (can be done before training without any impact on training efficiency). Based on both the original data and the newly constructed encoded data, the training phase can take advantage of the (n,k) property (Wait for the first k returned data) and hence improve the training speed. The training process does not require encoding/decoding operations, and hence significantly improves the training speed. Experimental results show that the training scheme based on constructed encoded data can achieve prediction accuracy approximating that of the centralized one and significantly reduce the latency compared to the layer-by-layer linear encoding and decoding scheme.

Activated biochar production from young coconut waste (Cocos nucifera) as bioadsorbent: a pathway through Artificial Neural Network (ANN) optimization.

This pioneering work explores the immense potential of young coconut waste, a continuously marginalized residue of the food and beverage industry, to serve as an indispensable feedstock in the production of biochar. Through an examination of the key carbonization factors that include time, temperature, and concentrations of the activating agent, KOH, the outcomes offer relevant insights that could be leveraged to maximize biochar production for tailored applications. This study stands out for its innovative use of Artificial Neural Network (ANN) approaches for predictive modeling. Fifty datasets, supplemented with secondary data obtained from the literature and experiments, were utilized for the purposes of training, testing, and validating the neural network model. Here, the datasets were processed utilizing the Deep Neural Network (DNN) framework, which was designed and implemented with the minimal loss function framework feasible. The architectural configuration comprises the following; an input layer, four hidden layers (128-neuron dense layer, batch normalization, and 64-neuron dense layer, batch normalization), a dropout layer, and an output layer. With an R2 of 0.8238 for biochar yield and 0.7324 for iodine number, the trained DNN model showed a relatively high degree of accuracy in making predictions.

Deep-neural-network model for predicting ground motion parameters using earthquake horizontal-to-vertical spectral ratios

This study proposed a deep-neural-network (DNN) model for seismic ground motion prediction by utilizing a unified strong motion database by the National Research Institute for Earth Science and Disaster Resilience, and earthquake horizontal-to-vertical spectral ratio (EHVR) database in Japan. The model aims to enhance the accuracy of predictions by incorporating the EHVRs for complementing site effects, and utilizing existing ground motion prediction equations (GMPE) as the base model for source and propagation path effects. The hybrid approach enables the prediction of peak ground accelerations (PGAs), peak ground velocities (PGVs), and 5% damped absolute acceleration response spectra (SAs). After classifying the training and test sets from the database, the trained DNN models were applied on the test set to evaluate the performance of the predicted results. The accuracy assessment by the residuals, R-squared ( R2), and root mean square error (RMSE) between the predicted and observed values in the test set revealed the superior performance of the proposed model compared with the traditional GMPE with proxy-based site effects such as V S30s especially in predicting both the spectral amplitude and shape of SAs.

Tuning parameters of deep neural network training algorithms pays off: a computational study

AbstractThe paper aims to investigate the impact of the optimization algorithms on the training of deep neural networks with an eye to the interaction between the optimizer and the generalization performance. In particular, we aim to analyze the behavior of state-of-the-art optimization algorithms in relationship to their hyperparameters setting to detect robustness with respect to the choice of a certain starting point in ending on different local solutions. We conduct extensive computational experiments using nine open-source optimization algorithms to train deep Convolutional Neural Network architectures on an image multi-class classification task. Precisely, we consider several architectures by changing the number of layers and neurons per layer, to evaluate the impact of different width and depth structures on the computational optimization performance. We show that the optimizers often return different local solutions and highlight the strong correlation between the quality of the solution found and the generalization capability of the trained network. We also discuss the role of hyperparameters tuning and show how a tuned hyperparameters setting can be re-used for the same task on different problems achieving better efficiency and generalization performance than a default setting.

Nondeterministic Features in Deep neural network design, training and inference

Neural networks are increasingly integral to scientific modeling and a wide range of real-world applications. However, standard neural networks often lack reliable certainty and confidence, and can be poorly calibrated, leading to nondeterministic results. To address these challenges, researchers have focused on understanding and quantifying uncertainty in neural network predictions. This work offers a comprehensive discussion of uncertainty estimation in neural networks, discussing recent advances, current challenges, and future research opportunities. This work explains key sources of uncertainty, challenges from model uncertainty and irreducible data uncertainty. We explore various methods for modeling these uncertainties, such as deterministic neural networks, Bayesian neural networks (BNNs), ensembles, and test-time data augmentation approaches. Additionally, the paper provides practical examples from fields like medical data analysis, robotics, and autonomous driving, illustrating the challenges and requirements associated with uncertainty in real-world applications. We also examine the limitations of current uncertainty quantification methods in safety-critical applications, and provide an outlook on future developments aimed at broader adoption of these methods in diverse domains. This paper serves as a valuable resource for both newcomers and those experienced in the field of uncertainty estimation in neural networks.

An approach of 2D convolutional neural network–based seismic data fault interpretation with linear annotation and pixel thinking

AbstractThis article introduces an improved method for geological fault interpretation utilizing a 2D convolutional neural network approach with a focus on coalbed horizons, dealing with the problem as an image classification task. At the beginning, linear annotations, reflecting geological fault features, are applied to multiple seismic sections. By considering texture differences between fault and non‐fault areas, we construct samples that represent these distinct zones for training deep neural networks. Initially, fault annotations are transformed into single dots to facilitate pixel‐based processing. To depict a specific dot's geological structure, we employ a matrix clipped around the point, determined by a combination of range and step parameters. Convolutional layers generate filters equivalent to seismic data transformation, streamlining the need for analysis and selection of seismic attributes. The article discusses enhancing the efficiency of 2D convolutional neural network–based fault interpretation by optimizing sample selection, data extraction and model construction procedures. Through the incorporation of data from two mining areas (totalling 27.09 km2) in sample creation, the overall accuracy exceeds 0.99. Recognition extends seamlessly to unlabelled sections, showcasing the innovative technical route and methodology of fault interpretation with linear annotation and pixel‐based thinking. This study presents a method that integrates planar and raster thinking, transitioning from vision‐oriented geological structure annotation to algorithm‐oriented pixel location. The proposed 2D convolutional neural network–based matrix‐oriented fault/non‐fault binary classification demonstrates feasibility and reproducibility, offering a new automated approach for fault detection in coalbeds through convolutional neural network algorithms.

Enhancing early Parkinson’s disease detection through multimodal deep learning and explainable AI: insights from the PPMI database

Parkinson’s is the second most common neurodegenerative disease, affecting nearly 8.5M people and steadily increasing. In this research, Multimodal Deep Learning is investigated for the Prodromal stage detection of Parkinson's Disease (PD), combining different 3D architectures with the novel Excitation Network (EN) and supported by Explainable Artificial Intelligence (XAI) techniques. Utilizing data from the Parkinson’s Progression Markers Initiative, this study introduces a joint co-learning approach for multimodal fusion, enabling end-to-end training of deep neural networks and facilitating the learning of complementary information from both imaging and clinical modalities. DenseNet with EN outperformed other models, showing a substantial increase in accuracy when supplemented with clinical data. XAI methods, such as Integrated Gradients for ResNet and DenseNet, and Attention Heatmaps for Vision Transformer (ViT), revealed that DenseNet focused on brain regions believed to be critical to prodromal pathophysiology, including the right temporal and left pre-frontal areas. Similarly, ViT highlighted the lateral ventricles associated with cognitive decline, indicating their potential in the Prodromal stage. These findings underscore the potential of these regions as early-stage PD biomarkers and showcase the proposed framework's efficacy in predicting subtypes of PD and aiding in early diagnosis, paving the way for innovative diagnostic tools and precision medicine.

MixTrain: accelerating DNN training via input mixing.

Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. An important factor contributing to the long training times is the increasing dataset complexity required to reach state-of-the-art performance in real-world applications. To address this challenge, we explore the use of input mixing, where multiple inputs are combined into a single composite input with an associated composite label for training. The goal is for training on the mixed input to achieve a similar effect as training separately on each the constituent inputs that it represents. This results in a lower number of inputs (or mini-batches) to be processed in each epoch, proportionally reducing training time. We find that naive input mixing leads to a considerable drop in learning performance and model accuracy due to interference between the forward/backward propagation of the mixed inputs. We propose two strategies to address this challenge and realize training speedups from input mixing with minimal impact on accuracy. First, we reduce the impact of inter-input interference by exploiting the spatial separation between the features of the constituent inputs in the network's intermediate representations. We also adaptively vary the mixing ratio of constituent inputs based on their loss in previous epochs. Second, we propose heuristics to automatically identify the subset of the training dataset that is subject to mixing in each epoch. Across ResNets of varying depth, MobileNetV2 and two Vision Transformer networks, we obtain upto 1.6 × and 1.8 × speedups in training for the ImageNet and Cifar10 datasets, respectively, on an Nvidia RTX 2080Ti GPU, with negligible loss in classification accuracy.

Improved region proposal network for enhanced few-shot object detection

Despite significant success of deep learning in object detection tasks, the standard training of deep neural networks requires access to a substantial quantity of annotated images across all classes. Data annotation is an arduous and time-consuming endeavor, particularly when dealing with infrequent objects. Few-shot object detection (FSOD) methods have emerged as a solution to the limitations of classic object detection approaches based on deep learning. FSOD methods demonstrate remarkable performance by achieving robust object detection using a significantly smaller amount of training data. A challenge for FSOD is that instances from novel classes that do not belong to the fixed set of training classes appear in the background and the base model may pick them up as potential objects. These objects behave similarly to label noise because they are classified as one of the training dataset classes, leading to FSOD performance degradation. We develop a semi-supervised algorithm to detect and then utilize these unlabeled novel objects as positive samples during the FSOD training stage to improve FSOD performance. Specifically, we develop a hierarchical ternary classification region proposal network (HTRPN) to localize the potential unlabeled novel objects and assign them new objectness labels to distinguish these objects from the base training dataset classes. Our improved hierarchical sampling strategy for the region proposal network (RPN) also boosts the perception ability of the object detection model for large objects. We test our approach and COCO and PASCAL VOC baselines that are commonly used in FSOD literature. Our experimental results indicate that our method is effective and outperforms the existing state-of-the-art (SOTA) FSOD methods. Our implementation is provided as a supplement to support reproducibility of the results https://github.com/zshanggu/HTRPN.11Early partial results of this work is presented in the 2023 ICCV Workshop on Visual Continual Learning (Shangguan and Rostami, 2023).

Synthetic ground motions in heterogeneous geologies from various sources: the HEMEWS-3D database

Abstract. The ever-improving performances of physics-based simulations and the rapid developments of deep learning are offering new perspectives to study earthquake-induced ground motion. Due to the large amount of data required to train deep neural networks, applications have so far been limited to recorded data or two-dimensional (2D) simulations. To bridge the gap between deep learning and high-fidelity numerical simulations, this work introduces a new database of physics-based earthquake simulations. The HEterogeneous Materials and Elastic Waves with Source variability in 3D (HEMEWS-3D) database comprises 30 000 simulations of elastic wave propagation in 3D geological domains. Each domain is parametrized by a different geological model built from a random arrangement of layers augmented by random fields that represent heterogeneities. Elastic waves originate from a randomly located pointwise source parametrized by a random moment tensor. For each simulation, ground motion is synthesized at the surface by a grid of virtual sensors. The high frequency of waveforms (fmax⁡=5 Hz) allows for extensive analyses of surface ground motion. Existing and foreseen applications range from statistical analyses of the ground motion variability and machine learning methods on geological models to deep-learning-based predictions of ground motion that depend on 3D heterogeneous geologies and source properties. Data are available at https://doi.org/10.57745/LAI6YU (Lehmann, 2023).