Training Deep Neural Networks Research Articles

Image generation of hazardous situations in construction sites using text-to-image generative model for training deep neural networks

There has been a persistent challenge in acquiring sufficient training image data for deep neural networks (DNNs) to enhance safety monitoring on construction sites. Given the prevalence of textual data in the construction sector and the capabilities of multi-modal AI systems, this paper presents the use of text-to-image models to generate training images that capture the relationships between objects involved in construction accidents. Through a systematic prompt design process, a synthetic dataset of 3585 images across 27 hazardous scenarios was generated. The efficacy of this method is demonstrated by the performance of DNN models trained on these virtual images, which achieved a mean Average Precision (mAP) of approximately 64% in object detection and 60% in segmentation tasks. This paper demonstrates the potential of text-to-image models in mitigating the scarcity of training data and enhancing the capability of DNNs to identify potential hazards.

A Smart Visual Sensor for Smoke Detection Based on Deep Neural Networks.

The automatic detection of smoke by analyzing the video stream acquired by traditional surveillance cameras is becoming a more and more interesting problem for the scientific community thanks to the necessity to prevent fires at the very early stages. The adoption of a smart visual sensor, namely a computer vision algorithm running in real time, allows one to overcome the limitations of standard physical sensors. Nevertheless, this is a very challenging problem, due to the strong similarity of the smoke with other environmental elements like clouds, fog and dust. In addition to this challenge, data available for training deep neural networks is limited and not fully representative of real environments. Within this context, in this paper we propose a new method for smoke detection based on the combination of motion and appearance analysis with a modern convolutional neural network (CNN). Moreover, we propose a new dataset, called the MIVIA Smoke Detection Dataset (MIVIA-SDD), publicly available for research purposes; it consists of 129 videos covering about 28 h of recordings. The proposed hybrid method, trained and evaluated on the proposed dataset, demonstrated to be very effective by achieving a 94% smoke recognition rate and, at the same time, a substantially lower false positive rate if compared with fully deep learning-based approaches (14% vs. 100%). Therefore, the proposed combination of motion and appearance analysis with deep learning CNNs can be further investigated to improve the precision of fire detection approaches.

Prior-DualGAN: Rain rendering from coarse to fine

The success of deep neural networks (DNN) in deraining has led to increased research in rain rendering. In this paper, we introduce a novel Prior-DualGAN algorithm to synthesize diverse and realistic rainy/non-rainy image pairs to improve DNN training for deraining. More precisely, the rain streak prior is first generated using essential rain streak attributes; then more realistic and diverse rain streak patterns are rendered by the first generator; finally, the second generator naturally fuses the background and generated rain streaks to produce the final rainy images. Our method has two main advantages: (1) the rain streak prior enables the network to incorporate physical prior knowledge, accelerating network convergence; (2) our dual GAN approach gradually improves the naturalness and diversity of synthesized rainy images from rain streak synthesis to rainy image synthesis. We evaluate existing deraining algorithms using our generated rain-augmented datasets Rain100L, Rain14000, and Rain-Vehicle, verifying that training with our generated rain-augmented datasets significantly improves the deraining effect. The source code will be released shortly after article’s acceptance.

Resampling the optical transmission matrix to accelerate the training of the diffractive deep neural network

The diffractive deep neural network (D2NN) enables all-optical implementation of machine learning tasks. During the training, the Rayleigh–Sommerfeld (RS) diffraction integral is employed for connecting neurons between neighboring hidden layers. The RS formula can be rewritten as a transmission matrix (TM), which allows for the parallel computation of multiple vectorized light fields. However, the TM has a large size, demanding substantial computational resources, and resulting in long training time. In this paper, we propose to resample the TM in free space based on the propagation invariant modes (PIMs), thereby reducing the size of the matrix, and accelerating the propagation simulations. This method enables the training of the large-scale D2NN with reduced memory requirements and fast speed.

Neural network architecture search model for thermal radiation in dense particulate systems

In the CFD-DEM simulation of the many dense particulate systems, particle-scale thermal radiation is an important heat transfer mode under high temperatures. In this work, neural network architecture search model with different layer connection matrix is applied to the view factor regression of the thermal radiation in dense particulate systems. Deep neural networks with 3346 feasible architectures are evaluated by the HyperBand algorithm to find the local optimal solution. Neural architecture search model trained by the big data of the view factor gives a good prediction of the macroscopic radiative properties and it is in general agreement with the empirical correlations and experimental data. The particle–wall radiation decreases strongly with the distance and the maximum interaction depth is about 2.0 times the sphere diameter. The trained deep neural network model provides an efficient data-driven closure to discuss the thermal radiation of the particle–particle and particle–wall interactions in particle bed.

Enhancing deep neural network training efficiency and performance through linear prediction

Deep neural networks have achieved remarkable success in various fields. However, training an effective deep neural network still poses challenges. This paper aims to propose a method to optimize the training effectiveness of deep neural networks, with the goal of improving their performance. Firstly, based on the observation that parameters (weights and bias) of deep neural network change in certain rules during training process, the potential of parameters prediction for improving training efficiency is discovered. Secondly, the potential of parameters prediction to improve the performance of deep neural network by noise injection introduced by prediction errors is revealed. And then, considering the limitations comprehensively, a deep neural network Parameters Linear Prediction method is exploit. Finally, performance and hyperparameter sensitivity validations are carried out on some representative backbones. Experimental results show that by employing proposed Parameters Linear Prediction method, as opposed to SGD, has led to an approximate 1% increase in accuracy for optimal model, along with a reduction of about 0.01 in top-1/top-5 error. Moreover, it also exhibits stable performance under various hyperparameter settings, shown the effectiveness of the proposed method and validated its capacity in enhancing network’s training efficiency and performance.

Deep learning in distinguishing pulmonary nodules as benign and malignant.

Due to the high mortality of lung cancer, the aim was to find convolutional neural network models that can distinguish benign and malignant cases with high accuracy, which can help in early diagnosis with diagnostic imaging. Patients who underwent tomography in our clinic and who were found to have lung nodules were retrospectively screened between January 2015 and December 2020. The patients were divided into two groups: benign (n=68; 38 males, 30 females; mean age: 59±12.2 years; range, 27 to 81 years) and malignant (n=29; 19 males, 10 females; mean age: 65±10.4 years; range, 43 to 88 years). In addition, a control group (n=67; 38 males, 29 females; mean age: 56.9±14.1 years; range, 26 to 81 years) consisting of healthy patients with no pathology in their sections was formed. Deep neural networks were trained with 80% of the three-class dataset we created and tested with 20% of the data. After the training of deep neural networks, feature extraction was done for these networks. The features extracted from the dataset were classified by machine learning algorithms. Performance results were obtained using confusion matrix analysis. After training deep neural networks, the highest accuracy rate of 80% was achieved with the AlexNET model among the models used. In the second stage results, obtained after feature extraction and using the classifier, the highest accuracy rate was achieved with the support vector machine classifier in the VGG19 model with 93.5%. In addition, increases in accuracy were noted in all models with the use of the support vector machine classifier. Differentiation of benign and malignant lung nodules using deep learning models and feature extraction will provide important advantages for early diagnosis in radiology practice. The results obtained in our study support this view.

Power Law in Deep Neural Networks: Sparse Network Generation and Continual Learning With Preferential Attachment.

Training deep neural networks (DNNs) typically requires massive computational power. Existing DNNs exhibit low time and storage efficiency due to the high degree of redundancy. In contrast to most existing DNNs, biological and social networks with vast numbers of connections are highly efficient and exhibit scale-free properties indicative of the power law distribution, which can be originated by preferential attachment in growing networks. In this work, we ask whether the topology of the best performing DNNs shows the power law similar to biological and social networks and how to use the power law topology to construct well-performing and compact DNNs. We first find that the connectivities of sparse DNNs can be modeled by truncated power law distribution, which is one of the variations of the power law. The comparison of different DNNs reveals that the best performing networks correlated highly with the power law distribution. We further model the preferential attachment in DNNs evolution and find that continual learning in networks with growth in tasks correlates with the process of preferential attachment. These identified power law dynamics in DNNs can lead to the construction of highly accurate and compact DNNs based on preferential attachment. Inspired by the discovered findings, two novel applications have been proposed, including evolving optimal DNNs in sparse network generation and continual learning tasks with efficient network growth using power law dynamics. Experimental results indicate that the proposed applications can speed up training, save storage, and learn with fewer samples than other well-established baselines. Our demonstration of preferential attachment and power law in well-performing DNNs offers insight into designing and constructing more efficient deep learning.

Physics-informed machine learning algorithms for forecasting sediment yield: an analysis of physical consistency, sensitivity, and interpretability.

The sediment transport, involving the movement of the bedload and suspended sediment in the basins, is a critical environmental concern that worsens water scarcity and leads to degradation of land and its ecosystems. Machine learning (ML) algorithms have emerged as powerful tools for predicting sediment yield. However, their use by decision-makers can be attributed to concerns regarding their consistency with the involved physical processes. In light of this issue, this study aims to develop a physics-informed ML approach for predicting sediment yield. To achieve this objective, Gaussian, Center, Regular, and Direct Copulas were employed to generate virtual combinations of physical of the sub-basins and hydrological datasets. These datasets were then utilized to train deep neural network (DNN), conventional neural network (CNN), Extra Tree, and XGBoost (XGB) models. The performance of these models was compared with the modified universal soil loss equation (MUSLE), which serves as a process-based model. The results demonstrated that the ML models outperformed the MUSLE model, exhibiting improvements in Nash-Sutcliffe efficiency (NSE) of approximately 10%, 18%, 32%, and 41% for the DNN, CNN, Extra Tree, and XGB models, respectively. Furthermore, through Sobol sensitivity and Shapley additive explanation-based interpretability analyses, it was revealed that the Extra Tree model displayed greater consistency with the physical processes underlying sediment transport as modeled by MUSLE. The proposed framework provides new insights into enhancing the accuracy and applicability of ML models in forecasting sediment yield while maintaining consistency with natural processes. Consequently, it can prove valuable in simulating process-related strategies aimed at mitigating sediment transport at watershed scales, such as the implementation of best management practices.

Clinical utility of artificial intelligence–augmented endobronchial ultrasound elastography in lymph node staging for lung cancer

ObjectiveEndobronchial ultrasound elastography produces a color map of mediastinal lymph nodes, with the color blue (level 60) indicating stiffness. Our pilot study demonstrated that predominantly blue lymph nodes, with a stiffness area ratio greater than 0.496, are likely malignant. This large-scale study aims to validate this stiffness area ratio compared with pathology. MethodsThis is a single-center prospective clinical trial where B-mode ultrasound and endobronchial ultrasound elastography lymph node images were collected from patients undergoing endobronchial ultrasound transbronchial needle aspiration for suspected or diagnosed non–small cell lung cancer. Images were fed to a trained deep neural network algorithm (NeuralSeg), which segmented the lymph nodes, identified the percent of lymph node area above the color blue threshold of level 60, and assigned a malignant label to lymph nodes with a stiffness area ratio above 0.496. Diagnostic statistics and receiver operating characteristic analyses were conducted. NeuralSeg predictions were compared with pathology. ResultsB-mode ultrasound and endobronchial ultrasound elastography lymph node images (n = 210) were collected from 124 enrolled patients. Only lymph nodes with conclusive pathology results (n = 187) were analyzed. NeuralSeg was able to predict 98 of 143 true negatives and 34 of 44 true positives, resulting in an overall accuracy of 70.59% (95% CI, 63.50-77.01), sensitivity of 43.04% (95% CI, 31.94-54.67), specificity of 90.74% (95% CI, 83.63-95.47), positive predictive value of 77.27% (95% CI, 64.13-86.60), negative predictive value of 68.53% (95% CI, 64.05-72.70), and area under the curve of 0.820 (95% CI, 0.758-0.883). ConclusionsNeuralSeg was able to predict nodal malignancy based on endobronchial ultrasound elastography lymph node images with high area under the receiver operating characteristic curve and specificity. This technology should be refined further by testing its validity and applicability through a larger dataset in a multicenter trial.

FedComm: Federated Learning as a Medium for Covert Communication

Proposed as a solution to mitigate the privacy implications related to the adoption of deep learning, Federated Learning (FL) enables large numbers of participants to successfully train deep neural networks without revealing the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">actual</i> private training data. To date, a substantial amount of research has investigated the security and privacy properties of FL, resulting in a plethora of innovative attack and defense strategies. This paper thoroughly investigates the communication capabilities of an FL scheme. In particular, we show that a party involved in the FL learning process can use FL as a covert communication medium to send an arbitrary message. We introduce FedComm, a novel covert-communication technique that enables robust sharing and transfer of targeted payloads within the FL framework. Our extensive theoretical and empirical evaluations show that FedComm provides a stealthy communication channel, with minimal disruptions to the training process. Our experiments show that FedComm successfully delivers 100% of a payload in the order of kilobits before the FL procedure converges. Our evaluation also shows that FedComm is independent of the application domain and the neural network architecture used by the underlying FL scheme.

An artificial intelligence approach for the estimation of conduction heat transfer using deep neural networks

PurposeThis study aims to use deep neural networks (DNNs) to learn the conduction heat transfer physics and estimate temperature distribution images in a physical domain without using any physical model or mathematical governing equation.Design/methodology/approachTwo novel DNNs capable of learning the conduction heat transfer physics were defined. The first DNN (U-Net autoencoder residual network [UARN]) was designed to extract local and global features simultaneously. In the second DNN, a conditional generative adversarial network (CGAN) was used to enhance the accuracy of UARN, which is referred to as CGUARN. Then, novel loss functions, introduced based on outlier errors, were used to train the DNNs.FindingsA UARN neural network could learn the physics of heat transfer. Within a few epochs, it reached mean and outlier errors that other DNNs could never reach after many epochs. The composite outlier-mean error as a loss function showed excellent performance in training DNNs for physical images. A UARN could excellently capture local and global features of conduction heat transfer, whereas the composite error could accurately guide DNN to extract high-level information by estimating temperature distribution images.Originality/valueThis study offers a unique approach to estimating physical information, moving from traditional mathematical and physical models to machine learning approaches. Developing novel DNNs and loss functions has shown promising results, opening up new avenues in heat transfer physics and potentially other fields.

PackMASNet: An information integration approach for quality inspection in industry 5.0

Industry 5.0 (I5.0) is increasingly governed by Operation Technology and Information Technology. It hosts Deep Learning (DL) solutions as a premier tool for the informatization of Quality Inspection (QI) of Mass Produced Products (MPP) and Mass Customized Products (MCP) at the on-premise edge devices. Besides, these ever-evolving MCPs in I5.0 require newer DL models to informatize customized quality parameters, mandating significant storage and computing resources, that is challenging to achieve at resource-constraint edge platform. This work proposes PackMASNet, a continual learning-based information integration scheme that sustainably incorporates the QI information of multiple MCPs in a single Deep Neural Network (DNN) under the following constraints: (i) resource-limited on-premise edge devices, (ii) no access to prior product data, and (iii) negligible deterioration of quality information of frequently manufactured MPPs. A lightweight Cosine-similarity threshold mechanism is incorporated to leverage the benefits of knowledge transfer (cloning) while avoiding its adverse impact in multi-varied MCPs. The scheme proposes DNN performance, storage, computation, data, and time efficient solutions for industries implementing the edge computing paradigm on I5.0. The efficacy of the proposed scheme is validated for the predictive QI of real-world use case in injection molding. The scheme preserves the frequently used information of MPP by 99.07%, saves computation cost by 132 times, reduces the DNN training time by 20.45%, and improves overall performance by 43.60% over the state-of-the-art schemes.

A Short Note on Physics-Guided GAN to Learn Physical Models without Gradients

This study briefly describes the concept of guided training of deep neural networks (DNNs) to learn physically reasonable solutions. The proposed method does not need the gradients of the physical equations, although the conventional physics-informed models need the gradients. DNNs are widely used to predict phenomena in physics and mechanics. One of the issues with DNNs is that their output does not always satisfy physical equations. One approach to consider with physical equations is adding a residual of the equations into the loss function; this is called physics-informed neural network (PINN). One feature of PINNs is that the physical equations and corresponding residuals must be implemented as part of a neural network model. In addition, the residual does not always converge to a small value. The proposed model is a physics-guided generative adversarial network (PG-GAN) that uses a GAN architecture, in which physical equations are used to judge whether the neural network’s output is consistent with physics. The proposed method was applied to a simple problem to assess its potential usability.

Evaluation of activated sludge settling characteristics from microscopy images with deep convolutional neural networks and transfer learning

Timely assessment and prediction of changes in microbial compositions leading to activated sludge settling problems, such as filamentous bulking (FB), can reduce water resource recovery facilities (WRRFs) upsets, operational challenges, and negative environmental impacts. This study presents a computer vision approach to assess activated sludge-settling characteristics based on Microscopy Images (MIs). We utilize MIs to train deep convolutional neural networks (CNN) using transfer learning to investigate the morphological properties of flocs and filaments. The methodology was tested on the offline MI dataset collected over two years at a full-scale industrial WRRF in Belgium. Various CNN architectures were tested, including Inception v3, ResNet18, ResNet152, ConvNeXt-nano, and ConvNeXt-S. The sludge volume index (SVI) was used as the final prediction variable, but the method can be easily adjusted to predict any other settling metric of choice. The best-performing CNN, ConvNeXt-nano, could predict SVI values with MAE (37.51 ± 4.02), MTD (11.65 ± 1.94), MAPE (0.18 ± 0.02), and R2 (0.75 ± 0.05). The model was tested in real-life FB events, where it identified early indicators of bulking by predictive surges in SVI values. We used an explainable AI technique, Eigen-CAM, to discover key morphological indicators of sludge bulking transitions. The findings highlight the SVI multimodality issue, where SVI readings as a unidimensional metric could not capture delicate shifts from good to poor sludge settling, while the model detected these subtle changes. The key morphological attributes of threshold conditions leading to FB were identified, which can provide actionable insight for preemptive WRRF management.

Unet-boosted Classifier – Multi-Task Architecture for Small Datasets Applied to Brain MRI Classification

The problem of training deep neural networks on small samples is especially relevant for medical issues. The paper examines the impact of pixel-wise marking of significant objects in the image, over the true class label, on the quality of the classification. To achieve better classification results on small samples, we propose a multitasking architecture – Unet-boosted classifier (UBC), that is trained simultaneously to solve classification and semantic segmentation problems. As the exploratory dataset, MRI images of patients with benign glioma and glioblastoma taken from the BRaTS 2019 data set are used. One horizontal slice of the MRI image containing a glioma is considered as the input (a total of 380 frames in the training set), and the probability of glioblastoma – as the output. Resnet34 was chosen as the baseline, trained without augmentations with a loss function based on cross-entropy. As an alternative solution, UBC-resnet34 is used – the same resnet34, boosted by a decoder built on the U-Net principle and predicting the pixels with glioma. The smoothed Sorensen-Dice coefficient (DiceLoss) is used as a decoder loss function. Results on the test sample: accuracy for the baseline reached 0.71, for the proposed model – 0.81, and the Dice score – 0.77. Thus, a deep model can be well trained even on a small data set, using the proposed architecture, provided that marking of the affected tissues in the form of a semantic mask is provided.

Computing the Plankton-Oxygen Dynamics Model Using Deep Neural Networks in the Context of Climate Change

This study proposes a novel approach to computing the plankton-oxygen dynamics model using deep neural networks (DNNs) within the context of climate change. By leveraging advanced computational methods, particularly deep learning algorithms, we aim to enhance our understanding of how plankton populations and oxygen concentrations interact in response to changing environmental conditions. The integration of DNNs offers several advantages, including the ability to capture complex nonlinear relationships and patterns from large datasets, making them well-suited for modeling dynamic systems such as aquatic ecosystems. By training DNNs on observational data and environmental variables, we can develop predictive models that simulate the behavior of plankton-oxygen dynamics under different climate scenarios. This research builds upon existing studies in ecological modeling and deep learning techniques to advance our knowledge of plankton-oxygen dynamics and their implications for ecosystem resilience in the face of climate change. By computationally modeling these dynamics, we can gain valuable insights into the mechanisms driving ecosystem responses to environmental stressors and inform conservation efforts and policy decisions.

Domain Generalization via Ensemble Stacking for Face Presentation Attack Detection

Face presentation attack detection (PAD) plays a pivotal role in securing face recognition systems against spoofing attacks. Although great progress has been made in designing face PAD methods, developing a model that can generalize well to unseen test domains remains a significant challenge. Moreover, due to the different types of spoofing attacks, creating a dataset with a sufficient number of samples for training deep neural networks is a laborious task. This work proposes a comprehensive solution that combines synthetic data generation and deep ensemble learning to enhance the generalization capabilities of face PAD. Specifically, synthetic data is generated by blending a static image with spatiotemporal-encoded images using alpha composition and video distillation. In this way, we simulate motion blur with varying alpha values, thereby generating diverse subsets of synthetic data that contribute to a more enriched training set. Furthermore, multiple base models are trained on each subset of synthetic data using stacked ensemble learning. This allows the models to learn complementary features and representations from different synthetic subsets. The meta-features generated by the base models are used as input for a new model called the meta-model. The latter combines the predictions from the base models, leveraging their complementary information to better handle unseen target domains and enhance overall performance. Experimental results from seven datasets—WMCA, CASIA-SURF, OULU-NPU, CASIA-MFSD, Replay-Attack, MSU-MFSD, and SiW-Mv2—highlight the potential to enhance presentation attack detection by using large-scale synthetic data and a stacking-based ensemble approach.

Explaining neural scaling laws

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origin and relationships between scaling exponents.

Iteration and stochastic first-order oracle complexities of stochastic gradient descent using constant and decaying learning rates

The performance of stochastic gradient descent (SGD), which is the simplest first-order optimizer for training deep neural networks, depends on not only the learning rate but also the batch size. They both affect the number of iterations and the stochastic first-order oracle (SFO) complexity needed for training. In particular, the previous numerical results indicated that, for SGD using a constant learning rate, the number of iterations needed for training decreases when the batch size increases, and the SFO complexity needed for training is minimized at a critical batch size and that it increases once the batch size exceeds that size. Here, we study the relationship between batch size and the iteration and SFO complexities needed for nonconvex optimization in deep learning with SGD using constant or decaying learning rates and show that SGD using the critical batch size minimizes the SFO complexity. We also provide numerical comparisons of SGD with the existing first-order optimizers and show the usefulness of SGD using a critical batch size. Moreover, we show that measured critical batch sizes are close to the sizes estimated from our theoretical results.