Training Deep Neural Networks Research Articles

Uncertainty-aware deep learning for monitoring and fault diagnosis from synthetic data

Deep neural networks (DNNs) are often coupled with physics-based and data-driven models to perform fault detection and health monitoring. The system models serve as digital surrogates that generate large quantities of data for training DNNs which would otherwise be difficult to obtain from the real-life system. In such a scenario, the uncertainty in the system model and in the DNN parameters will influence the predictions of the DNN. Here, we quantify the impact of this uncertainty on the performance of DNNs. The uncertainty from the system model is captured with two methods, namely assumed density filtering and heteroskedastic modelling. In addition to quantification, these methods allow training DNNs in an uncertainty-aware manner. The uncertainty in the DNN parameters is captured with Monte Carlo dropout. The proposed approach is demonstrated for fault diagnosis of electric power lines. Data generated from a physics-based model calibrated with real-life measurements is used to train three neural network architectures for fault diagnosis. The results reveal that uncertainty-aware models can provide 1% to 19% improvement in classification accuracy than their deterministic counterparts. The uncertainty-aware models also exhibit better robustness to uncertainty and, thus, offer more reliable models for deployment. Remarkably, the article provides a system-agnostic framework for uncertainty-aware training of DNN models for fault diagnosis and monitoring that explicitly accounts for the synthetic nature of training data.

Identifying and training deep learning neural networks on biomedical-related datasets.

This manuscript describes the development of a resources module that is part of a learning platform named 'NIGMS Sandbox for Cloud-based Learning' https://github.com/NIGMS/NIGMS-Sandbox. The overall genesis of the Sandbox is described in the editorial NIGMS Sandbox at the beginning of this Supplement. This module delivers learning materials on implementing deep learning algorithms for biomedical image data in an interactive format that uses appropriate cloud resources for data access and analyses. Biomedical-related datasets are widely used in both research and clinical settings, but the ability for professionally trained clinicians and researchers to interpret datasets becomes difficult as the size and breadth of these datasets increases. Artificial intelligence, and specifically deep learning neural networks, have recently become an important tool in novel biomedical research. However, use is limited due to their computational requirements and confusion regarding different neural network architectures. The goal of this learning module is to introduce types of deep learning neural networks and cover practices that are commonly used in biomedical research. This module is subdivided into four submodules that cover classification, augmentation, segmentation and regression. Each complementary submodule was written on the Google Cloud Platform and contains detailed code and explanations, as well as quizzes and challenges to facilitate user training. Overall, the goal of this learning module is to enable users to identify and integrate the correct type of neural network with their data while highlighting the ease-of-use of cloud computing for implementing neural networks. This manuscript describes the development of a resource module that is part of a learning platform named ``NIGMS Sandbox for Cloud-based Learning'' https://github.com/NIGMS/NIGMS-Sandbox. The overall genesis of the Sandbox is described in the editorial NIGMS Sandbox [1] at the beginning of this Supplement. This module delivers learning materials on the analysis of bulk and single-cell ATAC-seq data in an interactive format that uses appropriate cloud resources for data access and analyses.

Simulation-based inference on virtual brain models of disorders

Connectome-based models, also known as virtual brain models (VBMs), have been well established in network neuroscience to investigate pathophysiological causes underlying a large range of brain diseases. The integration of an individual’s brain imaging data in VBMs has improved patient-specific predictivity, although Bayesian estimation of spatially distributed parameters remains challenging even with state-of-the-art Monte Carlo sampling. VBMs imply latent nonlinear state space models driven by noise and network input, necessitating advanced probabilistic machine learning techniques for widely applicable Bayesian estimation. Here we present simulation-based inference on VBMs (SBI-VBMs), and demonstrate that training deep neural networks on both spatio-temporal and functional features allows for accurate estimation of generative parameters in brain disorders. The systematic use of brain stimulation provides an effective remedy for the non-identifiability issue in estimating the degradation limited to smaller subset of connections. By prioritizing model structure over data, we show that the hierarchical structure in SBI-VBMs renders the inference more effective, precise and biologically plausible. This approach could broadly advance precision medicine by enabling fast and reliable prediction of patient-specific brain disorders.

Artificial Neural Network-aided Mathematical Model for Predicting Soil Stress-strain Hysteresis Loop Evolution

This study presents a novel approach to forecasting the evolution of hysteresis stress-strain response of different types of soils under repeated loading-unloading cycles. The forecasting is made solely from the knowledge of soil properties and loading parameters. Our approach combines mathematical modeling, regression analysis, and Deep Neural Networks (DNNs) to overcome the limitations of traditional DNN training. As a novelty, we propose a hysteresis loop evolution equation and design a family of DNNs to determine the parameters of this equation. Knowing the nature of the phenomenon, we can impose certain solution types and narrow the range of values, enabling the use of a very simple and efficient DNN model. The experimental data used to develop and test the model was obtained through Torsional Shear (TS) tests on soil samples. The model demonstrated high accuracy, with an average R² value of 0.9788 for testing and 0.9944 for training.

Convergence of gradient descent for learning linear neural networks

We study the convergence properties of gradient descent for training deep linear neural networks, i.e., deep matrix factorizations, by extending a previous analysis for the related gradient flow. We show that under suitable conditions on the stepsizes gradient descent converges to a critical point of the loss function, i.e., the square loss in this article. Furthermore, we demonstrate that for almost all initializations gradient descent converges to a global minimum in the case of two layers. In the case of three or more layers, we show that gradient descent converges to a global minimum on the manifold matrices of some fixed rank, where the rank cannot be determined a priori.

Radiomics of pituitary adenoma using computer vision: a review.

Pituitary adenomas (PA) represent the most common type of sellar neoplasm. Extracting relevant information from radiological images is essential for decision support in addressing various objectives related to PA. Given the critical need for an accurate assessment of the natural progression of PA, computer vision (CV) and artificial intelligence (AI) play a pivotal role in automatically extracting features from radiological images. The field of "Radiomics" involves the extraction of high-dimensional features, often referred to as "Radiomic features," from digital radiological images. This survey offers an analysis of the current state of research in PA radiomics. Our work comprises a systematic review of 34 publications focused on PA radiomics and other automated information mining pertaining to PA through the analysis of radiological data using computer vision methods. We begin with a theoretical exploration essential for understanding the theoretical background of radionmics, encompassing traditional approaches from computer vision and machine learning, as well as the latest methodologies in deep radiomics utilizing deep learning (DL). Thirty-four research works under examination are comprehensively compared and evaluated. The overall results achieved in the analyzed papers are high, e.g., the best accuracy is up to 96% and the best achieved AUC is up to 0.99, which establishes optimism for the successful use of radiomic features. Methods based on deep learning seem to be the most promising for the future. In relation to this perspective DL methods, several challenges are remarkable: It is important to create high-quality and sufficiently extensive datasets necessary for training deep neural networks. Interpretability of deep radiomics is also a big open challenge. It is necessary to develop and verify methods that will explain to us how deep radiomic features reflect various physics-explainable aspects.

A stable and robust fault diagnosis method for bearing using lightweight batch normalization-free residual network

Abstract The conventional deep learning-based bearing fault diagnosis methods tend to utilize denoising modules to improve the fault diagnosis performance in noisy scenes. However, the addition of denoising modules will increase expensive computational costs, leading to a delayed acquisition of fault diagnosis results. This work proposed a lightweight batch normalization (BN)-free residual network without any denoising modules for bearing fault diagnosis which properly rescaled the weights in a standard initialization instead of BN to avoid the exploding gradient problem and vanishing gradient problem at the beginning of training for deep neural networks. Therefore, it prevents the undesirable properties caused by BN. Compared with other methods, the fault diagnosis performance of the proposed method can maintain a high level with different input sizes and batch sizes. Especially in noisy scenes, the testing accuracy of fault diagnosis on different bearing datasets can be improved by 13.54% and 7.74% using fewer parameters and floating point operations on different bearing datasets.

Fingerprint Identification Method Based on Convulsional Neural Networks

The article presents an advanced method of fingerprint identification based on convolutional neural network (CNN) technology. This work elaborately describes the development and implementation process of a specialized CNN architecture for detecting and verifying the authenticity of fingerprints. Utilizing the comprehensive Socofing dataset allowed for an in -depth analysis of the model’s ability to distinguish between genuine and fabricated fingerprints, where the model demonstrated impressive accuracy – up to 98.964%. Special attention is given to error analysis, including the false discovery and omission rates, pointing towards potential directions for further improvement. Besides highlighting the technical aspects and high identification accuracy, the article also addresses potential challenges and limitations that the method might encounter. This includes issues related to the imbalance and diversity of data in the Socofing set, as well as limitations associated with computational resources when training deep neural networks. Potential pathways for model optimization are discussed, particularly focusing on reducing the false omission rate, which could improve user experience in authentication. The concluding section of the article emphasizes the importance of the presented work for the security sector, where precise authentication of fingerprint images is critically needed. The obtained results can be considered a solid foundation for future scientific developments in this direction. Additionally, the need for systematic updates and modifications of the model is highlighted to adapt it to continually improved imitation techniques, ensuring its long-term relevance and effectiveness.

Strategic Geosteering Workflow with Uncertainty Quantification and Deep Learning: Initial Test on the Goliat Field Data

Continuous integration of real-time logging-while-drilling data into a subsurface model with relevant geological uncertainties enables strategic geosteering: a field-level optimization of the well-placement strategy. Model errors arising from oversimplified conceptual geological models and imperfect simulation of measurements result in unreliable subsurface-model updates. The model errors are particularly pronounced when synthetic measurements are approximated with a fast but imperfect model, such as a deep neural network (DNN).#xD;We present a practical data-assimilation workflow consisting of offline and online phases. The offline phase involves DNN training and building an uncertain prior near-well geo-model. The online phase utilizes the flexible iterative ensemble smoother (FlexIES) to perform real-time assimilation of extra-deep electromagnetic data while accounting for the model errors in the approximate DNN model. We demonstrate the proposed workflow on historic well-log data from the Goliat Field (Barents Sea). #xD;The median of our probabilistic estimation is on par with proprietary inversion, regardless of the number of layers in the chosen prior or the approximate DNN model. By estimating model errors, FlexIES automatically quantifies the uncertainty in the boundaries and resistivities of layers, which is not standard in proprietary inversion. #xD;This capability allows us to capture uncertainties more efficiently, thus providing input for future quantitative decision support methods. We demonstrate the potential of quantitive decision support by visually estimating the ahead-of-bit risk of reservoir exit that has occurred during the considered operation.

Image generation of hazardous situations in construction sites using text-to-image generative model for training deep neural networks

There has been a persistent challenge in acquiring sufficient training image data for deep neural networks (DNNs) to enhance safety monitoring on construction sites. Given the prevalence of textual data in the construction sector and the capabilities of multi-modal AI systems, this paper presents the use of text-to-image models to generate training images that capture the relationships between objects involved in construction accidents. Through a systematic prompt design process, a synthetic dataset of 3585 images across 27 hazardous scenarios was generated. The efficacy of this method is demonstrated by the performance of DNN models trained on these virtual images, which achieved a mean Average Precision (mAP) of approximately 64% in object detection and 60% in segmentation tasks. This paper demonstrates the potential of text-to-image models in mitigating the scarcity of training data and enhancing the capability of DNNs to identify potential hazards.

A Smart Visual Sensor for Smoke Detection Based on Deep Neural Networks.

The automatic detection of smoke by analyzing the video stream acquired by traditional surveillance cameras is becoming a more and more interesting problem for the scientific community thanks to the necessity to prevent fires at the very early stages. The adoption of a smart visual sensor, namely a computer vision algorithm running in real time, allows one to overcome the limitations of standard physical sensors. Nevertheless, this is a very challenging problem, due to the strong similarity of the smoke with other environmental elements like clouds, fog and dust. In addition to this challenge, data available for training deep neural networks is limited and not fully representative of real environments. Within this context, in this paper we propose a new method for smoke detection based on the combination of motion and appearance analysis with a modern convolutional neural network (CNN). Moreover, we propose a new dataset, called the MIVIA Smoke Detection Dataset (MIVIA-SDD), publicly available for research purposes; it consists of 129 videos covering about 28 h of recordings. The proposed hybrid method, trained and evaluated on the proposed dataset, demonstrated to be very effective by achieving a 94% smoke recognition rate and, at the same time, a substantially lower false positive rate if compared with fully deep learning-based approaches (14% vs. 100%). Therefore, the proposed combination of motion and appearance analysis with deep learning CNNs can be further investigated to improve the precision of fire detection approaches.

Prior-DualGAN: Rain rendering from coarse to fine

The success of deep neural networks (DNN) in deraining has led to increased research in rain rendering. In this paper, we introduce a novel Prior-DualGAN algorithm to synthesize diverse and realistic rainy/non-rainy image pairs to improve DNN training for deraining. More precisely, the rain streak prior is first generated using essential rain streak attributes; then more realistic and diverse rain streak patterns are rendered by the first generator; finally, the second generator naturally fuses the background and generated rain streaks to produce the final rainy images. Our method has two main advantages: (1) the rain streak prior enables the network to incorporate physical prior knowledge, accelerating network convergence; (2) our dual GAN approach gradually improves the naturalness and diversity of synthesized rainy images from rain streak synthesis to rainy image synthesis. We evaluate existing deraining algorithms using our generated rain-augmented datasets Rain100L, Rain14000, and Rain-Vehicle, verifying that training with our generated rain-augmented datasets significantly improves the deraining effect. The source code will be released shortly after article’s acceptance.

Resampling the optical transmission matrix to accelerate the training of the diffractive deep neural network

The diffractive deep neural network (D2NN) enables all-optical implementation of machine learning tasks. During the training, the Rayleigh–Sommerfeld (RS) diffraction integral is employed for connecting neurons between neighboring hidden layers. The RS formula can be rewritten as a transmission matrix (TM), which allows for the parallel computation of multiple vectorized light fields. However, the TM has a large size, demanding substantial computational resources, and resulting in long training time. In this paper, we propose to resample the TM in free space based on the propagation invariant modes (PIMs), thereby reducing the size of the matrix, and accelerating the propagation simulations. This method enables the training of the large-scale D2NN with reduced memory requirements and fast speed.

Neural network architecture search model for thermal radiation in dense particulate systems

In the CFD-DEM simulation of the many dense particulate systems, particle-scale thermal radiation is an important heat transfer mode under high temperatures. In this work, neural network architecture search model with different layer connection matrix is applied to the view factor regression of the thermal radiation in dense particulate systems. Deep neural networks with 3346 feasible architectures are evaluated by the HyperBand algorithm to find the local optimal solution. Neural architecture search model trained by the big data of the view factor gives a good prediction of the macroscopic radiative properties and it is in general agreement with the empirical correlations and experimental data. The particle–wall radiation decreases strongly with the distance and the maximum interaction depth is about 2.0 times the sphere diameter. The trained deep neural network model provides an efficient data-driven closure to discuss the thermal radiation of the particle–particle and particle–wall interactions in particle bed.

Enhancing deep neural network training efficiency and performance through linear prediction

Deep neural networks have achieved remarkable success in various fields. However, training an effective deep neural network still poses challenges. This paper aims to propose a method to optimize the training effectiveness of deep neural networks, with the goal of improving their performance. Firstly, based on the observation that parameters (weights and bias) of deep neural network change in certain rules during training process, the potential of parameters prediction for improving training efficiency is discovered. Secondly, the potential of parameters prediction to improve the performance of deep neural network by noise injection introduced by prediction errors is revealed. And then, considering the limitations comprehensively, a deep neural network Parameters Linear Prediction method is exploit. Finally, performance and hyperparameter sensitivity validations are carried out on some representative backbones. Experimental results show that by employing proposed Parameters Linear Prediction method, as opposed to SGD, has led to an approximate 1% increase in accuracy for optimal model, along with a reduction of about 0.01 in top-1/top-5 error. Moreover, it also exhibits stable performance under various hyperparameter settings, shown the effectiveness of the proposed method and validated its capacity in enhancing network’s training efficiency and performance.

Power Law in Deep Neural Networks: Sparse Network Generation and Continual Learning With Preferential Attachment.

Training deep neural networks (DNNs) typically requires massive computational power. Existing DNNs exhibit low time and storage efficiency due to the high degree of redundancy. In contrast to most existing DNNs, biological and social networks with vast numbers of connections are highly efficient and exhibit scale-free properties indicative of the power law distribution, which can be originated by preferential attachment in growing networks. In this work, we ask whether the topology of the best performing DNNs shows the power law similar to biological and social networks and how to use the power law topology to construct well-performing and compact DNNs. We first find that the connectivities of sparse DNNs can be modeled by truncated power law distribution, which is one of the variations of the power law. The comparison of different DNNs reveals that the best performing networks correlated highly with the power law distribution. We further model the preferential attachment in DNNs evolution and find that continual learning in networks with growth in tasks correlates with the process of preferential attachment. These identified power law dynamics in DNNs can lead to the construction of highly accurate and compact DNNs based on preferential attachment. Inspired by the discovered findings, two novel applications have been proposed, including evolving optimal DNNs in sparse network generation and continual learning tasks with efficient network growth using power law dynamics. Experimental results indicate that the proposed applications can speed up training, save storage, and learn with fewer samples than other well-established baselines. Our demonstration of preferential attachment and power law in well-performing DNNs offers insight into designing and constructing more efficient deep learning.

Physics-informed machine learning algorithms for forecasting sediment yield: an analysis of physical consistency, sensitivity, and interpretability.

The sediment transport, involving the movement of the bedload and suspended sediment in the basins, is a critical environmental concern that worsens water scarcity and leads to degradation of land and its ecosystems. Machine learning (ML) algorithms have emerged as powerful tools for predicting sediment yield. However, their use by decision-makers can be attributed to concerns regarding their consistency with the involved physical processes. In light of this issue, this study aims to develop a physics-informed ML approach for predicting sediment yield. To achieve this objective, Gaussian, Center, Regular, and Direct Copulas were employed to generate virtual combinations of physical of the sub-basins and hydrological datasets. These datasets were then utilized to train deep neural network (DNN), conventional neural network (CNN), Extra Tree, and XGBoost (XGB) models. The performance of these models was compared with the modified universal soil loss equation (MUSLE), which serves as a process-based model. The results demonstrated that the ML models outperformed the MUSLE model, exhibiting improvements in Nash-Sutcliffe efficiency (NSE) of approximately 10%, 18%, 32%, and 41% for the DNN, CNN, Extra Tree, and XGB models, respectively. Furthermore, through Sobol sensitivity and Shapley additive explanation-based interpretability analyses, it was revealed that the Extra Tree model displayed greater consistency with the physical processes underlying sediment transport as modeled by MUSLE. The proposed framework provides new insights into enhancing the accuracy and applicability of ML models in forecasting sediment yield while maintaining consistency with natural processes. Consequently, it can prove valuable in simulating process-related strategies aimed at mitigating sediment transport at watershed scales, such as the implementation of best management practices.

Clinical utility of artificial intelligence–augmented endobronchial ultrasound elastography in lymph node staging for lung cancer

ObjectiveEndobronchial ultrasound elastography produces a color map of mediastinal lymph nodes, with the color blue (level 60) indicating stiffness. Our pilot study demonstrated that predominantly blue lymph nodes, with a stiffness area ratio greater than 0.496, are likely malignant. This large-scale study aims to validate this stiffness area ratio compared with pathology. MethodsThis is a single-center prospective clinical trial where B-mode ultrasound and endobronchial ultrasound elastography lymph node images were collected from patients undergoing endobronchial ultrasound transbronchial needle aspiration for suspected or diagnosed non–small cell lung cancer. Images were fed to a trained deep neural network algorithm (NeuralSeg), which segmented the lymph nodes, identified the percent of lymph node area above the color blue threshold of level 60, and assigned a malignant label to lymph nodes with a stiffness area ratio above 0.496. Diagnostic statistics and receiver operating characteristic analyses were conducted. NeuralSeg predictions were compared with pathology. ResultsB-mode ultrasound and endobronchial ultrasound elastography lymph node images (n = 210) were collected from 124 enrolled patients. Only lymph nodes with conclusive pathology results (n = 187) were analyzed. NeuralSeg was able to predict 98 of 143 true negatives and 34 of 44 true positives, resulting in an overall accuracy of 70.59% (95% CI, 63.50-77.01), sensitivity of 43.04% (95% CI, 31.94-54.67), specificity of 90.74% (95% CI, 83.63-95.47), positive predictive value of 77.27% (95% CI, 64.13-86.60), negative predictive value of 68.53% (95% CI, 64.05-72.70), and area under the curve of 0.820 (95% CI, 0.758-0.883). ConclusionsNeuralSeg was able to predict nodal malignancy based on endobronchial ultrasound elastography lymph node images with high area under the receiver operating characteristic curve and specificity. This technology should be refined further by testing its validity and applicability through a larger dataset in a multicenter trial.

FedComm: Federated Learning as a Medium for Covert Communication

Proposed as a solution to mitigate the privacy implications related to the adoption of deep learning, Federated Learning (FL) enables large numbers of participants to successfully train deep neural networks without revealing the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">actual</i> private training data. To date, a substantial amount of research has investigated the security and privacy properties of FL, resulting in a plethora of innovative attack and defense strategies. This paper thoroughly investigates the communication capabilities of an FL scheme. In particular, we show that a party involved in the FL learning process can use FL as a covert communication medium to send an arbitrary message. We introduce FedComm, a novel covert-communication technique that enables robust sharing and transfer of targeted payloads within the FL framework. Our extensive theoretical and empirical evaluations show that FedComm provides a stealthy communication channel, with minimal disruptions to the training process. Our experiments show that FedComm successfully delivers 100% of a payload in the order of kilobits before the FL procedure converges. Our evaluation also shows that FedComm is independent of the application domain and the neural network architecture used by the underlying FL scheme.

An artificial intelligence approach for the estimation of conduction heat transfer using deep neural networks

PurposeThis study aims to use deep neural networks (DNNs) to learn the conduction heat transfer physics and estimate temperature distribution images in a physical domain without using any physical model or mathematical governing equation.Design/methodology/approachTwo novel DNNs capable of learning the conduction heat transfer physics were defined. The first DNN (U-Net autoencoder residual network [UARN]) was designed to extract local and global features simultaneously. In the second DNN, a conditional generative adversarial network (CGAN) was used to enhance the accuracy of UARN, which is referred to as CGUARN. Then, novel loss functions, introduced based on outlier errors, were used to train the DNNs.FindingsA UARN neural network could learn the physics of heat transfer. Within a few epochs, it reached mean and outlier errors that other DNNs could never reach after many epochs. The composite outlier-mean error as a loss function showed excellent performance in training DNNs for physical images. A UARN could excellently capture local and global features of conduction heat transfer, whereas the composite error could accurately guide DNN to extract high-level information by estimating temperature distribution images.Originality/valueThis study offers a unique approach to estimating physical information, moving from traditional mathematical and physical models to machine learning approaches. Developing novel DNNs and loss functions has shown promising results, opening up new avenues in heat transfer physics and potentially other fields.