Training Deep Learning Networks Research Articles

Using acoustic emission technique for structural health monitoring of laminate composite: A novel CNN-LSTM framework

This research has developed a real-time end-to-end deep learning model for structural health monitoring (SHM) method for composite impact damage diagnosis based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM). The acoustic emission (AE) signals collected under low-velocity impacts by means of piezoelectric sensors on composite materials are used for training deep learning networks. Based on the impact load curves, specimens are categorized into minor failure, intermediate failure, and severe failure. The convolved signals are segmented and reconstructed at a given length for the following LSTM module. The average accuracies for basic CNN, CNN– Recurrent Neural Network (RNN), CNN-LSTM, and CNN– Gated Recurrent Unit (GRU) are respectively 88.7 %, 92.6 %, 98 %, and 95.4 %. A sensitivity analysis on sub-signal length was conducted on the CNN-LSTM model, revealing that the model achieved its best performance when the sub-signal length was set at 16. The model attained prediction accuracies of 97.4 %, 100 %, and 100 %, respectively, for minor failure, intermediate failure, and severe failure cases.

Unsupervised speckle noise reduction technique for clinical ultrasound imaging.

Deep learning-based image enhancement has significant potential in the field of ultrasound image processing, as it can accurately model complicated nonlinear artifacts and noise, such as ultrasonic speckle patterns. However, training deep learning networks to acquire reference images that are clean and free of noise presents significant challenges. This study introduces an unsupervised deep learning framework, termed speckle-to-speckle (S2S), designed for speckle and noise suppression. This framework can complete its training without the need for clean (speckle-free) reference images. The proposed network leverages statistical reasoning for the mutual training of two in vivo images, each with distinct speckle patterns and noise. It then infers speckle- and noise-free images without needing clean reference images. This approach significantly reduces the time, cost, and effort experts need to invest in annotating reference images manually. The experimental results demonstrated that the proposed approach outperformed existing techniques in terms of the signal-to-noise ratio, contrast-to-noise ratio, structural similarity index, edge preservation index, and processing time (up to 86 times faster). It also performed excellently on images obtained from ultrasound scanners other than the ones used in this work. S2S demonstrates the potential of employing an unsupervised learning-based technique in medical imaging applications, where acquiring a ground truth reference is challenging.

A novel multi-scale violence and public gathering dataset for crowd behavior classification

Dependable utilization of computer vision applications, such as smart surveillance, requires training deep learning networks on datasets that sufficiently represent the classes of interest. However, the bottleneck in many computer vision applications lies in the limited availability of adequate datasets. One particular application that is of great importance for the safety of cities and crowded areas is smart surveillance. Conventional surveillance methods are reactive and often ineffective in enable real-time action. However, smart surveillance is a key component of smart and proactive security in a smart city. Motivated by a smart city application which aims at the automatic identification of concerning events for alerting law-enforcement and governmental agencies, we craft a large video dataset that focuses on the distinction between small-scale violence, large-scale violence, peaceful gatherings, and natural events. This dataset classifies public events along two axes, the size of the crowd observed and the level of perceived violence in the crowd. We name this newly-built dataset the Multi-Scale Violence and Public Gathering (MSV-PG) dataset. The videos in the dataset go through several pre-processing steps to prepare them to be fed into a deep learning architecture. We conduct several experiments on the MSV-PG datasets using a ResNet3D, a Swin Transformer and an R(2 + 1)D architecture. The results achieved by these models when trained on the MSV-PG dataset, 88.37%, 89.76%, and 89.3%, respectively, indicate that the dataset is well-labeled and is rich enough to train deep learning models for automatic smart surveillance for diverse scenarios.

Chest CT-based automated vertebral fracture assessment using artificial intelligence and morphologic features.

Spinal degeneration and vertebral compression fractures are common among the elderly that adversely affect their mobility, quality of life, lung function, and mortality. Assessment of vertebral fractures in chronic obstructive pulmonary disease (COPD) is important due to the high prevalence of osteoporosis and associated vertebral fractures in COPD. We present new automated methods for (1) segmentation and labelling of individual vertebrae in chest computed tomography (CT) images using deep learning (DL), multi-parametric freeze-and-grow (FG) algorithm, and separation of apparently fused vertebrae using intensity autocorrelation and (2) vertebral deformity fracture detection using computed vertebral height features and parametric computational modelling of an established protocol outlined for trained human experts. A chest CT-based automated method was developed for quantitative deformity fracture assessment following the protocol by Genant etal. The computational method was accomplished in the following steps: (1) computation of a voxel-level vertebral body likelihood map from chest CT using a trained DL network; (2) delineation and labelling of individual vertebrae on the likelihood map using an iterative multi-parametric FG algorithm; (3) separation of apparently fused vertebrae in CT using intensity autocorrelation; (4) computation of vertebral heights using contour analysis on the central anterior-posterior (AP) plane of a vertebral body; (5) assessment of vertebral fracture status using ratio functions of vertebral heights and optimized thresholds. The method was applied to inspiratory or total lung capacity (TLC) chest scans from the multi-site Genetic Epidemiology of COPD (COPDGene) (ClinicalTrials.gov: NCT00608764) study, and the performance was examined (n=3231). One hundred and twenty scans randomly selected from this dataset were partitioned into training (n=80) and validation (n=40) datasets for the DL-based vertebral body classifier. Also, generalizability of the method to low dose CT imaging (n=236) was evaluated. The vertebral segmentation module achieved a Dice score of.984 as compared to manual outlining results as reference (n=100); the segmentation performance was consistent across images with the minimum and maximum of Dice scores among images being.980 and.989, respectively. The vertebral labelling module achieved 100% accuracy (n=100). For low dose CT, the segmentation module produced image-level minimum and maximum Dice scores of.995 and.999, respectively, as compared to standard dose CT as the reference; vertebral labelling at low dose CT was fully consistent with standard dose CT (n=236). The fracture assessment method achieved overall accuracy, sensitivity, and specificity of 98.3%, 94.8%, and 98.5%, respectively, for 40,050 vertebrae from 3231 COPDGene participants. For generalizability experiments, fracture assessment from low dose CT was consistent with the reference standard dose CT results across all participants. Our CT-based automated method for vertebral fracture assessment is accurate, and it offers a feasible alternative to manual expert reading, especially for large population-based studies, where automation is important for high efficiency. Generalizability of the method to low dose CT imaging further extends the scope of application of the method, particularly since the usage of low dose CT imaging in large population-based studies has increased to reduce cumulative radiation exposure.

Evaluation of a deep image-to-image network (DI2IN) auto-segmentation algorithm across a network of cancer centers.

Due to manual OAR contouring challenges, various automatic contouring solutions have been introduced. Historically, common clinical auto-segmentation algorithms used were atlas-based, which required maintaining a library of self-made contours. Searching the collection was computationally intensive and could take several minutes to complete. Deep learning approaches have shown significant benefits compared to atlas-based methods in improving segmentation accuracy and efficiency in auto-segmentation algorithms. This work represents the first multi-institutional study to describe and evaluate an AI algorithm for the auto-segmentation of organs at risk (OARs) based on a deep image-to-image network (DI2IN). The AI-Rad Companion Organs RT (AIRC) algorithm (Siemens Healthineers, Erlangen, Germany) uses a two-step approach for segmentation. In the first step, the target organ region in the optimal input image is extracted using a trained deep reinforcement learning network (DRL), which is then used as input to create the contours in the second step based on DI2IN. The study was initially designed as a prospective single-center evaluation. The automated contours generated by AIRC were evaluated by three experienced board-certified radiation oncologists using a four-point scale where 4 is clinically usable and 1 requires re-contouring. After seeing favorable results in a single-center pilot study, we decided to expand the study to six additional institutions, encompassing eight additional evaluators for a total of 11 physician evaluators across seven institutions. One hundred and fifty-six patients and 1366 contours were prospectively evaluated. The five most commonly contoured organs were the lung (136 contours, average rating = 4.0), spinal cord (106 contours, average rating = 3.1), eye globe (80 contours, average rating = 3.9), lens (77 contours, average rating = 3.9), and optic nerve (75 contours, average rating = 4.0). The average rating per evaluator per contour was 3.6. On average, 124 contours were evaluated by each evaluator. 65% of the contours were rated as 4, and 31% were rated as 3. Only 4% of contours were rated as 1 or 2. Thirty-three organs were evaluated in the study, with 19 structures having a 3.5 or above average rating (ribs, abdominopelvic cavity, skeleton, larynx, lung, aorta, brachial plexus, lens, eye globe, glottis, heart, parotid glands, bladder, kidneys, supraglottic larynx, submandibular glands, esophagus, optic nerve, oral cavity) and the remaining organs having a rating of 3.0 or greater (female breast, proximal femur, seminal vesicles, rectum, sternum, brainstem, prostate, brain, lips, mandible, liver, optic chiasm, spinal cord, spleen). No organ had an average rating below 3. AIRC performed well with greater than 95% of contours accepted by treating physicians with no or minor edits. It supported a fully automated workflow with the potential for time savings and increased standardization with the use of AI-powered algorithms for high-quality OAR contouring.

Comparing multi-image and image augmentation strategies for deep learning-based prostate segmentation

During MR-Linac-based adaptive radiotherapy, multiple images are acquired per patient. These can be applied in training deep learning networks to reduce annotation efforts. This study examined the advantage of using multiple versus single images for prostate treatment segmentation. Findings indicate minimal improvement in DICE and Hausdorff 95% metrics with multiple images. Maximum difference was seen for the rectum in the low data regime, training with images from five patients. Utilizing a 2D U-net resulted in DICE values of 0.80/0.83 when including 1/5 images per patient, respectively. Including more patients in training reduced the difference. Standard augmentation methods remained more effective.

Training Universal Deep-Learning Networks for Electromagnetic Medical Imaging Using a Large Database of Randomized Objects.

Deep learning has become a powerful tool for solving inverse problems in electromagnetic medical imaging. However, contemporary deep-learning-based approaches are susceptible to inaccuracies stemming from inadequate training datasets, primarily consisting of signals generated from simplified and homogeneous imaging scenarios. This paper introduces a novel methodology to construct an expansive and diverse database encompassing domains featuring randomly shaped structures with electrical properties representative of healthy and abnormal tissues. The core objective of this database is to enable the training of universal deep-learning techniques for permittivity profile reconstruction in complex electromagnetic medical imaging domains. The constructed database contains 25,000 unique objects created by superimposing from 6 to 24 randomly sized ellipses and polygons with varying electrical attributes. Introducing randomness in the database enhances training, allowing the neural network to achieve universality while reducing the risk of overfitting. The representative signals in the database are generated using an array of antennas that irradiate the imaging domain and capture scattered signals. A custom-designed U-net is trained by using those signals to generate the permittivity profile of the defined imaging domain. To assess the database and confirm the universality of the trained network, three distinct testing datasets with diverse objects are imaged using the designed U-net. Quantitative assessments of the generated images show promising results, with structural similarity scores consistently exceeding 0.84, normalized root mean square errors remaining below 14%, and peak signal-to-noise ratios exceeding 33 dB. These results demonstrate the practicality of the constructed database for training deep learning networks that have generalization capabilities in solving inverse problems in medical imaging without the need for additional physical assistant algorithms.

DGU-HAU: A Dataset for 3D Human Action Analysis on Utterances

Constructing diverse and complex multi-modal datasets is crucial for advancing human action analysis research, providing ground truth annotations for training deep learning networks, and enabling the development of robust models across real-world scenarios. Generating natural and contextually appropriate nonverbal gestures is essential for enhancing immersive and effective human–computer interactions in various applications. These applications include video games, embodied virtual assistants, and conversations within a metaverse. However, existing speech-related human datasets are focused on style transfer, so they have limitations that make them unsuitable for 3D human action analysis studies, such as human action recognition and generation. Therefore, we introduce a novel multi-modal dataset, DGU-HAU, a dataset for 3D human action on utterances that commonly occurs during daily life. We validate the dataset using a human action generation model, Action2Motion (A2M), a state-of-the-art 3D human action generation model.

NetShield: An in-network architecture against byzantine failures in distributed deep learning

There is a growing trend of training deep learning networks on distributed clusters. Unfortunately, distributed deep learning (DDL) is prone to Byzantine failures where some nodes corrupt training by sending malicious gradients to the parameter server (PS). Existing works address this problem by implementing Byzantine defenses on the PS. However, Byzantine defenses come with large computational overhead, seriously affecting the DDL’s training performance. Moreover, malicious gradients are not identified until they are transmitted to the endpoint (PS), which leads to a waste of network resources and a decrease in communication efficiency.In this paper, we propose NetShield, a high-performance and universal in-network architecture against Byzantine failures in DDL. By exploiting the computing power of programmable switches, various Byzantine defenses are accelerated and the communication efficiency is increased. Based on NetShield, we improve two popular Byzantine-robust methods (the median-based method and the trimmed-mean-based method) and construct two switch-friendly defenders. We implement NetShield on a real testbed which includes a programmable switch. In comparison with traditional Byzantine defense deployments, NetShield reduces the total processing time by more than 2 times. The computing time is decreased by 2 ∼ 5 times and the traffic volume is reduced by half. Besides, NetShield shows a good universality by deploying two designed defenders flexibly.

Uncertainty-aware and dynamically-mixed pseudo-labels for semi-supervised defect segmentation

Deep learning-based defect segmentation is one of the important tasks of machine vision in automated inspection. Supervised learning methods have recently achieved remarkable performance on this task. However, the effectiveness of the supervised methods is limited by the scarcity and high cost of pixel-level annotation of training data. Semi-supervised learning methods have been proposed for training deep learning networks using a limited amount of labeled data along with additional unlabeled data for image segmentation. Most of these methods are based on consistency regularization and pseudo labeling, where the predictions on unlabeled samples often come with noise and are unreliable, resulting in poor segmentation performance. To alleviate this problem, we propose uncertainty-aware pseudo labels, which are generated from dynamically mixed predictions of multiple decoders that leverage a shared encoder network. The estimated uncertainty guides the pseudo-label-based supervision and regularizes the training when using the unlabeled samples. In our experiments on four public datasets for defect segmentation, the proposed method outperformed the fully supervised baseline and six state-of-the-art semi-supervised segmentation methods. We also conducted an extensive ablation study to demonstrate the effectiveness of our approach in various settings. The implementation code for this work is available at https://github.com/djene-mengistu/UAPS.

Prediction of Ovarian Cancer Response to Therapy Based on Deep Learning Analysis of Histopathology Images.

Ovarian cancer remains the leading gynecological cause of cancer mortality. Predicting the sensitivity of ovarian cancer to chemotherapy at the time of pathological diagnosis is a goal of precision medicine research that we have addressed in this study using a novel deep-learning neural network framework to analyze the histopathological images. We have developed a method based on the Inception V3 deep learning algorithm that complements other methods for predicting response to standard platinum-based therapy of the disease. For the study, we used histopathological H&E images (pre-treatment) of high-grade serous carcinoma from The Cancer Genome Atlas (TCGA) Genomic Data Commons portal to train the Inception V3 convolutional neural network system to predict whether cancers had independently been labeled as sensitive or resistant to subsequent platinum-based chemotherapy. The trained model was then tested using data from patients left out of the training process. We used receiver operating characteristic (ROC) and confusion matrix analyses to evaluate model performance and Kaplan-Meier survival analysis to correlate the predicted probability of resistance with patient outcome. Finally, occlusion sensitivity analysis was piloted as a start toward correlating histopathological features with a response. The study dataset consisted of 248 patients with stage 2 to 4 serous ovarian cancer. For a held-out test set of forty patients, the trained deep learning network model distinguished sensitive from resistant cancers with an area under the curve (AUC) of 0.846 ± 0.009 (SE). The probability of resistance calculated from the deep-learning network was also significantly correlated with patient survival and progression-free survival. In confusion matrix analysis, the network classifier achieved an overall predictive accuracy of 85% with a sensitivity of 73% and specificity of 90% for this cohort based on the Youden-J cut-off. Stage, grade, and patient age were not statistically significant for this cohort size. Occlusion sensitivity analysis suggested histopathological features learned by the network that may be associated with sensitivity or resistance to the chemotherapy, but multiple marker studies will be necessary to follow up on those preliminary results. This type of analysis has the potential, if further developed, to improve the prediction of response to therapy of high-grade serous ovarian cancer and perhaps be useful as a factor in deciding between platinum-based and other therapies. More broadly, it may increase our understanding of the histopathological variables that predict response and may be adaptable to other cancer types and imaging modalities.

Improving measurement of blood-brain barrier permeability with reduced scan time using deep-learning-derived capillary input function

PurposeIn Dynamic contrast-enhanced MRI (DCE-MRI), Arterial Input Function (AIF) has been shown to be a significant contributor to uncertainty in the estimation of kinetic parameters. This study is to assess the feasibility of using a deep learning network to estimate local Capillary Input Function (CIF) to estimate blood-brain barrier (BBB) permeability, while reducing the required scan time. Materials and MethodA total of 13 healthy subjects (younger (<40 y/o): 8, older (> 67 y/o): 5) were recruited and underwent 25-min DCE-MRI scans. The 25 min data were retrospectively truncated to 10 min to simulate a reduced scan time of 10 min. A deep learning network was trained to predict the CIF using simulated tissue contrast dynamics with two vascular transport models. The BBB permeability (PS) was measured using 3 methods: (i) Ca-25min, using DCE-MRI data of 25 min with individually sampled AIF (Ca); (ii) Ca-10min, using truncated 10min data with AIF (Ca); and (iii) Cp-10min, using truncated 10 min data with CIF (Cp). The PS estimates from the Ca-25min method were used as reference standard values to assess the accuracy of the Ca-10min and Cp-10min methods in estimating the PS values. ResultsWhen compared to the reference method(Ca-25min), the Ca-10min and Cp-10min methods resulted in an overestimation of PS by 217 ± 241 % and 48.0 ± 30.2 %, respectively. The Bland Altman analysis showed that the mean difference from the reference was 8.85 ± 1.78 (x10−4 min−1) with the Ca-10min, while it was reduced to 1.63 ± 2.25 (x10−4 min−1) with the Cp-10min, resulting in an average reduction of 81%. The limits of agreement also reduced by up to 39.2% with the Cp-10min. We found a 75% increase of BBB permeability in the gray matter and a 35% increase in the white matter, when comparing the older group to the younger group. ConclusionsWe demonstrated the feasibility of estimating the capillary-level input functions using a deep learning network. We also showed that this method can be used to estimate subtle age-related changes in BBB permeability with reduced scan time, without compromising accuracy. Moreover, the trained deep learning network can automatically select CIF, reducing the potential uncertainty resulting from manual user-intervention.

Deep metric learning framework combined with Gramian angular difference field image generation for Raman spectra classification based on a handheld Raman spectrometer

Rapid identification of unknown material samples using portable or handheld Raman spectroscopy detection equipment is becoming a common analytical tool. However, the design and implementation of a set of Raman spectroscopy-based devices for substance identification must include spectral sampling of standard reference substance samples, resolution matching between different devices, and the training process of the corresponding classification models. The process of selecting a suitable classification model is frequently time-consuming, and when the number of classes of substances to be recognised increases dramatically, recognition accuracy decreases dramatically. In this paper, we propose a fast classification method for Raman spectra based on deep metric learning networks combined with the Gramian angular difference field (GADF) image generation approach. First, we uniformly convert Raman spectra acquired at different resolutions into GADF images of the same resolution, addressing spectral dimension disparities induced by resolution differences in different Raman spectroscopy detection devices. Second, a network capable of implementing nonlinear distance measurements between GADF images of different classes of substances is designed based on a deep metric learning approach. The Raman spectra of 450 different mineral classes obtained from the RRUFF database were converted into GADF images and used to train this deep metric learning network. Finally, the trained network can be installed on an embedded computing platform and used in conjunction with portable or handheld Raman spectroscopic detection sensors to perform material identification tasks at various scales. A series of experiments demonstrate that our trained deep metric learning network outperforms existing mainstream machine learning models on classification tasks of different sizes. For the two tasks of Raman spectral classification of natural minerals of 260 classes and Raman spectral classification of pathogenic bacteria of 8 classes with significant noise, our suggested model achieved 98.05% and 90.13% classification accuracy, respectively. Finally, we also deployed the model in a handheld Raman spectrometer and conducted identification experiments on 350 samples of chemical substances attributed to 32 classes, achieving a classification accuracy of 99.14%. These results demonstrate that our method can greatly improve the efficiency of developing Raman spectroscopy-based substance detection devices and can be widely used in tasks of unknown substance identification.

Label fusion and training methods for reliable representation of inter-rater uncertainty

Medical tasks are prone to inter-rater variability due to multiple factors such as image quality, professional experience and training, or guideline clarity. Training deep learning networks with annotations from multiple raters is a common practice that mitigates the model’s bias towards a single expert. Reliable models generating calibrated outputs and reflecting the inter-rater disagreement are key to the integration of artificial intelligence in clinical practice. Various methods exist to take into account different expert labels. We focus on comparing three label fusion methods: STAPLE, average of the rater’s segmentation, and random sampling of each rater’s segmentation during training. Each label fusion method is studied using both the conventional training framework and the recently published SoftSeg framework that limits information loss by treating the segmentation task as a regression. Our results, across 10 data splittings on two public datasets (spinal cord gray matter challenge, and multiple sclerosis brain lesion segmentation), indicate that SoftSeg models, regardless of the ground truth fusion method, had better calibration and preservation of the inter-rater rater variability compared with their conventional counterparts without impacting the segmentation performance. Conventional models, i.e., trained with a Dice loss, with binary inputs, and sigmoid/softmax final activate, were overconfident and underestimated the uncertainty associated with inter-rater variability. Conversely, fusing labels by averaging with the SoftSeg framework led to underconfident outputs and overestimation of the rater disagreement. In terms of segmentation performance, the best label fusion method was different for the two datasets studied, indicating this parameter might be task-dependent. However, SoftSeg had segmentation performance systematically superior or equal to the conventionally trained models and had the best calibration and preservation of the inter-rater variability. SoftSeg has a low computational cost and performed similarly in terms of uncertainty to ensembles which require multiple models and forward passes. Our code is available at &lt;href='https://ivadomed.org'&gt;https://ivadomed.org&lt;/a&gt;.

Multiscale Attention Network Guided With Change Gradient Image for Land Cover Change Detection Using Remote Sensing Images

Learning performance is unsatisfactory when training deep-learning networks without prior-knowledge guidance. In this paper, a multi-scale change detection neural network guided by a change gradient image (CGI) was proposed. First, a multi-scale information attentional module was embedded in the backbone of UNet to achieve a multi-scale information fusion task of bi-temporal images. Second, the position channel attention module was promoted to make the neural network pay more attention to the spectral and spatial information in the multi-scale fused feature map. Finally, a change gradient guide module was proposed to optimize backpropagation and overcome the negative effects of pseudo-change. Compared with seven state-of-the-art methods using three pairs of real remote sensing images, the proposed approach could smoothen the salt-and-pepper noise from the detection maps and improve the detection accuracy. The quantitative improvements are about 1.67% and 3.00% in terms of overall accuracy and Kappa coefficient, respectively, thus confirming the feasibility and superiority of the proposed approach for detecting land cover change with remotely sensed images. Code: https://github.com/ImgSciGroup/MACGGNet.git.

XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments

Memory efficiency is crucial in training deep learning networks on resource-restricted devices. During backpropagation, forward tensors are used to calculate gradients. Despite the option of keeping those dependencies in memory until they are reused in backpropagation, some forward tensors can be discarded and recomputed later from saved tensors, so-called checkpoints . This allows, in particular, for resource-constrained heterogeneous environments to make use of all available compute devices. Unfortunately, the definition of these checkpoints is a non-trivial problem and poses a challenge to the programmer—improper or excessive recomputations negate the benefit of checkpointing. In this article, we present XEngine, an approach that schedules network operators to heterogeneous devices in low memory environments by determining checkpoints and recomputations of tensors. Our approach selects suitable resources per timestep and operator and optimizes the end-to-end time for neural networks taking the memory limitation of each device into account. For this, we formulate a mixed-integer quadratic program (MIQP) to schedule operators of deep learning networks on heterogeneous systems. We compare our MIQP solver XEngine against Checkmate [ 12 ], a mixed-integer linear programming (MILP) approach that solves recomputation on a single device. Our solver finds solutions that are up to 22.5% faster than the fastest Checkmate schedule in which the network is computed exclusively on a single device. We also find valid schedules for networks making use of both central processing units and graphics processing units if memory limitations do not allow scheduling exclusively to the graphics processing unit.

Enhancing resolution of micro-CT images of reservoir rocks using super resolution

Current hardware configuration of micro-CT detectors puts a lower limit on voxel size that can be acquired while maintaining a sufficiently large field of view. This limits the degree to which rock pores can be resolved in a micro-CT image and thus restricting the application envelope of Digital Rock technology. Super resolution techniques can refine voxel size while maintaining a sufficiently large field of view using pairs of low- and high-resolution images for training. However, for interpretation of quality of Digital Rock results, image quality is not determined by voxel size alone but by the degree to which a feature such as pore throat is resolved, which depends on both the physical size of the feature and voxel size. Furthermore, artificially down-sampling finer voxel size images to obtain images of coarser voxel size for training deep learning networks is not sufficient to capture the mapping between images acquired at different resolutions. This is especially true for reservoir rocks because the noise and artifacts introduced during imaging and reconstruction are more complex than that captured by simple down-sampling operation. We overcome these two limitations, by (1) using the ratio of pore throat size and voxel size (N) to group training dataset instead of voxel size and (2) using pairs of registered micro-CT images acquired using a state-of-the-art detector instead of synthetically down-sampled images. We show that combination of these two techniques produced images with better sharpness and contrast and enabled us to refine voxel size significantly beyond what is possible using the current imaging technology while maintaining the field of view.

Deep learning-based noise filtering toward millisecond order imaging by using scanning transmission electron microscopy

Application of scanning transmission electron microscopy (STEM) to in situ observation will be essential in the current and emerging data-driven materials science by taking STEM’s high affinity with various analytical options into account. As is well known, STEM’s image acquisition time needs to be further shortened to capture a targeted phenomenon in real-time as STEM’s current temporal resolution is far below the conventional TEM’s. However, rapid image acquisition in the millisecond per frame or faster generally causes image distortion, poor electron signals, and unidirectional blurring, which are obstacles for realizing video-rate STEM observation. Here we show an image correction framework integrating deep learning (DL)-based denoising and image distortion correction schemes optimized for STEM rapid image acquisition. By comparing a series of distortion corrected rapid scan images with corresponding regular scan speed images, the trained DL network is shown to remove not only the statistical noise but also the unidirectional blurring. This result demonstrates that rapid as well as high-quality image acquisition by STEM without hardware modification can be established by the DL. The DL-based noise filter could be applied to in-situ observation, such as dislocation activities under external stimuli, with high spatio-temporal resolution.

UnfairGAN: An enhanced generative adversarial network for raindrop removal from a single image

Image deraining is a new challenging problem in real-world applications, such as autonomous vehicles. In a bad weather condition of heavy rainfall, raindrops, mainly hitting glasses or windshields, can significantly reduce observation ability. Moreover, raindrops spreading over the glass can yield refraction’s physical effect, which seriously impedes the sightline or undermine machine learning systems. In this paper, we propose an enhanced generative adversarial network to deal with the challenging problems of raindrops. UnfairGAN is an enhanced generative adversarial network that can utilize prior high-level information, such as edges and rain estimation, to boost deraining performance. UnfairGAN can effectively conserve the essential details caused by heavy raindrops and remove artifacts caused by the instability of training the discriminator. This method is based on three main primary advantages. First, UnfairGAN consists of an advanced loss function of the discriminator that can improve the instabilities of traditional GAN. Second, UnfairGAN comprises a new advanced activation function that is able to increase the learning effectiveness of image classification and reconstruction. Finally, UnfairGAN is basically built on a new end-to-end cascade network, namely DRD-UNet, used to probe hierarchical features for image restoration effectively. When evaluating competing methods on the well-known Raindrop dataset, we achieve a peak signal-to-noise ratio value of 31.56 while retaining the essential details in the image. Besides, we introduce a new large image dataset (DeepRaindrops) for training deep learning networks of removing raindrops. In this dataset, our proposed method is superior to other state-of-the-art approaches of deraining raindrops regarding quantitative metrics and visual quality. Our source codes for UnfairGAN are available at https://github.com/ZeroZero19/UnfairGAN.git.

Synthetic seismic data for training deep learning networks

Deep learning is increasingly being used as a component of geoscience workflows for processing and interpreting seismic data. Training a supervised deep learning network is a data-hungry task: a significant number of data examples are needed and they must include labels. The data examples and their labels must have consistent patterns for the deep learning network to learn. Too few examples and/or poor-quality labels can lead to poor deep learning training results. One method to provide large quantities of training examples with high-quality labels is to create synthetic data. We discuss our techniques and experiences with our ongoing use of synthetic seismic data. We share our techniques as an open-source project concurrent with this paper at https://github.com/tpmerrifield/synthoseis. We hope that the geoscience community will share our enthusiasm for developing deep learning geoscience tools and for including synthetic seismic data in supervised deep learning training. We invite contributions from the geoscience community using the open-source model to collectively reduce the realism gap between synthetic data and field seismic data.