DL-based Method Research Articles

Method for clustering and identification of objects in laser scanning point clouds using dynamic logic

Today there is a gap between a presence of manifold new equipment on the market which provides streams of various digital data about the environment, in particular in the form of laser scanning point clouds, and the lack of adequate efficient methods and software for information extraction from such data. A solution to the problem of bridging this gap on the basis of neural modeling field theory and dynamic logic (DL) is proposed. We present a DL-based method of extracting and analyzing information in hybrid point clouds, which include not only spatial coordinates and intensity, but also the color of each point from multiple sources including terrestrial, mobile, and airborne laser scanning data. The proposed method is significant for creating a fundamental theoretical basis for new application algorithms and software for many new applications, including building information modeling and “smart city” environment. The proposed method is fairly new to solving various problems related to extracting semantically rich information from a nontraditional type of digital data, especially hybrid point clouds created from laser scanning. This method will allow to significantly expand the existing boundaries of knowledge in the field of extraction and analysis of information from various digital data, because neural modeling field theory and DL can improve the performance of relevant calculations and close the existing gap in analysis of digital images.

Proposing a deep learning-based method for improving the diagnostic certainty of pulmonary nodules in CT scan of chest.

To compare the performance of a deep learning (DL)-based method for diagnosing pulmonary nodules compared with radiologists' diagnostic approach in computed tomography (CT) of the chest. A total of 150 pathologically confirmed pulmonary nodules (60% malignant) assessed and reported by radiologists were included. CT images were processed by the proposed DL-based method to generate the probability of malignancy (0-100%), and the nodules were divided into the groups of benign (0-39.9%), indeterminate (40.0-59.9%), and malignant (60.0-100%). Taking the pathological results as the gold standard, we compared the diagnostic performance of the proposed DL-based method with the radiologists' diagnostic approach using the McNemar-Bowker test. There was a statistically significant difference between the diagnosis results of the proposed DL-based method and the radiologists' diagnostic approach (p < 0.001). Moreover, there was no statistically significant difference in the composition of the diagnosis results between the proposed DL-based method and the radiologists' diagnostic approach (all p > 0.05). The difference in diagnostic accuracy between the proposed DL-based method (70%) and radiologists' diagnostic performance (64%) was not statistically significant (p = 0.243). The proposed DL-based method achieved an accuracy comparable with the radiologists' diagnostic approach in clinical practice. Furthermore, its advantage in improving diagnostic certainty may raise the radiologists' confidence in diagnosing pulmonary nodules and may help clinical management. Therefore, the proposed DL-based method showed great potential in a certain clinical application. • Deep learning-based method for diagnosing the pulmonary nodules in computed tomography provides a higher diagnostic certainty.

Object-Oriented and Deep-Learning-Based High-Resolution Mapping from Large Remote Sensing Imagery

Land use and land cover (LULC) mapping is a basic research topic in geography. In deep-learning (DL)-based LULC mapping, there are primarily the following issues: training and testing samples for DL are typically annotated by indoor visual interpretation without field surveys; remotely sensed scene classification based on DL typically lacks fine geometric boundaries for ground objects; historical big data (HBD) (e.g., vector data) are underutilized in DL; studies of large-scale remote sensing mapping (LSRSM) using DL are rare. To solve above issues, this paper proposes an object-oriented (i.e., polygon-based) and DL-based (OODLB) image classification method assisted by HBD for LSRSM to serve for monitoring the soil erosion and water loss (SEWL) in the Yangtze River Basin that includes the following steps: (1) using HBD and OpenStreetMap data, ground objects are vectorized; (2) a remote sensing interpretation key database is established by field surveys and data augmentation; (3) object-oriented (i.e., polygon-based) and Inception-ResNet-V2-based LULC mapping is performed; (4) DL-based classification results are updated by man-machine mutual verification. The experimental results of one county of the Yangtze River Basin show state-of-the-art performance with an overall accuracy of 90.20% in comparison of 75.60% for eCognition. It provides an excellent framework for OODLB large-scene mapping.

Feasibility of deep learning-based fully automated classification of microsatellite instability in tissue slides of colorectal cancer.

High levels of microsatellite instability (MSI-H) occurs in about 15% of sporadic colorectal cancer (CRC) and is an important predictive marker for response to immune checkpoint inhibitors. To test the feasibility of a deep learning (DL)-based classifier as a screening tool for MSI status, we built a fully automated DL-based MSI classifier using pathology whole-slide images (WSIs) of CRCs. On small image patches of The Cancer Genome Atlas (TCGA) CRC WSI dataset, tissue/non-tissue, normal/tumor and MSS/MSI-H classifiers were applied sequentially for the fully automated prediction of the MSI status. The classifiers were also tested on an independent cohort. Furthermore, to test how the expansion of the training data affects the performance of the DL-based classifier, additional classifier trained on both TCGA and external datasets was tested. The areas under the receiver operating characteristic curves were 0.892 and 0.972 for the TCGA and external datasets, respectively, by a classifier trained on both datasets. The performance of the DL-based classifier was much better than that of previously reported histomorphology-based methods. We speculated that about 40% of CRC slides could be screened for MSI status without molecular testing by the DL-based classifier. These results demonstrated that the DL-based method has potential as a screening tool to discriminate molecular alteration in tissue slides.

Design of sparse arrays via deep learning for enhanced DOA estimation

This paper introduces an enhanced deep learning-based (DL) antenna selection approach for optimum sparse linear array selection for direction-of-arrival (DOA) estimation applications. Generally, the antenna selection problem yields a combination of subarrays as a solution. Previous DL-based methods designated these subarrays as classes to fit the problem into a classification problem to which a convolutional neural network (CNN) is employed to solve it. However, these methods sample the combination set randomly to reduce computational cost related to the generation of training data, and it often leads to sub-optimal solutions due to ill-sampling issues. Hence, in this paper, we propose an improved DL-based method by constraining the combination set to retain the hole-free subarrays to enhance the method’s performance and sparse subarrays rendered. Numerical examples show that the proposed method yields sparser subarrays with better beampattern properties and improved DOA estimation performance than conventional DL techniques.

Deep Learning-Based Denoising in High-Speed Portable Reflectance Confocal Microscopy.

Portable confocal microscopy (PCM) is a low-cost reflectance confocal microscopy technique that can visualize cellular details of human skin in vivo. When PCM images are acquired with a short exposure time to reduce motion blur and enable real-time 3D imaging, the signal-to-noise ratio (SNR) is decreased significantly, which poses challenges in reliably analyzing cellular features. In this paper, we evaluated deep learning (DL)-based approach for reducing noise in PCM images acquired with a short exposure time. Content-aware image restoration (CARE) network was trained with pairs of low-SNR input and high-SNR ground truth PCM images obtained from 309 distinctive regions of interest (ROIs). Low-SNR input images were acquired from human skin in vivo at the imaging speed of 180 frames/second. The high-SNR ground truth images were generated by registering 30 low-SNR input images obtained from the same ROI and summing them. The CARE network was trained using the Google Colaboratory Pro platform. The denoising performance of the trained CARE network was quantitatively and qualitatively evaluated by using image pairs from 45 unseen ROIs. CARE denoising improved the image quality significantly, increasing similarity with the ground truth image by 1.9 times, reducing noise by 2.35 times, and increasing SNR by 7.4 dB. Banding noise, prominent in input images, was significantly reduced in CARE denoised images. CARE denoising provided quantitatively and qualitatively better noise reduction than non-DL filtering methods. Qualitative image assessment by three confocal readers showed that CARE denoised images exhibited negligible noise more often than input images and non-DL filtered images. Results showed the potential of using a DL-based method for denoising PCM images obtained at a high imaging speed. The DL-based denoising method needs to be further trained and tested for PCM images obtained from disease-suspicious skin lesions.

Ground-truth-free deep learning for artefacts reduction in 2D radial cardiac cine MRI using a synthetically generated dataset

In this work, we consider the task of image reconstruction in 2D radial cardiac cine MRI using deep learning (DL)-based regularization. As the regularization is achieved by employing an image-prior predicted by a pre-trained convolutional neural network (CNN), the quality of the image-prior is of essential importance. The achievable performance of any DL-based method is limited by the amount and the quality of the available training data. For fast dynamic processes, obtaining good-quality MR data is challenging because of technical and physiological reasons. In this work, we try to overcome these problems by a transfer-learning approach which is motivated by a previously presented DL-method (XT,YT U-Net). There, instead of training the network on the whole 2D dynamic images, it is trained on 2D spatio-temporal profiles (xt,yt-slices) which show the temporal changes of the imaged object. Therefore, for the training and test data, it is more important that their spatio-temporal profiles share similar local features rather than being images of the same anatomy. This allows us to equip arbitrary data with simulated motion that resembles the cardiac motion and use it as training data. By doing so, it is possible to train a CNN which is applicable to cardiac cine MR data without using ground-truth cine MR images for training. We demonstrate that combining XT,YT U-Net with the proposed transfer-learning strategy delivers comparable performance to CNNs trained on cardiac cine MR images and in some cases even qualitatively surpasses these. Additionally, the transfer-learning strategy was investigated for a 2D and 3D U-Net. The images processed by the the CNNs were used as image-priors in the CNN-regularized iterative reconstruction. The XT,YT U-Net yielded visibly better results than the 2D U-Net and slightly better results than the 3D U-Net when used in combination with the presented transfer learning-strategy.

Analysis on SPECT myocardial perfusion imaging with a tool derived from dynamic programming to deep learning

Our study tries to propose a method that can rapidly process and calibrate data sets through our previous dynamic programming (DP) tool, manually adjust error-heavy labels, and then train neural network models to accurately segment gated SPECT MPI images. Lastly, LV contours can be accurately determined on the gated SPECT MPI images and corresponding LV volumes can also be evaluated. First, we used our previous DP-based tool to coarsely determine the myocardial profiles on SPECT MPI images. In order to obtain a more accurate LV volumes and LVEF, it was also necessary to manually adjust the profiles that do not meet the clinical requirements. Then the results of manual adjustment were used as the ground truths to train the end-to-end full convolution neural networks by deep learning (DL). The method proposed in this paper was evaluated with the Dice similarity coefficient and Hausdorff distance, and the results showed that the proposed method had a high similarity with the ground truth. The LV volumes and LVEFs estimated by different methods were calculated separately and compared each other, and the experimental results showed that the performance of the proposed DL-based method was superior to that of the DP-based method and close to that of QGS program. As a quantitative analysis tool, the proposed DL-based method is fully automatic and more accurate in the evaluation of LV volumes and LVEFs, so it is expected that the DL-based method may have a wide application prospect in clinical practices.

Single Exposure Optical Image Watermarking Using a cGAN Network

A single exposure optical image watermarking framework based on deep learning (DL) is proposed in this paper, and original watermark image information can be reconstructed from only single-frame watermarked hologram by using an end-to-end network with high-quality. First, the single exposure watermarked hologram is acquired with our presented phase-shifted interferometry based optical image watermarking (PSOIW) frame, and then all holograms and corresponding watermark images are constructed to the train datasets for the learning of an end-to-end conditional generative adversarial network (cGAN), finally retrieved the watermark image well with the trained cGAN network using only one hologram. This DL-based method greatly reduces the recording or transmitting data burden by 1/4 compared with our presented PSOIW technique, and may provide a new way for the real-time 3D image/video security applications. The feasibility and security of the proposed method are demonstrated by the optical experiment results.

Conventional Machine Learning versus Deep Learning for Magnification Dependent Histopathological Breast Cancer Image Classification: A Comparative Study with Visual Explanation.

Breast cancer is a serious threat to women. Many machine learning-based computer-aided diagnosis (CAD) methods have been proposed for the early diagnosis of breast cancer based on histopathological images. Even though many such classification methods achieved high accuracy, many of them lack the explanation of the classification process. In this paper, we compare the performance of conventional machine learning (CML) against deep learning (DL)-based methods. We also provide a visual interpretation for the task of classifying breast cancer in histopathological images. For CML-based methods, we extract a set of handcrafted features using three feature extractors and fuse them to get image representation that would act as an input to train five classical classifiers. For DL-based methods, we adopt the transfer learning approach to the well-known VGG-19 deep learning architecture, where its pre-trained version on the large scale ImageNet, is block-wise fine-tuned on histopathological images. The evaluation of the proposed methods is carried out on the publicly available BreaKHis dataset for the magnification dependent classification of benign and malignant breast cancer and their eight sub-classes, and a further validation on KIMIA Path960, a magnification-free histopathological dataset with 20 image classes, is also performed. After providing the classification results of CML and DL methods, and to better explain the difference in the classification performance, we visualize the learned features. For the DL-based method, we intuitively visualize the areas of interest of the best fine-tuned deep neural networks using attention maps to explain the decision-making process and improve the clinical interpretability of the proposed models. The visual explanation can inherently improve the pathologist’s trust in automated DL methods as a credible and trustworthy support tool for breast cancer diagnosis. The achieved results show that DL methods outperform CML approaches where we reached an accuracy between 94.05% and 98.13% for the binary classification and between 76.77% and 88.95% for the eight-class classification, while for DL approaches, the accuracies range from 85.65% to 89.32% for the binary classification and from 63.55% to 69.69% for the eight-class classification.

A Multitask Learning-Based Neural Network for Defect Detection on Textured Surfaces Under Weak Supervision

Automatic defect detection on textured surfaces has long been one of the hotspots in the computer vision community. Many methods based on deep learning (DL) have been proposed in the past few years although most are semantic segmentation-based methods that require a large number of accurately labeled samples. However, obtaining precise labels is time-consuming and costly. In this article, we propose a novel weakly supervised DL-based method to accurately segment and locate defects on textured surfaces. In the proposed method, we design one encoder for extracting features and two decoders for two related tasks. To ensure that the deep neural network benefits from multitask learning, the two decoders share the unique encoder. The main task is designed to restore the defects on textured surfaces, and the auxiliary task is designed to obtain the region of interest (ROI), which is employed to filter out noise in the main task. Subsequently, we can obtain residual maps by comparing the original images and the restored images. Finally, more accurate results can be obtained through the fusion of residual maps and ROIs. A series of experiments on the public defect detection dataset DAGM showed that our method demonstrates the best performance compared with other state-of-the-art methods.

Unsupervised Pre-Training of Imbalanced Data for Identification of Wafer Map Defect Patterns

Visual defect inspection and classification are significant steps of most manufacturing processes in the semiconductor and electronics industries. Known and unknown defects on wafer maps tend to cluster, and these spatial patterns provide valuable process information for supporting manufacturing in determining the root causes of abnormal processes. In previous studies, data augmentation-based deep learning (DL) techniques were most commonly used for the identification of wafer map defect patterns (WMDP). Data augmentation is an effective technique for improving the accuracy of modern image classifiers. However, current data augmentation implementations were manually designed for the WMDP problem. In this study, we propose a DL-based method with automatic data augmentation for the WMDP task. Basically, it focuses on learning effective discriminative features, from wafer maps, through a deep network structure. The network consists of a convolution-based variational autoencoder (CVAE) sequentially. First, we pre-trained the CVAE on large training data in an unsupervised manner. Second, we fine-tuned the encoder of the CVAE, which was followed by a neural network (NN) classifier, in a supervised manner. Additionally, we describe a simple procedure for automatically searching for improved data augmentation policies. The policy mainly consists of five image processing functions: rotation, flipping, shifting, shearing range, and zooming. The effectiveness of the proposed method was demonstrated through experimental results obtained from a simulation dataset and a real-world wafer map dataset (WM-811K). This study provides guidance for the application of deep learning in semiconductor manufacturing processes to improve product quality and yield.

Deep learning-based instance segmentation of cracks from shield tunnel lining images

This paper presents a deep learning (DL)-based method for the instance segmentation of cracks from shield tunnel lining images using a mask region-based convolutional neural network (Mask R-CNN) incorporated with a morphological closing operation. The Mask R-CNN herein is divided into a backbone architecture, a region proposal network (RPN), and a head architecture for specification, and the implementation details are introduced. Compared with the current image processing methods, the proposed DL-based method efficiently detects cracks in an image while simultaneously generating a high-quality segmentation mask for each crack. A shield tunnel lining image dataset is established for crack instance segmentation task. The established dataset contains a total of 1171 labelled crack instances in 761 images. The morphological closing operation was incorporated into a Mask R-CNN to form an integrated model to connect disjoint cracks that belong to one crack. Image tests were carried out among four trained models to explore the effect of the morphological closing operation, network depth, and feature pyramid network on crack segmentation performance, and a relative optimal model is found. The relative optimal model achieves a balanced accuracy of 81.94%, a F1 score of 68.68%, and an intersection over union (IoU) of 52.72% with respect to 76 test images.

Depth Estimation for Light-Field Images Using Stereo Matching and Convolutional Neural Networks.

The paper presents a novel depth-estimation method for light-field (LF) images based on innovative multi-stereo matching and machine-learning techniques. In the first stage, a novel block-based stereo matching algorithm is employed to compute the initial estimation. The proposed algorithm is specifically designed to operate on any pair of sub-aperture images (SAIs) in the LF image and to compute the pair’s corresponding disparity map. For the central SAI, a disparity fusion technique is proposed to compute the initial disparity map based on all available pairwise disparities. In the second stage, a novel pixel-wise deep-learning (DL)-based method for residual error prediction is employed to further refine the disparity estimation. A novel neural network architecture is proposed based on a new structure of layers. The proposed DL-based method is employed to predict the residual error of the initial estimation and to refine the final disparity map. The experimental results demonstrate the superiority of the proposed framework and reveal that the proposed method achieves an average improvement of in root mean squared error (RMSE), in mean absolute error (MAE), and in structural similarity index (SSIM) over machine-learning-based state-of-the-art methods.

An automatic framework for endoscopic image restoration and enhancement

Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs.

Clipping Noise Compensation with Neural Networks in OFDM Systems

The application of deep learning (DL) to solve physical layer issues has emerged as a prominent topic. In this paper, the mitigation of clipping effects for orthogonal frequency division multiplexing (OFDM) systems with the help of a Neural Network (NN) is investigated. Unlike conventional clipping recovery algorithms, which involve costly iterative procedures, the DL-based method learns to directly reconstruct the clipped part of the signal while the unclipped part is protected. Furthermore, an interpretation of the learned weight matrices of the neural network is presented. It is observed that parts of the network, in effect, implement transformations very similar to the (Inverse) Discrete Fourier Transform (DFT/IDFT) to provide information in both the time and frequency domains. The simulation results show that the proposed method outperforms existing algorithms for recovering clipped OFDM signals in terms of both mean square error (MSE) and Bit Error Rate (BER).

Amplitude and phase estimations of power system harmonics using deep learning framework

In this study, a new method for the analysis of harmonic components in the power system based on a deep learning (DL) framework is introduced. In the proposed method, both amplitudes and phases of the harmonic components can be estimated accurately, unlike most of the research work in the literature, which usually focus on estimating amplitudes only. A convolutional neural network (CNN) structure is used to estimate the phases and amplitudes of harmonics, although CNN is usually used for classification. It has been shown that the proposed DL-based method can satisfactorily estimate both amplitudes and phases of the power system harmonics inside a 20-ms window and this makes the proposed method suitable for possible real-time applications, such as active power filtering of the harmonics. It has also been shown that the proposed method is robust to fundamental frequency changes. Experiments on carefully-generated data sets to reflect the power system behaviour show that the proposed method demonstrates remarkably good performance in terms of estimation accuracy, especially for time-varying frequency cases. Average error for the amplitude estimation is obtained as 0.21% and that for the phase is 9°, which outperforms the other compared analyses methods in cases of fundamental frequency variations.

Accelerating GluCEST imaging using deep learning for B0 correction.

Glutamate weighted Chemical Exchange Saturation Transfer (GluCEST) MRI is a noninvasive technique for mapping parenchymal glutamate in the brain. Because of the sensitivity to field (B0 ) inhomogeneity, the total acquisition time is prolonged due to the repeated image acquisitions at several saturation offset frequencies, which can cause practical issues such as increased sensitivity to patient motions. Because GluCEST signal is derived from the small z-spectrum difference, it often has a low signal-to-noise-ratio (SNR). We proposed a novel deep learning (DL)-based algorithm armed with wide activation neural network blocks to address both issues. B0 correction based on reduced saturation offset acquisitions was performed for the positive and negative sides of the z-spectrum separately. For each side, a separate deep residual network was trained to learn the nonlinear mapping from few CEST-weighted images acquired at different ppm values to the one at 3 ppm (where GluCEST peaks) in the same side of the z-spectrum. All DL-based methods outperformed the "traditional" method visually and quantitatively. The wide activation blocks-based method showed the highest performance in terms of Structural Similarity Index (SSIM) and peak signal-to-noise ratio (PSNR), which were 0.84 and 25dB respectively. SNR increases in regions of interest were over 8dB. We demonstrated that the new DL-based method can reduce the entire GluCEST imaging time by ˜50% and yield higher SNR than current state-of-the-art.

A Hybrid Deep Learning Model for Protein–Protein Interactions Extraction from Biomedical Literature

The exponentially increasing size of biomedical literature and the limited ability of manual curators to discover protein–protein interactions (PPIs) in text has led to delays in keeping PPI databases updated with the current findings. The state-of-the-art text mining methods for PPI extraction are primarily based on deep learning (DL) models, and the performance of a DL-based method is mainly affected by the architecture of DL models and the feature embedding methods. In this study, we compared different architectures of DL models, including convolutional neural networks (CNN), long short-term memory (LSTM), and hybrid models, and proposed a hybrid architecture of a bidirectional LSTM+CNN model for PPI extraction. Pretrained word embedding and shortest dependency path (SDP) embedding are fed into a two-embedding channel model, such that the model is able to model long-distance contextual information and can capture the local features and structure information effectively. The experimental results showed that the proposed model is superior to the non-hybrid DL models, and the hybrid CNN+Bidirectional LSTM model works well for PPI extraction. The visualization and comparison of the hidden features learned by different DL models further confirmed the effectiveness of the proposed model.

Exploring Uncertainty Measures in Bayesian Deep Attentive Neural Networks for Prostate Zonal Segmentation.

Automatic segmentation of prostatic zones on multiparametric MRI (mpMRI) can improve the diagnostic workflow of prostate cancer. We designed a spatial attentive Bayesian deep learning network for the automatic segmentation of the peripheral zone (PZ) and transition zone (TZ) of the prostate with uncertainty estimation. The proposed method was evaluated by using internal and external independent testing datasets, and overall uncertainties of the proposed model were calculated at different prostate locations (apex, middle, and base). The study cohort included 351 MRI scans, of which 304 scans were retrieved from a de-identified publicly available datasets (PROSTATEX) and 47 scans were extracted from a large U.S. tertiary referral center (external testing dataset; ETD)). All the PZ and TZ contours were drawn by research fellows under the supervision of expert genitourinary radiologists. Within the PROSTATEX dataset, 259 and 45 patients (internal testing dataset; ITD) were used to develop and validate the model. Then, the model was tested independently using the ETD only. The segmentation performance was evaluated using the Dice Similarity Coefficient (DSC). For PZ and TZ segmentation, the proposed method achieved mean DSCs of 0.80±0.05 and 0.89±0.04 on ITD, as well as 0.79±0.06 and 0.87±0.07 on ETD. For both PZ and TZ, there was no significant difference between ITD and ETD for the proposed method. This DL-based method enabled the accuracy of the PZ and TZ segmentation, which outperformed the state-of-art methods (Deeplab V3+, Attention U-Net, R2U-Net, USE-Net and U-Net). We observed that segmentation uncertainty peaked at the junction between PZ, TZ and AFS. Also, the overall uncertainties were highly consistent with the actual model performance between PZ and TZ at three clinically relevant locations of the prostate.