U-Net Convolutional Neural Network Research Articles

Using a U-net convolutional neural network to map woody vegetation extent from high resolution satellite imagery across Queensland, Australia

Convolutional neural networks offer a new approach to classifying high resolution imagery. We use the U-net neural network architecture to map the presence or absence of trees and large shrubs across the Australian state of Queensland. From a state-wide mosaic of 1 m resolution 3-band Earth-i imagery, a selection of 827 squares (1 km2) are manually labeled for the presence of trees or large shrubs, and these are used to train the neural network. The training is intended to capture the textures which are primary visual cues of such vegetation. The trained neural network has an accuracy on independent data of around 90%. The resulting map over the whole of Queensland (1.73 million km2) is intended to be manually checked, and edited where necessary, to provide a high quality map of woody vegetation extent to serve a range of government policy objectives.

Deep Learning for SAR Image Despeckling

Speckle filtering is an unavoidable step when dealing with applications that involve amplitude or intensity images acquired by coherent systems, such as Synthetic Aperture Radar (SAR). Speckle is a target-dependent phenomenon; thus, its estimation and reduction require the individuation of specific properties of the image features. Speckle filtering is one of the most prominent topics in the SAR image processing research community, who has first tackled this issue using handcrafted feature-based filters. Even if classical algorithms have slowly and progressively achieved better and better performance, the more recent Convolutional-Neural-Networks (CNNs) have proven to be a promising alternative, in the light of the outstanding capabilities in efficiently learning task-specific filters. Currently, only simplistic CNN architectures have been exploited for the speckle filtering task. While these architectures outperform classical algorithms, they still show some weakness in the texture preservation. In this work, a deep encoder–decoder CNN architecture, focused in the specific context of SAR images, is proposed in order to enhance speckle filtering capabilities alongside texture preservation. This objective has been addressed through the adaptation of the U-Net CNN, which has been modified and optimized accordingly. This architecture allows for the extraction of features at different scales, and it is capable of producing detailed reconstructions through its system of skip connections. In this work, a two-phase learning strategy is adopted, by first pre-training the model on a synthetic dataset and by adapting the learned network to the real SAR image domain through a fast fine-tuning procedure. During the fine-tuning phase, a modified version of the total variation (TV) regularization was introduced to improve the network performance when dealing with real SAR data. Finally, experiments were carried out on simulated and real data to compare the performance of the proposed method with respect to the state-of-the-art methodologies.

Multi-sensor cloud and cloud shadow segmentation with a convolutional neural network

Cloud and cloud shadow segmentation is a crucial pre-processing step for any application that uses multi-spectral satellite images. In particular, disaster related applications (e.g., flood monitoring or rapid damage mapping), which are highly time- and data-critical, require methods that produce accurate cloud and cloud shadow masks in short time while being able to adapt to large variations in the target domain (induced by atmospheric conditions, different sensors, scene properties, etc.). In this study, we propose a data-driven approach to semantic segmentation of cloud and cloud shadow in single date images based on a modified U-Net convolutional neural network that aims to fulfil these requirements. We train the network on a global database of Landsat OLI images for the segmentation of five classes (“shadow”, “cloud”, “water”, “land” and “snow/ice”). We compare the results to state-of-the-art methods, proof the model's generalization ability across multiple satellite sensors (Landsat TM, Landsat ETM+, Landsat OLI and Sentinel-2) and show the influence of different training strategies and spectral band combinations on the performance of the segmentation. Our method consistently outperforms Fmask and a traditional Random Forest classifier on a globally distributed multi-sensor test dataset in terms of accuracy, Cohen's Kappa coefficient, Dice coefficient and inference speed. The results indicate that a reduced feature space composed solely of red, green, blue and near-infrared bands already produces good results for all tested sensors. If available, adding shortwave-infrared bands can increase the accuracy. Contrast and brightness augmentations of the training data further improve the segmentation performance. The best performing U-Net model achieves an accuracy of 0.89, Kappa of 0.82 and Dice coefficient of 0.85, while running the inference over 896 test image tiles with 44.8 s/megapixel (2.8 s/megapixel on GPU). The Random Forest classifier reaches an accuracy of 0.79, Kappa of 0.65 and Dice coefficient of 0.74 with 3.9 s/megapixel inference time (on CPU) on the same training and testing data. The rule-based Fmask method takes significantly longer (277.8 s/megapixel) and produces results with an accuracy of 0.75, Kappa of 0.60 and Dice coefficient of 0.72.

Segmentation of Brain MR Images by Using Fully Convolutional Network and Gaussian Mixture Model with Spatial Constraints

Accurate segmentation of brain tissue from magnetic resonance images (MRIs) is a critical task for diagnosis, treatment, and clinical research. In this paper, a novel algorithm (GMMD-U) that incorporates the modified full convolutional neural network U-net and Gaussian-Dirichlet mixture model (GMMD) with spatial constraints is presented. The proposed GMMD-U considers the local spatial relationships by assuming that the prior probability obeys the Dirichlet distribution. Specifically, GMMD is applied for extracting brain tissue that has a distinct intensity region and modified U-net is exploited to correct the wrong-classification areas caused by GMMD or other conventional approaches. The proposed GMMD-U is designed to take advantage of the statistical model-based segmentation techniques and deep neural network. We evaluate the performance of GMMD-U on a publicly available brain MRI dataset by comparing it with several existing algorithms, and the results reported reveal that the proposed framework can accurately detect the brain tissue from MRIs. The proposed learning-based integrated framework could be effective for brain tissue segmentation, which will be helpful for surgeons in brain disease diagnosis.

Application of deep learning for the quantitative assessment of bone marrow lesions (BMLs)

Purpose: MRI can be used to assess several structural changes related to knee OA (KOA). Software image analysis methods provide objective, fully quantitative measurements of these changes. However, such methods generally require a human reader and can be time consuming if the number of images in a study is large. Deep leaning (DL), a form of statistical machine learning, offers the potential for increased or full automation, therefore reducing reader time substantially. The goal of our study was to validate a DL method for segmenting (outlining) bone marrow lesions (BMLs) on knee MRI scans. Methods: We used the baseline, 12 mo. and 24 mo. scans from the 600 subjects included in the FNIH Study, a nested case/control sub study of the OAI. All 600 subjects had both BL and 24 month scans, while 582 were imaged at the 12 month time point. Therefore, the dataset included 2 or 3 scans for each of the 600 subjects, totaling 1782 scans. Selecting for subjects with a BML in at least one time point gave 1358 scans from 544 different subjects for further processing. These 544 subjects were randomly split into training, validation and test sets of 50% (673 scans, 272 subjects) / 25% (352 scans, 186 subjects), 25% (333 scans, 186 subjects), respectively. BMLs segmented using a previously-validated semi-automated (SA) method were used to train the DL algorithm. A U-Net convolutional neural network was trained on BML patches and corresponding segmentation mask pairs determined from the SA method. The U-Net model is a fully-convolutional network (FCN) architecture specifically designed for image segmentation tasks. The DL algorithm was provided with the general location of each BML. We compared the BML volumes determined with the DL software to the SA measurements. The Dice similarity coefficients (DSC) and Pearson’s R2 are reported. In Figure 1 we show a representative example of BMLs segmented with SA and DL. Results: The average DSC was 0.70 and Figure 2 is a plot comparing the SA to the DL methods. The Pearson’s R2 was 0.94. Figure 3 depicts typical segmentation results over a range of DSC values. The DL method is not fully automated since a reader is required to indicate the location of a BML. However, this step requires on average 30 seconds per scan compared to 5 minutes for the full SA method. Conclusions: We have shown that the DL approach can produce an accurate method to assess BML volume, offering substantial time savings (90%) over the SA method. Future work will investigate a fully automated method, which does not require a reader to initially locate BMLs.View Large Image Figure ViewerDownload Hi-res image Download (PPT)View Large Image Figure ViewerDownload Hi-res image Download (PPT)

Automated segmentation of 2D low-dose CT images of the psoas-major muscle using deep convolutional neural networks.

The psoas-major muscle has been reported as a predictive factor of sarcopenia. The cross-sectional area (CSA) of the psoas-major muscle in axial images has been indicated to correlate well with the whole-body skeletal muscle mass. In this study, we evaluated the segmentation accuracy of low-dose X-ray computed tomography (CT) images of the psoas-major muscle using the U-Net convolutional neural network, which is a deep-learning technique. Deep learning has been recently known to outperform conventional image-segmentation techniques. We used fivefold cross validation to validate the segmentation performance (n = 100) of the psoas-major muscle. For the intersection over union and CSA ratio, segmentation accuracies of 86.0 and 103.1%, respectively, were achieved. These results suggest that the U-Net network is competitive compared with the previous methods. Therefore, the proposed technique is useful for segmenting the psoas-major muscle even in low-dose CT images.

Deep-Learning Schemes for Full-Wave Nonlinear Inverse Scattering Problems

This paper is devoted to solving a full-wave inverse scattering problem (ISP), which is aimed at retrieving permittivities of dielectric scatterers from the knowledge of measured scattering data. ISPs are highly nonlinear due to multiple scattering, and iterative algorithms with regularizations are often used to solve such problems. However, they are associated with heavy computational cost, and consequently, they are often time-consuming. This paper proposes the convolutional neural network (CNN) technique to solve full-wave ISPs. We introduce and compare three training schemes based on U-Net CNN, including direct inversion, backpropagation, and dominant current schemes (DCS). Several representative tests are carried out, including both synthetic and experimental data, to evaluate the performances of the proposed methods. It is demonstrated that the proposed DCS outperforms the other two schemes in terms of accuracy and is able to solve typical ISPs quickly within 1 s. The proposed deep-learning inversion scheme is promising in providing quantitative images in real time.

Automated CT and MRI Liver Segmentation and Biometry Using a Generalized Convolutional Neural Network.

To assess feasibility of training a convolutional neural network (CNN) to automate liver segmentation across different imaging modalities and techniques used in clinical practice and apply this to enable automation of liver biometry. We trained a 2D U-Net CNN for liver segmentation in two stages using 330 abdominal MRI and CT exams acquired at our institution. First, we trained the neural network with non-contrast multi-echo spoiled-gradient-echo (SGPR)images with 300 MRI exams to provide multiple signal-weightings. Then, we used transfer learning to generalize the CNN with additional images from 30 contrast-enhanced MRI and CT exams.We assessed the performance of the CNN using a distinct multi-institutional data set curated from multiple sources (n = 498 subjects). Segmentation accuracy was evaluated by computing Dice scores. Utilizing these segmentations, we computed liver volume from CT and T1-weighted (T1w) MRI exams, and estimated hepatic proton- density-fat-fraction (PDFF) from multi-echo T2*w MRI exams. We compared quantitative volumetry and PDFF estimates between automated and manual segmentation using Pearson correlation and Bland-Altman statistics. Dice scores were 0.94 ± 0.06 for CT (n = 230), 0.95 ± 0.03 (n = 100) for T1w MR, and 0.92 ± 0.05 for T2*w MR (n = 169). Liver volume measured by manual and automated segmentation agreed closely for CT (95% limit-of-agreement (LoA) = [-298 mL, 180 mL]) and T1w MR (LoA = [-358 mL, 180 mL]). Hepatic PDFF measured by the two segmentations also agreed closely (LoA = [-0.62%, 0.80%]). Utilizing a transfer-learning strategy, we have demonstrated the feasibility of a CNN to be generalized to perform liver segmentations across different imaging techniques and modalities. With further refinement and validation, CNNs may have broad applicability for multimodal liver volumetry and hepatic tissue characterization.

Fully automated extraction of the fringe skeletons in dynamic electronic speckle pattern interferometry using a U-Net convolutional neural network

With the development of artificial intelligence technology, intelligent fringe processing is a goal of relevant researchers in optical interferometry. We propose an intelligent method to achieve fully automated extraction of the fringe skeletons in electronic speckle pattern interferometry (ESPI) based on U-Net convolutional neural network. In the proposed method, the network is first trained by the samples that consist of the noisy ESPI fringe patterns and the corresponding skeleton images. After training, the other multiframe ESPI fringe patterns are fed to the trained network simultaneously; the corresponding skeleton images can be obtained in batches. Using our method, it is not necessary to process fringe patterns frame by frame. Our method is especially suitable for multiframe fringe patterns processing. We apply the intelligent method to one computer-simulated and one real-dynamic ESPI measurement, respectively. For the simulated measurement, it takes just 40 s to obtain the skeleton images of 20 noisy ESPI fringe patterns using our method. Even for low-quality experimental obtained ESPI fringe patterns, our method can also give desired results.

Abstract WP68: Interpretable Deep Learning-based Characterization of Intracranial Hemorrhage on Head CT

Background: Machine learning algorithms have proven accurate in the detection of intracranial hemorrhage (ICH) on head CT. Most reported algorithms, however, are limited to binary detection and global lesion volume estimation. We developed a pipeline to additionally perform subtype classification for clinical risk stratification, while also providing visual displays to improve interpretability of algorithm output (“black box problem”) and enable clinical systems integration. Methods: We developed a convolutional neural network (CNN) for ICH detection and subtype classification. The algorithm was trained on 2,209 head CT exams (hemorrhage-negative controls and the 5 major ICH subtypes) and outputs class-activation heatmaps to aid interpretability. We quantify intraparenchymal hemorrhage (IPH) volume using a modified U-Net CNN trained on a cohort of 110 manually-segmented hematomas to precisely delineate lesion boundaries. Algorithm output was stored as DICOM segmentation objects which can be overlaid on native images within our existing viewer applications, enabling the user to dynamically adjust characteristics such as color and transparency. Results: Diagnostic performance for ICH detection was high with ROC area-under-curve of 0.98. Fig 1A shows a heatmap highlighting different ICH foci. IPH subtype classification performance is summarized in Fig 1B with area-under-curve of 0.95. Fig 1C shows IPH segmentation and volume estimation overlays with automatically generated text for the medical record. Dice similarity coefficient for segmentation was 0.87. Conclusion: We describe a machine-learning approach for ICH detection, subtype classification, and IPH-specific volume estimation. Utilizing easily interpretable heatmaps, dynamically-adjustable overlays, and automated result text generation, our pipeline is capable of providing accurate information optimized for clinical integration.

Optimal planar X-ray imaging soft tissue segmentation using a photon counting detector

A rigorous method for automated soft tissue segmentation using planar kilovoltage (kV) imaging, a photon counting detector (PCD), and a convolutional neural network is presented. The goal of the project was to determine the optimum number of energy bins in a PCD for soft tissue segmentation. Planar kV X-ray images of solid water (SW) phantoms with varying depth of cartilage were generated with a cone-beam analytical method and parallel-beam Monte Carlo simulations. Simulations were preformed using 2 to 5 PCD energy bins with equal photon fluence distribution. Simulated image signal to noise ratio (SNR) was varied between 10 to 250 measured after transmission through 4 cm of SW. Algorithms using non-linear as well as linear regression were used to predict the amount of cartilage for every pixel of the phantom. These algorithms were evaluated based on the mean squared error (MSE) between their prediction and the ground truth. The best algorithm was used to decompose randomly generated SW and cartilage images with an SNR of 100. These randomly generated images trained a U-Net convolutional neural network to segment the cartilage in the image. The results indicated the smallest MSE occurred for non-linear regression with 4 energy bins over all SNR. The trained U-Net was able to correctly segment all regions of cartilage for the smallest amount of cartilage used (4 mm) and segmented the region with > 99% categorical accuracy by pixel.

A Deep Learning Approach to Vascular Structure Segmentation in Dermoscopy Colour Images.

Background Atypical vascular pattern is one of the most important features by differentiating between benign and malignant pigmented skin lesions. Detection and analysis of vascular structures is a necessary initial step for skin mole assessment; it is a prerequisite step to provide an accurate outcome for the widely used 7-point checklist diagnostic algorithm. Methods In this research we present a fully automated machine learning approach for segmenting vascular structures in dermoscopy colour images. The U-Net architecture is based on convolutional networks and designed for fast and precise segmentation of images. After preprocessing the images are randomly divided into 146516 patches of 64 × 64 pixels each. Results On the independent validation dataset including 74 images our implemented method showed high segmentation accuracy. For the U-Net convolutional neural network, an average DSC of 0.84, sensitivity 0.85, and specificity 0.81 has been achieved. Conclusion Vascular structures due to small size and similarity to other local structures create enormous difficulties during the segmentation and assessment process. The use of advanced segmentation methods like deep learning, especially convolutional neural networks, has the potential to improve the accuracy of advanced local structure detection.

Fully-Automatic Synapse Prediction and Validation on a Large Data Set.

Extracting a connectome from an electron microscopy (EM) data set requires identification of neurons and determination of connections (synapses) between neurons. As manual extraction of this information is very time-consuming, there has been extensive research efforts to automatically segment the neurons to help guide and eventually replace manual tracing. Until recently, there has been comparatively little research on automatic detection of the actual synapses between neurons. This discrepancy can, in part, be attributed to several factors: obtaining neuronal shapes is a prerequisite for the first step in extracting a connectome, manual tracing is much more time-consuming than annotating synapses, and neuronal contact area can be used as a proxy for synapses in determining connections. However, recent research has demonstrated that contact area alone is not a sufficient predictor of a synaptic connection. Moreover, as segmentation improved, we observed that synapse annotation consumes a more significant fraction of overall reconstruction time (upwards of 50% of total effort). This ratio will only get worse as segmentation improves, gating the overall possible speed-up. Therefore, we address this problem by developing algorithms that automatically detect presynaptic neurons and their postsynaptic partners. In particular, presynaptic structures are detected using a U-Net convolutional neural network (CNN), and postsynaptic partners are detected using a multilayer perceptron (MLP) with features conditioned on the local segmentation. This work is novel because it requires minimal amount of training, leverages advances in image segmentation directly, and provides a complete solution for polyadic synapse detection. We further introduce novel metrics to evaluate our algorithm on connectomes of meaningful size. When applied to the output of our method on EM data from Drosphila, these metrics demonstrate that a completely automatic prediction can be used to effectively characterize most of the connectivity correctly.

Coronary Artery Segmentation by Deep Learning Neural Networks on Computed Tomographic Coronary Angiographic Images.

Coronary artery lumen delineation, to localize and grade stenosis, is an important but tedious and challenging task for coronary heart disease evaluation. Deep learning has recently been successful applied to many applications, including medical imaging. However for small imaged objects such as coronary arteries and their segmentation, it remains a challenge. This paper investigates coronary artery lumen segmentation using 3D U-net convolutional neural networks, and tests its utility with multiple datasets on two settings. We adapted the computed tomography coronary angiography (CTCA) volumes into small patches for the networks and tuned the kernels, layers and the batch size for machine learning. Our experiment involves additional efforts to select and test various data transform, so as to reduce the problem of overfitting. Compared with traditional normalization of data, we showed that subject-specific normalization of dataset was superior to patch based normalization. The results also showed that the proposed deep learning approach outperformed other methods, evaluated by the Dice coefficients.

Automatic lesion detection and segmentation of 18F-FET PET in gliomas: A full 3D U-Net convolutional neural network study.

IntroductionAmino-acids positron emission tomography (PET) is increasingly used in the diagnostic workup of patients with gliomas, including differential diagnosis, evaluation of tumor extension, treatment planning and follow-up. Recently, progresses of computer vision and machine learning have been translated for medical imaging. Aim was to demonstrate the feasibility of an automated 18F-fluoro-ethyl-tyrosine (18F-FET) PET lesion detection and segmentation relying on a full 3D U-Net Convolutional Neural Network (CNN).MethodsAll dynamic 18F-FET PET brain image volumes were temporally realigned to the first dynamic acquisition, coregistered and spatially normalized onto the Montreal Neurological Institute template. Ground truth segmentations were obtained using manual delineation and thresholding (1.3 x background). The volumetric CNN was implemented based on a modified Keras implementation of a U-Net library with 3 layers for the encoding and decoding paths. Dice similarity coefficient (DSC) was used as an accuracy measure of segmentation.ResultsThirty-seven patients were included (26 [70%] in the training set and 11 [30%] in the validation set). All 11 lesions were accurately detected with no false positive, resulting in a sensitivity and a specificity for the detection at the tumor level of 100%. After 150 epochs, DSC reached 0.7924 in the training set and 0.7911 in the validation set. After morphological dilatation and fixed thresholding of the predicted U-Net mask a substantial improvement of the DSC to 0.8231 (+ 4.1%) was noted. At the voxel level, this segmentation led to a 0.88 sensitivity [95% CI, 87.1 to, 88.2%] a 0.99 specificity [99.9 to 99.9%], a 0.78 positive predictive value: [76.9 to 78.3%], and a 0.99 negative predictive value [99.9 to 99.9%].ConclusionsWith relatively high performance, it was proposed the first full 3D automated procedure for segmentation of 18F-FET PET brain images of patients with different gliomas using a U-Net CNN architecture.

Segmentation of Brain Tumor Tissues in HGG and LGG MR Images Using 3D U-net Convolutional Neural Network

The quantitative assessment of tumor extent is necessary for surgical planning, as well as monitoring of tumor growth or shrinkage, and radiotherapy planning. For brain tumors, magnetic resonance imaging (MRI) is used as a standard for diagnosis and prognosis. Manually segmenting brain tumors from 3D MRI volumes is tedious and depends on inter and intra observer variability. In the clinical facilities, a reliable fully automatic brain tumor segmentation method is necessary for the accurate delineation of tumor sub regions. This article presents a 3D U-net Convolutional Neural Network for segmentation of a brain tumor. The proposed method achieves a mean dice score of 0.83, a specificity of 0.80 and a sensitivity of 0.81 for segmenting the whole tumor, and for the tumor core region a mean dice score of 0.76, a specificity of 0.79 and a sensitivity of 0.73. For the enhancing region, the mean dice score is 0.68, a specificity of 0.73 and a sensitivity of 0.77. From the experimental analysis, the proposed U-net model achieved considerably good results compared to the other segmentation models.

Use of 2D U-Net Convolutional Neural Networks for Automated Cartilage and Meniscus Segmentation of Knee MR Imaging Data to Determine Relaxometry and Morphometry.

Purpose To analyze how automatic segmentation translates in accuracy and precision to morphology and relaxometry compared with manual segmentation and increases the speed and accuracy of the work flow that uses quantitative magnetic resonance (MR) imaging to study knee degenerative diseases such as osteoarthritis (OA). Materials and Methods This retrospective study involved the analysis of 638 MR imaging volumes from two data cohorts acquired at 3.0 T: (a) spoiled gradient-recalled acquisition in the steady state T1ρ-weighted images and (b) three-dimensional (3D) double-echo steady-state (DESS) images. A deep learning model based on the U-Net convolutional network architecture was developed to perform automatic segmentation. Cartilage and meniscus compartments were manually segmented by skilled technicians and radiologists for comparison. Performance of the automatic segmentation was evaluated on Dice coefficient overlap with the manual segmentation, as well as by the automatic segmentations' ability to quantify, in a longitudinally repeatable way, relaxometry and morphology. Results The models produced strong Dice coefficients, particularly for 3D-DESS images, ranging between 0.770 and 0.878 in the cartilage compartments to 0.809 and 0.753 for the lateral meniscus and medial meniscus, respectively. The models averaged 5 seconds to generate the automatic segmentations. Average correlations between manual and automatic quantification of T1ρ and T2 values were 0.8233 and 0.8603, respectively, and 0.9349 and 0.9384 for volume and thickness, respectively. Longitudinal precision of the automatic method was comparable with that of the manual one. Conclusion U-Net demonstrates efficacy and precision in quickly generating accurate segmentations that can be used to extract relaxation times and morphologic characterization and values that can be used in the monitoring and diagnosis of OA. © RSNA, 2018 Online supplemental material is available for this article.

Optic disc and cup segmentation methods for glaucoma detection with modification of U-Net convolutional neural network

Glaucoma is the second leading cause of blindness all over the world, with approximately 60 million cases reported worldwide in 2010. If undiagnosed in time, glaucoma causes irreversible damage to the optic nerve leading to blindness. The optic nerve head examination, which involves measurement of cup-to-disc ratio, is considered one of the most valuable methods of structural diagnosis of the disease. Estimation of cup-to-disc ratio requires segmentation of optic disc and optic cup on eye fundus images and can be performed by modern computer vision algorithms. This work presents universal approach for automatic optic disc and cup segmentation, which is based on deep learning, namely, modification of U-Net convolutional neural network. Our experiments include comparison with the best known methods on publicly available databases DRIONS-DB, RIM-ONE v.3, DRISHTI-GS. For both optic disc and cup segmentation, our method achieves quality comparable to current state-of-the-art methods, outperforming them in terms of the prediction time.