Deep Learning-based Segmentation Methods Research Articles

NnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.

Biomedical imaging is a driver of scientific discovery and a core component of medical care and is being stimulated by the field of deep learning. While semantic segmentation algorithms enable image analysis and quantification in many applications, the design of respective specialized solutions is non-trivial and highly dependent on dataset properties and hardware conditions. We developed nnU-Net, a deep learning-based segmentation method that automatically configures itself, including preprocessing, network architecture, training and post-processing for any new task. The key design choices in this process are modeled as a set of fixed parameters, interdependent rules and empirical decisions. Without manual intervention, nnU-Net surpasses most existing approaches, including highly specialized solutions on 23 public datasets used in international biomedical segmentation competitions. We make nnU-Net publicly available as an out-of-the-box tool, rendering state-of-the-art segmentation accessible to a broad audience by requiring neither expert knowledge nor computing resources beyond standard network training.

Breast tumor segmentation in 3D automatic breast ultrasound using Mask scoring R-CNN.

Automatic breast ultrasound (ABUS) imaging has become an essential tool in breast cancer diagnosis since it provides complementary information to other imaging modalities. Lesion segmentation on ABUS is a prerequisite step of breast cancer computer-aided diagnosis (CAD). This work aims to develop a deep learning-based method for breast tumor segmentation using three-dimensional (3D) ABUS automatically. For breast tumor segmentation in ABUS, we developed a Mask scoring region-based convolutional neural network (R-CNN) that consists of five subnetworks, that is, a backbone, a regional proposal network, a region convolutional neural network head, a mask head, and a mask score head. A network block building direct correlation between mask quality and region class was integrated into a Mask scoring R-CNN based framework for the segmentation of new ABUS images with ambiguous regions of interest (ROIs). For segmentation accuracy evaluation, we retrospectively investigated 70 patients with breast tumor confirmed with needle biopsy and manually delineated on ABUS, of which 40 were used for fivefold cross-validation and 30 were used for hold-out test. The comparison between the automatic breast tumor segmentations and the manual contours was quantified by I) six metrics including Dice similarity coefficient (DSC), Jaccard index, 95% Hausdorff distance (HD95), mean surface distance (MSD), residual mean square distance (RMSD), and center of mass distance (CMD); II) Pearson correlation analysis and Bland-Altman analysis. The mean (median) DSC was 85% ± 10.4% (89.4%) and 82.1% ± 14.5% (85.6%) for cross-validation and hold-out test, respectively. The corresponding HD95, MSD, RMSD, and CMD of the two tests was 1.646±1.191 and 1.665±1.129mm, 0.489±0.406 and 0.475±0.371mm, 0.755±0.755 and 0.751±0.508mm, and 0.672±0.612 and 0.665±0.729mm. The mean volumetric difference (mean and±1.96 standard deviation) was 0.47cc ([-0.77, 1.71)) for the cross-validation and 0.23cc ([-0.23 0.69]) for hold-out test, respectively. We developed a novel Mask scoring R-CNN approach for the automated segmentation of the breast tumor in ABUS images and demonstrated its accuracy for breast tumor segmentation. Our learning-based method can potentially assist the clinical CAD of breast cancer using 3D ABUS imaging.

Automatic quantification of myocardium and pericardial fat from coronary computed tomography angiography: a multicenter study.

To develop a deep learning-based method for simultaneous myocardium and pericardial fat quantification from coronary computed tomography angiography (CCTA) for the diagnosis and treatment of cardiovascular disease (CVD). We retrospectively identified CCTA data obtained between May 2008 and July 2018 in a multicenter (six centers) CVD study. The proposed method was evaluated on 422 patients' data by two studies. The first overall study involves training model on CVD patients and testing on non-CVD patients, as well as training on non-CVD patients and testing on CVD patients. The second study was performed using the leave-center-out approach. The method performance was evaluated using Dice similarity coefficient (DSC), Jaccard index (JAC), 95% Hausdorff distance (HD95), mean surface distance (MSD), residual mean square distance (RMSD), and the center of mass distance (CMD). The robustness of the proposed method was tested using the nonparametric Kruskal-Wallis test and post hoc test to assess the equality of distribution of DSC values among different tests. The automatic segmentation achieved a strong correlation with contour (ICC and R > 0.97, p value < 0.001 throughout all tests). The accuracy of the proposed method remained high through all the tests, with the median DSC higher than 0.88 for pericardial fat and 0.96 for myocardium. The proposed method also resulted in mean MSD, RMSD, HD95, and CMD of less than 1.36mm for pericardial fat and 1.00mm for myocardium. The proposed deep learning-based segmentation method enables accurate simultaneous quantification of myocardium and pericardial fat in a multicenter study. • Deep learning-based myocardium and pericardial fat segmentation method tested on 422 patients' coronary computed tomography angiography in a multicenter study. • The proposed method provides segmentations with high volumetric accuracy (ICC and R > 0.97, p value < 0.001) and similar shape as manual annotation by experienced radiologists (median Dice similarity coefficient ≥ 0.88 for pericardial fat and 0.96 for myocardium).

Fast body part segmentation and tracking of neonatal video data using deep learning

Photoplethysmography imaging (PPGI) for non-contact monitoring of preterm infants in the neonatal intensive care unit (NICU) is a promising technology, as it could reduce medical adhesive-related skin injuries and associated complications. For practical implementations of PPGI, a region of interest has to be detected automatically in real time. As the neonates’ body proportions differ significantly from adults, existing approaches may not be used in a straightforward way, and color-based skin detection requires RGB data, thus prohibiting the use of less-intrusive near-infrared (NIR) acquisition. In this paper, we present a deep learning-based method for segmentation of neonatal video data. We augmented an existing encoder-decoder semantic segmentation method with a modified version of the ResNet-50 encoder. This reduced the computational time by a factor of 7.5, so that 30 frames per second can be processed at 960 × 576 pixels. The method was developed and optimized on publicly available databases with segmentation data from adults. For evaluation, a comprehensive dataset consisting of RGB and NIR video recordings from 29 neonates with various skin tones recorded in two NICUs in Germany and India was used. From all recordings, 643 frames were manually segmented. After pre-training the model on the public adult data, parts of the neonatal data were used for additional learning and left-out neonates are used for cross-validated evaluation. On the RGB data, the head is segmented well (82% intersection over union, 88% accuracy), and performance is comparable with those achieved on large, public, non-neonatal datasets. On the other hand, performance on the NIR data was inferior. By employing data augmentation to generate additional virtual NIR data for training, results could be improved and the head could be segmented with 62% intersection over union and 65% accuracy. The method is in theory capable of performing segmentation in real time and thus it may provide a useful tool for future PPGI applications.Graphical This work presents the development of a customized, real-time capable Deep Learning architecture for segmenting of neonatal videos recorded in the intensive care unit. In addition to hand-annotated data, transfer learning is exploited to improve performance.

Deep pyramid local attention neural network for cardiac structure segmentation in two-dimensional echocardiography.

Automatic semantic segmentation in 2D echocardiography is vital in clinical practice for assessing various cardiac functions and improving the diagnosis of cardiac diseases. However, two distinct problems have persisted in automatic segmentation in 2D echocardiography, namely the lack of an effective feature enhancement approach for contextual feature capture and lack of label coherence in category prediction for individual pixels. Therefore, in this study, we propose a deep learning model, called deep pyramid local attention neural network (PLANet), to improve the segmentation performance of automatic methods in 2D echocardiography. Specifically, we propose a pyramid local attention module to enhance features by capturing supporting information within compact and sparse neighboring contexts. We also propose a label coherence learning mechanism to promote prediction consistency for pixels and their neighbors by guiding the learning with explicit supervision signals. The proposed PLANet was extensively evaluated on the dataset of cardiac acquisitions for multi-structure ultrasound segmentation (CAMUS) and sub-EchoNet-Dynamic, which are two large-scale and public 2D echocardiography datasets. The experimental results show that PLANet performs better than traditional and deep learning-based segmentation methods on geometrical and clinical metrics. Moreover, PLANet can complete the segmentation of heart structures in 2D echocardiography in real time, indicating a potential to assist cardiologists accurately and efficiently.

A Quality Control System for Automated Prostate Segmentation on T2-Weighted MRI.

Computer-aided detection and diagnosis (CAD) systems have the potential to improve robustness and efficiency compared to traditional radiological reading of magnetic resonance imaging (MRI). Fully automated segmentation of the prostate is a crucial step of CAD for prostate cancer, but visual inspection is still required to detect poorly segmented cases. The aim of this work was therefore to establish a fully automated quality control (QC) system for prostate segmentation based on T2-weighted MRI. Four different deep learning-based segmentation methods were used to segment the prostate for 585 patients. First order, shape and textural radiomics features were extracted from the segmented prostate masks. A reference quality score (QS) was calculated for each automated segmentation in comparison to a manual segmentation. A least absolute shrinkage and selection operator (LASSO) was trained and optimized on a randomly assigned training dataset (N = 1756, 439 cases from each segmentation method) to build a generalizable linear regression model based on the radiomics features that best estimated the reference QS. Subsequently, the model was used to estimate the QSs for an independent testing dataset (N = 584, 146 cases from each segmentation method). The mean ± standard deviation absolute error between the estimated and reference QSs was 5.47 ± 6.33 on a scale from 0 to 100. In addition, we found a strong correlation between the estimated and reference QSs (rho = 0.70). In conclusion, we developed an automated QC system that may be helpful for evaluating the quality of automated prostate segmentations.

Abstract 854: An artificial intelligence pipeline for imaging mass cytometry data analysis and its application in ovarian cancer prognostic biomarker discovery

Abstract Imaging mass cytometry (IMC) is a powerful platform which enables high-dimensional, single-cell analysis of cell type and state. However, reliable methods used to analyze the IMC data remain to be developed. We seek to build an artificial intelligence (AI)-based analytics pipeline for imaging mass cytometry (IMC) to increase the accuracy of cell segmentation and spatial information extraction and apply the AI pipeline to analyze the IMC data derived from high-grade serous ovarian cancer samples for patient overall survival prediction. Multiplexed spatial analysis of the tumor microenvironment by IMC was performed on 41 formalin fixed paraffin embedded (FFPE) tissue samples obtained from treatment naïve high- grade serous ovarian cancer patients using a panel of 24 metal-tagged antibodies that are specific to tumor, immune and stromal cell markers. IMC data was collected using a a Helios CyTOF instrument equipped with Hyperion Imaging System (Fluidigm). Mask Region-Convolution Neural Network (Mask R-CNN) model was used for cell segmentation. Cell subtypes were derived by iterative phenograph clustering with different marker combinations and different subsets of cells in each iteration. Tumor area was calculated as a thresholded Gaussian blurred image of the density map of the center-of-mass of Keratin positive tumor cells. All cells with center-of-mass located within the tumor region were classified as intratumoral cells. The average cell composition in the nearest neighborhood of each cell type (distance between the center of mass of two cells &amp;lt; 20 μm) in the tumor cell compartment was computed. The results showed that cell segmentation by Mask R-CNN has a higher accuracy than traditional watershed segmentation. Significantly more granzymeB+ CD8+ T cells and CD11b+ Vista+ cells were found in long-term survivors comparing to short-term survivors in both tumor and stromal cell compartments of the tumor microenvironment. Additionally, CD196+, CD45RO+ and CD73+ cell densities in the tumor cell compartment were significantly lower in long-term than short-term survivors. Our results also showed that the mean numbers of CD73+ cells adjacent to Vista- CD4+ T cells, macrophages and B cells were significantly lower in long-term than short-term survivors. The mean number of granzymeB+ CD8+ T cells adjacent to Vista- CD4+ T cells was significantly higher in long-term than short-term survivors. These data demonstrated that the deep learning-based cell segmentation method achieved higher accuracy than the conventional watershed segmentation method. Furthermore, our AI pipeline can automatically extract cell count and cell neighborhood information in tumor. The cell count and neighborhood information could be further employed as features for machine learning to generate predictive biomarkers for ovarian cancer patient survival. Citation Format: Ying Zhu, Jianting Sheng, Sammy Ferri-Borgogno, Tsz-Lun Yeung, Jared K. Burks, Samuel C. Mok, Stephen T. Wong. An artificial intelligence pipeline for imaging mass cytometry data analysis and its application in ovarian cancer prognostic biomarker discovery [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 854.

A Review of Segmentation Algorithms Applied to B-Mode Breast Ultrasound Images: A Characterization Approach

Ultrasound imaging modality is used prominently for breast cancer screening and diagnosis because of its safety, portability, ease of use and low cost. Over the years, computer-assisted algorithms have been used to aid the radiologists for interpreting the ultrasound images. The presence of speckle adversely affects the ultrasound image quality because of which accurate segmentation of tumors has become a challenging task. In the present work, various machine learning (ML) and deep learning (DL) based approaches designed for segmenting breast ultrasound images have been reviewed over the past two decades using a characterization approach in terms of (a) datasets used, (b) pre-processing methods, (c) augmentation methods, (d) segmentation methods and (e) evaluation metrics used for the segmentation algorithms along with their brainstorming diagrams. The review presents the achievements made till date in the design of ML and DL based segmentation methods applied to breast ultrasound images and also highlights the directions in which the future research could be carried out.

CT-based multi-organ segmentation using a 3D self-attention U-net network for pancreatic radiotherapy.

Segmentation of organs-at-risk (OARs) is a weak link in radiotherapeutic treatment planning process because the manual contouring action is labor-intensive and time-consuming. This work aimed to develop a deep learning-based method for rapid and accurate pancreatic multi-organ segmentation that can expedite the treatment planning process. We retrospectively investigated one hundred patients with computed tomography (CT) simulation scanned and contours delineated. Eight OARs including large bowel, small bowel, duodenum, left kidney, right kidney, liver, spinal cord and stomach were the target organs to be segmented. The proposed three-dimensional (3D) deep attention U-Net is featured with a deep attention strategy to effectively differentiate multiple organs. Performance of the proposed method was evaluated using six metrics, including Dice similarity coefficient (DSC), sensitivity, specificity, Hausdorff distance 95% (HD95), mean surface distance (MSD) and residual mean square distance (RMSD). The contours generated by the proposed method closely resemble the ground-truth manual contours, as evidenced by encouraging quantitative results in terms of DSC, sensitivity, specificity, HD95, MSD and RMSD. For DSC, mean values of 0.91±0.03, 0.89±0.06, 0.86±0.06, 0.95±0.02, 0.95±0.02, 0.96±0.01, 0.87±0.05 and 0.93±0.03 were achieved for large bowel, small bowel, duodenum, left kidney, right kidney, liver, spinal cord and stomach, respectively. The proposed method could significantly expedite the treatment planning process by rapidly segmenting multiple OARs. The method could potentially be used in pancreatic adaptive radiotherapy to increase dose delivery accuracy and minimize gastrointestinal toxicity.

Lightweight Learning-Based Automatic Segmentation of Subretinal Blebs on Microscope-Integrated Optical Coherence Tomography Images

Subretinal injections of therapeutics are commonly used to treat ocular diseases. Accurate dosing of therapeutics at target locations is crucial but difficult to achieve using subretinal injections due to leakage, and there is no method available to measure the volume of therapeutics successfully administered to the subretinal location during surgery. Here, we introduce the first automatic method for quantifying the volume of subretinal blebs, using porcine eyes injected with Ringer's lactate solution as samples. Ex vivo animal study. Microscope-integrated optical coherence tomography was used to obtain 3D visualization of subretinal blebs in porcine eyes at Duke Eye Center. Two different injection phases were imaged and analyzed in 15 eyes (30 volumes), selected from a total of 37 eyes. The inclusion/exclusion criteria were set independently from the algorithm-development and testing team. A novel lightweight, deep learning-based algorithm was designed to segment subretinal bleb boundaries. A cross-validation method was used to avoid selection bias. An ensemble-classifier strategy was applied to generate final results for the test dataset. The algorithm performs notably better than 4 other state-of-the-art deep learning-based segmentation methods, achieving an F1 score of 93.86 ± 1.17% and 96.90 ± 0.59% on the independent test data for entry and full blebs, respectively. The proposed algorithm accurately segmented the volumetric boundaries of Ringer's lactate solution delivered into the subretinal space of porcine eyes with robust performance and real-time speed. This is the first step for future applications in computer-guided delivery of therapeutics into the subretinal space in human subjects.

Semantic segmentation using stride spatial pyramid pooling and dual attention decoder

Semantic segmentation is an end-to-end task that requires both semantic and spatial accuracy. It is important for deep learning-based segmentation methods to effectively utilize the high-level feature map whose semantic information is abundant and the low-level feature map whose spatial information is accurate. However, existing segmentation networks typically cannot take full advantage of these two kinds of feature maps, leading to inferior performance. This paper attempts to overcome this challenge by introducing two novel structures. On the one hand, we propose a structure called stride spatial pyramid pooling (SSPP) to capture multiscale semantic information from the high-level feature map. Compared with existing pyramid pooling methods based on the atrous convolution, the SSPP structure is able to gather more information from the high-level feature map with faster inference speed, which improves the utilization efficiency of the high-level feature map significantly. On the other hand, we propose a dual attention decoder consisting of a channel attention branch and a spatial attention branch to make full use of the high- and low-level feature maps simultaneously. The dual attention decoder can result in a more “semantic” low-level feature map and a high-level feature map with more accurate spatial information, which bridges the gap between these two kinds of feature maps and benefits their fusion. We evaluate the proposed model on several publicly available semantic image segmentation benchmarks including PASCAL VOC 2012, Cityscapes and COCO-Stuff. The qualitative and quantitative results demonstrate that our method can achieve the state-of-the-art performance.

4252 Automated Fetal Brain Volumetry on Clinical Fetal MRI Using Convolutional Neural Network

OBJECTIVES/GOALS: We seek to develop an automated deep learning-based method for segmentation and volumetric quantification of the fetal brain on T2-weighted fetal MRIs. We will evaluate the performance of the algorithm by comparing it to gold standard manual segmentations. The method will be used to create a normative sample of brain volumes across gestational ages. METHODS/STUDY POPULATION: We will adapt a U-Net convolutional neural network architecture for fetal brain MRIs using 3D volumes. After re-sampling 2D fetal brain acquisitions to 3mm3 3D volumes using linear interpolation, the network will be trained to perform automated brain segmentation on 40 randomly-sampled, normal fetal brain MRI scans of singleton pregnancies. Training will be performed in 3 acquisition planes (axial, coronal, sagittal). Performance will be evaluated on 10 test MRIs (in 3 acquisition planes, 30 total test samples) using Dice scores, compared to radiologists’ manual segmentations. The algorithm’s performance on measuring total brain volume will also be evaluated. RESULTS/ANTICIPATED RESULTS: Based on the success of prior U-net architectures for volumetric segmentation tasks in medical imaging (e.g. Duong et al., 2019), we anticipate that the convolutional neural network will accurately provide segmentations and associated volumetry of fetal brains in fractions of a second. We anticipate median Dice scores greater than 0.8 across our test sample. Once validated, the method will retrospectively generate a normative database of over 1500 fetal brain volumes across gestational ages (18 weeks to 30 weeks) collected at our institution. DISCUSSION/SIGNIFICANCE OF IMPACT: Quantitative estimates of brain volume, and deviations from normative data, would be a major advancement in objective clinical assessments of fetal MRI. Such data can currently only be obtained through laborious manual segmentations; automated deep learning methods have the potential to reduce the time and cost of this process.

Pelvic multi-organ segmentation on cone-beam CT for prostate adaptive radiotherapy.

The purpose of this study is to develop a deep learning-based approach to simultaneously segment five pelvic organs including prostate, bladder, rectum, left and right femoral heads on cone-beam CT (CBCT), as required elements for prostate adaptive radiotherapy planning. We propose to utilize both CBCT and CBCT-based synthetic MRI (sMRI) for the segmentation of soft tissue and bony structures, as they provide complementary information for pelvic organ segmentation. CBCT images have superior bony structure contrast and sMRIs have superior soft tissue contrast. Prior to segmentation, sMRI was generated using a cycle-consistent adversarial networks (CycleGAN), which was trained using paired CBCT-MR images. To combine the advantages of both CBCT and sMRI, we developed a cross-modality attention pyramid network with late feature fusion. Our method processes CBCT and sMRI inputs separately to extract CBCT-specific and sMRI-specific features prior to combining them in a late-fusion network for final segmentation. The network was trained and tested using 100 patients' datasets, with each dataset including the CBCT and manual physician contours. For comparison, we trained another two networks with different network inputs and architectures. The segmentation results were compared to manual contours for evaluations. For the proposed method, dice similarity coefficients and mean surface distances between the segmentation results and the ground truth were 0.96±0.03, 0.65±0.67mm; 0.91±0.08, 0.93±0.96mm; 0.93±0.04, 0.72±0.61mm; 0.95±0.05, 1.05±1.40mm; and 0.95±0.05, 1.08±1.48mm for bladder, prostate, rectum, left and right femoral heads, respectively. As compared to the other two competing methods, our method has shown superior performance in terms of the segmentation accuracy. We developed a deep learning-based segmentation method to rapidly and accurately segment five pelvic organs simultaneously from daily CBCTs. The proposed method could be used in the clinic to support rapid target and organs-at-risk contouring for prostate adaptive radiation therapy.

Promising crack segmentation method based on gated skip connection

This Letter proposes a promising deep learning-based method for crack segmentation based on gated skip connection. The proposed gated skip connection enables the decoder layers to promote crack-aware feature representations from the encoder layers by applying high weights on the crack-relevant features that come from the encoder layers and lower weights for irrelevant features. Unlike the related methods, the authors do not apply any pre-processing or refinement steps to improve the crack segmentation results. The proposed method beats the state-of-the-art methods with an open benchmark database (IoU of 87.5).

NDNet: Narrow While Deep Network for Real-Time Semantic Segmentation

The rapid development of autonomous driving in recent years presents many challenges for scene understanding. As an essential step towards scene understanding, semantic segmentation has received increased attention in the past few years. Although deep learning based approaches have achieved great success in improving the segmentation accuracy, most of them suffer from an inefficiency problem and can hardly be applied to real-time applications. In this paper, we analyze the computational cost of Convolutional Neural Network (CNN) and find that the inefficiency of CNNs is mainly caused by their wide structure rather than deep structure. In addition, the success of pruning based model compression methods proves that there are many redundant channels in CNNs. Thus, we design a narrow while deep backbone network to improve the efficiency of semantic segmentation. By casting our network to fully convolutional network (FCN32) segmentation architecture, the basic structure of most segmentation methods, we achieve 61.5% mIoU on Cityscapes validation dataset with only 4.2G floating-point operations (FLOPs) on 1024×2048 inputs, which already outperforms one of the earliest real-time deep learning based segmentation methods: ENet (58.3% mIoU, 3.8G FLOPs on 640×360 inputs). By further refining the output resolution of our network to the 1/8 of the input resolution with a simple encoder-decoder structure, we achieve 65.3% mIoU on Cityscapes test set with 14.0G FLOPs and 39.9 frames per second (FPS) on Titan X card. We have made our model publicly available at https://github.com/zgyang-hnu/NDNet.

3D-BoxSup: Positive-Unlabeled Learning of Brain Tumor Segmentation Networks From 3D Bounding Boxes.

Accurate segmentation is an essential task when working with medical images. Recently, deep convolutional neural networks achieved a state-of-the-art performance for many segmentation benchmarks. Regardless of the network architecture, the deep learning-based segmentation methods view the segmentation problem as a supervised task that requires a relatively large number of annotated images. Acquiring a large number of annotated medical images is time consuming, and high-quality segmented images (i.e., strong labels) crafted by human experts are expensive. In this paper, we have proposed a method that achieves competitive accuracy from a “weakly annotated” image where the weak annotation is obtained via a 3D bounding box denoting an object of interest. Our method, called “3D-BoxSup,” employs a positive-unlabeled learning framework to learn segmentation masks from 3D bounding boxes. Specially, we consider the pixels outside of the bounding box as positively labeled data and the pixels inside the bounding box as unlabeled data. Our method can suppress the negative effects of pixels residing between the true segmentation mask and the 3D bounding box and produce accurate segmentation masks. We applied our method to segment a brain tumor. The experimental results on the BraTS 2017 dataset (Menze et al., 2015; Bakas et al., 2017a,b,c) have demonstrated the effectiveness of our method.

Trophectoderm segmentation in human embryo images via inceptioned U-Net.

Trophectoderm (TE) is one of the main components of a day-5 human embryo (blastocyst) that correlates with the embryo's quality. Precise segmentation of TE is an important step toward achieving automatic human embryo quality assessment based on morphological image features. Automatic segmentation of TE, however, is a challenging task and previous work on this is quite limited. In this paper, four fully convolutional deep models are proposed for accurate segmentation of trophectoderm in microscopic images of the human blastocyst. In addition, a multi-scaled ensembling method is proposed that aggregates five models trained at various scales offering trade-offs between the quantity and quality of the spatial information. Furthermore, synthetic embryo images are generated for the first time to address the lack of data in training deep learning models. These synthetically generated images are proven to be effective to fill the generalization gap in deep learning when limited data is available for training. Experimental results confirm that the proposed models are capable of segmenting TE regions with an average Precision, Recall, Accuracy, Dice Coefficient and Jaccard Index of 83.8%, 90.1%, 96.9%, 86.61% and 76.71%, respectively. Particularly, the proposed Inceptioned U-Net model outperforms state-of-the-art by 10.3% in Accuracy, 9.3% in Dice Coefficient and 13.7% inJaccard Index. Further experiments are conducted to highlight the effectiveness of the proposed models compared to some recent deep learning based segmentation methods.

Deep learning-based method for SEM image segmentation in mineral characterization, an example from Duvernay Shale samples in Western Canada Sedimentary Basin

Texture-based feature extraction and object segmentation are challenging in image processing. In this study, the U-Net architecture developed for biomedical image analysis was used to evaluate geologic characteristics depicted within scanning electron microscope (SEM) images of shale samples. With a revised weight function, the U-Net architecture allowed for effective discrimination of clay aggregates mixed with matrix mineral particles and organic matter (OM). In training, a local variability weight based on spatial statistics was used to enhance the contrast between features across boundary in the loss function of U-Net system optimization, thereby improving the ability of U-Net to distinguish the geologic features specific to our research needs. The Tensorflow neural network library was used to create semantic segmentation and feature extraction models in mineral identification. In the application example of the Devonian Duvernay shale study, we prepared 8000 randomly sliced image cuts (256 × 256 pixels) from four masked image tiles (6144 × 6144 pixels) with tagged feature objects, among which 6400 are for training and the remaining 1600 held image slices for validation. In the validation, the average of intersection over union (IOU) reaches 91.7%. The trained model approved by validation was used for clay aggregate segmentation and mineral classification. Three hundred SEM image tiles of source rock samples from different maturities in the Duvernay Formation were processed using the proposed workflow. The results show that the clay aggregates are clearly separated from other matrix mineral particles with acceptable boundaries, although both exhibit indistinguishable grey-level pixels. This approach demonstrates that texture-based deep learning feature extraction is feasible, cost-effective and timely, and can help geoscientists gain new insights by quantitatively analyzing specific geological characteristics and features.

Deep learning-based segmentation of malignant pleural mesothelioma tumor on computed tomography scans: application to scans demonstrating pleural effusion.

Tumor volume is a topic of interest for the prognostic assessment, treatment response evaluation, and staging of malignant pleural mesothelioma. Many mesothelioma patients present with, or develop, pleural fluid, which may complicate the segmentation of this disease. Deep convolutional neural networks (CNNs) of the two-dimensional U-Net architecture were trained for segmentation of tumor in the left and right hemithoraces, with the networks initialized through layers pretrained on ImageNet. Networks were trained on a dataset of 5230 axial sections from 154 CT scans of 126 mesothelioma patients. A test set of 94 CT sections from 34 patients, who all presented with both tumor and pleural effusion, in addition to a more general test set of 130 CT sections from 43 patients, were used to evaluate segmentation performance of the deep CNNs. The Dice similarity coefficient (DSC), average Hausdorff distance, and bias in predicted tumor area were calculated through comparisons with radiologist-provided tumor segmentations on the test sets. The present method achieved a median DSC of 0.690 on the tumor and effusion test set and achieved significantly higher performance on both test sets when compared with a previous deep learning-based segmentation method for mesothelioma.

Knee menisci segmentation and relaxometry of 3D ultrashort echo time cones MR imaging using attention U-Net with transfer learning.

To develop a deep learning-based method for knee menisci segmentation in 3D ultrashort echo time (UTE) cones MR imaging, and to automatically determine MR relaxation times, namely the T1, T1ρ , and parameters, which can be used to assess knee osteoarthritis (OA). Whole knee joint imaging was performed using 3D UTE cones sequences to collect data from 61 human subjects. Regions of interest (ROIs) were outlined by 2 experienced radiologists based on subtracted T1ρ -weighted MR images. Transfer learning was applied to develop 2D attention U-Net convolutional neural networks for the menisci segmentation based on each radiologist's ROIs separately. Dice scores were calculated to assess segmentation performance. Next, the T1, T1ρ , relaxations, and ROI areas were determined for the manual and automatic segmentations, then compared. The models developed using ROIs provided by 2 radiologists achieved high Dice scores of 0.860 and 0.833, while the radiologists' manual segmentations achieved a Dice score of 0.820. Linear correlation coefficients for the T1, T1ρ , and relaxations calculated using the automatic and manual segmentations ranged between 0.90 and 0.97, and there were no associated differences between the estimated average meniscal relaxation parameters. The deep learning models achieved segmentation performance equivalent to the inter-observer variability of 2 radiologists. The proposed deep learning-based approach can be used to efficiently generate automatic segmentations and determine meniscal relaxations times. The method has the potential to help radiologists with the assessment of meniscal diseases, such as OA.