3D Fully Convolutional Network Research Articles

A skeleton context-aware 3D fully convolutional network for abdominal artery segmentation.

This paper aims to propose a deep learning-based method for abdominal artery segmentation. Blood vessel structure information is essential to diagnosis and treatment. Accurate blood vessel segmentation is critical to preoperative planning. Although deep learning-based methods perform well on large organs, segmenting small organs such as blood vessels is challenging due to complicated branching structures and positions. We propose a 3D deep learning network from a skeleton context-aware perspective to improve segmentation accuracy. In addition, we propose a novel 3D patch generation method which could strengthen the structural diversity of a training data set. The proposed method segments abdominal arteries from an abdominal computed tomography (CT) volume using a 3D fully convolutional network (FCN). We add two auxiliary tasks to the network to extract the skeleton context of abdominal arteries. In addition, our skeleton-based patch generation (SBPG) method further enables the FCN to segment small arteries. SBPG generates a 3D patch from a CT volume by leveraging artery skeleton information. These methods improve the segmentation accuracies of small arteries. We used 20 cases of abdominal CT volumes to evaluate the proposed method. The experimental results showed that our method outperformed previous segmentation accuracies. The averaged precision rate, recall rate, and F-measure were 95.5%, 91.0%, and 93.2%, respectively. Compared to a baseline method, our method improved 1.5% the averaged recall rate and 0.7% the averaged F-measure. We present a skeleton context-aware 3D FCN to segment abdominal arteries from an abdominal CT volume. In addition, we propose a 3D patch generation method. Our fully automated method segmented most of the abdominal artery regions. The method produced competitive segmentation performance compared to previous methods.

Fast 3D Liver Segmentation Using a Trained Deep Chan-Vese Model

This paper introduces an approach for 3D organ segmentation that generalizes in multiple ways the Chan-Vese level set method. Chan-Vese is a segmentation method that simultaneously evolves a level set while fitting locally constant intensity models for the interior and exterior regions. First, its simple length-based regularization is replaced with a learned shape model based on a Fully Convolutional Network (FCN). We show how to train the FCN and introduce data augmentation methods to avoid overfitting. Second, two 3D variants of the method are introduced, one based on a 3D U-Net that makes global shape modifications and one based on a 3D FCN that makes local refinements. These two variants are integrated in a full 3D organ segmentation approach that is capable and efficient in dealing with the large size of the 3D volumes with minimal overfitting. Experiments on liver segmentation on a standard benchmark dataset show that the method obtains 3D segmentation results competitive with the state of the art while being very fast and having a small number of trainable parameters.

APRNet: A 3D Anisotropic Pyramidal Reversible Network With Multi-Modal Cross-Dimension Attention for Brain Tissue Segmentation in MR Images.

Brain tissue segmentation in multi-modal magnetic resonance (MR) images is significant for the clinical diagnosis of brain diseases. Due to blurred boundaries, low contrast, and intricate anatomical relationships between brain tissue regions, automatic brain tissue segmentation without prior knowledge is still challenging. This paper presents a novel 3D fully convolutional network (FCN) for brain tissue segmentation, called APRNet. In this network, we first propose a 3D anisotropic pyramidal convolutional reversible residual sequence (3DAPC-RRS) module to integrate the intra-slice information with the inter-slice information without significant memory consumption; secondly, we design a multi-modal cross-dimension attention (MCDA) module to automatically capture the effective information in each dimension of multi-modal images; then, we apply 3DAPC-RRS modules and MCDA modules to a 3D FCN with multiple encoded streams and one decoded stream for constituting the overall architecture of APRNet. We evaluated APRNet on two benchmark challenges, namely MRBrainS13 and iSeg-2017. The experimental results show that APRNet yields state-of-the-art segmentation results on both benchmark challenge datasets and achieves the best segmentation performance on the cerebrospinal fluid region. Compared with other methods, our proposed approach exploits the complementary information of different modalities to segment brain tissue regions in both adult and infant MR images, and it achieves the average Dice coefficient of 87.22% and 93.03% on the MRBrainS13 and iSeg-2017 testing data, respectively. The proposed method is beneficial for quantitative brain analysis in the clinical study, and our code is made publicly available.

Vessel segmentation from volumetric images: a multi-scale double-pathway network with class-balanced loss at the voxel level.

Vessel segmentation from volumetric medical images is becoming an essential pre-step in aiding the diagnosis, guiding the therapy, and patient management for vascular-related diseases. Deep learning-based methods have drawn many attentions, but most of them did not fully utilize the multi-scale spatial information of vessels. To address this shortcoming, we propose a multi-scale network similar to the well-known multi-scale DeepMedic. It also includes a double-pathway architecture and a class-balanced loss at the voxel level (MDNet-Vb) to achieve both the computation efficiency and segmentation accuracy. The proposed network consists two parallel pathways to learn the multi-scale vessel morphology. Specifically, the pathway with a normal resolution uses three-dimensional (3D) U-Net fed with small inputs to learn the local details with relatively small storage and time consumption. The pathway with a low-resolution employs 3D fully convolutional network (FCN) fed with downsampled large inputs to learn the overall spatial relationships between vessels and adjacent tissues, and the morphological information of large vessels. To cope with the class-imbalanced issue in vessel segmentation, we propose a class-balanced loss at the voxel level with uniform sampling strategy. The class-balanced loss at the voxel level re-balances the loss function with a coefficient that is inversely proportional to the normalized effective number at the voxel level of each class. The uniform sampling strategy extracts training data by sampling uniformly from two classes in every epoch. Our MDNet-Vb outperforms several state-of-the-art methods including ResNet, DenseNet, 3D U-Net, V-Net, and DeepMedic with the highest dice coefficients of 72.91% and 69.32% on cardiac computed tomography angiography (CTA) dataset and cerebral magnetic resonance angiography (MRA) dataset, respectively. Among four different double-pathway networks, our network (3D U-Net+3D FCN) not only has the fewest training parameters and shortest training time, but also gets competitive dice coefficients on both the CTA and MRA datasets. Compared with classical losses, our class-balanced focal loss (FL-Vb) and dice coefficient loss at the voxel level (Dsc-Vb) alleviates class imbalanced issue by improving both the sensitivity and dice coefficient on the CTA and MRA datasets. Moreover, simultaneously training on two datasets shows that our method has the highest dice coefficient of 73.06% and 65.40% on CTA and MRA datasets, respectively, outperforming the commonly used methods, such as U-Net and DeepMedic, which demonstrates the generalization potential of our network for segmenting different blood vessels. Our MDNet-Vb method demonstrates its superiority over other state-of-the-art methods, on both cardiac CTA and cerebral MRA datasets. For the network architecture, the MDNet-Vb combined the 3D U-Net and 3D FCN, which dramatically reduces the network parameters yet maintains the segmentation accuracy. The class-balanced loss at the voxel level further improves accuracy by properly alleviating the class-imbalanced issue between different classes. In summary, MDNet-Vb is promising for vessel segmentation from various volumetric medical images.

STA3D: Spatiotemporally attentive 3D network for video saliency prediction

• Attention guiding is significant for video saliency prediction based on 3D CNN. • A spatiotemporally attentive 3D CNN for robust video saliency prediction is proposed. • An adaptive upsampling module for refining spatial features is proposed. • A frame-wise attention module for propagating temporal features is proposed. • The effectiveness of the proposed method is comprehensively evaluated. 3D fully convolutional networks (FCN), which jointly leverage the spatial and temporal cues, have achieved great success in video saliency prediction. However, they still have limitations in some challenging cases, e.g. fixation shift. To address this issue, we propose a SpatioTemporally Attentive 3D Network (STA3D) to selectively propagate the significant temporal features and refine the spatial features in 3D FCN for video saliency prediction. Extensive experiments on three standard datasets demonstrate the superiority of the proposed model against the state-of-the-art.

Label cleaning and propagation for improved segmentation performance using fully convolutional networks.

In recent years, fully convolutional networks (FCNs) have been applied to various medical image segmentation tasks. However, it is difficult to generate a large amount of high-quality annotation data to train FCNs for medical image segmentation. Thus, it is desired to achieve high segmentation performances even from incomplete training data. We aim to evaluate performance of FCNs to clean noises and interpolate labels from noisy and sparsely given label images. To evaluate the label cleaning and propagation performance of FCNs, we used 2D and 3D FCNs to perform volumetric brain segmentation from magnetic resonance image volumes, based on network training on incomplete training datasets from noisy and sparse annotation. The experimental results using pseudo-incomplete training data showed that both 2D and 3D FCNs could provide improved segmentation results from the incomplete training data, especially by using three orthogonal annotation images for network training. This paper presented a validation for label cleaning and propagation based on FCNs. FCNs might have the potential to achieve improved segmentation performances even from sparse annotation data including possible noises by manual annotation, which can be an important clue to more efficient annotation.

Brain segmentation based on multi-atlas and diffeomorphism guided 3D fully convolutional network ensembles

In this study, we proposed and validated a multi-atlas and diffeomorphism guided 3D fully convolutional network (FCN) ensemble model (M-FCN) for segmenting brain anatomical regions of interest (ROIs) from structural magnetic resonance images (MRIs). A novel multi-atlas and diffeomorphism based encoding block and ROI patches with adaptive sizes were used. In the multi-atlas and diffeomorphism based encoding block, both MRI intensity profiles and expert priors from deformed atlases were encoded and fed to the proposed FCN. Utilizing patches with adaptive sizes enabled more efficient network training and testing. To incorporate both local and global contextual information of a specific ROI, we employed a long skip connection between the layer of the encoding block and the layer of the encoding-decoding block. To relieve over-fitting of the proposed FCN model on the training data, we adopted an ensemble strategy in the learning procedure. Systematic evaluations were performed on two brain MRI datasets, aiming respectively at segmenting 14 subcortical and ventricular structures and 54 whole-brain ROIs. Compared with two state-of-the-art segmentation methods including a multi-atlas based segmentation method and an existing 3D FCN segmentation model, the proposed method exhibited superior segmentation performance.

Automatic segmentation and location learning of neonatal cerebral ventricles in 3D ultrasound data combining CNN and CPPN

Preterm neonates are highly likely to suffer from ventriculomegaly, a dilation of the Cerebral Ventricular System (CVS). This condition can develop into life-threatening hydrocephalus and is correlated with future neuro-developmental impairments. Consequently, it must be detected and monitored by physicians. In clinical routing, manual 2D measurements are performed on 2D ultrasound (US) images to estimate the CVS volume but this practice is imprecise due to the unavailability of 3D information. A way to tackle this problem would be to develop automatic CVS segmentation algorithms for 3D US data. In this paper, we investigate the potential of 2D and 3D Convolutional Neural Networks (CNN) to solve this complex task and propose to use Compositional Pattern Producing Network (CPPN) to enable Fully Convolutional Networks (FCN) to learn CVS location. Our database was composed of 25 3D US volumes collected on 21 preterm nenonates at the age of 35.8±1.6 gestational weeks. We found that the CPPN enables to encode CVS location, which increases the accuracy of the CNNs when they have few layers. Accuracy of the 2D and 3D FCNs reached intraobserver variability (IOV) in the case of dilated ventricles with Dice of 0.893±0.008 and 0.886±0.004 respectively (IOV = 0.898±0.008) and with volume errors of 0.45±0.42 cm3 and 0.36±0.24 cm3 respectively (IOV = 0.41±0.05 cm3). 3D FCNs were more accurate than 2D FCNs in the case of normal ventricles with Dice of 0.797±0.041 against 0.776±0.038 (IOV = 0.816±0.009) and volume errors of 0.35±0.29 cm3 against 0.35±0.24 cm3 (IOV = 0.2±0.11 cm3). The best segmentation time of volumes of size 320×320×320 was obtained by a 2D FCN in 3.5±0.2 s.

Machine Segmentation of Pelvic Anatomy in MRI-Assisted Radiosurgery (MARS) for Prostate Cancer Brachytherapy

To investigate machine segmentation of pelvic anatomy in magnetic resonance imaging (MRI)-assisted radiosurgery (MARS) for prostate cancer using prostate brachytherapy MRIs acquired with different pulse sequences and image contrasts. Two hundred 3-dimensional (3D) preimplant and postimplant prostate brachytherapy MRI scans were acquired with a T2-weighted sequence, a T2/T1-weighted sequence, or a T1-weighted sequence. One hundred twenty deep machine learning models were trained to segment the prostate, seminal vesicles, external urinary sphincter, rectum, and bladder using the MRI scans acquired with T2-weighted and T2/T1-weighted image contrast. The deep machine learning models consisted of 18 fully convolutional networks (FCNs) with different convolutional encoders. Both 2-dimensional and 3D U-Net FCNs were constructed for comparison. Six objective functions were investigated: cross-entropy, Jaccard distance, focal loss, and 3 variations of Tversky distance. The performance of the models was compared using similarity metrics, including pixel accuracy, Jaccard index, Dice similarity coefficient (DSC), 95% Hausdorff distance, relative volume difference, Matthews correlation coefficient, precision, recall, and average symmetrical surface distance. We selected the highest-performing architecture and investigated how the amount of training data, use of skip connections, and data augmentation affected segmentation performance. In addition, we investigated whether segmentation on T1-weighted MRI was possible with FCNs trained on only T2-weighted and T2/T1-weighted image contrast. Overall, an FCN with a DenseNet201 encoder trained via cross-entropy minimization yielded the highest combined segmentation performance. For the 53 3D test MRI scans acquired with T2-weighted or T2/T1-weighted image contrast, the DSCs of the prostate, external urinary sphincter, seminal vesicles, rectum, and bladder were 0.90 ± 0.04, 0.70 ± 0.15, 0.80 ± 0.12, 0.91 ± 0.06, and 0.96 ± 0.04, respectively, after model fine-tuning. For the 5 T1-weighted images, the DSCs of these organs were 0.82 ± 0.07, 0.17 ± 0.15, 0.46 ± 0.21, 0.87 ± 0.06, and 0.88 ± 0.05, respectively. Machine segmentation of the prostate and surrounding anatomy on 3D MRIs acquired with different pulse sequences for MARS low-dose-rate prostate brachytherapy is possible with a single FCN.

AdaEn-Net: An ensemble of adaptive 2D–3D Fully Convolutional Networks for medical image segmentation

Fully Convolutional Networks (FCNs) have emerged as powerful segmentation models but are usually designed manually, which requires extensive time and can result in large and complex architectures. There is a growing interest to automatically design efficient architectures that can accurately segment 3D medical images. However, most approaches either do not fully exploit volumetric information or do not optimize the model’s size. To address these problems, we propose a self-adaptive 2D–3D ensemble of FCNs called AdaEn-Net for 3D medical image segmentation that incorporates volumetric data and adapts to a particular dataset by optimizing both the model’s performance and size. The AdaEn-Net consists of a 2D FCN that extracts intra-slice information and a 3D FCN that exploits inter-slice information. The architecture and hyperparameters of the 2D and 3D architectures are found through a multiobjective evolutionary based algorithm that maximizes the expected segmentation accuracy and minimizes the number of parameters in the network. The main contribution of this work is a model that fully exploits volumetric information and automatically searches for a high-performing and efficient architecture. The AdaEn-Net was evaluated for prostate segmentation on the PROMISE12 Grand Challenge and for cardiac segmentation on the MICCAI ACDC challenge. In the first challenge, the AdaEn-Net ranks 9 out of 297 submissions and surpasses the performance of an automatically-generated segmentation network while producing an architecture with 13× fewer parameters. In the second challenge, the proposed model is ranked within the top 8 submissions and outperforms an architecture designed with reinforcement learning while having 1.25× fewer parameters.

Automatic Segmentation of Individual Tooth in Dental CBCT Images From Tooth Surface Map by a Multi-Task FCN

Accurate and automatic segmentation of individual tooth is critical for computer-aided analysis towards clinical decision support and treatment planning. Three-dimensional reconstruction of individual tooth after the segmentation also plays an important role in simulation in digital orthodontics. However, it is difficult to automatically segment individual tooth in cone beam computed tomography (CBCT) images due to the blurring boundaries of neighboring teeth and the similar intensities between teeth and mandible bone. In this work, we propose the use of a multi-task 3D fully convolutional network (FCN) and marker-controlled watershed transform (MWT) to segment individual tooth. The multi-task FCN learns to simultaneously predict the probability of tooth region and the probability of tooth surface. Through the combination of the tooth probability gradient map and the surface probability map as the input image, MWT is used to automatically separate and segment individual tooth. Twenty-five dental CBCT scans are used in the study. The average Dice similarity coefficient, Jaccard index, and relative volume difference are 0.936 (±0.012), 0.881 (±0.019), and 0.072 (±0.027), respectively, and the average symmetric surface distance is 0.363 (±0.145) mm for our method. The experimental results demonstrate that the multi-task 3D FCN combined with MWT can segment individual tooth of various types in dental CBCT images.

Synthesizing diffusion tensor imaging from functional MRI using fully convolutional networks

PurposeMedical image synthesis can simulate a target modality of interest based on existing modalities and has the potential to save scanning time while contributing to efficient data collection. This study proposed a three-dimensional (3D) deep learning architecture based on a fully convolutional network (FCN) to synthesize diffusion-tensor imaging (DTI) from resting-state functional magnetic resonance imaging (fMRI).Methods: fMRI signals derived from white matter (WM) exist and can be used for assessing WM alterations. We constructed an initial functional correlation tensor image using the correlation patterns of adjacent fMRI voxels as one input to the FCN. We considered T1-weighted images as an additional input to provide an algorithm with the structural information needed to synthesize DTI. Our architecture was trained and tested using a large-scale open database dataset (training n = 648; testing n = 293). ResultsThe average correlation value between synthesized and actual diffusion tensors for 38 WM regions was 0.808, which significantly improves upon an existing study (r = 0.480). We also validated our approach using two open databases. Our proposed method showed a higher correlation with the actual diffusion tensor than the conventional machine-learning method for many WM regions. ConclusionsOur method synthesized DTI images from fMRI images using a 3D FCN architecture. We hope to expand our method of synthesizing various other imaging modalities from a single image source.

Deeply supervised 3D fully convolutional networks with group dilated convolution for automatic MRI prostate segmentation.

Reliable automated segmentation of the prostate is indispensable for image-guided prostate interventions. However, the segmentation task is challenging due to inhomogeneous intensity distributions, variation in prostate anatomy, among other problems. Manual segmentation can be time-consuming and is subject to inter- and intraobserver variation. We developed an automated deep learning-based method to address this technical challenge. We propose a three-dimensional (3D) fully convolutional networks (FCN) with deep supervision and group dilated convolution to segment the prostate on magnetic resonance imaging (MRI). In this method, a deeply supervised mechanism was introduced into a 3D FCN to effectively alleviate the common exploding or vanishing gradients problems in training deep models, which forces the update process of the hidden layer filters to favor highly discriminative features. A group dilated convolution which aggregates multiscale contextual information for dense prediction was proposed to enlarge the effective receptive field of convolutional neural networks, which improve the prediction accuracy of prostate boundary. In addition, we introduced a combined loss function including cosine and cross entropy, which measures similarity and dissimilarity between segmented and manual contours, to further improve the segmentation accuracy. Prostate volumes manually segmented by experienced physicians were used as a gold standard against which our segmentation accuracy was measured. The proposed method was evaluated on an internal dataset comprising 40 T2-weighted prostate MR volumes. Our method achieved a Dice similarity coefficient (DSC) of 0.86±0.04, a mean surface distance (MSD) of 1.79±0.46mm, 95% Hausdorff distance (95%HD) of 7.98±2.91mm, and absolute relative volume difference (aRVD) of 15.65±10.82. A public dataset (PROMISE12) including 50 T2-weighted prostate MR volumes was also employed to evaluate our approach. Our method yielded a DSC of 0.88±0.05, MSD of 1.02±0.35mm, 95% HD of 9.50±5.11mm, and aRVD of 8.93±7.56. We developed a novel deeply supervised deep learning-based approach with a group dilated convolution to automatically segment the MRI prostate, demonstrated its clinical feasibility, and validated its accuracy against manual segmentation. The proposed technique could be a useful tool for image-guided interventions in prostate cancer.

Coronary Arteries Segmentation Based on 3D FCN With Attention Gate and Level Set Function

Coronary heart disease is one of the most serious health problems in the world nowadays. By segmenting and examining the coronary arteries in medical images, we can find artery stenosis and plaque, which are the main causes of this certain disease. Segmenting coronary arteries manually is time-consuming and subjective, and the traditional segmentation method requires a good initial point, which is also difficult to apply for the 3D coronary computed tomography angiography (CTA) data. In this paper, we propose a joint framework for coronary CTA segmentation based on deep learning and traditional level set method. A 3D fully convolutional network (FCN) is used to learn the 3D semantic features of coronary arteries, which provides an excellent initial point for the traditional level set. Moreover, an attention gate is added to the entire network, aiming to enhance the vessels and suppress irrelevant regions. The output of 3D FCN with attention gate is optimized by the level set to smooth the boundary to better fit the ground truth segmentation. The proposed algorithm is evaluated by the Jaccard index (JI) and the Dice similarity coefficient (DSC) scores. The experimental results show that the proposed framework provides significant better segmentation results than the vanilla 3D FCN intuitively and quantitively.

Multiple Organ Localization in CT Image Using Triple-Branch Fully Convolutional Networks

Organ localization is an essential prerequisite for many computed tomography (CT) image analysis tasks such as organ segmentation, lesion detection, and image registration. However, it is a challenging problem due to various causes such as the low-contrast organ boundaries, the large variations of organ appearance, the truncated organs, and the abnormalities. In this work, we propose an automatic method for multiple organ localization in CT image using a novel triple-branch fully convolutional network (FCN). First, the input CT image is extended to a three-channel image that contains both the original image and the enhanced organ structures using a density enhancement filter. Then, the extended three-channel image is fed to a 3D triple-branch FCN to parallelly predict the organs' presence probability curves along axial, coronal and sagittal directions. Finally, the bounding boxes of the target organs are computed based on these presence probability curves. The triple-branch FCN is fully implemented in 3D manner, thus it can take full advantage of the spatial context information in CT image to perform accurate organ localization. Only several basic convolutional neural networks (ConvNet) operations (such as convolution, pooling and softmax regression) are involved in this triple-branch FCN, thus it is easy-implemented and flexible to serve as a preprocessing module in other medical image analysis tasks. By enhancing the structural information in the CT image using the density enhancement filter, the performance of the triple-branch FCN can be further improved. The proposed method is evaluated on a challenging public dataset with 201 clinical abdominal/torso CT images to locate 11 body organs. As the experimental results showed, the proposed method achieves higher localization accuracy than the current state-of-the-art methods.

Change Detection in Hyperspectral Images Using Recurrent 3D Fully Convolutional Networks

Hyperspectral change detection (CD) can be effectively performed using deep-learning networks. Although these approaches require qualified training samples, it is difficult to obtain ground-truth data in the real world. Preserving spatial information during training is difficult due to structural limitations. To solve such problems, our study proposed a novel CD method for hyperspectral images (HSIs), including sample generation and a deep-learning network, called the recurrent three-dimensional (3D) fully convolutional network (Re3FCN), which merged the advantages of a 3D fully convolutional network (FCN) and a convolutional long short-term memory (ConvLSTM). Principal component analysis (PCA) and the spectral correlation angle (SCA) were used to generate training samples with high probabilities of being changed or unchanged. The strategy assisted in training fewer samples of representative feature expression. The Re3FCN was mainly comprised of spectral–spatial and temporal modules. Particularly, a spectral–spatial module with a 3D convolutional layer extracts the spectral–spatial features from the HSIs simultaneously, whilst a temporal module with ConvLSTM records and analyzes the multi-temporal HSI change information. The study first proposed a simple and effective method to generate samples for network training. This method can be applied effectively to cases with no training samples. Re3FCN can perform end-to-end detection for binary and multiple changes. Moreover, Re3FCN can receive multi-temporal HSIs directly as input without learning the characteristics of multiple changes. Finally, the network could extract joint spectral–spatial–temporal features and it preserved the spatial structure during the learning process through the fully convolutional structure. This study was the first to use a 3D FCN and a ConvLSTM for the remote-sensing CD. To demonstrate the effectiveness of the proposed CD method, we performed binary and multi-class CD experiments. Results revealed that the Re3FCN outperformed the other conventional methods, such as change vector analysis, iteratively reweighted multivariate alteration detection, PCA-SCA, FCN, and the combination of 2D convolutional layers-fully connected LSTM.

3D FULLY CONVOLUTIONAL NETWORKS FOR CO-SEGMENTATION OF TUMORS ON PET-CT IMAGES.

Positron emission tomography and computed tomography (PET-CT) dual-modality imaging provides critical diagnostic information in modern cancer diagnosis and therapy. Automated accurate tumor delineation is essentially important in computer-assisted tumor reading and interpretation based on PET-CT. In this paper, we propose a novel approach for the segmentation of lung tumors that combines the powerful fully convolutional networks (FCN) based semantic segmentation framework (3D-UNet) and the graph cut based co-segmentation model. First, two separate deep UNets are trained on PET and CT, separately, to learn high level discriminative features to generate tumor/non-tumor masks and probability maps for PET and CT images. Then, the two probability maps on PET and CT are further simultaneously employed in a graph cut based co-segmentation model to produce the final tumor segmentation results. Comparative experiments on 32 PET-CT scans of lung cancer patients demonstrate the effectiveness of our method.

An application of cascaded 3D fully convolutional networks for medical image segmentation

Recent advances in 3D fully convolutional networks (FCN) have made it feasible to produce dense voxel-wise predictions of volumetric images. In this work, we show that a multi-class 3D FCN trained on manually labeled CT scans of several anatomical structures (ranging from the large organs to thin vessels) can achieve competitive segmentation results, while avoiding the need for handcrafting features or training class-specific models.To this end, we propose a two-stage, coarse-to-fine approach that will first use a 3D FCN to roughly define a candidate region, which will then be used as input to a second 3D FCN. This reduces the number of voxels the second FCN has to classify to ∼10% and allows it to focus on more detailed segmentation of the organs and vessels.We utilize training and validation sets consisting of 331 clinical CT images and test our models on a completely unseen data collection acquired at a different hospital that includes 150 CT scans, targeting three anatomical organs (liver, spleen, and pancreas). In challenging organs such as the pancreas, our cascaded approach improves the mean Dice score from 68.5 to 82.2%, achieving the highest reported average score on this dataset. We compare with a 2D FCN method on a separate dataset of 240 CT scans with 18 classes and achieve a significantly higher performance in small organs and vessels. Furthermore, we explore fine-tuning our models to different datasets.Our experiments illustrate the promise and robustness of current 3D FCN based semantic segmentation of medical images, achieving state-of-the-art results.11Our code and trained models are available for download: github.com/holgerroth/3Dunet_abdomen_cascade.

3D multi-scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multi-modality MR Images.

Intervertebral discs (IVDs) are small joints that lie between adjacent vertebrae. The localization and segmentation of IVDs are important for spine disease diagnosis and measurement quantification. However, manual annotation is time-consuming and error-prone with limited reproducibility, particularly for volumetric data. In this work, our goal is to develop an automatic and accurate method based on fully convolutional networks (FCN) for the localization and segmentation of IVDs from multi-modality 3D MR data. Compared with single modality data, multi-modality MR images provide complementary contextual information, which contributes to better recognition performance. However, how to effectively integrate such multi-modality information to generate accurate segmentation results remains to be further explored. In this paper, we present a novel multi-scale and modality dropout learning framework to locate and segment IVDs from four-modality MR images. First, we design a 3D multi-scale context fully convolutional network, which processes the input data in multiple scales of context and then merges the high-level features to enhance the representation capability of the network for handling the scale variation of anatomical structures. Second, to harness the complementary information from different modalities, we present a random modality voxel dropout strategy which alleviates the co-adaption issue and increases the discriminative capability of the network. Our method achieved the 1st place in the MICCAI challenge on automatic localization and segmentation of IVDs from multi-modality MR images, with a mean segmentation Dice coefficient of 91.2% and a mean localization error of 0.62 mm. We further conduct extensive experiments on the extended dataset to validate our method. We demonstrate that the proposed modality dropout strategy with multi-modality images as contextual information improved the segmentation accuracy significantly. Furthermore, experiments conducted on extended data collected from two different time points demonstrate the efficacy of our method on tracking the morphological changes in a longitudinal study.

Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks.

Cerebral microbleeds (CMBs) are small haemorrhages nearby blood vessels. They have been recognized as important diagnostic biomarkers for many cerebrovascular diseases and cognitive dysfunctions. In current clinical routine, CMBs are manually labelled by radiologists but this procedure is laborious, time-consuming, and error prone. In this paper, we propose a novel automatic method to detect CMBs from magnetic resonance (MR) images by exploiting the 3D convolutional neural network (CNN). Compared with previous methods that employed either low-level hand-crafted descriptors or 2D CNNs, our method can take full advantage of spatial contextual information in MR volumes to extract more representative high-level features for CMBs, and hence achieve a much better detection accuracy. To further improve the detection performance while reducing the computational cost, we propose a cascaded framework under 3D CNNs for the task of CMB detection. We first exploit a 3D fully convolutional network (FCN) strategy to retrieve the candidates with high probabilities of being CMBs, and then apply a well-trained 3D CNN discrimination model to distinguish CMBs from hard mimics. Compared with traditional sliding window strategy, the proposed 3D FCN strategy can remove massive redundant computations and dramatically speed up the detection process. We constructed a large dataset with 320 volumetric MR scans and performed extensive experiments to validate the proposed method, which achieved a high sensitivity of 93.16% with an average number of 2.74 false positives per subject, outperforming previous methods using low-level descriptors or 2D CNNs by a significant margin. The proposed method, in principle, can be adapted to other biomarker detection tasks from volumetric medical data.