Deep Segmentation Models Research Articles

A design of interactive review for computer aided diagnosis of pulmonary nodules based on active learning

Automatic detection of pulmonary nodule based on computer tomography (CT) images can significantly improve the diagnosis and treatment of lung cancer. However, there is a lack of effective interactive tools to record the marked results of radiologists in real time and feed them back to the algorithm model for iterative optimization. This paper designed and developed an online interactive review system supporting the assisted diagnosis of lung nodules in CT images. Lung nodules were detected by the preset model and presented to doctors, who marked or corrected the lung nodules detected by the system with their professional knowledge, and then iteratively optimized the AI model with active learning strategy according to the marked results of radiologists to continuously improve the accuracy of the model. The subset 5-9 dataset of the lung nodule analysis 2016(LUNA16) was used for iteration experiments. The precision, F1-score and MioU indexes were steadily improved with the increase of the number of iterations, and the precision increased from 0.213 9 to 0.565 6. The results in this paper show that the system not only uses deep segmentation model to assist radiologists, but also optimizes the model by using radiologists' feedback information to the maximum extent, iteratively improving the accuracy of the model and better assisting radiologists.

Automated void detection in high resolution x-ray printed circuit boards (PCBs) images with deep segmentation neural network

Defect detection and quantification on printed circuit board (PCB) assemblies are crucial for ensuring the quality of electronic products. X-ray inspection is commonly used to detect internal defects, in which high-resolution gray-scale images are captured to reveal the internal structure, particularly the defect features, of an assembled PCB. To detect defect features, conventional computer vision techniques require hand-crafted feature filters and usually have a narrow application scope. This study introduces an automated method for detecting voids using supervised machine learning. The method employs end-to-end segmentation models to identify pixels that belong to the void and integrated circuit (IC) regions in X-ray PCB images. The segmentation results are then used to compute the percentage of voids within each IC, which is a key measure of the quality of the inspected PCB. Accurate predictions with an intersection over union of over 0.95 have achieved. A challenge in this development is the large size of the images, which can reach to millions of pixels, and the small scale of defect features. Training a deep segmentation model for large-scale images using conventional methods requires a lot of memory. Downscaling high-resolution images result in the loss of defect features. In this study, a unique training method utilizing the local nature of the defects is proposed, which reduces memory consumption and speeds up the training process. Additionally, to overcome the limited training data, various augmentation methods are applied and transfer learning is employed to construct segmentation models for different types of PCBs.

Automatic center identification of electron diffraction with multi-scale transformer networks

Selected area electron diffraction (SAED) is a widely used technique for characterizing the structure and measuring lattice parameters of materials. An autonomous analytic method has become an urgent demand for the large-scale SAED data produced from in-situ experiments. In this work, we realize the automatic processing for center identification with a proposed deep segmentation model named the multi-scale Transformer (MS-Trans) network. This algorithm enables robust segmentation of the central spots by combining a novel gated axial-attention module and multi-scale feature fusion. The proposed MS-Trans model shows high precision and robustness, enabling autonomous processing of SAED patterns without any prior knowledge. The application on in-situ SAED data of the oxidation process of FeNi alloy demonstrates its capability of implementing autonomous quantitative processing.© 2017 Elsevier Inc. All rights reserved.

CellSegUNet: an improved deep segmentation model for the cell segmentation based on UNet++ and residual UNet models

Cell nucleus segmentation is an important method that is widely used in the diagnosis and treatment of many diseases, as well as counting and identifying the cell nucleus. The main challenges when using this method are heterogeneous image intensities in the image, overlapping of cell nuclei, and noise. In order to overcome these difficulties, a hybrid segmentation model with attention block, CellSegUNet, is proposed, inspired by the advantageous points of UNet++ and Residual UNet models. With the proposed attention mechanism, semantic gaps that may occur are prevented by evaluating both horizontal and vertical features together. The serial and parallel connection of the convolutional blocks in the residual modules in the CellSegUNet model prevents data loss. Thus, features with stronger representation ability were obtained. The output layer, which is, especially proposed for the CellSegUNet model, calculated the differences between the data in each layer and the data in the input layer. The output value obtained from the layer level where the lowest value comes from constitutes the output of the whole system. At the same depth level, CellSegUNet versus UNet++ and ResUNet models were compared on Data Science Bowl (DSB), Sartorius Cell Instance Segmentation (SCIS), and Blood Cell Segmentation (BCS) datasets. With the CellSegUNet model, accuracy, dice, and jaccard metrics were obtained as 0.980, 0.970, 0.959 for the DSB dataset, 0.931, 0.957, 0.829 for the SCIS dataset and 0.976, 0.971, 0.927 for the BCS dataset, respectively. As a result, it is predicted that the proposed model can provide solutions to different segmentation problems.

Structure and Intensity Unbiased Translation for 2D Medical Image Segmentation.

Data distribution gaps often pose significant challenges to the use of deep segmentation models. However, retraining models for each distribution is expensive and time-consuming. In clinical contexts, device-embedded algorithms and networks, typically unretrainable and unaccessable post-manufacture, exacerbate this issue. Generative translation methods offer a solution to mitigate the gap by transferring data across domains. However, existing methods mainly focus on intensity distributions while ignoring the gaps due to structure disparities. In this paper, we formulate a new image-to-image translation task to reduce structural gaps. We propose a simple, yet powerful Structure-Unbiased Adversarial (SUA) network which accounts for both intensity and structural differences between the training and test sets for segmentation. It consists of a spatial transformation block followed by an intensity distribution rendering module. The spatial transformation block is proposed to reduce the structural gaps between the two images. The intensity distribution rendering module then renders the deformed structure to an image with the target intensity distribution. Experimental results show that the proposed SUA method has the capability to transfer both intensity distribution and structural content between multiple pairs of datasets and is superior to prior arts in closing the gaps for improving segmentation.

Can Deep Adult Lung Segmentation Models Generalize to the Pediatric Population?

Lung segmentation in chest X-rays (CXRs) is an important prerequisite for improving the specificity of diagnoses of cardiopulmonary diseases in a clinical decision support system. Current deep learning models for lung segmentation are trained and evaluated on CXR datasets in which the radiographic projections are captured predominantly from the adult population. However, the shape of the lungs is reported to be significantly different across the developmental stages from infancy to adulthood. This might result in age-related data domain shifts that would adversely impact lung segmentation performance when the models trained on the adult population are deployed for pediatric lung segmentation. In this work, our goal is to (i) analyze the generalizability of deep adult lung segmentation models to the pediatric population and (ii) improve performance through a stage-wise, systematic approach consisting of CXR modality-specific weight initializations, stacked ensembles, and an ensemble of stacked ensembles. To evaluate segmentation performance and generalizability, novel evaluation metrics consisting of mean lung contour distance (MLCD) and average hash score (AHS) are proposed in addition to the multi-scale structural similarity index measure (MS-SSIM), the intersection of union (IoU), Dice score, 95% Hausdorff distance (HD95), and average symmetric surface distance (ASSD). Our results showed a significant improvement (p < 0.05) in cross-domain generalization through our approach. This study could serve as a paradigm to analyze the cross-domain generalizability of deep segmentation models for other medical imaging modalities and applications.

ADNet++: A few-shot learning framework for multi-class medical image volume segmentation with uncertainty-guided feature refinement.

A major barrier to applying deep segmentation models in the medical domain is their typical data-hungry nature, requiring experts to collect and label large amounts of data for training. As a reaction, prototypical few-shot segmentation (FSS) models have recently gained traction as data-efficient alternatives. Nevertheless, despite the recent progress of these models, they still have some essential shortcomings that must be addressed. In this work, we focus on three of these shortcomings: (i) the lack of uncertainty estimation, (ii) the lack of a guiding mechanism to help locate edges and encourage spatial consistency in the segmentation maps, and (iii) the models' inability to do one-step multi-class segmentation. Without modifying or requiring a specific backbone architecture, we propose a modified prototype extraction module that facilitates the computation of uncertainty maps in prototypical FSS models, and show that the resulting maps are useful indicators of the model uncertainty. To improve the segmentation around boundaries and to encourage spatial consistency, we propose a novel feature refinement module that leverages structural information in the input space to help guide the segmentation in the feature space. Furthermore, we demonstrate how uncertainty maps can be used to automatically guide this feature refinement. Finally, to avoid ambiguous voxel predictions that occur when images are segmented class-by-class, we propose a procedure to perform one-step multi-class FSS. The efficiency of our proposed methodology is evaluated on two representative datasets for abdominal organ segmentation (CHAOS dataset and BTCV dataset) and one dataset for cardiac segmentation (MS-CMRSeg dataset). The results show that our proposed methodology significantly (one-sided Wilcoxon signed rank test, p<0.05) improves the baseline, increasing the overall dice score with +5.2, +5.1, and +2.8 percentage points for the CHAOS dataset, the BTCV dataset, and the MS-CMRSeg dataset, respectively.

Deep Learning-Based Multi-Modality Segmentation of Primary Gross Tumor Volume in CT and MRI for Nasopharyngeal Carcinoma

The delineation of primary gross tumor volume (GTV) of nasopharyngeal carcinoma (NPC) is an essential step for radiotherapy planning. In clinical practice, radiation oncologists manually delineate the GTV in planning CT with the help of diagnostic MRI. This is because NPC tumors are closely adjacent to many important anatomic structures, and CT and MRI provide complementary strength to accurately determine the tumor extension boundary. Manual delineation is time-consuming with the potential registration errors between MRI and CT decreasing the delineation accuracy. In this study, we propose a fully automated GTV segmentation method based on CT and MRI by first aligning MRI to CT, and then, segmenting the GTV using a multi-modality deep learning model. We collected 104 nasopharyngeal carcinoma patients with both planning CT and diagnostic MRI scans (T1 & T2 phases). An experienced radiation oncologists manually delineated the GTV, which was further examined by another senior radiation oncologist. Then, a coarse to fine cross-modality registration from MRI to CT was conducted as follows: (1) A rigid transformation was performed on MRI to roughly align MRI to CT with similar anatomic position. (2) Then, the region of interest (RoI) on both CT and rigid-transformed MRI were cropped. (3) A leading cross-modality deformable registration algorithm, named DEEDS, was applied on the cropped MRI and CT RoIs to find an accurate local alignment. Next, using CT and registered MRI as the combined input, a multi-modality deep segmentation network based on nnUNet was trained to generate the GTV prediction. 20% patients were randomly selected as the unseen testing set to quantitatively evaluate the performance. The quantitative NPC GTV segmentation performance is summarized in Table 1. The deep segmentation model using CT alone achieved reasonable high performance with 76.6% Dice score and 1.34mm average surface distance (ASD). When both CT and registered MRI were used, the segmentation model further improved the performance by 0.9% Dice score increase and 11% relative ASD error reduction, demonstrating the complementary strength of CT and MRI in determining NPC GTV. Notably, the achieved 77.5% Dice score and 1.19mm ASD by the multimodality model is among the top performing results reported in recent automatic NPC GTV segmentation using either CT or MRI modality. We developed a fully automated multi-modal deep-learning model for NPC GTV segmentation. The developed model can segment the NPC GTV in high accuracy. With further optimization and validation, this automated model has potential to standardize the NPC GTV segmentation and significantly decrease the workload of radiation oncologists in clinical practice.

Anatomy-Guided Deep Learning Model for Accurate and Robust Gross Tumor Volume Segmentation in Lung Cancer Radiation Therapy

In lung cancer radiation therapy, clinicians must outline the gross tumor volume (GTV) precisely on the planning computed tomography (pCT) for accurate radiation dose delivery. However, due to the limited contrast between tumor and normal tissues in lung parenchyma, accurate delineation of tumor boundaries is difficult leading to large inter-observer variation. In this study, we develop an anatomy-guided lung GTV deep segmentation model using a training cohort of multi-center datasets. The quantitative segmentation performance is evaluated on an independent dataset, where the inter-observer delineation variation is also assessed. We collected and curated four publicly available lung datasets with GTV annotations (Lung-PET-CT-Dx, LIDC-IDRI, NSCLC-Radiogenomics and RIDER-CT) for deep learning model development. A total of 871 CT scans of patients, who were diagnosed with T1-T4 NSCLC, were available for training after data curation. The GTV annotations of primary tumor were examined and edited by two experienced radiation oncologists following the RTOG 1106 protocol. An anatomy-guided deep learning model was proposed, which consisted two deep networks. The first deep network used CT scan as input and segmented 4 anatomic organs (airway, heart, pulmonary artery and pulmonary vein), while the second deep network took both CT scan and these pre-segmented 4 organs as input and segmented the lung GTV. With the help of anatomic priors from 4 pre-segmented organs, the second deep network could more easily locate the GTV. We used nnUNet as the deep segmentation network. For evaluation, we used NSCLC-Radiomics as the testing dataset, which contains 20 CT scans each annotated by 5 radiation oncologists. The auto-segmented GTV were compared against each of the manual GTV reference. Inter-observer variation was also assessed using the 5 manual GTV references. The proposed anatomic-guided lung GTV segmentation model achieved a mean Dice score of 82.4% and 95% Hausdorff distance (HD95) of 6.9mm when averaged cross 20 patients and 5 GTV references (Table 1), which outperformed the basic deep GTV segmentation model by markedly reducing 19.4% HD95 error. The performance of proposed model was also comparable to the inter-observer variation (Dice score: 82.4% vs. 81.9%, HD95 6.9 vs. 6.4mm), indicating that our model had similar reproducibility as human observers. We developed and tested an anatomy-guided deep learning model for segmenting GTV in NSCLC patients. The model achieves high quantitative segmentation performance, which is comparable to the human observer variation. It can be potentially used in radiotherapy practice to improve GTV delineation consistency and reduce workloads of radiation oncologists.

FTransCNN: Fusing Transformer and a CNN based on fuzzy logic for uncertain medical image segmentation

The accurate segmentation of medical images plays a crucial role in diagnosing and treating diseases. Although many methods now use multimodal joint segmentation, the joint use of segmentation features extracted by multiple models can lead to heterogeneity and uncertainty. Unreasonable fusion methods cannot exploit the advantages of multiple models and still lack good performance in segmentation. Therefore, this study proposes the FTransCNN model, which is composed of a convolutional neural network (CNN) and Transformer and is based on a fuzzy fusion strategy that jointly utilizes the features extracted by a CNN and Transformer through a new fuzzy fusion module. First, the CNN and Transformer act as the backbone network for parallel feature extraction. Second, channel attention is used to promote the global key information of Transformer to improve the feature representation ability, and spatial attention is used to enhance the local details of CNN features and suppress irrelevant regions. Third, the proposed model applies the Hadamard product to model fine-grained interactions between the two branches and uses the Choquet fuzzy integral to suppress heterogeneity and uncertainty in fused features. Fourth, FTransCNN employs fuzzy attention fusion module (FAFM) hierarchical upsampling to effectively capture both low-level spatial features and high-level semantic context. Finally, the new model obtains the final segmentation result by using the deconvolution and results in an improvement in segmentation. The experimental results on Chest X-ray and Kvasir-SEG dataset show that FTransCNN has better performance on segmentation tasks than the-state-of-the-art deep segmentation models.

Abstract LB067: A confident and operator-independent deep segmentation model to measure residual tumor volume in the follow-up MRIs for glioblastoma

Abstract Introduction: Magnetic resonance imaging (MRI) is the most common tool to examine glioblastoma. Preoperative MRI can be used to initial diagnosis, and surgery planning, while follow-up MRIs can be used to evaluate treatment responses, identify recurrency, and detect side effects. The follow-up MRIs are usually taken after the first-line therapy, such as maximal safe resection. In the past, radiologists manually segment the tumor regions from normal brain tissue on follow-up MRIs, which is time-consuming, error-prone, and challenging. Several deep-learning (DL) models have been developed utilizing preoperative images, but their performance has yet to be evaluated on follow-up MRIs. In this research, we built the largest follow-up MRI cohort (311 patients) to assess these DL models and their generalizability and performance on independent preoperative and follow-up images. We also made the first follow-up-based deep learning models for this specific task. Methods: All evaluation deep learning models (10 models) were trained by the Brain Tumor Segmentation challenge 2020 (BraTS’20) and evaluated by fifty pairs of preoperative and follow-up scans from our institution. The segmentation form our institution is evaluated by board certified radiologist. MRIs in the BraTS’20 dataset were all preoperative scans. After the evaluation, we randomly assigned 264 patients’ scans from our institution to the training dataset and 47 patients’ scans to the testing dataset. We compared three types of models in our follow-up deep learning model, including 1) UNet-3D, 2) UNet-3D+transfer-learning, and 3) UNet-3D+transfer-learning+baysian-learning. The benchmark for all models was the Dice similarity coefficient (DSC). DSC can measure the spatial overlap between model prediction and ground truth. The value of DSC is between zero to one. Zero means no overlap and one indicates complete overlap. Results: Our study demonstrates that the BraTS'20 trained models' performance decreased by 13.05% in independent preoperative MRI scans and 19.04% in follow-up MRI scans. The most significant mismatch regions were FLAIR hyperintense regions (3.68% drop in independent preoperative scans and 10.61% drop in independent follow-up scans) and Non-enhancing core (5.20% drop in independent preoperative scans and 11.99% drop in independent follow-up scans). Our best model can achieve the best DSC among three tumor regions compared to all evaluation models (FLAIR hyperintense regions: DSC 0.77 V.S. DSC 0.58; Enhancing tumor region: DSC 0.87 V.S. DSC 0.68; Non-enhancing tumor region: DSC 0.92 V.S. DSC 0.55). Conclusion: Maximal safe resection induced brain structure change, decreasing the performance of the preoperative-based DL model. Implementing a follow-up MRI-based segmentation model is essential to make accurate and generalizable results to address structural changes after maximal safe resection. Our follow-up DL model demonstrates the DSC score can be recovered. We commit to further developing the tool to assist radiologists in handling follow-up MRIs for glioblastoma patients. Citation Format: Kang Lin Hsieh, Tanjida Kabir, Luis Nunez-Rubiano, Yu-Chun Hsu, Yu Cai, Juan Rodriguez Quintero, Octavio Arevalo, Kangyi Zhao, Jackie Zhang, Jiguang Zhu Zhu, Roy Riascos, Xioaqian Jiang, Shayan Shams. A confident and operator-independent deep segmentation model to measure residual tumor volume in the follow-up MRIs for glioblastoma [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 2 (Clinical Trials and Late-Breaking Research); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(8_Suppl):Abstract nr LB067.

A Depth-Wise Separable U-Net Architecture with Multiscale Filters to Detect Sinkholes

Numerous variants of the basic deep segmentation model—U-Net—have emerged in recent years, achieving reliable performance across different benchmarks. In this paper, we propose an improved version of U-Net with higher performance and reduced complexity. This improvement was achieved by introducing a sparsely connected depth-wise separable block with multiscale filters, enabling the network to capture features of different scales. The use of depth-wise separable convolution significantly reduces the number of trainable parameters, making the training faster, while reducing the risk of overfitting. We used our developed sinkhole dataset and the available benchmark nuclei dataset to assess the proposed model’s performance. Pixel-wise annotation is laborious and requires a great deal of human expertise; therefore, we propose a fully deep convolutional autoencoder network that utilizes the proposed block to automatically annotate the sinkhole dataset. Our segmentation model outperformed the state-of-the-art methods, including U-Net, Attention U-Net, Depth-Separable U-Net, and Inception U-Net, achieving an average improvement of 1.2% and 1.4%, respectively, on the sinkhole and the nuclei datasets, with 94% and 92% accuracy, as well as a reduced training time. It also achieved 83% and 80% intersection-over-union (IoU) on the two datasets, respectively, which is an 11.8% and 9.3% average improvement over the above-mentioned models.

Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking

In routine laparoscopic surgeries, the fatigue operation and improper chief-assistant coordination of the assistant surgeon both reduce the stability of the endoscopic field of view (EFOV) and the surgical efficiency. In contrast, the assistant endoscope-holding robot (AEHR) can stably hold the endoscope and adjust the pose of endoscopic image sensor. However, the traditional method simplifies the tracking of surgical instrument tips (SITs) to the tracking of feature points, and lacks the depth optimization of monocular endoscopic image senor. Based on them, this paper proposes a surgical instrument tracking control method based on visual tracking features (VTFs) and hand-eye coordination (HEC) with feedback of multiple sensors, which can control the depth and axial rotational degrees of freedom (DOFs) of endoscope while tracking SITs. Firstly, the motion mapping Jacobian matrix between the endoscopic image sensor and the endoscope-holding manipulator (EHM) end-effector is constructed with joint encoder feedbacks, and the VTFs of endoscopic camera feedback are extracted based on the deep segmentation model of SITs. Secondly, the position adjustment model based on VTFs and the attitude adjustment model based on HEC are established. Thirdly, a complete set of visual servo control (VSC) software and hardware system for AEHRs is designed. Finally, the correctness, effectiveness and operability of the proposed method are comprehensively verified through the simulation and experiment of surgical instrument tracking. The visual tracking distance and radius error of endoscopic image sensor frames are reduced by no less than 34.6% and 42.59% under different experimental backgrounds, respectively.

DeepGA for automatically estimating fetal gestational age through ultrasound imaging

Accurate estimation of gestational age (GA) is vital for identifying fetal abnormalities. Conventionally, GA is estimated by measuring the morphology of the cranium, abdomen, and femur manually and inputting them into the classic Hadlock formula to assess fetal growth. However, this procedure incurs considerable overhead and suffers from bias caused by the operators, yielding suboptimal estimations. To address this challenge, we develop an automatic DeepGA model to achieve fully automatic GA prediction in an end-to-end manner. Our model uses a deep segmentation model (DeepSeg) to accurately identify and segment three critical tissues, including the cranium, abdomen, and femur, in which their morphology is automatically extracted. After that, we are able to directly estimate the GA via a deep regression model (DeepReg). We evaluate DeepGA on a large dataset, including 10,413 ultrasound images from 7113 subjects. It achieves superior performance over the traditional measurement approach, with a mean absolute estimation error (MAE) of 5 days. Our DeepGA model is a novel automatic solution on the basis of artificial intelligence learning that can help radiologists improve the performance of GA estimation in various clinical scenarios, thereby enhancing the efficiency of prenatal examinations.

Towards annotation-efficient segmentation via image-to-image translation.

An important challenge and limiting factor in deep learning methods for medical imaging segmentation is the lack of available of annotated data to properly train models. For the specific task of tumor segmentation, the process entails clinicians labeling every slice of volumetric scans for every patient, which becomes prohibitive at the scale of datasets required to train neural networks to optimal performance. To address this, we propose a novel semi-supervised framework that allows training any segmentation (encoder-decoder) model using only information readily available in radiological data, namely the presence of a tumor in the image, in addition to a few annotated images. Specifically, we conjecture that a generative model performing domain translation on this weak label - healthy vs diseased scans - helps achieve tumor segmentation. The proposed GenSeg method first disentangles tumoral tissue from healthy "background" tissue. The latent representation is separated into (1) the common background information across both domains, and (2) the unique tumoral information. GenSeg then achieves diseased-to-healthy image translation by decoding a healthy version of the image from just the common representation, as well as a residual image that allows adding back the tumors. The same decoder that produces this residual tumor image, also outputs a tumor segmentation. Implicit data augmentation is achieved by re-using the same framework for healthy-to-diseased image translation, where a residual tumor image is produced from a prior distribution. By performing both image translation and segmentation simultaneously, GenSeg allows training on only partially annotated datasets. To test the framework, we trained U-Net-like architectures using GenSeg and evaluated their performance on 3 variants of a synthetic task, as well as on 2 benchmark datasets: brain tumor segmentation in MRI (derived from BraTS) and liver metastasis segmentation in CT (derived from LiTS). Our method outperforms the baseline semi-supervised (autoencoder and mean teacher) and supervised segmentation methods, with improvements ranging between 8-14% Dice score on the brain task and 5-8% on the liver task, when only 1% of the training images were annotated. These results show the proposed framework is ideal at addressing the problem of training deep segmentation models when a large portion of the available data is unlabeled and unpaired, a common issue in tumor segmentation.

DCS-UNet: Dual-Path Framework for Segmentation of Reflux Esophagitis Lesions from Endoscopic Images with U-Net-Based Segmentation and Color/Texture Analysis

Automatic analysis of medical images and endoscopic images, in particular, is an attractive research topic in recent years. To achieve this goal, many tasks must be conducted for example lesion detection, segmentation, and classification. However, existing methods for such problems still face challenges due to the spreading characteristic of the lesions as well as some artifacts caused by motion, specularities, low contrast, bubbles, debris, bodily fluid, and blood. As a consequence, single segmentation or detection could not deal with those issues. Besides, although deep learning has achieved impressive results in many tasks, lacking a large annotated dataset could lead to overfitting problems. In this paper, we tackle these issues by taking particular characteristics of lesions of interest and advantages of deep segmentation models into account. We propose a dual-path framework (namely, DCS-UNet) that combines both segmentation and classification. We first segment lesions from the image using U-Net architecture with different encoder backbones (ResNet-50, VGG-16, DenseNet-201). The segmented regions will be then refined in the second path where we classify every patch in the whole image or the Inner or Outer regions extended from the contours given by the segmentation results. For the refinement scheme, we utilize the color-dependent and texture features as doctors’ advice and deploy the support vector machine (SVM) technique to classify a patch into disease region or not. Extensive experiments were conducted on three datasets of gastroesophageal reflux disease (GERD) endoscopic images as GradeM, GradeA, GradeB, which are defined as modified Los Angeles classification. The experimental results show the improvements of the proposed method compared to the single U-Net or a segmentation using the hand-designed features-based scheme. The proposed method improves mDice and mIoU by 0.5% and 0.36% on GradeA dataset, which is the most challenging dataset with ambiguous separations of GERD vs. normal regions, and by 0.82% and 0.81% on GradeM dataset, which is clearer to segment GERD regions. This proposed framework shows the potential to develop a diagnosis assistance system to help endoscopists reduce burden in examining GERD in the future.

Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors.

Clinical diagnosis of the pediatric musculoskeletal system relies on the analysis of medical imaging examinations. In the medical image processing pipeline, semantic segmentation using deep learning algorithms enables an automatic generation of patient-specific three-dimensional anatomical models which are crucial for morphological evaluation. However, the scarcity of pediatric imaging resources may result in reduced accuracy and generalization performance of individual deep segmentation models. In this study, we propose to design a novel multi-task, multi-domain learning framework in which a single segmentation network is optimized over the union of multiple datasets arising from distinct parts of the anatomy. Unlike previous approaches, we simultaneously consider multiple intensity domains and segmentation tasks to overcome the inherent scarcity of pediatric data while leveraging shared features between imaging datasets. To further improve generalization capabilities, we employ a transfer learning scheme from natural image classification, along with a multi-scale contrastive regularization aimed at promoting domain-specific clusters in the shared representations, and multi-joint anatomical priors to enforce anatomically consistent predictions. We evaluate our contributions for performing bone segmentation using three scarce and pediatric imaging datasets of the ankle, knee, and shoulder joints. Our results demonstrate that the proposed approach outperforms individual, transfer, and shared segmentation schemes in Dice metric with statistically sufficient margins. The proposed model brings new perspectives towards intelligent use of imaging resources and better management of pediatric musculoskeletal disorders.

An Effective Semi-Supervised Approach for Liver CT Image Segmentation.

Despite the substantial progress made by deep networks in the field of medical image segmentation, they generally require sufficient pixel-level annotated data for training. The scale of training data remains to be the main bottleneck to obtain a better deep segmentation model. Semi-supervised learning is an effective approach that alleviates the dependence on labeled data. However, most existing semi-supervised image segmentation methods usually do not generate high-quality pseudo labels to expand training dataset. In this paper, we propose a deep semi-supervised approach for liver CT image segmentation by expanding pseudo-labeling algorithm under the very low annotated-data paradigm. Specifically, the output features of labeled images from the pretrained network combine with corresponding pixel-level annotations to produce class representations according to the mean operation. Then pseudo labels of unlabeled images are generated by calculating the distances between unlabeled feature vectors and each class representation. To further improve the quality of pseudo labels, we adopt a series of operations to optimize pseudo labels. A more accurate segmentation network is obtained by expanding the training dataset and adjusting the contributions between supervised and unsupervised loss. Besides, the novel random patch based on prior locations is introduced for unlabeled images in the training procedure. Extensive experiments show our method has achieved more competitive results compared with other semi-supervised methods when fewer labeled slices of LiTS dataset are available.

MASS: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired CT and MRI images.

Training deep segmentation models for medical images often requires a large amount of labeled data. To tackle this issue, semi-supervised segmentation has been employed to produce satisfactory delineation results with affordable labeling cost. However, traditional semi-supervised segmentation methods fail to exploit unpaired multi-modal data, which are widely adopted in today's clinical routine. In this paper, we address this point by proposing Modality-collAborative Semi-Supervised segmentation (i.e., MASS), which utilizes the modality-independent knowledge learned from unpaired CT and MRI scans. To exploit such knowledge, MASS uses cross-modal consistency to regularize deep segmentation models in aspects of both semantic and anatomical spaces, from which MASS learns intra- and inter-modal correspondences to warp atlases' labels for making predictions. For better capturing inter-modal correspondence, from a perspective of feature alignment, we propose a contrastive similarity loss to regularize the latent space of both modalities in order to learn generalized and robust modality-independent representations. Compared to semi-supervised and multi-modal segmentation counterparts, the proposed MASS brings nearly 6% improvements under extremely limited supervision.

CSGNet: Cascade semantic guided net for retinal vessel segmentation

Retinal vessels are essential biomarkers for the early diagnosis of ophthalmic diseases. Accurate segmentation of retinal vessels is necessary for further quantitative analysis of fundus images. With the development of deep learning, many deep segmentation models have been proposed and show promising results. Nevertheless, segmentation performance is not satisfactory in tiny vessels and low-contrast vessels. A cascade semantic guided network (CSGNet) is proposed to address this problem. CSGNet follows a low-to-high pipeline, where low-resolution semantic-rich features are first learned. Then, higher-resolution features are learned under the guidance of low-resolution features. In this way, CSGNet is expected to learn both semantic-rich and detail-preserved high-resolution representations to facilitate the segmentation of thin vessels. Meantime, a multi-scale and multi-directional feature learning (M2FL) module is developed, which leverages two strip convolutions and two dilated convolutions to align with the shape of vessels and encode context information. Extensive experiments are conducted to validate the segmentation performance of the proposed method over six public datasets, including four photography datasets (CHASE_DB1, DRIVE, STARE, HRF) and two color scanning laser ophthalmoscopy datasets (IOSTAR, RC-SLO). Also, the proposed method is compared with several popular deep learning-based models. Experimental results show that the proposed method achieves comparable or superior segmentation performance compared with other deep learning-based models with fewer parameters and faster segmentation speed. Meantime, cross-training experiments demonstrate the competitive generalization capability of the proposed method.