3D U-Net Research Articles

Acute ischemic stroke lesion segmentation in non-contrast CT images using 3D convolutional neural networks

In this paper, an automatic algorithm aimed at volumetric segmentation of acute ischemic stroke lesion in non-contrast computed tomography brain 3D images is proposed. Our deep-learning approach is based on the popular 3D U-Net convolutional neural network architecture, which was modified by adding the squeeze-and-excitation blocks and residual connections. Robust pre-processing methods were implemented to improve the segmentation accuracy. Moreover, a special patches sampling strategy was used to address the large size of medical images and class imbalance and to stabilize neural network training. All experiments were performed using five-fold cross-validation on the dataset containing non-contrast computed tomography volumetric brain scans of 81 patients diagnosed with acute ischemic stroke. Two radiology experts manually segmented images independently and then verified the labeling results for inconsistencies. The quantitative results of the proposed algorithm and obtained segmentation were measured by the Dice similarity coefficient, sensitivity, specificity and precision metrics. The suggested pipeline provides a Dice improvement of 12.0 %, sensitivity of 10.2 % and precision 10.0 % over the baseline and achieves an average Dice of 62.8 ± 3.3 %, sensitivity of 69.9 ± 3.9 %, specificity of 99.7 ± 0.2 % and precision of 61.9 ± 3.6 %, showing promising segmentation results.

MisMatch: Calibrated Segmentation via Consistency on Differential Morphological Feature Perturbations With Limited Labels.

Semi-supervised learning (SSL) is a promising machine learning paradigm to address the ubiquitous issue of label scarcity in medical imaging. The state-of-the-art SSL methods in image classification utilise consistency regularisation to learn unlabelled predictions which are invariant to input level perturbations. However, image level perturbations violate the cluster assumption in the setting of segmentation. Moreover, existing image level perturbations are hand-crafted which could be sub-optimal. In this paper, we propose MisMatch, a semi-supervised segmentation framework based on the consistency between paired predictions which are derived from two differently learnt morphological feature perturbations. MisMatch consists of an encoder and two decoders. One decoder learns positive attention for foreground on unlabelled data thereby generating dilated features of foreground. The other decoder learns negative attention for foreground on the same unlabelled data thereby generating eroded features of foreground. We normalise the paired predictions of the decoders, along the batch dimension. A consistency regularisation is then applied between the normalised paired predictions of the decoders. We evaluate MisMatch on four different tasks. Firstly, we develop a 2D U-net based MisMatch framework and perform extensive cross-validation on a CT-based pulmonary vessel segmentation task and show that MisMatch statistically outperforms state-of-the-art semi-supervised methods. Secondly, we show that 2D MisMatch outperforms state-of-the-art methods on an MRI-based brain tumour segmentation task. We then further confirm that 3D V-net based MisMatch outperforms its 3D counterpart based on consistency regularisation with input level perturbations, on two different tasks including, left atrium segmentation from 3D CT images and whole brain tumour segmentation from 3D MRI images. Lastly, we find that the performance improvement of MisMatch over the baseline might originate from its better calibration. This also implies that our proposed AI system makes safer decisions than the previous methods.

A 3D Deep Convolutional Neural Network for Dose Distribution Prediction of Intracavitary HDR Brachytherapy of Cervical Cancer

The purpose of this work is to develop a voxel-wise dose prediction model using convolutional neural network (CNN) for cervical cancer high-dose-rate (HDR) intracavitary brachytherapy treatment planning with tandem-and-ovoid (T&O) or tandem-and-ring (T&R) applicators. A 3D U-NET CNN was implemented to generate voxel-wise dose predictions based on high-risk clinical target volume (HRCTV) and organs at risk (OAR) contour information. A multi-institutional cohort of 77 retrospective clinical HDR brachytherapy plans treated to a prescription dose in the range of 4.8-7.0 Gy/fx was used in this study. Those plans were randomly divided into 60%/20%/20% as training, validating, and testing cohorts. Data augmentation techniques were implemented in the training and validation cohort data to increase the number of plans to 252. The model was trained using the mean-squared loss function, Adam optimization algorithm, a learning rate of 0.001, 250 epochs, and a batch size of 8. The model performance was evaluated on the testing dataset by analyzing the outcomes in terms of mean dose values and derived dose-volume-histogram (DVH) indices from 3D dose distributions and comparing the generated dose distributions against the ground-truth dose distributions using dose statistics and clinically meaningful dosimetric indices. The proposed 3D U-Net model showed competitive accuracy in predicting 3D dose distributions that closely resemble the ground truth dose distributions. The average value of mean absolute error was 0.108±3.617 Gy for HRCTV, 0.074±1.315 Gy for bladder, 0.093±0.981 Gy for rectum, and 0.035±2.789 Gy for sigmoid. The median absolute error was 1.264 Gy for HRCTV, 0. 441 Gy for the bladder, 0.018 Gy for rectum, and 0.019 Gy for sigmoid. Our results showed that the predicted mean D2cc OAR doses in the bladder, rectum, and sigmoid were 4.02±0.93, 3.77±0.57 and 3.36±0.82 Gy in comparison to 4.35±0.98, 3.99±0.53, 3.75±0.83 Gy in clinical plans, respectively, and the mean individual differences (ΔD2cc) were 0.33±0.41, 0.22±0.16 and 0.39±0.45 Gy, respectively. The predicted D90 of the HRCTV was 6.72±0.99 Gy in comparison with 6.83±1.72 Gy in clinical plans. Results indicate a good potential to predict useful dosimetric indices and thus would facilitate an improvement in brachytherapy treatment planning workflow. The proposed model needs less than 5 seconds to predict a full 3D dose distribution of 64 × 64 × 64 voxels for any new patient plan, thus making it sufficient for near real-time applications and aid in decision-making in clinic. The proposed model can be used to predict 3D dose distributions for near real-time decision-making, before planning, for planning quality assurance, and for guiding future automated planning for improved plan consistency, quality, and planning efficiency.

Artificial Intelligence-Assisted Automated Applicator Digitization for Fully-Automated Gynecological High-Dose Rate Brachytherapy Treatment Planning

To automate the digitization of plastic and titanium applicators used in interstitial and hybrid gynecological (GYN) computed tomography (CT)-based high-dose-rate (HDR) brachytherapy procedures to accelerate the planning and reduce the potential for planning errors. Our hypothesis is that artificial intelligence can accurately automate the identification and digitization of plastic and titanium applicators used in HDR brachytherapy. Forty-eight patients who had received GYN procedures (7 tandem/ring: plastic applicators, 41 interstitial: titanium needles) were selected retrospectively. Patients were randomly split into training (n = 40) and test (n = 8) sets for this study. DICOM images and digitized needles from delivered plans were converted to 3D binary format. The points from each needle were transformed to individual contours and combined into a single binary mask using custom software. Using nnU-Net, a self-configuring deep convolutional neural network, 2D and 3D U-Net architectures were trained and ensembled. With the CT image as input, the nnU-Net model learned features to automatically segment the needle contours. Lastly, a 3D U-Net model was trained using 5 of the 7 tandem/ring cases (plastic applicators), with two reserved to evaluate this automated digitization. The models' performance was evaluated using the Dice Similarity Coefficient (DSC) and identification rate for individual needles. The model trained on 40 patients performed well on titanium needle cases [mean (+/- std. dev.) DSC = 0.738+/-0.034], but did not perform well on the tandem/ring cases [DSC = 0.408] in the test set. This model automatically identified 100% (54 out of 54) titanium needles but missed all plastic applicators from tandem/ring cases. Training a model with only a limited number of tandem/ring (plastic applicators) cases greatly improved segmentation accuracy [mean DSC = 0.646] for tandem/ring test cases. This model which was trained using only tandem/ring cases, automatically identified 7 out of 7 needles (100% vs 0% with previous model) from cases in the test set. The nnU-Net can automatically detect HDR needles with high confidence. Using applicator-specific identification models may improve digitization accuracy. Further evaluation of these tools on larger datasets will confirm the findings of this study.

Deep Learning for Automated Contouring of Primary Gross Tumor Volumes by MRI for Radiation Therapy of Brain Metastasis

Radiotherapy is one of the most effective methods for the treatment of brain metastases (BMs). Traditional manual delineation of primary gross tumor volumes (GTV) of multiple BMs (especially small metastases) in radiotherapy practice is extremely labor intensive and highly dependent on oncologists' experience, achieving the precise and efficient automatic delineation of BMs is of great significance for efficient and homogeneous one-stop adaptive radiotherapy. We retrospectively collected 62 MRI (non-enhanced T1-weighted sequences) sequences of 50 patients with BMs from January 2020 to July 2021. An automatic model (BUC-Net) for automatic delineation BMs was proposed in this work, which was based on deep learning by combining 3D bottler layer module and the cascade architecture to improve the accuracy and efficient of BMs' automatic delineation, especially for small metastases with tiny size and relatively low contrast. The prosed method was compared with the existing 3D U-Net (U-Net) and 3D U-Net Cascade (U-Net Cascade). The performance of our proposed method was evaluated by Dice similarity coefficient (DSC), 95% Hausdorff distance (HD95) and average surface distance (ASD) with human experts. The automatic segmentation results of BUC-Net evaluated with 310 BMs in 13 test patients was summarized in Table 1. These BMs in each test patient were automatically delineated by two types of contours: as a whole tumor contour (Whole-delineation) and the multiple tumor contours (Multiple-delineation). BUC-Net performed the best mean DSC and HD95, which is significantly outperformed U-Net (Whole-delineation: 0.911 & 0.894 of DSC, Multiple-delineation: 0.794 & 0.754 of DSC, P < 0.05 for both) and U-Net cascade (Whole-delineation: 0.947 & 7.141 of HD95, Multiple-delineation: 0.902 & 1.171 of HD95, P < 0.05 for both); Additionally, BUC-Net achieved the best mean ASD for Whole-delineation and comparable ASD (0.290 & 0.277, P > 0) for Multiple-delineation with U-Net Cascade. Our results showed that the proposed approach is promising for the automatic delineation of BMs in MRI, which can be integrated into a radiotherapy workflow to significantly shorten segmentation time.

Identification and Automated Delineation of Radioresistant Biological Tumor Volume in Nasopharyngeal Carcinoma Based on Magnetic Resonance Imaging Radiomics

Widespread use of intensity modulated radiotherapy (IMRT) has improved the tumor control rate of nasopharyngeal carcinoma (NPC). However, nearly 20% of the patients with local-advanced NPC would relapse after precise irradiation and 80% of the recurrent lesions occur within the high dose field, suggesting that there are radiation-resistant cancer cell subsets within the tumor. In this context, identification and contouring of radiation resistance region of NPC for dose escalation at primary IMRT could be advantageous. In this work, we proposed a two-step radiomics workflow to predict local relapse and the recurrent region of NPC before primary IMRT. In this single-center, retrospective study, pre-treatment magnetic resonance (MR) sequences of T1-weighted imaging (T1-w) and contrast-enhanced T1-weighted imaging (CET1-w) were collected from 800 patients of newly diagnosed and non-metastatic NPC between April 2009 and December 2015. The primary gross tumor volume (GTVp) of all patients and the actual recurrent lesion (GTVr) of patients who suffered from local recurrence were manually contoured for further analysis. A two-step complete radiomics workflow was designed to predict tumor recurrence and segment the region. First, least absolute shrinkage and selection operator (LASSO) was utilized for radiomics features selection of GTVp and support vector machine (SVM) was adopted to predict the recurrence. If the model predicts a recurrence, then the workflow utilizes an improved 3D U-Net to segment the recurrent region. Area under receiver operating characteristic curve (ROC-AUC) was used to evaluate the performance of tumor recurrence prediction, and Dice similarity coefficient (DSC) was used to assess the consistence between the actual and predicted GTVr. Of 800 NPC patients, 95 (11.9%) patients developed in-field local recurrence. For recurrence risk prediction, the SVM ensemble model (T1-w+CET1-w) was selected for further application with higher sensitivity. The average ROC-AUC, specificity, sensitivity of the SVM ensemble model in a 5-fold cross-validation and in the independent test set of 160 patients were 0.922, 0.922, 0.777 and 0.928, 0.915, 0.737, respectively. Moreover, for recurrent region segmentation, the multi-modality (T1-w+CET1-w) model was superior to the single-modality (T1-w or CET1-w) model. In an independent test set of 15 patients, the DSC, sensitivity and 95% Hausdorff Distance between actual and predicted GTVr was 0.549±0.176, 0.696±0.118 and 9.813±4.788 which was superior to 0.444±0.188, 0.497±0.218 and 12.047±5.361 of original 3D U-Net. The proposed two-step radiomics workflow showed a good performance in predicting tumor recurrence of NPC. The predicted location of the recurrence lesion was all accurate, but there was still a certain difference between the volume of the automated delineated and actual GTVr, which needed to be further optimized to be used as biological tumor volume.

Hybrid-Supervised Deep Learning for Proton-Acoustic Reconstruction for 3D In Vivo Proton Dose Verification

Proton-acoustic (PA) image has shown great potential to provide real-time 3D dose verification of proton therapy. However, the PA image quality suffers from severe limited view artifacts, which significantly impairs its accuracy for dose verification. In this study, we developed a hybrid-supervised deep learning method for PA reconstruction to address the limited-view issues. Our method consists of two stages. In the first stage, a transformer-based network was proposed to reconstruct initial pressure maps from protoacoustic signals. The network was first trained using supervision by the iteratively reconstructed pressure map and then fine-tuned using transfer learning and self-supervision based on the data fidelity constraint. In the second stage, the PA image was further enhanced by a 3D U-net. The final PA images were converted to dose maps using conversion coefficients derived from CT images. Data from 126 prostate cancer patients treated by proton therapy were collected under an IRB protocol and were split into 86 and 40 patients for model training and testing, respectively. Data of each patient contains the planning CT scan, the corresponding clinical treatment plan, and the dose map calculated by commercial software. The radiofrequency signals were generated by performing proton acoustic simulation based on CT images and the ground truth pressure map derived from the treatment plan. An ultrasound detector matrix with 64 × 64 size and 500kHz central frequency was simulated under the perineum to acquire the signals in the prostate area. In the testing results, the method's accuracy was evaluated using Root-mean-squared-error (RMSE) and structural-similarity-index-measure (SSIM) between the reconstructed and ground truth pressure map and dose distribution. Testing results showed that the reconstructed pressure map achieved an average RMSE/SSIM of 0.0292/0.96, demonstrating excellent 3D information with details. Dose maps derived from the pressure map achieved an average RMSE/SSIM of 0.018/0.99 with a gamma index of 94.7% and 95.7% for 1%/3 mm and 1%/5 mm criteria compared to the ground truth dose maps. The reconstruction time was 6s, which can be further reduced using GPU. Our study achieves start-of-the-art performance in the challenging task of direct reconstruction from limited-view radiofrequency signals, demonstrating the great promise of PA imaging as a highly efficient and accurate tool for in-vivo 3D proton dose verification. Such high-precision 3D online dose verification can substantially reduce the range uncertainties of proton therapy to significantly improve its precision and outcomes.

Mitigation of Dosimetric Uncertainty in MRI-Based Proton Planning Using Spot-Scanning Proton Arc (SPArc) Technique

MRI-based synthetic CT (SCT) images created using generative adversarial network (GAN) have been demonstrated to be feasible for intensity-modulated proton therapy (IMPT) planning. However, dose calculation accuracy can be uncertain in some regions within/near the target of head and neck patients due to the local CT number estimation error or sharp dose fall-off. This study investigated the feasibility of using the SPArc technique to mitigate such dosimetric uncertainty. A GAN using a 3D U-net as the generator and a 6-layer 3D convolutional neural network as the discriminator was trained with T1-weighted MR-CT image pairs from 162 nasopharyngeal carcinoma patients (14 for validation). The generator was used to generate SCT images from MR images for 7 test patients. For each test patient, the CT image was used to create a SPArc plan and an IMPT plan with the same clinical objectives. The SPArc plans (control point frequency sampling, arc trajectory, etc.) were optimized using a previously developed iterative approach. The dose distributions of both SPArc plans and IMPT plans were re-calculated on the SCT images and compared to the one calculated on the CT images. The dosimetric uncertainty was quantified using the gamma index. The 2%/2mm and 3%/3mm passing rates for SPArc plans were (96.9¡À2.7) % and (98.6¡À1.5) %, while the passing rates for IMPT plans were (94.0¡À3.9) % and (96.4+2.9) %. A significant reduction in dosimetric uncertainty was identified for SPArc plans (p ¡Ü0.021). Table 1 shows the passing rates for the 7 test individuals. SPArc can mitigate the uncertainty of dose calculation in MRI-based proton planning. Further research needs to validate these findings on a larger patient cohort. The study paves the road map for using MRI for SPArc planning.

Automated Contouring of Cervical Lymph Nodes and Clinical Target Volumes for Nasopharyngeal Carcinoma Based on Deep Learning and Experience Constraints

Application of artificial intelligence (AI) for automated contouring of tumor volumes and organs at risk (OARs) for radiotherapy of nasopharyngeal carcinoma (NPC) leads to improved contouring accuracy and efficiency. However, few studies have involved the automated contouring of gross tumor volume of cervical lymph nodes (GTVn) and clinical target volumes (CTVs). In this work, we proposed an AI automated contouring tool for GTVn and CTVs for radiotherapy of NPC on the plain scans of planning compute tomography (CT). In this retrospective study, plain scan datasets of planning CT covering the nasopharynx and neck from 139 patients with NPC between March 2022 and December 2022 were collected and divided into training, validation, and testing cohorts of 95, 24, and 20 patients, respectively. Ground truth contours of primary gross tumor volume (GTVp), GTVn (divided into GTVn_L in left neck and GTVn_R in right neck), CTVs (including high risk CTV1 contains GTVp and low risk CTV2 contains GTVp and cervical nodal levels) and OARs were delineated and were defined by consensus of two experts. We first proposed a three-dimensional (3D) U-net using GTVp and OARs as experience constrains to guide the automated delineation of GTVn and CTVs. The average Dice similarity coefficient (DSC) and average surface distance (ASD) were used to quantify the performance of the AI tool. Next, five prospective patients were enrolled for clinical evaluation of our AI tool. DSC between automated contours and radiation oncologist-revised contours and time consuming of the revision were record. Clinical characteristics of 139 retrospective and 5 prospective patients are list in Table 1. In the independent testing set of 20 patients, our AI tool showed high performance in GTVn and CTVs contouring when compared with the ground truth contours. The mean DSC were 0.73 ± 0.07, 0.74 ± 0.05, 0.93 ± 0.03, and 0.88 ± 0.03, and the mean ASD were 1.01 ± 0.43 mm, 1.14 ± 0.61 mm, 0.51 ± 0.13 mm, 1.17 ± 0.43 mm for GTVn_L, GTVn_R, CTV1 and CTV2, respectively. In the five prospective patients, mean DSC were 0.74 ± 0.07, 0.74 ± 0.10, 0.95 ± 0.01 and 0.89 ± 0.04, respectively. The median time consuming for GTVn and CTVs revision was 2minutes and 10 seconds (range, 1 minutes to 3 minutes). The proposed AI tool integrating clinical experience as constrains showed high accuracy for contouring GTVn and CTVs of NPC. With the assistance of AI contours, contouring efficiency could be probably increased, which is promising in online adaptive radiotherapy of NPC.

An Automated Brain Metastasis Detection and Segmentation System from MRI with a Large Multi-Institutional Dataset

Developments of automated systems for brain metastasis (BM) detection and segmentation from MRI for assisting early detection and stereotactic radiosurgery (SRS) have been reported but most based upon relatively small datasets from single institutes. This work aims to develop and evaluate a system using a large multi-institutional dataset, and to improve both identification of small/subtle BMs and segmentation accuracy of large BMs. A 3D U-Net system was trained and evaluated to detect and segment intraparenchymal BMs with a size > 2mm using 1856 MRI volumes from 1791 patients treated with SRS from seven institutions (1539 volumes for training, 183 for validation, and 134 for testing). All patients had 3D post-Gd T1w MRI scans pre-SRS. Gross tumor volumes (GTVs) of BMs for SRS were curated by each institute first. Then, additional efforts were spent to create GTVs for the untreated and/or uncontoured BMs, including central reviews by two radiologists, to improve accuracy of ground truth. The training dataset was augmented with synthetic BMs of 3773 MRIs using a 3D generative pipeline. Our system consists of two U-Nets with one using small 3D patches dedicated for detecting small BMs and another using large 3D patches for segmenting large BMs, and a random-forest based fusion module for combining the two network outputs. The first U-Net was trained with 3D patches containing at least one BM < 0.1 cm3. For detection performance, we measured BM-level sensitivity and case-level false-positive (FP) rate. For segmentation performance, we measured BM-level Dice similarity coefficient (DSC) and 95-percentile Hausdorff distance (HD95). We also stratified performances based upon BM sizes. For 739 BMs in the 134 testing cases, the overall lesion-level sensitivity was 0.870 with an average case-level FP of 1.34±1.92 (95% CI: 1.02-1.67). The sensitivity was >0.969 for the BMs >0.1 cm3, but dropped to 0.755 for the BMs < 0.1 cm3 (Table 1). The average DSC and HD95 for all detected BMs were 0.786 and 1.35mm. The worse performance for BMs > 20 cm3 was caused by a case with 83 cm3 GTV and artifacts in the MRI volume. We achieved excellent detection sensitivity and segmentation accuracy for BMs > 0.1 cm3, and promising performance for small BMs (<0.1cm3) with a controlled FP rate using a large multi-institutional dataset. Clinical utility for assisting early detection and SRS planning will be investigated. Table 1: Per-lesion detection and segmentation performance stratified by individual BM size. N is the number of BMs in each category.

GA-Net: A geographical attention neural network for the segmentation of body torso tissue composition

PurposeBody composition analysis (BCA) of the body torso plays a vital role in the study of physical health and pathology and provides biomarkers that facilitate the diagnosis and treatment of many diseases, such as type 2 diabetes mellitus, cardiovascular disease, obstructive sleep apnea, and osteoarthritis. In this work, we propose a body composition tissue segmentation method that can automatically delineate those key tissues, including subcutaneous adipose tissue, skeleton, skeletal muscle tissue, and visceral adipose tissue, on positron emission tomography/computed tomography scans of the body torso. MethodsTo provide appropriate and precise semantic and spatial information that is strongly related to body composition tissues for the deep neural network, first we introduce a new concept of the body area and integrate it into our proposed segmentation network called Geographical Attention Network (GA-Net). The body areas are defined following anatomical principles such that the whole body torso region is partitioned into three non-overlapping body areas. Each body composition tissue of interest is fully contained in exactly one specific minimal body area. Secondly, the proposed GA-Net has a novel dual-decoder schema that is composed of a tissue decoder and an area decoder. The tissue decoder segments the body composition tissues, while the area decoder segments the body areas as an auxiliary task. The features of body areas and body composition tissues are fused through a soft attention mechanism to gain geographical attention relevant to the body tissues. Thirdly, we propose a body composition tissue annotation approach that takes the body area labels as the region of interest, which significantly improves the reproducibility, precision, and efficiency of delineating body composition tissues. ResultsOur evaluations on 50 low-dose unenhanced CT images indicate that GA-Net outperforms other architectures statistically significantly based on the Dice metric. GA-Net also shows improvements for the 95% Hausdorff Distance metric in most comparisons. Notably, GA-Net exhibits more sensitivity to subtle boundary information and produces more reliable and robust predictions for such structures, which are the most challenging parts to manually mend in practice, with potentially significant time-savings in the post hoc correction of these subtle boundary placement errors. Due to the prior knowledge provided from body areas, GA-Net achieves competitive performance with less training data. Our extension of the dual-decoder schema to TransUNet and 3D U-Net demonstrates that the new schema significantly improves the performance of these classical neural networks as well. Heatmaps obtained from attention gate layers further illustrate the geographical guidance function of body areas for identifying body tissues. Conclusions(i) Prior anatomic knowledge supplied in the form of appropriately designed anatomic container objects significantly improves the segmentation of bodily tissues. (ii) Of particular note are the improvements achieved in the delineation of subtle boundary features which otherwise would take much effort for manual correction. (iii) The method can be easily extended to existing networks to improve their accuracy for this application.

What MRI Sequences are Necessary for Automated Neural Network-Based Metastasis Segmentation ― An Ablation Study

Brain metastasis (BM) delineation is a time-consuming process in both daily clinical practice and research. Automated BM segmentation algorithms can be used to assist in this task. Most approaches to brain tumor segmentation, such as algorithms trained on the BraTS challenge, use four magnetic resonance imaging (MRI) sequences as input, making them susceptible to missing or corrupted sequences and increase the number of sequences necessary for MRI RT planning. The goal of this project is to compare neural networks with different combinations of input sequences for the segmentation of the contrast-enhancing metastasis and the surrounding FLAIR hyperintense edema. All models were tested in a multicenter international external test cohort. This allows us to determine which MRI sequences are needed for effective automated segmentations. In total, we had T1-weighted sequences without (T1) and with contrast enhancement (T1-CE), T2-weighted sequences (T2), and T2 fluid-attenuated inversion recovery (FLAIR) sequences from 339 patients with at least one brain metastasis from seven centers available. Preprocessing yielded co-registered, skull-stripped sequences with an isotropic resolution of 1 millimeter. The contrast-enhancing metastasis as well as the surrounding FLAIR hyperintense edema were manually segmented to create reference labels. A baseline 3D U-Net with all four sequences as well as six additional U-Nets with different clinically plausible combinations (T1-CE; T1; FLAIR; T1-CE+FLAIR; T1-CE+T1+FLAIR; T1-CE+T1) of input sequences were trained on a cohort of 239 patients from two centers and subsequently tested on an external cohort of 100 patients from the remaining five centers. All models that included T1-CE in their selected sequences showed similar performance for metastasis segmentation with a median Dice similarity coefficient (DSC) of 0.93-0.96. T1-CE alone likewise achieved a performance of 0.96 (IQR 0.93-0.97). The model trained with only FLAIR performed worse (DSC = 0.73, IQR 0.54-0.84). For edema segmentation, models that included both T1-CE and FLAIR performed best (median DSC = 0.93), while the remaining four models without simultaneous inclusion of these two sequences (T1-CE; T1; FLAIR; T1-CE+T1) reached a median DSC of 0.81-0.89. Automatic segmentation of brain metastases with less than four input sequences is feasible with minimal or no loss of quality. A T1-CE-only protocol suffices for metastasis segmentation. In contrast, for edema segmentation, the combination of T1-CE and FLAIR seems to be important. Missing either T1-CE or FLAIR decreases performance. These findings may improve future imaging routines by omitting unnecessary sequences, thus speeding up procedures in daily clinical practice while allowing for optimal neural network-based target definitions.

Deep Learning and GAN-Synthesis for Auto-Segmentation of Pancreatic Cancer by Non-Enhanced CT for Adaptive Radiotherapy

In conventional adaptive radiotherapy (ART) for pancreatic cancer, contrast-enhanced CT (CECT) helps to more precisely delineate primary gross tumor volume (GTV) than non-enhanced CT (NECT). However, frequent use of contrast medium can damage kidneys and prolong treatment time. Moreover, traditional manual delineation is labor-intensive and highly dependent on the experience of oncologists. Currently, automatic delineation based on deep learning with Generative Adversarial Networks (GAN)-based CT synthesis is one of the most feasible solutions to these problems. A dataset of 35 pancreatic cancer patients was retrospectively collected from May 2021 to December 2022. All patients consist of a pair of NECT and CECT. We designed and developed an automatic delineation framework (Proposed) for GTV of pancreatic cancer based on Trans-cycleGAN and a modified 3D U-Net. TranscycleGAN can not only synthesize CECT from NECT, but can also augment the amount of CT images; then all real and synthesized CT images were used to train the modified 3D U-Net for automatic delineation of GTV; finally, our framework was able to automatically delineate GTV by NECT, but not only by CECT. Our framework was evaluated by dice similarity coefficient (DSC), 95% Harsdorff distance (95HD) and average surface distance (ASD) with oncologists' manual delineation ("gold standard"). The evaluation results were summarized in Table 1. The proposed framework achieved the best automatic delineation results by NECT, which was superior to that of CECT: 0.917 & 0.903 of DSC, 2.498mm & 3.029mm of HD95, 0.481mm & 0.534mm of ASD, p < 0.05 for DSC and HD95. Specifically, it is significantly superior to the automatic delineation results using U-Net by CECT 0.917 & 0.818 of DSC, 2.498mm & 13.228mm of HD95, 0.481mm & 3.633mm of ASD, p < 0.05 for DSC. We proposed an automatic delineation framework for contouring GTV in ART of pancreatic cancer based on deep learning and Trans-cycleGAN network. This framework could automatically delineate GTV and achieve better performance with NECT compared to CECT. Our method could not only reduce the use of contrast medium, but also increase the precision and effectiveness of tumor delineation, which could have a positive impact on precision radiotherapy.

A Comprehensive Deep Learning Framework for Automatic Target Volumes Segmentation in Post-Operative Stereotactic Partial Breast Irradiation (S-PBI)

In S-PBI, accurate delineation of post-surgical tumor bed volume (TBV) and clinical target volume (CTV) are crucial tasks to achieve effective radiotherapy outcomes. However, manual contouring is labor intensive, time consuming, and largely relies on the experience of clinicians. We aimed to propose a deep learning (DL) approach which mimics physicians' contouring practice to accurately segment target volumes in post-operative breast CT images. Our approach incorporated domain knowledge into a 3D U-Net based DL model for breast target volumes (TBV and CTV) delineation. Our TBV segmentation approach was inspired by the marker-guidance procedure in manual delineation, where the visual clues provided by the markers assist physicians in defining TBV. For this purpose, a distance-transformation coupled with a Gaussian filter was adopted to convert markers' locations on the CT images to saliency maps. Subsequently, the CT images and the corresponding saliency maps formed a two-channel input for the segmentation model. For CTV segmentation, TBV was incorporated as an input in addition to the CT images, guiding the model to encode the location-related image features. The architecture allowed the network to emulate the oncologist's manual delineation where CTV is derived from TBV via a margin expansion, followed by correcting the extensions for anatomical barriers of tumor invasion (e.g., skin, chest wall). We retrospectively collected 175 prone CT images from 35 post-operative breast cancer patients who received 5-fraction partial breast irradiation (PBI) regimen on a Co-60 prone based S-PBI unit. The 35 patients were randomly split into 25, 5, and 5 for model training, validation, and testing respectively. We evaluated the performance of the developed DL model on the testing dataset by comparing the predicted volumes with the manually delineated contours (ground truth) using Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), and average symmetric surface distance (ASD). For TBV segmentation, our model achieved mean (standard deviation) of 0.76 (±2.7), 6.76 (±1.83) mm, and 1.9 (±0.66) mm for DSC, HD95, and ASD respectively. For CTV segmentation, our model achieved 0.94 (±0.02), 2.46 (±0.5) mm, and 0.53 (±0.14) mm for DSC, HD95, and ASD respectively. The proposed auto-segmentation approach generated TBV and CTV masks in ∼11 seconds per CT volume, implying significantly improved efficiency compared to manual contouring. We developed a comprehensive DL framework mimicking clinical contouring practice for auto-segmentation of target volumes in S-PBI. The results demonstrated high levels of agreement between the predicted contours and physicians' manual contours. The approach is promising for improving the efficiency and accuracy of the on-line treatment planning workflow, such as adaptive based S-PBI.

Deep-Learning Based Auto-Segmentation for Liver Yttrium-90 Selective Internal Radiation Therapy

In resin Yttrium-90 (Y-90) selective internal radiation therapy (SIRT), liver volume sizes are needed for Y-90 activity calculations using the body-surface-area method, which are obtained from contours that are manually delineated in 3D images. The aim was to apply a deep-learning based auto-segmentation method for liver delineation for Y-90 SIRT. A deep-learning-based liver segmentation method was applied using the U-Net3D architecture, which is a 3D convolutional neural network (CNN) extended from the original 2D U-Net architecture for 3D objects in medical imaging. The segmentation model was trained on the Liver Tumor Segmentation (LiTS) dataset. The training data set contained 130 CT scans, and the test data set contained 70 CT scans. The model was deployed in the clinic using DICOM communication. Auto-segmentation of liver in the CT images of 18 SIRT patients was studied. The CT images were exported from clinical database to the segmentation model's DICOM location, where a monitoring software detected the incoming data and automatically ran the liver segmentation. The results were then returned to the original DICOM location where the CT images were stored. Auto-segmented liver contours were compared with physician manually-delineated contours. Dice similarity coefficient (DSC), mean distance to agreement (MDA), ratio of volume (RV), and ratio of activity (RA, ratio of activity calculated using an auto-segmented liver contour to the accurate activity calculated using a manually-delineated contour), were assessed. DSC, MDA, and RV are 0.942±0.014 (range: 0.908-0.959), 1.902±0.503 mm (range: 1.043-2.956 mm), and 0.988±0.039 (range: 0.901-1.045), respectively. RA is 1.001±0.003 (range: 0.993-1.007), which indicates that the activities calculated using the auto-segmented liver contours are close to the accurate activities. The segmentation model was able to successfully identify and segment livers in the CT images, and provide accurate and reliable results. The proposed method is beneficial for clinical use as it can process large amounts of data quickly and efficiently, and can be easily deployed in a clinical environment using DICOM communication.

Automated Sarcopenia Assessment and Outcomes in Head and Neck Cancer with Deep Learning Analysis of Cervical Neck Skeletal Muscle

Sarcopenia is an established prognostic factor in patients diagnosed with head and neck cancers (HNC), typically measured by the skeletal muscle index (SMI) from abdominal muscle mass at L3. While sarcopenia assessment could inform HNC management, it remains impractical, time- and labor-intensive, and operator-dependent. To overcome these challenges, we developed an automated deep learning (DL) platform to calculate SMI at L3 by quantifying cross-sectional cervical skeletal muscle area (SMA) at C3 through auto-segmentation, externally validated it, and evaluated associations with clinical outcomes. Eight hundred twenty-one patients diagnosed with HNC from multiple institutes from 1999-2013, treated with definitive chemoradiation with baseline pre-treatment CT scans, were included for model development (335 training, 96 tuning) and for independent testing (48 internal, and 342 external). Ground truth single-slice segmentations of SM at the mid-C3 vertebral level were manually annotated by radiation oncologists using an established protocol. A multi-stage DL pipeline was developed, with a 2D DenseNet to select the middle slice of C3 section and a 2D UNet to segment the SM, from which SMA was calculated. The model was evaluated using the Dice Similarity Coefficient (DC) for the internal test set, and human acceptability testing on the external test set was performed by two radiation oncologists not involved in annotations. SMI was calculated from C3 SMA based on prior literature, and sarcopenia was defined by an established, sex-specific SMI cutoff. Sarcopenia associations with overall survival (OS) and toxicities were assessed on the external dataset with Cox and logistic multivariable regressions, as indicated. Model DC on the internal test set as 0.90 [95% CI: 0.90-0.91], with an intra-class coefficient of 0.96 for SMA. Human acceptability testing showed a pass rate of 94.4%. Of the 342 patients in the clinical analysis, 261 (76.3%) patients had sarcopenia. Five-year survival was 84.4% in patients without sarcopenia vs 73.1% in patients with sarcopenia (HR 2.21, p = 0.028) (median f/u: 44 mo (IQR: 25 - 66 mo)). On multivariable regression, sarcopenia (HR 2.06, p = 0.037), ACE-27 score 2+ (HR 2.25, p = 0.001), non-oropharynx diagnosis (HR 3.96, p<0.001), and T3-4 stage (HR 2.37, p<0.001) were associated with worse OS. Sarcopenia was associated with longer PEG tube duration on multivariable analysis (HR 1.59, p = 0.003), along with ACE-27 score (HR 1.20, p = 0.012) and non-oropharynx primary site (HR 1.46, p = 0.034). Sarcopenia was associated with higher risk of having PEG tube at last follow up (OR 2.25, p = 0.046). An observed increase in risk of hospitalization <3 months after RT was non-significant (OR 2.18, p = 0.117). We developed and externally validated a fully-automated platform for sarcopenia assessment that can be used on routine HNC imaging. This algorithm is positioned for prospective testing to determine if use will inform HNC management.

Automatic orbital segmentation using deep learning-based 2D U-net and accuracy evaluation: A retrospective study

The purpose of this study was to verify whether the accuracy of automatic segmentation (AS) of computed tomography (CT) images of fractured orbits using deep learning (DL) is sufficient for clinical application.In the surgery of orbital fractures, many methods have been reported to create a 3D anatomical model for use as a reference. However, because the orbit bone is thin and complex, creating a segmentation model for 3D printing is complicated and time-consuming.Here, the training of DL was performed using U-Net as the DL model, and the AS output was validated with Dice coefficients and average symmetry surface distance (ASSD). In addition, the AS output was 3D printed and evaluated for accuracy by four surgeons, each with over 15 years of clinical experience. One hundred twenty-five CT images were prepared, and manual orbital segmentation was performed in all cases. Ten orbital fracture cases were randomly selected as validation data, and the remaining 115 were set as training data. AS was successful in all cases, with good accuracy: Dice, 0.860 ± 0.033 (mean ± SD); ASSD, 0.713 ± 0.212 mm. In evaluating AS accuracy, the expert surgeons generally considered that it could be used for surgical support without further modification. The orbital AS algorithm developed using DL in this study is extremely accurate and can create 3D models rapidly at low cost, potentially enabling safer and more accurate surgeries.

Comprehensive deep learning-based framework for automatic organs-at-risk segmentation in head-and-neck and pelvis for MR-guided radiation therapy planning

Introduction: The excellent soft-tissue contrast of magnetic resonance imaging (MRI) is appealing for delineation of organs-at-risk (OARs) as it is required for radiation therapy planning (RTP). In the last decade there has been an increasing interest in using deep-learning (DL) techniques to shorten the labor-intensive manual work and increase reproducibility. This paper focuses on the automatic segmentation of 27 head-and-neck and 10 male pelvis OARs with deep-learning methods based on T2-weighted MR images.Method: The proposed method uses 2D U-Nets for localization and 3D U-Net for segmentation of the various structures. The models were trained using public and private datasets and evaluated on private datasets only.Results and discussion: Evaluation with ground-truth contours demonstrated that the proposed method can accurately segment the majority of OARs and indicated similar or superior performance to state-of-the-art models. Furthermore, the auto-contours were visually rated by clinicians using Likert score and on average, 81% of them was found clinically acceptable.

Application of 3D U-Net-based neutral hydrogen foreground reduction method

Deep learning techniques are gradually making their way into the field of cosmology, bringing new ideas to the foreground subtraction of cosmological signals. This study aims to separate neutral hydrogen signals from the foreground. Traditional methods include principal component analysis (PCA) and singular value decomposition (SVD), but it is not realistic to use these methods alone to completely separate the foreground from the 21 centimeter signal in the universe, Therefore, we use a deep learning method, namely the 3D U-Net network to facilitate separation. We add residual modules to 3D U-Net to avoid network degradation issues, and also considered system effects in the foreground of 3D U-Net subtraction. We use a Gaussian beam model and use power spectrum analysis to compare the results. The analysis of the simulated data training results is shown in Fig. 8. It has been proven that using PCA processing followed by 3D U-Net for foreground subtraction has better consistency than using PCA alone. Therefore, the use of deep learning methods provides a new and effective method for observing neutral hydrogen (HI) to eliminate HI foreground and reveal physical phenomena in the universe.

A whole-body diffusion MRI normal atlas: development, evaluation and initial use

BackgroundStatistical atlases can provide population-based descriptions of healthy volunteers and/or patients and can be used for region- and voxel-based analysis. This work aims to develop whole-body diffusion atlases of healthy volunteers scanned at 1.5T and 3T. Further aims include evaluating the atlases by establishing whole-body Apparent Diffusion Coefficient (ADC) values of healthy tissues and including healthy tissue deviations in an automated tumour segmentation task.MethodsMulti-station whole-body Diffusion Weighted Imaging (DWI) and water-fat Magnetic Resonance Imaging (MRI) of healthy volunteers (n = 45) were acquired at 1.5T (n = 38) and/or 3T (n = 29), with test-retest imaging for five subjects per scanner. Using deformable image registration, whole-body MRI data was registered and composed into normal atlases. Healthy tissue ADCmean was manually measured for ten tissues, with test-retest percentage Repeatability Coefficient (%RC), and effect of age, sex and scanner assessed. Voxel-wise whole-body analyses using the normal atlases were studied with ADC correlation analyses and an automated tumour segmentation task. For the latter, lymphoma patient MRI scans (n = 40) with and without information about healthy tissue deviations were entered into a 3D U-Net architecture.ResultsSex- and Body Mass Index (BMI)-stratified whole-body high b-value DWI and ADC normal atlases were created at 1.5T and 3T. %RC of healthy tissue ADCmean varied depending on tissue assessed (4–48% at 1.5T, 6–70% at 3T). Scanner differences in ADCmean were visualised in Bland-Altman analyses of dually scanned subjects. Sex differences were measurable for liver, muscle and bone at 1.5T, and muscle at 3T. Volume of Interest (VOI)-based multiple linear regression, and voxel-based correlations in normal atlas space, showed that age and ADC were negatively associated for liver and bone at 1.5T, and positively associated with brain tissue at 1.5T and 3T. Adding voxel-wise information about healthy tissue deviations in an automated tumour segmentation task gave numerical improvements in the segmentation metrics Dice score, sensitivity and precision.ConclusionsWhole-body DWI and ADC normal atlases were created at 1.5T and 3T, and applied in whole-body voxel-wise analyses.