Consensus Contours Research Articles

Deep learning-assisted interactive contouring of lung cancer: Impact on contouring time and consistency

Background and PurposeTo evaluate the impact of a deep learning (DL)-assisted interactive contouring tool on inter-observer variability and the time taken to complete tumour contouring. Materials and MethodsNine clinicians contoured the gross tumour volume (GTV) using the PET-CT scans of 10 non-small cell lung cancer (NSCLC) patients, either using DL-assisted or manual contouring tools. After contouring a case using one contouring method, the same case was contoured one week later using the other method. The contours and time taken were compared. ResultsUse of the DL-assisted tool led to a statistically significant decrease in active contouring time of 23 % relative to the standard manual segmentation method (p < 0.01). The mean observation time for all clinicians and cases made up nearly 60 % of interaction time for both contouring approaches. On average the time spent contouring per case was reduced from 22 min to 19 min when using the DL-assisted tool. Additionally, the DL-assisted tool reduced contour variability in the parts of tumour where clinicians tended to disagree the most, while the consensus contour was similar whichever of the two contouring approaches was used. ConclusionsA DL-assisted interactive contouring approach decreased active contouring time and local inter-observer variability when used to delineate lung cancer GTVs compared to a standard manual method. Integration of this tool into the clinical workflow could assist clinicians in contouring tasks and improve contouring efficiency.

Fully automated radiotherapy treatment planning: A scan to plan challenge

Background and PurposeOver the past decade, tools for automation of various sub-tasks in radiotherapy planning have been introduced, such as auto-contouring and auto-planning. The purpose of this study was to benchmark what degree of automation is possible. Materials and MethodsA challenge to perform automated treatment planning for prostate and prostate bed radiotherapy was set up. Participants were provided with simulation CTs and a treatment prescription and were asked to use automated tools to produce a deliverable radiotherapy treatment plan with as little human intervention as possible. Plans were scored for their adherence to the protocol when assessed using consensus expert contours. ResultsThirteen entries were received. The top submission adhered to 81.8% of the minimum objectives across all cases using the consensus contour, meeting all objectives in one of the ten cases. The same system met 89.5% of objectives when assessed with their own auto-contours, meeting all objectives in four of the ten cases. The majority of systems used in the challenge had regulatory clearance (Auto-contouring: 82.5%, Auto-planning: 77%). Despite the ‘hard’ rule that participants should not check or edit contours or plans, 69% reported looking at their results before submission. ConclusionsAutomation of the full planning workflow from simulation CT to deliverable treatment plan is possible for prostate and prostate bed radiotherapy. While many generated plans were found to require none or minor adjustment to be regarded as clinically acceptable, the result indicated there is still a lack of trust in such systems preventing full automation.

Does consensus contour improve robustness and accuracy in 18F-FDG PET radiomic features?

PurposeThe purpose of our study is to validate the robustness and accuracy of consensus contour in 2-deoxy-2-[18\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{18}$$\\end{document}F]fluoro-D-glucose (18\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{18}$$\\end{document}F-FDG) PET radiomic features.Methods225 nasopharyngeal carcinoma (NPC) and 13 extended cardio-torso (XCAT) simulated data were enrolled. All segmentation were performed with four segmentation methods under two different initial masks, respectively. Consensus contour (ConSeg) was then developed using the majority vote rule. 107 radiomic features were extracted by Pyradiomics based on segmentation and the intraclass correlation coefficient (ICC) was calculated for each feature between masks or among segmentation, respectively. In XCAT ICC between segmentation and simulated ground truth were also calculated to access the accuracy.ResultsICC varied with the dataset, segmentation method, initial mask and feature type. ConSeg presented higher ICC for radiomic features in robustness tests and similar ICC in accuracy tests, compared with the average of four segmentation results. Higher ICC were also generally observed in irregular initial masks compared with rectangular masks in both robustness and accuracy tests. Furthermore, 19 features (17.76%) had ICC ≥ 0.75 in both robustness and accuracy tests for any of the segmentation methods or initial masks. The dataset was observed to have a large impact on the correlation relationships between radiomic features, but not the segmentation method or initial mask.ConclusionsThe consensus contour combined with irregular initial mask could improve the robustness and accuracy in radiomic analysis to some extent. The correlation relationships between radiomic features and feature clusters largely depended on the dataset, but not segmentation method or initial mask.

Clinical evaluation of a deep learning CBCT auto-segmentation software for prostate adaptive radiation therapy

PurposeAim of the present study is to characterize a deep learning-based auto-segmentation software (DL) for prostate cone beam computed tomography (CBCT) images and to evaluate its applicability in clinical adaptive radiation therapy routine. Materials and methodsTen patients, who received exclusive radiation therapy with definitive intent on the prostate gland and seminal vesicles, were selected. Femoral heads, bladder, rectum, prostate, and seminal vesicles were retrospectively contoured by four different expert radiation oncologists on patients CBCT, acquired during treatment. Consensus contours (CC) were generated starting from these data and compared with those created by DL with different algorithms, trained on CBCT (DL-CBCT) or computed tomography (DL-CT). Dice similarity coefficient (DSC), centre of mass (COM) shift and volume relative variation (VRV) were chosen as comparison metrics. Since no tolerance limit can be defined, results were also compared with the inter-operator variability (IOV), using the same metrics. ResultsThe best agreement between DL and CC was observed for femoral heads (DSC of 0.96 for both DL-CBCT and DL-CT). Performance worsened for low-contrast soft tissue organs: the worst results were found for seminal vesicles (DSC of 0.70 and 0.59 for DL-CBCT and DL-CT, respectively). The analysis shows that it is appropriate to use algorithms trained on the specific imaging modality. Furthermore, the statistical analysis showed that, for almost all considered structures, there is no significant difference between DL-CBCT and human operator in terms of IOV. ConclusionsThe accuracy of DL-CBCT is in accordance with CC; its use in clinical practice is justified by the comparison with the inter-operator variability.

Human factors in the clinical implementation of deep learning-based automated contouring of pelvic organs at risk for MRI-guided radiotherapy.

Deep neural nets have revolutionized the science of auto-segmentation and present great promise for treatment planning automation. However, little data exists regarding clinical implementation and human factors. We evaluated the performance and clinical implementation of a novel deep learning-based auto-contouring workflow for 0.35T magnetic resonance imaging (MRI)-guided pelvic radiotherapy, focusing on automation bias and objective measures of workflow savings. An auto-contouring model was developed using a UNet-derived architecture for the femoral heads, bladder, and rectum in 0.35T MR images. Training data was taken from 75 patients treated with MRI-guided radiotherapy at our institution. The model was tested against 20 retrospective cases outside the training set, and subsequently was clinically implemented. Usability was evaluated on the first 30 clinical cases by computing Dice coefficient (DSC), Hausdorff distance (HD), and the fraction of slices that were used un-modified by planners. Final contours were retrospectively reviewed by an experienced planner and clinical significance of deviations was graded as negligible, low, moderate, and high probability of leading to actionable dosimetric variations. In order to assess whether the use of auto-contouring led to final contours more or less in agreement with an objective standard, 10 pre-treatment and 10 post-treatment blinded cases were re-contoured from scratch by three expert planners to get expert consensus contours (EC). EC was compared to clinically used (CU) contours using DSC. Student's t-test and Levene's statistic were used to test statistical significance of differences in mean and standard deviation, respectively. Finally, the dosimetric significance of the contour differences were assessed by comparing the difference in bladder and rectum maximum point doses between EC and CU before and after the introduction of automation. Median (interquartile range) DSC for the retrospective test data were 0.92(0.02), 0.92(0.06), 0.93(0.06), 0.87(0.04) for the post-processed contours for the right and left femoral heads, bladder, and rectum, respectively. Post-implementation median DSC were 1.0(0.0), 1.0(0.0), 0.98(0.04), and 0.98(0.06), respectively. For each organ, 96.2, 95.4, 59.5, and 68.21 percent of slices were used unmodified by the planner. DSC between EC and pre-implementation CU contours were 0.91(0.05*), 0.91*(0.05*), 0.95(0.04), and 0.88(0.04) for right and left femoral heads, bladder, and rectum, respectively. The corresponding DSC for post-implementation CU contours were 0.93(0.02*), 0.93*(0.01*), 0.96(0.01), and 0.85(0.02) (asterisks indicate statistically significant difference). In a retrospective review of contours used for planning, a total of four deviating slices in two patients were graded as low potential clinical significance. No deviations were graded as moderate or high. Mean differences between EC and CU rectum max-doses were 0.1±2.6Gy and -0.9±2.5Gy for pre- and post-implementation, respectively. Mean differences between EC and CU bladder/bladder wall max-doses were -0.9±4.1Gy and 0.0±0.6Gy for pre- and post-implementation, respectively. These differences were not statistically significant according to Student's t-test. We have presented an analysis of the clinical implementation of a novel auto-contouring workflow. Substantial workflow savings were obtained. The introduction of auto-contouring into the clinical workflow changed the contouring behavior of planners. Automation bias was observed, but it had little deleterious effect on treatment planning.

A Technique to Enable Efficient Adaptive Radiation Therapy: Automated Contouring of Prostate and Adjacent Organs

The purpose of this work was to investigate the use of a segmentation approach that could potentially improve the speed and reproducibility of contouring during magnetic resonance-guided adaptive radiation therapy. The segmentation algorithm was based on a hybrid deep neural network and graph optimization approach that also allows rapid user intervention (Deep layered optimal graph image segmentation of multiple objects and surfaces [LOGISMOS]+just enough interaction [JEI]). A total of 115 magnetic resonance-data sets were used for training and quantitative assessment. Expert segmentations were used as the independent standard for the prostate, seminal vesicles, bladder, rectum, and femoral heads for all 115 data sets. In addition, 3 independent radiation oncologists contoured the prostate, seminal vesicles, and rectum for a subset of patients such that the interobserver variability could be quantified. Consensus contours were then generated from these independent contours using a simultaneous truth and performance level estimation approach, and the deviation of Deep LOGISMOS+JEI contours to the consensus contours was evaluated and compared with the interobserver variability. The absolute accuracy of Deep LOGISMOS+JEI generated contours was evaluated using median absolute surface-to-surface distance which ranged from a minimum of 0.20 mm for the bladder to a maximum of 0.93 mm for the prostate compared with the independent standard across all data sets. The median relative surface-to-surface distance was less than 0.17 mm for all organs, indicating that the Deep LOGISMOS+JEI algorithm did not exhibit a systematic under- or oversegmentation. Interobserver variability testing yielded a mean absolute surface-to-surface distance of 0.93, 1.04, and 0.81 mm for the prostate, seminal vesicles, and rectum, respectively. In comparison, the deviation of Deep LOGISMOS+JEI from consensus simultaneous truth and performance level estimation contours was 0.57, 0.64, and 0.55 mm for the same organs. On average, the Deep LOGISMOS algorithm took less than 26 seconds for contour segmentation. Deep LOGISMOS+JEI segmentation efficiently generated clinically acceptable prostate and normal tissue contours, potentially limiting the need for time intensive manual contouring with each fraction.

Does consensus contours improve robustness and accuracy on ^{18}F-FDG PET imaging tumor delineation?

Purpose:The aim of this study is to explore the robustness and accuracy of consensus contours with 225 nasopharyngeal carcinoma (NPC) clinical cases and 13 extended cardio-torso simulated lung tumors (XCAT) based on 2-deoxy-2-[^{18}F]fluoro-D-glucose (^{18}F-FDG) PET imaging.Methods:Primary tumor segmentation was performed with two different initial masks on 225 NPC ^{18}F-FDG PET datasets and 13 XCAT simulations using methods of automatic segmentation with active contour, affinity propagation (AP), contrast-oriented thresholding (ST), and 41% maximum tumor value (41MAX), respectively. Consensus contours (ConSeg) were subsequently generated based on the majority vote rule. The metabolically active tumor volume (MATV), relative volume error (RE), Dice similarity coefficient (DSC) and their respective test–retest (TRT) metrics between different masks were adopted to analyze the results quantitatively. The nonparametric Friedman and post hoc Wilcoxon tests with Bonferroni adjustment for multiple comparisons were performed with P< 0.05 considered to be significant.Results:AP presented the highest variability for MATV in different masks, and ConSeg presented much better TRT performances in MATV compared with AP, and slightly poorer TRT in MATV compared with ST or 41MAXin most cases. Similar trends were also found in RE and DSC with the simulated data. The average of four segmentation results (AveSeg) showed better or comparable results in accuracy for most cases with respect to ConSeg. AP, AveSeg and ConSeg presented better RE and DSC in irregular masks as compared with rectangle masks. Additionally, all methods underestimated the tumour boundaries in relation to the ground truth for XCAT including respiratory motion.Conclusions:The consensus method could be a robust approach to alleviate segmentation variabilities, but did not seem to improve the accuracy of segmentation results on average. Irregular initial masks might be at least in some cases attributable to mitigate the segmentation variability as well.

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation.

Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Participants who contoured region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. ROIs were evaluated against contours using Dice similarity coefficient (DSC). The expert interobserver DSC ( ) was calculated as an acceptability threshold between and . To determine the number of nonexperts required to match the for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the . For all cases, the DSC values for versus were higher than comparator expert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H&N, 3 and 5 for GYN, and 3 for GI. Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

Prospective imaging comparison of anatomic delineation with rapid kV cone beam CT on a novel ring gantry radiotherapy device

IntroductionA kV imager coupled to a novel, ring-gantry radiotherapy system offers improved on-board kV-cone-beam computed tomography (CBCT) acquisition time (17–40 seconds) and image quality, which may improve CT radiotherapy image-guidance and enable online adaptive radiotherapy. We evaluated whether inter-observer contour variability over various anatomic structures was non-inferior using a novel ring gantry kV-CBCT (RG-CBCT) imager as compared to diagnostic-quality simulation CT (simCT). Materials/methodsSeven patients undergoing radiotherapy were imaged with the RG-CBCT system at breath hold (BH) and/or free breathing (FB) for various disease sites on a prospective imaging study. Anatomy was independently contoured by seven radiation oncologists on: 1. SimCT 2. Standard C-arm kV-CBCT (CA-CBCT), and 3. Novel RG-CBCT at FB and BH. Inter-observer contour variability was evaluated by computing simultaneous truth and performance level estimation (STAPLE) consensus contours, then computing average symmetric surface distance (ASSD) and Dice similarity coefficient (DSC) between individual raters and consensus contours for comparison across image types. ResultsAcross 7 patients, 18 organs-at-risk (OARs) were evaluated on 27 image sets. Both BH and FB RG-CBCT were non-inferior to simCT for inter-observer delineation variability across all OARs and patients by ASSD analysis (p < 0.001), whereas CA-CBCT was not (p = 0.923). RG-CBCT (FB and BH) also remained non-inferior for abdomen and breast subsites compared to simCT on ASSD analysis (p < 0.025). On DSC comparison, neither RG-CBCT nor CA-CBCT were non-inferior to simCT for all sites (p > 0.025). ConclusionsInter-observer ability to delineate OARs using novel RG-CBCT images was non-inferior to simCT by the ASSD criterion but not DSC criterion.

Interobserver Agreement among Multiple Generalists is Comparable to that of Recognized Experts: Prospective Acceptability Benchmarks from the C3RO Crowdsourced Initiative

<h3>Purpose/Objective(s)</h3> Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a public crowdsourced challenge engaging radiation oncologists across various expertise levels in cloud-based image-segmentation. A challenge in artificial intelligence (AI) development is the relative paucity of multi-expert observer datasets sufficiently large to train deep learning; consequently, we sought to characterize whether aggregate segmentations generated from large numbers of generalists could meet or exceed expert interobserver agreement, the current "gold standard." <h3>Materials/Methods</h3> Participants who contoured at least one region of interest (ROI) for the C3RO breast or sarcoma challenge were identified as generalist or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLE_generalist ROIs were evaluated against STAPLE_expert contours using Dice Similarity Coefficient (DSC). The expert interobserver DSC (IODSC_expert) (i.e., pairwise median DSC across experts) was calculated as a performance acceptability threshold between STAPLE_generalist and STAPLE_expert. To determine the number of generalists required to match the IODSC_expert for each ROI, a single STAPLE_bootstrap consensus contour was generated for a 10-fold random-bootstrap using a variable number of generalists (between 2-25) and then compared to the IODSC_expert. <h3>Results</h3> The breast challenge yielded contours from 124 generalists and 8 experts. The DSC for STAPLE_generalist versus STAPLE_expert were higher than their respective expert IODSC_expert for all ROIs, including the axilla (STAPLE_generalist/IODSC_expert, 0.86/0.68), chest wall (0.91/0.67), heart (0.97/0.9), supraclavicular nodes (0.77/0.57), internal mammary nodes (0.66/0.46), left brachial plexus (0.46/0.2), and left anterior descending artery (0.62/0.32). The sarcoma case had contours from 61 generalists and 4 experts. The DSC between STAPLE_generalist and STAPLE_expert were higher than their respective IODSC_expert for GTV (0.97/0.94) and CTV (0.76/0.69), but not genitalia (0.60/0.66). The theoretical minimum number of generalist segmentation needed to cross the IODSC_expert acceptability threshold ranged between 2-4 for breast and 2-5 for sarcoma ROIs. <h3>Conclusion</h3> Multi-generalist-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Analyses suggest that 5+ generalists could potentially generate consensus ROIs with DSC performance approximating an individual expert, suggesting multi-generalist segmentations as a feasible AI input. Future research will explore whether these observations are site-specific and/or generalizable to more granular surface metrics.

Contouring Variations and the Role of Deep Learning-Based Auto-Contouring in Breast Cancer Radiation Therapy: A Multi-Institutional Planning Study

<h3>Purpose/Objective(s)</h3> To quantify variations in target and organ-at-risk (OAR) contouring and to investigate whether intervention with deep learning-based auto-contour could improve the interobserver variations. <h3>Materials/Methods</h3> Two breast cancer cases (one with breast conservation surgery and the other with mastectomy with immediate implant-based reconstruction) were distributed with clinical information to multiple institutions that responded to participate in KROG 21-01 (1^st phase). Six months after the 1^st phase data was collected, auto-contour sets were generated for two cases using the deep-learning model that had previously been developed and verified, and the participants were asked to edit and submit the finalized auto-contours (2^nd phase). Consensus contour sets (i.e., ground truth) were generated and approved by members of KROG. Study endpoints were Dice similarity coefficient (DSC), 95% Hausdorff distance (HD), and added path length (APL). <h3>Results</h3> Thirty-one physicians from 31 institutions participated in this study. Statistically significant improvements (<i>p</i> <.05) in DSC, HD, or APL were observed in most target volumes (CTV_Ax_level II and III, CTV_IMN, and CTV_breast) and some OARs (heart, contralateral_breast, Lung_L and R, A_LAD, and thyroid) in the 2^nd phase compared to the 1^st phase. All OARs had a high level of agreement with less variability, except for A_LAD. Despite the high level of agreement in PTV (average interquartile range of DSC and HD; 0.08 and 7.90 mm, respectively), the lymph node CTVs had a considerable variance (the highest being CTV_Ax_level III). For each structure, areas with substantial contouring deviations were identified and shown using heatmaps. <h3>Conclusion</h3> Our findings demonstrated the potential of deep-learning based auto-contours in radiation therapy quality assurance programs for reducing the interobserver variations in multi-center studies.

Suitability of propagated contours for adaptive replanning for head and neck radiotherapy

PurposeAdaptive radiotherapy relies on rapid recontouring for replanning. Contour propagation offers workflow efficiencies, but the impact of using unedited propagated OAR contours directly during re-optimisation is unclear. MethodsPlans for ten head and neck patients were created on the planning CT scan. OAR contours for the spinal cord, brainstem, parotids and larynx were then propagated to five shading-corrected CBCTs equally spaced throughout treatment using five commercial packages. Two reference contours were created on the CBCTs by (1) a clinician and (2) a geometric consensus from the propagated contours. Treatment plans were re-optimised on each CBCT for each set of contours, and the DVH statistic differences to the reference contours were calculated. The spread of DVH statistic differences between the 5th and 95th percentiles was quantified. ResultsThe spread of DVH statistic differences was 3.7 Gy compared to the clinician contour and 3.3 Gy compared to the consensus contour for the brainstem (and PRV) and 2.4 Gy and 2 Gy for the spinal cord (and PRV), across all 5 auto-contouring solutions. The parotids and larynx showed differences of 3.7 Gy compared to the clinician and 0.9 Gy to the consensus contour, with the larger difference for the clinician possibly caused by uncertainty in the clinician standard due to poor image quality on the CBCTs. ConclusionsPropagated OAR contours can be used safely for adaptive radiotherapy replanning, however, where organ doses are close to clinical tolerance then the contours should be reviewed for accuracy regardless of the propagation software used.

Interobserver agreement among multiple generalists or specialists are comparable to that of recognized experts: Prospective acceptability benchmarks for H&N from the C3RO crowdsourced initiative

Background A challenge in the development of artificial intelligence (AI) is the lack of multi-expert observer datasets large enough to train deep learning models; this is particularly true for head and neck (H&N) cases, which have high interobserver segmentation variability. As such, we created Contouring Collaborative for Consensus in Radiation Oncology (C3RO), a crowdsourced challenge engaging international radiation oncologists in cloud-based contouring, to evaluate whether collective contours generated from large numbers of non-experts could meet or exceed expert interobserver agreement, the current "gold standard," in the segmentation of a H&N case. Methods Participants who contoured at least one region of interest (ROI) for the C3RO H&N challenge were categorized as generalist, self-identified specialist, or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEgeneralist ROIs or STAPLEspecialist ROIs were evaluated against STAPLEexpert contours using Dice Similarity Coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as a performance acceptability threshold between STAPLEgeneralist or STAPLEspecialist versus STAPLEexpert. To determine the number of generalists required to match the IODSCexpert for each ROI, a single STAPLEbootstrap consensus contour was generated for a 10-fold random-bootstrap using a variable number of generalists (between 2-25) and then compared to the IODSCexpert. Results This H&N challenge yielded contours from 58 generalists, 8 self-identified specialists, and 15 experts. The DSC for STAPLEgeneralist or STAPLEspecialist versus STAPLEexpert were both higher than their respective expert IODSCexpert for most ROIs, including the right parotid (STAPLEgeneralist/ STAPLEspecialist/ IODSCexpert, 0.95/ 0.94/ 0.87), left parotid (0.94/ 0.91/ 0.86), muscle constrictors (0.8/ 0.6/ 0.58), larynx (0.9/ 0.91/ 0.67), primary gross tumor volume (GTVp) (0.83/ 0.86/ 0.78), right submandibular gland (0.83/ 0.89/ 0.78), left submandibular gland (0.93/ 0.94/ 0.85), and clinical tumor volume 2 (CTV2) (0.84/ 0.82/ 0.7). The DSC for both STAPLEgeneralist and STAPLEspecialist were lower than their respective IODSCexpert for CTV1 (0.68/ 0.65/ 0.85). Interestingly, the DSC for STAPLEgeneralist was higher (0.96), while STAPLEspecialist was lower (0.8) when compared to the IODSCexpert (0.9) for the nodal GTV (GTVn). For the brainstem, the DSC for STAPLEspecialist (0.86) exceeded IODSCexpert (0.82), while the DSC for STAPLEgeneralist did not (0.8). The theoretical minimum number of generalist segmentation needed to cross the IODSCexpert acceptability threshold ranged between 2-5 for all H&N ROIs. Discussion Results show that 5+ generalists could potentially create consensus ROIs with performance approximating an individual expert, facilitating feasible mechanism to improve AI algorithm development for H&N. A challenge in the development of artificial intelligence (AI) is the lack of multi-expert observer datasets large enough to train deep learning models; this is particularly true for head and neck (H&N) cases, which have high interobserver segmentation variability. As such, we created Contouring Collaborative for Consensus in Radiation Oncology (C3RO), a crowdsourced challenge engaging international radiation oncologists in cloud-based contouring, to evaluate whether collective contours generated from large numbers of non-experts could meet or exceed expert interobserver agreement, the current "gold standard," in the segmentation of a H&N case. Participants who contoured at least one region of interest (ROI) for the C3RO H&N challenge were categorized as generalist, self-identified specialist, or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEgeneralist ROIs or STAPLEspecialist ROIs were evaluated against STAPLEexpert contours using Dice Similarity Coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as a performance acceptability threshold between STAPLEgeneralist or STAPLEspecialist versus STAPLEexpert. To determine the number of generalists required to match the IODSCexpert for each ROI, a single STAPLEbootstrap consensus contour was generated for a 10-fold random-bootstrap using a variable number of generalists (between 2-25) and then compared to the IODSCexpert. This H&N challenge yielded contours from 58 generalists, 8 self-identified specialists, and 15 experts. The DSC for STAPLEgeneralist or STAPLEspecialist versus STAPLEexpert were both higher than their respective expert IODSCexpert for most ROIs, including the right parotid (STAPLEgeneralist/ STAPLEspecialist/ IODSCexpert, 0.95/ 0.94/ 0.87), left parotid (0.94/ 0.91/ 0.86), muscle constrictors (0.8/ 0.6/ 0.58), larynx (0.9/ 0.91/ 0.67), primary gross tumor volume (GTVp) (0.83/ 0.86/ 0.78), right submandibular gland (0.83/ 0.89/ 0.78), left submandibular gland (0.93/ 0.94/ 0.85), and clinical tumor volume 2 (CTV2) (0.84/ 0.82/ 0.7). The DSC for both STAPLEgeneralist and STAPLEspecialist were lower than their respective IODSCexpert for CTV1 (0.68/ 0.65/ 0.85). Interestingly, the DSC for STAPLEgeneralist was higher (0.96), while STAPLEspecialist was lower (0.8) when compared to the IODSCexpert (0.9) for the nodal GTV (GTVn). For the brainstem, the DSC for STAPLEspecialist (0.86) exceeded IODSCexpert (0.82), while the DSC for STAPLEgeneralist did not (0.8). The theoretical minimum number of generalist segmentation needed to cross the IODSCexpert acceptability threshold ranged between 2-5 for all H&N ROIs. Results show that 5+ generalists could potentially create consensus ROIs with performance approximating an individual expert, facilitating feasible mechanism to improve AI algorithm development for H&N.

Deep learning-based GTV contouring modeling inter- and intra- observer variability in sarcomas

Background and purposeThe delineation of the gross tumor volume (GTV) is a critical step for radiation therapy treatment planning. The delineation procedure is typically performed manually which exposes two major issues: cost and reproducibility. Delineation is a time-consuming process that is subject to inter- and intra-observer variability. While methods have been proposed to predict GTV contours, typical approaches ignore variability and therefore fail to utilize the valuable confidence information offered by multiple contours. Materials and methodsIn this work we propose an automatic GTV contouring method for soft-tissue sarcomas from X-ray computed tomography (CT) images, using deep learning by integrating inter- and intra-observer variability in the learned model. Sixty-eight patients with soft tissue and bone sarcomas were considered in this evaluation, all underwent pre-operative CT imaging used to perform GTV delineation. Four radiation oncologists and radiologists performed three contouring trials each for all patients. We quantify variability by defining confidence levels based on the frequency of inclusion of a given voxel into the GTV and use a deep convolutional neural network to learn GTV confidence maps. ResultsResults were compared to confidence maps from the four readers as well as ground-truth consensus contours established jointly by all readers. The resulting continuous Dice score between predicted and true confidence maps was 87% and the Hausdorff distance was 14 mm. ConclusionResults demonstrate the ability of the proposed method to predict accurate contours while utilizing variability and as such it can be used to improve clinical workflow.

Oesophageal IGRT considerations for SBRT of LA-NSCLC: barium-enhanced CBCT and interfraction motion

BackgroundTo determine the optimal volume of barium for oesophageal localisation on cone-beam CT (CBCT) for locally-advanced non-small cell lung cancers (NSCLC) and quantify the interfraction oesophageal movement relative to tumour.MethodsTwenty NSCLC patients with mediastinal and/or hilar disease receiving radical radiotherapy were recruited. The first five patients received 25 ml of barium prior to their planning CT and alternate CBCTs during treatment. Subsequent five patient cohorts, received 15 ml, 10 ml and 5 ml. Six observers contoured the oesophagus on each of the 107 datasets and consensus contours were created. Overall 642 observer contours were generated and interobserver contouring reproducibility was assessed. The kappa statistic, dice coefficient and Hausdorff Distance (HD) were used to compare barium-enhanced CBCTs and non-enhanced CBCTs. Oesophageal displacement was assessed using the HD between consensus contours of barium-enhanced CBCTs and planning CTs.ResultsInterobserver contouring reproducibility was significantly improved in barium-enhanced CBCTs compared to non-contrast CBCTs with minimal difference between barium dose levels. Only 10 mL produced a significantly higher kappa (0.814, p = 0.008) and dice (0.895, p = 0.001). The poorer the reproducibility without barium, the greater the improvement barium provided. The median interfraction HD between consensus contours was 4 mm, with 95% of the oesophageal displacement within 15 mm.Conclusions10 mL of barium significantly improves oesophageal localisation on CBCT with minimal image artifact. The oesophagus moves substantially and unpredictably over a course of treatment, requiring close daily monitoring in the context of hypofractionation.

Automated Contour Propagation of the Prostate From pCT to CBCT Images via Deep Unsupervised Learning

To develop and evaluate a deep unsupervised learning (DUL) framework based on a regional deformable model for automated prostate contour propagation from planning computed tomography (pCT) to cone-beam CT (CBCT).A DUL model was introduced to map the prostate contour from pCT to on-treatment CBCT. The DUL framework used a regional deformable model via narrow band mapping to augment the conventional strategy. 251 anonymized CBCT images from prostate cancer patients were retrospectively selected and divided into three sets: 180 were used for training, 12 for validation, and 59 for testing. The testing dataset was divided into two Groups. Group one contained 50 CBCT volumes, with one physician-generated prostate contour on CBCT image. Group two contained 9 CBCT images, each including prostate contours delineated by four independent physicians and a consensus contour generated using the STAPLE method. Results were compared between the proposed DUL and physician-generated contours through the Dice similarity coefficients (DSC), the Hausdorff distances, and the distances of the center-of-mass.The average DSCs between DUL-based prostate contours and reference contours for test data in Group one and Group two-consensus were 0.83 ± 0.04, and 0.85 ± 0.04, respectively. Correspondingly, the mean center-of-mass distances were 3.52 mm ± 1.15 mm, and 2.98 mm ± 1.42 mm, respectively.This novel DUL technique can automatically propagate the contour of the prostate from pCT to CBCT. The proposed method shows that highly accurate contour propagation for CBCT-guided adaptive radiotherapy is achievable via the deep learning technique.

Contouring Collaborative for Consensus in Radiation Oncology (C3RO): An International Crowdsourcing Challenge to Improve Radiotherapy Contour Delineation

Contour variation has been correlated with inferior disease control and increased toxicity. 1 Segedin B Petric P. Uncertainties in target volume delineation in radiotherapy - are they relevant and what can we do about them?. Radiol Oncol. 2016; 50: 254-262 Crossref PubMed Scopus (55) Google Scholar While automated algorithms could decrease variability and improve quality, development has been limited by the availability of large multi-segmented datasets. 2 Mak RH Endres MG Paik JH Sergeev RA Aerts H Williams CL et al. Use of Crowd Innovation to Develop an Artificial Intelligence-Based Solution for Radiation Therapy Targeting. JAMA Oncol. 2019; 5: 654-661 Crossref PubMed Scopus (32) Google Scholar This project aims to conduct a crowdsourced challenge informed by behavioral economics to engage radiation oncologists worldwide in cloud-based contouring. We hypothesize that recruitment strategies and incentives will vary by practice setting. We hypothesize that, among physicians, high volume and specialization will be associated with greater similarity to the consensus contour.

Automated contour propagation of the prostate from pCT to CBCT images via deep unsupervised learning.

To develop and evaluate a deep unsupervised learning (DUL) framework based on a regional deformable model for automated prostate contour propagation from planning computed tomography (pCT) to cone-beam CT (CBCT). We introduce a DUL model to map the prostate contour from pCT to on-treatment CBCT. The DUL framework used a regional deformable model via narrow-band mapping to augment the conventional strategy. Two hundred and fifty-one anonymized CBCT images from prostate cancer patients were retrospectively selected and divided into three sets: 180 were used for training, 12 for validation, and 59 for testing. The testing dataset was divided into two groups. Group 1 contained 50 CBCT volumes, with one physician-generated prostate contour on CBCT image. Group 2 contained nine CBCT images, each including prostate contours delineated by four independent physicians and a consensus contour generated using the STAPLE method. Results were compared between the proposed DUL and physician-generated contours through the Dice similarity coefficients (DSCs), the Hausdorff distances, and the distances of the center-of-mass. The average DSCs between DUL-based prostate contours and reference contours for test data in group 1 and group 2 consensus were 0.83±0.04, and 0.85±0.04, respectively. Correspondingly, the mean center-of-mass distances were 3.52mm±1.15mm, and 2.98mm±1.42mm, respectively. This novel DUL technique can automatically propagate the contour of the prostate from pCT to CBCT. The proposed method shows that highly accurate contour propagation for CBCT-guided adaptive radiotherapy is achievable via the deep learning technique.

MRI prostate contouring is not impaired by the use of a radiotherapy image acquisition set-up. An intra- and inter-observer paired comparative analysis with diagnostic set-up images

PurposeThe use of MRI for radiotherapy planning purposes is growing but image acquisition using radiotherapy set-ups has impaired image quality. Whether differences in image acquisition set-up could modify organ contouring has not been evaluated. Therefore, we aimed to evaluate differences in contouring between paired of image sets that were acquired in the same scanning session using different parameters. Material and methodsTen patients underwent RT treatment planning with MRI co-registration. MRI was carried out using two different set-ups during the same session, MRI radiotherapy set-ups and MRI diagnostic set-ups. Prostates and rectums were retrospectively contoured in both image sets by 5 radiation oncologists and 4 radiologists. Intra-observer analysis was carried out comparing organ volumes, the Dice coefficient and hausdorff distance values between two contouring rounds. Inter-observer analysis was carried out by comparing individual contours to a generated STAPLE consensus contour, which is considered the gold standard reference. ResultsNo significant differences were observed between MRI acquisition set-ups. Significant differences were observed for the dice and hausdorff parameters, comparing individual contours to the STAPLE consensus contour, when analysing diagnostic images between rounds, although raw values were similar. ConclusionProstate and rectum contours did not differ significantly when using diagnostic or radiotherapy MRI acquisition set-ups.

Contouring Variability in Pediatric Abdominal Neuroblastoma: Results of a National Quality Assurance Exercise

To quantify the inter-clinician contouring variability in pediatric abdominal neuroblastoma based on a national pre-trial quality assurance exercise for a multi-center clinical trial. As part of the quality assurance process for the IMAT Neuroblastoma prospective Phase 1/2 trial (ISRCTN10746820), completion of a pre-trial benchmark contouring case was required. Diagnostic MRI, pre-surgical CT, radiotherapy planning CT, and a short clinical history were supplied to each center. Clinicians were requested to delineate the virtual gross tumor volume (GTV) defined as the reconstructed tumor volume seen on the post induction chemotherapy, pre-surgical imaging, as well as any immediately adjacent persistently enlarged lymph nodes, and edited to exclude uninvolved normal organs. Clinical target and planning target volumes (CTV and PTV) were then generated prior to submission and relevant organs at risk (OAR) were also delineated. Contours were reviewed centrally by a trial clinician, who provided feedback in a narrative report with images used for visual clarification. A consensus GTV contour was determined from all submissions using the simultaneous truth and performance level estimation (STAPLE) algorithm at the 95% confidence level. DICE similarity coefficient (DSC) and mean Hausdorff distance were calculated for each submitted contour compared to the STAPLE consensus contour. Clinicians from 12/16 pediatric radiotherapy centers in the United Kingdom submitted contours for review. GTV contours were classified as unacceptable deviations in 3/12 cases and required subsequent amendment prior to acceptance. Narrative detail was provided in 10/12 reviews and identified regions of under-contouring (9/10) and over-contouring (3/10). Under-contouring was identified in the para-aortic nodal region (5/9), retrocrural nodal region (1/9), unspecified regions of residual calcification (2/9), and more generally (2/9). Over-contouring was identified in uninvolved organs (2/3) and more generally (1/3). Substantial variation was present in the submitted GTVs when compared to the consensus contour. Averaged over all clinicians, the DSC was 0.61 ± 0.15 (mean ± 1 S.D.) and mean Hausdorff distance was 4.54 ± 2.25 mm. Good agreement was present for the organs at risk, however. Considerable variability in GTV contouring has been identified between clinicians, primarily in the extent to which nodes and residual calcification had been included. This supports the use of prospective contour review for target volumes in future abdominal neuroblastoma clinical trials. Limited variation in OAR contouring suggests that quality assurance could be streamlined for future trials.