Quality Assurance of Contouring for NRG Oncology/RTOG 1308 Clinical Trial Based on Automated Segmentation with Deep Active Learning

Abstract

High quality data for clinical trials requires that contours are generated in compliance with definitions in protocol. The current NCI-funded Imaging and Radiation Oncology Core (IROC) workflow includes manual review of the submitted contours, which is time-consuming and subjective. In this project, we developed an automated quality assurance (QA) system based on a segmentation model trained with deep active learning to detect contouring errors automatically and improve the current clinical workflow. The study used data that included a golden atlas with 36 cases from ‘Lung CT Segmentation Challenge 2017’ and 110 cases from NRG Oncology/RTOG 1308, which was divided into three groups: the first 70 cases enrolled were used as the candidate set, the following 40 cases were randomly assigned to a validation set (20 cases) and a test set (20 cases). The OARs included heart, esophagus, spinal cord, left and right lung. The proposed QA system was based on automated segmentation that consisted of four steps. First, we trained a convolutional neural networks (CNN) segmentation model with the golden atlas, even though it does not represent the whole population. This deficiency is overcome with the second step where we selected quality images from the candidate set to be added to the training set for fine-tuning of the model. The image selection strategy was based on the representativeness, defined with a parameter combined with the ‘uncertainty’ and ‘accuracy’ (quantities for image segmentation performance evaluation). We utilized the top 30% samples from the candidate set from the ranking of the representativeness and repeated the first two steps twice. Third, after the fine-tuning, we evaluated the accuracy of the segmentation model on the validation set, of which the contours were verified for its accuracy. The metrics included Dice and Hausdorff distance. The mean and standard deviation (σ) of the two metrics for each OAR were calculated and the QA passing criteria were set with a threshold of mean +/−1.96σ. Finally, we applied the fine-tuned CNN model and the decision criteria to the test set to assess the performance of the QA system. The quantitative metrics included balanced accuracy, sensitivity, specificity and the area under the receiving operator characteristic curve (AUC). The proposed QA method achieved promising contour error detection, with balanced accuracy of 0.96, 0.95, 0.96, 0.97 and 0.97, sensitivity of 0.95, 0.98 0.96, 1 and 1, and specificity of 0.98, 0.92, 0.97, 0.94 and 0.94, AUC of 0.96, 0.95, 0.96, 0.97 and 0.94, for heart, esophagus, spinal cord, left and right lung, respectively. The CT slices with error-containing contours can be detected for further evaluation. We have created a system that can automatically detect contour errors for multi-center clinical trial quality assurance. The implementation of such a system in clinical trials can provide consistent and objective evaluations with much reduced investigator intervention.