An Independent and Automatic Quality Assurance Tool for Auto-Segmentation

Abstract

Deep-learning based auto-segmentation has been shown to be efficient and accurate or consistent for radiotherapy planning. However, in current practice, manual review of auto-segmented contours before their clinical use is necessary and can be labor intensive and time consuming. This study aims to develop an independent and automatic contour quality assurance (QA) tool that can quickly identify inaccurate or inconsistent contour slices from auto-segmented contours, eliminating the need of manual QA process. The automatic contour QA tool starts from building a model, which includes: (1) pre-processing images, (2) extracting quantitative geometric and multi-scale texture features from the test contours and the feature changes between 4-mm inner/outer shells and a core region of the test contours on a slice-by-slice basis, (3) selecting a subset of organ-specific discriminative features that maximizes the model’s predictive performance using a recursive feature elimination method, and (4) determining the best performing model among 12 different machine learning-based supervised classification models. The process was tested using an MRI head and neck dataset. Contours of the parotid glands and submandibular glands on T2-weighted non-contrast MRIs from 31 patients were utilized, where 21 were used for training and 10 for testing. During model training, manually-delineated contours for each training case were utilized as accurate samples. The inaccurate samples were collected from a set of deformable image registration (DIR) propagated contours using a labeling criterion of mean distance to agreement ≥ 2 mm or Hausdorff distance ≥ 8mm as compared to ground truth contours. For the 10 MRIs in the test set, two sets of contours using a deep-learning based auto-segmentation software and DIR propagation were used to evaluate the model performance. All the contour slices were labeled using the same labeling criterion and manually checked to avoid mislabeling. The classification accuracy was used to measure a model’s performance. For each contour slice, a total of 437 features were extracted. At the training stage, 58, 41, 72, and 56 features were automatically selected for left and right parotid gland and submandibular glands contours, respectively. The best model had an accuracy of 89% for the four organs. The time required to evaluate one patient dataset is less than 30 seconds using a workstation equipped with a i7-6700 CPU. The developed automatic contour QA tool can quickly identify accurate and inaccurate contour slices with high accuracy from auto-segmented contours in head and neck. This tool is being integrated into a deep learning-based auto-segmentation package, to eventually develop a fully automatic and robust auto-segmentation pipeline for radiotherapy planning and delivery.