Comparative Evaluation of Four Software Packages for Automated Contouring

Abstract

<h3>Purpose/Objective(s)</h3> We are an academic radiation oncology center and treat 1500-2000 patients annually. The majority of target volume definition and organ at risk (OAR) contouring is performed by radiation oncologists (RO) and is estimated to require 10-15 hours per week for each RO. Artificial intelligence (AI) solutions for contouring are a possible method of reducing the time required. Several packages are commercially available. The purpose of this exercise was to evaluate 4 packages to identify one which would meet our department's need. The primary objective was to calculate the potential time savings associated with each of the 4 packages. The second objective was to perform a qualitative evaluation and comparison. <h3>Materials/Methods</h3> Approval from the hospital's Research Ethics Committee was obtained. CT Simulation datasets for 15 patients were anonymized for use as test cases for AI contouring. This included 5 datasets from each of 3 sites – head and neck (H+N), breast/thorax, and prostate/pelvis. 4 potential vendors provided access to their product for automated contouring of OARs +/- target volumes of each of the 15 datasets. Analysis was as follows:<ul><li>1.One RO used a stopwatch to assess the time taken for them to contour each of the OARs +/- targets on each dataset</li><li>2.The AI contours from each of the vendors for each of the datasets was reviewed.</li><li>3.The time taken to correct each auto-contoured organ to make it clinically useful was timed. These times were summed to give an overall time for each vendor for each dataset.</li><li>4.The absolute time saving was calculated by subtracting the result of step (3) from step (1)</li><li>5.The percentage time saving was calculated by dividing this value by the baseline time for contouring by the RO</li><li>6.For the purpose of qualitative assessment, 2 ROs reviewed each dataset and assigned an objective score of 1-5 to each auto- contoured organ (e.g., 5 = no, or almost no, edits required; clinically useful as is)</li><li>7.The scores for each organ in each of the 15 datasets were averaged to give a mean value for the software package.</li></ul> <h3>Results</h3> Baseline contouring time was 32.8 minutes for H+N, 23.0 minutes for breast/thorax, and 64.8 minutes for prostate/pelvis. One package was removed from consideration after assessment of H+N only as it was deemed to be of insufficient standard to be of clinical use. The average time savings for each of the remaining 3 software packages were 15.7, 17.9, and 12.8 minutes for H+N, 3.1, 13.1, and 11.0 minutes for breast/thorax, and 32.6, 35.2, and 36.2 minutes for prostate/pelvis. Percentage time savings ranged from 39-55% for H+N, 13-57% for breast/thorax, and 50-56% for prostate/pelvis. The overall qualitative assessments on a 5-point scale were 3.5, 4.0, and 3.6. <h3>Conclusion</h3> The AI contouring solutions were associated with meaningful savings in the time taken to contour. The introduction of one of these products into clinical practice within our department will permit ROs to focus more of their time on other clinical activities.