A Workshop-Based Evaluation of a Tool for Online Feedback During Contouring

Abstract

<h3>Purpose/Objective(s)</h3> Inter-observer variability (IOV) in target volume delineation is one of the largest sources of uncertainty in radiotherapy. It can be reduced by following guidelines and training, but the effects of training fade over time. Our aim was to evaluate a tool that provides automatic feedback during delineation and its impact on breast and lymph node IOV, using contouring workshops. <h3>Materials/Methods</h3> Automatic online feedback was presented via warning annotations displayed on the delineated contours when the observer crossed medial, lateral and skin limits. These limits were generated based on a machine learning solution that learnt guideline limits from 150 expertly contoured cases. Warning annotations were also shown when the users were using incorrect local window/level settings, based on analysis of the displayed grey values close to the contour. The system was tested in two 2-day contouring workshops (WS1, WS2). During WS1, we assessed acceptability of the tool via qualitative feedback. Prior to WS2, participants were asked to contour two cases (breast and lymph nodes). During the first day, training was provided on breast delineation following international guidelines. The participants then delineated two additional cases. The final day included additional training, discussions, and general feedback. For each workshop, the tool was enabled for one case. To evaluate IOV, the standard deviation (SD) of the perpendicular distances from every point on each observer's contour to the median of the contours was calculated. SDs for the cases with/without feedback were compared pairwise per observer and volume. We report changes in SD before and after training, and defined <i>improvement</i> when SD reduced by >2mm. <h3>Results</h3> Feedback after WS1 showed acceptability of the tool and its potential to influence observers even with minimal training. In WS2, 10 participants, out of 16, delineated all cases (138 contours). Training improved consistency in 23% of analyzed contours for cases when the tool was disabled, and only in 14% when the tool was enabled. SD changes <2mm were 63% for both cases. <h3>Conclusion</h3> Real-time automated feedback during contouring is feasible and has the potential to reduce IOV. Our results suggest that it can have similar effects to training. Its use for day-to-day contouring could act as a permanent reminder of guidelines, which may mitigate the fading effect seen after training. This could help to improve contouring by automatically providing anatomical limits to clinically defined target volumes and promoting correct window-level usage.