Evaluation of Intra-Observer Variation for Deep Learning Generated Head and Neck Organs at Risk Segmentation

Abstract

<h3>Purpose/Objective(s)</h3> Supervised deep learning based automated delineating studies often rely on human experts for providing gold-standard contours and evaluating the contours delineated by deep learning models. Hence, the degree of intra-observer variation may affect the quality of deep learning-based delineating research. This study aims to evaluate the intra-observer variation in the contour reviewing process. <h3>Materials/Methods</h3> We used 30 testing patients from an institution and enrolled a senior radiation oncologist who has initially delineated the contours of head and neck OARs of these patients as gold-standard. The testing treatment plan of each patient is composed of 13 head and neck OARs randomly selected organs from three sources: gold-standard, deep learning generated OAR contours and delineation by another radiation oncologist. The real contouring source for each OAR was kept unknown to this physician, and the physician had to decide if a particular OAR on each patient's treatment plan requires revision or not. The physician's decision on each OAR presented was then recorded and analyzed. <h3>Results</h3> 15% gold-standard contours required further editing, reflecting the intra-observer variation in deciding whether the revision of an OAR is needed. For deep learning generated contours, 43% requires revision, which is slightly higher than that in the earlier unblind assessment by this physician, where 37% deep learning generated contours required revision among the 13 OARs. However, since the number of deep learning generated contours requiring revision from two times study is close, it indicates that our intra-observer variation is within a small range. Moreover, compared with deep learning generated contours, a noticeable higher number of another human reader's contours requires revision (55% vs 43% of deep learning generated contours), reflecting that deep learning generated contours' quality is generally better than the human reader's in the blind assessment. <h3>Conclusion</h3> This study has demonstrated that inherent intra-observer variation exists in a deep learning based automated delineating study. The observer disapproves 15% of own contouring in the blind reviewing experiment. Deep learning generated OAR contours remain comparable editing rate in the blind process and are of better quality than another human reader's manual contours.