Evaluation of reader performance during interpretation of breast cancer screening: the Recall and detection Of breast Cancer in Screening (ROCS) trial study design

Abstract

The magnitude of the tradeoff between recall rate (RR) and cancer detection rate (CDR) in breast-cancer screening is not clear, and it is expected to depend on target population and screening program characteristics. Multi-reader multi-case research studies, which may be used to estimate this tradeoff, rely on enriched datasets with artificially high prevalence rates, which may bias the results. Furthermore, readers participating in research studies are subject to “laboratory” effects, which can alter their performance relative to actual practice. The Recall and detection Of breast Cancer in Screening (ROCS) trial uses a novel data acquisition system that minimizes these limitations while obtaining an estimate of the RR-CDR curve during actual practice in the Dutch National Breast Cancer Screening Program. ROCS involves collection of at least 40,000 probability-of-malignancy ratings from at least 20 radiologists during interpretation of approximately 2,000 digital mammography screening cases each. With the use of custom-built software on a tablet, and a webcam, this data was obtained in the usual reading environment with minimal workflow disruption and without electronic access to the review workstation software. Comparison of the results to short- and medium-term follow-up allows for estimation of the RR-CDR and receiver operating characteristics curves, respectively. The anticipated result of the study is that performance-based evidence from practice will be available to determine the optimal operating point for breast-cancer screening. In addition, this data will be useful as a benchmark when evaluating the impact of potential new screening technologies, such as digital breast tomosynthesis or artificial intelligence.Key Points• The ROCS trial aims to estimate the recall rate–cancer detection rate curve during actual screening practice in the Dutch National Breast Cancer Screening Program.• The study design is aimed at avoiding the influence of the “laboratory effect” in usual observer performance studies.• The use of a tablet and a webcam allows for the acquisition of probability of malignancy ratings without access to the review workstation software.