S237 Validation of Machine Learning Tool to Determine Post-Polypectomy Surveillance Intervals After Screening Colonoscopy in a Large Health System

Abstract

Introduction: Accurate determination and documentation of post-polypectomy surveillance intervals after screening colonoscopy is essential to reduce colorectal cancer (CRC) incidence and mortality. However, this remains clinically challenging particularly in light of recent post-polypectomy guidelines that increase the number of surveillance intervals. In order to improve guideline-concordant surveillance after CRC screening, we sought to validate a previously developed natural language processing (NLP) algorithm that automates determination of post-polypectomy colonoscopy surveillance intervals. Methods: UCLA Health is a large, academic healthcare system performing over 15,000 screening colonoscopies per year. We previously developed an automated NLP pipeline to extract and analyze relevant polyp features from free-text colonoscopy (polyp number, size, location) and pathology reports (polyp histology) to determine appropriate post-polypectomy surveillance interval based on 2020 USMSTF guidelines (Figure 1). To validate the algorithm, we used a random selection of screening colonoscopies performed between 6/2/2020 and 2/2/2021 for individuals at average-risk for CRC. We determined the performance (sensitivity, specificity, PPV, F-score, accuracy) of the NLP algorithm to identify guideline-concordant surveillance intervals compared to manual chart review and characterized common errors made by the algorithm. Results: Our validation cohort (n = 469) was 50.3% female and mean age was 57.9 (SD = 7.3). The most common surveillance interval per chart review was 10-years (60.1%), followed by 7-10 years (22.8%) (Table 1). The NLP pipeline identified the appropriate surveillance interval after screening colonoscopy with an overall 82.6% sensitivity (range: 68.8-99.6%), 98.3% specificity (95.7-99.8%), 82.4% PPV (42.3-97.2%), 81.6% F-score (52.4-98%), and 97.6% accuracy (95.5-99.8%) (Table 1). Among the 58 cases misclassified by the NLP algorithm, 14 (25.9%) were allocated a longer surveillance interval than the manual gold standard. Conclusion: We developed an automated NLP algorithm that is moderately sensitive and highly specific at identifying appropriate surveillance intervals after screening colonoscopy. Future work will refine the algorithm to increase sensitivity and specificity. This tool will help providers and health systems achieve the ASGE/ACG Taskforce screening colonoscopy quality indicator goal that > 90% screening colonoscopies document a guideline-concordant surveillance interval.Figure 1.: Automated NLP Pipeline to Determine Guideline-Concordant Post-Polypectomy Surveillance Intervals (Gupta et al. 2020; US Multi-Society Task Force (MSTF)). HP, hyperplastic polyp; TA, tubular adenoma; SSP, sessile serrated polyp; TSA, total serrated polyp; TVA, tubulovillous adenoma; HGD, high-grade dysplasia.Table 1.: A) Surveillance colonoscopy interval determination from manual chart review versus NLP pipeline, N=469. B) Performance of the NLP algorithm to identify appropriate surveillance interval after screening colonoscopy, N=469