0610 Development of a Rule-Based Text Mining Algorithm to Identify Sleep Complaints in Primary Care Progress Notes

Abstract

Abstract Introduction Sleep complaints are among the most common reasons to seek medical attention, yet sleep disorders are largely underdiagnosed in primary care settings. The ability to process large collections of unstructured clinical notes might offer an opportunity to promote screening of patients suffering from significant sleep disorders. The goal of this study was to develop a simple rule-based algorithm to identify sleep complaints in progress notes from primary care encounters and validate the performance of the algorithm against manual chart review. Methods De-identified progress notes of a random sample of patients with primary care encounters at the University of Kansas Health System in 2019 were extracted from the institution’s clinical research data warehouse (Healthcare Enterprise Repository for Ontological Narration). Review of 163 notes from patients enriched for presence (N=95) or absence (N=68) of sleep disorders based on the International Classification of Disease (ICD)-10 code hierarchy G47 guided the development of a vocabulary of sleep complaints and symptoms, including corresponding negation terms. The vocabulary was used to design a rule-based, regular expression matching algorithm, which was evaluated against manual chart review of the same patient cohort (training dataset). An independent set of notes from another sample of patients with primary care encounters (N=77; testing dataset) was also manually reviewed and used to assess the validation performance of the algorithm. Results In the training dataset, the algorithm had a sensitivity=75%; specificity=91%, positive predictive value (PPV)=90%, and a negative predictive value (NPV)=87%. The area under the receiver operating characteristics (AUC) curve in the training set was 0.84. When the algorithm was evaluated in the testing dataset, we found a sensitivity=53%, specificity=91%, PPV=78%, and NPV=77%. The AUC in the testing dataset was 0.78. Conclusion A simple pattern matching algorithm designed to identify sleep complaints in primary care progress notes showed good performance in the training set and acceptable performance in the testing set. Further refinement of this algorithm with potential incorporation of natural language processing might offer a feasible approach to screen patients for underdiagnosed sleep disorders using primary care clinical notes. Support (If Any) NIH CTSA UL1TR002366.