Abstract 12454: Weak Supervision Enables Scalable Post-Market Surveillance on Medical Wearables

Abstract

Introduction: With the advent of consumer-facing devices that can render atrial fibrillation (AF) pre-diagnosis, medical wearables now have the potential to affect diagnosis rates and medical care. Post-market surveillance is necessary to understand the impact of wearables on patient outcomes and health care utilization, but is hindered by the lack of codified terms in EHR that capture wearable use. Research Questions: Constructing a post-market surveillance system therefore requires a classifier that identifies mentions of AF pre-diagnosis in unstructured EHR data. However, fine-tuning classifiers require large, hand-labeled training sets that can be costly to generate. It is unclear whether a scalable surveillance system could be built for wearables. Methods: Two weakly-supervised approaches were evaluated to reduce clinical note labeling overhead: one using programmatic codes, and the other using prompts to large language models (LLMs). Probabilistically labeled notes were then used to fine-tune a classifier, which identified patients with AF pre-diagnosis mentions. A retrospective cohort study was conducted, where the characteristics and care patterns of patients identified by the classifier were compared against those who did not receive pre-diagnosis. Results: Label model derived from prompt-based labeling heuristics using LLMs (precision = 0.67, recall = 0.83, F1 = 0.74) nearly achieved the performance of code-based heuristics (precision = 0.84, recall = 0.72, F1 = 0.77), while cutting down the cost to create a labeled training set. The classifier learned on the labeled notes accurately identified patients with AF pre-diagnosis (precision = 0.85, recall = 0.81, F1 = 0.83). Those patients who received pre-diagnosis exhibited different demographic and comorbidity characteristics, and were enriched for anticoagulation and eventual diagnosis of AF. At the index diagnosis, existence of pre-diagnosis did not stratify patients on clinical characteristics, but did correlate with anticoagulant prescription. Conclusion: Our work establishes the feasibility of an EHR-based surveillance system for wearable devices that render AF pre-diagnosis. Weakly-supervised approaches can be effective in reducing test set labeling costs.