Abstract 9330: Machine Learning Can Enable the Automation of Clinical Endpoint Adjudication

Abstract

Introduction: Clinical Endpoint Adjudication is a critical component of Cardiovascular (CV) clinical outcome trials and relies on manual decision making to adjudicate clinical events, including CV death. This process is resource intensive and subject to human-driven variance. As part of a broader initiative to automate data collection and analysis of clinical endpoints, we evaluated whether machine learning (ML) algorithms could replicate the outcomes of expert adjudication for CV death. Algorithms were trained on data from THEMIS (NCT01991795), a large, randomized trial comparing ticagrelor to placebo in patients with Type 2 Diabetes Mellitus. Methods: We deployed a deep learning, biomedical named entity (NE) extraction model called BERN to extract relevant NE’s from clinical text of THEMIS events. NE’s were transformed into a sparse, numerical matrix and concatenated with event-level structured features to form a data set of 962 events. Expert adjudicator consensus for CV death was used as ground truth. We trained models using grid search and cross validation, experimenting with XGBoost, Random Forest, Logistic Regression, and Naïve Bayes. Performance was assessed on a 25% (240 of 962) validation subset of the data excluded from training. Metrics used to evaluate performance were Precision, Recall, Accuracy and Area Under the Receiver Operating Characteristic Curve (AUC). Results: Best performance was observed on models trained using Naïve Bayes (>97% ROC AUC on validation data), see Table. Top ranked features for classifying CV death included site investigator decision, sex, and NE’s associated with diagnoses or symptoms, such as “edema” and “chest pain.” Conclusion: With high consistency between automated and expert adjudication, our models demonstrate that machine learning may augment or even replace clinician adjudication in CV outcome trials. Subsequent research will focus on pursuing ML approaches to adjudicate other outcome events.