Structured Electronic Health Record Research Articles

Use of Electronic Health Record-Based Measures to Assess Quality of Care for Pediatric Agitation.

Acute agitation during pediatric mental health emergency department (ED) visits presents safety risks to patients and staff. We previously convened multidisciplinary stakeholders who prioritized 20 proposed quality measures for pediatric acute agitation management. Our objectives were to assess feasibility of evaluating performance on these quality measures using electronic health record (EHR) data and to examine performance variation across 3 EDs. At a children's hospital and 2 nonchildren's hospitals, we assessed feasibility of evaluating quality measures for pediatric acute agitation management using structured EHR data elements. We retrospectively evaluated measure performance during ED visits by children 5 to 17 years old who presented for a mental health condition, received medication for agitation, or received physical restraints from July 2020 to June 2021. Bivariate and multivariable regression were used to examine measure performance by patient characteristics and hospital. We identified 2785 mental health ED visits, 275 visits with medication given for agitation, and 35 visits with physical restraints. Performance was feasible to measure using EHR data for 10 measures. Nine measures varied by patient characteristics, including 4.87 times higher adjusted odds (95% confidence interval 1.28-18.54) of physical restraint use among children with versus without autism spectrum disorder. Four measures varied by hospital, with physical restraint use varying from 0.5% to 3.3% of mental health ED visits across hospitals. Quality of care for pediatric acute agitation management was feasible to evaluate using EHR-derived quality measures. Variation in performance across patient characteristics and hospitals highlights opportunities to improve care quality.

Clinical risk prediction using language models: benefits and considerations.

The use of electronic health records (EHRs) for clinical risk prediction is on the rise. However, in many practical settings, the limited availability of task-specific EHR data can restrict the application of standard machine learning pipelines. In this study, we investigate the potential of leveraging language models (LMs) as a means toincorporate supplementary domain knowledge for improving the performance of various EHR-based risk prediction tasks. We propose two novel LM-based methods, namely "LLaMA2-EHR" and "Sent-e-Med." Our focus is on utilizing the textual descriptions within structured EHRs to make risk predictions about future diagnoses. We conduct a comprehensive comparison with previous approaches across various data types and sizes. Experiments across 6 different methods and 3 separate risk prediction tasks reveal that employing LMs to represent structured EHRs, such as diagnostic histories, results in significant performance improvements when evaluated using standard metrics such as area under the receiver operating characteristic (ROC) curve and precision-recall (PR) curve. Additionally, they offer benefits such as few-shot learning, the ability to handle previously unseen medical concepts, and adaptability to various medical vocabularies. However, it is noteworthy that outcomes may exhibit sensitivity to a specific prompt. LMs encompass extensive embedded knowledge, making them valuable for the analysis of EHRs in the context of risk prediction. Nevertheless, it is important to exercise caution in their application, as ongoing safety concerns related to LMs persist and require continuous consideration.

Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative

ObjectiveAdvances in artificial intelligence have enabled the development of predictive models for glaucoma. However, most work is single-center and uncertainty exists regarding the generalizability of such models. The purpose of this study was to build and evaluate machine learning (ML) approaches to predict glaucoma progression requiring surgery using data from a large multicenter consortium of electronic health records (EHR). DesignCohort study. Participants36,548 patients with glaucoma, as identified by ICD codes from six academic eye centers participating in the Sight OUtcomes Research Collaborative (SOURCE). MethodsWe developed machine learning models to predict whether glaucoma patients would progress to glaucoma surgery in the coming year (identified by CPT codes) using the following modeling approaches: 1) penalized logistic regression (lasso, ridge, and elastic net); 2) tree-based models (random forest, gradient boosted machines, XGBoost), and 3) deep learning models. Model input features included demographics, diagnosis codes, medications, and clinical information (intraocular pressure, visual acuity, refractive status, and central corneal thickness) available from structured EHR data. One site was reserved as an “external site” test set (N=1550); of the patients from the remaining sites, 10% each were randomly selected to be in development and test sets, with the remaining 27999 reserved for model training. Main Outcome and MeasuresEvaluation metrics included area under the receiver operating characteristic curve (AUROC) on the test set and the external site. Results6019 (16.5%) of 36,548 patients underwent glaucoma surgery. Overall, the AUROC ranged from 0.735-0.771 on the random test set and from 0.706-0.754 on the external test site, with the XGBoost and random forest model performing best, respectively. There was greatest performance decrease from the random test set to the external test site for the penalized regression models. ConclusionsML models developed using structured EHR data can reasonably predict whether glaucoma patients will need surgery, with reasonable generalizability to an external site. Additional research is needed to investigate the impact of protected class characteristics such as race or gender on model performance and fairness.

A cross-modal clinical prediction system for intensive care unit patient outcome

Electronic Health Records (EHRs) are becoming an encyclopedia of clinical concepts, encompassing diagnoses, medications and clinical descriptions to describe patients’ health conditions. The abundance of EHR data presents unprecedented opportunities for leveraging machine learning in clinical predictive tasks. However, the complex process of acquiring healthcare data in some settings like Intensive Care Unit (ICU) gives rise to various challenges in mining EHR, such as modality diversity, heterogeneity, data silo and data quality problems, etc. In this paper, to overcome the data heterogeneity and data quality issues, we propose a Code-Text cross-modal Contrastive Learning (CTCL) framework specifically designed for predicting ICU patient outcomes. This framework leverages both structured (multi-view medical codes) and unstructured (clinical notes) EHR data to extract patient representations. To capture the complicated intra and inter interactions among medical codes, CTCL employs a multi-view graph convolution network to extract code encodings and a cross-view contrastive loss to regularize the model parameters. To enhance patient representations across different modalities, a cross-modal contrastive learning module is introduced. Experimental evaluations on the eICU-CRD database demonstrate that CTCL outperforms state-of-the-art competitors on the ICU mortality prediction task and in-hospital readmission prediction task. Furthermore, ablation studies and discussions demonstrate the effectiveness of each module of the framework.

Associations between blood pressure control and clinical events suggestive of nutrition care documented in electronic health records of patients with hypertension

BackgroundClinical events suggestive of nutrition care found in electronic health records (EHRs) are rarely explored for their associations with hypertension outcomes.MethodsLongitudinal analysis using structured EHR data from primary care visits at a health system in the US from December 2017—December 2020 of adult patients with hypertension (n = 4,237) tested for associations between last visit blood pressure (BP) control (≤ 140 Systolic BP and ≤ 90 Diastolic BP) and ≥ 1 nutrition care clinical event operationalized as (overweight or obesity (BMI > 25 or 30, respectively) diagnoses, preventive care visits, or provision of patient education materials (PEM)). Descriptive statistics and longitudinal targeted maximum likelihood estimation (LTMLE) models were conducted to explore average treatment effects (ATE) of timing and dose response from these clinical events on blood pressure control overall and by race.ResultsThe median age was 62 years, 29% were male, 52% were Black, 25% were from rural areas and 50% had controlled BP at baseline. Annual documentation of overweight/obesity diagnoses ranged 3.0–7.8%, preventive care visits ranged 6.2–15.7%, and PEM with dietary and hypertension content were distributed to 8.5–28.8% patients. LTMLE models stratified by race showed differences in timing, dose, and type of nutrition care. Black patients who had nutrition care in Year 3 only compared to none had lower odds for BP control (ATE -0.23, 95% CI: -0.38,-0.08, p = 0.003), preventive visits in the last 2 years high higher odds for BP control (ATE 0.31, 95% CI: 0.07,0.54, p = 0.01), and early or late PEMs had lower odds for BP control (ATE -0.08, 95% CI: -0.15,-0.01, p = 0.03 and ATE -0.23, 95% CI: -0.41,-0.05, p = 0.01, respectively).ConclusionsIn this study, clinical events suggestive of nutrition care are significantly associated with BP control, but are infrequent and effects differ by type, timing, and patient race. Preventive visits appear to have the most effect; additional research should include examining clinical notes for evidence of nutrition care among different populations, which may uncover areas for improving nutrition care for patients with chronic disease.

Comparing Natural Language Processing and Structured Medical Data to Develop a Computable Phenotype for Patients Hospitalized Due to COVID-19: Retrospective Analysis.

Throughout the COVID-19 pandemic, many hospitals conducted routine testing of hospitalized patients for SARS-CoV-2 infection upon admission. Some of these patients are admitted for reasons unrelated to COVID-19, and incidentally test positive for the virus. Because COVID-19-related hospitalizations have become a critical public health indicator, it is important to identify patients who are hospitalized because of COVID-19 as opposed to those who were admitted for other indications. We compared the performance of different computable phenotype definitions for COVID-19 hospitalizations that use different types of data from the electronic health records (EHR), including structured EHR data elements, provider notes, or a combination of both data types. We conducted a retrospective data analysis utilizing clinician chart review-based validation at a large academic medical center. We reviewed and analyzed the charts of 586 hospitalized individuals who tested positive for SARS-CoV-2 during January 2022. We used LASSO regression and Random Forests to fit classification algorithms that incorporated structured EHR data elements, provider notes, or a combination of structured data and provider notes. We used natural language processing to incorporate data from provider notes. The performance of each model was evaluated based on Area Under the Receiver Operator Characteristic (AUROC) and an associated decision rule based on sensitivity and positive predictive value. We also identified top words and clinical indicators of COVID-19-specific hospitalization and assessed the impact of different phenotyping strategies on estimated hospital outcome metrics. Based on a chart review, 38% of 586 patients were determined to be hospitalized for reasons other than COVID-19 despite having tested positive for SARS-CoV-2. A computable phenotype that used provider notes had significantly better discrimination than one that used structured EHR data elements (AUROC: 0.894 vs 0.841, P < .001), and performed similarly to a model that combined provider notes with structured data elements (AUROC: 0.894 vs 0.893). Assessments of hospital outcome metrics significantly differed based on whether the population included all hospitalized patients who tested positive for SARS-CoV-2 versus those who were determined to have been hospitalized due to COVID-19. These findings highlight the importance of cause-specific phenotyping for COVID-19 hospitalizations. More generally, this work demonstrates the utility of natural language processing approaches to derive information related to patient hospitalizations in cases where there may be multiple conditions that could serve as the primary indication for hospitalization.

Using artificial intelligence to identify patients with migraine and associated symptoms and conditions within electronic health records

BackgroundReal-world evidence (RWE)—based on information obtained from sources such as electronic health records (EHRs), claims and billing databases, product and disease registries, and personal devices and health applications—is increasingly used to support healthcare decision making. There is variability in the collection of EHR data, which includes “structured data” in predefined fields (e.g., problem list, open claims, medication list, etc.) and “unstructured data” as free text or narrative. Healthcare providers are likely to provide more complete information as free text, but extracting meaning from these fields requires newer technologies and a rigorous methodology to generate higher-quality evidence. Herein, an approach to identify concepts associated with the presence and progression of migraine was developed and validated using the complete patient record in EHR data, including both the structured and unstructured portions.Methods“Traditional RWE” approaches (i.e., capture from structured EHR fields and extraction using structured queries) and “Advanced RWE” approaches (i.e., capture from unstructured EHR data and processing by artificial intelligence [AI] technology, including natural language processing and AI-based inference) were evaluated against a manual chart abstraction reference standard for data collected from a tertiary care setting. The primary endpoint was recall; differences were compared using chi square.ResultsCompared with manual chart abstraction, recall for migraine and headache were 66.6% and 29.6%, respectively, for Traditional RWE, and 96.8% and 92.9% for Advanced RWE; differences were statistically significant (absolute differences, 30.2% and 63.3%; P < 0.001). Recall of 6 migraine-associated symptoms favored Advanced RWE over Traditional RWE to a greater extent (absolute differences, 71.5–88.8%; P < 0.001). The difference between traditional and advanced techniques for recall of migraine medications was less pronounced, approximately 80% for Traditional RWE and ≥ 98% for Advanced RWE (P < 0.001).ConclusionUnstructured EHR data, processed using AI technologies, provides a more credible approach to enable RWE in migraine than using structured EHR and claims data alone. An algorithm was developed that could be used to further study and validate the use of RWE to support diagnosis and management of patients with migraine.

1287-P: Identifying When Incident Diabetes Was Diagnosed in Children and Young Adults, Using Natural Language Processing of Clinical Notes

Identifying when an incident diabetes (DM) diagnosis was made is complicated using retrospective, structured electronic health record (EHR) data alone. Unstructured clinical notes have been underused but contain valuable information that could complement traditional methods. However, manually reviewing clinical notes is time-consuming. We developed and validated a simple rule-based Natural Language Processing (NLP) method to extract incident DM timing from clinical notes. In a single center we used structured EHR data to identify a cohort (age &amp;lt;45 as of 12/31/19) with likely type 1 (T1D) or type 2 (T2D) DM based on 2016-2019 records: (≥1 T1D ICD-10 code and insulin and no other DM medication) or ([≥2 T2D and no T1D codes] or [≥1 T2D code and a DM medication besides insulin or metformin]). This cohort had 2,654 patients (548,316 clinical notes, 2003-present). We randomly selected 58,450 clinical notes (1,465 patients) as a training set to look for relevant text patterns. We handcrafted the rules into our NLP tool. We required 3 distinct concepts at the sentence level to determine an incident DM diagnosis: DM (not, e.g., epilepsy), an onset attribute (e.g., “diagnosed in”), and a temporal component (e.g., 8/2008). We pre-defined all related keywords and date formats for these concepts in our training notes. We then tested the NLP algorithm against manual review in an independent set of 100 randomly selected patients from the cohort. Analysis was at the patient level (true+: ≥1 true+ note per patient). NLP in the training set found 1,268 patients with at least 1 of the 3 concepts and 826 patients with all 3. In the test set, we excluded 4 patients without substantive notes. NLP correctly detected incident DM timing in 73 of 96 patients. The NLP had recall 88%, specificity 77%, precision (PPV) 96%, and NPV 50%. NLP was helpful in finding incident DM timing and may complement structured EHR queries for identifying incident DM. Refinement of our NLP algorithm is ongoing. Disclosure A.Wong: None. V.W.Zhong: None. M.Rosenman: None. Funding Centers for Disease Control and Prevention (1U18DP006693-01-00)

Leveraging Natural Language Processing to Improve Electronic Health Record Suicide Risk Prediction for Veterans Health Administration Users

Background: Suicide risk prediction models frequently rely on structured electronic health record (EHR) data, including patient demographics and health care usage variables. Unstructured EHR data, such as clinical notes, may improve predictive accuracy by allowing access to detailed information that does not exist in structured data fields. To assess comparative benefits of including unstructured data, we developed a large case-control dataset matched on a state-of-the-art structured EHR suicide risk algorithm, utilized natural language processing (NLP) to derive a clinical note predictive model, and evaluated to what extent this model provided predictive accuracy over and above existing predictive thresholds.Methods: We developed a matched case-control sample of Veterans Health Administration (VHA) patients in 2017 and 2018. Each case (all patients that died by suicide in that interval, n = 4,584) was matched with 5 controls (patients who remained alive during treatment year) who shared the same suicide risk percentile. All sample EHR notes were selected and abstracted using NLP methods. We applied machine-learning classification algorithms to NLP output to develop predictive models. We calculated area under the curve (AUC) and suicide risk concentration to evaluate predictive accuracy overall and for high-risk patients.Results: The best performing NLP-derived models provided 19% overall additional predictive accuracy (AUC = 0.69; 95% CI, 0.67, 0.72) and 6-fold additional risk concentration for patients at the highest risk tier (top 0.1%), relative to the structured EHR model.Conclusions: The NLP-supplemented predictive models provided considerable benefit when compared to conventional structured EHR models. Results support future structured and unstructured EHR risk model integrations.

Development and validation of algorithms to build an electronic health record based cohort of patients with systemic sclerosis.

To evaluate methods of identifying patients with systemic sclerosis (SSc) using International Classification of Diseases, Tenth Revision (ICD-10) codes (M34*), electronic health record (EHR) databases and organ involvement keywords, that result in a validated cohort comprised of true cases with high disease burden. We retrospectively studied patients in a healthcare system likely to have SSc. Using structured EHR data from January 2016 to June 2021, we identified 955 adult patients with M34* documented 2 or more times during the study period. A random subset of 100 patients was selected to validate the ICD-10 code for its positive predictive value (PPV). The dataset was then divided into a training and validation sets for unstructured text processing (UTP) search algorithms, two of which were created using keywords for Raynaud's syndrome, and esophageal involvement/symptoms. Among 955 patients, the average age was 60. Most patients (84%) were female; 75% of patients were White, and 5.2% were Black. There were approximately 175 patients per year with the code newly documented, overall 24% had an ICD-10 code for esophageal disease, and 13.4% for pulmonary hypertension. The baseline PPV was 78%, which improved to 84% with UTP, identifying 788 patients likely to have SSc. After the ICD-10 code was placed, 63% of patients had a rheumatology office visit. Patients identified by the UTP search algorithm were more likely to have increased healthcare utilization (ICD-10 codes 4 or more times 84.1% vs 61.7%, p < .001), organ involvement (pulmonary hypertension 12.7% vs 6% p = .011) and medication use (mycophenolate use 28.7% vs 11.4%, p < .001) than those identified by the ICD codes alone. EHRs can be used to identify patients with SSc. Using unstructured text processing keyword searches for SSc clinical manifestations improved the PPV of ICD-10 codes alone and identified a group of patients most likely to have SSc and increased healthcare needs.

Bridging the Gap between Medical Tabular Data and NLP Predictive Models: A Fuzzy-Logic-Based Textualization Approach

The increasing use of electronic health records (EHRs) generates a vast amount of data, which can be leveraged for predictive modeling and improving patient outcomes. However, EHR data are typically mixtures of structured and unstructured data, which presents two major challenges. While several studies have focused on using machine learning models to predict patient outcomes, these models often require data to be in a structured format, which may lead to the loss of important information. On the other hand, unstructured data, such as narrative reports, can be noisy and challenging for natural language processing applications and interoperability. Therefore, there is a need to bridge the gap between structured EHR data and NLP-based predictive models. In this paper, we propose a fuzzy-logic-based pipeline that generates medical narratives from structured EHR data and evaluates its performance in predicting patient outcomes. The pipeline includes a feature selection operation and a reasoning and inference function that generates medical narratives. We then extensively evaluate the generated narratives using transformer-based NLP models for a patient-outcome-prediction task. We furthermore assess the interpretability of the generated text using Shapley values. Our approach has demonstrated comparable performance to the benchmark baseline models with an F1-score of 93.7%, while exhibiting slightly improved results in terms of recall. The model demonstrated proficiency in the preservation of information and interpretability inherited from nuanced and structured narratives. To the best of our knowledge, this is the first study to demonstrate the ability to transform tabular data into text to apply NLP for a prediction task.

Natural language processing to identify reasons for sex disparity in statin prescriptions

BackgroundStatins are the cornerstone of treatment of patients with atherosclerotic cardiovascular disease (ASCVD). Despite this, multiple studies have shown that women with ASCVD are less likely to be prescribed statins than men. The objective of this study was to use Natural Language Processing (NLP) to elucidate factors contributing to this disparity. MethodsOur cohort included adult patients with two or more encounters between 2014 and 2021 with an ASCVD diagnosis within a multisite electronic health record (EHR) in Northern California. After reviewing structured EHR prescription data, we used a benchmark deep learning NLP approach, Clinical Bidirectional Encoder Representations from Transformers (BERT), to identify and interpret discussions of statin prescriptions documented in clinical notes. Clinical BERT was evaluated against expert clinician review in 20% test sets. ResultsThere were 88,913 patients with ASCVD (mean age 67.8±13.1 years) and 35,901 (40.4%) were women. Women with ASCVD were less likely to be prescribed statins compared with men (56.6% vs 67.6%, p <0.001), and, when prescribed, less likely to be prescribed guideline-directed high-intensity dosing (41.4% vs 49.8%, p <0.001). These disparities were more pronounced among younger patients, patients with private insurance, and those for whom English is their preferred language. Among those not prescribed statins, women were less likely than men to have statins mentioned in their clinical notes (16.9% vs 19.1%, p <0.001). Women were less likely than men to have statin use reported in clinical notes despite absence of recorded prescription (32.8% vs 42.6%, p <0.001). Women were slightly more likely than men to have statin intolerance documented in structured data or clinical notes (6.0% vs 5.3%, p=0.003). ConclusionsWomen with ASCVD were less likely to be prescribed guideline-directed statins compared with men. NLP identified additional sex-based statin disparities and reasons for statin non-prescription in clinical notes of patients with ASCVD.

Prediction of Brain Metastases Development in Patients With Lung Cancer by Explainable Artificial Intelligence From Electronic Health Records.

Early detection of brain metastases (BMs) is critical for prompt treatment and optimal control of the disease. In this study, we seek to predict the risk of developing BM among patients diagnosed with lung cancer on the basis of electronic health record (EHR) data and to understand what factors are important for the model to predict BM development through explainable artificial intelligence approaches accurately. We trained a recurrent neural network model, REverse Time AttentIoN (RETAIN), to predict the risk of developing BM using structured EHR data. To interpret the model's decision process, we analyzed the attention weights in the RETAIN model and the SHAP values from a feature attribution method, Kernel SHAP, to identify the factors contributing to BM prediction. We developed a high-quality cohort with 4,466 patients with BM from the Cerner Health Fact database, which contains over 70 million patients from more than 600 hospitals. RETAIN uses this data set to achieve the best area under the receiver operating characteristic curve at 0.825, a significant improvement over the baseline model. We also extended a feature attribution method, Kernel SHAP, to structured EHR data for model interpretation. Both RETAIN and Kernel SHAP can identify important features related to BM prediction. To the best of our knowledge, this is the first study to predict BM using structured EHR data. We achieved decent prediction performance for BM prediction and identified factors highly relevant to BM development. The sensitivity analysis demonstrated that both RETAIN and Kernel SHAP could discriminate unrelated features and put more weight on the features important to BM. Our study explored the potential of applying explainable artificial intelligence for future clinical applications.

External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems.

Implementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population. PICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs. The general wards of two large hospitals, one an academic medical center and the other a community hospital. The model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital. None. The hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE's performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861-0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275-0.320) at the first hospital; AUROC 0.875 (0.851-0.902), AUPRC 0.339 (0.281-0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions. Important differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.

Abstract P3-03-16: A methodology for using real-world data from electronic health records to assess chemotherapy administration in women with breast cancer

Abstract Introduction Chemotherapy administration in real-world cancer care can differ extensively from clinical trials. It is important to understand real-world practice to identify dose reductions, delays, regimen changes and early discontinuations that impact cancer outcomes. Such variables require knowledge of intended regimens, which may not be well-documented in structured data in electronic health records (EHRs). We examined EHR data from the Kaiser Permanente Northern California (KPNC) site of the Optimal Breast Cancer Chemotherapy Dosing (OBCD) study to develop a process to identify each patient’s intended regimen. Methods In this study of women diagnosed and treated with primary stage I-IIIA breast cancer at KPNC from 2006-2019, and ages 18+y at diagnosis, we analyzed treatment patterns using structured EHR data on the drugs, dosages, and dates at which they were administered (from which intervals and total length can be derived). Chemotherapy agents were identified using the NCI’s CANMED database augmented with other sources. We used these data to categorize patients into the 22 drug combinations described in the National Cancer Care Network (NCCN) guidelines for breast cancer treatment. Within these 22 drug combinations, women were then subcategorized into 45 distinct chemotherapy administration schedules, defined as NCCN guideline regimens (NGRs). For this step, algorithms were developed that categorized patients into NGRs if they received the exact regimen described in the guidelines. For the second step, we conducted a manual review of the EHR data for patients who were unable to be categorized. This enabled us to gradually loosen the criteria (in terms of cycle intervals or number of cycles) so patients whose chemotherapy administration aligned closely with NGRs were categorized into each of the 45 NGRs. Clear patterns emerged of regimens that were administered to multiple patients, despite being outside of the NCCN guidelines, which we have defined as non-standard NGRs. For example, in the drug combination TC (cyclophosphamide and docetaxel) the NGR was TC every 21 days for 4 cycles. We found approximately 1 in 10 patients received 6 cycles, which we defined as a non-NGR. For the remaining uncategorized patients, medical chart abstraction was undertaken as a third step, at which point patients were categorized into either existing regimens or new non-NGRs if their intended regimen had not been previously described in the guidelines. Results Among 31,418 women with breast cancer, 12,427 (39.6%) received chemotherapy. We determined the intended chemotherapy regimens for 6,559 (52.8%) receiving the 45 NGRs using EHR data. We further expanded the algorithms through a manual review of the EHR data, which enabled us to categorize 2,977 (24.0%) additional women into their intended regimens. Abstracted medical notes were reviewed for the remaining patients for whom we had not been able to identify the intended regimen. Across both the manual review and abstraction processes, we were able to identify additional non-standard NGR regimens. In total, 9,536 (76.7%) of women were categorized into their intended regimen through the algorithm/manual review process, while 2891 (23.3%) of women underwent medical chart abstraction to identify the intended regimen. Conclusion Here, we describe the challenges and approaches to operationalize complex, real-world data to identify intended chemotherapy regimens at a granularity and scale not seen previously. We are adapting this method at a second OBCD study site, Kaiser Permanente Washington, where all women have undergone medical chart abstraction. We hope this methodology leads to increased feasibility and efficiency of use of large-scale clinical data, in turn improving cancer care delivery, patient outcome evaluation, and other real-world questions. Citation Format: Jenna Bhimani, Kelli O’Connell, Rachael P. Burganowski, Isaac J. Ergas, Marilyn J. Foley, Grace B. Gallagher, Jennifer J. Griggs, Narre Heon, Tatjana Kolevska, Yuriy Kotsurovskyy, Candyce H. Kroenke, Kanichi G. Nakata, Sonia Persaud, Donna R. Rivera, Janise M. Roh, Sara Tabatabai, Emily Valice, Erin J. Bowles, Elisa V. Bandera, Lawrence H. Kushi, Elizabeth D. Kantor. A methodology for using real-world data from electronic health records to assess chemotherapy administration in women with breast cancer [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P3-03-16.

Electronic health record-based prediction models for in-hospital adverse drug event diagnosis or prognosis: a systematic review.

We conducted a systematic review to characterize and critically appraise developed prediction models based on structured electronic health record (EHR) data for adverse drug event (ADE) diagnosis and prognosis in adult hospitalized patients. We searched the Embase and Medline databases (from January 1, 1999, to July 4, 2022) for articles utilizing structured EHR data to develop ADE prediction models for adult inpatients. For our systematic evidence synthesis and critical appraisal, we applied the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). Twenty-five articles were included. Studies often did not report crucial information such as patient characteristics or the method for handling missing data. In addition, studies frequently applied inappropriate methods, such as univariable screening for predictor selection. Furthermore, the majority of the studies utilized ADE labels that only described an adverse symptom while not assessing causality or utilizing a causal model. None of the models were externally validated. Several challenges should be addressed before the models can be widely implemented, including the adherence to reporting standards and the adoption of best practice methods for model development and validation. In addition, we propose a reorientation of the ADE prediction modeling domain to include causality as a fundamental challenge that needs to be addressed in future studies, either through acquiring ADE labels via formal causality assessments or the usage of adverse event labels in combination with causal prediction modeling.

428. Deep-Learning Based Predictive Model for Patients with Positive MRSA Cultures Using Time-Series Electronic Health Records

Abstract Background Methicillin-resistant Staphylococcus aureus (MRSA) is one of the common pathogens leading to significant morbidity and mortality in the hospital. This pathogen requires specific empirical antibiotics. Hence, identifying the personalized risks of this pathogen likely optimizes the usage of those antibiotics. Deep-learning based (DL) models are shown to be useful for modeling time-series electronic health record (EHR) data, and thus offering potential for predicting individualized risk for MRSA infection. Methods Longitudinal data were retrospectively retrieved from EHR in Memorial Hermann System, Houston, Tx. Patient encounters, demographics, diagnostic &amp; procedure codes, antibiotics use, and microbiology data were extracted between 1/2018 and 4/2021. Inpatient and outpatient data were included. We randomly identified roughly equal numbers of patients with positive culture (Cx) for MRSA, MSSA, other pathogens, and negative Cx. We set a 2-week window for the prediction, and any first culture within the window was used as an index Cx. Some patients had multiple predictions over time and were included into both MRSA vs. non-MRSA groups. Our team developed a DL model platform (Pyorch_EHR) for clinical outcomes predictions using structured EHR data. In this project, the outcome is MRSA positivity taken within 2 weeks from the index Cx. Datasets are split into 50, 30, and 20% for training, validation, and test, respectively. Other machine learning models; logistic regression (LR) and light GBM (LGBM) were used for comparison. Pytorch ver. 1.7.1 and Sklearn ver. 0.24.2 are used. Results A total of 8164 patients and 22,563 patients were identified as MRSA and non-MRSA groups, respectively. Table 1 summarizes the key features of the patient’s characteristics. After model training using train and validation datasets, our model achieved an AUC of 91.8 in test dataset, whereas AUC of 86.0 and 88.2 in LR and LGBM, respectively. Figure 1 shows the cumulative incidence of 2 week MRSA positivity. Our model precisely stratified the risks. Figure 1:Cumulative Incidence of Positive MRSA Cultures Based on Risk Stratificationa Conclusion Our DL based model exhibited excellent performance in the prediction of 2 week MRSA positivity. Our model precisely categorized patients into low, medium, and high-risk. This work should be validated with other data sources and high-risk subgroups. Disclosures All Authors: No reported disclosures.

Abstract 12092: Natural Language Processing to Identify Reasons for Gender Disparities in Statin Use

Introduction: Statins are life-saving medications for patients with atherosclerotic cardiovascular disease (ASCVD), but women with ASCVD are persistently less likely to be prescribed statins than men. This study aims to use Natural Language Processing (NLP) to further elucidate patient and provider factors contributing to this disparity. Methods: The study cohort included patients with &gt;2 ASCVD encounters between 2014 and 2021 within a multisite electronic health record (EHR) in Northern California. Data from a random sample of our cohort (N = 942) was manually annotated to develop a benchmark deep learning natural language processing (NLP) approach, Clinical Bidirectional Encoder Representations from Transformers (BERT); 80% of these were used for model training and 20% for testing. After reviewing structured EHR data (e.g. prescriptions, allergies), BERT was used to identify and interpret discussions of statins in clinical notes. Results: Of 88,913 patients with ASCVD (mean age 67.8 ± 13.1 years), 35,901 (40.4%) were women. Women were less likely to be prescribed statins (56.6% vs. 67.6%, p &lt; 0.001). Only 18.6% of nonallergic patients without a statin prescription had a mention of statins in unstructured EHR text. Statin use through unstructured text was less likely to be identified among women than men (32.8% vs. 42.6%, p &lt; 0.001). Reasons for statin nonuse did not significantly differ by gender (Figure). Conclusions: Women with ASCVD were less likely to be use statins, and this disparity became more pronounced with the inclusion of unstructured data. An NLP approach revealed actionable reasons for statin nonuse. Future studies should leverage these approaches to monitor and track statin adherence by combining structured and unstructured data in real-world populations.

Abstract 11934: Natural Language Processing Models Can Be Trained to Accurately Recognize the Presence of Disease Within Clinical Notes

Introduction: Automated methods to “label” the presence of disease such as atrial fibrillation (AF) within electronic health record (EHR) data are important for secondary uses such as population health management. Labels generated from billing code patterns (e.g. “at least 2 relevant ICD codes used within 1 year”) can be accurate within one health system, but may fail to generalize across systems due to variations in coding practices. Hypothesis: We hypothesized that a natural language processing (NLP) model could be trained to detect the presence of atrial fibrillation using unstructured clinical notes by learning at scale from labels generated from a validated structured EHR and billing code definition. If successful, this could facilitate scaling disease label methods across large amounts of clinical data without suffering from inaccuracies due to variations in coding practices. Methods: We collected clinical notes from a regional health system into a training set of roughly 29 million code-labeled episodes and a separate hold-out set of roughly 1.8 million code-labeled episodes. We trained our NLP model on the training set and computed model performance compared to the code-based labels on the hold-out set. We also performed targeted blinded chart reviews. This work was conducted under IRB approval. Results: The model achieved an AUPRC of 0.91 with 87% recall and 89% precision and learned to distinguish challenging confounders despite not being explicitly trained for these tasks (Fig). Blinded review of select episodes showed that the NLP model was correct 90% of the time when it labeled the episode as positive for atrial fibrillation despite the code-based label (incorrectly) labeling it negative. Conclusions: NLP models can learn to automatically label the presence or absence of atrial fibrillation within clinical notes and may lead to greater accuracy and generalizability relative to code-based labeling methods.

Derivation and Validation of the Management and Supervision Tool (MaST) Risk of Crisis (RoC) Algorithm Using Electronic Health Record (EHR) Data

AimsThe Management and Supervision Tool (MaST) helps NHS mental health care professionals identify patients who are most likely to need psychiatric hospital admission or home treatment, due to severe mental illness, through a Risk of Crisis (RoC) algorithm driven by electronic health record (EHR) data analytics. We describe the derivation and validation of the MaST RoC algorithm, and its implementation to support preventative mental healthcare in the NHS.MethodsThe RoC algorithm was developed and evaluated with EHR data from six UK NHS trusts using Ordered Predictor List propensity scores informed by a priori weightings from pre-existing literature, as well as real-world evidence evaluating the associations of clinical risk factors with mental health crisis using NHS EHR data. Mental health crisis was defined as admission to a psychiatric hospital or acceptance to a community crisis service within a 28-day period. Predictor variables included age, gender, accommodation status, employment status, Mental Health Act (MHA) status (under section or Community Treatment Order), and previous mental health service contacts (including hospital admissions and crisis services). Data were analysed using Ordered Predictor List propensity scores. The algorithm was derived using structured EHR data from 2,620 patients in a single NHS trust and externally validated using data from 107,879 patients in five other NHS trusts. Qualitative and quantitative data on feasibility, acceptability and system efficiency impacts of MaST implementation were obtained through staff surveys and local audits.ResultsThe factors associated with greatest propensity for mental health crisis included recent previous crisis, multiple previous crises, higher number of mental health service contacts in recent weeks, MHA section, accommodation status and employment status. The RoC algorithm identified 64% and 80% crises in its top quintile. Sentiment analysis of staff surveys suggested that the use of MaST improved productivity by reducing time taken to access patient information to support caseload management that was previously difficult to obtain through manual review of EHRs. The systems efficiency audit revealed a reduction in duration of crisis and inpatient admissions following MaST implementation.ConclusionThe MaST RoC algorithm supports the identification of people more likely to use crisis services in NHS mental health trusts, is feasible to implement, and improves systems efficiency. EHR-derived algorithms can support real-world clinical practice to improve outcomes in people receiving NHS mental healthcare.