ObjectivesElectronic health record (EHR) order preference lists and order sets potentially improve efficiency but have limited utility in complex primary care settings. We assessed adoption, impact on ordering efficiency, and clinician perceptions of a comprehensive set of nested order panels (xOrders) for adult primary care.MethodsIn Phase 1 (gradual implementation), 404 xOrders were released (November 29, 2020-September 25, 2021). Beginning of Phase 2 (rapid implementation), 630 xOrders were released with an additional 253 xOrders added (September 26, 2021-June 24, 2023). Three outcomes captured adoption: xOrders used per week; number of clinician users per week; and percent of xOrders of all orders. Impact of xOrders on times in orders per encounter per clinician was evaluated with mixed effects interrupted time series. t-Tests evaluated differences between low, moderate, and high utilizers. A survey captured clinicians’ perceptions in November 2022.ResultsxOrders were used 536 (SD, 245) times/week and by 57(15) clinicians/week in Phase 2. xOrders as a percent of all orders ranged from 0% to 31% across clinicians. Time spent in orders per encounter decreased by 14 ± 5 s (P =.01) from Phase 1 to 2 for high utilizers, decreased by 7(3) s (P=.05) for moderate utilizers, and increased by 1(3) s for low utilizers (P=.81); low and high utilizers were significantly different (P=.02). Most (77%) survey respondents agreed that xOrders improved ordering efficiency.Discussion and ConclusionsDespite yielding time savings and positive clinician feedback, the xOrder intervention showed limited adoption and impact, suggesting the need for expanded content and increased adoption to realize larger efficiency gains.

ObjectivesThis study aims to develop a detailed understanding of provider and Information Technology (IT) operations staff experiences and attitudes regarding patients’ ability to edit their data. This includes understanding barriers to developing a process to write back data into the electronic health record (EHR) as well as a concrete set of recommendations on incorporating patient-generated data into the EHR.Materials and MethodsRealRisks, our team’s Fast Healthcare Interoperability Resources-compliant web-based patient decision aid, was utilized as an exemplar platform in which patients can access EHR data and review, correct, and contribute patient-derived data when specific elements are missing. An interview guide was developed and semi-structured interviews of 9 participants (physicians n = 4, IT operations staff n = 5) at Columbia University Irving Medical Center were carried out to understand the feasibility of writing back patient-entered edits into the EHR using the RealRisks decision aid.ResultsProviders and IT operations staff reported varied knowledge of how patients interact with their data but collectively stated a need for increasing EHR accuracy that prioritizes provider-patient communication. Participants supported a write-back process and had specific suggestions for implementation mechanisms (such as the option to upload test results when submitting changes).DiscussionProviders and IT operations staff maintained that existing data management routes used for external data incorporation should be utilized, and that providers should screen edit requests to ensure EHR quality and accuracy.ConclusionWhile participants felt a write-back of patient-derived data would be helpful, future studies should directly assess nursing staff and advanced practice providers as well as patient perspectives to ensure equity and efficacy.

ObjectivesTo develop a multimodal pediatric critical care datamart supporting predictive modeling and decision support tool development, integrating high-resolution physiologic and clinical data and future clinical deployment.Materials and MethodsWe developed a continuously expanding datamart integrating electronic health record data, high-resolution telemetry, and extracorporeal membrane oxygenation (ECMO)-domain datasets. The platform links static and longitudinal time-series variables with expert-curated neurologic outcomes for both ECMO and non-ECMO patients, enabling trajectory-based analyses.ResultsThe datamart currently includes 25 762 pediatric patients, of whom 395 received ECMO support. The datamart captures granular longitudinal physiologic, laboratory, medication, and telemetry data suitable for dynamic predictive modeling.DiscussionExisting ECMO prognostication tools rely on static variables and lack appropriate control cohorts. This datamart enables trajectory-based multimodal modeling, that reflects evolving physiology and neurologic outcomes.ConclusionThis platform provides a scalable foundation for predictive modeling across pediatric critical care, beyond ECMO, to support precision decision-making and outcomes research.

To explore the relationship between ambulatory physician electronic health record (EHR) use characteristics and proxies for physician efficiency. A longitudinal cohort study was conducted to examine physician-month EHR use metadata in 413 US organizations between May 2019 and April 2022. A multi-model machine learning classifier was developed to predict physician efficiency. The main outcomes of the study were physician efficiency, measured as the proportion of same-day chart completion by specialty, and productivity, measured as daily patient visit volume, both segmented into quintiles. The study included 218 610 unique physicians with 5 193 385 physician-month observations from 413 organizations with an average chart completion efficiency of 72.9% and 10.8 visits per scheduled day. The primary ML analysis achieved an accuracy of 0.74 in classifying physician-months with high chart completion efficiency and highlighted associations with key features, such as inbox message turnaround time <1.5 days and after-hours documentation <25 min/scheduled day. A secondary analysis achieved an accuracy of 0.84 in classifying physician-months with high visit volumes, indicating that factors such as EHR time outside scheduled hours <4.1 min/visit and clinical review time <3.2 min/visit were associated with higher visit volumes. Implementing specific EHR use measures with distinct thresholds, such as inbox management and after-hours documentation, could help target interventions to enhance productivity, providing actionable insights to create balanced and efficient work environments that improve patient care and reduce EHR time.

To explore how Spanish-speaking caregivers navigate translation barriers in patient portals and to assess their perspectives on improving language accessibility. This qualitative study was conducted at a pediatric academic health system. Semi-structured interviews were conducted with Spanish-speaking caregivers of children with chronic conditions, and inductive thematic analysis was used to generate themes. Twenty caregivers participated. Three key themes emerged: (1) Caregivers rely on online machine translation tools, which can be inaccurate and time-consuming; (2) Caregivers frequently depend on children and family members for translation, raising concerns about comprehension and appropriateness; (3) Caregivers expressed strong interest in timely and accurate translation features within patient portals to enhance accessibility. Spanish-speaking caregivers develop workarounds to access medical information, but these strategies pose risks to patient safety and exacerbate digital health inequities. While AI-powered machine translation offers a potential solution, concerns about accuracy, regulatory compliance, and equitable implementation must be addressed. Spanish-speaking caregivers face significant challenges in accessing health information through patient portals. Health systems should prioritize integrated translation solutions, leveraging AI-driven tools while ensuring accuracy and equitable implementation to improve language accessibility.

This study evaluates the usefulness of explicit syntactic knowledge, integrated via a neural mechanism, in improving the accuracy of named entity recognition in the domain of biomedical text processing. Syntactic structure of a text can be helpful to determine whether a certain part of the text is an entity or not. Parsing is an essential technique in natural language processing (NLP) that can be utilized to determine the syntactic structure of sentences in human languages. We propose to infuse syntactic knowledge through the attention mechanism using dependency parsing and sequence labelling parsing, as well as the multi-task learning paradigm. Experiments were conducted on five datasets: MTSamples, VAERS, NCBI-disease, BC2GM, and JNLPBA. We demonstrate improvements in the F1 score over the current state of the art on 3 out of 5 datasets (MTSamples, VAERS, and NCBI). We reduce the number of mismatches with gold labels in particular in the n-dash and parentheses tokens and in compound and adjective modifier dependencies. Syntactic features improve NER accuracy in attention-based neural systems, and parsing as sequence labelling brings additional benefits.

The expansion of artificial intelligence (AI)-enabled clinical decision support (CDS) requires nurses to interpret complex model outputs. However, their cognitive readiness remains underexplored, particularly in terms of their understanding of statistics. To assess nurses' understanding of key statistical concepts underlying AI predictions and their relationship to health numeracy. An organizational approach study involving 180 nurses from 6 medical-surgical units at a tertiary hospital, preparing to implement an AI fall-prediction model. Statistical knowledge was evaluated using a heuristic vignette based on fuzzy-trace theory, assessing both verbatim (literal) and gist (meaning-based) understanding of sensitivity, specificity, and CIs. Health numeracy was measured using the Lipkus Objective Numeracy Scale, Numeracy Understanding in Medicine Instrument: short form, and Subjective Numeracy Scale. Analyses included ANOVA and Kruskal-Wallis and Wilcoxon rank-sum tests, with thematic analyses applied to the qualitative concerns of nurses. Overall statistical knowledge was moderate (mean = 85.56, 95% CI, 82.64-88.46). Gist knowledge lagged verbatim knowledge, especially about CIs. Nurses with advanced degrees had higher verbatim scores (P = .0108), while bachelor-level nurses performed better on discrete-choice tasks related to gist (P = .0124). Numeracy was not significantly associated with the understanding of statistics. Nurses overrode predictions due to cognitive mismatch, requesting greater model transparency, input rationale, and risk-threshold explanations. Despite displaying adequate numeracy, nurses' conceptual grasp of statistical concepts may hinder the safe application of AI CDS system outputs. These findings underscore the need for targeted education and a cognitive-fit-driven interface design to support the trustworthy use of AI in nursing practice.

Traditional clinical trial enrollment relies on manual screening and coordinator-led recruitment, creating scalability barriers in high-volume perioperative environments. This study evaluated whether a fully automated, electronic health record (EHR)-integrated clinical decision support (CDS) system could identify eligible patients and engage clinicians in real time without manual screening or dedicated research staff. In this prospective implementation study, predefined respiratory-risk criteria were computed within the UCLA Perioperative Data Warehouse and transmitted to the EHR via Healthcare Level Seven interfaces. Patients meeting inclusion criteria automatically triggered Best Practice Advisories (BPAs) recommending an intervention. Outcomes included system accuracy in eligibility identification, provider adherence to BPA recommendations, and technical performance metrics. The automated system processed 10 592 eligible patients and achieved 51.2% provider adherence (5424 patients) to CDS prompts without coordinator involvement. BPA allocation accuracy was 69.7% among patients recovering in the post-anesthesia care unit and 59.4% when including unanticipated ICU transfers. Adherence varied significantly by care team composition, with full teams (attending + CRNA + resident) achieving 57.4% adherence compared with 42.2% for solo attendings. Workflow factors were stronger predictors of adherence than patient clinical characteristics, indicating minimal selection bias. Fully automated, EHR-integrated CDS can enable large-scale, workflow-embedded enrollment into implementation-focused studies. While not a substitute for research designs requiring consent or randomization, this framework demonstrates a scalable approach for automated prescreening and CDS-driven prompting that reduces reliance on coordinator-dependent processes and supports real-world implementation science.

To assess the interpretability and acceptance of Shapley values for making artificial intelligence/machine learning (AI/ML) tools more transparent, interpretable, and useful to clinicians. Structured assessments were conducted with 30 clinicians (15 providers; 15 nurses; 8 assessments per clinician) to evaluate their ability to understand interventional Shapley Additive exPlanations (SHAP) values, a type of Shapley value that provides individualized variable importance scores and ascertain their perspective on SHAP value utility for the use of an AI/ML sepsis diagnostic. Participants were shown the diagnostic interface for real clinical scenarios with de-identified patient data with and without SHAP values. The primary outcomes were clinician ability to correctly interpret SHAP values and clinician self-reported improvement in their understanding of how the AI/ML algorithm produced its result. Participants correctly interpreted SHAP values in 235 of 240 assessments (98%; CI, 95%-99%) and reported SHAP values improved their understanding of how the algorithm produced its result in every case (240/240; 100%; CI, 99%-100%). Participants were unanimous (30/30) in preferring the interface with SHAP values over the interface without. Clinician participants strongly preferred the device interface with SHAP values, were unanimous in reporting SHAP values improved their understanding of the AI/ML diagnostic, and scored nearly perfectly when asked to interpret SHAP values. These results suggest health care providers value transparency into AI/ML algorithms designed for clinical use, and that Shapley values are a useful approach to providing that transparency, which in turn may improve tool adoption and clinical utility.

The COVID-19 pandemic caused a major health crisis worldwide significantly impacting mental well-being. In this study, our objective is to assess the resilience of pre-pandemic depression level prediction models when applied to COVID-19 era data. We leverage advanced Machine Learning (ML) and Explainable Artificial Intelligence (XAI) techniques to identify the key factors impacting the shifts in depression levels during the pandemic. We aim to align the later identification with interventions and preparedness for future pandemics. We use, in this study, a data-driven methodology using National Health Interview Survey (NHIS) household survey data, explicitly covering the years 2019-2022. The NHIS data is used to build both the pre-pandemic (2019) and COVID-19 (2020-2022) models discussed in our comparative evaluation. Various ML techniques are supported (1) upstream, using feature selection methods to reduce both irrelevance and the high dimensionality of social-nature data, and (2) downstream, by an XAI-based approach to gain insight into the pandemic-associated phenomena that mostly impacted the mental health of individuals. In our empirical experiments, we use over 100000 entries across the 4 yearly datasets, where we apply an 80%-20% training/testing split for models building and evaluation. The outcomes of our empirical study show that classifiers trained solely on pre-COVID-19 data performed poorly when applied to COVID-19 era data. Conversely, models retrained on pandemic-specific data demonstrated high performance. In particular, the Random Forest (RF) classifier achieved the best performance, recording an average accuracy of 98.10% across the COVID-19 era datasets. With respect to the depression key factors' identification, XAI techniques provided actionable insights, revealing that features such as Delayed Medical Care, Family Poverty, Participation in Social Activities, and Marital Status were the most influential factors contributing to depression challenges during the pandemic. The significant decline in the performance of pre-pandemic models on COVID-19 data reveals the profound impact of the pandemic on mental health, highlighting the need for new predictive models tailored to crisis circumstances. The built RF model, uses appropriate pandemic data, performed accurately during the COVID-19 era with an accuracy of 98.1%. XAI techniques confirmed that factors such as delayed medical care, family poverty, job loss, and reduced social involvement were critical drivers that impacted the decline in mental health during the pandemic.