Electronic Health Record Data Quality Research Articles

Electronic Health Record Data Quality and Performance Assessments: Scoping Review.

Electronic health records (EHRs) have an enormous potential to advance medical research and practice through easily accessible and interpretable EHR-derived databases. Attainability of this potential is limited by issues with data quality (DQ) and performance assessment. This review aims to streamline the current best practices on EHR DQ and performance assessments as a replicable standard for researchers in the field. PubMed was systematically searched for original research articles assessing EHR DQ and performance from inception until May 7, 2023. Our search yielded 26 original research articles. Most articles had 1 or more significant limitations, including incomplete or inconsistent reporting (n=6, 30%), poor replicability (n=5, 25%), and limited generalizability of results (n=5, 25%). Completeness (n=21, 81%), conformance (n=18, 69%), and plausibility (n=16, 62%) were the most cited indicators of DQ, while correctness or accuracy (n=14, 54%) was most cited for data performance, with context-specific supplementation by recency (n=7, 27%), fairness (n=6, 23%), stability (n=4, 15%), and shareability (n=2, 8%) assessments. Artificial intelligence-based techniques, including natural language data extraction, data imputation, and fairness algorithms, were demonstrated to play a rising role in improving both dataset quality and performance. This review highlights the need for incentivizing DQ and performance assessments and their standardization. The results suggest the usefulness of artificial intelligence-based techniques for enhancing DQ and performance to unlock the full potential of EHRs to improve medical research and practice.

Enhancing Data Quality through Electronic Health Records in Prison Settings: Towards a Learning Health System

Abstract Utilizing electronic health records (EHRs) in prison settings offers a unique opportunity to establish a learning health system (LHS) aimed at enhancing healthcare quality for incarcerated individuals. Despite the challenges inherent in prison healthcare, leveraging EHRs can facilitate data-driven decision-making and foster continuous improvement initiatives. This study describes the results of an intervention aimed at improving EHR data quality in prisons and explores the potential to enhance the development of a robust LHS. We assessed EHR data quality in 2015 (38 prisons) and 2022 (35 prisons) in the Netherlands over six months. Data were extracted, de-identified, and analyzed across demographic, medication use, (history of) diagnosis, and healthcare usage categories. We measured indicator scores and evaluated variation between prisons in data quality. This project was commissioned by the Custodial Institutions Agency. In 2015, age and gender were recorded for 93% of prisoners, increasing to 100% in 2022. Diagnostic coding use rose from an average of 82% in 2015 to 98% in 2022, with reduced variation between prisons. Consultation frequency per inmate ranged from 3 to 13, reflecting challenges in constructing a uniform denominator due to institutional differences. Mental health problems accounted for 17.7% of recorded disease episodes in 2022. In addition, consultations were most often related to oral health (4.4%), drug abuse (3.2%), sleep (1.8%), fear of diseases (1.6%), and respiratory infections (1.5%). Our findings indicate improvements in EHR data quality in Dutch prisons between 2015 to 2022. Integrating EHR data reuse into prison healthcare systems is pivotal for realizing the vision of an LHS in correctional facilities. This study underscores the potential of advancing EHR recording habits and the reuse of EHR data to enhance healthcare quality in prison settings.

Transparency of an Honest Data Broker in Providing Electronic Health Record Data Sufficient for Reuse

Electronic Health Records (EHRs) offer a rich data source for clinical researchers to assess a wide variety of treatments and outcomes. Researchers can use Honest Data Brokers (HDBs) to gain access to EHR data for research. Unfortunately, HDBs’ data analysts can inadvertently overlook nuances in clinical workflows when generating request code for data capture and cause data extraction errors that negatively impact EHR data quality. This paper presents findings from interviews with clinical researchers who have had experience with requesting EHR data from the Regenstrief Institute Data Core (RDC), an HDB with data analysts who provide authorised access to EHR data of nearly 25 million patients in the state of Indiana. Our participants wanted greater transparency when they had questions about the quality of the datasets they received. Participants wanted their data analysts to double check the data in their system, explain how they extracted the data, let them visit the data in their system, and/or view/edit the code used for data extraction. We offer a set of recommendations for HDBs on how to provide greater transparency to clinical researchers about the processes used to generate the EHR data they receive and discuss future directions for research.

Assessing the Effect of Electronic Health Record Data Quality on Identifying Patients With Type 2 Diabetes: Cross-Sectional Study.

Increasing and substantial reliance on electronic health records (EHRs) and data types (ie, diagnosis, medication, and laboratory data) demands assessment of their data quality as a fundamental approach, especially since there is a need to identify appropriate denominator populations with chronic conditions, such as type 2 diabetes (T2D), using commonly available computable phenotype definitions (ie, phenotypes). To bridge this gap, our study aims to assess how issues of EHR data quality and variations and robustness (or lack thereof) in phenotypes may have potential impacts in identifying denominator populations. Approximately 208,000 patients with T2D were included in our study, which used retrospective EHR data from the Johns Hopkins Medical Institution (JHMI) during 2017-2019. Our assessment included 4 published phenotypes and 1 definition from a panel of experts at Hopkins. We conducted descriptive analyses of demographics (ie, age, sex, race, and ethnicity), use of health care (inpatient and emergency room visits), and the average Charlson Comorbidity Index score of each phenotype. We then used different methods to induce or simulate data quality issues of completeness, accuracy, and timeliness separately across each phenotype. For induced data incompleteness, our model randomly dropped diagnosis, medication, and laboratory codes independently at increments of 10%; for induced data inaccuracy, our model randomly replaced a diagnosis or medication code with another code of the same data type and induced 2% incremental change from -100% to +10% in laboratory result values; and lastly, for timeliness, data were modeled for induced incremental shift of date records by 30 days to 365 days. Less than a quarter (n=47,326, 23%) of the population overlapped across all phenotypes using EHRs. The population identified by each phenotype varied across all combinations of data types. Induced incompleteness identified fewer patients with each increment; for example, at 100% diagnostic incompleteness, the Chronic Conditions Data Warehouse phenotype identified zero patients, as its phenotypic characteristics included only diagnosis codes. Induced inaccuracy and timeliness similarly demonstrated variations in performance of each phenotype, therefore resulting in fewer patients being identified with each incremental change. We used EHR data with diagnosis, medication, and laboratory data types from a large tertiary hospital system to understand T2D phenotypic differences and performance. We used induced data quality methods to learn how data quality issues may impact identification of the denominator populations upon which clinical (eg, clinical research and trials, population health evaluations) and financial or operational decisions are made. The novel results from our study may inform future approaches to shaping a common T2D computable phenotype definition that can be applied to clinical informatics, managing chronic conditions, and additional industry-wide efforts in health care.

Development and initial validation of a data quality evaluation tool in obstetrics real-world data through HL7-FHIR interoperable Bayesian networks and expert rules.

The increasing prevalence of electronic health records (EHRs) in healthcare systems globally has underscored the importance of data quality for clinical decision-making and research, particularly in obstetrics. High-quality data is vital for an accurate representation of patient populations and to avoid erroneous healthcare decisions. However, existing studies have highlighted significant challenges in EHR data quality, necessitating innovative tools and methodologies for effective data quality assessment and improvement. This article addresses the critical need for data quality evaluation in obstetrics by developing a novel tool. The tool utilizes Health Level 7 (HL7) Fast Healthcare Interoperable Resources (FHIR) standards in conjunction with Bayesian Networks and expert rules, offering a novel approach to assessing data quality in real-world obstetrics data. A harmonized framework focusing on completeness, plausibility, and conformance underpins our methodology. We employed Bayesian networks for advanced probabilistic modeling, integrated outlier detection methods, and a rule-based system grounded in domain-specific knowledge. The development and validation of the tool were based on obstetrics data from 9 Portuguese hospitals, spanning the years 2019-2020. The developed tool demonstrated strong potential for identifying data quality issues in obstetrics EHRs. Bayesian networks used in the tool showed high performance for various features with area under the receiver operating characteristic curve (AUROC) between 75% and 97%. The tool's infrastructure and interoperable format as a FHIR Application Programming Interface (API) enables a possible deployment of a real-time data quality assessment in obstetrics settings. Our initial assessments show promised, even when compared with physicians' assessment of real records, the tool can reach AUROC of 88%, depending on the threshold defined. Our results also show that obstetrics clinical records are difficult to assess in terms of quality and assessments like ours could benefit from more categorical approaches of ranking between bad and good quality. This study contributes significantly to the field of EHR data quality assessment, with a specific focus on obstetrics. The combination of HL7-FHIR interoperability, machine learning techniques, and expert knowledge presents a robust, adaptable solution to the challenges of healthcare data quality. Future research should explore tailored data quality evaluations for different healthcare contexts, as well as further validation of the tool capabilities, enhancing the tool's utility across diverse medical domains.

The benefit of chart curation of electronic health record (EHR) data: Comparing data completeness between curated and structured data.

11165 Background: The main objective of an EHR is to effectively standardize clinical workflow. However, EHR data is also utilized extensively as secondary data for clinical research and quality improvement. Healthcare data exists in different forms such as paper, digital images, and notes, only some of which are recorded as structured data in the EHR. It is therefore essential to curate detailed information from unstructured data so that the patient’s journey can be understood in depth for research and quality studies. Methods: Patients diagnosed with metastatic breast cancer (mBC) between 01-Jan-2020 and 31-Dec-2022 were identified from the Integra Connect PrecisionQ de-identified database of 3 million cancer patients across 500 sites of care across ~80% community oncology and ~20% academic practices. Manual curation was conducted for a sample of these patients and the fill rates of crucial elements were captured as part of this study. Results: A total of 13,763 mBC patients were identified during this study period. The availability of data for the different variables was assessed from structured data in 13,120 patients. Additional information was obtained by manual curation for 643 patients. Information on staging and grade were available for 62.8% in the structured data, while curation increased the availability of these details to 99.5% among those 643 patients. The fill rates for patients’ tumor size and nodal status, as defined by the T and N values, were found to be 99% in the curated data compared to 60% in the structured data. Similarly, the fill rates were more than 98% for estrogen, progesterone, and HER2 receptor by curation, while they were only approximately 65% in the structured data. HER2 low, HER2-ve, and HER2+ve status was identified in 8%, 45.2%, and 10.1% by structured data and in 57.1%, 84.1%, and 14.0%, respectively, by curated data. Furthermore, the fill rates via curation were 97.1% for HER2 immunohistochemistry tests compared to 49.0% for structured data, while the fill rates for in-situ hybridization (ISH) or fluorescent in-situ hybridization (FISH) tests were 61.3% for curated data compared to 28.4% for structured data. This discrepancy is caused by the fact that IHC and FISH/ISH tests are primarily found only in PDF format. Conclusions: This study highlights the need for curation in order to maximize the utilization of EHR data for secondary research purposes and quality studies. Natural language processing and augmented curation methods can further enhance the quality of EHR data for secondary research.

Generating Synthetic Electronic Health Record Data Using Generative Adversarial Networks: Tutorial.

Synthetic electronic health record (EHR) data generation has been increasingly recognized as an important solution to expand the accessibility and maximize the value of private health data on a large scale. Recent advances in machine learning have facilitated more accurate modeling for complex and high-dimensional data, thereby greatly enhancing the data quality of synthetic EHR data. Among various approaches, generative adversarial networks (GANs) have become the main technical path in the literature due to their ability to capture the statistical characteristics of real data. However, there is a scarcity of detailed guidance within the domain regarding the development procedures of synthetic EHR data. The objective of this tutorial is to present a transparent and reproducible process for generating structured synthetic EHR data using a publicly accessible EHR data set as an example. We cover the topics of GAN architecture, EHR data types and representation, data preprocessing, GAN training, synthetic data generation and postprocessing, and data quality evaluation. We conclude this tutorial by discussing multiple important issues and future opportunities in this domain. The source code of the entire process has been made publicly available.

Development and validation of a rule-based algorithm to identify periodontal diagnosis using structured electronic health record data.

To develop and validate an automated electronic health record (EHR)-based algorithm to suggest a periodontal diagnosis based on the 2017 World Workshop on the Classification of Periodontal Diseases and Conditions. Using material published from the 2017 World Workshop, a tool was iteratively developed to suggest a periodontal diagnosis based on clinical data within the EHR. Pertinent clinical data included clinical attachment level (CAL), gingival margin to cemento-enamel junction distance, probing depth, furcation involvement (if present) and mobility. Chart reviews were conducted to confirm the algorithm's ability to accurately extract clinical data from the EHR, and then to test its ability to suggest an accurate diagnosis. Subsequently, refinements were made to address limitations of the data and specific clinical situations. Each refinement was evaluated through chart reviews by expert periodontists at the study sites. Three-hundred and twenty-three charts were manually reviewed, and a periodontal diagnosis (healthy, gingivitis or periodontitis including stage and grade) was made by expert periodontists for each case. After developing the initial version of the algorithm using the unmodified 2017 World Workshop criteria, accuracy was 71.8% for stage alone and 64.7% for stage and grade. Subsequently, 16 modifications to the algorithm were proposed and 14 were accepted. This refined version of the algorithm had 79.6% accuracy for stage alone and 68.8% for stage and grade together. Our findings suggest that a rule-based algorithm for suggesting a periodontal diagnosis using EHR data can be implemented with moderate accuracy in support of chairside clinical diagnostic decision making, especially for inexperienced clinicians. Grey-zone cases still exist, where clinical judgement will be required. Future applications of similar algorithms with improved performance will depend upon the quality (completeness/accuracy) of EHR data.

Estimating the prevalence of diabetic retinopathy in electronic health records with massive missing labels

ObjectiveThe paper aims to address the problem of massive unlabeled patients in electronic health records (EHR) who potentially have undiagnosed diabetic retinopathy (DR). It is desired to estimate the actual DR prevalence in EHR with 96 % missing labels. Materials and methodsThe Cerner Health Facts data are used in the study, with 3749 labeled DR patients and 97,876 unlabeled diabetic patients. This extensive dataset spans the demographics of the United States over the past two decades. We implemented state-of-art positive-unlabeled learning methods, including ensemble-based support vector machine, ensemble-based random forest, and Bayesian finite mixture modeling. ResultsThe estimated DR prevalence in the population represented by Cerner EHR is approximately 25 % and the classification techniques generally achieve an AUC of around 87 %. As a by-product, a predictive inference on the risk of DR based on a patient's personalized medical information is derived. DiscussionMissing labels is a common issue for EHR data quality. Ignoring these missing labels can lead to biased results in the analyses of EHR data. The problem is especially severe in the context of DR. It is thus important to use machine learning or statistical tools to identify the unlabeled patients. The tool in this paper helps both data analysts and clinicians in their practices.

An EHR Data Quality Evaluation Approach Based on Medical Knowledge and Text Matching

Introduction: Recently, medical artificial intelligence based on Electronic Health Records (EHR) is a significant research field, and EHR data has been widely used in clinical decision support systems and medical diagnosis systems. However, because EHR are used to record the patient's disease information and are not primarily designed for research and discovery, the utility of EHR for research will be hindered by data quality problems. Therefore, it is a meaningful and challenging task to evaluate the data quality of EHR before they are used in medical artificial intelligence. Most of the current EHR data quality evaluation methods are based on some conventional evaluation indicators, and rarely consider the introduction of clinical evidence.Materials and methods: we propose an EHR data quality evaluation approach based on clinical evidence and a deep text matching model. First, based on the medical knowledge of the particular disease, we establish the list of standard clinical evidence descriptions including typical symptoms and special signs, etc. Then we find the relevant clinical evidence from the EHR based on the text matching model, and finally evaluate the quality of the EHR based on the quantity and quality of the relevant clinical evidence found.Results: The experimental results of more than 1,000 EHR for two types of diseases show that our approach can effectively distinguish high-quality EHR from low-quality EHR, and the high-quality EHR found generally contains sufficient and consistent information related to disease diagnosis.Conclusions: Experiments results on a real-world dataset demonstrate the effectiveness of our EHR data quality evaluation approach based on medical knowledge and text matching.

Electronic health record data quality variability across a multistate clinical research network.

Electronic health record (EHR) data have many quality problems that may affect the outcome of research results and decision support systems. Many methods have been used to evaluate EHR data quality. However, there has yet to be a consensus on the best practice. We used a rule-based approach to assess the variability of EHR data quality across multiple healthcare systems. To quantify data quality concerns across healthcare systems in a PCORnet Clinical Research Network, we used a previously tested rule-based framework tailored to the PCORnet Common Data Model to perform data quality assessment at 13 clinical sites across eight states. Results were compared with the current PCORnet data curation process to explore the differences between both methods. Additional analyses of testosterone therapy prescribing were used to explore clinical care variability and quality. The framework detected discrepancies across sites, revealing evident data quality variability between sites. The detailed requirements encoded the rules captured additional data errors with a specificity that aids in remediation of technical errors compared to the current PCORnet data curation process. Other rules designed to detect logical and clinical inconsistencies may also support clinical care variability and quality programs. Rule-based EHR data quality methods quantify significant discrepancies across all sites. Medication and laboratory sources are causes of data errors.

Abstract A020: Assessment of EHR data quality variabilities among different racial groups in the cohort of de novo stage IV breast cancer

Abstract Introduction: Despite the decline in overall mortality and incidence of cancer in the US population, disparities in cancer care still largely exist within certain groups. The proportion of racial and ethnic minorities recruited to participate in cancer research is persistently lower than the US population. The rapid adoption of electronic health records (EHRs) systems has enabled computational cohort identification and eligibility screening for facilitating cancer research. However, EHR data is known to suffer from data quality issues due to numerous factors, such as care access. Patients with limited care access due to low socioeconomic status would naturally be sparsely represented in EHR data relative to those of higher status. Prospective enrollees may be disqualified due to biased or incomplete EHR data at the screening stage. Methods: To systematically identify data quality variabilities in EHRs, we implemented an information score (i-score) to reflect the density, variability, and irregularity of patients’ data. The pattern can be measured by estimating the variability of the time gaps between observations. For a patient observed n times, the encountered observation times are represented as x(1)…x(n). The relative time interval g for each observed encounter can be defined as g1 = [x(i+1)-x(i)]/[x(n)-x(1)], for I = 1…n - 1. The average amount of information of each observation is then defined as I, between 0 and 1, calculated as: I = 2/n+(n-2)/n[1-sqrt((n-1)Var{gi;i=1…n-1})]. An equality-spaced observation would receive a high i-score. The lower the i-score, the higher the data irregularity. We conducted a case study in a cohort of de novo stage IV breast cancer (DNIV) assembled by a previously validated phenotyping algorithm (Wang AACR 2020). The cohort contains 1,918 DNIV cases between 2004 and 2018 at Mayo Clinic Rochester. We used the i-score to assess patient visits, breast cancer diagnosis codes, and narrative documentation, achieved by the extraction of stage, surgery, de novo, and metastasis information based on the phenotyping definitions. The summary statistics of the i-scores were compared among four racial groups: African American (3%), Asian (1%), Caucasian (87%), and Other (9%). ANOVA F test was used to compare the group differences. Results and Conclusion: The mean i-scores for visit, diagnosis and narrative documentation were 0.712, 0.200, and 0.056 for African American, 0.412, 0.301, and 0.073 for Asian, 0.438, 0.278, and 0.056 for Caucasian, and 0.519, 0.343, and 0.043 for the other category, respectively. The F test indicated significant differences between visit (p<0.001) and diagnosis (p=0.0267) data. No difference was found in narrative documentation (p=0. 8591). In conclusion, there was a substantial i-score variation discovered, especially within the African American group, suggesting a strong need to address EHR quality disparity for cancer research. Citation Format: Sunyang Fu, Liwei Wang, Folakemi T. Odedina, Hongfang Liu. Assessment of EHR data quality variabilities among different racial groups in the cohort of de novo stage IV breast cancer [abstract]. In: Proceedings of the 15th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2022 Sep 16-19; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2022;31(1 Suppl):Abstract nr A020.

Quantifying Electronic Health Record Data Quality in Telehealth and Office-Based Diabetes Care.

Data derived from the electronic health record (EHR) are commonly reused for quality improvement, clinical decision-making, and empirical research despite having data quality challenges. Research highlighting EHR data quality concerns has largely been examined and identified during traditional in-person visits. To understand variations in data quality among patients managing type 2 diabetes mellitus (T2DM) with and without a history of telehealth visits, we examined three EHR data quality dimensions: timeliness, completeness, and information density. We used EHR data (2016-2021) from a local enterprise data warehouse to quantify timeliness, completeness, and information density for diagnostic and laboratory test data. Means and chi-squared significance tests were computed to compare data quality dimensions between patients with and without a history of telehealth use. Mean timeliness or T2DM measurement age for the study sample was 77.8 days (95% confidence interval [CI], 39.6-116.4). Mean completeness for the sample was 0.891 (95% CI, 0.868-0.914). The mean information density score was 0.787 (95% CI, 0.747-0.827). EHR data for patients managing T2DM with a history of telehealth use were timelier (73.3 vs. 79.8 days), and measurements were more uniform across visits (0.795 vs. 0.784) based on information density scores, compared with patients with no history of telehealth use. Overall, EHR data for patients managing T2DM with a history of telehealth visits were generally timelier and measurements were more uniform across visits than for patients with no history of telehealth visits. Chronic disease care relies on comprehensive patient data collected via hybrid care delivery models and includes important domains for continued data quality assessments prior to secondary reuse purposes.

Quantifying the Electronic Health Record Burden in Head and Neck Cancer Care.

Background Although the main task of health care providers is to provide patient care, studies show that increasing amounts of time are spent on documentation. Objective To quantify the time and effort spent on the electronic health record (EHR) in head and neck cancer care. Methods Cross-sectional time–motion study. Primary outcomes were the percentages of time spent on the EHR and the three main tasks (chart review, input, placing orders), number of mouse events, and keystrokes per consultation. Secondary outcome measures were perceptions of health care providers regarding EHR documentation and satisfaction. Results In total, 44.0% of initial oncological consultation (IOC) duration and 30.7% of follow-up consultation (FUC) duration are spent on EHR tasks. During 80.0% of an IOC and 67.9% of a FUC, the patient and provider were actively communicating. Providers required 593 mouse events and 1,664 keystrokes per IOC and 140 mouse events and 597 keystrokes per FUC, indicating almost 13 mouse clicks and close to 40 keystrokes for every minute of consultation time. Less than a quarter of providers indicated that there is enough time for documentation. Conclusion This study quantifies the widespread concern of high documentation burden for health care providers in oncology, which has been related to burnout and a decrease of patient–clinician interaction. Despite excessive time and effort spent on the EHR, health care providers still felt this was insufficient for proper documentation. However, the need for accurate and complete documentation is high, as reuse of information becomes increasingly important. The challenge is to decrease the documentation burden while increasing the quality of EHR data.

Chasm Between Cancer Quality Measures and Electronic Health Record Data Quality.

The Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) requires eligible clinicians to report clinical quality measures (CQMs) in the Merit-Based Incentive Payment System (MIPS) to maximize reimbursement. To determine whether structured data in electronic health records (EHRs) were adequate to report MIPS CQMs, EHR data aggregated by ASCO's CancerLinQ platform were analyzed. Using the CancerLinQ health technology platform, 19 Oncology MIPS (oMIPS) CQMs were evaluated to determine the presence of data elements (DEs) necessary to satisfy each CQM and the DE percent population with patient data (fill rates). At the time of this analysis, the CancerLinQ network comprised 63 active practices, representing eight different EHR vendors and containing records for more than 1.63 million unique patients with one or more malignant neoplasms (1.73 million cancer cases). Fill rates for the 63 oMIPS-associated DEs varied widely among the practices. The average site had at least one filled DE for 52% of the DEs. Only 35% of the DEs were populated for at least one patient record in 95% of the practices. However, the average DE fill rate of all practices was 23%. No data were found at any practice for 22% of the DEs. Since any oMIPS CQM with an unpopulated DE component resulted in an inability to compute the measure, only two (10.5%) of the 19 oMIPS CQMs were computable for more than 1% of the patients. Although EHR systems had relatively high DE fill rates for some DEs, underfilling and inconsistency of DEs in EHRs render automated oncology MIPS CQM calculations impractical.

Evaluating a Prototype Clinical Decision Support Tool for Chronic Pain Treatment in Primary Care.

The Chronic Pain Treatment Tracker (Tx Tracker) is a prototype decision support tool to aid primary care clinicians when caring for patients with chronic noncancer pain. This study evaluated clinicians' perceived utility of Tx Tracker in meeting information needs and identifying treatment options, and preferences for visual design. We conducted 12 semi-structured interviews with primary care clinicians from four health systems in Indiana. The interviews were conducted in two waves, with prototype and interview guide revisions after the first six interviews. The interviews included exploration of Tx Tracker using a think-aloud approach and a clinical scenario. Clinicians were presented with a patient scenario and asked to use Tx Tracker to make a treatment recommendation. Last, participants answered several evaluation questions. Detailed field notes were collected, coded, and thematically analyzed by four analysts. We identified several themes: the need for clinicians to be presented with a comprehensive patient history, the usefulness of Tx Tracker in patient discussions about treatment planning, potential usefulness of Tx Tracker for patients with high uncertainty or risk, potential usefulness of Tx Tracker in aggregating scattered information, variability in expectations about workflows, skepticism about underlying electronic health record data quality, interest in using Tx Tracker to annotate or update information, interest in using Tx Tracker to translate information to clinical action, desire for interface with visual cues for risks, warnings, or treatment options, and desire for interactive functionality. Tools like Tx Tracker, by aggregating key information about past, current, and potential future treatments, may help clinicians collaborate with their patients in choosing the best pain treatments. Still, the use and usefulness of Tx Tracker likely relies on continued improvement of its functionality, accurate and complete underlying data, and tailored integration with varying workflows, care team roles, and user preferences.

Assessing quality and agreement of structured data in automatic versus manual abstraction of the electronic health record for a clinical epidemiology study

Objective We evaluate data agreement between an electronic health record (EHR) sample abstracted by automated characterization with a standard abstracted by manual review. Study Design and Setting We obtain data for an epidemiology cohort study using standard manual abstraction of the EHR and automated identification of the same patients using a structured algorithm to query the EHR. Summary measures of agreement (e.g., Cohen’s kappa) are reported for 12 variables commonly used in epidemiological studies. Results Best agreement between abstraction methods is observed among demographic characteristics such as age, sex, and race, and for positive history of disease. Poor agreement is found in missing data and negative history, suggesting potential impact for researchers using automated EHR characterization. EHR data quality depends upon providers, who may be influenced by both institutional and federal government documentation guidelines. Conclusion Automated EHR abstraction discrepancies may decrease power and increase bias; therefore, caution is warranted when selecting variables from EHRs for epidemiological study using an automated characterization approach. Validation of automated methods must also continue to advance in sophistication with other technologies, such as machine learning and natural language processing, to extract non-structured data from the EHR, for application to EHR characterization for clinical epidemiology.

Quality of Hospital Electronic Health Record (EHR) Data Based on the International Consortium for Health Outcomes Measurement (ICHOM) in Heart Failure: Pilot Data Quality Assessment Study.

BackgroundThere is increasing recognition that health care providers need to focus attention, and be judged against, the impact they have on the health outcomes experienced by patients. The measurement of health outcomes as a routine part of clinical documentation is probably the only scalable way of collecting outcomes evidence, since secondary data collection is expensive and error-prone. However, there is uncertainty about whether routinely collected clinical data within electronic health record (EHR) systems includes the data most relevant to measuring and comparing outcomes and if those items are collected to a good enough data quality to be relied upon for outcomes assessment, since several studies have pointed out significant issues regarding EHR data availability and quality.ObjectiveIn this paper, we first describe a practical approach to data quality assessment of health outcomes, based on a literature review of existing frameworks for quality assessment of health data and multistakeholder consultation. Adopting this approach, we performed a pilot study on a subset of 21 International Consortium for Health Outcomes Measurement (ICHOM) outcomes data items from patients with congestive heart failure.MethodsAll available registries compatible with the diagnosis of heart failure within an EHR data repository of a general hospital (142,345 visits and 12,503 patients) were extracted and mapped to the ICHOM format. We focused our pilot assessment on 5 commonly used data quality dimensions: completeness, correctness, consistency, uniqueness, and temporal stability.ResultsWe found high scores (>95%) for the consistency, completeness, and uniqueness dimensions. Temporal stability analyses showed some changes over time in the reported use of medication to treat heart failure, as well as in the recording of past medical conditions. Finally, the investigation of data correctness suggested several issues concerning the characterization of missing data values. Many of these issues appear to be introduced while mapping the IMASIS-2 relational database contents to the ICHOM format, as the latter requires a level of detail that is not explicitly available in the coded data of an EHR.ConclusionsOverall, results of this pilot study revealed good data quality for the subset of heart failure outcomes collected at the Hospital del Mar. Nevertheless, some important data errors were identified that were caused by fundamentally different data collection practices in routine clinical care versus research, for which the ICHOM standard set was originally developed. To truly examine to what extent hospitals today are able to routinely collect the evidence of their success in achieving good health outcomes, future research would benefit from performing more extensive data quality assessments, including all data items from the ICHOM standards set and across multiple hospitals.

Clinical comparison between trial participants and potentially eligible patients using electronic health record data: A generalizability assessment method

ObjectiveTo present a generalizability assessment method that compares baseline clinical characteristics of trial participants (TP) to potentially eligible (PE) patients as presented in their electronic health record (EHR) data while controlling for clinical setting and recruitment period. MethodsFor each clinical trial, a clinical event was defined to identify patients of interest using available EHR data from one clinical setting during the trial’s recruitment timeframe. The trial’s eligibility criteria were then applied and patients were separated into two mutually exclusive groups: (1) TP, which were patients that participated in the trial per trial enrollment data; (2) PE, the remaining patients. The primary outcome was standardized differences in clinical characteristics between TP and PE per trial. A standardized difference was considered prominent if its absolute value was greater than or equal to 0.1. The secondary outcome was the difference in mean propensity scores (PS) between TP and PE per trial, in which the PS represented prediction for a patient to be in the trial. Three diverse trials were selected for illustration: one focused on hepatitis C virus (HCV) patients receiving a liver transplantation; one focused on leukemia patients and lymphoma patients; and one focused on appendicitis patients. ResultsFor the HCV trial, 43 TP and 83 PE were found, with 61 characteristics evaluated. Prominent differences were found among 69% of characteristics, with a mean PS difference of 0.13. For the leukemia/lymphoma trial, 23 TP and 23 PE were found, with 39 characteristics evaluated. Prominent differences were found among 82% of characteristics, with a mean PS difference of 0.76. For the appendicitis trial, 123 TP and 242 PE were found, with 52 characteristics evaluated. Prominent differences were found among 52% of characteristics, with a mean PS difference of 0.15. ConclusionsDifferences in clinical characteristics were observed between TP and PE among all three trials. In two of the three trials, not all of the differences necessarily compromised trial generalizability and subsets of PE could be considered similar to their corresponding TP. In the remaining trial, lack of generalizability appeared present, but may be a result of other factors such as small sample size or site recruitment strategy. These inconsistent findings suggest eligibility criteria alone are sometimes insufficient in defining a target group to generalize to. With caveats in limited scalability, EHR data quality, and lack of patient perspective on trial participation, this generalizability assessment method that incorporates control for temporality and clinical setting promise to better pinpoint clinical patterns and trial considerations.

Factors affecting clinicians' adherence to principles of diagnosis documentation: A concept mapping approach for improved decision-making.

The quality of data in electronic health records (EHRs) depends on adherence of clinicians to principles of diagnosis documentation. A concept mapping (CM) approach was used to extract factors related to quality of clinicians' documentation that govern EHR data quality. Influential factors extracted from brainstorming sessions were sorted by individual participants, followed by a quantitative analysis using multidimensional scaling and cluster analysis to categorise sorted factors. Finally, a questionnaire was used to elicit the importance-feasibility of the extracted factors. Results were visualised by cluster maps and Go-Zone plots. Factors were classified into seven clusters: "knowledge about International Classification of Diseases and clinical coding," "need for facilitators and guidelines," "explaining the importance of the issue and defining responsibilities," "cooperation of other personnel," "codify legal requirements," "workload" and "clinical obstacles," as ranked by importance. To enhance the quality of EHR data, a collaboration between physicians, nurses, managers and EHR developers is required. CM is an acceptable approach to meet this objective. Our findings highlight the significance of clinical coding knowledge, awareness about its importance and applicability and use of well-structured information systems. In combination, these three factors can have a strong positive impact on the quality of EHR data. A list of solutions is provided for policymakers, and two interventions suggested, based on the findings of this study, including the adoption of EHRs that incorporate documentation guidelines. We further propose updated clinical training programs and a monitoring and feedback mechanism to facilitate the EHR documentation process.