Observational Healthcare Data Research Articles

Statistical challenges in systematic evidence generation through analysis of observational healthcare data networks

Observational healthcare databases offer significant potential for generating evidence about the effects of medical products. Recent US and international efforts attempt to coordinate analysis activities across observational data networks for active drug safety surveillance and comparative effectiveness research. Several statistical challenges exist that require further research to inform the appropriate use of observational data for identifying and evaluating temporal relationships between medical product exposures and outcomes. This review highlights current work across the areas of data quality, methods development, and performance evaluation that promise to contribute to our understanding of best practices for observational analyses moving forward.

Using evidence from observational healthcare data to inform a trial design

Clinical trials are expensive. Understanding where a trial is needed and what questions to address is therefore important. We ask if pilot work using observational healthcare data can be used to inform a trial. Symptoms of significant psychological distress are common in cancer patients but are often left untreated. It is unknown to what extent such distress symptoms persist over time and require treatment. Should a clinical trial be devised to determine if patients would benefit from treatment? How many and what patients are likely to suffer from persistent distress and to require treatment? On what criteria should patients be included in a trial, and when should trial outcomes be evaluated? We address these questions using repeated, psychological symptom data that were routinely collected from over 20,000 outpatients who attended selected oncology clinics in Scotland. However, the limitations to routinely collected observational healthcare data are many. We approach the analysis within a missing data context and develop a Bayesian model in WinBUGS to estimate the posterior predictive distribution for the incomplete longitudinal responses and covariates under both MAR and MNAR mechanisms and use this model to generate multiply imputed datasets for analysis in SAS. The findings are compared with those from a purpose-designed, prospective, clinical study in the same population. Meaningful analysis of observational healthcare data requires subject-matter knowledge, good quality data and appropriate statistical methods. With flexible modelling techniques now more practicable, we believe pilot work with such data could help inform future trial designs.

Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership

Expanded availability of observational healthcare data (both administrative claims and electronic health records) has prompted the development of statistical methods for identifying adverse events associated with medical products, but the operating characteristics of these methods when applied to the real-world data are unknown. We studied the performance of eight analytic methods for estimating of the strength of association-relative risk (RR) and associated standard error of 53 drug-adverse event outcome pairs, both positive and negative controls. The methods were applied to a network of ten observational healthcare databases, comprising over 130 million lives. Performance measures included sensitivity, specificity, and positive predictive value of methods at RR thresholds achieving statistical significance of p < 0.05 or p < 0.001 and with absolute threshold RR > 1.5, as well as threshold-free measures such as area under receiver operating characteristic curve (AUC). Although no specific method demonstrated superior performance, the aggregate results provide a benchmark and baseline expectation for risk identification method performance. At traditional levels of statistical significance (RR > 1, p < 0.05), all methods have a false positive rate >18%, with positive predictive value <38%. The best predictive model, high-dimensional propensity score, achieved an AUC = 0.77. At 50% sensitivity, false positive rate ranged from 16% to 30%. At 10% false positive rate, sensitivity of the methods ranged from 9% to 33%. Systematic processes for risk identification can provide useful information to supplement an overall safety assessment, but assessment of methods performance suggests a substantial chance of identifying false positive associations.

The Use of Observational Health-Care Data to Identify and Report on Off-Label Use of Biopharmaceutical Products

Significant efforts are under way to develop analytic surveillance systems that can leverage observational health-care data to monitor biopharmaceutical products. The systems are being designed to identify safety-related events in a post-approval setting, when the products are already on the market.An additional application of such a system would be the use of observational data to help identify and track off-label use of marketed biopharmaceutical products in specific populations.The opportunities and challenges associated with such a system, as well as the underlying data sources themselves, are reviewed in this article.

Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data

Many challenges arise when conducting a sequentially monitored medical product safety surveillance evaluation using observational electronic data captured during routine care. We review existing sequential approaches for potential use in this setting, including a continuous sequential testing method that has been utilized within the Vaccine Safety Datalink (VSD) and group sequential methods, which are used widely in randomized clinical trials. Using both simulated data and preliminary data from an ongoing VSD evaluation, we discuss key sequential design considerations, including sample size and duration of surveillance, shape of the signaling threshold, and frequency of interim testing. All designs control the overall Type 1 error rate across all tests performed, but each yields different tradeoffs between the probability and timing of true and false positive signals. Designs tailored to monitor efficacy outcomes in clinical trials have been well studied, but less consideration has been given to optimizing design choices for observational safety settings, where the hypotheses, population, prevalence and severity of the outcomes, implications of signaling, and costs of false positive and negative findings are very different. Analytic challenges include confounding, missing and partially accrued data, high misclassification rates for outcomes presumptively defined using diagnostic codes, and unpredictable changes in dynamically accessed data over time (e.g., differential product uptake). Many of these factors influence the variability of the adverse events under evaluation and, in turn, the probability of committing a Type 1 error. Thus, to ensure proper Type 1 error control, planned sequential thresholds should be adjusted over time to account for these issues.

Surveying US Observational Data Sources and Characteristics for Drug Safety Needs

Abstract Background: Active surveillance for potential drug safety issues requires access to observational healthcare data to identify and evaluate the association between drugs and subsequent outcomes. This study assessed the availability and content of observational data, as well as the interest of the data holder in participating in drug safety surveillance. Methods: Between January and June 2009 we conducted a two-stage online survey of data holders.The surveys evaluated characteristics of the data sources and populations, as well as the experience of the organization in conducting drug safety evaluations and research. The first-stage ten-question survey was designed to capture basic information about the size of source population and types of data elements available and gauge interest in further participation in this research. The second-stage 70-question survey provided a comprehensive, quantitative and qualitative summary of population characteristics, data capture of drugs, health outcomes, procedures, visits and laboratory results, as well as a descriptive review of the organization’s evaluation and research experience, infrastructure and process and interest in drug safety surveillance. Results: Of the 70 US organizations that were contacted, 37 agreed to participate in the first stage of the survey. Of these, 26 reported access to data for over 1 million persons, and 9 reported more than 10 million persons. Respondents reflected a wide range of data stakeholders, including claims data aggregators, hospital benchmarking organizations, fiscal intermediaries that reimburse claims and integrated healthcare systems. Among the 21 organizations completing the second-stage survey, the aggregate data represented 540 million persons, 257 million of them currently active. Data sources showed significant variation in size (ranging from 34 000 to 160 million lives) and demographics of underlying populations, length of follow-up and completeness of data capture. Respondents had a median of 9 years coverage across the population, with a maximum of 51 years, and databases utilizing claims data tended to have a lower average length of capture than those using electronic health records. Data holders varied in interest and ability to conduct drug safety surveillance. Conclusions: Multiple observational databases are needed to meet the requirements for an active drug safety surveillance system. Developing a network of disparate data sources requires careful consideration of the complementary nature of the underlying data and the necessary infrastructure to support coordination across the diverse stakeholder groups.