Pitfalls of Using Administrative Data for Research

Abstract

It has been estimated that more than 14 million colonoscopy procedures are performed annually in the United States for screening, surveillance, and evaluation of symptoms [1]. The effectiveness of gastrointestinal (GI) procedures depends on appropriate utilization, technical competence, recognition and management of pathology, a low rate of missed lesions of importance, and a low rate of adverse events—all elements of quality [2]. As we move towards health care systems which reward quality (‘‘pay for performance’’) over quantity, we need to think about how quality can be assessed and monitored. It would be ideal if we could use existing databases to extract important measures of quality. Administrative databases provide an inexpensive tool for looking at health care utilization in health care systems. Claims data are extremely accurate to determine if a procedure or intervention was performed and who received the procedure. However, there is only limited experience with using administrative data to measure important quality indicators [3]. Most often, research databases are required to measure quality [4]. Fisher et al. [5] explored using administrative data from the Department of Veterans Affairs (VA) to determine if the indication for colonoscopy can be accurately determined. Establishing the procedure indication is an essential first step in determining whether the procedure is appropriate and if the findings are consistent with established benchmarks. For example, there are new data that colonoscopy surveillance may be overused in low-risk patients and under-used in high-risk patients [6]. If the surveillance indication can be established with administrative data, one could then determine if the surveillance interval is appropriate, based on current guidelines. In the case of colonoscopy screening, the endpoint of interest is adenoma detection rate (ADR). Data from screening colonoscopy studies has indicated that first-time screening exams should identify adenomas in at least 15% of women, and 25% of men, with variation based on age [7]. If a system like the VA wants to understand how colonoscopy screening is being utilized, and if outcomes (ADR) are meeting expectations, it would be essential to determine if the colonoscopy is being performed for screening. When measured over time, such data could monitor changes in practice and resource utilization. Previous attempts to use VA administrative data revealed that algorithms were poorly sensitive (70%) and specific (72%) for identification of primary screening indication [8]. Lacking precision, attempts to determine if procedures were appropriate and if outcomes (i.e., ADR) reached expected benchmarks are fraught with inaccuracy. Fisher et al. tested modifications of the previous algorithm by El-Serag et al. [8], and validated results with chart review. The El-Serag algorithm was more accurate than the revised versions, but with sensitivity of 83–90% and specificity of 57–76%, many patients undergoing screening would not be identified, and many others would be misclassified. These data highlight some, but not all, of the limitations of using administrative data for research. A recent study of colonoscopy highlights some of the limitations of using administrative data for research and quality assurance. Baxter et al. [9] used claims data in Ontario to determine if prior colonoscopy reduced mortality of colorectal cancer (CRC) in a case–control, population-based study. The study analyzed data from 1993 to 2003. More than 10,000 cancer cases were identified, and D. Lieberman (&) Division of Gastroenterology and Hepatology, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Portland, OR 97239, USA e-mail: lieberma@ohsu.edu