Effect of response format for clinical vignettes on reporting quality of physician practice

Thao Pham,Fréderic Lioté,Carine Roy,Philippe Ravaud,Pierre Durieux,Xavier Mariette

doi:10.1186/1472-6963-9-128

Abstract

BackgroundClinical vignettes have been used widely to compare quality of clinical care and to assess variation in practice, but the effect of different response formats has not been extensively evaluated. Our objective was to compare three clinical vignette-based survey response formats – open-ended questionnaire (A), closed-ended (multiple-choice) questionnaire with deceptive response items mixed with correct items (B), and closed-ended questionnaire with only correct items (C) – in rheumatologists' pre-treatment assessment for tumor-necrosis-factor (TNF) blocker therapy.MethodsStudy design: Prospective randomized study. Setting: Rheumatologists attending the 2004 French Society of Rheumatology meeting. Physicians were given a vignette describing the history of a fictitious woman with active rheumatoid arthritis, who was a candidate for therapy with TNF blocking agents, and then were randomized to receive questionnaire A, B, or C, each containing the same four questions but with different response formats, that asked about their pretreatment assessment. Measurements: Long (recommended items) and short (mandatory items) checklists were developed for pretreatment assessment for TNF-blocker therapy, and scores were expressed on the basis of responses to questionnaires A, B, and C as the percentage of respondents correctly choosing explicit items on these checklists. Statistical analysis: Comparison of the selected items using pairwise Chi-square tests with Bonferonni correction for variables with statistically significant differences.ResultsData for all surveys distributed (114 As, 118 Bs, and 118 Cs) were complete and available for analysis. The percentage of questionnaire A, B, and C respondents for whom data was correctly complete for the short checklist was 50.4%, 84.0% and 95.0%, respectively, and was 0%, 5.0% and 5.9%, respectively, for the long version. As an example, 65.8%, 85.7% and 95.8% of the respondents of A, B, and C questionnaires, respectively, correctly identified the need for tuberculin skin test (p < 0.0001).ConclusionIn evaluating clinical practice with use of a clinical vignette, a multiple-choice format rather than an open-ended format overestimates physician performance. The insertion of deceptive response items mixed with correct items in closed-ended (multiple-choice) questionnaire failed to avoid this overestimation.

Highlights

Clinical vignettes have been used widely to compare quality of clinical care and to assess variation in practice, but the effect of different response formats has not been extensively evaluated
We conducted a prospective randomized study aimed at comparing three response modalities for a vignette-based survey: open-ended questionnaire, closed-ended questionnaire and closedended questionnaire with deceptive response items mixed with correct items
Physicians were asked to answer four questions about their pre-treatment assessment, considering that TNFblocking treatment was planned: 1) what specific data are you searching for in this patient's history? 2) What clinical data are you personally searching for during the physical examination? 3) Which biological, radiographic or other tests do you request? 4) What other preventive measures do you take? Physicians were given these questions in one of three questionnaire formats: open-ended questionnaire [see Additional file 1], closed-ended questionnaire with deceptive response items mixed with correct items (n = 73) [see Additional file 2], closed-ended questionnaire with only correct items (n = 35) [see Additional file 3]

Summary

Introduction

Clinical vignettes have been used widely to compare quality of clinical care and to assess variation in practice, but the effect of different response formats has not been extensively evaluated. Improvement of quality of clinical practice needs quality measurements These measurements must be accurate, valid and feasible. Methods include chart extraction, standardized patients and clinical vignettes. Compared to standardized patients and chart extraction, clinical vignettes are an accurate, valid, feasible and inexpensive tool to measure quality of health care [4,5]. Clinical vignettes have been used widely to compare quality of clinical care and to assess variation in practice across countries, health care systems, specialties or clinicians [6,7,8,9,10,11]

Objectives

Methods

Results

Discussion

Conclusion