Impact of tablet-scoring and immediate score sheet review on validity and educational impact in an internal medicine residency Objective Structured Clinical Exam (OSCE)

Abstract

Introduction: The Objective Structured Clinical Examination (OSCE) is used globally for formative and summative purposes. The objective of this study was to examine the impact of tablet-scoring on sources of validity evidence for an Internal Medicine residency OSCE.Methods: We compared paper-scored OSCEs from 2014 to tablet-scored OSCEs in 2015 for missing data, amount of comments, and time to pass/fail decision. We then examined in 2016 the impact on learning of showing residents their score sheets and asking them to write an action plan immediately after an OSCE.Results: Tablet-scoring significantly reduced stations with missing data from 1.8 to 0.2%, and stations without comments from 42 to 28% with an increase in word count per comment. Time to official results reduced from 3 weeks to 12 h with tablet-scoring. Residents who wrote a learning plan after reviewing their OSCE score sheets were more likely (with medium to large effect sizes) to pursue further studying and/or change their behavior (e.g. history taking or physical examination) in the clinical environment.Conclusions: OSCE tablet-scoring improved many sources of validity evidence, especially educational impact with timeliness of feedback supporting a change in behavior, a hard to achieve goal of educational interventions.