Measuring the Quality of the OSCE in a Chiropractic Programme: A Review of Metrics and Recommendations.

Abstract

The Objective Structured Clinical Examination (OSCE) is a commonly used assessment of clinical skill, and ensuring the quality and reliability in OSCEs is a complex, and on-going process. This paper discusses scoring schemas and reviews checklists and global rating scales (GRS) for marking. Also detailed are post-examination quality assurance metrics tailored to smaller cohorts, with an illustrative dataset. A deidentified OSCE dataset, from stations with both a checklist and GRS, of 24 examinees from a 2021 cohort was assessed using the following metrics: Cut-scores or pass-rates, number-of-failures, R2, inter-grade discrimination, and between-group-variation. The results were used to inform a set of implementable recommendations to improve future OSCEs. For most stations the calculated cut-score calculated was higher than the traditional pass of 50% (58.9.8-68.4%). Number-of-failures were low for traditional pass rates and cuts-scores (0.00-16.7%), excepting Lab Analysis where number-of-failures was 50.0%. R2 values ranged from 0.67-0.97, but proportion of total variance was high (67.3-95.9). These data suggest there were potential missed teaching concepts, that station marking was open to examiner interpretation, and there were inconsistencies in examiner marking.Recommendations included increasing checklist detail and using a weighted marking scale, separating some stations into dichotomous and key-feature checklists, using GRSs specific to each station, and reviewing all future OSCEs with the metrics described to guide refinements. The analysis used revealed several potential issues with the OSCE assessment. These findings informed recommendations to improve the quality of future examinations.