Fine-tuning the standard setting of objective structured practical examinations in clinical anatomy.

Abstract

The objective structured practical examination (OSPE) is a reliable assessment of practical skills in anatomy teaching. It is often administered as low-stake assessments to track progress at multiple time points in anatomy curricula. Standard-setting OSPEs to derive a pass mark and to ensure assessment quality and rigor is a complex task. This study compared standard-setting outcomes of clinical anatomy OSPEs determined by traditional criterion-referenced (Ebel) and norm-referenced ("mean minus standard deviation") methods in comparison to hybrid methods which apply both criterion-referenced and norm-referenced approaches in setting examination standards. The hybrid approaches utilized included the "Cohen method" and an adaptation of the "Taylor's method," which is an improvement on the Cohen method. These diverse standard-setting methods were applied retrospectively to 16 anatomy OSPEs conducted over 4 years for first- and second-year medical students in a graduate Doctor of Medicine Program at Griffith Medical School, Australia; and the pass marks, failure rates, and variances of failure rates were compared. The application of the adaptation of Taylor's method to standard set OSPEs produced pass marks and failure rates comparable to the Ebel method, whereas the variability of failure rates was higher with the Ebel method than with the Cohen and Taylor's methods. This underscores this study's adaptation of Taylor's method as a suitable alternative to the widely accepted but resource intensive, panel-based criterion-referenced standard-setting methods such as the Ebel method, where panelists with relevant expertise are unavailable, particularly for the multiple low-stakes OSPEs in an anatomy curriculum.