Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia.

Lauren Chong,Matthew Haywood,Boaz Shulruf,Barbara-Ann Adelstein,Silas Taylor

doi:10.3352/jeehp.2018.15.17

Abstract

PurposeThe biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established.MethodsWe compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance.ResultsFive hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores.ConclusionWe suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.

Highlights

The sources of bias that may influence objective structured clinical examination (OSCE) scores are well understood, and include the halo [1], ‘hawk-dove’ [2], contrast [3], and site [4] effects, in addition to the examiner’s familiarity with the students [1] and the duration of the examining period [1]
Criteria and data Our OSCE focused on 3 domains graded across 9 criteria: generic communication skills (4 items); clinical communication skills; and physical examination skills (2 items)
Physical examination domain marking We propose that examination skills may be less susceptible to examiner bias because the process of patient physical examination is well-documented, fairly prescriptive, and widely accepted [8]

Summary

Introduction

The sources of bias that may influence objective structured clinical examination (OSCE) scores are well understood, and include the halo [1], ‘hawk-dove’ [2], contrast [3], and site [4] effects, in addition to the examiner’s familiarity with the students [1] and the duration of the examining period [1]. It can be argued that ‘background’ and ‘experience’ are conflated in some studies, in much the same way that medical student examiners marking more generously than both ‘teaching doctors’ or ‘senior academic(s)’ may be explained in terms of their clinical and examining inexperience [7]. These 2 variables are not necessarily correlated, and the general failure in the literature to accurately distinguish among examiner characteristics with respect to experience, occupation, and rank/seniority makes it almost impossible to draw any inferences regarding the relative importance of these variables

Methods

Results

Conclusion