Differential rater function over time (DRIFT) during student simulations

Abstract

Background The field of paramedicine continues to advance in scope. Simulation training is frequently used to teach and evaluate students. Simulation examinations are often evaluated using a standardised global rating scale (GRS) that is reliable and valid. However, differential rater function over time (DRIFT) has not been evaluated when using the GRS during simulations. Aims This study aimed to assess if DRIFT arises when applying the GRS. Methods Data were collected at six simulation evaluations. Raters were randomly assigned to evaluate several students at the same station. Each station lasted 12 minutes and there was a total of 11 stations. A model to test DRIFT scores was created and was tested against both a leniency and perceptual model. Findings Of the models explored, one that included students, the rater, and the dimensions had the greatest evidence (–3151 Bayes factors). This model was then tested against leniency (K=–9.1 dHart) and perceptual models (K=–7.1 dHart). This suggests a substantial finding against DRIFT; however, the tested models used a wide parameter so the possibility of a minor effect is not fully excluded. Conclusion DRIFT was not found; however, further studies with multiple centres and longer evaluations should be conducted.