Differential rater function over time (DRIFT) during student simulations

Abstract

Background: The field of paramedicine continues to advance in scope. Simulation training is frequently used to teach and evaluate students. Simulation examinations are often evaluated using a standardised global rating scale (GRS) that is reliable and valid. However, differential rater function over time (DRIFT) has not been evaluated when using the GRS during simulations. Aims: This study aimed to assess if DRIFT arises when applying the GRS. Methods: Data were collected at six simulation evaluations. Raters were randomly assigned to evaluate several students at the same station. Each station lasted 12 minutes and there was a total of 11 stations. A model to test DRIFT scores was created and was tested against both a leniency and perceptual model. Findings: Of the models explored, one that included students, the rater, and the dimensions had the greatest evidence (−3151 Bayes factors). This model was then tested against leniency (K=−9.1 dHart) and perceptual models (K=−7.1 dHart). This suggests a substantial finding against DRIFT; however, the tested models used a wide parameter so the possibility of a minor effect is not fully excluded. Conclusion: DRIFT was not found; however, further studies with multiple centres and longer evaluations should be conducted.