Virtual reality informs clinical observation tool.

Abstract

Assessing trainees' skills via workplace-based assessments is challenging given the lack of psychometrically valid instruments. Our team previously developed an observation instrument to assess residents' competencies in behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI) though its reliable usage required expert raters (e.g., paediatric psychologists). Our purpose was to establish validity evidence for a modified tool for clinicians. This study utilised data from a prior educational trial of a virtual reality (VR)-based behavioural health curriculum for paediatric residents. First, group interviews were conducted with clinicians to modify the expert instrument. Next, recorded VR simulations (n = 10) were scored to assess interrater reliability between clinicians. Finally, a physician used the tool to assess its ability to discriminate between residents' skill levels (n = 55). Modifications during group interviews included a change from frequency counts for items related to MI to the binary outcome of present/absent and the addition of an entrustment item. On interrater-reliability testing, the clinician tool demonstrated mostly substantial or near perfect agreement for items related to BHAG. MI items demonstrated a range of agreement. Using 55 recorded VR simulations, the clinician tool discriminated between trained versus untrained residents in BHAG skills (p = 0.002) and level of entrustment (p = 0.001). Differences between groups in MI adherence was not statistically significant (p = 0.095) as it was on the expert instrument. VR demonstrated potential as a novel approach for obtaining validity evidence for tool development to support workplace-based assessments. Further work assessing usage of the clinician tool in real-world settings is warranted.