Assessment of Competency-Based Behavioral Health Anticipatory Guidance Skills Among Pediatric Residents: the Role of Virtual Reality

Abstract

Assessing pediatric residents’ behavioral health anticipatory guidance (BHAG) competencies in outpatient settings is challenging given limited opportunities to directly observe physicians’ skills and the lack of accepted psychometrically sound tools. Virtual reality (VR) provides a strategy to design scenarios based on specific clinical occurrences to observe physicians’ competencies, aiding in the development of direct observation assessment instruments. This study was conducted at a large, urban pediatric primary care center. A VR curriculum was created to train pediatric residents on evidence-based BHAG competencies and motivational interviewing (MI) skills. The curriculum included three VR simulations, of increasing complexity, focused on caregiver concerns about tantrums. Residents verbally counseled graphical caregiver representatives (avatars) whose verbal and non-verbal responses were driven by a facilitator in real-time. We used Kane’s model to establish validity evidence for a direct observation instrument that included an assessment of interrater reliability by pediatric psychologists. A total of 42 VR observations were performed in 14 unique final year pediatric residents in May 2020. Interrater reliability was strong (kappa = 0.94). Resident’s scores on the instrument improved with progression through the curriculum that included deliberate practice with feedback. A range of scores were observed in each scenario. VR may provide an innovative approach to collecting validity evidence on a direct observation assessment instrument when opportunities for clinical observations are limited or when a competency is complex. VR’s ability to standardize simulated encounters for learners provides advantages to instrument development.