Development and Effectiveness of a Rater Training Curriculum for Evaluating Student Medical Spanish Oral Proficiency Using the Physician Oral Language Observation Matrix

Abstract

To improve language-concordant, equitable healthcare for Spanish speakers, many United States medical schools offer medical Spanish education. However, there is no clinically contextualized, standardized approach to medical student language assessment. This article reports on the development and effectiveness of a training curriculum to prepare raters to use a new rubric, the Physician Oral Language Observation Matrix (POLOM), to reliably rate medical student Spanish oral proficiency after observing videorecorded encounters between students and standardized patients. Curriculum effectiveness was primarily evaluated by examining each rater trainee’s level of agreement with expert consensus POLOM ratings (i.e., inter-rater reliability as measured by the generalizability theory dependability coefficient). Out of the study’s nine rater trainees, who were from either medical or linguistic professional backgrounds, five proceeded to the calibration phase, and four attained the reliability threshold required for calibration. The paper concludes that this rater training curriculum can successfully train raters to use the POLOM reliably when evaluating medical student Spanish oral proficiency during videorecorded healthcare encounters. To allow for improved assessment of student language skills prior to use in patient care, future efforts should focus on POLOM validity assessment and larger scale rater recruitment, training, calibration, and maintenance.