The Validity of Standardized Patient Assessment Using Faculty-Physician Global Ratings as the Gold-Standard Criterion

Abstract

The purpose of the study was to provide a direct evaluation of the validity of standardized patient (SP) assessment, using global ratings by faculty physicians as the gold-standard criterion. Five faculty physicians independently observed and rated video-taped performances of 44 medical students on the seven SP cases which comprise the fourth-year assessment administered at The Morchand Center of Mount Sinai School of Medicine to students in the eight member schools in the New York City Consortium. A more fundamental purpose of the study was to determine the reliability of the gold-standard ratings, by examining the agreement among the ratings assigned by the five different faculty-physician raters and by evaluating the reliability of the mean ratings of the panel of raters. The validity results are encouraging, with validity coefficients showing correlations between full examination scores and the gold-standard ratings in the 60s. At the least, the validity coefficients are high enough to warrant optimism about the possibility of increasing them to the commonly recommended .80 level, based on further studies to identify those items that best reflect the gold-standard ratings to be used to develop and refine the examination. But most encouraging, the reliabilities of the ratings achieved the recommended .80 level for the full examination ratings and were very close at the case level with inter-rater reliabilities in the 70s. These high interrater reliabilities indicate that faculty-physician ratings of performance on SP cases and examinations can provide a reliable gold-standard criterion for validating and possibly improving SP assessment, despite the common perception to the contrary. The study represents an important new direction for validation and refinement of SP assessment.