A Large-Scale Study of the Reliabilities of Checklist Scores and Ratings of Components of Interpersonal and Communication Skills Evaluated on a SP Examination

Abstract

Checklists and rating scales are both used for assessing examinees in standardized patient (SP) examinations. The common presumption, though, is that checklists are more objective than and hence superior to rating scales with respect to such psychometric properties as reliability. Two related articles, however, have warned that “objectified methods do not inherently provide more reliable scores,” and our recent study of interpersonal and communication skills supports this claim with checklist and rating-scale reliabilities of. 65 and. 76, respectively, and an intercorrelation of. 82. The present study extended this research to different aspects or components of interpersonal and communication skills, using a 26-item checklist consisting of five sections of 3 to 7 items each, with each section representing a different component of interpersonal and communication skills. At the end of each section, the SP was asked to provide a global rating of the component indicated by the items in that section. Analyses were performed on data for 1,048 fourth-year medical students at the eight-school New York City Consortium, tested on the 7-case SP examination administered by The Morchand Center at Mount Sinai School of Medicine. The intercase reliabilities for the checklist scores and ratings, respectively, were:. 33 and. 39 for eliciting information,. 33 and. 39 for nonverbal behaviour,. 55 and. 52 for patient education,. 48 and. 45 for professional manner, and. 49 and. 52 for patient satisfaction. The correlations between the checklist scores and ratings for the five components were. 66,. 60,. 80,. 69, and. 75, respectively. The pattern of these results provides further support for the claim that “objectified” measures, like check lists, “do not inherently provide more reliable scores.”