Inter-Rater Reliability of Web-Based Calibrated Peer Review within a Pharmacy Curriculum

Abstract

Objective. To assess the inter-rater reliability of student evaluations submitted through a web-based calibrated peer review system versus faculty evaluations in a Doctor of Pharmacy (PharmD) curriculum. Methods. Two calibrated peer review assignments were integrated into two sequential pharmacotherapy-based courses in the first professional year of the PharmD curriculum. To assess the inter-rater reliability of student peer and self-evaluations compared to faculty evaluations, 50 student submissions from each assignment were randomly selected for faculty evaluation, using the same evaluation criteria used by students. Statistical analysis was performed to assess the difference between the scores given by students versus faculty members and the correlation between the combined and individual assignment scores given by students versus faculty members. Results. Mean faculty scores did not significantly differ from student peer or self-evaluation scores. When the assignments were combined, there was a significant fair correlation between scores assigned by peers and those assigned by faculty members, and between scores assigned by students (self-evaluation) and faculty members. For individual assignments, one showed significant positive correlations for both peer and self-assessment compared to faculty, while the other assignment did not. Conclusion. Incorporating longitudinal calibrated peer review activities into the PharmD curriculum demonstrated fair inter-rater reliability between scores assigned by pharmacy student and faculty members. Further research on calibrated peer review is needed in pharmacy education to evaluate its reliability and utility. However, the current study suggests that the use of a web-based calibrated peer review system within a pharmacy curriculum could provide additional opportunities for critical evaluation and student self-assessment, as well as encourage more open-ended communication activities without increasing the grading burden on faculty members.