Assessment of the validity of peer scores and peer feedback in an online peer assessment platform (Kritik)

Abstract

Background and purposeOnline peer assessment platforms aim to improve the accuracy of peer scores (numerical scores based on a rubric) and hold students accountable for peer feedback (written comments). We evaluated the validity of peer scores and peer feedback using the online platform, Kritik. Educational activity and settingThe course was a two-credit hour online infectious diseases pharmacotherapy elective consisting of twelve third-year students enrolled in a four-year doctor of pharmacy program. Students analyzed patient cases and created video presentations of their therapeutic care plan weekly. Each student scored three peers' presentations using a rubric and provided peer feedback in Kritik. The instructor independently scored the presentations. The students' presentation scores, which was the weighted average of three peers' scores, was compared to the instructor's score. Students also rated the peer feedback they received using two Likert-type scales (feedback-on-feedback [FoF] ratings). Two faculty rated 97 randomly selected peer feedback (written comments) and recorded their FoF ratings separately. Students completed an anonymous course evaluation and exit survey. FindingsThe Pearson correlation coefficient between weighted peer scores and instructor scores for 91 presentations was r = 0.880. There was fair agreement in FoF ratings between students and faculty based on weighted kappa. All students would recommend the course and had a positive experience with peer assessment and the platform. SummaryWeighted peer scores strongly correlated with instructor scores, and students held each other accountable for peer feedback in Kritik. Our findings should be confirmed in different contexts and settings.