Reliability of a modified single-point rubric for assessment of integrated pharmacotherapy cases

Abstract

Background: The Pharmacists’ Patient Care Process (PPCP) offers a method to integrate pharmacotherapy cases between disciplines. However, establishing a consistent grading standard across multiple faculty and disciplines creates a challenge. The purpose of this study was to describe the reliability of a single-point rubric for assessing integrated cases among interdisciplinary graders. Methods: A single-point rubric was developed to provide students with formative and summative feedback on existing integrated cases requiring written PPCP notes. The rubric was calibrated retrospectively on a sample of deidentified cases from two integrated pharmacotherapy courses. Following calibration, 20 submissions were evaluated by an interdisciplinary team of faculty. The following year, four new evaluators graded 12 submissions to ensure continued reliability. Results: Fleiss' kappa was used to determine if there was an agreement between instructors’ judgement on the final grade classification of the original 20 submissions. There was almost perfect agreement between the instructors’ judgements, κ= .868. Each section of the PPCP and overall numerical score all showed significant agreement using Kendall’s W. Results were similar the following year with four new graders. Conclusion: This study demonstrates the continued reliability of a modified single-point rubric to evaluate written cases involving multiple disciplines.