Comparing The Reliability Of Two Peer Evaluation Instruments

Abstract

This paper presents an analysis of student peer evaluations in project teams to compare the reliability of two different evaluation procedures. The project teams consist of junior-level students in a mechanical engineering design course taught by Layton for five semesters in 1997, 1998, and 1999. The peer-evaluation instruments were used by students to evaluate their teammates’ contributions to the team’s deliverables—oral and written presentations of their solution to a technical design problem. The first instrument is an adaptation of the one advocated by Brown, in which students use a prescribed list of terms such as “excellent,” “very good,” “satisfactory,” and so forth. The second form, by Layton, asked students to assign a numerical rating (from 0 to 5) to 10 different aspects of contribution to the team. Analysis of variance was used to study the reliability of each of the instruments, using a special form by Crocker and Algina to study inter-rater reliability. The similarity of the reliability coefficients of the two instruments (ρi=0.34 for Brown’s instrument and ρi=0.41 for Layton’s instrument) strengthens the assumption made in the first study—that data from the two instruments are similar enough to be normalized for comparison. At the same time, the higher reliability of Layton’s instrument lends credence to Layton and Ohland’s conclusion that focusing on identified behavioral characteristics of good teamwork (as Layton’s instrument does) can improve peer evaluation. Layton’s instrument accomplishes this to an extent, yielding a modest improvement in reliability. More focused attempts to define teamwork success behaviorally, such as the modification of Brown’s instrument by Kaufman et al., may yield further improvements in reliability. The overall reliability of the two instruments validates such instruments as repeated measures of a consistent trait.