A Comparison of Rubrics and Graded Category Rating Scales with Various Methods Regarding Raters’ Reliability

Abstract

The measurement of psychological attributes can be performed in two ways. One is to give examinees a number of stimuli (questions) assumed to stimulate the psychological attribute in focus; in this case, examinees respond to predetermined standard response categories depending on the magnitude of the attribute in themselves. This method, called direct self-report, simply covers tests in which examinees choose the correct answer from options available to them. In indirect selfreport, examinees are similarly provided with a stimulus, but they are not offered standard response categories - they can create responses themselves (Erkuc, 2012). Examples of indirect self-report include open-ended questions and performance tasks which aim at measuring skills such as writing, critical thinking and creativity, where responses are structured by examinees themselves.In tests which require examinees to select the correct answer, scoring can be carried out by scoring a correct answer as 1 and an incorrect answer as 0. In other words, there is no degree of correct answers. This is a condition that increases objectivity and reliability in scoring. However, as regards tests in which responses are constructed by examinees themselves, answers close to the correct answer are likely, as well as the correct answer. This situation adversely affects objectivity and reliability in scoring. In order to overcome the adverse effect, graded-category rating scales and rubrics are widely used for the scoring of such tools (Haladyna, 1997; Jonsson & Svingby, 2007; Rezaei & Lovorn, 2010).Graded-Category Rating ScalesGraded-Category Rating Scales (GCRSs) are scoring tools containing the criteria regarding properties intended to be measured in student work along with success levels regarding these criteria. GCRSs offer dimensions for scoring and the range of scoring regarding student work (Haladyna, 1997). In this way, differentiation of the scores assigned by the raters can be prevented to a certain degree. However, since performance levels of success pertaining to the criterion are not defined, success measured as three points by one rater may be measured as two points by another. This situation can be considered as a major weakness of GCRSs. Conversely, their biggest advantage is their ease of preparation and quick scoring (Haladyna, 1997).RubricsRubrics are scoring tools used to define the criteria against which student work is evaluated, and the level to which their performance corresponds (Goodrich, 1996). The most distinctive aspect of rubrics in comparison to GCRSs is that they provide performance definition of each criterion for different levels of success. In other words, rubrics contain separate definitions made for each performance level regarding the criteria. In this context, rubrics have the advantage of giving feedback for students while enhancing objectivity in scoring for teachers. This ultimately contributes to a more standardized and objective determination, not varying from rater to rater.Rubrics can be prepared for a certain scope (task-specific rubric). They can also be developed in order to score general skills such as writing and communication (generic rubric). In addition to these, there are two types of rubrics based on the structure of the tool (Haladyna, 1997). The first of these is the holistic rubric. In holistic rubrics, one single point is given to the student's entire performance and descriptions are available for all performance levels. Such rubrics are used in situations where small mistakes in student performance can be ignored and focus is placed on the whole performance (Arter & McTighe, 2001; Kutlu, Dogan, & Karakaya, 2010). The other type is the analytical rubric. Used more widely, the analytical rubric is a scoring tool that provides information about the achievement levels of student performance in various dimensions. Thus, it can provide a profile of the strengths and weaknesses of students in a certain area (Gronlund, 1998). …