Abstract
This study examined the rater severity of instructors using a multi-trait rubric in a freshman composition course offered in a private university in Kuwait. Use of standardized multi-trait rubrics is a recent development in this course and student feedback and anchor papers provided by instructors for each essay exam necessitated the assessment of rater effects, including severity/leniency and restriction of range in ratings among instructors. Data were collected from three instructors teaching the same course in Summer 2019, who rated the first midterm exam essays of their students and shared the scores with the researcher. Also, two students from each class were randomly selected and a total of six papers were marked by all instructors for anchoring purposes. Many-facet Rasch model (MFRM) was employed for data analysis. The results showed that although the raters used the rubric consistently during scoring across all examinees and tasks, they differed in their degree of leniency and severity, and tended to assign scores of 70 and 80 more frequently than the other scores. The study shows that composition instructors may differ in their rating behavior and this may cause dissatisfaction, creating a sense of unfairness among the students of severe instructors. The findings of this study are expected to help writing departments to monitor their inter-rater reliability and consistency in their ratings. The most practical way to achieve this is by organizing rater training workshops.
Highlights
National Council of Teachers of English (NCTE) proposes that writing is a complex skill learned over a long period of time, through a wide range of assignments, and with copious and significant feedback (Anson, Filkins, Hicks, O'Neill, Pierce, & Winn, 2013)
Direct writing assessment is challenged, because unlike the straightforward multiple-choice assessment, the assessment of student writing, in English as a second language (ESL) classes, is a challenging task for writing instructors (Huang & Foote, 2010), and there is plenty of evidence that raters from different backgrounds seem to weigh assessment criteria quite differently when they are scoring their students’ essays (Barkaoui, 2010)
The aim of this paper is to examine the rating behavior of instructors while they were using multi-trait scoring rubrics in a first-year composition course (ENG 100) in a private university in Kuwait
Summary
National Council of Teachers of English (NCTE) proposes that writing is a complex skill learned over a long period of time, through a wide range of assignments, and with copious and significant feedback (Anson, Filkins, Hicks, O'Neill, Pierce, & Winn, 2013). Students must gain this complex skill in order to meet the requirements of higher education, demands of a twenty-first-century workforce, and the realization of meaningful lives. Direct writing assessment is challenged, because unlike the straightforward multiple-choice assessment, the assessment of student writing, in English as a second language (ESL) classes, is a challenging task for writing instructors (Huang & Foote, 2010), and there is plenty of evidence that raters from different backgrounds seem to weigh assessment criteria quite differently when they are scoring their students’ essays (Barkaoui, 2010)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
