Examining the Differential Rater Functioning in the Process of Assessing Writing Skills of Middle School 7th Grade Students

Aslıhan Erman Aslanoğlu,Mehmet Şata

doi:10.17275/per.21.88.8.4

Aslıhan Erman Aslanoğlu, Mehmet Şata

Open Access

https://doi.org/10.17275/per.21.88.8.4

Copy DOI

Abstract

When students present writing tasks that require higher order thinking skills to work, one of the most important problems is scoring these writing tasks objectively. The fact that raters give scores below or above their performance based on several environmental factors affects the consistency of the measurements. Inconsistencies in scoring negatively affect the validity and reliability of student performance and cause the scores obtained to be questioned. In regard to the validity and reliability of these measurements, it is significant to identify the rater behavior and correct the sources of error. This study aims to analyze the differential rater functioning (DRF), which is one of the problematic rater behaviors, in evaluating compositions written by middle school 7th-grade students within the scope of the Turkish course. 86 students attending a public school were participated the study. Students' compositions were rated using an analytical rubric by 8 teachers from different institutions. In this correlational research, the many facet Rasch model was used, and five variables including students, raters’ and, students’ gender, students’ qualification, and evaluation criteria were examined. it was examined whether the raters show DRF on an individual and group basis based on the dual interaction analysis, including the gender of the student x rater and the student's competence x rater. The findings have revealed that DRF at the group level does not interfere with the measurements, while the individual level DRF is involved in the measurements. It was determined that the level of DRF mixing in the measurements of successful students was the lowest. Especially rigid and lenient raters were found to show DRF. In the present study, it was observed that the raters showing DRF was also the most lenient raters, while these raters did not show DRF in terms of the gender of the student.

Highlights

Writing can be defined as writing down of the information that individuals organize in their minds via putting this organized information on paper (Englert & Mariage, 2003)
This study aims to investigate the effect of the differential rater function, which is one of the rater errors involved in the measurements as a result of the interactions between rater x student qualities in the process of evaluating the academic writing skills of middle school students
In the study, it was aimed to reveal the interference level of the differential rater function, which is one of the rater errors in the process of evaluating the academic writing skills of students in primary education. previous literature suggests in relation to this that Differential Rater Function (DRF) is mostly investigated in the process of evaluating academic writing skills in the second language (Eckes, 2005; Engelhard, & Myford, 2003; Farrokhi, Esfandiari, & Schaefer, 2012; Johnson and Lim, 2009; Kondo-Brown, 2002)

Summary

Introduction

Writing can be defined as writing down of the information that individuals organize in their minds via putting this organized information on paper (Englert & Mariage, 2003). .Erman Aslanoğlu, M.Şata knowledge, feelings, and thoughts In this respect, writing is an important tool for individuals to express themselves. Researchers state that in the teaching process, effective use of writing better enables students to express the newly learned information and their thoughts, which improves their written communication skills as well as their academic success in other courses (Erhardt & Meade, 2005; Lam, Au, Leung, & Li Tsang., 2011). Students are expected to perform the processes of designing, organizing, drafting, editing, and editing effectively while they are writing (Englert, 1991). In this respect, the thinking method that students use while creating a written product is the cognitive aspect of writing, and the method of checking when producing a written product are the metacognitive aspects of writing (Collins, 2000). Writing is a process that requires individuals to use higher order thinking skills defined as associating these skills with individual characteristics by using more than one skill simultaneously (Marzono, 2001)

Objectives

Methods

Results

Conclusion