Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students

İsmail Karakaya,Mehmet Şata

doi:10.21031/epod.842094

Abstract

This study aimed to examine the effect of rater training on the differential rater function (rater error) in the process of assessing the academic writing skills of higher education students. The study was conducted with a pre-test and post-test control group quasi-experimental design. The study group of the research consisted of 45 raters, of whom 22 came from experimental, and 23 came from control groups. The raters were pre-service teachers who did not participate in any rater training before, and it was investigated that they had similar experiences in assessment. The data were collected using an analytical rubric developed by the researchers and an opinion-based writing task prepared by the International English Language Testing System (IELTS). Within the scope of the research, the compositions of 39 students that were written in a foreign language (English) were assessed. Many Facet Rasch Model was used for the analysis of the data, and this analysis was conducted under the Fully Crossed Design. The findings of the study revealed that the given rater training was effective on differential rater function, and suggestions based on these results were presented.

Highlights

Academic writing is defined as a type of text in which thoughts are logically structured and justified (Bayat, 2014)
T statistics are used for interactions that are significant in interaction between sources of variability in Many Facet Rasch Model (MFRM)
When the t statistic takes a negative value, it is defined as differential rater severity; when it takes a positive value, it refers to differential rater leniency

Summary

Introduction

Academic writing is defined as a type of text in which thoughts are logically structured and justified (Bayat, 2014) According to another definition, academic writing is defined as explaining the individual's views, ideas, feelings, observations, experiments, and experiences based on his/her world of thought, congruent with the rules of the language by planning them in accordance with the individual’s interest towards the chosen subject (Göçer, 2010). Academic writing is defined as explaining the individual's views, ideas, feelings, observations, experiments, and experiences based on his/her world of thought, congruent with the rules of the language by planning them in accordance with the individual’s interest towards the chosen subject (Göçer, 2010) It can be seen from these definitions that academic writing requires many skills, and it has a complex process. In this context, when academic writing is considered as the realization of higher-level mental skills, it is important to assess academic writing validly and reliably (Carter, Bishop & Kravits, 2002)

Objectives

Methods

Results

Conclusion