The influence of rater empathy, age and experience on writing performance assessment

Pilvi Alp,Hille Pajupuu,Anu Epner

doi:10.31743/lingbaw.5647

Pilvi Alp, Hille Pajupuu + Show 1 more

Open Access

https://doi.org/10.31743/lingbaw.5647

Copy DOI

Abstract

Assessment reliability is vital in language testing. We have studied the influence of empathy, age and experience on the assessment of the writing component in Estonian Language proficiency examinations at levels A2–C1, and the effect of the rater properties on rater performance at different language levels. The study included 5,270 examination papers, each assessed by two raters. Raters were aged 34–73 and had a rating experience of 3–15 years. The empathy level (EQ) of all 26 A2–C1 raters had previously been measured by Baron-Cohen and Wheelwright’s self-report questionnaire. The results of the correlation analysis indicated that in case of regular training (and with three or more years of experience), the rater’s level of empathy, age and experience did not have a significant effect on the score.

Highlights

Assessment reliability is vital in language testing
The Estonian language proficiency examinations are high-stakes tests. Success in these tests provides an opportunity to apply for Estonian citizenship and enhances competitiveness in the labour market as the language proficiency examinations are to be passed for proving the level of language proficiency required for employment in certain positions
In order to assess the empathy level of the raters, we used the Baron-Cohen and Wheelwright’s (2004) self-assessment questionnaire Empathy Quotient (EQ), which was translated into Estonian

Summary

Introduction

We have studied the influence of empathy, age and experience on the assessment of the writing component in Estonian Language proficiency examinations at levels A2–C1, and the effect of the rater properties on rater performance at different language levels. The results of the correlation analysis indicated that in case of regular training (and with three or more years of experience), the rater’s level of empathy, age and experience did not have a significant effect on the score. Language proficiency tests are standardised tests which include the four skills: reading and listening comprehension, writing and speaking. Raters use holistic rating scales which rely on the criteria of the specific language proficiency levels. Raters need to be consistent in order to be reliable, that is, the scores given by different raters for the same performance coincide or differ only minimally (inter-rater reliability) (e.g., Alderson, Clapham, & Wall, 1996; Luoma, 2004; Weir, 2005)

Objectives

Methods

Results

Conclusion