Abstract
This study investigates errors in a sample of 50 written performances of Austrian learners of English collected in the 2009 baseline study for the Austrian Educational Standards-Based Writing Test for English at grade 8 (E8 Standards Writing Test). The research aims to contribute to the validation of this large-scale assessment by studying the relationship between errors (described using the Scope – Substance error taxonomy) and human ratings awarded to writing performances. The results add to the validity evidence of the E8 Standards Writing Test. There is a negative relationship between human ratings and the presence of errors; a low error density is associated with higher ratings and a high error density with lower ratings. Substance word, clause, and text error densities play an important role in the rating in most dimensions; errors with a larger scope also have a strong effect. By highlighting aspects of errors to which raters seem to be sensitive, these findings constitute evidence of context validity. At the same time, the findings are relevant to theory-based validity by concretising areas of competence that learners need to develop in order to receive higher ratings. While errors are important determinants of the ratings, additional factors, presumably positive features, must be at play as the accuracy of the regression models is low to moderate. This should in fact be the case since the E8 rating scale refers to negative as well as positive features.
Highlights
Analytic rating scales for writing, and the rating scale used in the Austrian Educational Standards-Based Writing Test for English at grade 8 (E8 Standards Writing Test) in particular, describe levels of performance on each of several dimensions by means of so-called performance level descriptors (e.g. Cizek and Bunch 2007, 46)
This research aims to contribute to the validation of this test by investigating the role that errors play in the rating of writing performances using an analytic rating scale
The study identifies the most common errors in the writing of Austrian learners of English at the age of around 14 years and, in particular, focuses on the relationship between errors and human ratings awarded to writing performances generated in the context of the E8 Standards Writing Test
Summary
Analytic rating scales for writing, and the rating scale used in the Austrian Educational Standards-Based Writing Test for English at grade 8 (E8 Standards Writing Test) in particular, describe levels of performance on each of several dimensions by means of so-called performance level descriptors (e.g. Cizek and Bunch 2007, 46). Cizek and Bunch 2007, 46) These are supposed to guide trained raters in the process of judging the writing performances with the aim of maximising intra- and interrater agreement, which, in turn, is considered as an indication of validity. This study aims to shed light on this aspect of the rating process in the E8 Standards Writing Test. This research aims to contribute to the validation of this test by investigating the role that errors play in the rating of writing performances using an analytic rating scale. The study identifies the most common errors in the writing of Austrian learners of English at the age of around 14 years and, in particular, focuses on the relationship between errors and human ratings awarded to writing performances generated in the context of the E8 Standards Writing Test. The errors in the writing performances are identified and categorised by means of the Scope – Substance error taxonomy (Dobrić and Sigott 2014)
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
