Examining the validity of an analytic rating scale for a Spanish test for academic purposes using the argument-based approach to validation

Abstract

Abstract Rating scales are used to assess the performance of examinees presented with open-ended tasks. Drawing on an argument-based approach to validation, this study reports on the development of an analytic rating scale designed for a Spanish test for academic purposes. The study is one of the first that sets out the detailed scale development and validation activities for a rating scale for Spanish as a second language. The rating scale was grounded in a communicative competence model and developed and validated over two phases. The first version was trialed by five raters, and its quality was analyzed by means of many-facet Rasch measurement. Based on the raters’ experience and on the statistical results, the rating scale was modified and a second version was trialed by six raters. After the rating process, raters were sent an online questionnaire in order to collect their opinions and perceptions of the rating scale, the training and the feedback provided during the rating process. The results suggest the rating scale was of good quality and raters’ comments were generally positive, although they mentioned that more samples and training were needed. The study has implications for rating scale development and validation for languages other than English.