The development and refinement of a rating scale for beginner students’ foreign language speaking skills

Abstract

Interactional oral language proficiency is a core component of modern foreign language education with ‘speaking fluently’ a central learning objective (Eisenmann & Summer, 2012). In spite of its importance, speaking is a difficult skill to assess (Tajeddin et al., 2011; Yan, 2014), and there is a gap in literature regarding the reliable and valid assessment of the very basic level of speaking skills. This paper reports on the development process of an assessment instrument, which was conducted as action research and involved both quantitative and qualitative data collection and an analysis. The learning environment and its related technology-enhanced out-of-class practice environment within which the research was conducted, focus on beginner foreign language students. Activities include computer-mediated communication and face-to-face oral activities. In order to address the need for a valid and reliable assessment instrument, a first version of the instrument was created and subsequently used to assess both computer-mediated communication and face-to-face oral activities. The reliability of the instrument was investigated during two action research cycles by means of studying the consistency, consensus estimates, and intra-rater reliability. The results from the two cycles of investigation informed changes to the instrument, and this ultimately resulted in two assessment instruments that differentiate between technology-enhanced activities and personal interaction. Similar to Gruhn and Weideman (2017), this study was of an exploratory nature and additional design principles would have to be evaluated over a longer period of time.