CAF indices and human ratings of oral performances in an opinion-based monologue task

Chie Ogawa

doi:10.1186/s40468-022-00154-9

Abstract

This study explored two assessment approaches to oral performances: analytical complexity, accuracy, and fluency (CAF) indices and human raters’ evaluations. CAF indices are frequently used in second-language speaking (L2) research; however, because tasks are communicative and goal-oriented, the degree to which students achieve such communicative goals must also be included. By incorporating human ratings of monologue organization and perceived CAF into speaking assessments, researchers can better understand the relationship between the analytical CAF indices and human ratings of a monologue task. The participants consisted of 48 English as a Foreign Language (EFL) students in a Japanese university. Their oral performances of 2-min opinion-based monologues were audio-recorded and then transcribed and analyzed using CAF measures. In addition, 11 human raters evaluated the same recordings in terms of the following criteria: topic organization, complexity, accuracy, and fluency. These ratings were then analyzed using the many-facet Rasch measurement (MFRM). Multiple linear regression results showed that fluency accounted for a significant amount of the human ratings, but other measures (lexis, complexity, accuracy) explained only a small portion of the variance. This study concluded with implications regarding L2 testing in speaking assessments.

Highlights

Second-language acquisition (SLA) scholars have been increasingly focusing on speaking performance assessment
Raters’ assessment of oral performances To address research question 1 (How do analytic rating scales based on organization and CAF evaluate opinion-based monologue tasks?), the manyfacet Rasch measurement (MFRM) provided insights into how human raters perceived Japanese university students’ oral performances in opinion-based tasks
The results showed that analytical fluency measures accounted for a significant amount of human rater evaluation (53% of variance), but the other analytical measures explained only a small portion of the variances (1.5–3.4%)

Summary

Introduction

Second-language acquisition (SLA) scholars have been increasingly focusing on speaking performance assessment. ?” Oral performances are assessed primarily to evaluate the extent of a test-taker’s ability to successfully convey meaning through speech or whether their speaking ability meets the minimum (2022) 12:4 requirements of the proficiency level. The following section discusses two methods of evaluating second-language (L2) oral performances: measuring recorded data using complexity, accuracy, and fluency (CAF) and human raters’ oral performance. Complexity, accuracy, and fluency indices CAF indices have been routinely used as indicators of learners’ oral proficiency and language acquisition (Housen et al, 2012). Most task-based studies have employed analytical measures for objective evaluations of transcribed speech data using CAF indices, because L2 performance has multiple components as conveyed by the concepts of complexity, accuracy, and fluency

Methods

Results

Discussion

Conclusion