Exploring the Potential of a Spoken Dialog System-Delivered Paired Discussion Task for Assessing Interactional Competence

Abstract

ABSTRACT While several studies support the appropriateness of tasks delivered through Spoken Dialog Systems (SDSs) for L2 oral communication assessment, few studies have explored the construct of interactional competence (IC) as measured by such tasks. This study aims to address this gap in the literature by investigating the use of IC features by high- and low-ability test-takers in performance on an SDS-delivered paired discussion task, along with the perceived appropriateness of the SDS as a conversation partner. Data collected from 30 test-takers and three trained raters included test-taker task responses, task scores, and semi-structured interviews. The analysis of the task responses for IC features based on a coding scheme adapted from Galaczi (2014) showed that the majority of the features included in the construct definition in the study were consistently observable in the task responses, and the use of observed features generally varied with score levels as expected. The qualitative analysis of the interviews revealed that test-takers generally perceived the system as an appropriate partner for oral communication assessment, although they noted several technical limitations that could affect their task performance. This study helps us better understand the construct of IC underlying SDS-delivered tasks and offers implications for designing and validating these tasks.