Efficient measurement of spoken English skills: Design and validation of a 6-minute assessment

Abstract

Efficient measurement of spoken language skills can improve educational and commercial services. Traditional spoken language assessments often require 30–90 minutes. We report the development and validation of a prototype mobile-browser test of spoken English, “APro”, that participants complete in 6–8 minutes. The APro test presents a stratified random draw of 20 interactive tasks (from pool of 377) to participants whose spoken responses are automatically scored, scaled, and reported. APro aggregates skill-specific measures from disparate task types such as open descriptions and story retellings. Each skill-specific score was independently validated, then combined into an overall spoken English score. APro scoring was trained on 30,000 responses from 354 non-native pilot-test participants who connected remotely from their nine home countries on five continents. The final APro instrument was validated on an independent sample of 79 participants resident in seven countries. In the validation set, human and machine scoring of same-test APro administrations correlated r = 0.98, suggesting accurate speech processing. APro scores predicted much of the score variance in traditional concurrent assessments from Oxford, r = 0.86, and Pearson, r = 0.92. Finally, we present detailed results on two new interactive tasks and share changes to the content, procedure, and scoring that yield improved accuracy at shorter duration. [Work supported in part by Google LLC.]