The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans?

Abstract

Despite the relevance of collaborative problem solving (CPS), there are limited empirical results on the assessment of CPS. In 2015, the large-scale Programme for International Student Assessment (PISA) first assessed CPS with virtual tasks requiring participants to collaborate with computer-simulated agents (human-to-agent; H-A). The approach created dynamic CPS situations while standardizing assessment conditions across participating countries. However, H-A approaches are sometimes regarded as poor substitutes for natural collaboration, and only a few studies have identified if the collaborations with agents capture real dynamics of human interactions. To address this, we validated the original PISA 2015 CPS assessment by investigating the effects of replacing computer agents with real students in classroom tests (human-to-human; H-H). We obtained the original PISA 2015 CPS tasks from the OECD and replaced agents with real students to provide more real-life collaboration environments with less control over conversations; the H-H was less constrained than the H-A but still limited by predefined sets of possible answers from which the humans’ would make selections. The interface remained nearly identical to the original PISA 2015 CPS assessment. Students were told the types of collaboration partners, namely humans versus agents. We applied structural equation modeling and multivariate analyses of variance to a sample of 386 students to identify the dimensionality of the CPS construct and compared the effects in CPS performance accuracy and number of behavioral actions. Results indicated no significant differences between type of collaboration partner. However, students performed a larger number of actions when collaborating with a human agent.