Assessing conversation quality, reasoning, and problem solving with computer agents

Abstract

Conversational agents have been developed to help students learn in computerlearning environments with collaborative reasoning and problem solving. Conversational agents were used in the 2015 Programme for International Student Assessment (PISA) collaborative problem-solving assessments, where a human interacted with one, two or three agents. This chapter reviews advances in conversational agents and how they can help students learn by engaging in collaborative reasoning and problem solving. Using the example of AutoTutor it demonstrates how dialogues can mimic the approaches of expert human tutors. Conversations with intelligent systems are quite different depending on the number of agents involved. A human interacting with only one computer agent during a dialogue needs to continuously participate in the exchange. In trialogues there are two agents, so there are more options available to the human (including social loafing and vicarious observation) and the conversation patterns can be more complex, illustrated by Operation ARIES!, which uses a number of patterns in teaching students the basics of research methodology. Conversational agent systems use online, continuous, formative assessment of human abilities, achievements, and psychological states, tracked during the course of the conversations. Some of these formative assessment approaches are incorporated in the PISA 2015 assessment of collaborative problem solving. However, this chapter focuses on formative assessment in learning environments rather than on summative assessments.