Non-intrusive assessment of learners’ prior knowledge in dialogue-based intelligent tutoring systems

Vasile Rus,Dan Ştefănescu

doi:10.1186/s40561-016-0025-3

Abstract

This article describes a study whose goal was to assess students’ prior knowledge level with respect to a target domain based solely on characteristics of the natural language interaction between students and conversational Intelligent Tutoring Systems (ITSs). We report results on data collected from two conversational ITSs: a micro-adaptive-only ITS and a fully-adaptive (micro- and macro-adaptive) ITS. These two ITSs are in fact different versions of the state-of-the-art conversational ITS DeepTutor ( http://www.deeptutor.org ). Our models rely on both dialogue and session interaction features including time on task, student generated content features (e.g., vocabulary size or domain specific concept use), and pedagogy-related features (e.g., level of scaffolding measured as number of hints). Linear regression models were explored based on these features in order to predict students’ knowledge level, as measured with a multiple-choice pre-test, and yielded in the best cases an r=0.949 and adjusted r-square =0.833. We discuss implications of our findings for the development of future ITSs.

Highlights

Assessment is a key element in education in general and in Intelligent Tutoring Systems (ITSs; (Rus et al 2013)) in particular because fully adaptive tutoring presupposes accurate assessment (Chi et al 2001; Woolf 2008)
We focus in this article on assessing students’ prior knowledge in dialogue-based ITSs based on characteristics of the tutorial dialogue interaction between students and such systems
The goal of our work presented here was to investigate to what degree we can automatically infer students’ knowledge level directly from their performance while engaging in problem solving with the help of an ITS

Summary

Introduction

Assessment is a key element in education in general and in Intelligent Tutoring Systems (ITSs; (Rus et al 2013)) in particular because fully adaptive tutoring presupposes accurate assessment (Chi et al 2001; Woolf 2008). The pre-test serves two purposes: enabling macro-adaptation in ITSs, i.e. the selection of appropriate instructional tasks for a student based on student’s knowledge state before the tutoring session starts, and, when paired with a post-test, establishing a baseline from which the student progress is gauged by computing learning gains (post- minus pre-test score). This widely used pre-test/post-test experimental framework is often necessary in order to infer whether the treatment was effective relative to the control

Objectives

Results

Conclusion