Jim Greer\u2019s and Mary Mark\u2019s Reviews of Evaluation Methods for Adaptive Systems: a Brief Comment about New Goals

Abstract

Mark and Greer’s (International Journal of Artificial Intelligence in Education, 4(2/3), 129–153, 1993) review was very influential in setting out effective goals and methods for evaluating adaptive educational systems of all kinds. A later review brought the story up to date (Greer, International Journal of Artificial Intelligence in Education, 26(1), 387–392, 2016). The current paper explores a new range of evaluative goals which go beyond the quality of learning outcomes, learning efficiency, transfer, retention, and short-term motivation. While learner satisfaction has been downgraded over the years as a reliable indicator of learning quality, it cannot be wholly ignored in terms of wider issues such as the learner’s developing metacognitive and meta-affective insight, regulatory competence and longer-term motivation. These factors lead on to such evaluable issues as the learner’s appetite for further learning of the kind just experienced as well as for learning in general. The rise in the use of data analytics and the increasing use of AIED and computer-based learning systems in schools and universities has led to the development of orchestration systems to assist the teacher to manage their students using such systems. Orchestration systems raise new kinds of evaluation goal, such as the balance of activity, cooperation and agency between the human teacher and the adaptive systems, as well as between the learner, the systems, the teacher and, indeed, other learners. Further evaluable goals include the degree to which the teacher is alerted to the learning difficulties of the learners, the degree to which the teacher’s scarce and valuable time is being used efficiently, and the degree to which the orchestration system can be used as a reflective device for teachers to examine their own practice.