A Framework for Assessing the Complexity of Auto Generated Questions from Ontologies

Abstract

Automatic difficulty calibration (ADC) is the application of computational techniques to estimate the difficulty levels of assessment questions before administering them. Compared to traditional difficulty calibration approaches, ADC eliminates the need for pretesting, and minimises the time and efforts typically involved in manual calibration. In recent years, Ontology-based Automatic Question Generation (OAQG) has emerged as a powerful tool to generate assessment questions effortlessly and in massive numbers with minimal human intervention. Despite these benefits, not being able to control the characteristics of generated questions hinders their suitability to be used in pedagogical settings. However, much of the research up to now has tended to focus on the quantity of the questions rather than their quality. Indeed, most current automatic question generators produce simple questions which consist of a few facts, and simply test the recall of knowledge. Furthermore, the majority of existing frameworks are mostly technical and are not supported with a strong theoretical underpinning. In this paper, we propose a novel framework to assess the complexity of ontology-based, automatically generated questions. We discuss various aspects that are involved in determining the complexity of assessment questions, and attempt to quantify important characteristics of question complexity through the use of novel ontological metrics. To further support the plausibility of our computational framework, we shed light on its consistency with theories from education and cognitive psychology. This will provide us with a solid theoretical foundation which ensures that questions are generated according to principled methods that are grounded in theories of learning and cognition. The proposed approach is agnostic to different domains and independent of the question format, therefore, is highly general and applicable to a variety of contexts.