Knowledge assessment: tapping human expertise by the QUERY routine

Abstract

The QUERY procedure is designed to systematically question an expert, and construct the unique knowledge space consistent with the expert's responses. Such a knowledge space can then serve as the core of a knowledge assessment system. The essentials of the theory of knowledge spaces are given here, together with the theoretical underpinnings of the QUERY procedure. A full scale application of the procedure is then described, which consists in constructing the knowledge spaces of five expert-teachers, pertaining to 50 mathematics items of the standard high school curriculum. The results show that the technique is applicable in a realistic setting. However, the analysis of the data indicates that, despite a good agreement across experts concerning item difficulty and other coarse measures, the constructed knowledge spaces obtained for the different experts are not as consistent as one might expect or hope. Some experts appear to be considerably more skillful than others at generating a usable knowledge space, at least by this technique.