Higher-order thinking skills assessment in 3D virtual learning environments using motifs and expert data

Abstract

The research reported in this paper addresses the problem of assessing higher-order thinking skills, such as reflective and creative thinking, within the context of virtual learning environments. Assessment of these skills requires process-based observations and evaluation, as the output-based methods have been found to be insufficient. Virtual learning environments offer a wealth of data on the process, which makes them good candidates for process-based evaluation, but the existing assessment methods in these environments have shortcomings, such as reliance on large data sets, inability to offer specific feedback on actions, and the lack of consideration for how actions are integrated into bigger tasks. Demonstrating and confirming the ability of three-dimensional virtual learning environments to work with process metrics for assessment, we propose and evaluate the use of motifs as an assessment tool. Motifs are short and meaningful combination of metrics. Combining time-ordered motifs with a similarity analysis between expert and learner data, our proposed approach can potentially offer feedback on specific actions that the learner takes, as opposed to single output-based feedback. It can do so without the use of large training datasets due to reliance on expert data and similarity analysis. Through a user study, we found out that such a motif-based approach can be effective in the assessment of higher-order thinking skills while addressing the identified shortcomings of previous work. We also address the limited research on similarity-based analysis methods, compare their effectiveness, and show that utilizing different similarity measures for different tasks may be a more effective approach. Our proposed method facilitates and encourages the involvement of instructors and course designers through the definition of motifs and expert problem-solving paths.