LOsMonitor: A Machine Learning Tool for Analyzing and Monitoring Cognitive Levels of Assessment Questions

Abstract

Developing effective assessments is a critical component of quality instruction. Assessments are effective when they are well-aligned with the learning outcomes, can confirm that all intended learning outcomes are attained, and their obtained grades are accurately reflecting the level of student achievement. Developing effective assessments is not easy, especially when considering the large number of assessments that instructors are required to develop each semester. To facilitate the process of developing effective assessments, this article introduces a novel tool called “LOsMonitor.” The tool utilizes machine learning and text mining to classify the cognitive level of assessment questions and learning outcomes according to Bloom's revised taxonomy. It uses the classification results to show statistics and charts that could allow management and instructors to judge the quality of assessment questions and monitor the cognitive levels at which the students are being assessed. The classification performance of LOsMonitor was evaluated in terms of accuracy, recall, and precision. A focus group was also used to assess the usability and usefulness of LOsMonitor. Besides, a case study was conducted to test how the tool would perform in a real-world scenario. The evaluation results indicate that LOsMonitor can be very helpful in developing effective assessments. It was able to discover and report various issues in assessments that instructors did not notice. Instructors who participated in the focus group reported that LOsMonitor would facilitate their quality assurance work and help them to ensure better alignment between assessments and learning outcomes.