The calculus dashboard — leveraging intelligent tutor techniques to provide automated fine-grained student assessment

Abstract

Automatic grading systems, such as WebWork, are becoming much more widely used as they relieve the instructor from needing to grade student work, provide students with automatic feedback, and can allow for immediate resubmission. They have also been shown to improve the effectiveness of teaching and learning. In this paper, we apply Item Response Theory (IRT) to a large WebWork Calculus homework dataset to provide a skill level for each student and item characteristics curves for each problem which we then show accurately predict the probability a given student will get a particular problem correct. A student's skill level at the end of a course represents a kind of summative assessment which can be used to accurately predict how well they would answer future questions, and hence is perhaps a better indicator of subject mastery than the grade on the final exam. We also apply the Performance Factors Analysis (PFA) approach to our data and use it to provide a more fine-grained analysis of the student's mastery. PFA requires labeling each problem with a set of skills that are required to solve that problem. It produces a formula that accurately predicts the probability that students will correctly answer a new question based on their previous answers. We use the PFA approach to produce a dashboard for every student which isolates their mastery of different skills. Our dataset consisted of 703,743 attempted solutions to 243 different questions by 1609 students in 87 sections of Calculus I at a large university. Both the IRT and PFA approaches produced accurate predictions, and PFA enabled a dashboard to be constructed for each of the students.