Uncertainty of Skill Estimates in Operational Deep Knowledge Tracing

Abstract

This paper reviews challenges and opportunities of using deep knowledge tracing (DKT) in a production hybrid instructional and assessment system (HIAS) for measuring learner skills. An empirical data analysis using K-12 Mathematics practice data from an operational HIAS showed that the DKT estimates in the training set were well calibrated, although the estimates in a holdout set overestimated student skill mastery at lower ability levels and underestimated mastery at higher ability levels. Examination of expected vs. observed plots across ability levels provided a means for evaluating the generalizability of the predictions to new students and suggested a stratified sampling approach may be needed for creating training and validation sets that better represented the full practice data. Estimation accuracy was further examined in a parameter recovery study, with response data simulated from patterns in the empirical data. The study showed that in general, the DKT procedure had high estimation accuracy, with an absolute mean difference of approximately 0.03 units on the [0, 1] probability scale and a Pearson correlation of +0.89 between true and estimated values. To examine uncertainty in the estimates, 80% and 95% confidence intervals were constructed around the mean estimate for each true ability level. As expected, results showed less uncertainty at ability levels where more response data were generated, indicating the DKT approach is sensitive to ability level information in the training data. Finally, the DKT approach was able to detect learning across attempts within a skill, suggesting this approach may be preferred over traditional measurement approaches in scenarios where ability levels change over time.