Predictive learning analytics in online education: A deeper understanding through explaining algorithmic errors

Abstract

Existing Predictive Learning Analytics (PLA) systems utilising machine learning models show they can improve teacher practice and, at the same time, student outcomes. The accuracy, and related errors, of these systems can negatively influence their adoption. However, little effort has been made to investigate the errors made by the underlying models. This study focused on errors of models predicting students at risk of not submitting their assignments. We analysed two groups of error when the model was confident about the prediction: (a) students predicted to submit their assignment, yet they did not (False Negative; FN), and (b) students predicted not to submit their assignment yet they did (False Positive; FP). We followed the principles of thematic analysis to analyse interview data from 27 students whose predictions presented FN or FP errors. Findings revealed the significance of unexpected events occurring during studies that can affect students' behaviour and cannot be foreseen and accounted for in PLA, such as changes in family and work responsibilities, unexpected health issues and computer problems. Interview data helped identify new data sources, which could be integrated into predictions to mitigate some of the errors, such as study loan application information. Some other sources, e.g. capturing student knowledge at the start of the course, would require changes in the learning design of courses. Our insights showcase the importance of complimenting AI-based systems with human intelligence. In our case, these were both the interviewed students providing insights, as well potential users of these systems, e.g. teachers, who are aware of contextual factors, invisible to ML algorithms. We discuss the implications for improving predictions, learning design and teacher training in using PLA in their practice. • Predictions with large errors are present even in highly accurate predictive models. • False Positive and False Negative errors need to be analysed separately. • Large errors are mostly explained by unexpected events and data not being collected. • Student interviews can reveal reasons of errors, otherwise not captured by the data. • Human intelligence is important to compliment current AI-based systems in education.