Valid Prediction Intervals for Course Grades with Conformal Prediction

Abstract

Conformal prediction is a statistical-learning framework that complements predictions with a reliable measure of confidence [22]. It allows the construction of prediction intervals for any type of regression models under the assumption that the data are exchangeable. Conformal prediction intervals have provably valid frequentist coverage for finite data; i.e. they contain the true value of the response variable for any test data with probability at least 1 - ε for any significance level ε.Due to the validity of its prediction intervals, we argue that the conformal-prediction framework has to be used for course-grade prediction. In this paper, we experiment with two conformal predictors: Inductive Conformal Machines (ICMs) [11] and Cross-Conformal Machines (CCMs) [21]. They are compared in terms of informational efficiency (width and stability of prediction intervals) and computational efficiency on course-grade prediction tasks from a liberal education program of Maastricht University, the Netherlands. We show that ICMs enjoy substantial computational benefits while CCMs has a better informational efficiency. Moreover, CCMs extend the applicability of the conformal-prediction framework to course-grade prediction tasks for data sets of as few as 30 instances.