Is England’s Office for Students likely to falsely identify courses as below threshold on the B3 progression metric?

Abstract

ABSTRACT Rising inflation and slow growth globally have led to governments focussing on value for money, and in the sphere of higher education policy that is often translated into graduate outcomes. In England, the Office for Students (OfS) has created the progression metric which for full-time, first degree and UK domiciled students requires courses to have 60% of graduates with positive outcomes 15 months after graduation. This paper utilises simulations to evaluate the likelihood that the OfS’s approach will fail to identify courses below the threshold (false positives) and incorrectly identify courses at or above the threshold (false negatives). The simulation adapts the levels of positive outcomes within the population (20-90%), sample size (40–1000), percentage sampled (30-90%) and crucially confidence intervals applied (90%, 95% and 99%) to identify courses below threshold. The three main findings are: the choice of 90% CI minimises the likelihood of false positives compared to 95% or 99% CIs; second, larger samples are essential to reduce likelihood of false positives; and third, preferring the 95% confidence interval instead of 99% for taking regulatory action will result in more courses incorrectly identified as below threshold. Governments that choose a statistical approach to regulate educational standards, like the application of confidence intervals to identify provision below quality standards, should utilise simulations to check the likelihood of false positives and false negatives since failure to do so could lead to both: (a) courses below standard not being identified and (b) regulatory action taken incorrectly against provision at or above standards.