Will artificial intelligence revolutionise the student evaluation of teaching? A big data study of 1.6 million student reviews

Abstract

This article presents the first-ever big data study of the student evaluation of teaching (SET) using artificial intelligence (AI). We train natural language processing (NLP) models on 1.6 million student evaluations from the US and the UK. We address two research questions: (1) are these models able to predict student ratings from the student textual feedback, and (2) can AI-powered SET eliminate the problems of the traditional SET based on Likert scale surveys. We test these NLP models on a new dataset of 12,386 university reviews from the UK and on 155 SET reviews from a university that agreed to run a pilot AI project. The trained NLP models exhibited high prediction accuracy, and they learnt two biases from humans, those of extreme responding and assigning higher ratings to less demanding courses. In the future, universities will likely adopt many AI-based tools that have proved successful in client management in other sectors. However, our results make a strong case against using AI as a black box for performativity purposes. It should remain a useful tool for university administrators who are aware of the AI weaknesses documented here.