Missing the forest for the trees: investigating factors influencing student evaluations of teaching

Abstract

Student evaluations of teaching (SETs) feature prominently in higher education and can impact an academic’s career. As a result, they have attracted considerable research attention in order to identify evidence of bias and the influence of factors beyond an educator’s control. This study investigates the influence of seven factors on a large dataset of student evaluations (N = 376,805) of academics teaching at an Australian university. Students were invited to rate their experience at the end of each teaching period using an online survey instrument. The following factors are analysed comparing means between relevant groups to verify if: i) SET is dominated by students with strong feelings; ii) revenge reviews are given by angry students; iii) larger units are rated lower than smaller units; iv) different expectations/ratings are given by students of different gender and backgrounds; v) reticence of international students lowers overall ratings; vi) bigoted students skew results for some staff; and, vii) SET surveys during examinations disadvantaging academics teaching units with examinations. Overall, while statistically significant differences were found, they represented only small or trivial effects, with medium effects in only two limited cases. The results highlight the importance of explicitly reporting effect size of statistically significant results, and the benefits of representing differences visually in ways that avoid over-emphasising differences.