Can we trust teaching evaluations when response rates are not high? Implications from a Monte Carlo simulation

Abstract

ABSTRACT Online teaching evaluations have replaced in-class teaching evaluations as the new norm for assessing teaching performance. Faculty raise concerns regarding lower response rates compared to traditional in-class evaluations. A low response rate implies an underrepresentation of the target student population and threatens the validity of the evaluation. The literature comparing online with in-class evaluations focuses on the average evaluation ratings and concludes, with few exceptions, that the two methods produce statistically indifferent results. The accuracy of an evaluation, however, is not addressed. This study employs Monte Carlo simulation to investigate the effects of various evaluation conditions, including response rates, class sizes, and observed evaluation scores, on the accuracy of the evaluations. The complex distribution of evaluation ratings are carefully simulated based on a real dataset, and the effects of the investigated factors on evaluation accuracy are calculated and analyzed. Implications for assessing teaching performance are discussed.