Using the sampling margin of error to assess the interpretative validity of student evaluations of teaching

Abstract

We present an equation, derived from standard statistical theory, that can be used to estimate sampling margin of error for student evaluations of teaching (SETs). We use the equation to examine the effect of sample size, response rates and sample variability on the estimated sampling margin of error, and present results in four tables that allow users to assess the interpretative validity (IV) of a SET score for a specific evaluation context. In this framework, a small margin of error, e.g. 3% of the range, suggests a greater precision, or IV in a score, whereas a large margin of error, e.g. 10% of the range, suggests a lower IV. We review the SET literature and summarise five ways that low response rates may be ameliorated. We also compare our results to previous published studies. Our findings matched, but greatly extended, prior results.