Exploring Variability Sources in Student Evaluation of Teaching via Many-Facet Rasch Model

Abstract

Evaluating quality of teaching is important in nearly every higher education institute. The most common way of assessing teaching effectiveness takes place through students. Student Evaluation of Teaching (SET) is used to gather information about students’ experiences with a course and instructor’s performance at some point of semester. SET can be considered as a type of rater mediated performance assessment where students are the raters and instructors are the examinees. When performance assessment becomes a rater mediated assessment process, extra measures need to be taken into consideration in order to create more reliable and fair assessment practices. The study has two main purposes; (a) to examine the extent to which the facets (instructor, student, and rating items) contribute to instructors’ score variance and (b) to examine the students’ judging behavior in order to detect any potential source of bias in student evaluation of teaching by using the Many-Facet Rasch Model. The data set includes one thousand 235 students’ responses from 254 courses. The results show that a) students greatly differ in the severity while rating instructors, b) students were fairly consistent in their ratings, c) students as a group and individual level are tend to display halo effect in their ratings, d) students are clustered at the highest two categories of the scale and e) the variation in item measures is fairly low. The findings have practical implications for the SET practices by improving the psychometric quality of measurement.