Multidimensional Students' Evaluations of Teaching Effectiveness

Abstract

The literature on students' evaluations of teaching effectiveness (SETE) consists of thousands of studies and dates back to 1920s and earlier |13~. In reviews of this research, Marsh |10, 12, 13, 16~ concluded that SETEs are: (a) multidimensional; (b) reliable and stable; (c) primarily a function of instructor who teaches a course rather than of course that is taught; (d) relatively valid against a variety of indicators of effective teaching; (e) relatively unaffected by a variety of variables hypothesized as potential biases to ratings; and (f) seen to be useful by faculty as feedback about their teaching, by students for use in course selection, by administrators for use in personnel decisions, and by researchers. Based on his review, Marsh argued that SETEs are probably the most thoroughly studied of all forms of personnel evaluation, and one of best in terms of being supported by empirical |13, p. 369~. SETEs, like teaching that they represent, are a multidimensional construct (for example, an instructor may be organized but lack enthusiasm). This contention is supported by common sense and a considerable body of empirical research |13~. The SETE literature contains several examples of well-constructed instruments with clearly defined factor structures that provide measures of distinct SETE scales. In his review, Marsh |13~ noted that systematic approach used in development of these instruments and similarity in factors identified by each supports their construct validity. The Students' Evaluation of Educational Quality instrument that is focus of present investigation appears to measure most broadly representative set of scales and to have strongest factor analytic support of these instruments |10, 11, 12, 13~. Marsh and Hocevar |17~ factor analyzed SEEQ responses for a total group of 24,158 sets of class average ratings and for each of 21 different subgroups selected to differ in terms of instructor rank, course level (undergraduate and graduate), and academic discipline. The set of 9 factors that SEEQ is designed to measure was identified in all 22 factor analyses, and factor scores based on total group were highly correlated with factor scores based on separate analyses of each of 21 subgroups (mean r |is greater than~ 0.99). The results demonstrated that SEEQ measures a broadly representative set of evaluation factors and supported generalizability of SEEQ factor structure. A logical extension of multidimensionality of SETEs is that a given instructor has a distinct profile of SETE ratings (for example, high in organization but low in enthusiasm) that generalizes over time and across different courses. There is apparently no research that has examined this hypothesis, and so purpose of this investigation is to test it. Support for existence of a distinguishable profile that is specific to each instructor has important implications for understanding SETEs, for use of SETEs as feedback, and for relation of SETEs to other criteria, such as student learning. The necessary starting point of such research is to determine whether instructors have distinguishable SETE profiles, and this is purpose of present investigation. Generalizability of SETEs Studies of generalizability of SETEs have typically considered global ratings, total scores, or separate analyses of specific SETE dimensions. Researchers |for example, 4, 6, 8, 9~ have examined correlations between ratings of same instructor in different offerings of same course, same instructor in different courses, and different instructors teaching same course in an attempt to disentangle relative influence of course and instructor. This research examined generality of SETEs and relative importance of instructor who teaches course and course that is being taught. For example, for overall instructor and course ratings, correlations between ratings of different instructors teaching same course (one estimate of course effect) were -0. …