Use of student ratings to benchmark universities: Multilevel modeling of responses to the Australian Course Experience Questionnaire (CEQ).

Abstract

Recently graduated university students from all Australian Universities rate their overall departmental and university experiences (DUEs), and their responses (N = 44,932, 41 institutions) are used by the government to benchmark departments and universities. We evaluate this DUE strategy of rating overall departments and universities rather than individual teachers, and we juxtapose it with the traditional use of student ratings to evaluate individual teachers (SETs). Multilevel analyses of DUE overall ratings were not able to discriminate well between universities or departments--few universities or departments differed significantly from the grand mean. Although the a priori 5-factor structure for this DUE instrument was reasonably well-defined at the individual student level, none of the 5 factors separately or in combination discriminated well between departments or universities. In contrast to this pattern of results, we review studies showing that SETs do reliably differentiate between teachers and are valid in relation to many criteria of effective teaching. However, casual reviews of these research literatures should not use this support for SETs to justify the use of DUE-type strategies. We conclude that DUE-type ratings should be used with great caution, if at all, and should not be seen as an alternative to SETs