Measuring Student Experience: Relationships between Teaching Quality Instruments (TQI and Course Experience Questionnaire (CEQ)

Abstract

The results of the Course Experience Questionnaire (CEQ) in Australian tertiary institutions have been available for a number of years and have provided the administration of these institutions with valuable information as to students’ perceptions of their courses. In addition to the CEQ, all of these institutions survey their students at the subject level. The purpose of this study has been to determine the degree to which the responses recorded on these subject level Teaching Quality Indicators (TQI) are related to the CEQ. In the analysis, we have used two approaches to cast some light on the degree to which the responses on the TQI can be used to anticipate and establish responses on the subsequent CEQ.Prior to performing these studies, we first sampled the extensive literature that describes influences on responses to TQIs. From this survey, we found that there are many factors which influence students’ responses, other than a specific lecturer’s performance in class. We have used the literature on the TQIs because there is almost no literature concerning students’ responses to course level surveys such as the CEQ. Nonetheless we have used the results of this survey to determine how we can control for these factors when modeling responses to the CEQ.There were two major hurdles which needed to be addressed in the performance of this analysis across institutions. First, there is no consistent TQI for all institutions. Most of these surveys have been assembled using questions with different origins that have varied over time and that are individual to each institution. Secondly, even if the TQIs were constant and consistent across all institutions, it would be necessary to account for any other factors, such as the year level and the field of study, that may influence the specific subject TQIs and the overall course evaluation. Our first analysis addresses, in a systematic manner, the first difficulty by investigating the nature of the TQIs in use in Australian tertiary institutions. Through the use of a website designed as a clearing house for information on these surveys, we have collected examples of the TQIs from 39 higher education institutions in Australia. We then categorised these different instruments to develop a composite new TQI survey that captures the characteristics of those in use.Using the new TQI, we then compared the responses of students at the four participating institutions (Flinders, Melbourne, Tasmania, and Wollongong). The students were asked to respond to a survey that included questions relating to teaching, generic skills and overall course satisfaction (from the CEQ), questions from the local TQI, and questions from the new TQI based on the composite question types found in our analysis of all Australian TQIs. We found that few of the institution-specific TQI’s elicited responses that matched the CEQ and that even the new TQI did not provide many solid matches. Thus we concluded that, in most cases, the items on the TQI and the CEQ are measuring different factors.For the CEQ responses from The University of Melbourne, we matched the students’ experiences in order to determine the degree to which their responses could have been anticipated. From this analysis we found that those students who completed the CEQ were most responsive to situations where their expectations are not met. We also found that characteristics of subjects that influence TQI responses are also factors that influence CEQ responses. We found that older students, those that receive higher marks than their peers, and those who study full-time in their last year, are more likely to rate a course more highly. Whilst students who take more than one course, who are in subjects with higher than average enrollments, and that received lower than average TQI ratings, are less likely to rate their course highly.The conclusion from this study is that TQIs at different institutions are not designed in a consistent manner and that only a small portion of the CEQ responses could be predicted by these TQI. However, just as with the studies of the TQIs, we can establish that course characteristics such as: the level of the degree, the Faculty and Department in which the course was taken, the course description, the industry and duties of those who have found employment after completing their course, all strongly influence the CEQ.