Use Of Student Ratings Research Articles

A systematic review of factors influencing student ratings in undergraduate medical education course evaluations.

BackgroundStudent ratings are a popular source of course evaluations in undergraduate medical education. Data on the reliability and validity of such ratings have mostly been derived from studies unrelated to medical education. Since medical education differs considerably from other higher education settings, an analysis of factors influencing overall student ratings with a specific focus on medical education was needed.MethodsFor the purpose of this systematic review, online databases (PubMed, PsycInfo and Web of Science) were searched up to August 1st, 2013. Original research articles on the use of student ratings in course evaluations in undergraduate medical education were eligible for inclusion. Included studies considered the format of evaluation tools and assessed the association of independent and dependent (i.e., overall course ratings) variables. Inclusion and exclusion criteria were checked by two independent reviewers, and results were synthesised in a narrative review.ResultsTwenty-five studies met the inclusion criteria. Qualitative research (2 studies) indicated that overall course ratings are mainly influenced by student satisfaction with teaching and exam difficulty rather than objective determinants of high quality teaching. Quantitative research (23 studies) yielded various influencing factors related to four categories: student characteristics, exposure to teaching, satisfaction with examinations and the evaluation process itself. Female gender, greater initial interest in course content, higher exam scores and higher satisfaction with exams were associated with more positive overall course ratings.ConclusionsDue to the heterogeneity and methodological limitations of included studies, results must be interpreted with caution. Medical educators need to be aware of various influences on student ratings when developing data collection instruments and interpreting evaluation results. More research into the reliability and validity of overall course ratings as typically used in the evaluation of undergraduate medical education is warranted.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-015-0311-8) contains supplementary material, which is available to authorized users.

¿Para qué sirve la evaluación de la docencia? Un estudio exploratorio de las creencias de los estudiantes

El propósito de este trabajo fue explorar las creencias de los estudiantes acerca de la evaluación de la docencia, sus fines, utilidad y validez. Participaron 163 estudiantes de carreras de las Ciencias Sociales y Humanidades de una universidad pública mexicana. Los resultados del estudio indican que los estudiantes consideran que la evaluación de la docencia tiene un fin sumativo, de control y supervisión del trabajo del docente, que tiene poca utilidad y los cuestionarios de evaluación no son válidos. Asimismo, de manera secundaria consideran que los fines de la evaluación docente son de tipo formativo. Se concluye que las políticas de evaluación de la docencia vistas a través de los estudiantes, no han tenido el efecto de mejorar la calidad de la docencia. Asimismo que las creencias de los estudiantes en torno a la evaluación docente son semejantes a los hallazgos producidos por la investigación reciente que critica el uso de los cuestionarios para evaluar la docencia con el fin de elevar la calidad de la enseñanza. Se discuten los resultados.

Exploring Non-Instructional Factors in Student Evaluations

The use of student ratings to measure instructors’ teaching performance and effectiveness in tertiary education has been an important but controversial tool in the improvement of teaching quality during the past few decades. This is an attempt to explore non-instructional factors of student evaluations by discussing and reviewing relevant literature with regard to the most common non-instructional factors in student ratings. Moreover, semi-structured interviews were used with 14 college instructors. The findings show that most of the teachers support the use of student evaluations as a means of quality control and teaching improvement. However, the great majority of teachers expressed their concerns about the non-instructional factors which affect student ratings and make them meaningless. They reported that gender, time of evaluation, expected grades, nationality of the instructor, and other factors can affect student ratings. The study proposes some recommendations which might make student evaluation practices more useful and informative.

Use of student ratings to benchmark universities: Multilevel modeling of responses to the Australian Course Experience Questionnaire (CEQ).

Recently graduated university students from all Australian Universities rate their overall departmental and university experiences (DUEs), and their responses (N = 44,932, 41 institutions) are used by the government to benchmark departments and universities. We evaluate this DUE strategy of rating overall departments and universities rather than individual teachers, and we juxtapose it with the traditional use of student ratings to evaluate individual teachers (SETs). Multilevel analyses of DUE overall ratings were not able to discriminate well between universities or departments--few universities or departments differed significantly from the grand mean. Although the a priori 5-factor structure for this DUE instrument was reasonably well-defined at the individual student level, none of the 5 factors separately or in combination discriminated well between departments or universities. In contrast to this pattern of results, we review studies showing that SETs do reliably differentiate between teachers and are valid in relation to many criteria of effective teaching. However, casual reviews of these research literatures should not use this support for SETs to justify the use of DUE-type strategies. We conclude that DUE-type ratings should be used with great caution, if at all, and should not be seen as an alternative to SETs

Supplemental Material for Use of Student Ratings to Benchmark Universities: Multilevel Modeling of Responses to the Australian Course Experience Questionnaire (CEQ)

Supplemental Material for Use of Student Ratings to Benchmark Universities: Multilevel Modeling of Responses to the Australian Course Experience Questionnaire (CEQ)

What do students consider useful about student ratings?

Although measures of student ratings of instruction have been developed and examined empirically, students' attitudes about the utility of these ratings have yet to be meaningfully quantified. Using survey responses from 1229 students at a major Canadian university, we developed a psychometrically sound measure of the usefulness of student ratings. A principal components analysis with oblique rotation was conducted and three factors (Instructor characteristics, Course characteristics and Instructor's relative ranking) were obtained that explained 64% of the variance in six iterations. The usefulness of these three types of ratings information differs according to several student and course characteristics. The present results provide evidence for both reliability and validity of the instrument as an indicator of the utility of teaching evaluation for students.

The Utility of Student Ratings of Instruction for Students, Faculty, and Administrators: A "Consequential Validity" Study

Students, faculty and administrators at a major Canadian university were surveyed to investigate the utility or "consequential validity" of student ratings of instructors. Of the 1,229 (approximately equal number of males and females) students and alumni, about half (52%) indicated that they had never used the ratings, but of those who did use it, many (47%) reported using it several times to select courses and/or instructors. The majority (84%) of faculty members (n = 357) gave favorable responses about the usefulness of student ratings for improving quality of teaching. Paradoxically, even though faculty members were positive about the student ratings, they did not generally use them to make changes in their teaching. The majority (87%) of administrators (n = 52) stated that they use the student ratings for various purposes including decisions about faculty merit and tenure. Students, faculty and administrators considered the overall course instruction to be the most useful type of information derived from the student ratings. The results of the present study indicate that while the utility of data from student ratings of instructors is quite variable, there is evidence of "consequential validity" particularly from administrators.

Integrating Theoretical Perspectives Of Student Rating Behavior

This paper examines attribution, grade-leniency/stringency, motivation, and construct validity theories of student-rating behavior. Previous research has tended to treat these theories as mutually exclusive. However, our findings suggest that these theoretical perspectives should be integrated into a unified theory of student-rating behavior. The R-squares found in this study are larger than in any previous research, which we attribute largely to the rigor of using structural equation modeling (SEM) to analyze the data. The significant bias that was found in this study suggests the need to rethink the use of student ratings when evaluating faculty performance.

Evaluation Of College Faculty: What Do Accounting Students Really Think?

Student opinion surveys of teaching are widely implemented at the university level for many reasons. Faculty members use the input from students to improve their teaching while administrators often use the information for merit, and tenure and promotion decisions. The extensive use of student ratings is in part the result of the increased interest in improving teaching and the growing demand for greater accountability on the part of colleges and universities. At issue is the concern that students are asked to evaluate faculty on components of teaching deemed important by divisions of institutional research and not necessarily by faculty themselves. Final approval of student opinion surveys may be prematurely granted without sufficient input from consumers. The purpose of this paper is to explore issues surrounding the wide spread use of student opinion of teaching surveys for evaluative purposes and to describe the role of student ratings in judging teaching effectiveness. An extensive review of literature was used to construct a survey administered to 165 students enrolled in undergraduate and graduate accounting courses. Students were asked to rank effective teaching practices. The results of this survey guide faculty and administrators to understand, from students’ perspective, effective teaching practices. Conclusions are drawn concerning the appropriate use of the information obtained from student evaluations and the impact the student ratings have on the quality of instruction in colleges and universities.

Effectiveness of Consultation on Student Ratings Feedback: A Meta-Analysis

Consultation on student ratings is recognized as an important strategy to support university teachers in learning from student ratings feedback. However, there is little evidence to suggest which practices and strategies are most important for successful consultation. As the use of student ratings increases, such evidence becomes increasingly important. The authors of this article performed an exploratory meta-analysis to determine the effects of various consultation strategies and to identify the conditions under which consultative feedback is most effective in improving teaching effectiveness. The results indicated that the various approaches to consultation are not equally effective. The authors suggest that only with use of certain consultation strategies might the benefits of support with student ratings feedback be maximized.

Attitudes of Nigerian secondary school teachers to student evaluation of teachers

This study was designed to assess the attitudes of Nigerian secondary school teachers to student evaluation of teachers (SET), and to find out if the attitudes expressed were influenced by teacher characteristics such as gender, professional status, geographical location, academic qualification and teaching experience. The study was a survey, and made use of a researcher-designed questionnaire as research instrument. Teachers in one state in Nigeria were sampled, as similar conditions apply for teachers in all states in Nigeria. A stratified random technique was used to select 480 teachers from 20 schools. Four hypotheses were tested using t-test and four-way ANOVA. The findings were that: (i) Nigerian teachers expressed significantly negative attitudes to SET when the results of such evaluation were to serve both formative and summative purposes, although the attitudes were more negative when the purpose was summative; and (ii) the attitudes expressed by Nigerian teachers to SET were not significantly influenced by the gender, academic qualification and teaching experience of the teachers under the two forms of purposes. However, under both purposes, teachers in urban areas were found to express relatively more negative attitudes to SET than their counterparts in rural areas. It was concluded that Nigerian teachers are not too different from their counterparts abroad in treating SET with serious caution. It was recommended that Nigerian school policy makers and administrators should exercise great care in the use of student ratings of teachers, especially in personnel decisions.

Changing the Agenda for Research into Students' Views about University Teaching: Four shortcomings of SRT research

Despite the increasing use of student ratings of teaching in higher education as an indicator of teaching quality, issues pertaining to the validity and reliability, rather than how best to use student ratings continue to dominate the research agenda for this topic. It is important to be aware of biases in ratings, but after 75 years of research and debate, it seems unlikely that academics will ever agree on whether student ratings should be used in promotion and tenure decisions. There is sufficient agreement, however, that they are useful for teaching improvement purposes. This article argues for a shift in emphasis of research, as there are unresolved and emerging issues that research has not yet explored sufficiently. Four shortcomings of student ratings are identified as in need of this attention in order to increase the practical usefulness of student ratings.

Documenting Teaching Excellence within the Peer Review of Teaching

The initial autumn email from a communication scholar chairing a department's tenure committee was welcomed and welcoming. I'll be happy to review your candidate's teaching scholarship, I wrote back. I requested artifacts on which I could make an evidence-based peer review - syllabi, course assignments and readings, copies of papers the assistant professor had graded including comments returned to the students, lesson plans, examples of creative computer, visual or audio presentations and tutorials, student assessments - in short, a teaching portfolio. Thanks anyway, the committee chair responded. We can't provide that material. The committee was anxious to reach a decision. The faculty member under review, the committee chair wanted me to know, always received strong student ratings, some of the best in the department. The tenure committee was planning its peer review around student evaluations and the individual's reputation, as well as upon the two classroom visits a committee member conducted to observe the assistant professor at work. My role, apparently, would have been to comment on reputation. I don't know the fate of the faculty member under review, or even the individual's name. The procedures briefly outlined, however, are worthy of discussion. Can the evidence described form the basis of a valid peer review? Let's examine the data that held the tenure committee's interest. What elements of teaching excellence are being rewarded? Which are being left out? Where is the emphasis? Student ratings are frequently used as a basis for peer review (the questionable use of student ratings for salary setting and the problems inherent in using any assessment for both formative and summative purposes are issues for another occasion). Although the research is not unanimous, student reviews carefully tailored to elicit opinions about teaching performance in a specific: instructional environment and administered with focused instructions and a clear explanation of what is at stake, may provide data that is both reliable and valid. Fears that student ratings reward the funniest faculty performers and the easiest graders are unfounded. In fact, students give the highest marks when they feel they have learned the most. The infamous bubble sheet can and often does provide an important data point for those graphing teaching performance. Student ratings are easy to use, but they are also subject to easy abuse. Students are not peers. Their assessments are not based upon the kinds of deep familiarity with, and mastery of, either teaching and learning or the field of communication that resides in the professional educational cohort we think of as peers. Student ratings tend to paint with broad brush strokes. Neither the bubble sheets nor open-ended student questionnaires capture a faculty member's professional pedagogical and curricular judgments, or the course preparation, mentoring, tutoring, grading effort, and assessment reflection that take place outside of the classroom. Student ratings also fail the innovation test. Many of us have had personal experience with at least one botched experiment in which our best attempts at new curriculum or pedagogy turn out well below expectations. Student ratings do not distinguish, however, between time-tested classroom delivery and the development of new pedagogies and curricular enhancements that carry the potential of discovery and the risk of poor classroom reception. Student Testimonials provide an additional and related data point a review committee of peers may consider. Although rarely critical, testimonials afford valuable context when viewed as one tile in a larger assessment mosaic. Unlike student ratings, there is little to suggest they are representative and, as indicators of teaching excellence, they may lack both validity and reliability. Classroom Observation, which a cynical colleague of mine refers to as drive-by assessments, are common. …

Disciplinary, Gender and Course Year Influences on Student Perceptions of Teaching: Explorations and implications

The prevalent use of student ratings in teaching evaluations, particularly the reliability of such data, has been debated for many years. Reports in the literature indicate that there are many factors influencing student perceptions of teaching. Three of these factors were investigated at the University of Western Australia, namely the broad discipline group, course/unit year level and student gender. Data collected over 3 years were analysed. The outcomes of this study confirmed results reported by other workers in the field that there are differences in ratings of students in different discipline groups and at different year levels. It also provided a possible explanation for the mixed results reported in studies of student gender in relation to student ratings.

The Use of Student Ratings of Teacher Behaviors With Academically Talented High School Students

The present study examined the validity of student ratings of teaching within a two-cohort sample of academically talented high school students (N = 851). Results from the 1997 cohort indicated that low-inference teaching behaviors, as rated by high school students, form factors that are similar to the factors found with college students. Further, student ratings of low-inference (i.e., observable) behaviors were significant predictors of global ratings of teaching effectiveness, whereas expected grade in a course and course workload did not contribute meaningfully to the prediction of global ratings of teaching effectiveness. Teaching behaviors and global ratings of effectiveness also did not contribute to the prediction of expected grade beyond the contribution of student ability. These findings were replicated with the 1998 cohort. Low-inference teaching behaviors may be a useful way for teachers of academically talented and regular education high school students to receive feedback on their teaching.

The development of a student rating of teaching effectiveness questionnaire

The use of student ratings of teaching effectiveness has become the most widely used, and often the only, source of information to assess and improve teaching effectiveness. Important prerequisites for questionnaires for student ratings should, therefore, be reliability and validity. The research reported in this article presents a methodology for developing a teaching effectiveness rating instrument, based on identified dimensions of teaching effectiveness believed to be important. Results obtained from 11 570 respondents of the RAU student rating questionnaire were factor analysed to an oblique simple structure. A first-order solution with four factors resulted: 1) Preparation of lectures; 2) Presentation of lectures; 3) Interaction with students during lectures; and 4) Assessment practices. One second-order factor was also found, representing General Teaching Effectiveness. The research results support the conclusion that the Rand Afrikaans University (RAU) student rating questionnaire is a reliable and valid instrument to implement for formative teaching assessment. South African Journal of Higher Education Vol.15(1) 2001: 84-91

The Dimensions of Students’ Perceptions of Teaching Effectiveness

The use of student ratings of instructional quality is enhanced by an understanding of the nature of the underlying dimensions. In the current investigation, confirmatory factor analysis procedures were used to assess the fit of the original solution for the Student’s Perceptions of Teaching Effectiveness questionnaire to a more recent sample of more than 7,000 university classes. Furthermore, a new exploratory factor analysis was used to examine the factor pattern after excluding certain items. The latter solution was crossvalidated on an additional sample. The analyses provided a clear interpretation of six first-order and two second-order dimensions of instructional quality that are useful across a broad range of university courses. The dimensions of teaching quality obtained by researchers are examined and compared to the results of the current study. Implications for the evaluation of perceived teaching quality are discussed.

Student Evaluation of Faculty: Effects of Purpose on Pattern

Much of the existing research addressing student ratings of faculty has focused on psychometric properties of the rating process or on physical characteristics of courses and individuals. Less explored is the cognitive processes of students, as evaluators, and the administrative purposes for which outcomes from student evaluations are used. To determine if students used different cognitive sets as a function of the explicit purpose for which ratings are obtained, a field experiment was conducted. Explicit purposes for course evaluations were manipulated to include a control condition, a formative condition (instructional improvement), and a summative condition (salary increase). These conditions served as the classification variable in a discriminant analysis and six different dimensions of classroom performance, as assessed by students, served as the discriminating variables. Results indicate that the purpose of assessment fails to influence the pattern of faculty ratings provided by students. From an administrative perspective, these findings have practical implications both for the collection of student evaluations and for the use of student ratings.

Are student ratings of instruction useful?

Despite the lead article^Rs title ^SValidity Concerns and Usefulness of Student Ratings of Instruction^T (Greenwald 1997) in the American Psychologist^R^Rs special section on teacher ratings, the papers did not provide direct evidence on ^Susefulness.^T There is no evidence that the use of teacher ratings improves learning in the long run. The papers do not show that the effects would improve the allocation of effort between teaching and research, or that the quality of the educational experience will be better, or that students and faculty will be happier. Given the evidence to date, the case for student ratings is weak. I raise some questions about usefulness, with a particular emphasis on the ratings^R effects on learning.

Student Evaluation of College Teaching Effectiveness: a brief review

ABSTRACT This paper presents a brief review of the existing research on student written evaluations of the teaching performance of college and university instructors. First, a short historical background is given. Next, arguments are presented which have been advanced for and against the use of student written evaluations as a valid indicator of teaching effectiveness, followed by a discussion of various background characteristics which have been thought to influence student ratings. Student and faculty reaction to the use of student ratings is discussed, and finally suggestions for further investigation are given.