Student Rating Data Research Articles

Assessing and Evaluating Anatomy Instruction: Strategies for Anatomy Educators

Effective instruction is crucial to student learning in anatomy education. While anatomy teaching contexts have expanded and accreditation standards have become more focused on student learning, teaching evaluation practices have not changed. Across the US, higher education institutions have come to rely primarily or solely on students’ ratings of instruction to evaluate teaching. While more than 80 years of research has demonstrated that end‐of‐term student ratings provide results that accurately and reliably reflect students’ collective perceptions of the learning environments created by their instructors, researchers have long recommended that students’ perception data not be the only source of evidence used in the evaluation of teaching effectiveness. Students’ ratings are important and should not be excluded, but they should never be the only source of information used to evaluate teaching. Anatomy faculty do have options. In this session, we will discuss actions faculty can take to use student ratings data for formative assessment (i.e., improvement before evaluation by administrators or colleagues), as well as strategies that can be used to corroborate (or contradict) student ratings used in promotion or reappointment processes. Faculty participants in this session will learn about strategies for controlling the narrative about their teaching through systematic interpretation, targeted responses to ratings, and collection of additional evidence of teaching effectiveness from other sources. Administrators and peers will realize that they have the power to to ensure that student ratings data are used appropriately in evaluative contexts.

Interpreting and using student ratings data: Guidance for faculty serving as administrators and on evaluation committees

This article is about the accurate interpretation of student ratings data and the appropriate use of that data to evaluate faculty. Its aim is to make recommendations for use and interpretation based on more than 80 years of student ratings research. As more colleges and universities use student ratings data to guide personnel decisions, it is critical that administrators and faculty evaluators have access to research-based information about their use and interpretation.The article begins with an overview of common views and misconceptions about student ratings, followed by clarification of what student ratings are and are not. Next are two sections that provide advice for two audiences—administrators and faculty evaluators—to help them accurately, responsibly, and appropriately use and interpret student ratings data. A list of administrator questions is followed by a list of advice for faculty responsible for evaluating other faculty members’ records.

Alternative Specifications of Multivariate Multilevel Probit Ordinal Response Models

Multivariate multilevel models for ordinal variables are quite complex with respect to both interpretation and estimation. The specification in terms of a multivariate latent distribution and a set of thresholds helps in the interpretation of the variance-covariance parameters. However, most existing estimation algorithms for multilevel models can be used only if the model is reparameterized as a univariate model with an additional dummy bottom level. Moreover, the univariate formulation allows the model to be cast in the framework of Generalized Linear Latent and Mixed Models ( Rabe-Hesketh, Pickles, & Skrondal, 2001a ), a rather general class that includes, as special cases, structural equations and factor models. This article outlines the multivariate latent distribution specification and the corresponding interpretation issues; it then shows the univariate formulation, along with some alternative parameterizations that are useful in the estimation phase. An application to student ratings data illustrates the interpretation of the parameters and the estimation procedures, with a discussion of some computational issues.

Grading leniency is a removable contaminant of student ratings.

It is well established that students' evaluative ratings of instruction correlate positively with expected course grades. The authors identify 4 additional data patterns that, collectively, discriminate among 5 theories of the grades--ratings correlation. The presence of all 4 of these markers in student ratings data (obtained at University of Washington) was most consistent with the theory that the grades--ratings correlation is due to an unwanted influence of instructors' grading leniency on ratings. This conclusion justifies use of a statistical correction--illustrated here with actual ratings data--to remove the unwanted inflation of ratings produced by lenient grading. Additional research can profitably seek other inappropriate influences on ratings to identify more opportunities for validity-enhancing adjustments.

The Dependability of Student Evaluations of Teaching Effectiveness: Matching the Conclusions to the Design

Carbno (1981) used generalizability theory as a conceptual framework for analyzing student ratings data from two courses taught by a single instructor. This article argues that unwarranted conclusions were reached in the study because its design included an improper operationalization of the object of measurement, given the problems addressed, and because the sample sizes employed were too small.

Speech and drama training for lectures as a means of improving university teaching

Although considerable resources have been invested in programs for the improvement of college and university teaching, there is little data available on the effectiveness of these programs. The present study assessed the value of speech and drama training as a method of improving lecturing skills in university teachers. It was found that teachers who received speech and drama training showed significant improvement in student ratings of classroom teaching relative to control teachers. Rival hypotheses involving greater motivation to improve in experimental teachers and generalized placebo and self-help effects were ruled out on the basis of student rating data from previous courses and differential findings for target and nontarget rating scales. It is concluded that speech and drama training can make a small but significant contribution to the improvement of university teaching.

The stability of teacher performance in the same course over time

Traditional reliability studies report the stability of teaching performance, as measured by student course/instructor evaluations, as ranging from .40 to .90, depending upon the study. Recently, generalizability theory has been applied to these rating data to more specifically pinpoint sources of error in student course evaluations. The purpose of the present study is to (1) familiarize the reader with some of the work that has been done with student rating data using generalizability theory and (2) to illustrate the use of the theory on a problem that in the past has been examined using traditional reliability estimates. Specifically, the paper addresses the stability of teacher performance across time in the same course.

Dimensions of Effective Teaching: Factors or Artifacts

A current concern in evaluating faculty from student rating data has been to identify dimensions of teaching behaviors. Although recent studies have identified factors which are highly stable across different groups of teachers and students, the specific type of data underlying the factor analyses may have yielded dimensions which are implicit in the student raters rather than the teachers. In the present study, although the factor structures of rating items were found to be highly congruent over different subject matters and teaching roles, the factors were also highly related to students' implicit categories of teaching behaviors. The meaning of these results is discussed by considering the potential influences of rater's implicit theories on the different types of rating data which can be factor analyzed.

THE GENERALIZABILITY OF STUDENT RATINGS OF INSTRUCTION: ESTIMATION OF THE TEACHER AND COURSE COMPONENTS

Previous studies (Gillmore, Kane, and Naccarato, 1976; Kane, Crooks, and Gillmore, 1976; Kane, Gillmore, and Crooks, 1976) have shown that generalizability theory (Cronbach, Gleser, Nanda, and Rajaratnam, 1972) can be a valuable tool in analyzing the dependability of student ratings of instruction. Specifically these studies have discussed three distinct generalizability coefficients within the common paradigm under which these ratings are collected. The coefficient which is appropriate in a given setting depends upon whether one wishes to generalize over both students and items, or only over one or the other. Furthermore, the characteristics of each of these coefficients can be easily traced as a function of changes in the number of raters and items. In addition, the multifaceted interpretation which underlies generalizability theory was useful in showing that the variability of student responses is consistently greater than the variability within students, indicating that increasing the number of students generally has a much greater impact on dependability than increasing the number of items. Perhaps the most important practical conclusion arising from the series of analyses reported was that, even for the most conservative case, when generalization is over both students and items, generalizability coefficients of .70 and greater were found when a class was rated by fifteen or more students, responding to four or more items. This result was reported in the above studies for three distinct instruments in two locations for various disciplinary groupings. The object of measurement for the above studies was the class, defined as a particular course (e.g., Psychology 101) taught by a particular teacher. However, the class as the object of measurement confounds the teacher effect with the course effect. This confounding is acceptable for some uses of student ratings, e.g. feedback to the teacher on his performance in a particular course, but does not permit estimation of the dependability of student ratings for two important applications of such ratings. First in evaluating a teacher's instructional effectiveness for a promotional decision, one would probably want to generalize over all courses that the teacher might teach rather than focusing on results of one or more specific courses. Second, in evaluating a specific curriculum offering, one would probably want to assess the course's rating by students, generalizing over the universe of all possible teachers who might teach it. Analyses in which the teacher and course effects are confounded do not allow an assessment of the dependability of student rating data for either of these purposes.

Implicit Theories in Student Ratings

Identifying generalizable dimensions of teaching from student ratings data has been both practically and theoretically important in the study of teaching. However, an examination of the methods employed reveals that many studies may have identified dimensions which were implicit in the student raters rather than the instructors. The current study compared various methods of identifying teaching factors and found substantive similarity of the dimensions defined from students’ implicit theories and various types of correlational data. The results are interpreted both with respect to the rating process and to the nature of the factors identified in previous student ratings research.

The localization, diagnostic, and monitoring functions of student ratings in a model for improving university teaching

A diversity of student questionnaires are used by colleges and universities to provide data on faculty teaching performance. Yet the purposes for collecting this data are frequently unclear, and at times superficial. Rarely is student rating data used as a tool to improve faculty teaching. A more relevant approach incorporates a variety of types of student ratings into a model for improving university teaching. One type of student rating data is used to identify broad instructional problem areas. Another type pinpoints probable causes and solutions for the instructional problems. Instructional improvement procedures are designed on the basis of this data. A third type of student rating data evaluates the instructional improvement procedures and indicates when modifications are needed. In addition to these three types of student ratings, and the generation of appropriate questionnaires, this paper presents an overview of the teaching improvement model and discusses its effectiveness.

Implicit Theories in Student Ratings

Identifying generalizable dimensions of teaching from student ratings data has been both practically and theoretically important in the study of teaching. However, an examination of the methods employed reveals that many studies may have identified dimensions which were implicit in the student raters rather than the instructors. The current study compared various methods of identifying teaching factors and found substantive similarity of the dimensions defined from students’ implicit theories and various types of correlational data. The results are interpreted both with respect to the rating process and to the nature of the factors identified in previous student ratings research.

On the Invalidity of Student Ratings for Administrative Personnel Decisions

With an increased demand for more careful assessment of teaching, administrators are incorporating student ratings of instructional effectiveness into their personnel decisions. Yet a variety of findings makes questionable the validity of this use of student ratings. The purpose of this paper is to review these findings to place the administrative use of student ratings into a proper perspective. Comparability of student ratings, sources of invalidity, and a proposal for the use of student rating data are discussed.

RESEARCH: The Ongoing Debate: Student Evaluation of Teaching

Teaching competence is not easily documented The university teacher is one of few professionals whose work is seldom observed by his peers. His reputation is often based more on hearsay than on substantive evidence. For this reason, promotion committees frequently make final tenure decisions without seriously considering information about teaching. To counter this tendency, many campuses have recently instituted a system of student instructional ratings. Unfortunately, this approach has little more credibility with faculty than hearsay system. Even though students are only people who observe a teacher's course daily, many faculty seriously question reliability and validity of student instructional ratings. My present purpose is to analyze these issues in light of recent research developments and to indicate what implications this work may have for faculty evaluation process. Although there are several acceptable procedures for assessing reliability of a measuring instrument, most meaningful index of reliability for student rating data is extent to which students agree about strengths and weaknesses of different teachers. In my Science report of October 5, 1973, 13 different instructors were rated by students in two multisection calculus courses. The ability of students to discriminate among teachers on several different traits was examined. When asked to comment on clarity of teacher's presentations, students were consistent in selecting same teachers as and worst instructors on this trait. Statistical analysis of this outcome indicated that a difference as large as that observed would be expected on a chance basis less than one time in a million. When students were asked to comment on work load in their calculus classes, differences in their mean ratings for each instructor were also extraordinarily reliabla These results suggest that students can clearly discriminate among performances of different instructors. To analyze this reliability issue in more detail, I re-examined some data from an investigation which I had previously reported in Spring 1973 issue of American Educational Research Journal. Ratings from 72 students who had enrolled in both biology and organic chemistry were analyzed. The students had been asked to respond on a sevenpoint rating scale to item the teacher explained concepts clearly and made good use of examples and illustrations. Their responses were analyzed with a two-way analysis of variance (2 instructors x 72 raters). The point of interest was whether variation among ratings could be attributed to differences among students or to differences between teachers. My analysis indicated that 44 percent of variance could be attributed to differences between teachers, 5 percent of variance was attributable to differences among student raters in their tendencies to use high or low ratings in general, and 51 percent of variance was attributable to differences among students in way they perceived two teachers. This result clearly indicates that student ratings provide considerable information about teacher. It is a rare event when an independent variable in a behavioral science experiment accounts for more than 10 percent of total variance. The fact that 44 percent of variance in my ratings was related to instructors argues strongly that students can function as reliable observers. This demonstration that students can make consistent ratings does not necessarily mean, however, that ratings are valid in sense of indicating which instructors are best This validity issue is more difficult to analyze than reliability issue. Some faculty insist that their real goal in is to impart philosophical values or to inculcate a special attitude toward learning rather frhfln to simply help student to master subject matter. Intangible outcomes such as these are not easily measured, and therefore it is difficult to determine whether teachers who successfully accomplish these goals are also ones who are rated highly by their students. Only if we define teaching in a relatively narrow way, such as preparing students to solve calculus problems on a written examination, is it feasible to determine if some teachers do a more adequate job than others. In Science, I recently reported evidence indicating that teachers who were best at preparing their students for a calculus exam were also ones who received highest student ratings. Since there is contradictory evidence on this point, let us consider this research and its implications in more detail.