Reliability Of Student Ratings Research Articles

Assessing the reliability of self‐ and peer rating in student group work

Peer and self‐ratings have been strongly recommended as the means to adjust individual contributions to group work. To evaluate the quality of student ratings, previous research has primarily explored the validity of these ratings, as indicated by the degree of agreement between student and teacher ratings. This research describes a Generalizability Theory framework to evaluate the reliability of student ratings in terms of the degree of consistency among students themselves, as well as group and rater effects. Ratings from two group projects are analyzed to illustrate how this method can be applied. The reliability of student ratings differs for the two group projects considered in this research. While a strong group effect is present in both projects, the rater effect is different. Implications of this research for classroom assessment practice are discussed.

Using Generalizability Theory to Assess the Reliability of Student Ratings of Academic Advising

The coefficient alpha procedure has been used frequently to assess the reliability of student ratings of academic advising, despite the fact that in many cases, coefficient alpha is not an appropriate procedure for providing reliability information on this type of measurement data. Data were collected from 15 postsecondary institutions’ use of the Survey of Academic Advising, and a more advanced framework, that is, the generalizability theory, was used to analyze the reliability of the data. The results of the study demonstrate that the reliability of student ratings of advising can be estimated more appropriately and accurately by using the procedures of generalizability theory. Advantages and disadvantages of the generalizability theory approach in assessing the reliability of student ratings of advising are discussed in comparison with those of the coefficient alpha procedure.

Validity of Student Ratings of Field Instruction

Abstract Researchers have suggested that practicum planning might be improved by considering student ratings of the practicum. However, deriving decision aids from empirical studies on the field experience is problematic because validity and reliability of student ratings has not been demonstrated. The study reported in this article compared student ratings of field instructor behavior with field instructor self reports. Shulman's (1982) supervision questionnaire was mailed to a random sample of 40 social work graduate students and field instructors. Student ratings of field instructors were more strongly associated with students' global satisfaction than with instructor self-reports. Better indicators of practicum quality are needed.

Validity and Reliability of Student Ratings of Faculty

(1983). Validity and Reliability of Student Ratings of Faculty. Teaching of Psychology: Vol. 10, No. 1, pp. 46-46.

Alternatives to Student Ratings of College Teaching

It seems clear that the use of systematic student ratings of college teaching has increased since Astin and Lee's study [1] of a decade ago. Seldin [40] found that more than half of the private liberal arts colleges that he surveyed now use systematic student ratings. The collection and use of other kinds of teaching evaluation data (i.e., informal student opinions, classroom visits, colleagues opinions, scholarly research and publication, student examination performance, course syllabi and examinations, long term follow-up of students, alumni opinions, and graduate distributions) seems to have decreased in Seldin's study, with the exception of committee evaluations. Likewise, Riggs [37] reported a high use of student ratings in AACTE schools (86 percent in 1975). It would appear that student ratings have become an important, if not the most important, kind of systematically collected data considered in evaluating college teaching. The use of student ratings as a means of evaluating college teaching has caused a considerable stir in higher education. Researchers have conducted numerous studies aimed at examining the validity of student ratings in terms of such validity criteria as student achievement, course grades, and instructor behavior [13, 15, 21, 22, 30, 49, 53]. In general, the results have been mixed and the validity issue still appears to be far from settled. While the research regarding the reliability of student ratings is generally supportive [13], Seiler et al. [39] report that the reliability of student ratings over time is low (accounts for less than 40 percent of the variance). Some faculty unions have limited, through collective bargain-

Rating Stability Shown after Feedback of Prior Ratings of Instruction

Student evaluation of college professors is a phe nomenon that dates back to the Middle Ages. At the University of Padua in Italy, for example, the students hired their own professors (Werdell, 1967). But tradi tion in American education has not provided much in the way of student evaluation of professors. Student evaluation in the United States began at Harvard in 1924 when students published a Confidential Guide to Courses (Eble, 1970) which gave a review of students' ratings of courses, professors, examinations, etc. This publication is still doing a thriving business. At about the same time, the University of Washington began an evaluation program, and in 1954 the University of Michigan launched a professor evaluation program (Slobin and Nichols, 1969). Developing scales for student evaluation of instruc tion began about 20 years ago in a real systematic man ner with the Purdue Rating Scale (Remmers and Elliott, 1950). This is a graphic ten point rating scale consisting of ten qualities of a teacher. The scale can be used to develop profiles for each faculty member against norms that have been developed. Factor analytic studies of instructional rating scales (Isaacson, Mc Keachie, Milholland, et al., 1963, 1964; Cosgrove, 1959; Cof?man, 1954; Hoffman, 1963) generally found from three to seven consistent factors, with the following five factors common to most rating scales: (1) ability in presentation, (2) stability and fairness in work load, (3) organization, (4) positive response and feedback, (5) enthusiastic, friendly and constructive manner. The fact that students can be ''experts in evaluating instruction has long been in contention among faculty and students (McKeachie, 1969a). There are many arguments, most of which are not supported by evidence, that the student does not recognize effective and good instruction. Some statements to support this argument are: cannot really evaluate a teacher until they have left college and obtained some perspective on what was really valuable to and rate teachers on their personalities?not on how much they've learned (McKeachie, 1969b, p. 214). Thus, a curious and rather enigmatic teacher/student status conflict seems to have developed. Further enumeration of frequent objections to stu dent evaluations of teaching and the subsequent lack of evidence for them have been stated by Slobin and Nichols (1969) as follows: (1) Student ratings are influenced by variables irrelevant to teaching. But studies show that such factors as age of student, sex of stu dent and instructor, students' grades, etc. are not cor related with student ratings of instruction. (2) Student ratings reflect only the instructor's personality. This may be true if the rating forms are poorly constructed, but it is possible to construct questionnaires which do indeed tap areas other than personality. (3) Students cannot evaluate the goals of teaching. Students are not being asked to set the goals but are asked to evaluate how well the teacher is achieving his goals. (4) A man should be judged by his peers. Student evaluations do not violate this. Peers are not expected to be the best judges of the goals of teaching. (5) Overemphasis on teaching has bad consequences. This could be an objec tion only if good teaching is not essential. Slobin and Nichols (1969) quoted E. R. Guthrie as saying, It is well to remember that student evaluation is continuous and inescapable. The only question is whether or not we care to know what it is. Many of the questions about student evaluations have been partially answered, if not resolved. Reliability of student ratings has been shown to be high if there are 25 raters or more (Shock, Kelly, and Remmers, 1927). Student grades, past, present or expected, are generally unrelated to evaluation of instructors (Rem mers, 1930; Elliott, 1950; Voeks and French, 1960; Garverick and Carter, 1962). While the student may not be competent to judge the instructor's knowledge and mastery of concepts in his field, it appears that the student should be able to judge how well the instructor gets his subject across to the student. Solomon (1964) observed that the student is able to reasonably assess classroom performance. Thus a basis of reliability in student rating of instructors appears to have been es tablished to a moderate degree. Further reliability ques tions about stability of instructor ratings are posed; e.g. what is the consistency of an instructor's rating on teaching a given course when measured twice in the same semester (same students) compared to his rating on the same course the following semester (different group of students) ? And what are the effects on rating, stability after the instructor has been given the results of prior ratings? The purpose of this study was twofold: first to de termine whether students' ratings of instructors, on certain selected variables, are reliable over a period of Ill

Negative Numbers and Order of Numbers in Student Ratings

This study is one in a program of empirical examinations of the effects of format variables on student ratings of effectiveness of college teachers and courses. This study investigated the influence of negative numbers and order of numbers on the level and reliability of student ratings of teaching effectiveness. N o studies of student rating scales were found in which negative numbers or order of numbers were systematically investigated. However, Follman, Lucoff, Small, and Power (1973) in an earlier scudy, found that negative phrasing of items led to much lower mean scores than did positive phrasing. Therefore, the objective of this study was to investigate furcher different formats, specifically negative numbers (+2, +1, 0, -1, 2 , and reverse order ( 5 , 4, 3, 2, 1) and natural order (1, 2, 3, 4, 5 ) . Two classes at the University of South Florida in May, 1973 constituted the student samples. Instructor 1 conducted an undergraduate guidance class, Instructor 2 an undergraduate science class. Within each class, students were randomly assigned to the three treatment groups. Ss were told that they were participating in a study of scale formats and they were asked to rate the instructor and course as they ordinarily would if no study were being conducted. The rating scale was a 17-item teacher and course scale developed at the University. Ratings were quanrified 5, 4, 3, 2, 1. Ns, reliability estimates, and means, respectively, for Instructor 1 were: 25,.88, and 4.26 for the reverse order; 18, .90, and 4.20 for the natural order; and 21, .85, and 4.43 for negarlve numbers. Ns, reliability estimates, and means, respectively, for Instrucror 2 were: 21, .93, and 3.18 for the reverse order; 25, .94, and 2.96 for the natural order; and 23, .90, and 3.17 for negative numbers. Separate analyses of variance indicated no significant differences in total score for either instructor. Significant differences were found on four individual items for Instructor 1, none for Instructor 2. Ir is concluded that nature and order of numbers used on teacher rating scales is not important despite a slight tendency for lower ratings to be associated with the natural format.