Ratings Of Instruction Research Articles

Faculty Forum

Past research indicates several factors influencing teaching evaluation ratings instructors receive. We analyzed teaching evaluations from psychology courses during fall and spring semesters of 2003–2004 to determine if class size, class level, instructor gender, number of publications (faculty instructors), average grade given by the instructor, and instructor rank predicted teaching evaluation ratings. Entering predictor variables into a multiple regression analysis concurrently, results indicated that only average grade given and instructor rank significantly predicted instructor ratings. Specifically, higher average grades given by the instructor predicted higher ratings, and graduate teaching assistants received higher overall ratings than faculty instructors.

Hot and Easy in Florida: The Case of Economics Professors

The objective of this study is threefold. First, we seek to investigate whether overall quality ratings recorded by students on RateMyProfessors.com (henceforth RMP) are positively related to the perceived easiness and hotness RMP ratings for 173 economics professors at public universities in Florida. Second, the overall quality ratings on RMP are investigated to determine whether they are positively correlated with the overall in-class instructor ratings for a subset of public Florida universities. Third, this subset of overall in-class instructor ratings is examined to determine whether the ratings are significantly related to the easiness and hotness ratings recorded on RMP. For a sample of 173 overall quality ratings on RMP, our results indicate that students who utilize RMP reward easy and hot economics professors with a higher rating. On average, for every additional 10 percent of total ratings that indicate a professor is hot, that professor's RMP overall quality rating increases by approximately 0.125 points on a 1 to 5 scale. Similarly, for every additional point of perceived easiness on a 1 to 5 scale, the RMP overall quality rating increases by about 0.50 points. Moreover, the correlation coefficient between the RMP overall quality ratings and the overall in-class ratings for a subsample of 60 economics professors at five public universities in Florida is only 0.47. However, when regressing the overall in-class instructor ratings on easiness and hotness ratings from RMP, easiness is significant, but hotness is not. Interestingly, when the overall in-class instructor ratings are regressed on perceived easiness, helpfulness, and clarity ratings from RMP, only helpfulness is significant. The findings reported here support the view documented previously in the literature that students reward professors based on a criterion that arguably increases teaching productivity - helpfulness outside the classroom.

The Utility of Student Ratings of Instruction for Students, Faculty, and Administrators: A "Consequential Validity" Study

Students, faculty and administrators at a major Canadian university were surveyed to investigate the utility or "consequential validity" of student ratings of instructors. Of the 1,229 (approximately equal number of males and females) students and alumni, about half (52%) indicated that they had never used the ratings, but of those who did use it, many (47%) reported using it several times to select courses and/or instructors. The majority (84%) of faculty members (n = 357) gave favorable responses about the usefulness of student ratings for improving quality of teaching. Paradoxically, even though faculty members were positive about the student ratings, they did not generally use them to make changes in their teaching. The majority (87%) of administrators (n = 52) stated that they use the student ratings for various purposes including decisions about faculty merit and tenure. Students, faculty and administrators considered the overall course instruction to be the most useful type of information derived from the student ratings. The results of the present study indicate that while the utility of data from student ratings of instructors is quite variable, there is evidence of "consequential validity" particularly from administrators.

Measuring Teaching Effectiveness in a Pre-Clinical Multi-Instructor Course: A Case Study in the Development and Application of a Brief Instructor Rating Scale

Background: Despite widespread use, misunderstandings persist about student evaluations of teaching. These evaluations have not been well examined in the common medical school setting of the multi-instructor, preclinical lecture course.Purpose: The study evaluated the psychometrics of a brief student evaluation of a teaching instrument developed for a multi-instructor 2nd-year course and described its application.Methods: An 11-item instrument was developed and administered to 276 students to evaluate 27 lecturers per year in 3 years of an introductory clinical psychiatry course. A fully crossed research design allowed for a thorough analysis of variability in ratings.Results: Generalizability analysis showed good reliability and relatively large Student × Lecturer interactions. Profile analysis generated distinct lecturer teaching profiles.Conclusions: Judicious use of a psychometrically sound student evaluation of a teaching instrument can be used to assist faculty and course development. Administering the evaluation instrument to an entire class produces no better reliability than administration to randomly selected subgroups of students.

Transfer of Manual Flying Skills From PC-Based Simulation to Actual Flight-Comparison of In-Flight Measured Data and Instructor Ratings

Three groups of novice pilots received training to fly aerobatic maneuvers in a light aircraft. Trainees in the control group received in-flight instruction and were given the usual briefings before each flight. Trainees in the 2 experimental groups received extra training: Each in-flight lesson was preceded by PC-based simulated flight. A total of 2,053 maneuvers were analyzed on the basis of both flight-data recordings and instructor ratings. I hypothesized that complex manual flying skills, learned on the ground, transfer to the aircraft. The results provide no objective support for this hypothesis. There were no significant differences in flying skills between the three groups as measured by the flight-data recordings. However, both experimental (PC) groups managed to fly significantly more maneuvers in the same amount of flight time in the aircraft and received better instructor ratings. I analyze these differences in detail. In the discussion, I compare the findings with published transfer experiments with PC-based simulation.

Bridging the Gap between Reduced Training Time and Maintained Operational Demands

As a consequence of decreased defense budgets, the training time of conscript combat boat 90 operators has recently been reduced. The effect has been an increased number of drop–outs, detachments, and accidents during the training. The need to improve this precarious situation is urgent. This encouraged a study with the purpose of measuring abilities identified as important for being successful in the training, and hence a high-performing and safe combat boat 90 operator. The objective was to improve the selection process in order to minimize drop–outs, detachments, and accidents during the training. The test battery consisted of psychometric tests measuring spatial and verbal working memory, problem solving ability, risk willingness, and social capacity. The results of the psychometric tests were compared with instructor ratings of performance during the training. The only test that showed to reliably predict a successful combat boat 90 operator was the test that measured verbal working memory.

Association amongst factors thought to be important by instructors in dental education and perceived effectiveness of these instructors by students.

It is hypothesised that dental educators have perceptions of their roles as effective teachers. It is expected that subject matter expertise would be amongst the components of such personal philosophies of education, but it is unclear whether faculty member self-perceptions carry over into student ratings of instructors' effectiveness. A 20-item survey of 'Teaching Characteristics' was completed by 86% of full-time and 64% of the part-time faculty members at the University of the Pacific. Respondents distributed 100 points amongst the descriptions of what makes an effective instructor. The responses were factor-analysed, resulting in four general faculty 'types' that explained about 50% of the variance in ratings: expert, enthusiast, judicial and good soldier. Student ratings for the 2 years running up to the date of the survey administration were used to gauge student perceptions of instructor effectiveness. Faculty members who placed emphasis on expertise as key to being a good instructor received significantly lower ratings for teacher effectiveness from students than did other faculty members. Faculty members who conceived their roles as motivating students, explaining difficult concepts, displaying interest in the subject, showing compassion and caring, and being proactive tended to receive high ratings for teaching effectiveness from students.

Association amongst factors thought to be important by instructors in dental education and perceived effectiveness of these instructors by students

It is hypothesised that dental educators have perceptions of their roles as effective teachers. It is expected that subject matter expertise would be amongst the components of such personal philosophies of education, but it is unclear whether faculty member self‐perceptions carry over into student ratings of instructors’ effectiveness. A 20‐item survey of ‘Teaching Characteristics’ was completed by 86% of full‐time and 64% of the part‐time faculty members at the University of the Pacific. Respondents distributed 100 points amongst the descriptions of what makes an effective instructor. The responses were factor‐analysed, resulting in four general faculty ‘types’ that explained about 50% of the variance in ratings: expert, enthusiast, judicial and good soldier. Student ratings for the 2 years running up to the date of the survey administration were used to gauge student perceptions of instructor effectiveness. Faculty members who placed emphasis on expertise as key to being a good instructor received significantly lower ratings for teacher effectiveness from students than did other faculty members. Faculty members who conceived their roles as motivating students, explaining difficult concepts, displaying interest in the subject, showing compassion and caring, and being proactive tended to receive high ratings for teaching effectiveness from students.

Making the Grade with Students: The Case for Accessibility

AbstractStudent‐based teaching evaluations are one critical component of instructor and course evaluation in most universities. The purpose of this study is to critically examine several factors that drive student‐based ratings of instructors and courses in an undergraduate mechanical engineering program. The findings are based on a statistical study of student ratings in 181 mechanical engineering courses during a three and one‐half year period, two mid‐semester classroom interviews, and anecdotal evidence garnered from personal experience. A key rating that appears strongly linked with the overall instructor and course rating is the “accessibility of the instructor.” The instructor accessibility factor outweighs others often cited by conventional wisdom, such as perceived course workload or expected grade in the course. Rationale for the strong correlation between instructor rating and perceived instructor accessibility is discussed. Several suggestions for how to improve instructor accessibility and resulting student‐based teaching evaluations are provided.

A Preliminary Analysis of Psychophysiological Variables and Nursing Performance in Situations of Increasing Criticality

To examine the relationship between psychological, physiological, and performance variables in intensive care unit (ICU) nurses in situations of increasing criticality. Psychophysiological variables and endotracheal suctioning performance were examined in a classroom, a skills laboratory, and an ICU. Situation-specific anxiety (state anxiety) and the predisposition to view situations as threatening (trait anxiety), cognitive appraisal, and heart rate were measured and compared with self-appraisal and a nurse instructor's ratings of suctioning performance. Baseline data were obtained during class on 45 novice ICU nurses. Twenty-six nurses provided complete data, which included being videotaped and monitored in the classroom, in the skills laboratory performing endotracheal suctioning, and in the ICU during suctioning. High state anxiety significantly predicted poor ICU suctioning performance (P<.04). Nurses high in state and trait anxiety, worry, and heart rate performed poorly compared with less anxious nurses. Nurses in this study who performed best had a mean heart rate of 94 beats/min. Those nurses who are high state anxious, high trait anxious, and worried and who had a faster heart rate performed less well than their more relaxed peers. Nurses with high state anxiety may be at risk for attrition, burnout, medical errors, and poor performance in other ICU nursing tasks.

A cross‐cultural comparison of instructor communication in American and German classrooms

This study addressed patterns and influence of student perceptions of instructor power use, affinity‐seeking, and nonverbal immediacy in American and in German classrooms. American instructors were seen to be significantly higher in power use, affinity‐seeking, and nonverbal immediacy than German instructors. American students reported significantly higher perceptions of cognitive learning than those reported by German students. Instructor affinity‐seeking and nonverbal immediacy were strongly correlated with student affective/cognitive learning and ratings of instructor for American and German students. The influence of instructor referent power, affinity‐seeking, and nonverbal immediacy on student cognitive learning was significantly higher for American students.

Are Student Ratings of Teaching Effectiveness Influenced by Instructors' English Language Proficiency?

We use data from norming the third edition of the Test of Understanding College Economics to exam ine whether instructors for whom English is a second language (ESLs) receive lower student ratings of teaching effectiveness in principles of economics courses, holding constant what students learn in the course. The results suggest that student ratings of ESL instructors are, on average, about 0.4 points lower, on a scale of 1.0 to 5.0, than the ratings of native English speaking instructors. Most of this deficit can be attributed to differences in how the two groups of instructors teach their courses and evaluate the knowledge of their students.

Determinants of Student Evaluations of Global Measures of Instructor and Course Value

Using structural equation modeling (LISREL) with a sample of business students, this study addresses important issues with respect to student evaluations. After reviewing past research and using exploratory factor analysis, confirmatory factor analysis suggests the existence of five latent variables in student evaluations: organization, workload/difficulty, expected/fairness of grading, instructor liking/concern, and perceived learning. Next, adding structural paths, these variables are used to model the formation of student evaluations. The models developed demonstrate that student evaluations may lack discriminant validity, the extent to which a measure does not correlate with other constructs it is not supposed to measure (e.g., expected/fairness of grading does have a large impact on ratings of teaching ability). Moreover, the use of student evaluations is compounded by the common usage of global measures in student evaluation practice. Here, the use of overall course value and instructor rating are contrasted. The results are disparate, raising questions about the use of summative measures (i.e., of what should they be composed, if used at all).

Can preclinical medical students be integrated into the continuing medical education process by instructing prehospital care providers?

Objective: To develop a model introducing medical students (MS) to the continuing medical education (CME) process while simultaneously developing a curriculum to enhance the relevant surgical anatomy knowledge base of the advanced prehospital care provider. Methods: A CME curriculum for teaching human anatomy was developed and approved by the governing state agencies for prehospital education. The curriculum focused on structures relevant to the prehospital care of the trauma patient in a case based format using common scenarios presented by surgery and emergency medicine faculty. Five year-one medical students who completed gross anatomy served as teaching staff and were given a structures list one week prior to the CME course. Human cadavers were prosected by the medical students prior to the CME program under the guidance of the surgical faculty. Course attendees and medical student staff were anonymously surveyed at the end of the program (rating scale 1 = low to 5 = high). Prehospital providers were given a multiple-choice posttest and surveyed at 3 months after the course with regard to applicability to their current practice. Data are means ± SD. Results: Nineteen licensed practicing paramedics attended the course. All of the paramedics scored above the 85% passing cutoff on the posttest (95.6% ± 6.2%). Instructor qualities were rated highly (4.62 ± 0.49) with no instructor rating less than a 3. MS believed themselves well prepared to teach (5 ± 0), and spent 2 ± 0.81 hours in preparation. They were only infrequently faced with questions they were not well prepared to answer (1.25 ± 0.5) and would uniformly participate in CME offerings in the future (5 ± 0). The CME program improved the MS view of CME (3 ± 0), prehospital education (3 ± 1.4), and the surgeon as educator (3.25 ± 1.5). At 3 months, the paramedics felt that the CME program significantly impacted the care they rendered (4.37 ± 0.76), and improved their understanding of injury complexes (4.53 ± 0.61), and resuscitation (4.26 ± 0.73). The cadaver course was uniformly recommended to coworkers (5 ± 0). Conclusions: This model provided prehospital care providers direct contact with clinically relevant human anatomy, enhanced their understanding of pertinent anatomy, and positively impacted their patient care. MS were introduced to the CME process and found it to be one with which they would become reinvolved. Furthermore, the MS felt prepared to present human anatomy, met the expectations of the course attendees, improved their understanding of prehospital education, and positively altered their perception of the surgeon as an educator. This process holds promise as both a model for prehospital education and as a tool for integrating MS into the role of allied health educator early in their career.

The influence of instructor gender on learning and instructor ratings

This paper uses two large databases to determine if students of a given gender learn more from instructors of the same gender and if instructors of a given gender receive higher ratings from students of the same gender compared to students of the opposite gender. The Indiana University (IU) data set was collected over a six-year period from the fall of 1984 to the spring of 1990 for associate instructors teaching introductory microeconomics and macroeconomics courses. A subset of the data collected in norming the third edition of the Test of Understanding College Economics is the TUCE III data set. The results of this study do not lend support for a positive gender-match role model effect concerning student learning and instructor ratings. This effect appeared only in the IU micro data set but did not appear on a consistent yearly basis. The effect did not appear at all in the IU macro data or either of the TUCE III data sets.

The interrelationship of student ratings of instructors' immediacy, verbal aggressiveness, homophily, and interpersonal attraction

This study explored the relationships of immediacy and verbal aggression with homophily and interpersonal attraction in the instructional context. Immediacy was negatively related to verbal aggression and positively related to all dimensions of homophily and interpersonal attraction. Verbal aggression was also found to be negatively related to both dimensions of homophily and all three dimensions of interpersonal attraction. It is recommended that the relationship between verbal aggression and immediacy be examined in greater detail, including possible’ interactions between these variables and various instructional outcomes.

Student Ratings of Instruction in Distance Learning and On-Campus Classes

Student ratings in 2 special education courses offered on campus and off campus using different means of instruction, including electronic media, were compared. Ratings also were compared when distance classes were taught at local and remote facilities. End-of-course student evaluations were examined using a counterbalanced design. A comparison of outcome measures revealed no difference in the overall course means. Outcome measures for on-campus students versus off-campus students for the 2 courses were examined, but no differences were found in the overall ratings. Course, instructor, teaching, and communication ratings were similar across settings and courses. Implications for future research in evaluating instruction at a distance are suggested.

Student Rating Myths Versus Research Facts from 1924 to 1998

Sixteen of the most common myths regarding student ratings of instructors and instruction are looked at from the perspective of the research that has been conducted on them over the past seventy-four years. It is concluded that the myths are, on the whole, myths. However, suggestions are made about how the information regarding the myths can be used to both improve and document instructional effectiveness.

Psychotherapists' Perceptions of Their Trainers and Training Emphases

ABSTRACT Psychotherapy training is currently a neglected area of empirical study (Binder, Bongar, Messer, Strupp, Lee, & Peake, 1993; Binder & Strupp, 1993; Henry, Strupp, Butler, Schacht, & Binder, 1993; Strupp, 1993). This study empirically investigated trainees' perceptions of the emphases placed by instructors and supervisors on therapeutic skills which have been shown in the past to contribute to both positive and negative patient outcome. The Therapist Capacity Questionnaire (Korner & Goldberg, in press) was utilized for that purpose. Fifty three members of a state psychological association completed the questionnaire. A significant difference was found between the participants' ratings of instructors and supervisors on all but two capacities. This finding highlights the importance of the supervisory role in training. Instructor and supervisor ratings by the participants appeared to correspond to the participants' importance ratings of capacities indicating that trainees may have internalized what they perceived to be important to their trainers.

“How'm I Doing?” Problems with Student Ratings of Instructors and Courses

“How'm I Doing?” Problems with Student Ratings of Instructors and Courses