Teaching Evaluations Research Articles

Do You Trust Broken Mirrors? Confronting Structural Inequalities in Student Evaluations of Teaching in Higher Education

ABSTRACT In recent years, the presence of foreign-born female academics (FBFA) in U.S. higher education has noticeably increased. Despite this progress in inclusion and diversity, they continue to face significant challenges related to race and gender biases in education. These challenges are evident when student evaluations of teaching (SETs) are the mechanism that evaluates the effectiveness of their teaching. Adopting a duo-ethnography methodology, researchers examined the effects of SETs in their professional lives, exploring the intersections of their experiences in their academic roles. Results highlight the potential biases in SETs, emphasizing the need to question the trustworthiness of these measures.

Motivational factors and course difficulty as predictors of teaching evaluations: a study of U.S. undergraduates

ABSTRACT Student evaluations of teaching (SETs) are widely used in higher education to assess instructional effectiveness and inform faculty personnel decisions. However, the factors influencing SETs remain a topic of ongoing research. This study investigated the predictive role of students’ motivation-related perceptions, as conceptualised by the MUSIC model (eMpowerment, Usefulness, Success, Interest, and Caring), and course difficulty in explaining variability in course and instructor evaluations. Participants included 2,949 undergraduate students from 30 courses across various disciplines at a large public university in the United States. Hierarchical linear modelling analyses revealed that students’ perceptions of the MUSIC components were significant predictors of both course and instructor evaluations at the student and course levels. Perceived course difficulty was negatively associated with evaluations, but the effect was smaller compared to the motivation-related variables. The final models explained substantial proportions of variance in course (67.6%) and instructor (65.2%) evaluations at the course level. The findings highlight the importance of fostering a motivationally supportive classroom environment to enhance students’ perceptions of empowerment, usefulness, success, interest, and caring, which in turn, can lead to more favourable evaluations of teaching effectiveness. SETs should be used as a tool to inform instructors’ pedagogical decisions rather than for personnel evaluation. Limitations and future research directions are discussed, including examining the influence of student diversity on SETs and referencing more recent literature.

Evaluation of content knowledge and instructor impacts in a middle school outreach program: lessons from Brain Explorers

Despite general agreement that science outreach is important, the effectiveness of science outreach programs on participants' learning often is not assessed. Brain Explorers is a neuroscience outreach program that allows medical students to partner with Biomedical Sciences faculty to develop lesson plans with learning objectives, interactive experiments for middle schoolers, and assessments of learning. These lessons are then implemented through a community-level intervention in which medical students teach their lesson plans to middle schoolers and assist them in performing activities to reinforce the concepts. Importantly, the efficacy of these active learning interventions is assessed. Throughout the program's evolution, a variety of different forms of assessment have been utilized to examine student understanding. While the goals of outreach programs are varied, here we have focused on the evaluation of content knowledge gains laying out three unique evaluation methods including post-event, immediate pre- and post-event, and spaced pre- and post-event evaluation. Specifically, using Brain Explorers lessons as examples we explore the practicality and feasibility of various learning assessments in outreach programs, begin to examine the impacts of participation in these programs on the medical school instructors and encourage others in the field to implement assessment of their programs.

Can you mitigate gender bias in student evaluations of teaching? Evaluating alternative methods of soliciting feedback

Student evaluations of teaching can be unreliable indicators of effective teaching and affected by implicit bias. We conduct a randomized experiment at a selective U.S. liberal arts college in which we vary both the instrument and timing at which we solicit student feedback to assess whether either intervention can mitigate gender disparities in qualitative evaluation comments. External evaluators with expertise in teaching evaluation scored the positivity, specificity, and constructiveness of student comments in control and treatment groups. We find neither intervention is successful in mitigating gender disparities. Students that receive an alternate prompt which highlights the potential for implicit bias and asks them to articulate and apply their own criteria for effective teaching penalize female faculty at similar rates as those receiving a more standard evaluation prompt. We also find no evidence that delaying solicitation of student feedback until the start of the next semester, when students face less stress and fatigue, reduces the gender gap in the positivity of feedback. Instead, we find substantial evidence that the positivity of qualitative student evaluations is affected by student, instructor, and course characteristics across all study conditions. The results suggest that bias in student evaluations may be difficult to mitigate.

Exploration of Strategies for Applying Life Based Teaching in High School Political Courses

Since the promulgation and implementation of the "General High School Ideological and Political Curriculum Standards" (2017 edition, revised in 2020) (hereinafter referred to as the "Curriculum Standards"), the combination of education and life has become one of the requirements of the curriculum standards. Life oriented teaching has been increasingly valued, promoting the concept of teaching from life to life, which is conducive to students mastering real ideological and political knowledge and establishing the concept of applying what they have learned. Therefore, it is necessary and feasible to use life oriented teaching in high school politics classes. Exploring life oriented teaching strategies for high school political courses can start from four aspects: updating teachers' educational concepts, using life oriented case materials, creating activity-based classrooms with a life oriented atmosphere, and pursuing life oriented teaching evaluations. This will promote students' comprehensive development and individual socialization, and strengthen their correct understanding of the importance of high school political courses.

A topic model analysis of students' gendered expectations using surveyed critiques of lecturers

Student evaluations of teaching (SET) have been examined over the years to better understand the student experience, with an increasing portion of literature exploring the presence of implicit bias in SET surveys against minority gender and ethnic lecturers. This study explores free-text comments made by students from a large public university in Australia over the period 2010–2016, using a semi-supervised statistical approach. Data were collected via surveys administered online at the end of each course to every student officially enrolled in that course via the learning management system, and completion of the surveys was voluntary. We build a probabilistic topic model which incorporates student and lecturer characteristics into the topic formation process. We make statistical inference on the effects of gender and cultural or language backgrounds based on the topic and word prevalence probabilities. The results showed clear separation of topics discussed between male and female lecturers. From a gendered perspective, our topic analyses have found that students are significantly more likely to critique female lecturers to improve on structural aspects of the course, as well as aspects of time management and control of the lecturing environment. In comparison, male lecturers were significantly more likely to be critiqued on specific aspects related to lecture delivery. Lecturers from non-English speaking backgrounds were more likely to be both critiqued and praised for the clarity of their delivery.

Enhancement of academic programs and student guidance based on Generative AI (GenAI) and Student Evaluations of Teaching (SETs)

Enhancement of academic programs and student guidance based on Generative AI (GenAI) and Student Evaluations of Teaching (SETs)

Bias intervention messaging in student evaluations of teaching: The role of gendered perceptions of bias

Many studies have documented discrepancies in student evaluation of teaching ratings between male and female instructors and between ethnic majority and minority instructors. Given the importance of such ratings to academic careers and the likelihood of potential intergroup bias, it is crucial that institutions consider approaches to mitigate such biases. Several recent studies have found that simple bias mitigation messaging can be effective in reducing gender and other biases. In the present research, students enrolled in several large Faculty of Science undergraduate courses at an Australian university were recruited on a volunteer basis via the course learning management system. Half of the participants were randomly assigned an intervention message highlighting potential biases relating to gender and language background. Data from 185 respondents were analysed using Bayesian ordinal regression models assessing the impact of message exposure on evaluation scores. Reading a bias intervention message caused students to significantly adjust their scores, with the nature of that change dependent on student and instructor characteristics. Among male students, the bias intervention message significantly increased scores for all except male instructors with English speaking backgrounds, for whom there was no significant impact of the message. In contrast, among female students, the bias intervention message significantly decreased scores for male instructors with English speaking backgrounds only. The sample showed an overall decrease in scores in the intervention group relative to the control group. This is the first study to detect a negative impact of bias intervention messaging on SET scores. Our results suggest students may not acknowledge their own potential bias towards instructors with whom they share similar demographic backgrounds. In conclusion, bias intervention messaging may be a simple method of mitigating bias, but it may lead to consequences in which one or more groups receive lower ratings as a result of the correction.

Finding the Needle in the Haystack: Can Natural Language Processing of Students' Evaluations of Teachers Identify Teaching Concerns?

Institutions rely on student evaluations of teaching (SET) to ascertain teaching quality. Manual review of narrative comments can identify faculty with teaching concerns but can be resource and time-intensive. To determine if natural language processing (NLP) of SET comments completed by learners on clinical rotations can identify teaching quality concerns. Single institution retrospective cohort analysis of SET (n = 11,850) from clinical rotations between July 1, 2017, and June 30, 2018. The performance of three NLP dictionaries created by the research team was compared to an off-the-shelf Sentiment Dictionary. The Expert Dictionary had an accuracy of 0.90, a precision of 0.62, and a recall of 0.50. The Qualifier Dictionary had lower accuracy (0.65) and precision (0.16) but similar recall (0.67). The Text Mining Dictionary had an accuracy of 0.78 and a recall of 0.24. The Sentiment plus Qualifier Dictionary had good accuracy (0.86) and recall (0.77) with a precision of 0.37. NLP methods can identify teaching quality concerns with good accuracy and reasonable recall, but relatively low precision. An existing, free, NLP sentiment analysis dictionary can perform nearly as well as dictionaries requiring expert coding or manual creation.

COVID-19 reshaped students’ perceptions of teaching excellence but not their evaluations of teaching

Teaching excellence (TE) at the tertiary level has been extensively researched, with student perceptions of its frequency and the educator attributes that comprise it, increasingly important for student retention and learning, and institutional quality and benchmarking purposes. Despite its myriad of issues, student evaluation of teaching instruments (SETs) are the most commonly used proxies for TE. University students enrolled in medicine, nursing and health science courses were surveyed about TE-associated educator attributes before and after the COVID-19 pandemic. Students were also surveyed about the frequency of TE they experienced, and this was compared with metrics for associated unit SETs. Students’ 2022 perceptions of the frequency of TE were significantly lower than those for 2017, both overall and based on years of university study. Conversely, the mean 2022 SET was significantly higher than that of 2017, indicating a discordance between the perceived frequency of TE and SETs. Applying a framework with validated TE ‘themes’, we found a significant difference between pre- and post-pandemic values for the overall cohorts and non-first year students’ weightings of educator attributes that best describe TE. No such difference was found for first year students. These results strengthen calls for tertiary institutions to develop more authentic TE criteria, bespoke and dynamic SETs that more accurately reflect TE, and relevant educator professional development that will enhance students’ learning gain and overall university experience. The implementation of such initiatives will be increasingly important in a rapidly changing and more disrupted tertiary environment.

Comparative analysis of student evaluations: Exploring the effects of course modality and type in graduate business courses

This study compares student evaluations across face-to-face and online modalities, as well as qualitative versus quantitative courses, in a large sample of graduate-level courses from 2008 to 2023 at a US business school. The findings reveal significantly lower course and instructor evaluation ratings despite similar class performance in online modality. Quantitative courses received notably lower satisfaction ratings than qualitative courses. Importantly, when accounting for modality and course type, grades do not influence student evaluations. Maintaining fair comparisons by restricting evaluations solely to the same modality and course type is crucial for assessing college professors’ teaching quality through student evaluation ratings.

Strategies for Boxing Teaching in Competitive Sports Schools

To improve the quality of boxing teaching in competitive sports schools, it is necessary to first analyze the strengths and weaknesses of boxing teaching in competitive sports schools, and then optimize the teaching strategy. The strategies include innovating the boxing curriculum system and content, building learning communities inside and outside the classroom, personalizing the teaching forms and methods, using multimedia to broaden the teaching path, focusing on the integration of boxing teaching leagues, and diversifying the teaching incentives and evaluations.

Success in Asynchronous Courses Through Increasing Student’s Social Presence

Online education has fundamentally altered the educational landscape, enhancing accessibility and flexibility for learners. However, a notable limitation of asynchronous learning modalities is the diminished interpersonal engagement inherent in traditional face-to-face pedagogical contexts. This empirical inquiry investigates the impact of deliberately integrating highly interactive instructional activities within online asynchronous developmental mathematics courses. Three principal thematic categories emerged through the iterative design process of these activities. Each category is delineated and exemplified, followed by exploring course policies conducive to student achievement in interactive learning contexts. Following a grounded theory approach, this study scrutinizes course outcomes alongside student and instructor evaluations at the semester’s end. Findings suggest incorporating social interactive activities yields positive effects on course outcomes.

Evaluation of Instructors’ Demographic Variations on a Web-based Platform for Connecting with Practitioners

Exploration of demographic variations is required to develop dynamic web platforms that cater to the varying preferences of diverse users. Hence, this study evaluated instructors’ demographic variations on a web-based platform for connecting with practitioners for student development. Both objective and subjective measures were adopted to investigate age- and gender-related differences in gaze behavior, task completion time, perceived cognitive load, perceived usability, and trust. Compared to male instructors, female instructors had higher fixation counts, longer task completion times, and statistically significant longer fixation duration. Female instructors gave higher usability and trust ratings but reported a higher cognitive workload. Compared to Generation Y instructors, Generation X instructors had longer fixation duration, higher fixation count, and statistically longer task completion time. Generation X instructors reported high cognitive load, lower usability, and trust ratings. The study also reveals demographic differences in parameters that instructors focused on while connecting with practitioners via a web platform.

Exploring the Relationship of Cognitive and Motivational Factors to Sonography Student Performance

Objectives: The purpose of this mixed-methods study was to explore the cognitive and motivational challenges sonography students encounter by examining the relationships between sonography students’ levels of self-regulation and self-efficacy and their performance in a 16-week introductory vascular sonography skills laboratory course. Materials and Methods: Measures for the study were designed to yield qualitative and quantitative data related to student goals, strategies, and course performance and were generated by both students and faculty. Qualitative data from the study included student self-reports of self-regulatory strategies and instructor evaluations of student performance, while quantitative data were provided by instructor and student ratings of performance, student self-efficacy ratings, and student reports on their use of deliberate practice. Results: The data showed that many students learning how to perform sonographic examinations encountered motivational challenges that appeared to be tied both to their self-efficacy and their ability to effectively use self-regulatory activities. Conclusion: The findings of this study suggest possibilities for improving sonography’s curricular models and teaching strategies toward the goal of making sonography instruction more efficient and effective.

Consideration of student evaluations of teaching (SET) and learning: perspectives of learning and teaching leaders through the ethical lens of “first, do no harm”

This paper explores institutional leaders’ perceptions of learning and teaching involved in facilitating and assessing Student Evaluations of Teaching (SET) survey instruments across Australian regional universities. It focuses on how they understand the function of SET, strategies used to mitigate bias, and potential residual harm. Through adopting a combination of inductive and deductive research processes and a thematic analysis through the ethical lens of nonmaleficence (first, do no harm), we report that leaders in learning and teaching perceive SET as a form of surveillance and Quality Assurance ‘performance’, recognise inherent biases inhabited in SET reports, and identify how these biases negatively impact academics through a lack of systematic harm mitigation strategies. The paper’s critical – and novel – contributions include an increased understanding of how SET inflicts harm towards women and other marginalised academic groups through systematic and authorised microaggressions and how SET contravenes universities’ duty of care to employees. It recommends an expansion of the principle of nonmaleficence beyond potential harm to research subjects, including those who undertake research or evaluation (such as academics), particularly if these impact them.

Career challenges of international female faculty in US universities: from a linguistic profiling perspective

PurposeThe purpose of this study is to review how international female faculty experience linguistic challenges and bias in their US university careers.Design/methodology/approachBy reviewing related literature, the authors explore the career challenges of international female faculty including hiring, promotion and tenure and leadership opportunities from a linguistic profiling perspective.FindingsInternational female faculty have relatively few hiring opportunities, specifically when institutions and fields openly accept linguistic profiling and bias and are less likely to hire non-native English-speaking international faculty. In the promotion and tenure process, international female faculty have struggled with standard academic English criteria and poor teaching evaluations from students because of the faculty’s different English usage such as word choice, grammar and pragmatics. In terms of leadership opportunities, international female professors have faced linguistic bias that non-native English faculty members are not competent, credible, intelligent or skilled because they speak accented English.Originality/valueThis study can help researchers and career development practitioners by adding linguistic profiling specific diversity and inclusion perspectives to existing literature. The findings expand the perspectives and practices related to the career challenges of international female faculty due to linguistic profiling.

The Persistence of Gender Bias in Student Evaluations of Teaching: The Role of Gender Stereotypes

The Persistence of Gender Bias in Student Evaluations of Teaching: The Role of Gender Stereotypes

Thinking or talking about teaching? Student evaluation as an occasion for dialogue or reflection on teaching

Using student evaluations of teaching (SETs) as a quality assessment tool have long been a highly debated topic among both practitioners and scholars of higher education teaching. Rather than focus on whether SETs provide reliable measurements of quality, we explore what happens when SETs are implemented as or intended as quality development tools. We report from a qualitative study of how higher education teachers and managers perceive and use SETs as part of developing their own teaching, and focus on how the dialogue around teaching, which was intended as a part of the quality assessment procedure, is actualized (or not) in practice.

Small programs, big challenges: Reimagining the evaluation of clinical teaching in genetic counseling.

Within the health professions education system, a significant proportion of teaching and learning occurs in the clinical setting. As such, the need to measure effective teaching for accreditation standards, faculty development, merit pay, academic promotion, and for monitoring the safety of the learning environment has led to numerous universities developing instruments to evaluate teaching effectiveness in this context. To date; however, these instruments typically focus on the student perspective, despite evidence demonstrating that student evaluations of teaching (SETs) lack correlation with learning outcomes and are not a true measure of teaching effectiveness. This issue is further exacerbated in small health professional training programs, such as genetic counseling, where clinical teachers may only supervise 1-3 students per year. As a result, not only are SETs more confounded due to small sample sizes, but a direct conflict exists between respecting learner anonymity and providing timely and relevant feedback to faculty. In such contexts, even using SETs to evaluate the nature of the learning environment may be unreliable due to student concerns about identifiability and fear of retaliation for unfavorable evaluation. This paper will review the literature regarding SETs, barriers to this process within the clinical setting, and the unintended downstream consequences. Options for addressing issues related to the use of SETs will be considered, with particular focus on the process of reflection and the use of teaching consultations or peer support groups as a means to improve teaching effectiveness in this learning environment.