Student Evaluations Of Teaching Research Articles

Do You Trust Broken Mirrors? Confronting Structural Inequalities in Student Evaluations of Teaching in Higher Education

ABSTRACT In recent years, the presence of foreign-born female academics (FBFA) in U.S. higher education has noticeably increased. Despite this progress in inclusion and diversity, they continue to face significant challenges related to race and gender biases in education. These challenges are evident when student evaluations of teaching (SETs) are the mechanism that evaluates the effectiveness of their teaching. Adopting a duo-ethnography methodology, researchers examined the effects of SETs in their professional lives, exploring the intersections of their experiences in their academic roles. Results highlight the potential biases in SETs, emphasizing the need to question the trustworthiness of these measures.

Motivational factors and course difficulty as predictors of teaching evaluations: a study of U.S. undergraduates

ABSTRACT Student evaluations of teaching (SETs) are widely used in higher education to assess instructional effectiveness and inform faculty personnel decisions. However, the factors influencing SETs remain a topic of ongoing research. This study investigated the predictive role of students’ motivation-related perceptions, as conceptualised by the MUSIC model (eMpowerment, Usefulness, Success, Interest, and Caring), and course difficulty in explaining variability in course and instructor evaluations. Participants included 2,949 undergraduate students from 30 courses across various disciplines at a large public university in the United States. Hierarchical linear modelling analyses revealed that students’ perceptions of the MUSIC components were significant predictors of both course and instructor evaluations at the student and course levels. Perceived course difficulty was negatively associated with evaluations, but the effect was smaller compared to the motivation-related variables. The final models explained substantial proportions of variance in course (67.6%) and instructor (65.2%) evaluations at the course level. The findings highlight the importance of fostering a motivationally supportive classroom environment to enhance students’ perceptions of empowerment, usefulness, success, interest, and caring, which in turn, can lead to more favourable evaluations of teaching effectiveness. SETs should be used as a tool to inform instructors’ pedagogical decisions rather than for personnel evaluation. Limitations and future research directions are discussed, including examining the influence of student diversity on SETs and referencing more recent literature.

A practitioner reflection and response to students’ perceptions of assessment at Higher Education

Assessments are an integral component of university programmes. They have the formative function of being instrumental in gauging the level of student engagement, whilst also providing opportunity for feedback to enhance students’ learning. Moreover, assessments have the summative function of providing grades on which degree classifications are based. Therefore, it is crucial that assessments are presented in a format that engage students. Listening to the student voice is an essential step in designing appropriate assessments. The aim of this reflection piece is to review and critique how the student voice is captured. While students have various informal and formal opportunities to voice their views, here we chose to focus on processes that result in recorded data, namely, Student Evaluation of Teaching (SET) surveys, Module Evaluation Surveys (MES), and research on students’ experience of assessments. We conclude by outlining an example of our adjustments to assessments, based on the student voice.

Can you mitigate gender bias in student evaluations of teaching? Evaluating alternative methods of soliciting feedback

Student evaluations of teaching can be unreliable indicators of effective teaching and affected by implicit bias. We conduct a randomized experiment at a selective U.S. liberal arts college in which we vary both the instrument and timing at which we solicit student feedback to assess whether either intervention can mitigate gender disparities in qualitative evaluation comments. External evaluators with expertise in teaching evaluation scored the positivity, specificity, and constructiveness of student comments in control and treatment groups. We find neither intervention is successful in mitigating gender disparities. Students that receive an alternate prompt which highlights the potential for implicit bias and asks them to articulate and apply their own criteria for effective teaching penalize female faculty at similar rates as those receiving a more standard evaluation prompt. We also find no evidence that delaying solicitation of student feedback until the start of the next semester, when students face less stress and fatigue, reduces the gender gap in the positivity of feedback. Instead, we find substantial evidence that the positivity of qualitative student evaluations is affected by student, instructor, and course characteristics across all study conditions. The results suggest that bias in student evaluations may be difficult to mitigate.

A topic model analysis of students' gendered expectations using surveyed critiques of lecturers

Student evaluations of teaching (SET) have been examined over the years to better understand the student experience, with an increasing portion of literature exploring the presence of implicit bias in SET surveys against minority gender and ethnic lecturers. This study explores free-text comments made by students from a large public university in Australia over the period 2010–2016, using a semi-supervised statistical approach. Data were collected via surveys administered online at the end of each course to every student officially enrolled in that course via the learning management system, and completion of the surveys was voluntary. We build a probabilistic topic model which incorporates student and lecturer characteristics into the topic formation process. We make statistical inference on the effects of gender and cultural or language backgrounds based on the topic and word prevalence probabilities. The results showed clear separation of topics discussed between male and female lecturers. From a gendered perspective, our topic analyses have found that students are significantly more likely to critique female lecturers to improve on structural aspects of the course, as well as aspects of time management and control of the lecturing environment. In comparison, male lecturers were significantly more likely to be critiqued on specific aspects related to lecture delivery. Lecturers from non-English speaking backgrounds were more likely to be both critiqued and praised for the clarity of their delivery.

Enhancement of academic programs and student guidance based on Generative AI (GenAI) and Student Evaluations of Teaching (SETs)

Enhancement of academic programs and student guidance based on Generative AI (GenAI) and Student Evaluations of Teaching (SETs)

Bias intervention messaging in student evaluations of teaching: The role of gendered perceptions of bias

Many studies have documented discrepancies in student evaluation of teaching ratings between male and female instructors and between ethnic majority and minority instructors. Given the importance of such ratings to academic careers and the likelihood of potential intergroup bias, it is crucial that institutions consider approaches to mitigate such biases. Several recent studies have found that simple bias mitigation messaging can be effective in reducing gender and other biases. In the present research, students enrolled in several large Faculty of Science undergraduate courses at an Australian university were recruited on a volunteer basis via the course learning management system. Half of the participants were randomly assigned an intervention message highlighting potential biases relating to gender and language background. Data from 185 respondents were analysed using Bayesian ordinal regression models assessing the impact of message exposure on evaluation scores. Reading a bias intervention message caused students to significantly adjust their scores, with the nature of that change dependent on student and instructor characteristics. Among male students, the bias intervention message significantly increased scores for all except male instructors with English speaking backgrounds, for whom there was no significant impact of the message. In contrast, among female students, the bias intervention message significantly decreased scores for male instructors with English speaking backgrounds only. The sample showed an overall decrease in scores in the intervention group relative to the control group. This is the first study to detect a negative impact of bias intervention messaging on SET scores. Our results suggest students may not acknowledge their own potential bias towards instructors with whom they share similar demographic backgrounds. In conclusion, bias intervention messaging may be a simple method of mitigating bias, but it may lead to consequences in which one or more groups receive lower ratings as a result of the correction.

Teacher-tailored student evaluation of teaching as a formative lens for reflection on teaching: academics’ perspectives

ABSTRACT This paper explores how teacher-tailored student evaluation of teaching (TT-SET) facilitates academics’ reflection on and adaptation of their teaching practices. This case study involved 11 academics teaching English at a Vietnamese university, using interviews for data collection and thematic analysis for interpretation. The findings show that TT-SETs were perceived as promoting academics’ reflection on and understanding of teaching practices, motivating adjustments in current practice and plans for future changes. The efficacy of TT-SETs in fostering reflection varied depending on academics’ perceptions of their formative intent, validity, and reliability. Academics’ developing understanding of learning’s reciprocal nature and reflection can enhance their comprehension of the nature of learning. Recommendations for a successful implementation of formative TT-SET include refining TT-SET procedures by clearly delineating their formative goals and enhancing student engagement to ensure high-quality feedback. This study underscores the significance of tailored feedback mechanisms in supporting continuous professional development among academics.

Finding the Needle in the Haystack: Can Natural Language Processing of Students' Evaluations of Teachers Identify Teaching Concerns?

Institutions rely on student evaluations of teaching (SET) to ascertain teaching quality. Manual review of narrative comments can identify faculty with teaching concerns but can be resource and time-intensive. To determine if natural language processing (NLP) of SET comments completed by learners on clinical rotations can identify teaching quality concerns. Single institution retrospective cohort analysis of SET (n = 11,850) from clinical rotations between July 1, 2017, and June 30, 2018. The performance of three NLP dictionaries created by the research team was compared to an off-the-shelf Sentiment Dictionary. The Expert Dictionary had an accuracy of 0.90, a precision of 0.62, and a recall of 0.50. The Qualifier Dictionary had lower accuracy (0.65) and precision (0.16) but similar recall (0.67). The Text Mining Dictionary had an accuracy of 0.78 and a recall of 0.24. The Sentiment plus Qualifier Dictionary had good accuracy (0.86) and recall (0.77) with a precision of 0.37. NLP methods can identify teaching quality concerns with good accuracy and reasonable recall, but relatively low precision. An existing, free, NLP sentiment analysis dictionary can perform nearly as well as dictionaries requiring expert coding or manual creation.

Student Evaluation of Teachers’ Effectiveness (SETE) scale: translation, cross-cultural adaptation and psychometric properties in a Latin American sample

IntroductionTeacher effectiveness is a burgeoning field. Those responsible for educational policies seem increasingly committed to this fact, since it is one of the most important factors that influence the success of a university student; for this reason, the study of this topic has gained relevance in recent years. Therefore, an instrument with adequate psychometric properties is needed to measure this construct in Spanish-speaking countries.ObjectiveThe study had the purpose of translating, adapting, and evaluating the internal structure, providing evidence of reliability and validity of the Student Evaluation of Teachers’ Effectiveness (SETE) Scale in a Latin American sample.MethodsAn instrumental study was carried out and through convenience sampling, data were collected from 1,000 university students from South America belonging to a private educational network. The participants were between 18 and 40 years old (M = 21.25, SD = 2.99). Analyzes used Structural Equation Modeling (SEM) with AMOS 24 statistical software.ResultsConfirmatory Factor Analysis provided a 4-factor, 28-item fit model (CMIN/DF = 4.359; CFI = 0.956; SRMR = 0.030; RMSEA = 0.058). The results demonstrated good internal consistency (α = between 0.927 and 0.961; CR = between 0.927 and 0.962; AVE = between 0.646 and 0.799). Evidence of validity and reliability was obtained for the total sample.DiscussionThis adaptation and validation of the SETE scale makes it a valid, useful, reliable, and necessary tool that can be considered to evaluate teachers’ effectiveness from the perspective of university students.

Towards an improved of teaching practice using Sentiment Analysis in Student Evaluation

Student evaluation of teaching (SET) is an ad-hoc way of assessing teaching effectiveness in higher education institutions. In this paper, we present an approach to analyzing sentiments expressed in SET comments using a large language model (LLM). By employing natural language processing techniques, we extract and analyze sentiments expressed by students when the course has ended, aiming to provide educators and administrators with valuable insights into teaching quality and elements to improve teaching practice. Our study demonstrates the effectiveness of LLMs in sentiment analysis of comments, highlighting their potential to enhance the evaluation process. Our experiments with a crowdsourced tagged dataset show a 93% of accuracy in the classification of feedback messages. We discuss the implications of our findings for educational institutions and propose future directions for research in this domain.

Consideration of student evaluations of teaching (SET) and learning: perspectives of learning and teaching leaders through the ethical lens of “first, do no harm”

This paper explores institutional leaders’ perceptions of learning and teaching involved in facilitating and assessing Student Evaluations of Teaching (SET) survey instruments across Australian regional universities. It focuses on how they understand the function of SET, strategies used to mitigate bias, and potential residual harm. Through adopting a combination of inductive and deductive research processes and a thematic analysis through the ethical lens of nonmaleficence (first, do no harm), we report that leaders in learning and teaching perceive SET as a form of surveillance and Quality Assurance ‘performance’, recognise inherent biases inhabited in SET reports, and identify how these biases negatively impact academics through a lack of systematic harm mitigation strategies. The paper’s critical – and novel – contributions include an increased understanding of how SET inflicts harm towards women and other marginalised academic groups through systematic and authorised microaggressions and how SET contravenes universities’ duty of care to employees. It recommends an expansion of the principle of nonmaleficence beyond potential harm to research subjects, including those who undertake research or evaluation (such as academics), particularly if these impact them.

Student Ratings of Instruction: Updating Measures to Reflect Recent Scholarship

Student Ratings of Instruction: Updating Measures to Reflect Recent Scholarship

The Persistence of Gender Bias in Student Evaluations of Teaching: The Role of Gender Stereotypes

The Persistence of Gender Bias in Student Evaluations of Teaching: The Role of Gender Stereotypes

Thinking or talking about teaching? Student evaluation as an occasion for dialogue or reflection on teaching

Using student evaluations of teaching (SETs) as a quality assessment tool have long been a highly debated topic among both practitioners and scholars of higher education teaching. Rather than focus on whether SETs provide reliable measurements of quality, we explore what happens when SETs are implemented as or intended as quality development tools. We report from a qualitative study of how higher education teachers and managers perceive and use SETs as part of developing their own teaching, and focus on how the dialogue around teaching, which was intended as a part of the quality assessment procedure, is actualized (or not) in practice.

Small programs, big challenges: Reimagining the evaluation of clinical teaching in genetic counseling.

Within the health professions education system, a significant proportion of teaching and learning occurs in the clinical setting. As such, the need to measure effective teaching for accreditation standards, faculty development, merit pay, academic promotion, and for monitoring the safety of the learning environment has led to numerous universities developing instruments to evaluate teaching effectiveness in this context. To date; however, these instruments typically focus on the student perspective, despite evidence demonstrating that student evaluations of teaching (SETs) lack correlation with learning outcomes and are not a true measure of teaching effectiveness. This issue is further exacerbated in small health professional training programs, such as genetic counseling, where clinical teachers may only supervise 1-3 students per year. As a result, not only are SETs more confounded due to small sample sizes, but a direct conflict exists between respecting learner anonymity and providing timely and relevant feedback to faculty. In such contexts, even using SETs to evaluate the nature of the learning environment may be unreliable due to student concerns about identifiability and fear of retaliation for unfavorable evaluation. This paper will review the literature regarding SETs, barriers to this process within the clinical setting, and the unintended downstream consequences. Options for addressing issues related to the use of SETs will be considered, with particular focus on the process of reflection and the use of teaching consultations or peer support groups as a means to improve teaching effectiveness in this learning environment.

Perspectives and discourses on teaching evaluations in a South African university

ABSTRACT The paper reports on a study that explored selected lecturers’ perspectives and discourses on a university's Student Evaluation of Teaching (SET) policy in South Africa; particularly what the policy prioritised in terms of purpose and evaluation processes. It also reports on the lecturers’ reflections on the additional questions they included in the self-designed evaluation tools. A questionnaire, informal group conversations, and extended observations were used to collect data, and Latour (2005). Reassembling the Social: An Introduction to Actor-Network Theory. Oxford University Press and Latour (2013). An Inquiry Into Modes of Existence. Translated by C. Porter. London: Harvard University Press works helped make sense of the lecturers’ perspectives and discourses. Findings indicate a partial grasp of what the SET policy promotes. Lecturers’ understanding seemed to emphasise teaching evaluations’ professional development and accountability functions. Little attention was paid to the context in which teaching and learning occurred. The conclusion suggests ways in which the guidance given to lecturers could be improved to help them understand and work more effectively with their university’s SET policy.

If (my) 6 was (your) 9. Reporting heterogeneity in student evaluations of teaching

Student Evaluations of Teaching (SET) are subjective measures of student satisfaction that are often used to assess teaching quality. In this paper, we show that heterogeneity in students’ reporting styles challenges SET validity. Using administrative data that enable us to track all evaluations produced by each student, we isolate student-specific reporting scales. We show that reporting heterogeneity explains at least one third of the within-course variation in SET. We also document that students sort across elective courses according to their reporting style. As a result, the average evaluation of two otherwise identical electives can differ only because of heterogeneity in the reporting style of students attending them. Using a simulation exercise, we show that this type of sorting coupled with large sampling variability severely alter the ranking of courses within a major, calling into question the use of SET to incentivise teachers.

Performance and Perception of Creativity and Academic Achievement in Elementary School Students: A Normal Mixture Modeling Study

ABSTRACTCreativity can be assessed using various methods, including divergent thinking performance, self‐ratings, and teacher ratings. However, these measures may not always align, as they may not consistently identify creative potential in the same manner. The present study aimed to identify latent subgroups of students based on their observed originality, creative self‐efficacy, teacher‐rated originality, academic achievement in reading and mathematics, and demographic background characteristics. Data were collected from 243 elementary school students. We applied the normal mixture technique to classify participants into latent subgroups. Five latent subgroups of students were identified: Overconfident Low Performers, Creative High Achievers, Under‐Confident Below‐Average Achievers, Mathematically Oriented Students, and Calibrated Above‐Average Achievers. Female students tended to fall disproportionately into the subgroup of Creative High Achievers. Students receiving free/reduced lunch had a lower probability of being Creative High Achievers. Special education students had a higher probability of falling into the subgroup Overconfident Low Performers. Teacher ratings of students' originality were more in line with student academic performance rather than with their performance‐based originality scores. Students' self‐ratings of creativity bifurcated across subgroups, with Creative High Achievers and Overconfident Low Performers reporting the highest self‐ratings of originality, despite displaying very different levels of performance on the divergent thinking assessment.

Adopting Trauma-Informed Approaches to Teaching & Learning in Health & Exercise Sciences: a Case Study

The School of Health & Exercise Sciences at the University of British Columbia Okanagan offers a competency-based undergraduate program that prepares students for professional practice in kinesiology and allied health, clinical exercise physiology, and/or health behaviour change. In delivering this program, we aim to 1) provide a safe and inclusive learning environment for all students, and 2) intentionally prepare students for equity-minded and anti-oppressive professional practice. To support both of these goals, we have recently embarked on a multipronged initiative to implement trauma-informed approaches to teaching and learning across our program. Trauma-informed approaches are critical due to the high prevalence of trauma exposure in both the university student and general adult population. Since people from historically, persistently or systemically marginalized groups are more likely to have experienced trauma, trauma-informed approaches should be considered an essential tool for supporting equity and inclusion in higher education. Finally, trauma-informed practice is especially relevant in a discipline like ours, where learning activities and professional practice commonly involve close examination of the body, touching, and other potentially triggering interactions and events. We have taken several steps to support the adoption of trauma-informed teaching practices in our School. These include characterizing the need for trauma-informed approaches in our laboratory courses, selecting an appropriate framework to guide our recommendations, creating an educator guide for designing and facilitating trauma-informed learning experiences, and developing and launching a training program for instructors and teaching assistants. We have also made progress with integrating learning about trauma-informed approaches into our curriculum. To support backward design, we defined one program-level competency related to trauma-informed practice, with five associated learning outcomes. These learning outcomes have been mapped across the curriculum to support mastery of the competency by graduation. This year, we employed two different instructional approaches (large vs. small group and instructor-led vs. guest expert-led sessions) in one first- and fourth-year course (n=209 and 100 students, respectively). Formative assessments indicate that individual learning outcomes were attained by 72-91% of students. Summative assessments of learning and student evaluations of instruction and perceived learning are forthcoming. Here we describe a comprehensive effort to revise both pedagogy and curriculum in support of trauma-informed teaching and learning in health and exercise sciences. By sharing our process, early successes, and lessons learned, we offer a valuable example for educators and educational designers across disciplines related to human anatomy, physiology, and health. This work is happening on the traditional, ancestral, and unceded territory of the Syilx Okanagan people and is supported by the UBCO School of Health & Exercise Sciences and the UBCO Sexual Violence Prevention & Response Offce. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.