Student Evaluations Of Teaching Research Articles

Is it possible to redress noninstructional biases in student evaluation of teaching surveys? Quantitative analysis in accounting and finance courses

Several studies have reported that student evaluation of teaching (SET) presents important problems. First, depending on the area, there are significant differences in the evaluations. Second, numerous noninstructional biases exist, such as when those teachers who award better grades obtain better SETs. Correcting the rankings by considering these biases (e.g., adjusting SETs according to the class grade) has been proposed. In this paper, we analyse a third problem: it is impossible to correct the biases because they are specific to each area, level, and even class. On a sample of 15,439 SETs, we compared the biases present in two very close areas (accounting and finance) and at two levels (undergraduate and postgraduate). Then, we used a procedure based on the analysis of residuals in OLS models to eliminate area- and level-specific biases. However, there are still latent biases apparently linked to each specific group of students.

Measuring the Accuracy and the Potential for Standardizing Students’ Evaluations of Teaching: The Use of Measurable Bench-Marking Questions

Student evaluations of teaching (SETs) are widely used to quantitatively assess the competence of university lecturers. SETs can be used formatively to direct lecturers in improving their teaching, and more summatively in determining the suitability of a candidate for employment, confirmation in post or tenure, promotion, and performance-based salary adjustment or reward. The validity and suitability of SETs remains heavily contested, yet few papers provide workable recommendations to improve the validity of the tool or the interpretation. This results in perpetuating distrust for SETs, and, in cases, lecturers behaving to artificially enhance their scores. We present a method to assess the accuracy of SETs, either informally by the lecturer in understanding the outputs, or formally by the administrative body that distributes SETs and their scores. We provide recommendations for standardizations of score outputs using bench-marking questions that can highlight biased responses, and to adjust output scores and assessment selections accordingly.

Student motivations, perceptions and opinions of participating in student evaluation of teaching surveys: a scoping review

Several times each year the teaching performance of academics at higher education institutions are evaluated through anonymous, online student evaluation of teaching (SET) surveys. Universities use SETs to inform decisions about staff promotion and tenure, but low student participation levels make the surveys impractical for this use. This scoping review aims to explore student motivations, perceptions and opinions of SET survey completion. Five EBSCO® databases were searched using key words. Thematic analysis of a meta-synthesis of qualitative findings derived from 21 papers identified five themes: (i) the value students’ place on SET, (ii) the knowledge that SET responses are acted upon to improve teaching, (iii) assurance of survey confidentiality and anonymity, (iv) incentives for completing SET, and (v) survey design and timing of survey release. Perceptions, knowledge and attitudes about the value of SET are essential factors in motivating students to engage and complete SETs, particularly if surveys are easy to interpret, time for completion is incentivised and responses are valued.

Students’ online evaluation of teaching and system continuance usage intention: new directions from a multidisciplinary perspective

Student evaluation of teaching (SET), a major tool for assessing teaching quality in higher education is a crucial research topic. Among 13 studies published about online peer SET in Assessment & Evaluation in Higher Education over the past two decades, ease of use, clarity and helpfulness of SET information were repeatedly tested. This study introduces theory from the information systems field to give a multi-disciplinary view in testing how students’ perceptions may affect their intention to continue to use an online peer SET system. While past studies focused on ratemyprofessor.com, offering results from USA, Canada and UK, this study aimed to provide Asian insight by using data from Taiwan. Based on 364 student members of the selected website, findings indicated that disconfirmation of SET information significantly affected perceived usefulness, trust and satisfaction, ultimately shaping continuance usage intention of the online peer SET system. Practical implications for online peer SET website managers and institutional SET managers are discussed.

Why It Is Wrong to Use Student Evaluations of Professors as a Measure of Teaching Effectiveness in Personnel Assessments: An Unjust Risk of Harm Account

Abstract I argue that university supervisors should not use student evaluations of teachers (SETs) as a measure of teaching effectiveness in personnel assessments because the evidence suggests SETs likely violate several duties university supervisors have toward their instructional employees. I focus on the duty to not knowingly impose a wrongful risk of harm on nonconsenting and innocent others. Many university employers impose a wrongful risk of harm on instructors by not using relevant, merit-based performance indicators that have adequate construct validity, by using uncorrected indicators that likely perpetuate discrimination, and by incentivizing instructors to do wrong. The use of SETs imposes unjust risk of harm on all instructors, but the risk is higher for women, minorities, and those in precarious, teaching-focused roles. In conclusion, I tentatively suggest some other means of evaluating student learning and assessing instructors.

Student evaluation of teaching: reactions of Australian academics to anonymous non-constructive student commentary

Within Australian higher education, student evaluation of teaching (SET) is regularly conducted and data are utilised for quality control and staff appraisal. Within current methodologies, students can anonymously provide further feedback as written commentary. There is now growing evidence that, once this narrative becomes derogatory or abusive, it may have the potential to create harm. To investigate staff reactions to receiving anonymous non-constructive commentary, a one group point in time design was constructed, and a survey conducted. Participants (N = 741) from a broad cross-section of Australian universities responded to Likert questions asking about their reactions. A significant impact was revealed according to age for mental health, stress and professional confidence, with younger and tenured academics indicating the most vulnerability. There were no differences across gender. Non-health disciplines with teaching loads greater than 50% reported an impact of anonymous SET on mental health and professional confidence. Being casually or seasonally employed or from an ethnic background was shown to have a significant effect on professional confidence. Findings suggest that the potential for higher education academics to be harmed via this process is a continued risk and highlights the need for review and reform of SET systems and protocols.

Online Parameter Estimation for Student Evaluation of Teaching.

Student evaluation of teaching (SET) assesses students' experiences in a class to evaluate teachers' performance in class. SET essentially comprises three facets: teaching proficiency, student rating harshness, and item properties. The computerized adaptive testing form of SET with an established item pool has been used in educational environments. However, conventional scoring methods ignore the harshness of students toward teachers and, therefore, are unable to provide a valid assessment. In addition, simultaneously estimating teachers' teaching proficiency and students' harshness remains an unaddressed issue in the context of online SET. In the current study, we develop and compare three novel methods-marginal, iterative once, and hybrid approaches-to improve the precision of parameter estimations. A simulation study is conducted to demonstrate that the hybrid method is a promising technique that can substantially outperform traditional methods.

Contextualizing gender disparities in online teaching evaluations for professors.

Student evaluation of teaching (SET) is widely used to assess teaching effectiveness in higher education and can significantly influence professors' career outcomes. Although earlier evidence suggests SET may suffer from biases due to the gender of professors, there is a lack of large-scale examination to understand how and why gender disparities occur in SET. This study aims to address this gap in SET by analyzing approximately 9 million SET reviews from RateMyProfessors.com under the theoretical frameworks of role congruity theory and shifting standards theory. Our multiple linear regression analysis of the SET numerical ratings confirms that women professors are generally rated lower than men in many fields. Using the Dunning log-likelihood test, we show that words used in student comments vary by the gender of professors. We then use BERTopic to extract the most frequent topics from one- and five-star reviews. Our regression analysis based on the topics reveals that the probabilities of specific topics appearing in SET comments are significantly associated with professors' genders, which aligns with gender role expectations. Furtherly, sentiment analysis indicates that women professors' comments are more positively or negatively polarized than men's across most extracted topics, suggesting students' evaluative standards are subject to professors' gender. These findings contextualize the gender gap in SET ratings and caution the usage of SET in related decision-making to avoid potential systematic biases towards women professors.

New frontiers in student evaluations of teaching: university efforts to design and test a new instrument for student feedback

Student evaluations of teaching (SETs) are a ubiquitous feature of higher education. However, scholars have presented numerous challenges to the accuracy, validity, reliability and objectivity of SETs as a measure of teaching effectiveness. Given the potential for bias, the use of SETs in professional review may constitute a form of institutional discrimination. Therefore, institutions of higher learning need to develop, adopt and refine better methods for collecting and using student feedback. This paper describes the steps taken by a mid-sized comprehensive university in the USA over a three-year period to do that. We describe the work of our committee dealing with this issue, how we collaborated with the rest of the university to enact change, and the Learning Environment Survey (LENS) system that the university eventually selected and modified. We also report findings from a pilot study of the new instrument, which was favorably received by both students and faculty, and make recommendations for other institutions of higher education.

Factors Affecting Students Evaluation of Teachers: A Case Study Research in Albania

The reliability, efficacy, and ethics of students “judging” their professors at university are considered to be controversial and political issues. Students’ evaluations of teaching (SET) instruments are used by universities to rate teachers and teaching. However, we question whether or not such instruments are adequate. And to what degree? What are the limitations? Are there any biased interpretations in the responses of the students? In this article, we address some of the issues concerning the evaluation of teachers by students at the University of Durres, Albania, such as the direct correlation between grading and positive evaluation, gender of professors, et al. A quantitative methodology was used for the realization of this study. The instrument used in the study was a questionnaire designed to explore students' and academic staff attitudes, regarding students’ evaluations of teachers. The article presents the study results and recommendations for better, more effective SETs instruments that limit ethical implications in these evaluative processes.
 
 Received: 22 January 2023 / Accepted: 2 March 2023 / Published: 5 March 2023

Establishing a Student Evaluation of Online Teaching and Learning Framework Through Analysis of Existing Instruments

This study aims to establish a framework for student evaluation of online teaching and learning (SEOTL) through an analysis of SEOTL instruments implemented by universities and colleges. From a list of 131 R1 and 135 R2 institutions, we searched, reviewed, and identified 27 instruments for student evaluation of online teaching. A five-dimensional evaluation framework with 24 categories of elements was developed through an analysis of these instruments. There were 278 evaluation elements among the 27 instruments. We found that most instruments focus more on the Course and Instructor dimensions, with Instructor Facilitation and Learning Goals and Objectives elements occurring most frequently. However, Organization and Technology dimensions with Advising Availability and Adequacy, Registration Procedures, Support Services, and Online Help Desk elements were least included. This study has implications for administrators, instructors, instructional designers, and students.

Revisiting student evaluation of teaching during the pandemic

ABSTRACT The pandemic has placed unprecedented pressures upon staff and students alike. Yet performance management of academics including Student Evaluation of Teaching (SET) persists. The American Association of University Professors (AAUP) has intervened on this issue. We develop new methods enabling better treatment of pandemic-era SET. Analysis of UK National Student Survey (NSS) data suggests 85% of institutions meet reasonable performance expectations during the pandemic. Results emphasize the need for a more sensitive treatment of pandemic-era SET.

An exploratory analysis of major dropdowns in Student Evaluation of Teaching ratings in higher education

We present an exploratory study of Student Evaluation of Teaching (SET) surveys that focused on analyzing the major dropdowns that sometimes occur in SET scores of those university teachers who teach the same subject for at least two consecutive years. Specifically, the purpose of this study was to identify those SET survey items associated with major dropdowns in teacher evaluations. The sample comprised SET surveys ratings from one Spanish University over five years, for a total of 2,284 different teacher-subject pairs under analysis. The 5% percentile of the differences between the teacher and subject SET ratings in two consecutive years was used as the cut-off point to identify major dropdowns. Frequency tables were used for analysis. The results highlighted SET survey items related to teaching methodology as those most associated with major dropdowns in teachers’ ratings. In particular, the item concerning teachers’ compliance with class schedules showed the highest frequency of occurrence. Beyond the specific scores obtained or the specific items in which major dropdowns may take place, in this study we discuss the importance of incorporating major dropdowns analysis to SET surveys reports in the future.

Quantifying halo effects in students’ evaluation of teaching: a response to Michela

In Cannon and Cipriani (2022) we contributed to the literature on halo effects in student evaluations of teaching (SETs) by proposing and implementing a method to separate the effect of halo effects in student responses from an external measure of the item being assessed. Our paper has been criticised by Michela (2022). Many of his comments about problems with SETs are not directly relevant as they discuss issues other than halo. We re-visit our data and confirm that our conclusion that halo does not necessarily make SETs uninformative is correct. However, we do find heterogeneity in the importance of halo between SETs from two different campuses.

Hidden gems in student evaluations of teaching– a treasure trove of opportunities to improve teaching outcomes in accounting

Professors often overlook or even ignore the negative feedback in their Student Evaluations of Teachings (SETs) for many reasons. Most notably, negative feedback appears unimportant if the instructor is overtly amicable or teaches well. However, there is potential for teaching improvements through a detailed evaluation of students' negative feedback. In this study, we present empirical support that the negative feedback in SETs can be useful to understand how students perceive elements of classroom justice through the evaluation of negative comments by students. We qualitatively explore negative comments from multiple entry-level accounting courses at the post-secondary level to identify potential avenues for improvement in teaching, encourage all instructors to take negative comments seriously, and read them with a positive frame of mind. After presenting the themes derived from our analysis, we discuss teaching implications and potential for improvements in the classroom.

The effect of teaching style and academic motivation on student evaluation of teaching: Insights from social cognition.

Student evaluation of teaching (SET) is ubiquitous in higher education as a metric for assessing teachers, gaining student feedback, and informing faculty personnel decisions. It is thus imperative to examine the dimensions along which a teacher is judged. This study tested the application of the universal dimensions of social judgment (i.e., warmth and competence) in SET. A total of 108 psychology undergraduates (M age = 23.63, SD age = 3.14) in Singapore rated a fictitious teacher (i.e., either relationship-oriented or task-oriented) based on their interactions over a programmed online chat. Participants responded to the social judgment measures of warmth and competence and rated their academic motivation. Results indicated a higher SET rating for a relationship-oriented than a task-oriented teacher. Further, student academic motivation mediated the link between teaching style and judgment of competence. The findings extend the supremacy of warmth in the context of SET, thus supporting the application of social cognition literature to educational research. In addition, the findings suggest that fostering a match in task goals between a teacher and student improves ratings of teacher competence.

A Narrative Review of Student Evaluations of Teaching in Decolonial Praxis: Implications for Occupational Therapy Higher Education

Student evaluations of teaching (SETs) are the primary source for evaluating teaching effectiveness and are used for deciding tenure and promotion. However, as efforts to engage in a decolonial critique of higher education amplify, the use of SETs in teaching and learning requires scrutiny. A narrative review was used to address the research question of SET biases in decolonial praxis and what insights may be useful for OT decolonial praxis. We identify and describe two content areas: (a) SET biases and (b) recommendations for alternatives promoting OT decolonial praxis. A total of 92 articles were sourced from five databases. Of the 92 articles, 44 met the inclusion criteria: peer-reviewed across disciplines, written in English, research conducted in the US, and published between 2011–2021. SETs scores are affected by factors beyond the influence of the instructor. Twenty-nine factors contributing to SETs biases were grouped into three main categories: SETs biases against instructors, other biases from students, and SETs biases in processes. Alternative methodological approaches are highlighted that may mitigate the identified biases for application in decolonial praxis in OT higher education.

Bias in Student Ratings of Instruction: A Systematic Review of Research from 2012 to 2021

Student ratings of instruction (SRI) are commonly used to evaluate courses and teaching in higher education. Much debate about their validity in evaluating teaching exists, which is due to concerns of bias by factors unrelated to teaching quality (Spooren et al., 2013). Our objective was to identify peer-reviewed original research published in English from January 1, 2012, to March 10, 2021, on potential sources of bias in SRIs. Our systematic review of 63 articles demonstrated strong support for the continued existence of gender bias, favoring male instructors and bias against faculty with minority ethnic and cultural backgrounds. These and other biases must be considered when implementing SRIs and reviewing results. Critical practices for reducing bias when using SRIs include implementing bias awareness training and avoiding use of SRIs as a singular measure of teaching quality when making decisions for teaching development or hiring and promotion.

Institutional Approaches to Evaluate Teaching Effectiveness: The Role of Summative Peer Review of Teaching for Promotion and Tenure

A growing body of literature has identified student evaluations of teaching (SETs) as introducing bias against minority faculty members and not serving as a reliable or valid measure of teaching effectiveness. This lack of reliability and validity presents issues for university tenure and promotion committees, as these institutional processes necessarily require accurate, objective, and holistically informed modes of evaluation to recognize teaching achievements. Summative peer review of teaching (SPRT) is an alternative mode of assessment that aims to provide evidence of teaching effectiveness to inform promotion and tenure. SPRT, as an institutional practice, has been adopted at a small cohort of institutions of higher education, marking a potential shift in practice. This article examines SETs to articulate the problematic elements introduced by SETs, specifically to examine if SPRT can serve as a viable alternative. By describing the SPRT processes that four institutions have taken, the authors aim to articulate these emerging approaches to collecting evidence of teaching effectiveness. In this descriptive work, it is our secondary contention that SPRT, through intentional design and facilitation, can offer a process that does not introduce bias in the same way as SETs and thus, can also be used to satisfy the growing need for practices that help achieve, in part, institutional goals related to equity, diversity, and inclusion (EDI).

Impact of social media distraction on student evaluation of teacher effectiveness

PurposeThe study examines the impact of social media distraction on the relationship between student engagement and student evaluations of teachers (SET) in management education, part of the STEM disciplines. While it is widely accepted that student engagement and evaluation of teachers are two important constructs of active learning systems, their links, especially in the context of management education, have not been explored. This study seeks to fill this research gap as teachers' appraisal and career growth greatly depend on SET across higher education institutes worldwide.Design/methodology/approachData were collected from 247 student participants from higher education institutes/universities using a questionnaire-based survey. Further, 15 students were interviewed to glean insights from their feelings regarding social media distractions and how it impacts their engagement and evaluation of teachers. For the quantitative and qualitative analyses, multiple regression and thematic analysis were applied, respectively. A word tree was developed to determine the association between experiences and contexts.FindingsThe results indicate that social media distraction moderates the relationship between students' cognitive engagement and their evaluation of teaching faculty. Social media distractions disrupt student engagement inside the classroom, leading to wrong/biased assessments of teachers' performance. Additionally, cognitive and emotional engagement significantly impacted student evaluations of the management college faculty.Originality/valueThe authors’ findings enable educators in higher education systems, specifically management education, to rethink the faculty feedback system and social media distractions impeding student engagement other than SET drawbacks.