Use Of Student Evaluations Research Articles

Making Sense of Today’s Use of Student Evaluations of Teaching (SET)

Making Sense of Today’s Use of Student Evaluations of Teaching (SET)

Nurturing learning development through student feedback

Student evaluations of both teaching and student services are increasingly embedded into higher education. There is debate surrounding the reliability, effectiveness, and bias of such evaluations (Hefferman, 2022) and National Student Survey (NSS) results (Office for Students, 2022) show that students typically respond poorly to questions relating to their learning community and their opportunities to give feedback on their experiences (student voice).
 
 After receiving ethical approval to conduct research on evaluations, four students and a member of staff worked together to address how staff and students within the School of Mathematics and Statistics engage with student evaluations. Two surveys were conducted, the first aimed at staff (63 responses, 90% response rate) and the second at students (53 responses, 17% response rate). The results suggested that both staff and students agreed that evaluations are necessary and useful in building relationships. While staff implement the feedback they receive, students currently do not see it, and their learning may not benefit from being part of this process.
 
 When asked to describe the purpose of student evaluations, participating staff expressed that they provide students with opportunities to have direct input to courses, influence their learning environment, and feel part of the school. Students expressed that their feedback could improve a course’s content, quality and delivery, and provide a learning opportunity for lecturers. Students indicated a preference for informal mid-term feedback since they could see their feedback acted upon in real time. In response, we propose the use of student evaluations as a feedback dialogue tool to encourage and enhance relationships between staff and students and help develop self-regulated learning. We will exemplify a feedback system that uses short, direct, and frequent surveys that students complete at the time of learning (Rowland, 2021), providing time to reflect on learning and creating a line of dialogic communication with the lecturer who can respond to the feedback to inform future learning. The system is applicable to any continuous student-staff learning-focused interaction.

Student Evaluations of Teaching: Understanding Limitations and Advocating for a Gold Standard for Measuring Teaching Effectiveness

The arbitrator’s decision in Ryerson University v Ryerson Faculty Association [2018] CanLII 58446 (ON LA) rejected use of Student Evaluations of Teaching (SETs) for academic confirmation and promotion purposes. SETs provide largely quantitative data in response to pre-determined institutional, generic questions using a Likert scale applicable to all teaching modes. SETs may be efficient, but commonly low response rates mean the data is often statistically invalid. Studies of SETs suggest gender, age, race, and other biases are widespread, and they discourage teaching innovation because academics fear student backlash in SET scores. Consequently, SETs are of little value to academics for their professional development, confirmation or promotion, or as evidence for teaching grant or awards processes. The continuing impact of the COVID-19 pandemic on traditional models of teaching has forced many changes in teaching, learning and pedagogy, often with a temporary suspension of SETs to allow teachers to innovate without negative impact on professional development measures. This presents a unique opportunity for us to revisit how the effectiveness of teaching and learning is measured. Academic teaching staff still need evidence of teaching effectiveness, as do sessional staff looking for continued employment and/or a career in academia. This paper discusses the strengths and weaknesses of SETs; seeks to equip law academics to advocate for other measures of teaching effectiveness that better reflect their contribution to student learning; and to pave the way for law discipline and institutional level changes that support a gold standard in measuring teaching effectiveness beyond reliance on SETs, for the benefit of teachers in law and other disciplines.

Evaluating and Improving the Formative Use of Student Evaluation of Teaching

Student Evaluations of Teaching (SETs) are a ubiquitous tool in higher education. Though they are not effective means of evaluating teaching ability, they are useful in formative teaching development of a teaching career. We characterise the current formative use of and attitudes towards SETs by instructors across all disciplines (STEM and non-STEM). We found that tenured instructors used SETs for formative development more than untenured, and that non-STEM instructors had more negative associations with SETs than STEM instructors. Based upon these data, we make recommendations to redesign the SET instrument and change the way in which the data are used to support formative teaching development.

Teacher Appraisers' Perceptions of Teacher Appraisal

This study investigates teacher appraisers' perceptions of teacher appraisal in terms of its purpose, methods, and criteria. Fourteen teacher-appraisers were administered semi-structured interviews to gain an in-depth understanding of their perceptions of appraisal. Analysis of the responses revealed that participants seemed to advocate the use of teacher appraisal for both professional improvement and decision-making purposes. Participants also seemed to agree about the value of classroom observation as an objective method of evaluation. Still, they were divided regarding the use of student evaluation, student achievement, and other indi-rect observation methods, especially for decision-making purposes. As for evaluation criteria, the results revealed that while teacher-appraisers paid attention to all aspects of the teaching process in their evaluation, their focus remained mainly on those skills exhibited during the actual delivery of teaching.

We can work it out: faculty interpretation of student evaluations of teaching

ABSTRACT Despite previous literature strongly criticising the use of student evaluations of teaching (SETs), it is unlikely that the frequency of SETs will be decreased for university instructors. However, little attention has been given to how we can make SETs work as a tool for improving teaching effectiveness. This research explores how faculty members of a large professional master’s programme in Brazil interpret the results of their SETs. The findings show that qualitative questions combined with quantitative scores are effective in improving classes. It also shows that faculty critically assess students’ feedback by disregarding emotional positive and negative comments and looking for suggestions in comments. Instructors with lower scores have difficulty interpreting feedback and tend to attribute their performance to nonresponse bias. This research contributes to the SET literature by showing that a combination of qualitative and quantitative questions can be effective in improving teaching quality but that some instructors may need support in interpreting SET results to change their teaching.

Investigating faculty perception of instrument used for evaluating teaching effectiveness: A case of two universities in Sunyani

This study aimed at examining the perception of faculty members towards the use of student evaluations of faculty members as a basis to assess teaching effectiveness and for decision-making for annual faculty review. The study was conducted with the participation of 62 full-time members at Catholic University and Sunyani Technical University. Quantitative and qualitative data were collected through questionnaires and interview guides. The main findings revealed that faculty members positively perceive the current Students’ Evaluation of Faculty (SEF) instrument as effective in evaluating teaching effectiveness but do not perceive the use of the current instrument as effective in decision-making for annual review. The study also revealed that there was no association between the objectives for the adoption and usage of the Students’ Evaluation of Faculty (SEF) instrument for private and public universities. Finally, the findings show that faculty are challenged both administratively and humanly in their endeavour to effectively implement SEF. Based on the findings, it was concluded that the evaluation method and instrument needed to be revised further to include a multidimensional procedure. This multifaceted method should produce independent instruments for annual faculty reviews and teaching improvement. Keywords: Faculty evaluation, perception, teaching effectiveness, decision-making.

We Can Work it Out: Faculty Interpretation of Student Evaluations of Teaching

Despite previous literature strongly criticizing the use of student evaluations of teaching (SETs), it is unlikely that the frequency of SETs will be decreased for university instructors. However, little attention has been given to how we can make SETs work as a tool for improving teaching effectiveness. This research explores how faculty members of a large professional master’s program in Brazil interpret the summative and formative results of their SETs. The findings show that formative comments combined with summative scores are effective in improving classes. It also shows that faculty critically assess students’ feedback by disregarding emotional positive and negative comments and looking for suggestions in comments. Less effective instructors have difficulty interpreting feedback and tend to attribute their performance to nonresponse bias. This research contributes to the SET literature by showing that a combination of formative and summative questions can be effective in improving teaching quality but that some instructors may need support in interpreting SET results to change their teaching.

Evaluating Cluster-Level Factor Models with lavaan and Mplus

Background: Researchers frequently use the responses of individuals in clusters to measure cluster-level constructs. Examples are the use of student evaluations to measure teaching quality, or the use of employee ratings of organizational climate. In earlier research, Stapleton and Johnson (2019) provided advice for measuring cluster-level constructs based on a simulation study with inadvertently confounded design factors. We extended their simulation study using both Mplus and lavaan to reveal how their conclusions were dependent on their study conditions. Methods: We generated data sets from the so-called configural model and the simultaneous shared-and-configural model, both with and without nonzero residual variances at the cluster level. We fitted models to these data sets using different maximum likelihood estimation algorithms. Results: Stapleton and Johnson’s results were highly contingent on their confounded design factors. Convergence rates could be very different across algorithms, depending on whether between-level residual variances were zero in the population or in the fitted model. We discovered a worrying convergence issue with the default settings in Mplus, resulting in seemingly converged solutions that are actually not. Rejection rates of the normal-theory test statistic were as expected, while rejection rates of the scaled test statistic were seriously inflated in several conditions. Conclusions: The defaults in Mplus carry specific risks that are easily checked but not well advertised. Our results also shine a different light on earlier advice on the use of measurement models for shared factors.

What student evaluations are not: Scholarship of Teaching and Learning using student evaluations

In this Editorial, we stay committed to the objective of the Journal of University Teaching and Learning Practice regarding sharing, evaluating, and developing stronger evidence-based practice papers by focusing on the topic of national and institutional student evaluations. We create an important theoretical and practical foundation for authors considering publishing with our Journal on studies that utilise student surveys as their primary method of data collection. The editorial begins by providing a comprehensive overview of the history and emergence of student evaluations dating back to medieval times, we trace the evolution of student evaluations to present day looking at the rationale behind the induction of such tools. Following this, we discuss the validity of student evaluations through an exploration of factors such as student satisfaction, the timing of when student surveys are administered, and the idiosyncrasies regarding paper-based and online evaluations. We then further discuss the reliability of student evaluations by contextualising what student evaluations do not say and uncover how various forms of bias can influence the ways student evaluations are both completed and interpreted. Through this we assert that due to confounding factors of bias that influence the results of student evaluations, they cannot always be thought of as wholly objective data collection tools. This then leads into our discussion of the contemporary social contexts within which student evaluations are situated and how both micro and macro dynamics influence student experiences of teaching and learning, where we contest that broader external factors experienced by students can skew the ways teaching is both perceived and evaluated. We conclude our Editorial by critically envisioning a new direction for future manuscript submissions to our Journal. We assert that although the use of student evaluations as evidence of teaching practice may be inherently flawed, there nonetheless remains merit in their use following critical and reflexive engagement throughout the research process. As such, we are hopeful that our critical review of student evaluation-based scholarship may be utilised to leverage higher quality research output.

Qualitative Analysis of Students’ Evaluations – The Case of the South East European University (SEEU) in North Macedonia

Abstract The use of student evaluations of teaching (SET) has become a widespread practice in higher education despite inconclusive evidence reported in the literature around its validity. Not surprisingly, the question of the validity of SET continues to be a current debate in higher education, pointing to the need for more research in this area. This paper is a part of a larger scale study, which aims to contribute to broadening the knowledge and understanding of SET validity by analysing the process within the South East European University (SEEU) in North Macedonia in order to determine whether student evaluations are objective and critical. A likert scale questionnaire, containing 9 questions, was designed for the purpose of the analysis. The questionnaire was sent to all students from the five (5) Faculties: Business and Economics, Law, Contemporary Sciences and Technologies, Contemporary Social Sciences and Faculty of Languages, Cultures and Communications, in both campuses, Tetovo and Skopje. Three hundred and thirty three (333) students participated in the survey. Statistical Package for Social Science (SPSS) was used for analysing the results. Findings revealed that the information students received about the reputation, experience and qualifications of the professors had the highest influence on their perceptions, which in turn influenced the evaluations. As an addition, the present paper also compares two methods on a data set of actual SET. For illustrative purposes, only data from one faculty have been analysed. It is shown that the traditional method of considering the average values can misrepresent a teacher’s performance as it can be highly sensitive to any extreme grades, being either very positive or very negative.

Use of Student Evaluation of Teaching (SET) Survey to Evaluate Effectiveness of Teaching in a Leadership Course among Dental Students over Three Years.

Leadership courses are being increasingly integrated into dental curricula. The study aimed to assess the validity and reliability of student evaluation of teaching (SET) instrument among dental students and to evaluate the effectiveness of teaching in a new leadership course over a period of three years. This cross-sectional study was conducted on fourth-year undergraduate dental students (N = 260) who took a practice management course over three consecutive years from 2014 to 2016. A 29-item SET questionnaire was administered among students who were willing to participate in the study. Out of 260 students, 185 returned completed surveys and the response rate was 71.15%. Factor analysis (principal component analysis) showed the validity of four dimensions of the SET instrument. Total variance explained by four dimensions was 62.80%. Cronbach's alpha for the instrument was 0.95 and each dimension had fairly high internal consistency (>0.80). Treating students with respect (94%), accepting different viewpoints of students (94.1%), being flexible/open-minded (92.5%), and preparedness in the course (91.9%) were the most common effective teaching traits. Over the period of three years, 16 items showed improvement in teaching and there was a significant improvement in four items (P < 0.05). In conclusion, it was found that SET is a valid instrument to evaluate the effectiveness of teaching in nonclinical courses in dentistry. This instrument should be used longitudinally to compare the effectiveness of teaching.

Current Use of Student Evaluations of Teaching in Anatomy Education

Anatomy, histology, and embryology (or anatomical sciences) provide a framework for understanding normal function as well as disease and its underlying mechanisms. Thus, they are often taught (by basic scientists) in the context of their clinical relevance. Delivery formats and settings vary widely: anatomy, histology, and embryology can be combined in a single course or integrated into other courses at different points in a curriculum. Anatomy labs are taught with the expectation (by students) that faculty know every detail, be it taught in the form of dissection, prosection, or virtually. The uniqueness of teaching contexts in anatomical sciences education can make it challenging to document effective teaching and to obtain feedback from students that provide growth opportunities to develop as an educator. Educators in anatomical sciences are charged with facilitating learning of skills and knowledge that are foundational and can be applied to many health sciences fields. Because anatomical sciences can be taught virtually as well as practically, additional professional development needs arise. But carving out time for development activities is challenging. Some educators teach several different levels of learners, while managing donor programs, and coordinating laboratory use, all competing demands on their time. Simultaneously, curricular changes have reduced anatomical sciences curricular hours to an all‐time low, and education is shifting towards active learning modalities with increased digital learning and reduced classroom and laboratory teaching. These modalities, if not applied expertly, can increase student pushback. Topics explored in this presentation include if and how our methods of evaluating teaching effectiveness have evolved with the changed landscape in which anatomical sciences faculty teach, and how to support faculty in their development as educators by constructive use of student evaluations of teaching.

Discrepancies in purposes of student course evaluations: what does it mean to be \u201csatisfied\u201d?

Student evaluation of teaching is a multipurpose tool that aims to improve and assure educational quality. Improved teaching and student learning are central to educational enhancement. However, use of evaluation data for these purposes is less robust than expected. This paper explores how students and teachers perceive how different student evaluation methods at a Norwegian university invite students to provide feedback about aspects relevant to their learning processes. We discuss whether there are characteristics of the methods themselves that might affect the use of student evaluation. For the purpose of this study, interviews with teachers and students were conducted, and educational documents were analysed. Results indicated that evaluation questions in surveys emerged as mostly teaching-oriented, non-specific and satisfaction-based. This type of question did not request feedback from students about aspects that they considered relevant to their learning processes. Teachers noted limitations with surveys and said such questions were unsuitable for educational enhancement. In contrast, dialogue-based evaluation methods engaged students in discussions about their learning processes and increased students’ and teachers’ awareness about how aspects of courses improved and hindered students’ learning processes. Students regarded these dialogues as valuable for their learning processes and development of communication skills. The students expected all evaluations to be learning oriented and were surprised by the teaching focus in surveys. This discrepancy caused a gap between students’ expectations and the evaluation practice. Dialogue-based evaluation methods stand out as a promising alternative or supplement to a written student evaluation approach when focusing on students’ learning processes.

Group-devised screenplays and film projects in Higher Education curriculum

Group work is a well-recognised pedagogical practice in Higher Education (HE), meant to facilitate peer collaboration in contexts simulating realistic industry situations. While group work is credited with helping students to develop a range of communication, social and negotiation skills, it can also disenfranchise students, leading to disorganisation and creative disputes. With increased numbers of students taking up HE film and television and creative writing degrees, group work is necessary to prepare students for industry practice. However, with limited time within a semester to produce such artefacts as screenplays and short films, group-based projects can backfire, creating tensions and conflicts, and resulting in decreased student satisfaction. This study discusses the process of implementation and evaluation of a redesigned screenwriting model, where the act of restructuring the group work component of the studies to ensure that all students participate in the writing of the team’s screenplay helped shift the unequal power balance between students and boost individual student agency. While contextualised in the field of screenwriting and filmmaking, the findings pertaining to group work dynamics and the mindful use of student evaluations to improve teaching can be transported to other disciplines and contexts. Personal skills developed through HE group work, such as collaboration and negotiation, may be applied to other real-world industry and international settings.

Student Evaluations of Teaching: Is There a Relationship between Student Feedback on Teaching and the Student Final Grade?

The use of Student Evaluations of Teaching (SET) has become widespread practice in higher education despite inconclusive evidence reported in the literature around its validity. Not surprisingly, the question of the validity of SET continues to be a current debate in higher education, pointing to more research to be conducted in this area. The current study contributes to broadening knowledge and understanding on the validity of SET by drawing on an online unit evaluation completed by students (n=2430 out of total student enrolment of N=7757) in one university across three postgraduate education programs over a two-year period, to determine whether there is a relationship between student feedback on teaching and student final unit grade. Findings revealed that students who achieved very high or very low final unit grades did not participate in the SET, while students who achieved Pass or Credit grades partook in the SET, thus providing feedback. This indicates that teaching and evaluating staff need to be aware that a large subset of their students that are not providing feedback to staff to improve the quality of their courses.

On the Use of Student Evaluation of Teaching: A Longitudinal Analysis Combining Measurement Issues and Implications of the Exercise

Multi item questionnaires are widely used to collect students’ evaluation of teaching at university. This article makes an attempt to analyse students’ evaluation on a broad perspective. Its main aim is to adjust the evaluations from a wide range of factors which jointly may influence the teaching process: academic year peculiarities, course characteristics, students’ characteristics and item dimensionality. By setting the analysis in a generalised mixed models framework a large flexibility is introduced in the measurement of the quality of university teaching in students’ perception. In that way we consider (1) the effects of potential confounding factors which are external to the process under evaluation; (2) the dependency structure across units in the same clusters; (3) the assessment of real improvement in lecturers’ performance over time and (4) the uncertainty related to the use of an overall indicator to assess the global level of quality of the teaching as it has been assessed by the students. The implications related to a misuse of the evaluation results in implementing university policies are discussed comparing point versus interval estimates and adjusted versus unadjusted indicators.

Should Student Evaluation of Teaching Play a Significant Role in the Formal Assessment of Dental Faculty? Two Viewpoints

Student evaluation of teaching (SET) is often used in the assessment of faculty members’ job performance and promotion and tenure decisions, but debate over this use of student evaluations has centered on the validity, reliability, and application of the data in assessing teaching performance. Additionally, the fear of student criticism has the potential of influencing course content delivery and testing measures. This Point/Counterpoint article reviews the potential utility of and controversy surrounding the use of SETs in the formal assessment of dental school faculty. Viewpoint 1 supports the view that SETs are reliable and should be included in those formal assessments. Proponents of this opinion contend that SETs serve to measure a school's effectiveness in support of its core mission, are valid measures based on feedback from the recipients of educational delivery, and provide formative feedback to improve faculty accountability to the institution. Viewpoint 2 argues that SETs should not be used for promotion and tenure decisions, asserting that higher SET ratings do not correlate with improved student learning. The advocates of this viewpoint contend that faculty members may be influenced to focus on student satisfaction rather than pedagogy, resulting in grade inflation. They also argue that SETs are prone to gender and racial biases and that SET results are frequently misinterpreted by administrators. Low response rates and monotonic response patterns are other factors that compromise the reliability of SETs.

Maximising resource allocation in the teaching laboratory: understanding student evaluations of teaching assistants in a team-based teaching format

ABSTRACTMinimal research papers have investigated the use of student evaluations on the laboratory, a learning medium usually run by teaching assistants with little control of the content, delivery and equipment. Finding the right mix of teaching assistants for the laboratory can be an onerous task due to the many skills required including theoretical and practical know-how, troubleshooting, safety and class management. Using larger classes with multiple teaching assistants, a team-based teaching (TBT) format may be advantageous. A rigorous three-year study across twenty-five courses over repetitive laboratory classes is analysed using a multi-level statistical model considering students, laboratory classes and courses. The study is used to investigate the effectiveness of the TBT format, and quantify the influence each demonstrator has on the laboratory experience. The study found that TBT is effective and the lead demonstrator most influential, influencing up to 55% of the laboratory experience evaluation.

Evaluating Engagement Online

Abstract. Past experimental research has shown that women are penalized with harsh evaluations when they violate gender prescriptions to be nurturing and helpful. Instructor participation in an asynchronous online discussion forum and end-of-class evaluation data from 360 courses was used to test the hypothesis that students would penalize female, but not male, online instructors based on their classroom engagement. Results showed a penalties effect in student ratings for low-participating female, but not male, instructors in gender-balanced courses. The results demonstrate the differential impact of instructor engagement on male and female evaluations, shedding light on when and why gender bias is found in student evaluations. Implications for the use of student evaluations of faculty are discussed.