Ratings Of Instruction Research Articles

Learning geographical information systems through first principles of instruction – effects on student experiences and geocapabilities

ABSTRACT The development of geocapabilities has been a concern in geographical education, and the application of geospatial technologies such as geographical information systems (GIS) has been popularized in both geography and non-geography courses. However, there are still gaps in how GIS courses could be designed to promote students’ geocapabilities. In order to facilitate GIS teaching and learning rooted in educational theory, we designed a course on story maps based on the First Principles of Instruction. We examine students’ perceptions of learning experiences and geocapabilities through two rounds of course delivery and student survey. According to student feedback and instructor ratings, the designed course activities supported their learning of subject content and developed both of their geography-related and general skills. However, the objective for the development of geocapabilities was found to be too broad or vague for students. We discuss the application of First Principles of Instruction, the relevance of considering geocapabilities, and the implication of the study findings for GIS pedagogy.

Strengthening the writing component in upper level economics electives

Writing-intensive economics courses are courses that use writing as a central feature of instruction and evaluation of students’ work. This study presents and evaluates teaching improvements on these basic structures. Using teaching evaluations data from eight public finance sections taught between Fall of 2012 and Spring of 2019, an ordered probit model estimates the value, as rated by students, of the following improvements to enhance the writing quality of student papers: (1) explicit grading rubrics; (2) funding a dedicated course specialist consultant (CSC) from the writing center; (3) engaging librarians in creating a literature review guide; (4) instructor blogging. The empirical model estimates the contribution of the improvements to the overall instructor and course evaluation ratings as well as to the writing component student ratings. Empirical results show that grading rubrics had a statistically significant positive effect on student assessments of writing component design and writing skills development. Engaging a dedicated CSC improves writing skills ratings. Instructor blogging had a statistically significant positive effect on instructor ratings while literature review guide improved course ratings. Only an expected grade had a significant positive effect on all student rated course outcomes. The goal of this paper is to provide ideas for enhancing economics content through writing.

Sustainable education on improving the quality of peer assessment: design and implementation of an online deep learning-based peer assessment system

PurposeAs global issues such as climate change, economic growth, social equality and the wealth gap are widely discussed, education for sustainable development (ESD) allows every human being to acquire the knowledge, skills, attitudes and values necessary to shape a sustainable future. It also requires participatory teaching and learning methods that motivate and empower learners to change their behavior and take action for sustainable development. Teachers have begun rating pupils based on peer assessment for open evaluation. Peer assessment enables students to transition from passive to active feedback recipients. The assessors improve critical thinking and encourage introspection, resulting in more significant recommendations. However, the quality of peer assessment is variable, resulting in reviewers not recognizing the remarks of other reviewers, therefore the benefits of peer assessment cannot be fulfilled. In the past, researchers frequently employed post-event questionnaires to examine the effects of peer assessment on learning effectiveness, which did not accurately reflect the quality of peer assessment in real time.Design/methodology/approachThis study employs a multi-label model and develops a self-feedback system in order to use the AIOLPA system in the classroom to enhance students' learning efficacy and the validity of peer assessment.FindingsThe research findings indicate that the better peer assessment through the rapid feedback system, for the evaluator, encourages more self-reflection and attempts to provide more ideas, so bringing the peer rating closer to the instructor rating and assisting the evaluator. Improve self-evaluation and critical thinking for the evaluator, peers make suggestions and comments to help improve the work and support the growth of students' learning effectiveness, which can lead to more suggestions and an increase in the work’s quality.Originality/value ESD consequently promotes competencies like critical thinking, imagining future scenarios and making decisions in a collaborative way. This study builds an online peer assessment system with a self-feedback mechanism capable of classifying peer comments, comparing them with scores in a consistent manner and providing prompt feedback to critics.

Student and Instructor Ratings in Geographic Information Systems: A Comparative Analysis

Geographic information system (GIS) education empowers engineering students to make informed decisions, integrate comprehensive data, and communicate effectively through maps and visualizations. In GIS education, it is common to employ problem-based learning, which can benefit from the advantages of peer assessment methods. Among the benefits of peer assessment are the enhancement of students’ capacity for analysis and synthesis, improvement in organizational and professional communication skills, and the development of critical judgement. However, a consequence of its application is that there may be variation in students’ final grades, with no consensus in the literature on this matter. This paper explores the extent to which the application of peer ratings among students can modify student grades in the field of GIS education. This was achieved by conducting an experiment in which undergraduate engineering students assessed two problem-based learning activities carried out by their peers in two different basic GIS courses. The ratings obtained after the peer assessment were compared with the grades given by the instructors. The results allowed us to debate whether the teaching benefits of this strategy compensate for the differences between the students’ grades and those given by instructors. Although no clear pattern was found in the mean ratings awarded by the two groups of evaluators, the results show that student engagement in peer assessment was high. This experience has demonstrated that the assessments of the two groups complement each other and allow students to gain a better understanding of their ratings and how to improve their skills.

Course-level factors and undergraduate engineering students’ ratings of instruction

ABSTRACT Post-course instructor ratings are a common practice at universities in Europe, Australia, and North America. Rather than solely describing teaching practice, however, such ratings may be associated with a range of non-pedagogical factors. We explored engineering students’ instructor ratings at a large United States institution, investigating relationships between overall instructor rating and some of those non-pedagogical factors, including subject area, class size, and students’ course grades. Instructor ratings were more favourable in humanities courses than in science or math courses, and students gave higher ratings to instructors who taught smaller class sizes. The strongest relationship existed between overall rating and students’ course grades: students who received A’s rated instructors an average 0.84 points higher on a 6-point scale than students who received F’s. Students who withdrew from a course provided the lowest instructor ratings. When instructor ratings are used as metrics of teaching ability in discussions related to promotion, tenure, or salary adjustments, there is an inherent inequity in the system that must be acknowledged beyond already well-documented biases, as non-pedagogical factors often outside of the instructor’s control may be significantly associated with those ratings.

Easy expectations and racial bias in economics instructor ratings

This paper uses Rate My Professors (RMP) data for instructors at two Canadian universities to investigate the determinants of economics instructor ratings, including the impact of easy grading expectations (or “easy expectations”) and various potential student biases related to ethnicity, gender, and accent. Regression analysis, including random effects panel data analysis and multilevel modelling, indicates that easier courses with lower difficulty levels and higher grades awarded to students are significant determinants of better instructor ratings. In addition, lower difficulty levels and higher grades tend to be associated with contract instructors compared to full-time instructors. The effect of instructor accent was insignificant. Our findings suggest that the ratings of economics instructors suffer from the same biases related to course difficulty, possibly attributable to “easy expectations,” and racial bias, as has been generally found in student ratings across academic disciplines. To the extent that instructor ratings are driven by “easy expectations” and racial bias, and that RMP ratings are consistent with formal university instructor ratings, the case for basing promotions, tenure decisions, and salary raises on average instructor ratings is weak.

A Comparison of Cognitive and Social Presence in Online Graduate Courses: Asynchronous vs. Synchronous Modalities

Over the last decade, online courses have continued to expand, and students in higher education are being offered increased access to technology and communication tools in online learning programs. This action research study analyzed the impact of two distinct types of online course instruction (100% asynchronous and weekly online synchronous meetings) on learning outcomes, including cognitive and social presence, knowledge gained, and student perceptions. Study participants consisted of graduate students enrolled in online sections of a course on program evaluation. Four sections of the course were available: two included a synchronous meeting using web-conferencing, and two used an asynchronous format. A quasi-experimental design was used and included a pre-post test knowledge assessment, a modified version of the Community of Inquiry questionnaire (CoI), and end-of-course student evaluations. The mean ratings of the CoI in this study ranged from 3.75–4.60 out of 5. There was a significant difference in the cognitive presence scores for synchronous (M=4.26, SD=.529 asynchronous (M=4.47, SD=.454) conditions; t(97)=-2.07, p =.041. Our results suggest when students learn in an asynchronous format, they have a higher cognitive presence. The average scores on the knowledge pre-test were the same for both sections but post-test scores were slightly higher in the asynchronous section. Instructor ratings were high for all courses. These findings may offer valuable implications to higher education programs that have recently transitioned to online teaching modalities.

Behavioral Indicator-Based Initial Flight Training Competency Assessment Model

Ensuring training safety is paramount to flight schools. In response to the inadequacy of traditional flight training assessment for comprehensive quantitative evaluation of cadet competency, an initial flight training competency assessment standard based on behavioral indicators was developed and optimized using the VENN model. Firstly, the Assessor Score Measurement Form (ASMF) was constructed according to the requirements of the Training Evaluation Worksheet specification, such as typical subjects, observations, and completion criteria. Secondly, based on the basic principles of the experience of the flight expert and the Competency-Based Training and Assessment (CBTA), a matrix of correlations between the observations and each competency-based behavioral indicator was created to construct a competency assessment matrix. In addition, a two-dimensional model for representing competency items characterized by behavioral indicators was established and an optimization model for competency assessment criteria was constructed. Finally, through combining actual flight training data, the proposed method was validated in the flight screening check phase. The results show that the optimized flight training competency assessment scheme can be well quantified and matched to real instructor ratings with an accuracy of 84%. The assessment worksheet, the assessment matrix, and the VENN competency rating model can be adapted to the different teaching requirements of each flight phase, achieving a perfect match between the behavioral indicators and the competency items, which is highly versatile. The proposed model can more accurately reflect the core competencies of flight trainees, enable quantitative assessment of behavioral indicators and competency items, and provide support for subsequent training of trainees.

Development and Evaluation of Digital Learning Tools Promoting Applicable Knowledge in Economics and German Teacher Education

Digital teaching interventions allow for tailor-made university teaching. This is especially relevant for teacher education, where applicable professional teaching knowledge needs to be promoted for later professional success. Digital teaching tools have been shown to be a promising supplement for this purpose. Even though the corresponding demands in teacher education have been increasing in recent years, the need to develop digital learning tools usable in instruction is still urgent. The TWIND project develops digital learning tools for teacher education and evaluates them in a quasi-experimental design. The present work investigates the usability and application of these newly developed tools. Sixty-three trainee teachers worked independently over four weeks with one of two digital learning tools, focusing on either ‘Multilingualism in Classrooms’ or ‘Professional Communication in Classrooms.’ This study includes a pre-post-test of pedagogical knowledge facets as well as student and instructor ratings on the digital tools. The digital learning tools led to a positive change in the respective target facets of pedagogical knowledge. The student and instructor feedback reflected positively on the usability and usefulness of the new digital tools. Based on these findings, the limitations of the study as well as implications for further research and teacher education practice have been outlined.

Using Clinical Simulation to Assess MSW Students’ Engagement Skills

ABSTRACT Social work educators are compelled to use reliable and valid methods to assess student learning outcomes. This study adapted a clinical simulation by integrating traditional role-play of case scenarios and elements of the Objective Structured Clinical Examination, which is often used to assess students’ practice skills. Master of Social Work (MSW) students’ engagement skills were assessed at the end of the first semester of their foundation year using the clinical simulation. Participants were first-year MSW students from two theory and practice courses. The study demonstrated the value of using a clinical simulation in practice courses to assess engagement skills. Correlations of self, peer, and instructor performance scores were compared and interrater reliability was assessed. Peers rated their classmates’ performance the highest, and some peer ratings were negatively correlated with instructor ratings. There was higher correspondence in self- and instructor ratings in some skills and poor correspondence in others. Peers tended to rate their classmates’ performance as showing intermediate proficiency, while self- and instructor ratings of engagement skills indicated basic proficiency. Overall, there was poor interrater reliability. Challenges relating to the implementation of a clinical simulation and its implications for social work education are examined.

A Randomized, Controlled Pilot Study of Mindfulness-Based Stress Reduction in Healthy Older Adults

ABSTRACT Objectives As the aging population increases, it is critical to find ways to sustain older adults’ health and well-being. Mindfulness-Based Stress Reduction (MBSR) may be one approach, but its effects are difficult to discern because few studies have conducted randomized controlled trials with an active control group and blinded examiners. We begin to address these gaps with a pilot study examining the feasibility of conducting an MBSR intervention with an active control condition in healthy older adults. Methods Participants were randomly assigned to one of two classes, MBSR or Brain Health education. Classes were matched for time, format, and instructor. The study examined acceptability, practicality, implementation, and preliminary efficacy using a range of participant questionnaires, instructor ratings, cognitive measures assessed by blinded examiners, and attendance. Results Both MBSR and the Brain Health class evidenced high rates of recruitment, participant satisfaction, and retention. Implementation procedures were successful, and preliminary results revealed similar levels of efficacy across both classes. Conclusions This study demonstrates the feasibility of an MBSR intervention in healthy older adults. Clinical Implications MBSR, with its focus on improving stress and self-awareness, has the potential to be an approach that can improve aging adults’ health and coping skills.

Qualitative Oral-Presentation Feedback: Comparisons from Business Professionals, Instructors, and Student Peers

Previous studies compare quantitative feedback ratings of student peers and instructors, but new presentation-feedback technologies enable qualitative-feedback comparison. This study extends previous research by comparing qualitative feedback and business professionals’ feedback. Compared to the professionals, the instructors provided similar feedback types and sentiment; students, however, de-emphasized message delivery and made fewer suggestions for improvement. The results suggest that students may need additional practice in critiquing message delivery and in suggesting needed improvements in their peers’ oral presentations. The study also provides a methodology using the new technologies for instructors to calibrate their own and their students’ feedback with professionals’ feedback.

A multicenter study of gender bias in student evaluations of teaching in pharmacy programs

IntroductionStudent evaluations of teaching (SET) are widely used to assess effectiveness of teaching. Studies conducted to assess the presence of gender bias in SET have produced mixed results. The purpose of this study was to evaluate the presence and degree of gender bias in SET of didactic courses in United States pharmacy programs. MethodsA three-year, retrospective, multi-institutional analysis of anonymous SET data were analyzed from required didactic courses. Analysis included gender, SET questions, mean scale score, and number of students responding to each question. A hierarchical linear model was used to compare the gender difference with the normalized SET scores as the outcome. ResultsA total of 2114 SET scores were included from seven pharmacy schools across eight campuses. Analysis of the results revealed that the combined data were skewed secondary to one institution whose results fell significantly outside the mean. When this school was excluded, the difference between SET scores did not differ by gender, b = 0.021, t(1,702) = 0.69, P = .49, with similar SET scores for female faculty (mean = 4.41, SD = 0.35, range = 2.54–5) and male faculty (mean = 4.44, SD = 0.32, range = 2.67–5). ConclusionsAfter secondary analysis, the aggregated data showed no significant difference between ratings of male and female instructors. However, there were differences within individual programs. This illustrates the importance of applying assessment principles to SET to determine the presence of bias so that continuous quality improvement strategies may be applied.

ExploringEAPinstructors' evaluation of classroom‐based integrated essays

Due to its authenticity as an academic writing task, integrated writing assessment has become widely used for assessing the writing ability of English for academic purposes (EAP) students (Plakans & Gebril, 2017). However, apart from validation studies focusing on a common set of standardized test rubrics (e.g., Chan, Inoue, & Taylor, 2015), little research has explored the construct of integration or how to assess it effectively in EAP contexts. To provide second language (L2) writing researchers and practitioners with an empirically based model of source integration, this study explored EAP instructors' orientations when assessing integrated essays and investigated the relationship between instructor ratings and textual measures of source use. Six experienced EAP instructors first evaluated two sample argumentative essays written by undergraduate English L2 students and provided comments through a stimulated recall interview. Next, they evaluated 48 additional argumentative essays using an analytic rubric that included dimensions of source use and provided comments for each essay. Finally, the essays were analyzed for linguistic and rhetorical features associated with source integration. Triangulation of data sources revealed that the instructors oriented to three aspects of source integration: using source information to support personal claims, incorporating source‐text language in students' own words, and representing source content accurately. Their ratings were correlated with text‐based measures of students' source use. Implications for teaching and assessing integrated writing in EAP settings are discussed.

Do Personality Traits Predict Training Performance?

Abstract. Most test batteries for the selection of ab initio air traffic controllers (ATCOs) include a personality questionnaire. However, the literature suggests that the relationship between personality and job performance is not straightforward: Correlations are found only for a limited set of personality factors, depend on job characteristics, and are at best small to moderate in size. Moreover, the relationship between personality scores and performance can be curvilinear rather than linear. This research investigates whether a personality test used for the selection of ATCOs predicts subsequent training performance. Training performance was measured in terms of instructor ratings ( N = 262) and training outcome ( N = 431). The rated training performance correlated positively with general self-efficacy and team orientation, and negatively with negativity and vulnerability. For some traits (i.e., general self-efficacy and rebelliousness), curvilinear relationships were found. A comparison of trait scores between students who had successfully completed and those who had failed the training showed significant differences for 10 of the 17 measured traits, all of which were in the expected direction.

Course Quality Improvement in Design Education

Teaching design in technology and engineering education is one of the key <em>Standards for Technological and Engineering Literacy</em> (International Technology and Engineering Educators Association, 2021). Still, it can be challenging to provide a high-quality educational experience aligned with the standards required in introductory technology and engineering courses. Even after thorough preparation and delivering valuable content to students, students may feel the course was of low quality because it was required or not interesting. Our study investigated how improving the design thinking in technology courses based on the Expectancy Value Theory of motivation can impact students’ perceived course quality, as evidenced by course and instructor ratings. This study included nearly 1000 students in 25 sections. Quantitative analysis was completed using a t-test to measure differences in end-of-course evaluation scores for each section, and qualitative analysis of student evaluation responses was done through thematic analysis method. Our study showed that after making changes to a course based on Expectancy Value Theory, students’ ratings of both the course and instructor increased significantly. Expectancy Value Theory focuses the instructors’ course improvement efforts on three aspects which were manageable and resulted in significant improvement. <div> </div>

Gender Bias Interacts With Instructor Disfluency to Negatively Affect Student Evaluations of Teaching

Recent research has shown that instructor fluency can impact student judgments of learning and instructor ratings but has no real effect on actual learning. In addition, women tend to receive lower course and instructor evaluations than men. In the current study, we examined how instructor fluency and instructor gender influenced instructor evaluations and student learning. Participants watched a short lecture video. The speaker was either male or female and was either fluent (i.e., even paced in their speech) or disfluent (i.e., disorganized, made mistakes). Following the video, participants evaluated the instructor and took a quiz over the lecture. Results indicated that disfluency negatively affected quiz scores, but instructor gender did not. Participants rated the female speaker significantly lower than the male speaker, but only when the speaker was disfluent. These results are explained through the lens of attributional gender bias.

Getting in Synch: Understanding Student Perceptions of Synchronous Online Instruction

Aim/Purpose: This study examines the impact of transitioning from in-person classrooms to remote online business education and provides analysis of key factors impacting course and instructor ratings as well as strategies for higher education institutions to provide engaging instruction. Background: “Zoom”ing into teaching and moving out of traditional classrooms during the COVID-19 pandemic has been a path full of twists and has impacted student perceptions of courses as well as instructors. One challenge has been to make the quality of synchronous online instruction perceived by students as positive as classroom-delivered ones. Methodology: We analyze primary data collected in the course evaluation process from Business &amp; Accounting students over six semesters between Fall 2019 to Spring 2022, covering pre-pandemic instruction in the classroom and the conversion to virtual instruction via Zoom. A total of 1782 observations for 38 courses were examined using mean comparison, regression and correlation analyses, and pairwise comparisons. Contribution: We provide insights from the evaluation of those instructors who were able to make their Zoom-delivered courses perceived by students as equivalent or better than room-delivered ones. Specifically, clear presentation, stimulating delivery, providing feedback and encouraging discussion were positively correlated with successful online classes. Findings: We find that there is a clear downward shift in course and instructor ratings as the change to synchronous online delivery was made. However, in the Spring of 2022, even though instructors and students were still not completely back in the classroom, both instructor and course ratings moved back closer to the pre-pandemic levels. The parameters associated with instructor ratings, such as providing feedback, clear presentations, stimulating sessions, and encouraging discussion, showed similar downward fluctuations. Also, aspects related to course content were affected by the transition to online modality, including training on critical thinking quantitative analysis, research and writing abilities, and overall usefulness of the content. Moore’s model of Transactional Distance helps explain these changes. Recommendations for Practitioners: We recommend that practitioners allow sufficient time for students and faculty to learn through online instruction delivery and supply training for both populations in adapting to learning in this delivery mode. Recommendation for Researchers: The disruption in higher education caused by COVID-19 has provided a wealth of information on the pluses and minuses of online delivery. Careful inspection of trends can help provide guidance to higher education leaders. Impact on Society: One of the many changes the COVID-19 pandemic brought was the opportunity to try alternate ways of connecting and learning. This study shows how this experience can be used to guide the future of higher education. Future Research: Further research is needed to explore the in-depth reactions of students and faculty to the switch from classroom to online delivery, to explore whether these findings can be more broadly applied to other subjects and other types of universities.

Classroom management competency enhancement for student teachers using a fully immersive virtual classroom

The purpose of this study is to examine whether pre-service teachers studying classroom management (CM) in a virtual reality (VR)-supported setting enhance their CM competencies more than students do in a setting using conventional methods. With this aim in mind and to address a lack of available situations of practicing and reflecting CM competencies besides simply gaining theoretical knowledge about CM in education, we integrated a novel fully immersive VR application in selected CM courses. We evaluated the development of self-assessed and instructor-rated CM competencies and the learning quality in the different learning conditions. Additionally, we evaluated the presence, social presence, believability and utility of the VR application and the VR-assisted and video-assisted course.Participants were pre-service teachers (n = 55) of the University of Würzburg who participated in a quasi-experimental pre-test/post-test intervention. The students were randomly assigned to one of two intervention groups: the test group used the virtual classroom Breaking Bad Behaviors (BBB) during the term (n = 39), and the comparison group's learning was video-assisted (n = 16).The instructor rating shows significant differences between the VR group and the video group, the two points of measurement and for the interaction between the condition and time of measurement. It demonstrates a highly significant improvement in CM competencies in the VR setting between the pre-test and post-test (p < 0.001, Cohen's d = 1.06) as compared to the video-based setting. The participants of the VR setting themselves rated their CM competencies in the post-test significantly higher than in the pre-test (p = 0.02, Cohen's d = 0.39). Interestingly, the video group also rated themselves better in the post-test (p = 0.02, Cohen's d = 0.67), which reveals that self-assessment and external assessment show different results. In addition, we observed that even if both groups gained similar theoretical knowledge, the CM competencies developed to a greater degree in the VR-based settings. The participants rated the CM training system a useful tool to evaluate and reflect on individual teacher actions. Its immersion contributes to a high presence and the simulation of realistic scenarios in a CM course. These findings suggest that VR-based settings can lead to a higher benefit in the enhancement of pre-service teachers’ CM competencies.

Mitigating implicit bias in student evaluations: A randomized intervention

AbstractWe conduct a randomized control trial to assess the efficacy of utilizing modified introductory language in student evaluations of instruction to mitigate implicit bias. Students are randomly assigned within courses to three treatment arms and shown so‐called “cheap talk” scripts referencing implicit bias, the high stakes associated with student evaluations, and the combination of the two. We analyze both the impact assignment of the treatment has on completion rates as well as the effect on average instructor rating. Our analysis indicates assignment has statistically significant effects on the likelihood of response for those assigned the combined treatment, though the effects are heterogeneous with respect to both instructor and student race/ethnicity and gender. We further find the high‐stakes treatment leads to higher average scores for racial/ethnic minority instructors with no significant effects from the implicit bias and combined scripts.