Ratings Of Instruction Research Articles

Selecting trainee pilots: Predictive validity of the WOMBAT situational awareness pilot selection test

The WOMBAT pilot selection test is widely used to select candidates for pilot training programs. Despite use in many countries, little information is available regarding the predictive validity of the test. This study was designed to test the ability of the WOMBAT test to predict performance outcomes in a sample of ab-initio pilots. Sixty students commenced the study in 3 cohorts, and completed the WOMBAT test before their performance in the training program was evaluated through flight time to solo, flight time to licence level achievement, and instructor ratings of performance. Higher WOMBAT total scores were significantly related to reduced time to solo, achieving flight licences, as well as some early ratings of performance by flight instructors. Further research now needs to examine the exact nature of the skills and abilities that the test indexes in order to further improve pilot selection and training procedures.

Collaborative course development: A comparison of business and non-business students’ perceptions of class experience

Prior research has demonstrated significant increases in student engagement and learning outcomes when faculty and students share course development and implementation practices. This study is an empirical investigation of collaborative course development (CCD) effects on four dependent variables, specifically: student's felt engagement, perceived learning, course satisfaction, and ratings of instructors. It utilized an interdisciplinary course and teaching method comparison conducted over one academic year. The experimental study included six courses that were each taught with one control-group class that employed traditional, instructor-led course designs and one experimental class for a total of twelve control and experimental conditions (in business and non-business courses). Courses were selected for inclusion in the study based on the availability of a course having two sections offered during the academic year for the participating faculty. Each CCD experimental course carefully employed six student-centered CCD techniques, namely: student-faculty collaborative syllabus and assessment building; elective choice assignments; competitive experimentation; flexible lectures; and creative theory building. We hypothesized that an emphasis on faculty–student collaboration facilitated through student empowerment and choice would positively influence the four dependent variables across all subjects. However, the findings revealed an interdisciplinary-contextual effect. The CCD techniques were significantly more effective in business courses compared to non-business courses. These findings enhance our understanding of teacher collaboration with students and student preferences for collaborative education approaches in business (but not in the non-business courses).

Comparability of students’ writing performance on TOEFL iBT and in required university writing courses

Investigating the comparability of students’ performance on TOEFL writing tasks and actual academic writing tasks is essential to provide backing for the extrapolation inference in the TOEFL validity argument (Chapelle, Enright, & Jamieson, 2008). This study compared 103 international non-native-English-speaking undergraduate students’ performance on two TOEFL iBT® writing tasks with their performance in required writing courses in US universities as measured by instructors’ ratings of student proficiency, instructor-assigned grades on two course assignments, and five dimensions of writing quality of the first and final drafts of those course assignments: grammatical, cohesive, rhetorical, sociopragmatic, and content control. Also, the quality of the writing on the TOEFL writing tasks was compared with the first and final drafts of responses to written course assignments using a common analytic rubric along the five dimensions. Correlations of scores from TOEFL tasks (Independent, Integrated, and the total Writing section) with instructor ratings of students’ overall English proficiency and writing proficiency were moderate and significant. However, only scores on the Integrated task and the Writing section were correlated with instructor-assigned grades on course assignments. Correlations between scores on TOEFL tasks and all dimensions of writing quality were positive and significant, though of lower magnitude for final drafts than for first drafts. The TOEFL scores were most highly correlated with cohesive and grammatical control and had the lowest correlations with rhetorical organization. The quality of the writing on the TOEFL tasks was comparable to that of the first drafts of course assignment but not the final drafts. These findings provide backing for the extrapolation inference, suggesting that the construct of academic writing proficiency as assessed by TOEFL “accounts for the quality of linguistic performance in English-medium institutions of higher education” (Chapelle, Enright, & Jamieson, 2008, p. 21).

Do We See Eye to Eye? Moderators of Correspondence Between Student and Faculty Evaluations of Day-to-Day Teaching

Students and instructors show moderate levels of agreement about the quality of day-to-day teaching. In the present study, we replicated and extended this finding by asking how correspondence between student and instructor ratings is moderated by time of semester and student demographic variables. Participants included 137 students and 5 instructors. On 10 separate days, students and instructors rated teaching effectiveness and challenge level of the material. Multilevel modeling indicated that student and instructor ratings of teaching effectiveness converged overall, but more advanced students and Caucasian students converged more closely with instructors. Student and instructor ratings of challenge converged early but diverged later in the semester. These results extend our knowledge about the connection between student and faculty judgments of teaching.

Influential Factors of Student Evaluations of Teaching in a Nursing Program

We used a logistic regression model to analyze data from routinely collected student evaluation of teaching from 2,211 nursing students over 8 semesters. The results showed that not all aspects of teaching are equally influential in their effects on students' overall rating of instructors' teaching as excellent. Influential aspects include motivation, individual assistance, presentation of concepts, environment, and practical application. The most influential evaluation aspect is motivation.

Women and Men Together in Recruit Training.

Although men and women recruits to the Australian Army have trained in mixed-gender platoons since 1995, restrictions on women joining the combat arms were only removed in 2016. As part of a longitudinal study starting with recruit training, this article examined recruit records collected before 2016 with the aims of delineating (1) the relative performance of women versus men in mixed-gender platoons and (2) the relative performance of men in mixed-gender platoons versus all-male platoons. De-identified instructor ratings for 630 females and 4,505 males who completed training between 2011 and 2015 were obtained. Recruits were distributed across 128 platoons (averaging 41.6 members, SD = 8.3) of which 75% contained females, in proportions from 5% to 45%. These analyses were conducted under defense ethics approval DPR-LREP 069-15. Factor analyses revealed that instructor ratings generally loaded onto a single factor, accounting 77.2% of the variance. Consequently, a composite recruit performance score (range 1-5) was computed for 16 of 19 competencies. Analyses of the scores revealed that the distributions of the scores for females and males overlapped considerably. Observed effects were negligible to small in size. The distributions were all centered between 3.0 and 3.5. In mixed-gender platoons, 51% of the females and 52% of the males fell in this band, and 44% of recruits in all-male platoons had scores in this band. The lower three bands (1.0-3.0) contained a slightly greater proportion of females (18%) than males in either mixed-gender platoons (12%) or all-male platoons (12%). Conversely, the upper three bands (3.5-5.0) contained a slightly smaller percentage of females (31%) than males in either mixed-gender platoons (36%) or all-male platoons (44%). Although scores for females were reliably lower than those of males in mixed-gender platoons, χ2 (4) = 16.01, p < 0.01, the effect size (V = 0.07) did not reach the criterion for even a small effect (0.10). For male recruits, those in mixed-gender platoons had scores that were reliably lower than in all-male platoons, χ2 (4) = 48.38, p < 0.001; its effect size (V = 0.11) just exceeded the criterion for a small effect (0.10). Further analyses revealed that male scores had a near-zero correlation (r = -0.033) with the proportion of females in platoons (0-45%). This large-scale secondary analysis of instructor ratings of female and male recruits provides a platform for monitoring the integration of women into the combat arms. The analyses revealed nearly complete overlap in the performance of female versus male recruits. The detected gender-related differences were negligible to small in size. These small differences must be viewed with considerable caution. They may be artifacts of rater bias or other uncontrolled features of the rating system, which was designed for reporting individual recruit performance rather than aggregate analyses. Even with these limitations, this baseline snapshot of recruit performance suggests that, at recruit training, women and men are already working well together, which bodes well for their subsequent integration into the combat arms.

Discussion Boards in the Face-to-Face Classroom, Laboratory, and Clinical

We used a logistic regression model to analyze data from routinely collected student evaluation of teaching from 2,211 nursing students over 8 semesters. The results showed that not all aspects of teaching are equally influential in their effects on students' overall rating of instructors' teaching as excellent. Influential aspects include motivation, individual assistance, presentation of concepts, environment, and practical application. The most influential evaluation aspect is motivation.

Administrative and structural changes in student evaluations of teaching and their effects on overall instructor scores

Using nine years of student evaluation of teaching (SET) data from a large US research university, we examine whether changes to the SET instrument have a substantial impact on overall instructor scores. Our study exploits four distinct natural experiments that arose when the SET instrument was changed. To maximise power, we compare the same course/instructor before and after each of the four changes occurred. We find that switching from in-class, paper course evaluations to online evaluations generates an average change of −0.14 points on a five-point scale, or 0.25 standard deviations (SDs) in the overall instructor ratings. Changing labelling of the scale and the wording of the overall instructor question generates another decrease in the average rating: −0.15 of a point (0.27 SDs). In contrast, extending the evaluation period to include the final examination and offering an incentive (early grade release) for completing the evaluations do not have a statistically significant effect on the overall instructor rating. The cumulative impact of these individual changes is −0.29 points (0.52 SDs). This large decrease shows that SET scores are not comparable over time when instruments change. Therefore, administrators should measure and account for such changes when using historical benchmarks for evaluative purposes (e.g. appointments and compensation).

Translating Concept into Act forMulti-Cultural Practice: Comparison of Students’ and Field Instructors’ Perceptions of Diversity Training Effectiveness

Education for culturally competent practice increasingly is a responsibility for social work educators.Using data collected for an evaluation of the field education component of a large,Midwestern social work program, the purpose of this study is to shed light on students’ application in the field practicum setting of classroom training in culturally competent practice. Responses were obtained from field instructors (n=76) and students (n=70). Students reported learning in areas dealing with diversity at statistically significant levels; however, instructor ratings of student competence were significantly lower than student ratings. Results are discussed in light of necessary attitudes, knowledge, and skills. Implications for program monitoring and improvement, education, and further research are discussed.

The effect of class session length on student performance, homework, and instructor evaluations in an introductory accounting course

ABSTRACTThe author examines the effect of class format on student performance, time spent on homework, and instructor evaluations. The findings are unique and add to a limited body of literature. Each group comprised similar undergraduate students who received the same number of instructional hours, same assignments and were taught by the same instructor. Students in the one-day-a-week class performed significantly better than students in the two-days-a-week class. Students in the one-day-a-week class spent more time on online homework, although the difference was not statistically significant. Differences in instructor evaluation ratings were nominal.

An Examination of Motivation Model Components in Face-to-Face and Online Instruction

Introduction. The MUSIC Model of Academic Motivation consists of psychological components (i.e., empowerment, usefulness, success, interest, and caring) that have been derived from research and theory as ones that are critical to student engagement in academic settings. The purpose of this study was to: (1) determine whether men and women rate these psychological components differently in traditional face-to-face and online courses, and (2) determine which of the psychological components best predicts men and women’s effort, instructor ratings, course ratings, and achievement in a face-to-face and online course.Method. Participants included 245 students in a face-to-face course and 218 students in an online course who completed a questionnaire in a Personal Health course at a large public university in the U.S. I conducted correlation, analysis of variance (ANOVA), and regression analyses to examine the data.Results: Although several similarities between sexes and course types (face-to-face and online) existed for the components of the MUSIC model, I also documented many differences. Overall, the pattern of results indicated that the components of the MUSIC model were significantly related to students’ effort, instructor ratings, and course ratings. In all conditions, instructor rating was best predicted by academic caring, course rating was best predicted by situational interest, and achievement was best predicted by perceptions of success.Conclusion. To maximize students’ effort and achievement, as well as their instructor and course ratings in both face-to-face and online courses, instructors should consider the components of the MUSIC Model of Academic Motivation.

Changes in syllabus tone affect warmth (but not competence) ratings of both male and female instructors

The syllabus is often the first meaningful piece of information that students receive about a course. Previous research has indicated that students form more positive impressions of a course instructor after reading a syllabus that has been manipulated to convey information in a friendly, rather than unfriendly, tone (Harnish and Bridges in Soc Psychol Educ 14:319–330, 2011). While a friendly syllabus leads to increased perceptions of instructor warmth and approachability, it is unclear from this previous research whether a friendly syllabus may also lead to decreases in the perceived competence of the instructor. Thus, we aimed to clarify whether changes in syllabus tone affect perceptions of instructor competency. We also wished to explore the possibility of gender bias affecting these syllabus-based impressions of instructors, and to examine whether differences in syllabus tone impact the impressions formed of male and female instructors in the same way. Participants read a friendly or unfriendly course syllabus from either a male, female, or gender-unspecified instructor. Regardless of instructor gender, participants receiving the friendly syllabus perceived the instructor as being more approachable, more caring, and more motivating, but not any more or less competent, compared to those receiving the unfriendly syllabus. While instructors will be relieved to know that efforts to appear friendly on a course syllabus do not appear to negatively impact student perceptions of instructor competence, more research is needed to examine the potential role of gender bias on students’ initial impressions of instructors.

Is RateMyProfessors.com Unbiased? A Look at the Impact of Social Modelling on Student Online Reviews of Marketing Classes

This article looks at how marketing student ratings of instructors and classes on online rating sites such as RateMyProfessor.com can be biased by prior student ratings of that class. Research has identified potential sources of bias of online student reviews administered by universities. Less has been done on the sources of bias inherent in a ratings site where those doing the rating can see prior ratings. To measure how student online ratings of a course can be influenced by existing online ratings, the study used five different prior ratings experiment conditions: mildly negative prior ratings, strongly negative prior ratings, mildly positive prior ratings, strongly positive prior ratings, and a control condition of no prior ratings. Results of this study suggest prior online ratings, both positive and negative, do affect subsequent online ratings and bias them. There are several implications. First, both negative and positive ratings can have an impact biasing subsequent ratings. Second, sometimes negative prior ratings must be strong in valence in order to bias subsequent ratings whereas even mildly positive ratings can have an impact. Last, this bias can potentially influence student course selection.

The effect of white coats and gender on medical students\u2019 perceptions of physicians

BackgroundDespite the fact that medical schools spend a considerable effort to rate clinical instructors, there is limited evidence regarding the effect of physical characteristics on instructor ratings. White coats have been shown to alter patients’ perceptions of physicians although it has not been determined if preceptors who wear white coats are rated differently than their colleagues.MethodsSecond year medical students were administered a questionnaire with four clinical scenarios depicting medical errors accompanied by a picture of a physician of different sexes and ethnicities. The packages were randomized so that the physicians depicted either had or did not have a white coat.ResultsWhite coats did not alter the perception of physicians’ ratings by medical students although sex and ethnicity/case were associated with the perception of trustworthiness, physician management, competence, professionalism and the perception of medical error.ConclusionsPhysical characteristics may alter students’ ratings of physicians.

Assessing Entrustable Professional Activities During the Psychiatry Clerkship.

Within 10years, the Association of American Medical Colleges envisions graduating medical students will be entrusted by their school to perform 13 core entrustable professional activities (EPAs) without direct supervision. The authors focused on eight EPAs that appear most relevant to clinical training during the psychiatry clerkship at their institution to evaluate whether students assess themselves as making progress in EPAs during this clerkship, to see how students' self-assessments compare with the clerkship director's assessments, and to see if weaknesses in the curriculum were found. An EPA-assessment scale was designed (ratings 1 to 5) to assess progress toward entrustment in each EPA. Medical students completed pre- and post-psychiatry clerkship self-assessments. The clerkship director independently assessed each student's progress in EPAs utilizing assessment methods already present in the curriculum. Seventy of 116 students (60.3%) completed both pre- and post-clerkship self-assessments. These ratings increased significantly from pre- to post-clerkship, representing large effect sizes from 0.83 to 1.13. The largest mean rating increase was observed for EPA 2, Prioritize a differential diagnosis following a clinical encounter. Mean post-clerkship self-assessment ratings were significantly higher than mean post-clerkship instructor ratings for seven of the eight EPAs. The results suggest training during the psychiatry clerkship can contribute to the professional development of medical students in the eight EPAs studied but that student self-assessments tend to be higher than those of the clerkship director. Further study is needed of the relative value and role of student self-assessments versus faculty assessments of progress in EPAs.

Should Students Have the Power to Change Course Structure?

In the present article, we describe a course exercise in which students were administered four course evaluation forms throughout the semester, on which they provided their overall impressions of the class as well as their desire to change certain aspects of the course. Critically, during the semester, a total of three changes were made to the structure of the course as voted on by the students. Compared to the previous semester where students completed only end-of-semester evaluations, improvements in exam performance as well as instructor ratings were observed. Furthermore, students indicated that the changes made throughout the semester improved the course, and they hoped that other classes would adopt a similar classroom developmental strategy. This supports a growing body of evidence, suggesting that midsemester feedback is crucial for optimizing the learning environment for the student, particularly when concrete changes are made after the administration of course feedback.

Optimum Frequency of Peer Evaluations in Capstone Design Courses

Peer evaluations are commonly used in design courses for developmental and evaluative purposes. Peer ratings are however often higher than the instructor ratings and this can induce fears regarding their reliability. This study examined whether it may be possible to increase the agreement between peer and instructor ratings by increasing the frequency of the peer assessments. The premise was that peers may provide less lenient assessments if the impact of single evaluations on the final grade is reduced by increasing the number of evaluations. Increasing the number of peer evaluations in our senior design course from two to six per year did not increase the accuracy of the peer ratings but provided other benefits such as earlier identification of dysfunctional teams, elimination of free riding and more frequent developmental feedback. Peer and instructor ratings can be normalized however to yield similar indicators of the relative performance of teammates. The frequency and timing of peer evaluations are critical to obtain meaningful results and maximize impact on team dynamics

Instructional Management Skills and Effectiveness of Physical Education Instructors in Higher Education Institutions in Cebu City, Philippines

Teaching has become multiple jobs. It could be in myriad aspects and in harmony with the context of effectiveness and improvement course. It is concerned with a wider range of outcomes, social, and affective. The study determined the instructional management skills and level of effectiveness of the PE instructors of the three higher education institutions of Cebu City, Philippines. A total of 24 instructors coming from three selected institutions participated in the study. There were 14 instructors from the University A, seven instructors from University B, and three instructors from University C. The study utilized a descriptive research method. It uses a researcher-designed survey tool to gather data on the profile of physical education instructors; their instructional effectiveness as well their management skills. There was correlation analysis on the profile and instructional management skills and the level of effectiveness. The ratings of the PE instructors of the three universities on the instructional management skills and instructional effectiveness were very good. Age, gender, education, teaching experiences, and seminar had nothing to do with instructional management skills and effectiveness of the PE instructors. However, there was a significant relationship between their instructional management skills and their instructional effectiveness. The majority of the PE teachers of the three institutions were positive on the given support of the administration in the form of facilities, materials, and equipment. Multimedia resources should be enriched to upgrade learning in physical education.

No Second Chance to Make a First Impression: The “Thin‐Slice” Effect on Instructor Ratings and Learning Outcomes in Higher Education

Prior research has found strong and persistent effects of instructor first impressions on student evaluations. Because these studies look at real classroom lessons, this finding fits two different interpretations: (1) first impressions may color student experience of instruction regardless of lesson quality, or (2) first impressions may provide valid evidence for instructional quality. By using scripted lessons, we experimentally investigated how first impression and instruction quality related to learning and evaluation of instruction among college students. Results from two studies indicate that quality of instruction is the strongest determinant of student factual and conceptual learning, but that both instructional quality and first impressions affect evaluations of the instructor. First impressions matter, but our findings suggest that lesson quality matters more.

Understanding the effect of response rate and class size interaction on students evaluation of teaching in a higher education

AbstractObjective: This study aims to investigate the interaction between response rate and class size and its effects on students’ evaluation of instructors and the courses offered at a higher education Institution in Saudi Arabia. Study Design: A retrospective study design was chosen. Methods: One thousand four hundred and forty four different courses belonging to all the Colleges (N = 21) located across seven different campuses of University of Dammam (UOD) were considered in this study. All the course evaluation surveys (CES) (N = 168,574) conducted during the academic year 2013–2014 were analyzed to investigate response rate and class size interaction effect on students evaluation of courses and instructors. Results: It is observed that when the class size is at the medium level, the ratings of instructors and courses increase as the response rate increases. On the contrary; when the class size is small, a high response rate is required for the evaluation of instructors and at least medium response r...