Teaching Effectiveness Scores Research Articles

Validation of the Learner Engagement Instrument for Continuing Professional Development.

Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables. Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or online) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. All statistical methods accounted for repeated measures by participants. 415 unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with Comparative Fit Index ≥0.99, Standardized Root Mean Square Residual ≤0.031, and Root Mean Square Error of Approximation ≤0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive-in-class domain (50-rater G-coefficient 0.55 to 0.66). All findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho 0.58), and a small correlation was found with participant age (rho 0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form. This study confirms a 4-domain model of learner engagement, and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.

The Physiologic and Emotional Effects of 360-Degree Video Simulation on Head-Mounted Display Versus In-Person Simulation: A Noninferiority, Randomized Controlled Trial.

A key simulation component is its capability to elicit physiological changes, improving recall. The primary aim was to determine whether parasympathetic responses to head-mounted display simulations (HMDs) were noninferior to in-person simulations. The secondary aims explored sympathetic and affective responses and learning effectiveness. The authors conducted a noninferiority trial. Hospital providers who did not use chronotropic medications, have motion sickness, or have seizures were included. The authors randomized participants to in-person or HMD simulation. Biometric sensors collected respiratory sinus arrhythmia and skin conductance levels to measure parasympathetic and sympathetic states at baseline, during, and after the simulation. Affect was measured using a schedule. The authors measured 3-month recall of learning points and used split-plot analysis of variance and Mann-Whitney U tests to analyze. One hundred fifteen participants qualified, and the authors analyzed 56 in each group. Both groups experienced a significant change in mean respiratory sinus arrhythmia from baseline to during and from during to afterward. The difference of change between the groups from baseline to during was 0.134 (95% confidence interval = 0.142 to 0.410, P = 0.339). The difference of change from during the simulation to after was -0.060 (95% confidence interval = -0.337 to 0.217, P = 0.670). Noninferiority was not established for either period. Sympathetic arousal did not occur in either group. Noninferiority was not established for the changes in affect that were demonstrated. The mean scores of teaching effectiveness and achievement scores were not different. Although a parasympathetic and affective response to the video simulation on an HMD did occur, it was not discernibly noninferior to in-person in this study.

Results of a Preceptor Improvement Project.

The majority of medical students receive some of their training with a community preceptor. Nearly all of these preceptors are motivated by a desire to give back to their profession through teaching and they want to learn how to teach more effectively. Designing effective educational programs to improve preceptor teaching is important to upholding the quality of medical education. We designed an educational program consisting of readings, short videos, handouts and posters, as well as one-on-one sessions with a trained standardized medical student. The standardized student visited the community physician's office both before and after the preceptor engaged with the educational materials related to a subject area of the preceptor's choosing. We assessed the preceptor's teaching using three tools: self-evaluation, student reporting of observed behaviors, and an overall rating of teaching effectiveness. Thirteen preceptors took part in this the educational intervention. Per the self-assessment, preceptors showed improved teaching competency for all items on the questionnaire, with 12 of the 21 items (57%) showing statistically significant results (P<.05). The standardized student reported that preceptors used more of eight desired teaching behaviors in the second mock preceptor encounter than they used in the first, with a mean increase of 1.46 (P=.001). Overall teaching effectiveness scores increased, with a mean increase of 1.15 (P=.001) on the 10-point scale. Moreover, all participants indicated they were either satisfied or highly satisfied with the program and that they would recommend this program to a colleague. Our preceptor-improvement intervention led to demonstrable improvement in preceptor teaching, as measured by preceptor evaluation and evaluation by the standardized student. More research is needed to see if these results can be replicated and, in particular, to determine which aspects of the intervention were most useful.

Characteristics of Effective Continuing Medical Education for Physician Assistants and Nurse Practitioners in Hospital Medicine

The purpose of this study was to describe participant characteristics and effective teaching methods at a national continuing medical education (CME) conference on hospital medicine for physician assistants (PAs) and nurse practitioners (NPs). In this cross-sectional study, participants provided demographic information and teaching effectiveness scores for each presentation. Associations between teaching effectiveness score and presentation characteristics were determined. In total, 163 of 253 participants (64.4%) completed evaluations of 28 presentations. Many of the participants were younger than 50 years (69.0%), had practiced for fewer than 5 years (41.5%), and worked in nonacademic settings (76.7%). Teaching effectiveness scores were significantly associated with the use of clinical cases (perfect scores for 68.8% of presentations with clinical cases vs. 59.8% without; P = .04). Many PAs and NPs at an HM CME conference were early-career clinicians working in nonacademic settings. Presenters at CME conferences in hospital medicine should consider using clinical cases to improve their teaching effectiveness among PA and NP learners.

Validation of a Teaching Effectiveness Assessment in Psychiatry Continuing Medical Education.

Little is known about factors associated with effective continuing medical education (CME) in psychiatry. The authors aimed to validate a method to assess psychiatry CME teaching effectiveness and to determine associations between teaching effectiveness scores and characteristics of presentations, presenters, and participants. This cross-sectional study was conducted at the Mayo Clinic Psychiatry Clinical Reviews and Psychiatry in Medical Settings. Presentations were evaluated using an eight-item CME teaching effectiveness instrument, its content based on previously published instruments. Factor analysis, internal consistency and interrater reliabilities, and temporal stability reliability were calculated. Associations were determined between teaching effectiveness scores and characteristics of presentations, presenters, and participants. In total, 364 participants returned 246 completed surveys (response rate, 67.6%). Factor analysis revealed a unidimensional model of psychiatry CME teaching effectiveness. Cronbach α for the instrument was excellent at 0.94. Item mean score (SD) ranged from 4.33 (0.92) to 4.71 (0.59) on a 5-point scale. Overall interrater reliability was 0.84 (95% CI, 0.75-0.91), and temporal stability was 0.89 (95% CI, 0.77-0.97). No associations were found between teaching effectiveness scores and characteristics of presentations, presenters, and participants. This study provides a new, validated measure of CME teaching effectiveness that could be used to improve psychiatry CME. In contrast to prior research in other medical specialties, CME teaching effectiveness scores were not associated with use of case-based or interactive presentations. This outcome suggests the need for distinctive considerations regarding psychiatry CME; a singular approach to CME teaching may not apply to all medical specialties.

Response Rate and Teaching Effectiveness in Institutional Student Evaluation of Teaching: A Multiple Linear Regression Study

<p>It is important to consider the question of whether teacher-, course-, and student-related factors affect student ratings of instructors in Student Evaluation of Teaching (SET) in English Language Teaching (ELT). This paper reports on a statistical analysis of SET in two large EFL programmes at a university setting in the Sultanate of Oman. I carried out a multiple regression analysis to address the research questions of whether instructor sex, class size, course type and percent participation would affect teaching effectiveness scores, and whether or not response rate can be predicted by instructor sex, class size and course type. The study utilizes a dataset of over 2000 student ratings obtained from an SET survey covering the period from Fall 2011 through to Spring 2014in these two programmes. Results indicated that the modeled predictors showed extremely low bias towards both teaching quality scores and response rate. Although the effect sizes of these results are extremely small, they are still significant due to the large sample size (comprising over 2000). The findings also suggest that contrary to common parlance in some quarters claiming students’ unreliable ratings, this analysis has shown that students can judge teaching effectiveness and do not allow other teacher-, course- and student-related factors to bias their responses. The study’s significance stems from the fact that it adds to instructional evaluation in ELT, a field characterized by a clear lack of research on SET.</p>

Associations between teaching effectiveness scores and characteristics of presentations in hospital medicine continuing education.

There is little research regarding characteristics of effective continuing medical education (CME) presentations in hospital medicine (HM). Therefore, we sought to identify associations between validated CME teaching effectiveness scores and characteristics of CME presentations in the field of HM. This was a cross-sectional study of participants and didactic presentations from a national HM CME course in 2014. Participants provided CME teaching effectiveness (CMETE) ratings using an instrument with known validity evidence. Overall CMETE scores (5-point scale: 1 = strongly disagree; 5 = strongly agree) were averaged for each presentation, and associations between scores and presentation characteristics were determined using the Kruskal-Wallis test. The threshold for statistical significance was set at P < 0.05. A total of 277 out of 368 participants (75.3%) completed evaluations for the 32 presentations. CMETE scores (mean [standard deviation]) were significantly associated with the use of audience response (4.64 [0.16]) versus no audience response (4.49 [0.16]; P = 0.01), longer presentations (≥30 minutes: 4.67 [0.13] vs <30 minutes: 4.51 [0.18]; P = 0.02), and larger number of slides (≥50: 4.66 [0.17] vs <50: 4.55 [0.17]; P = 0.04). There were no significant associations between CMETE scores and use of clinical cases, defined goals, or summary slides. To our knowledge, this is the first study regarding associations between validated teaching effectiveness scores and characteristics of effective CME presentations in HM. Our findings, which support previous research in other fields, indicate that CME presentations may be improved by increasing interactivity through the use of audience response systems and allowing longer presentations.

Success of a Faculty Development Program for Teachers at the Mayo Clinic.

There has been limited research on the improvement of underperforming clinical teachers. To determine whether a faculty development program could improve the evaluations of clinical teachers in an internal medicine residency program. A total of 123 teachers completed faculty development at the Mayo Clinic from 2009 to 2012. The faculty enhancement and education development program (FEED) consists of 6 interactive, small group, 2-hour sessions taught by experienced Mayo Clinic faculty over 1year. These sessions address the following competencies: asking questions, diagnosing learners, giving feedback, using teaching frameworks, recognizing learning styles, and providing clinical supervision. Resident-of-faculty Mayo teaching effectiveness (MTE) scores have previously demonstrated content, internal structure, and criterion validity. Teachers were grouped into the top 80% or the bottom 20%, according to baseline MTE scores. Mixed linear models were used to compare these groups regarding changes in MTE scores after completion of FEED. Results were adjusted for teacher age, sex, medical specialty, academic rank, and teaching awards. For all participants combined, the adjusted MTE scores (mean; standard error) improved from baseline (3.80; 0.04) to completion of FEED (3.93; 0.04; P < .001). However, the bottom 20% had a significantly greater improvement in scores than the top 80% (score-change difference = 0.166, P < .001). We describe a low-intensity faculty development intervention that benefited all clinical teachers, but was particularly effective for underperforming teachers in internal medicine. The approach may be suitable for adoption or adaptation in other graduate medical education programs.

57. The power of power: Comparative evaluations of medical residency training across teaching sites and programs at the University of Torontos

To meet accountability and accreditation requirements, teaching partners and the faculty postgraduate office required more robust and integrated feedback on teaching and assessment. The web-based evaluation system known as POstgraduate Web Evaluation and Registration (POWER) was implemented in 2004/05 by most residency training programs, using their existing forms and scoring scales. At start up, over 250 different evaluation forms and 85 varying scoring scales were in operation across programs for the In-Training Evaluation Reports (ITERs) and resident-completed evaluations for Rotation Evaluation Scores (RES) and Teaching Effectiveness Scores (TES). &#x0D; The POWER Evaluation Working Group was formed to develop a methodology to gather and consolidate evaluations to report on medical residents, their teachers, and rotations in a clear, consistent user-friendly format, map general questions against CanMEDS roles and Family Medicine principles, and convert all scoring scales to a consistent 5 point Likert scale. A standardized naming protocol was developed to map rotation services to individual teaching sites.&#x0D; The 2004/05 analysis of these evaluations (2004/05 Annual POWER Report: Lessons Learned) provide baseline data to begin monitoring trends in resident and faculty performance, assess the quality of programs and identify areas for improvement by CanMEDS standards and CFPC principles. Mean scores, standard deviations and number of evaluations were presented by teaching site and program.&#x0D; Consolidation of evaluations by program and teaching site provides valuable feedback to hospitals and programs wishing to standardize and improve their assessment systems, and to postgraduate medical offices who must maintain evaluation standards and illustrate trends for accreditation purposes. Future activities include: standardizing evaluation forms starting July 2007, improving scoring consistency and accuracy, improve participation rates and timeliness of responses, develop a procedure/case log tracking system, and trend analysis.&#x0D; Afrin LB, Arana GW, Medio FJ, Ybarra AF, Clarke HS Jr. Improving oversight of the graduate medical education enterprise: one institution’s strategies and tools. Academic Medicine 2006 (May); 81(5):419-25.&#x0D; Benjamin S, Robbins LI, Kung S. Online Sources for assessment and evaluation. Academic Psychiatry 2006 (Nov-Dec); 30(6):498-504.&#x0D; Rosenberg ME, Watson K, Paul J Miller W, Harris I, Valdivia TD. Development and Implementation of a web-based evaluation system for an internal medicine residency program. Academic Medicine 2001 (Jan); 76(1):92-5.

Clinical teaching and clinical outcomes: teaching capability and its association with patient outcomes

There is little research on the impact of medical education on patient outcome. We studied whether teaching capability is associated with altered short-term patient outcomes. We performed a multicentre retrospective cross-sectional study involving 40 clinician teachers who had attended on the general internal medicine services in hospitals affiliated with the University of Toronto along with the clinical outcomes of consecutive patients treated for community-acquired pneumonia, congestive heart failure, chronic obstructive pulmonary disease and gastrointestinal bleeding (n = 4377) between 1999 and 2001. Doctors were characterised by teaching effectiveness scores (n = 677) as high-rated or low-rated according to house staff ratings. There was no correlation between the teaching effectiveness scores and the mean length of stay for those patients treated for community-acquired pneumonia (high-rated = 10.3 versus low-rated = 8.1 days, P = 0.058), congestive heart failure (high-rated = 10.1 versus low-rated = 9.9 days, P = 0.978), chronic obstructive pulmonary disease (high-rated = 9.4 versus low-rated = 9.9 days, P = 0.419) and gastrointestinal bleeding (high-rated = 6.3 versus low-rated = 6.8 days, P = 0.741). In addition, we observed no significant correlation between teaching effectiveness scores and 7-day, 28-day and 1-year readmission rates for all pre-specified diagnoses. There is no large correlation between teaching effectiveness scores and short-term patient outcomes, suggesting that doctor teaching capabilities, as perceived by house staff, does not generally impact clinical care.

Self-Efficacy and Teaching Effectiveness

Interpreting changing levels of self-efficacy while improving effective teaching behaviors can be one goal of teacher education programs. Undergraduate education majors often begin their teacher education programs with high levels of self-efficacy (a belief in one's capabilities). These levels of self-efficacy frequently decline as preservice teachers progress through their curriculum and make the transition to in-service teaching. A slight decline in these levels can be interpreted as a novice teacher's greater understanding of the complexity of the teaching process. Eighteen preservice string teachers evaluated their levels of self-efficacy with the Teacher Efficacy Scale (Guskey and Passaro, 1994) three times during an academic year. The preservice teachers were videotaped three times during regular teaching assignments at a community music program. They self-evaluated the teaching episodes by the means of a Music Teaching Observation Form (Kelly, 1984). Experienced educators also evaluated the tapes. Group and individual means indicated that ratings of teaching effectiveness by both preservice teachers and experienced educators increased while levels of self-efficacy decreased slightly. Several specific items from the Teacher Efficacy Scale and Music Teaching Teaching Observation Form had significant changes. Changing levels in self-efficacy did not have an influence on increased teaching effectiveness scores for this specific group of preservice teachers.

Teaching Effectiveness and Social Skill Development

The purpose of this study was to determine whether there were any relationships between adjudicated teaching effectiveness scores, assessed by the Survey of Teaching Effectiveness (Hamann &amp; Baker, 1995), an observation-based assessment instrument, and social skills scores, assessed by the Social Skills Inventory (Riggio, 1989), a self-report instrument, of preservice teachers. Subjects for the study, N = 138, were music education and elementary/secondary education students preparing to be teachers at three universities in Ohio and Oklahoma. From regression analysis, it was found that (a) Emotional Expressivity, an individual's skill in nonverbal communication, (b) Emotional Sensitivity, an individual's skill in receiving and interpreting nonverbal communication of others, and (c) Social Control, an individual's ability to engage others in social discourse, were related to teaching effectiveness among preservice teachers.

Teaching effectiveness of surgeons

Evaluation of clinical teaching is essential for feedback to teachers, and for reliable and valid data for the purpose of promotion and tenure. The teaching effectiveness of 43 surgeons was evaluated continuously over a 9-year period. Stability of Teaching Effectiveness Scores (TES) over the 9-year period, correlation with age, and the changes after academic promotion were analyzed. A total of 3,750 evaluations were completed. The average of 10 evaluations per surgeon per year gave an intraclass correlation of 0.65. The mean TES did not show any significant change over the 9 years. The majority of the "good" and "average" surgeons maintained their TES ratings, and most of the "poor" group improved their TES. The age of the surgeon was not a factor. Most surgeons had a decrease in their TES after academic promotion. A reliable system of teaching evaluation was achieved. Teaching scores were stable over time for good and average teachers, whereas the poor teachers tended to improve their scores.

A Methodology for Evaluating College Teaching Effectiveness

SUMMARY SUMMARY In this paper a multidimensional model for evaluating college teaching effectiveness is developed to select the best faculty member for a teaching award. The model is based on the assumption that teaching effectiveness is a linear function of five variables: (a) students’ evaluation; (b) graduating students’ evaluation; (c) chairman's input; (d) colleagues’ input; (e) course file. These variables have been quantified to obtain teaching effectiveness scores for the faculty members eligible for the award contest. The scores of the five variables are standardized prior to differential weighting. The reason for incorporating these five standardized weighted variables into the model is to ensure the fairness and objectivity of this evaluation process. The model has been implemented in a college to select a faculty for an outstanding teacher award.