Differences In Evaluation Scores Research Articles

Taking Our Pulse: Examining 5 Years of Data from the Association of Pediatric Program Directors Research and Scholarship Learning Community

BackgroundThe Association of Pediatric Program Directors Research and Scholarship Learning Community (RSLC) prioritizes and comprehensively reviews medical education surveys directed to residency program leadership. Each survey is reviewed by two members of RSLC and the Chair and Vice Chair using a standardized scoring rubric and a limited number of surveys are accepted per cycle. MethodsInternal review data from 2015 to 2020 were analyzed to determine factors associated with survey acceptance, and for surveys accepted for distribution, determine factors associated with response rates or dissemination status. One-Way analysis of variance (ANOVA) assessed differences in evaluation scores by initial determination status. T-tests and Pearson Product Correlation assessed associations between evaluation scores and response rates by dissemination status. ResultsThe majority (47/81; 58%) of surveys submitted to RSLC are eventually accepted for distribution. Response rates for distributed surveys varied between 14% and 73%. Most (35/47; 74%) surveys distributed through RSLC are presented (62%) and/or published (60%). Higher review scores were associated with acceptance for distribution but not with response rates or dissemination status. ConclusionMost surveys are eventually accepted by RSLC for distribution and those distributed often get published or presented despite variability in response rates.

Assessment of medical students\u2019 clinical performance using high-fidelity simulation: comparison of peer and instructor assessment

BackgroundHigh-fidelity simulators are highly useful in assessing clinical competency; they enable reliable and valid evaluation. Recently, the importance of peer assessment has been highlighted in healthcare education, and studies using peer assessment in healthcare, such as medicine, nursing, dentistry, and pharmacy, have examined the value of peer assessment. This study aimed to analyze inter-rater reliability between peers and instructors and examine differences in scores between peers and instructors in the assessment of high-fidelity-simulation-based clinical performance by medical students.MethodsThis study analyzed the results of two clinical performance assessments of 34 groups of fifth-year students at Ajou University School of Medicine in 2020. This study utilized a modified Queen’s Simulation Assessment Tool to measure four categories: primary assessment, diagnostic actions, therapeutic actions, and communication. In order to estimate inter-rater reliability, this study calculated the intraclass correlation coefficient and used the Bland and Altman method to analyze agreement between raters. A t-test was conducted to analyze the differences in evaluation scores between colleagues and faculty members. Group differences in assessment scores between peers and instructors were analyzed using the independent t-test.ResultsOverall inter-rater reliability of clinical performance assessments was high. In addition, there were no significant differences in overall assessment scores between peers and instructors in the areas of primary assessment, diagnostic actions, therapeutic actions, and communication.ConclusionsThe results indicated that peer assessment can be used as a reliable assessment method compared to instructor assessment when evaluating clinical competency using high-fidelity simulators. Efforts should be made to enable medical students to actively participate in the evaluation process as fellow assessors in high-fidelity-simulation-based assessment of clinical performance in situations similar to real clinical settings.

Student Potential for Self-assessment in a Clinical Dentistry Practical Training Course on Communication Skills

This study aimed to investigate student potential for self-assessment in a clinical dentistry practical training course focused on communication skills. Participants were 124 fourth-year students (70 males, 54 females; all Japanese) in 2017 and 2018 at the Nippon Dental University, School of Life Dentistry at Niigata. Participating students belonged to different cohorts in 2017 and 2018. Participants were asked to complete a self-evaluation sheet at the end of each unit of the course. Their self-evaluation scores and the faculty evaluation scores for each student for Units 1-1, 1-2, and 1-3 were statistically analyzed. The results showed that females tended to rate themselves significantly higher than males. Furthermore, there were significant differences in evaluation scores between students and faculty for nine of 11 evaluation items for male students and 10 of 11 items for female students in Unit 1-3. Faculty expectations increased from Unit 1-1 to Unit 1-3, although students were satisfied with their performance and had a sense of achievement. However, students’ actual performance was below faculty expectations, suggesting faculty evaluations were stricter than students’ self-evaluation. Self-assessment may enhance students’ ability for self-directed learning and may also inform how faculty can effectively educate dental students. Dental educators should support students to increase their levels of self-efficacy, which will enhance their self-evaluation skills.

Do I Sound Funny to You? The Role of Instructor Gender and Accent on Student Learning and Evaluation in a Digital Learning Environment

In academia, student evaluation of instructors is the most utilized assessment of educators’ effectiveness, which in turn, can influence faculty promotion, tenure and career trajectory. However, ample body of literature demonstrates that students’ implicit bias and stereotyped expectations can negatively impact their evaluation of instructors in the classroom and in online learning environments. Studies on gender and age bias revealed that digital resources and instructors are evaluated differently based on the voice qualities that carry different age and gender characteristic. Further, in our previous study, male voice containing ethnic characteristics expressed in different accents showed differences in student evaluations despite the consistently positive learning outcomes. In the current, IRB‐exempt study (#17‐1510), the experimental condition expanded to test students’ learning outcomes and evaluation of digital resources narrated by a female voice with different ethnic qualities. Four identical digital learning resources introducing histology, embryology and anatomy of the pituitary were narrated by a female voice actor in 4 different accents (American, Chinese, Indian, British). Professional science students who had not yet learned any aspect of pituitary gland were recruited to participate in a randomized single blind study. Participants were randomly assigned to 4 groups (A, B, C, D) and received one of the four digital resources. All students completed a pre and post‐quiz before and after tutorial viewing and a resource and instructor evaluation. Learning outcomes were assessed by pre and post‐quiz comparison using Two‐Way ANOVAs. Quantitative data from evaluations were analyzed by Two‐way ANOVAs with Tukey post hoc tests where appropriate. Internal reliability of the evaluation items was measured across 4 groups by Cronbach’s Alpha (0.88, 0.94 for digital resource and instructor, respectively).126 students completed the study (A, n=33, B, n=28 C, n=32. D, n=30). All groups performed significantly better on the post‐quiz compared to the pre‐quiz (p<0.05; partial η2 = 0.53), but the performance increase across the 4 groups was not significantly different (p>0.05) suggesting that learning occurs from the digital tutorials and that narrator accent has no effect on learning. There was no significant difference in evaluation scores for the tutorial or the instructor across the 4 groups (p>0.05). Interestingly, all 4 groups rated the tutorials higher than the instructor (p<0.05; partial η2 = 0.14), suggesting that students value the digital resources higher than the instructor narrating them. This study reveals for the first‐time, student learning and evaluation of a female accented digital instructor, which may have important implications in the increasingly diverse and digital world of education.

The Role of Gender in Ophthalmology Resident Evaluations

Abstract Background Gender affects various aspects of medical training. Prior research in surgical specialties has shown that female residents are given less positive feedback, granted less autonomy in the operating room, perform fewer procedures, and achieve competency milestones at a slower rate as compared with their male counterparts. Purpose The purpose of this research is to evaluate whether gender affects ophthalmology resident evaluations at a single institution. Methods Ophthalmology resident evaluations at a single residency program from 2010 to 2018 were reviewed. Data were collected on faculty gender, resident gender, and year of resident training. A linear mixed-effects model was utilized to analyze the degree to which differences in evaluation scores could be predicted from demographic data, while accounting for multiple sources of nonindependence of data. Results A total of 490 evaluations for 43 residents by 34 faculty were analyzed. Evaluations consisted of up to 23 questions graded on a scale from 0 (poor) to 9 (excellent). Female residents received marginally higher scores than male residents on average (coefficient of male residents = −0.2). Both male and female residents received marginally lower scores from male faculty than from female faculty on average (coefficient of male faculty = −0.21). Male faculty also appeared to have scored male residents lower to a greater degree than did female faculty (coefficient of male faculty by male resident interaction = −0.14), though this result was sensitive to model specifications. There was no significant interaction between year of resident training and gender. Conclusion In contrast to other procedural specialties, female residents appear to have been graded at a similar level or higher than male residents on average. Male faculty gave slightly lower scores to both male and female residents than female faculty did. Male faculty also may have graded male residents marginally lower than female residents to a greater degree than female faculty did.

Female Surgeons as Counter Stereotype: The Impact of Gender Perceptions on Trainee Evaluations of Physician Faculty

Similar to women in Science, Technology, Engineering and Mathematics disciplines, women in medicine are subject to negative stereotyping when they do not adhere to their sex-role expectations. These biases may vary by specialty, largely dependent on the gender's representation in that specialty. Thus, females in male-dominated surgical specialties are especially at risk of stereotype threat. Herein, we present the role of gender expectations using trainee evaluations of physician faculty at a single academic center, over a 5-year period (2010-2014). Using Graduate Medical Education evaluation data of physician faculty from MedHub, we examined the differences in evaluation scores for male and female physicians within specialties that have traditionally had low female representation (e.g., surgical fields) compared to those with average or high female representation (e.g., pediatrics). Stanford Medicine residents and fellows' MedHub ratings of their physician faculty from 2010 to 2014. A total of 3648 evaluations across 1066 physician faculty. Overall, female physicians received lower median scores than their male counterparts across all specialties. When using regression analyses controlling for race, age, rank, and specialty-specific characteristics, the negative effect persists only for female physicians in specialties with low female representation. This finding suggests that female physicians in traditionally male-dominated specialties may face different criteria based on sex-role expectations when being evaluated by trainees. As trainee evaluations play an important role in career advancement decisions, dictate perceptions of quality within academic medical centers and affect overall job satisfaction, we propose that these differences in evaluations based merely on gender stereotypes could account, in part, for the narrowing pipeline of women promoted to higher ranks in academic medicine.

The Online Evaluation of Courses: Impact on Participation Rates and Evaluation Scores

At one of Ontario’s largest universities, the University of Ottawa, course evaluations involve about 6,000 course sections and over 43,000 students every year. This paper-based format requires over 1,000,000 sheets of paper, 20,000 envelopes, and the support of dozens of administrative staff members. To examine the impact of a shift to an online system for the evaluation of courses, the following study sought to compare participation rates and evaluation scores of an online and paper-based course evaluation system. Results from a pilot group of 10,417 students registered in 318 courses suggest an average decrease in participation rate of 12–15% when using an online system. No significant differences in evaluation scores were observed. Instructors and students alike shared positive reviews about the online system; however, they suggested that an in-class period be maintained for the electronic completion of course evaluations.

Prospective Randomized Crossover Study of Telesimulation Versus Standard Simulation for Teaching Medical Students the Management of Critically Ill Patients

The objective was to evaluate the comparative effectiveness of telesimulation versus standard simulation in teaching medical students the management of critically ill patients. Prospective, randomized crossover study of 32 fourth-year medical students at a university medical simulation center. Students were randomized to the standard simulation (SIM) or telesimulation (TeleSIM) group between September 2014 and February 2015. The SIM group experience included participating in a live, fully immersive simulation case followed by debriefing with their SIM cohort and a live TV Internet connection to the TeleSIM group. The TeleSIM group experience included remotely observing the live simulation case at an off-site location, followed by a shared group debriefing via live TV Internet connection. Subject assessment was performed with a written evaluation tool. During a second instructional session, the students crossed over and participated in a different simulation scenario and assessment. Mean evaluation scores were calculated along with 95% confidence intervals (CIs) and were analyzed via linear regression. Our secondary outcome was a survey evaluating the perceptions and attitudes held between the two simulation modalities. Of 33 eligible students, 32 participated in the study (97.0%). We found no significant difference in the mean evaluation scores of the two groups: SIM group mean= 96.6% (95% CI= 94.5%-98.6%) and TeleSIM group mean= 96.8% (95% CI= 94.8%-98.9%). We also found no significant difference in the favorability of teaching modality (TeleSIM vs. SIM) on the survey. In our prospective randomized crossover study evaluating telesimulation versus standard simulation, we found no significant difference in evaluation scores among the two groups. There was also no significant difference found in the favorability of one teaching modality on a posteducational session survey. Our data support and highlight the capability of telesimulation to provide educational benefit to learners who do not have direct access to simulation resources.

Evaluating Evaluation Measure Stability

This paper presents a novel way of examining the accuracy of the evaluation measures commonly used in information retrieval experiments. It validates several of the rules-of-thumb experimenters use, such as the number of queries needed for a good experiment is at least 25 and 50 is better, while challenging other beliefs, such as the common evaluation measures are equally reliable. As an example, we show that Precision at 30 documents has about twice the average error rate as Average Precision has. These results can help information retrieval researchers design experiments that provide a desired level of confidence in their results. In particular, we suggest researchers using Web measures such as Precision at 10 documents will need to use many more than 50 queries or will have to require two methods to have a very large difference in evaluation scores before concluding that the two methods are actually different.

How Does Value Added Compare to Student Growth Percentiles?

We compare teacher evaluation scores from a typical value-added model to results from the Colorado Growth Model (CGM), which 16 states currently use or plan to use as a component of their teacher performance evaluations. The CGM assigns a growth percentile to each student by comparing each student's achievement to that of other students with similar past test scores. The median (or average) growth percentile of a teacher's students provides the measure of teacher effectiveness. The CGM does not account for other student background characteristics and excludes other features included in many value-added models used by states and school districts. Using data from the District of Columbia Public Schools (DCPS), we examine changes in evaluation scores across the two methods for all teachers and for teacher subgroups. We find that use of growth percentiles in place of value added would have altered evaluation consequences for 14% of DCPS teachers. Most differences in evaluation scores based on the two methods are not related to the characteristics of students’ teachers.

Evaluation of Motion Correction Processing in Equine Bone Scintigraphy by Scheffé’s Method of Paired Comparisons

Equine bone scintigraphy is usually performed with horses in standing position under sedation. However, swaying motion often leads to poor-quality images. To examine the usefulness of motion correction (MC) processing, equine bone scintigrams were evaluated using Scheffé's method of paired comparisons. A significant difference in evaluation scores was detected by analysis of variance (F test, P<0.01). According to all observers, Yardstick analysis scores were higher for images use of MC processing than for those no use of MC processing, for all parts. Overall scores of 5 observers were as follows: without MC 100% acquisition time (AT, lowest), use of MC with 25% AT, MC 50% AT, MC 75% AT and MC 100% AT (highest). Thus, MC processing shortens AT in equine bone scintigraphy, and it contributes to a reduction in the external radiation exposure of nurses/technicians.

The Effects of Gender and Age on Evaluation of Trainees and Faculty in Gastroenterology

We studied whether differences exist in evaluation scores of faculty and trainees in gastroenterology (GI) based on the gender of the evaluator or evaluatee, or the evaluator-evaluatee gender pairing. We examined evaluations of faculty and trainees (GI fellows and internal medicine residents rotating on GI services), using mixed linear models to assess effects of the four possible evaluator-evaluatee gender pairings. Potential confounding variables were adjusted for, and random effects were used to account for repeated assessments. For internal medicine (IM) residents, no difference in evaluation scores based on gender was found. Resident age was negatively associated with performance rating, while percentage correct on the in-training examination (ITE) was positively associated. For GI fellows, the interaction between evaluator and evaluatee gender was significant. Fellow age and international medical graduate (IMG) status were negatively associated with performance rating, while ITE percentage correct was positively associated. For faculty, no difference was found in evaluation scores by IM residents based on the gender of the evaluated faculty or the evaluating resident, although the interaction between the evaluator and the evaluatee gender was significant. Gender had a significant marginal effect on faculty scores by GI fellows, with female faculty receiving lower scores. The interaction between evaluator and evaluatee gender was also significant for evaluations by fellows. Faculty age was negatively associated with performance rating. Gender, age, and ITE performance are associated with evaluation scores of GI trainees and faculty at our institution. The interaction of evaluator and evaluatee gender appears to play a more critical role in evaluation scoring than the gender of the evaluatee or evaluator in isolation.

Comparison of 128-Section Single-Shot Technique with Conventional Spiral Multisection CT for Imaging of the Temporal Bone

Computed tomography is an essential modality for imaging of the temporal bone. Newest generation scanners allow the coverage of large examination volumes with a single gantry rotation. The objective of this study was to compare a 128-section SST (1 single rotation of the x-ray tube) with conventional spiral MSCT (ultra-high-resolution mode) for imaging of the temporal bone. Fifty-four temporal bones in 27 patients were scanned with both a conventional MSCT and 128-section SST. After blinding and randomization of both examinations, 2 observers assessed the visualization of 38 anatomic structures (eg, various segments of the facial nerve canal, mallear ligaments) by using multiplanar reconstructions in the axial and coronal planes. The differences in evaluation scores obtained for the 2 techniques were analyzed by using a Wilcoxon signed rank test, with a P value of < .05 considered significant. For both methods, imaging time and radiation exposure were noted. Overall visualization of anatomic structures did not differ significantly between the 2 techniques (P > .05). When we compared the anatomic structures separately, there was better visualization of the lateral mallear ligament with MSCT, whereas the cochlear septa were ranked higher with SST (P < .05). Imaging time and average DLP for MSCT were 12.3 seconds and 306 mGy cm, respectively; for SST, values they were 1 second and 64 mGy cm, respectively (ie, a dose reduction of 79%). For imaging of the temporal bone with adequate diagnostic quality, 128-section SST can be used. The main advantages over MSCT are the dramatic reductions of imaging time and radiation exposure, which are particularly important when scanning uncooperative patients or children.

Volatile compounds of vegetarian soybean kapi, a fermented Thai food condiment

Kapi is a traditional shrimp paste used as a food condiment in Thailand. Several vegetarian soybean kapi, S1-S5, were fermented from various bacterial starter cultures isolated from commercial shrimp paste. The volatile compounds of S1-S5 were analyzed using SPME coupled with gas chromatography/mass spectrometry and compared to three samples of commercial vegetarian kapi (J1-J3) and commercial shrimp pastes (K1-K3). 124 volatile compounds consisting of aldehydes, alcohols, ketones, acids and esters, N-containing compounds, aromatic compounds, S-containing compounds, miscellaneous, indoles and hydrocarbons were identified. Principle component analysis and cluster analysis separated the volatile profile of the fermented samples into four groups. Vegetarian soybean kapi, S1, S4 and S5 produced from Bacillus subtilis IS4, TISTR10 and TISTR1, respectively, were classified into the following groups containing commercial kapi (J1, J2, K2 and K3), that had a predominance of indole, S-containing and N-containing compounds. Sensory evaluation of S1 showed a strong kapi odor with higher scores among the vegetarian soybean kapi and there were no significant differences in evaluation scores between S1 and commercial vegetarian kapi J1-J3. These data demonstrate that B. subtilis IS4 can be employed as a starter culture to produce an acceptable vegetarian soybean kapi substitute for shrimp paste kapi. Key words : Vegetarian kapi, volatile compounds, fermentation, principal component analysis, cluster analysis.