Postgraduate Year Research Articles

Want doctors to use VR simulation? Make it mandatory, accessible, educationally valuable, and enjoyable!

Background Virtual reality (VR) simulation training is mandatory for postgraduate year 1–2 doctors at the author’s hospital trust. Despite this, a preceding quantitative study demonstrated uptake below required levels. While the educational value of VR simulation has been highlighted, little attention has been paid to participant utilisation in postgraduate curricula. With the increasing development and incorporation of VR-based clinical education, it is essential to understand the factors influencing how frequently postgraduate doctors utilise it so that its potential can be maximised. Methods A qualitative study design was employed. All 108 postgraduate year 1–2 doctors from the 2020-21 training year were invited for a semi-structured interview. Interviews continued until data saturation was reached in the form of informational redundancy. Reflexive thematic analysis was conducted. Results A total of 17 interviews were conducted. Four main themes that influenced participation in VR simulation were identified: (1) the mandatory nature encouraged participation but led to negative perceptions as a tick-box exercise; (2) there were multiple challenges to accessing the resource; (3) the scenarios were felt to have limited educational value; and (4) there was untapped potential in drawing benefits from VR as an enjoyable leisure activity. Conclusions Recommendations from these findings include: (1) VR simulation should be mandatory but with a degree of learner autonomy; (2) sessions should be integrated into doctors’ rotas as protected time; (3) more challenging scenarios ought to be created aligned with postgraduate courses, examinations, and specialty training, and (4) presented as a difficulty level system akin to gaming experiences.

Current state of undergraduate medical school training in transfusion medicine and its impact on postgraduate trainee knowledge.

Studies have described poor transfusion medicine (TM) knowledge in postgraduate trainees. The impact of undergraduate medical TM education on postgraduate knowledge is unclear. Canadian medical schools were surveyed on the number of hours dedicated to TM teaching and topics covered by curricula during 2016-2020. Postgraduate trainees attending Transfusion Camp in 2021 completed a pretest of 20 multiple-choice questions. The survey results and pretest scores were compared to evaluate the association between undergraduate medical TM education and pretest scores. The survey was completed by 16 of 17 Canadian medical schools. The number of hours (h) of TM teaching were <2 h (25%), 3-4 h (25%), and >4 h (50%). Twelve of 19 Transfusion Camp topics were covered in ≥50% of schools. Eleven medical schools provided ethics approvals/waivers to include trainee pretest scores in the analysis (N = 200). The median pretest scores by medical school ranged from 48% to 70%. No association was found between number of TM teaching hours and average pretest scores (p = .60). There was an association between higher postgraduate year level and individual pretest score (p < .0001). The analysis by topic demonstrated questions where trainees from different schools performed uniformly well or poorly; other topics showed considerable variation. Variation in quantity and content of undergraduate TM teaching exists across Canadian medical schools. In this limited assessment, the number of TM teaching hours was not associated with performance on the pretest. This study raises the opportunity to re-evaluate the delivery (content, timing, consistency) of TM education in undergraduate medical schools.

Assessing Team Performance: A Mixed-Methods Analysis Using Interprofessional in situ Simulation.

Optimizing the performance of emergency department (ED) teams impacts patient care, but the utility of current, team-based performance assessment tools to comprehensively measure this impact is underexplored. In this study we aimed to 1) evaluate ED team performance using current team-based assessment tools during an interprofessional in situ simulation and 2) identify characteristics of effective ED teams. This mixed-methods study employed case study methodology based on a constructivist paradigm. Sixty-three eligible nurses, technicians, pharmacists, and postgraduate year 2-4 emergency medicine residents at a tertiary academic ED participated in a 10-minute in situ simulation of a critically ill patient. Participants self-rated performance using the Team Performance Observation Tool (TPOT) 2.0 and completed a brief demographic form. Two raters independently reviewed simulation videos and rated performance using the TPOT 2.0, Team Emergency Assessment Measure (TEAM), and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). Following simulations, we conducted semi-structured interviews and focus groups with in situ participants. Transcripts were analyzed using thematic analysis. Eighteen team-based simulations took place between January-April 2021. Raters' scores were on the upper end of the tools for the TPOT 2.0 (R1 4.90, SD 0.17; R2 4.53, SD 0.27, IRR [inter-rater reliability] 0.47), TEAM (R1 3.89, SD 0.19; R2 3.58, SD 0.39, IRR 0.73), and Ottawa GRS (R1 6.6, SD 0.56; R2 6.2, SD 0.54, IRR 0.68). We identified six themes from our interview data: team member entrustment; interdependent energy; leadership tone; optimal communication; strategic staffing; and simulation empowering team performance. Current team performance assessment tools insufficiently discriminate among high performing teams in the ED. Emergency department-specific assessments that capture features of entrustability, interdependent energy, and leadership tone may offer a more comprehensive way to assess an individual's contribution to a team's performance.

Impact of COVID-19-Induced Academic Stress on Insomnia and Suicidal Ideation among Taiwanese Health Trainees and Junior Doctors.

BACKGROUND Clinical training for allied health trainees (AHTs) and postgraduate-year (PGY) doctors needed to go online during the outbreak of coronavirus disease 2019 (COVID-19), which may have caused academic stress and consequent outcomes among this cohort. MATERIAL AND METHODS To evaluate academic-related stress, clinical confidence, psychological distress, and insomnia, an online survey-based study was conducted among Taiwanese AHTs and PGY doctors between July and December, 2022, during the COVID-19 pandemic. The survey included the 21-item Depression, Anxiety, and Stress Scale (DASS-21), the Insomnia Severity Index (ISI), and self-designed questions. It was distributed using convenience sampling and snowball sampling and was completed by 522 participants. RESULTS Structural equational modelling showed that academic stress was negatively associated with clinical confidence (standardized coefficient [ß]=-0.382, p<0.001). Clinical confidence was negatively associated with psychological distress (ß=-0.397, p<0.001), which was associated with insomnia (ß=0.648, p<0.001). Additionally, clinical confidence and psychological distress were the significant mediators. Results indicated that higher academic stress was associated with higher level of insomnia via the mediation of clinical confidence and psychological distress. CONCLUSIONS Academic stress related to changes in clinical training may have led to insomnia among AHTs and PGY doctors during the pandemic. Factors to reduce academic stress should be investigated to promote good mental health while providing sufficient clinical training, especially during events that can cause increased stress (eg, epidemics, pandemics).

Is ChatGPT ready for primetime? Performance of artificial intelligence on a simulated Canadian urology board exam.

Generative artificial intelligence (AI) has proven to be a powerful tool with increasing applications in clinical care and medical education. ChatGPT has performed adequately on many specialty certification and knowledge assessment exams. The objective of this study was to assess the performance of ChatGPT 4 on a multiple-choice exam meant to simulate the Canadian urology board exam. Graduating urology residents representing all Canadian training programs gather yearly for a mock exam that simulates their upcoming board-certifying exam. The exam consists of written multiple-choice questions (MCQs) and an oral objective structured clinical examination (OSCE). The 2022 exam was taken by 29 graduating residents and was administered to ChatGPT 4. ChatGPT 4 scored 46% on the MCQ exam, whereas the mean and median scores of graduating urology residents were 62.6%, and 62.7%, respectively. This would place ChatGPT's score 1.8 standard deviations from the median. The percentile rank of ChatGPT would be in the sixth percentile. ChatGPT scores on different topics of the exam were as follows: oncology 35%, andrology/benign prostatic hyperplasia 62%, physiology/anatomy 67%, incontinence/female urology 23%, infections 71%, urolithiasis 57%, and trauma/reconstruction 17%, with ChatGPT 4's oncology performance being significantly below that of postgraduate year 5 residents. ChatGPT 4 underperforms on an MCQ exam meant to simulate the Canadian board exam. Ongoing assessments of the capability of generative AI is needed as these models evolve and are trained on additional urology content.

Risk factors and outcomes associated with external ventricular drain related hemorrhage

ObjectiveThe objective of this study was to determine risk factors predictive of external ventricular drain (EVD)-related hemorrhage and the association of such hemorrhages with mortality, discharge disposition, length of stay (LOS), and total cost. MethodsAfter Institutional Review Board approval, data was collected retrospectively for adult patients requiring EVD placement from 2015 to 2018 at the authors’ institution. Collected data included demographic patient information, peri-procedural factors, and relevant post-procedural measures. Computerized tomography (CT) images and associated radiologic reports were independently reviewed, identifying hemorrhages accompanying EVD placement. ResultsFrom this 487-patient sample, 85 (17.5 %) patients had hemorrhages, including asymptomatic hemorrhages identified on imaging alone. A univariable analysis of patient parameters in the overall cohort was performed to identify possible predictors of hemorrhage. Age (p = 0.002), Charlson Comorbidity Index (CCI) (p < 0.001), platelet count (p = 0.002), presence of uremia (p = 0.035), and the number of times the EVD was replaced (p < 0.001) were associated with hemorrhage in univariable models. The experience of the resident surgeon based on post-graduate year (PGY level) and the number of attempts/passes needed for EVD placement were not associated with hemorrhage risk. Significant predictor of hemorrhage confirmed in a multivariable analysis only included the number of times the EVD was replaced (OR = 2.78, adjusted p < 0.001). Outcomes between EVD-related hemorrhage versus no hemorrhage groups, including mortality, discharge disposition, LOS, and cost, were compared. EVD-related hemorrhage was found to be associated with increased mortality (OR = 3.58, adjusted p < 0.001) and decreased likelihood of discharge home (OR = 0.13, adjusted p = 0.030) in the associated multivariable regressions. ConclusionThe number of times an EVD was replaced was associated with EVD-related hemorrhage outcome. EVD-related hemorrhage is associated with increased mortality and a decreased likelihood of being discharged home.

A Longitudinal Assessment of Resident and New Graduate Well-Being According to Length of Training: A Report From the Length of Training Pilot in Family Medicine.

Two surveys captured data included in these analyses. One was a resident survey that included the Mayo Clinic physician-expanded Well-Being Index (eWBI) administered annually during the In-Training Examination (2014-2019). The second was administered to graduates 1 year after completion of training between 2016 and 2022 and included the same well-being questions. Response rates ranged between 77.7% and 96.8%. The eWBI summary scores for burnout were highest in postgraduate year 1 (PGY1) and did not differ statistically according to length of training (PGY1: 2.02 in 3-year [3YR] programs vs 1.93 in 4-year [4YR] programs, P=.55; postgraduate year 2 [PGY2]: 2.42 in 3YR programs vs 2.38 in 4YR programs, P=.83; postgraduate year 3 [PGY3]: 2.18 in 3YR programs vs 2.28 in 4YR programs, P=.59; and 2.34 in postgraduate year 4 [PGY4] for those in 4YR programs), though some statistical differences were noted for three items. New graduates' eWBI summary scores before the COVID-19 pandemic were 1.77 among 3YR graduates and 1.66 among 4YR graduates (P=.59). These scores were higher during COVID-19 at 1.89 for 3YR graduates and 2.02 for 4YR graduates (P=.62). Length of training was not associated with differences in well-being before or during COVID-19. We found no associations between length of training and physician well-being during training or among new graduates before or during COVID-19.

Surgeon and Surgical Trainee Experiences After Adverse Patient Events

Adverse patient events are inevitable in surgical practice. To characterize the impact of adverse patient events on surgeons and trainees, identify coping mechanisms, and assess whether current forms of support are sufficient. In this mixed-methods study, a validated survey instrument was adapted and distributed to surgical trainees from 7 programs, and qualitative interviews were conducted with faculty from 4 surgical departments in an urban academic health system. The personal impact of adverse patient events, current coping mechanisms, and desired forms of support. Of 216 invited trainees, 93 (43.1%) completed the survey (49 [52.7%] male; 60 [64.5%] in third postgraduate year or higher; 23 [24.7%] Asian or Pacific Islander, 6 [6.5%] Black, 51 [54.8%] White, and 8 [8.6%] other race; 13 [14.0%] Hispanic or Latinx ethnicity). Twenty-three of 29 (79.3%) invited faculty completed interviews (13 [56.5%] male; median [IQR] years in practice, 11.0 [7.5-20.0]). Of the trainees, 77 (82.8%) endorsed involvement in at least 1 recent adverse event. Most reported embarrassment (67 of 79 trainees [84.8%]), rumination (64 of 78 trainees [82.1%]), and fear of attempting future procedures (51 of 78 trainees [65.4%]); 28 of 78 trainees (35.9%) had considered quitting. Female trainees and trainees who identified as having a race and/or ethnicity other than non-Hispanic White consistently reported more negative consequences compared with male and White trainees. The most desired form of support was the opportunity to discuss the incident with an attending physician (76 of 78 respondents [97.4%]). Similarly, faculty described feelings of guilt and shame, loss of confidence, and distraction after adverse events. Most described the utility of confiding in peers and senior colleagues, although some expressed unwillingness to reach out. Several suggested designating a departmental point person for event debriefing. In this mixed-methods study of the personal impact of adverse events on surgeons and trainees, these events were nearly universally experienced and caused significant distress. Providing formal support mechanisms for both surgical trainees and faculty may decrease stigma and restore confidence, particularly for underrepresented groups.

The effect of social support and resource support on emotional exhaustion, insomnia, and suicidal ideation among allied health trainees and post-graduate year doctors in Taiwan

BackgroundCOVID-19-related stigmatization refers to COVID-19-related judgements by others that devalue the individual. Such stigmatization towards healthcare workers may cause psychological burden and negative consequences. Such stigmatization may have particularly overwhelmed allied health trainees (AHTs) and post-graduate year doctors (PGYDs) because they just started their medical career. Social support and resource support have been reported to benefit psychological health and reduce stigmatization. Therefore, the present study used a cross-sectional study design to investigate the association between perceived stigma, self-stigma, psychological distress, and negative outcomes (including emotional exhaustion, insomnia and suicidal ideation) among AHTs and PGYDs in Taiwan.MethodsAn online survey distributed between July and December, 2022 received 522 responses. Variables were assessed using the 21-item Depression, Anxiety and Stress Scale, Insomnia Severity Index and a series of self-designed questions to assess social support, resource support, perceived stigma, self-stigma, emotional exhaustion, and suicidal ideation.ResultsStructural equation modeling showed that perceived stigma was associated with self-stigma (standardized coefficient [β] = 0.428, p < 0.001), and self-stigma was associated with psychological distress (β = 0.197, p < 0.001), as well as being associated with emotional exhaustion, insomnia, and suicidal ideation (β = 0.349, 0.556 and 0.212, all p-values < 0.001). While social support and resource support were negatively associated with perceived stigma (β= − 0.175 and − 0.152, p < 0.01), additional associations were found between social support and emotional exhaustion (β= − 0.093, p < 0.001), as well as between resource support and insomnia (β= − 0.120, p < 0.001).ConclusionsThe results showed that COVID-19 related stigmatization was correlated to the detrimental consequences of emotional exhaustion, insomnia and suicidal ideation. Clear paths regarding the associations of social support and resource support with the three negative associations were found as the possible solutions. Strategies to reduce the stigmatization and these negative outcomes, or improve the psychological health will benefit AHTs and PGYDs in maintaining a healthy mental status.

Individualized throughput metric reports for emergency medicine residents: Impact on time to disposition and resident perceptions.

Research suggests that quantitative metric reports can improve the clinical performance of emergency physicians; however, few studies have examined their effects on physicians in training. The primary study objective was to assess the effects of providing emergency medicine (EM) residents with individualized throughput metrics with regard to emergency department (ED) disposition times. We performed a single-center, retrospective, observational study from January 2021 to December 2022 examining ED disposition times before and after providing upper-level EM residents individualized throughput metrics. Residents received monthly reports of three specific metrics averaged over the preceding 6 months: (1) median time from room to discharge order (Rm2Dc), (2) median time from return of all results to discharge order (Rlts2Dc), and (3) median time from room and to consult order for hospitalization (Rm2Hosp). Overall mean values of the three metrics before and during metric sharing were compared via independent t-test and stratified by level of training and time of year. Adjusted analysis was performed to control for temporal differences between study periods. Testing was conducted at α = 0.05 level of significance. A total of 35 unique residents were included in the analysis. Overall, mean disposition times were not significantly different before and during reporting of metrics: Rm2Dc (154.8 min vs. 148.9 min, p = 0.109), Rslt2Dc (46.5 min vs. 45.1 min, p = 0.522), and Rm2Hosp (141.7 min vs. 135.7 min, p = 0.257). Subgroup analysis yielded similar results, aside from a significant decrease in mean Rm2Hosp in the postgraduate year-3 (PGY-3) group (145.8 min vs. 124.1 min, p = 0.004). Analysis with adjusted means yielded results similar to those observed with unadjusted data. Overall, individualized throughput metrics were not correlated with decreased average times to ED disposition for upper-level EM residents; however, in the subset of hospitalized patients seen by PGY-3 residents, we observed a mean decrease of 21.7 min to consultation.

AI-assisted colonoscopy in gastroenterology fellowship training: An interim analysis of a randomized controlled trial.

e15642 Background: Artificial intelligence (AI) assisted colonoscopy may increase adenoma detection rates (ADR) for endoscopists, but its role for trainees in learning endoscopy has not been delineated. This randomized control trial explores the impact of AI on ADR by year of fellowship and anatomical location. Methods: In a registered randomized controlled trial (NCT05423964), gastroenterology fellows were assigned to AI (GI Genius, Medtronic; Minneapolis, MN) vs. conventional colonoscopy(CC). We included all symptomatic or screening colonoscopies with adequate bowel prep by trainees (postgraduate year 4-6). Baseline demographics, polyp pathology, size in 5mm increments, and location, and ADR was collected and compared for AI versus CC arms. A multivariate regression was used to identify factors influencing ADR of the right colon. Results: Of 685 colonoscopies between March 15th to July 28th, 2023, a total of 381 procedures performed by 17 fellows met criteria. Patient characteristics were similar between arms. In the CC arm, there was no difference in ADR among fellow years (1st 39% vs. 2nd 33% vs. 3rd 36%, p = 0.63; Right colon: 1st 31% vs 2nd 21% vs 3rd 30%, P = 0.21). Within 2nd year fellows, there was higher ADR using AI vs. CC for screening colonoscopies (52% vs 17%, p = 0.025). In the right colon, ADR was significantly higher using AI vs. CC for screening alone colonoscopies within 1st and 2nd year fellows (1st 50% vs 13%, p = 0.04; 2nd 57% vs 9%, p &lt; 0.001). 2nd year fellows also increased ADR in the right colon using AI vs CC for symptomatic and screening colonoscopies (44% vs 21%, p = 0.001). There was no difference in procedural time with addition of AI, but a trend to longer withdrawal time. Multivariable logistic regression analysis also showed AI assisted colonoscopies was associated with enhanced ADR in right colon for trainees performing colonoscopy for symptomatic and screening (OR [95%CI]: 2 [1.3-3.3]) and screening alone (OR [95%CI]: 8.3 [2.5-26.6]). Conclusions: This randomized control trial shows AI assisted colonoscopies improve ADR, particularly in the right colon for postgraduate trainees. We propose a use case for AI assisted colonoscopy guiding targeted improvement of ADR in in the right colon which has been highlighted by the American Gastroenterology Association as an area of need. Clinical trial information: NCT05423964 . [Table: see text]

Successful Implementation of the AIUM Standardized 4-Year Residency Ultrasound Curriculum in Obstetrics and Gynecology: Lessons Learned and Way Forward.

Significant disparities in sonographic education exist in Obstetrics and Gynecology programs in the United States. To address the lack of standardization in ultrasound teaching, the American Institute of Ultrasound in Medicine (AIUM) assembled a multi-society task force in 2018 that resulted in the publication of a Consensus Report outlining a standardized ultrasound curriculum and competency assessment. The primary objective of the study was to implement the AUIM standardized curriculum within an Obstetrics and Gynecology residency program at a tertiary medical center, and report on the early implementation experience. The secondary objective was to assess current national practices in ultrasound training in Obstetrics and Gynecology residency programs. Study design was an effectiveness-implementation hybrid study of a structured ultrasound educational program based on the 2018 AIUM Consensus Report. The theoretical portion of the curriculum consisted of 26 hours of didactic lectures distributed over 4 years. For the practical component, residents received instruction from faculty and sonographers and independently acquired the required images as part of their regular clinical work. Simulator app access was also provided for additional opportunities to practice and acquire images. Deidentified images for the required competencies were uploaded to a WhatsApp group for review. Theoretical knowledge was assessed with a series of multiple-choice exams. Practical skills were assessed through grading of images submitted through WhatsApp as well as direct observations of resident scans. At the completion of 4 years, the assessment of both theoretical knowledge and practical skills was carried out using the ISUOG Basic Training Examination. National patterns in the area of ultrasound education were assessed with a four-question electronic survey of Obstetrics and Gynecology program directors in the United States. Analysis of both portions of the study was primarily descriptive. Four Obstetrics and Gynecology residents completed the curriculum spanning postgraduate years 1 through 4. The number of competencies completed ranged from 7 to 19 out of 41. All of the residents passed the theoretical and practical components of the ISUOG Basic Training Examination. For the residency program director survey portion of the study, response rate was 20% (59 of 290). Among the respondents, 55.9% reported having a structured curriculum and 34.4% reported implementing the AIUM curriculum. The two most commonly cited obstacles to implementation of structured ultrasound teaching were uncertainty with how to start the process (49.2%) and lack of faculty engagement (49.2%). Implementation of the AIUM Curriculum within the framework of an Obstetrics and Gynecology residency program at a tertiary medical center is feasible. In our experimental group, completion of the curriculum resulted in passing of an internationally recognized validated assessment. Obstacles to implementation in our program included a wide range of engagement and participation among residents, and limited opportunities to obtain some of the required competencies. Identification of obstacles to implementation from the perspective of residency program directors is critical for developing targeted approaches to ensure widespread implementation. Successful standardized validated sonographic training of graduating Obstetrics and Gynecology residents is crucial for this operator-dependent modality that has tremendous implications on patient safety and clinical care.

Resident-as-Teacher: Can It Be Done With an E-Learning Module?

Background Resident-as-teacher initiatives are traditionally specialty-specific and performed in-person, limiting ability to disseminate essential teaching skills to all residents. Objective The aim of this study was to develop, implement, and evaluate a resident-as-teacher interactive e-learning module on growth mindset and coaching. Methods The module was designed and implemented between August 2022 and March 2023. It was distributed to postgraduate year (PGY) 1 residents in all specialties at a large academic institution. Completion rates, Likert ratings, and answers to 2 open-ended questions were used for assessment. Descriptive statistics and 1-way analysis of variance with Sîdák correction for multiple comparisons were performed on Likert ratings. Responses to open-ended questions were evaluated using content analysis. Results The module was completed by all 277 PGY-1 residents (100%), with the evaluation completed by 276 of 277 (99.6%) residents. Mean rating of the module's relevance to the role of resident teacher was 4.06±0.90 (5-point Likert scale), with general surgery residents rating the module less favorably compared to all specialties (3.28±1.06; P<.01; 95% CI 0.26-1.30). Open-ended comments revealed that residents most liked the delivery of relevant teaching strategies and the interactive design of the module. The most common area for suggested improvement was the addition of content such as teaching in challenging situations. Time needed for design, implementation, and evaluation was 80 hours total. Conclusions An e-learning module offers an interactive platform for teaching skills and was found to be an acceptable method of instruction for residents.

Looking Beyond the Numbers: A Comparison of Operative Self-Efficacy, Supervision, and Case Volume in General Surgery Residency.

Background A national survey of general surgery residents revealed significant self-assessed deficits in preparation for independent practice, with only 7.7% of graduating postgraduate year 5 residents (n=1145) reporting self-efficacy for all 10 commonly performed operations surveyed. Objective We sought to understand why this phenomenon occurs. We hypothesized that self-efficacy would be positively correlated with both operative independence and case volume. Methods We compared 3 independent datasets: case information for the same 10 previously surveyed operations for residents graduating in 2020 (dataset 1), operative independence data obtained through the SIMPL OR app, an operative self-assessment tool (dataset 2), and case volume data obtained through the Accreditation Council for Graduate Medical Education National Data Report (dataset 3). Operations were categorized into high, middle (mid), and low self-efficacy tiers; analysis of variance was used to compare operative independence and case volume per tier. Results There were significant differences in self-efficacy between high (87.7%), mid (68.3%), and low (25.4%) tiers (P=.008 [95% CI 6.2, 32.7] for high vs mid, P<.001 for high vs low [49.1, 75.6], and P<.001 for mid vs low [28.7, 57.1]). The percentage of cases completed with operative independence followed similar trends (high 32.7%, mid 13.8%, low 4.9%, P=.006 [6.4, 31.4] for high vs mid, P<.001 [15.3, 40.3] for high vs low, P=.23 [-4.5, 22.3] for mid vs low). The total volume of cases decreased from high to mid to low self-efficacy tiers (average 91.8 to 20.8 to 11.1) but did not reach statistical significance on post-hoc analysis. Conclusions In this analysis of US surgical residents, operative independence was strongly correlated with self-efficacy.

Association between rural exposure/experience and practice location 10 years postgraduation, stratified by specialty: evidence from a cohort study of graduates from nine Australian universities

ObjectiveThis study aims to determine the associations between specialty type and practice location at postgraduate year 10 (PGY10), matched with PGY5 and PGY8 work locations, and earlier rural exposure/experience.Design and...

Large language foundation models encode clinical radiation oncology domain knowledge: Performance on the American College of Radiology Standardized Examination.

e13585 Background: The integration of Large Language Models (LLMs) into healthcare and medical education will represent a significant paradigm shift, offering transformative potential in how medical knowledge is accessed and assimilated. These models, however, have not yet been systematically trained, tested, or validated on complex medical information such as sub-specialty medical examinations. This study explores the performance of seven major LLMs in clinical radiation oncology using residency in-training exams. Methods: In this study, the 2021 American College of Radiology (ACR) Radiation Oncology In-Training Exam (TXIT) was used to evaluate the performance of various LLMs, including OpenAI's GPT-3.5-turbo, GPT-4, GPT-4-turbo, three Meta's Llama-2 models (7 billion, 13 billion, and 70 billion parameter), and Google's PaLM-2-text-bison. The ACR provided the publicly available national scoring for this exam. The exam comprised 298 questions across 13 domains, including clinical radiation oncology (195 questions, 65.4%). The exam was processed through each LLM via an application programming interface. LLM-generated answers were analyzed by clinical disease sites and compared to Radiation Oncology trainee performance and stratified by Post-Graduate Year (PGY) 2-5. Results: LLMs showed varied performance in the overall clinical radiation oncology domain, with OpenAI's GPT-4-turbo having the best performance with 68.0% correct answers, GPT-4 61.0%, GPT-3.5-turbo 48.0%, PaLM-2-text-bison 40.0%, and then the three Llama-2 models (70b 37.0%, 13b 38%, 7b 26%). GPT-4-turbo performed superiorly to lower-level (PGY2 51.6%, PGY3 61.6%) and comparably to upper-level radiation oncology trainees (PGY4 64.1%, PGY5 68.3%). Notably, GPT-4-turbo demonstrated 7.0% improvement over its predecessor GPT-4. LLMs scored the lowest in the gastrointestinal, genitourinary, and gynecology domains and highest in the bone and soft tissue, central nervous system and eye, and head, neck, and skin domains. Conclusions: GPT-4-turbo demonstrates clinical accuracy comparable to upper-level and superior to lower-level trainees in nearly all clinical domains. Conversely, Llama2 foundation models demonstrate overall worse performance than Level 1 (PGY2) trainees. Score discrepancies across disease site domains may be due to data availability, complexity of medical conditions, quality and quantity of training datasets, and interdisciplinary data inputs. Future research will need to evaluate the performance of models that are fine-tune trained in clinical oncology. This study also underscores the need for rigorous validation of LLM-generated information against established medical literature and expert consensus, necessitating expert oversight in their application in medical education and practice.

Comparing Perceptions of Virtual and In-Person Interviews: Results From a National Survey of Internal Medicine Residents.

Background Since 2020, virtual interviews have become the typical way in which applicants assess residency programs. It is unknown whether the change from in-person to virtual interviews has been associated with changes in perceptions of the quality of information gathered by prospective applicants. Objective To ascertain perspectives on the satisfaction with, quality of, and accuracy of information gathered by internal medicine (IM) residency applicants from virtual and in-person interviews. Methods Twenty-nine thousand, seven hundred and seventy-six residents from US and Puerto Rico residency programs sitting for the 2022 American College of Physicians Internal Medicine In-Training Examination (IM-ITE) were surveyed. An optional, 5-question survey was administered at the end of the examination. Responses were analyzed based on interview format-virtual (postgraduate year [PGY]-1-2) or in-person (PGY-3)-and PGY. Results Of 29 776, 23 161 residents responded to the survey (77.8% response rate). Regardless of PGY, respondents reported a high degree of satisfaction with the quality of information gathered from their interview day, though there was a statistically significant difference between virtual and in-person [somewhat/very satisfied: In-person 5938 of 7410 (80.1%); 95% CI [79.2, 81.0] vs virtual 12 070 of 15 751 (76.6%); 95% CI [76.0, 77.3]:P<.001]. Residents in all PGYs reported sessions with residents and one-on-one interviews as the most important factors when creating their rank lists. Conclusions We found differences in satisfaction and perceptions of the quality of information gathered between IM residents who participated in virtual and in-person interviews. However, regardless of format, most respondents reported satisfaction with their interview experience.

Hospital Use of a Web-Based Clinical Knowledge Support System and In-Training Examination Performance Among Postgraduate Resident Physicians in Japan: Nationwide Observational Study.

The relationship between educational outcomes and the use of web-based clinical knowledge support systems in teaching hospitals remains unknown in Japan. A previous study on this topic could have been affected by recall bias because of the use of a self-reported questionnaire. We aimed to explore the relationship between the use of the Wolters Kluwer UpToDate clinical knowledge support system in teaching hospitals and residents' General Medicine In-Training Examination (GM-ITE) scores. In this study, we objectively evaluated the relationship between the total number of UpToDate hospital use logs and the GM-ITE scores. This nationwide cross-sectional study included postgraduate year-1 and -2 residents who had taken the examination in the 2020 academic year. Hospital-level information was obtained from published web pages, and UpToDate hospital use logs were provided by Wolters Kluwer. We evaluated the relationship between the total number of UpToDate hospital use logs and residents' GM-ITE scores. We analyzed 215 teaching hospitals with at least 5 GM-ITE examinees and hospital use logs from 2017 to 2019. The study population consisted of 3013 residents from 215 teaching hospitals with at least 5 GM-ITE examinees and web-based resource use log data from 2017 to 2019. High-use hospital residents had significantly higher GM-ITE scores than low-use hospital residents (mean 26.9, SD 2.0 vs mean 26.2, SD 2.3; P=.009; Cohen d=0.35, 95% CI 0.08-0.62). The GM-ITE scores were significantly correlated with the total number of hospital use logs (Pearson r=0.28; P<.001). The multilevel analysis revealed a positive association between the total number of logs divided by the number of hospital physicians and the GM-ITE scores (estimated coefficient=0.36, 95% CI 0.14-0.59; P=.001). The findings suggest that the development of residents' clinical reasoning abilities through UpToDate is associated with high GM-ITE scores. Thus, higher use of UpToDate may lead physicians and residents in high-use hospitals to increase the implementation of evidence-based medicine, leading to high educational outcomes.

Creating a win-win: A novel layered learning approach for assessing first-year pharmacy students' communication skills

Background and purposeIndividualized assessment of students in skills-based courses is essential for practice readiness, however recruiting evaluators is challenging. Our school of pharmacy offers a teaching certificate program for postgraduate year one pharmacy residents (PGY1 residents; PGY1s) which requires completion of a teaching experience. The longitudinal layered learning assessment experience (LLLAE) was designed to meet instructional needs for individualized assessment of first-year pharmacy students' communication skills and provide a meaningful teaching opportunity for PGY1s. This manuscript describes the implementation and evaluates the impact of the LLLAE. Educational activity and settingPGY1s were invited to participate in the yearlong LLLAE. Faculty developed PGY1s' skills through training sessions, direct observation, and debrief sessions. PGY1s evaluated students and provided feedback using pre-defined criteria during 3 summative assessments (capstones). Capstones were common pharmacy practice scenarios in which students interact with an actor serving as a standardized patient or provider. PGY1s completed an end-of-year survey to self-rate their improvement in evaluating students, providing feedback, and confidence. FindingsTwenty-two PGY1s participated in the LLLAE over 2 years. They evaluated 73.2% of total capstone interactions. The end-of-year survey was completed by 20 PGY1s (90.9% response rate). All respondents indicated improvement in their skills to evaluate students, ability to provide feedback, and confidence. Scores were 4 or 5 (scale of 1 to 5) across all measures. Key contributors fostering improvement were the training and debrief sessions, faculty feedback, pre-defined criteria, and multiple practice opportunities. SummaryThis novel layered learning approach was a win-win for faculty and PGY1s. The approach improved feasibility for faculty to continue individualized student assessment while mentoring early career pharmacists. PGY1s gained an opportunity to contribute to student growth, learn from experienced faculty, and develop skills for practice. Additionally, students benefited from individualized feedback.

Preparedness of Speech Language Pathologists and Occupational Therapists to Treat Pediatric Feeding Disorder: A Cross-Sectional Survey.

Pediatric feeding disorder (PFD) is increasingly common and is often treated by speech language pathologists (SLPs) and occupational therapists (OTs) in the community setting. However, the preparedness of these disciplines to effectively address PFD is relatively unknown. A national (US), online survey was disseminated to providers who assess and treat PFD. For the present analysis, the responses of SLPs (N = 418) and OTs (N = 195) related to their clinical background, educational background, post-graduate training, and self-rated clinical effectiveness were statistically analyzed and compared across the two disciplines. Both SLPs and OTs report feeling underprepared to work with PFD clients immediately following their academic training, but time spent in post-graduate training and years of clinical practice both significantly (p < 0.0001) increased feelings of effectiveness in assessing and treating PFD. Most SLPs and OTs pursued self-directed learning activities to increase competence, with the most common activities being article review, podcasts, and peer case review, although SLPs were significantly more likely to use podcasts (p < 0.0001) and peer review (p = 0.0004) than OTs. The most common barriers for providers were financial, time, travel, and institutional support barriers. While PFD is a key practice area of both SLPs and OTs, both provider groups feel unprepared and under-supported in providing competent care to these patients upon graduation. Future research and policy should support advancements in training for current SLPs and OTs related to PFD and address current barriers to a specialized educational pathway.