Trained Examiners Research Articles

Artificial Intelligence Large Language Models Are Nearly Equivalent to Fourth-Year Orthopaedic Residents on the Orthopaedic In-Training Examination: A Cause for Concern or Excitement?

Background The rapid improvement of generative artificial intelligence (AI) models in medical domains including answering board-style questions warrants further investigation regarding their utility and accuracy in answering orthopaedic surgery written board questions. Previous studies have analyzed the performance of ChatGPT alone on board exams, but a head-to-head analysis of multiple current AI models has yet to be performed. Hence, the objective of this study was to compare the utility and accuracy of various large language models (LLMs) in answering Orthopaedic Surgery In-Training Exam (OITE) written board questions to each other as well as orthopaedic surgery residents. Methods A complete set of questions from the OITE 2022 exam was inputted into various LLMs and results were calculated and compared against orthopaedic surgery residents nationally. Results were analyzed by overall performance and question type. Type A questions related to knowledge and recall of facts, Type B questions involved diagnosis and analysis of information, and Type C questions focused on the evaluation and management of diseases, requiring knowledge and reasoning to develop treatment plans. Results Google Gemini was the most accurate tool answering 69.9% of questions correctly. Google Gemini also performed superiorly to ChatGPT and Claude on Type A (76.9%) and Type C questions (67.4%), with Claude performing superiorly on Type B questions (70.7%). Questions without images were answered with greater accuracy compared to those with images (65.9% vs. 34.1%). All LLMs performed above the average of a first-year orthopaedic surgery intern, with Google Gemini and Claude performance approaching that of fourth- and fifth-year orthopaedic surgery residents. Conclusion The study assessed LLMs like Google Gemini, ChatGPT, and Claude against orthopaedic surgery residents on the OITE. Results showed that these LLMs perform on par with orthopaedic surgery residents, with Google Gemini achieving best performance overall and in Type A and C questions while Claude performed best in Type B questions. LLMs have the potential to be used to generate formative feedback and interactive case studies for orthopaedic trainees.

Online Education in Orthopaedic Surgery: A Current Concept Review

This allows for blended learning and flipped classroom opportunities for the reduced in-person didactic time that orthopaedic surgical faculty have with residents today. Exam preparation for the Orthopaedic In-Training Exam (OITE) and the American Board of Orthopaedic Surgeons (ABOS) is now very commonly done with online platforms. Journal club activities have also shifted to the virtual space; podcasts provide a new way to learn when on the go, and virtual and augmented reality technologies are starting to make their way into surgical training. Evidence for these learning methods is steadily growing, although more extensive studies are needed to determine their efficacy. This review will assess the evidence available for these online education methods in orthopaedic surgery.

Perceived impact of abortion access on quality of cancer care and on future practice location decisions among oncology fellow physicians: An ASCO State of Cancer Care in America study.

43 Background: The Supreme Court decision in Dobbs v. Jackson Women’s Health Organization (2022) allows states to ban or limit abortion without health exceptions for pregnant persons. Notably, about 1 out of every 1000 pregnant persons has cancer and abortion is utilized in up to 28% of pregnant patients with cancer. To investigate the impact of this court decision on oncology practice, we studied perceptions among oncology fellows of the effect of abortion restrictions on quality care and whether they anticipate care implications to influence their decisions on future practice location. Methods: Fellows who took the 2024 ASCO Medical Oncology In-Training Exam (ITE) were invited to take an optional post-ITE survey assessing perceived (1) effect of abortion policies on ability to provide 9 key aspects of quality cancer care, including care access and delays, trust, and professional well-being and (2) impact of abortion restrictions on quality care in decisions of future practice location. Agreement (Agree, Strongly Agree) and impact on future practice location (Somewhat likely, Very likely) were compared by respondent gender and abortion policies (banned/under legal contention vs legal) in their own or neighboring state using chi-squared analysis. Results: Of 1884 US medical oncology fellows who took the ITE, 1254 (67%, inc. 49% male, 46% female) participated in this survey; 34% trained in states with bans, 44% in states adjacent to states with bans, and 22% in and adjacent to states where abortion is legal. Overall, 33% of fellows report that abortion restrictions affected their ability to deliver at least one aspect of quality care, including 22% reporting limited availability/delays of drugs and 28% reporting poorer job satisfaction. For all 9 aspect of quality care, agreement was strongly associated with local abortion status (p<0.001). For example, 37% of fellows in abortion-restricted states report reduced patient trust and relationship quality vs. 20% among those in states adjacent to bans vs. 18% among those in and adjacent to legal states (p<0.001). Over half (51%) of fellows report that they are likely to consider the impact of local abortion policies on quality care when deciding future practice location (28% were neutral, 21% unlikely to consider). More fellows training in states where abortion is legal report being likely to consider the impact of local abortion policies on care in future practice location decisions (55%) than those training in abortion-restricted states (45%) (p<0.001). More women (58%) than men (45%) report being likely to consider the impact of local abortion policies on quality care in future practice location (p<0.001). Conclusions: Findings provide early insights into the impact of abortion restrictions both on key aspects of quality care as well as the geographic distribution of future oncologists.

Does exposure matter? Association between exposure to clinical trial conduct during training and intent to participate in clinical trials post-training among oncology fellows: An ASCO State of Cancer Care in America study.

69 Background: Clinical trials are part of standard cancer care. Yet, a quarter of oncology fellows in a 2023 survey reported feeling underprepared to conduct research after training and nearly half did not identify research as important to their practice. In a 2024 survey, we explore the impact of exposure to research activities in fellowship on intentions to conduct research post-training. Methods: Fellows who took the 2024 ASCO Medical Oncology In-Training Exam (ITE) were invited to an optional IRB-exempted post-exam survey to measure (1) research exposure during training and (2) anticipated research participation once in practice. Responses of moderate and great amount are combined as frequent (vs. never, rarely, occasionally as not frequent). Rates of frequent research engagement in future practice are compared by frequent/not frequent research exposure during fellowship, expected practice setting, and program year using chi-squared analysis. Results: Of the 1884 US medical oncology fellows taking the ITE, 1491 (79%) participated in this study, 33% 1st year, 37% 2nd year, 39% 3rd year. 891 (60%) expect their first post-training role to be in academics, 527 (35%) in a non-academic practice, and 67 (5%) in other settings. Anticipated engagement in enrollment and management of participants on clinical trials post-training was associated with exposure to those activities in fellowship and with expected practice setting. Overall, 827 (63%) anticipate frequent participation in research once in practice. 531 (76%) of 701 with frequent exposure in training compared to 296 (37%) of 790 with not frequent exposure anticipate frequent participation in practice (p<0.001). 610 (78%) of fellows expecting to practice in an academic setting vs. 192 (40%) in a non-academic setting anticipate frequent future participation in practice (p<0.001). For research leadership activities (writing clinical research protocols, applying for clinical research grants, serving as principal investigator), 622 (49%) anticipate frequent participation once in practice. 390 (79%) of 519 with frequent vs. 229 (29%) of 855 with not frequent exposure to leadership activities during fellowship anticipate frequent future engagement (p<0.001). 523 (67%) of fellows expecting to practice in an academic setting vs. 76 (16%) in a non-academic setting anticipate frequent leadership engagement after training (p<0.001). Anticipated future participation did not vary by program year for either type of research activity. Conclusions: Findings show an urgent need to increase exposure to research during fellowship training to improve participation in clinical trials by practicing oncologists. This is a critical component of improving patient access to research.

Curricular Innovation to Improve Performance on Internal Medicine In-Training Exam Nephrology Items

Curricular Innovation to Improve Performance on Internal Medicine In-Training Exam Nephrology Items

Burnout is not associated with trainee performance on the Vascular Surgery In-Training Exam

ObjectiveThe Vascular Surgery In-Training Examination (VSITE) is a yearly exam evaluating vascular trainees’ knowledge base. Although multiple studies have evaluated variables associated with exam outcomes, few have incorporated training program-specific metrics. The purpose of this study is to evaluate the impact of the learning environment and burnout on VSITE performance. MethodsData was collected from a confidential, voluntary survey administered after the 2020 to 2022 VSITE as part of the SECOND Trial. VSITE scores were calculated as percent correct then standardized per the American Board of Surgery. Generalized estimating equations with robust standard errors and an independent correlation structure were used to evaluate trainee and program factors associated with exam outcomes. Analyses were further stratified by integrated and independent training paradigms. ResultsA total of 1385 trainee responses with burnout data were collected over 3 years (408 in 2020, 459 in 2021, 498 in 2022). On average, 46% of responses reported at least weekly burnout symptoms. On unadjusted analysis, burnout symptoms correlated with a 14 point drop in VSITE score (95% confidence interval [CI], −24 to −4; P = .006). However, burnout was no longer significant after adjusted analysis. Instead, higher postgraduate year level, being in a relationship, identifying as male gender with or without kids, identifying as non-Hispanic white, larger programs, and having a sense of belonging within a program were associated with higher VSITE scores. ConclusionsDespite high rates of burnout, trainees generally demonstrate resilience in gaining the medical knowledge necessary to pass the VSITE. Performance on standardized exams is associated with trainee and program characteristics, including availability of support systems and program belongingness.

Critical Care Exposure and Preparedness Among Internal Medicine Residents: Results from the American College of Physicians In-Training Exam Survey

Critical Care Exposure and Preparedness Among Internal Medicine Residents: Results from the American College of Physicians In-Training Exam Survey

Performance of Language Models on the Family Medicine In-Training Exam.

Artificial intelligence (AI), such as ChatGPT and Bard, has gained popularity as a tool in medical education. The use of AI in family medicine has not yet been assessed. The objective of this study is to compare the performance of three large language models (LLMs; ChatGPT 3.5, ChatGPT 4.0, and Google Bard) on the family medicine in-training exam (ITE). The 193 multiple-choice questions of the 2022 ITE, written by the American Board of Family Medicine, were inputted in ChatGPT 3.5, ChatGPT 4.0, and Bard. The LLMs' performance was then scored and scaled. ChatGPT 4.0 scored 167/193 (86.5%) with a scaled score of 730 out of 800. According to the Bayesian score predictor, ChatGPT 4.0 has a 100% chance of passing the family medicine board exam. ChatGPT 3.5 scored 66.3%, translating to a scaled score of 400 and an 88% chance of passing the family medicine board exam. Bard scored 64.2%, with a scaled score of 380 and an 85% chance of passing the boards. Compared to the national average of postgraduate year 3 residents, only ChatGPT 4.0 surpassed the residents' mean of 68.4%. ChatGPT 4.0 was the only LLM that outperformed the family medicine postgraduate year 3 residents' national averages on the 2022 ITE, providing robust explanations and demonstrating its potential use in delivering background information on common medical concepts that appear on board exams.

Team-Based Learning as a Feasible, Acceptable, and Effective Educational Method for Internal Medicine Trainees - A Multi-phase Intervention and Cohort Comparison.

Team-based learning (TBL) is an active learning strategy gaining traction in medical education. However, studies demonstrating successful incorporation into Graduate Medical Education (GME) curricula are limited. To assess the feasibility, acceptability and efficacy of Infectious Disease (ID) TBL sessions within an Internal Medicine (IM) residency curriculum as part of a traditional 60-minute conference. We conducted a prospective cohort study of TBL implementation assessing acceptability and feasibility (Phase 1), and efficacy (Phase 2). Phase 1 included 101 IM residents and eight TBL naïve faculty. Phase 2 included aggregate cohort IM In-Training Exam (ITE) data before (2008-2013) and after (2014-2019) TBL implementation. Eight TBL sessions were delivered once or twice weekly during 60-minute noon conferences. We assessed feasibility by measuring individual Readiness Assurance Test (iRAT) completion rates and inclusion of TBL elements in each session; acceptability through attendance, perceived effectiveness rating and attitudes about TBL; efficacy by comparing ITE data for the overall ID content and specific TBL associated learning objectives. Seventy-five of 93 (80%) residents attended at least one session. All TBL elements were successfully incorporated each session. Of those surveyed, 86% rated the TBL sessions as facilitating their learning "very (4)" or "extremely (5)" well on a 5-point Likert scale (p<0.001). ITE mean percent correct scores of total ID content as well as TBL associated learning objective performance were both significantly higher for the post-TBL cohort among PGY-2 (76.2 vs 62.3; 76.2 vs 62.6) and PGY-3 (73 vs 64.5; 76.2 vs 64.5) IM residents (p<0.05; p<0.001 respectively). Implementing a complete TBL pedagogy within the traditional noontime conference hour in GME is feasible, acceptable to residents and faculty, and associated with improved learning efficacy demonstrated through improved ITE scores.

Large language foundation models encode clinical radiation oncology domain knowledge: Performance on the American College of Radiology Standardized Examination.

e13585 Background: The integration of Large Language Models (LLMs) into healthcare and medical education will represent a significant paradigm shift, offering transformative potential in how medical knowledge is accessed and assimilated. These models, however, have not yet been systematically trained, tested, or validated on complex medical information such as sub-specialty medical examinations. This study explores the performance of seven major LLMs in clinical radiation oncology using residency in-training exams. Methods: In this study, the 2021 American College of Radiology (ACR) Radiation Oncology In-Training Exam (TXIT) was used to evaluate the performance of various LLMs, including OpenAI's GPT-3.5-turbo, GPT-4, GPT-4-turbo, three Meta's Llama-2 models (7 billion, 13 billion, and 70 billion parameter), and Google's PaLM-2-text-bison. The ACR provided the publicly available national scoring for this exam. The exam comprised 298 questions across 13 domains, including clinical radiation oncology (195 questions, 65.4%). The exam was processed through each LLM via an application programming interface. LLM-generated answers were analyzed by clinical disease sites and compared to Radiation Oncology trainee performance and stratified by Post-Graduate Year (PGY) 2-5. Results: LLMs showed varied performance in the overall clinical radiation oncology domain, with OpenAI's GPT-4-turbo having the best performance with 68.0% correct answers, GPT-4 61.0%, GPT-3.5-turbo 48.0%, PaLM-2-text-bison 40.0%, and then the three Llama-2 models (70b 37.0%, 13b 38%, 7b 26%). GPT-4-turbo performed superiorly to lower-level (PGY2 51.6%, PGY3 61.6%) and comparably to upper-level radiation oncology trainees (PGY4 64.1%, PGY5 68.3%). Notably, GPT-4-turbo demonstrated 7.0% improvement over its predecessor GPT-4. LLMs scored the lowest in the gastrointestinal, genitourinary, and gynecology domains and highest in the bone and soft tissue, central nervous system and eye, and head, neck, and skin domains. Conclusions: GPT-4-turbo demonstrates clinical accuracy comparable to upper-level and superior to lower-level trainees in nearly all clinical domains. Conversely, Llama2 foundation models demonstrate overall worse performance than Level 1 (PGY2) trainees. Score discrepancies across disease site domains may be due to data availability, complexity of medical conditions, quality and quantity of training datasets, and interdisciplinary data inputs. Future research will need to evaluate the performance of models that are fine-tune trained in clinical oncology. This study also underscores the need for rigorous validation of LLM-generated information against established medical literature and expert consensus, necessitating expert oversight in their application in medical education and practice.

Development of a neonatal cardiac curriculum for neonatal-perinatal medicine fellowship training.

The topic of neonatal cardiovascular care in neonatal-perinatal medicine (NPM) fellowship training has continued to transform due to increased complexity of patients, development of specialized units, continued Accreditation Council for Graduate Medical Education requirements, and clinical practice variation across centers that care for neonates with congenital heart disease (CHD). We developed a neonatal cardiac curriculum comprised of eight interactive sessions with novel active learning concepts specific to our NPM fellows. A self-assessment survey in comfort in managing infants with CHD and perceived competency in neonatal cardiology topics was performed by all neonatology fellows at baseline and after completion of the curriculum. The American Board of Pediatrics Subspecialty In-training Exam (SITE) scores for fellows were compared to that of the national average. The average comfort score (0-100) of the first-year fellows increased from 33 to 76, and that of the second and third-year fellows increased from 72 to 86, and 75 to 86, respectively. The first-year fellows improved their competency score by 44 points (3 standard deviations), the second-year fellows improved their score by about 26 points (one standard deviation), and there was an overall 9-point increase in the competency score of all fellows (one standard deviation). The average local SITE score was lower than the national average before the initiation of this curriculum, became nearly equal to the national average score at the end of the first year the curriculum was implemented, and has progressively become higher since then. Due to the variable clinical exposure and differing practice models of managing CHD a neonatal cardiac curriculum may be beneficial to NPM trainees.

Validity evidence for a novel multiple-choice question bank in the field of child abuse pediatrics

BackgroundThough child abuse pediatrics has been a board-certified subspecialty for 15 years, there are few formalized board preparation resources available. ObjectiveThe purpose of this project was to establish a multiple-choice question bank with sufficient validity evidence for use in preparation for the child abuse pediatrics board examination. Participants and settingThe question bank was distributed via an electronic child abuse pediatrics mailing list. Participants completing the entire question bank included 27 board-certified child abuse pediatricians (CAPs), 19 board-eligible CAPs, and 18 CAP fellows. MethodsWe used Messick's framework to conduct the validity investigation, which includes five components: content evidence, response process, internal structure, relation to other variables, and consequences. Item analyses included difficulty index, discrimination index, and distractor analysis. We used Cronbach's alpha to estimate internal consistency reliability. We conducted linear regressions of scores on the question bank compared to in-training exam scores and career stage. ResultsEighty-four participants completed part of the question bank, and 64 completed the entire question bank. Of the original 117 questions (“items”), 94 met inclusion criteria. The mean score among board-certified CAPs was 80 %, and among participants reporting passing third-year ITE scores was 81 %. Correlation coefficient of scores on this question bank by career stage was r = 0.94, and by year of fellowship was r = 0.99. Cronbach's alpha for internal consistency reliability was 0.83. ConclusionsThis multiple-choice question bank is the first question bank with a robust validity investigation for use by child abuse pediatrics trainees.

Screening for Conscientiousness at Start of Residency: Early Detection of Professionalism Concerns.

Early identification of residents who may require remediation could help prevent problems for both individual residents and their programs. Our aim in this project was to identify whether residents prone to challenges with professionalism could be identified early. For 3 years we tracked onboarding tasks completed by residents between official match and the first day of residency to develop a tool that would help identify residents who may be at risk for problems with timely chart completion and subsequent remediation. We compared residents' "at risk" scores with in-training exam (ITE) scores to differentiate between professionalism (task completion issues) and medical knowledge. Our at-risk tool successfully predicted timely chart completion rates and need for remediation within our residency program (Fischer's exact P<.001), but was not predictive of ITE scores. Using readily accessible information, we can identify residents at risk for poor task completion performance, who may benefit from extra support and development in the area of organizational skill. Early identification may increase opportunities for early intervention.

Testing the Effects of Individual Residents' Retrieval Practice on Standardized Examination Scores.

Residents have limited time and much to learn. Retrieval practice-studying using exam-style review questions-is a powerful educational tool for aggregate groups of learners, but it has not been tested for individual graduate medical education (GME) trainees. We conducted a retrospective observational study examining how individual residents' self-directed retrieval practice affected their learning. We used linear regression models to compare the number of exam-style multiple-choice practice questions each resident answered per year to their scores on annual In-Training Exams (ITE). We found that the higher a resident's ITE score was at the start of a year, the fewer practice questions they answered that year for both first- (p = 0.023) and second-year (p = 0.020) trainees. Then, the more questions a resident answered in a year, the more their ITE score increased over that year for both first- and second-year trainees (p = 0.026 and 0.025, respectively). Residents' prior ITE scores also independently predicted their subsequent ITE scores (p = 0.024 in the first and 0.007 in the second year), and the effect of their baseline scores was larger than the effect of the number of practice questions they answered. Individual residents' prior exam scores significantly predict their future exam scores, and their independent retrieval practice using exam-style review questions also significantly predicts the annual improvement in their scores. These findings add to the growing evidence supporting retrieval practice in GME, and they can inform individualized educational coaching for GME trainees.

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ±63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

Foundations of Emergency Medicine: Impact of a Standardized, Open-access, Core Content Curriculum on In-Training Exam Scores.

Learners frequently benefit from modalities such as small-group, case-based teaching and interactive didactic experiences rather than passive learning methods. These contemporary techniques are features of Foundations of Emergency Medicine (FoEM) curricula, and particularly the Foundations I (F1) course, which targets first-year resident (PGY-1) learners. The American Board of Emergency Medicine administers the in-training exam (ITE) that provides an annual assessment of EM-specific medical knowledge. We sought to assess the effect of F1 implementation on ITE scores. We retrospectively analyzed data from interns at four EM residency programs accredited by the Accreditation Council for Graduate Medical Education. We collected data in 2021. Participating sites were geographically diverse and included three- and four-year training formats. We collected data from interns two years before (control group) and two years after (intervention group) implementation of F1 at each site. Year of F1 implementation ranged from 2015-2018 at participating sites. We abstracted data using a standard form including program, ITE raw score, year of ITE administration, US Medical Licensing Exam Step 1 score, Step 2 Clinical Knowledge (CK) score, and gender. We performed univariable and multivariable linear regression to explore differences between intervention and control groups. We collected data for 180 PGY-1s. Step 1 and Step 2 CK scores were significant predictors of ITE in univariable analyses (both with P < 0.001). After accounting for Step 1 and Step 2 CK scores, we did not find F1 implementation to be a significant predictor of ITE score, P = 0.83. Implementation of F1 curricula did not show significant changes in performance on the ITE after controlling for important variables.

Perspectives From the RadDiscord Annual Survey: Overview of the Top Study Tools and Evaluation of Study Time and Various Resources

Each year, senior radiology residents take the American Board of Radiology Qualifying (Core) exam to evaluate competency. Approximately 10% of first-time examinees will fail this exam (1). Understanding factors that contribute to success will help residency program directors and trainees prepare for future exams. RadDiscord (www.raddiscord.org), an international radiology educational community, is in the unique position to evaluate different study materials and resources. The goal of this paper is to report the results from the RadDiscord survey and analyze the factors that correlate with higher exam performance and passing. Following the February 2021, June 2021, and June 2022 exams, RadDiscord members were provided an anonymous survey, collecting information on study resources and exam scores. The collected data were analyzed using various statistical methods. Both descriptive and inferential analyses were performed. A total of 318 residents responded (95% passed). Significant variability in Qualifying (Core) exam performance and perceived quality of internal didactics existed between program types. Residents who did less than 2000 practice questions performed lower on the exam. The Diagnostic Radiology In-Training (DXIT) exam was the most predictive for passing and performance. Qualifying (Core) exam performance negatively correlated with study time, though certain residents did receive some benefit from study time. Many factors correlate with passing and Qualifying (Core) exam performance. Residency programs with fewer resources should consider alternative ways to support residents beyond offering study time. Residents who complete at least 2000 practice questions are more likely to pass and DXIT results can be a useful gauge to identify exam readiness.

Assessing the impact of adopting a competency-based medical education framework and ACGME-I accreditation on educational outcomes in a family medicine residency program in Abu Dhabi Emirate, United Arab Emirates.

Competency-Based Medical Education (CBME) is now mandated by many graduate and undergraduate accreditation standards. Evaluating CBME is essential for quantifying its impact, finding supporting evidence for the efforts invested in accreditation processes, and determining future steps. The Ambulatory Healthcare Services (AHS) family medicine residency program has been accredited by the Accreditation Council of Graduate Medical Education-International (ACGME-I) since 2013. This study aims to report the Abu Dhabi program's experience in implementing CBME and accreditation. Compare the two residents' cohorts' performance pre-and post-ACGME-I accreditation.Study the bi-annually reported milestones as a graduating residents' performance prognostic tool. All residents in the program from 2008 to 2019 were included. They are called Cohort one-the intake from 2008 to 2012, before the ACGME accreditation, and Cohort two-the intake from 2013 to 2019, after the ACGME accreditation, with the milestones used. The mandatory annual in-training exam was used as an indication of the change in competency between the two cohorts. Among Cohort two ACGME-I, the biannual milestones data were studied to find the correlation between residents' early and graduating milestones. A total of 112 residents were included: 36 in Cohort one and 76 in Cohort two. In Cohort one, before the ACGME accreditation, no significant associations were identified between residents' graduation in-training exam and their early performance indicators, while in Cohort two, there were significant correlations between almost all performance metrics. Early milestones are correlated with the graduation in-training exam score. Linear regression confirmed this relationship after controlling the residents' undergraduate Grade Point Average (GPA). Competency development continues to improve even after residents complete training at Post Graduate Year, PGY4, as residents' achievement in PGY5 continues to improve. Improved achievement of residents after the introduction of the ACGME-I accreditation is evident. Additionally, the correlation between the graduation in-training exam and graduation milestones, with earlier milestones, suggests a possible use of early milestones in predicting outcomes.

Wellbeing and burnout among oncology fellows compared to prior fellows and current practicing oncologists: An ASCO State of Cancer Care in America (SOCCA) trend analysis.

47 Background: Burnout impacts oncology clinicians across their professional lifecycle. To better understand clinician wellbeing and the role of fellowship training, we characterized trends in oncology fellows’ burnout and perception of preparedness during training. Methods: As an ASCO SOCCA initiative, all medical oncology fellows taking the 2023 ASCO Medical Oncology In-Training Exam (ITE) were invited to complete an optional post-ITE professional wellbeing survey. Fellows were asked 1 item on fatigue (on a 0-10 Likert scale, 10 = “as bad as can be”), 1 item on satisfaction with work-life integration (WLI) and 2 Maslach Burnout Inventory (MBI) items on emotional exhaustion (EE) and depersonalization (DP), each scored on a 1-7 scale, &gt;5 being high. Per convention, respondents with a high EE and/or DP were categorized as having burnout. 2023 ITE responses were compared to (1) 2013 ASCO Fellows Post-ITE survey (n=1345),1 (2) 2012-13 ASCO survey of practicing oncologists (n=1117),2 and (3) 2023 ASCO survey of practicing oncologists (n=328; 26% tagged as early career, using age &lt;40 as proxy; 64% in academia). Results: 2231 fellows who took the ITE in 2023. Responses from 1568 first (33%), second (36%) and third year (31%) fellows who responded to at least one wellbeing item are included in this analysis. Compared to 10 years prior, oncology fellows in 2023 reported less burnout (20% in 2023 vs 34% in 2013, p&lt;.001) and higher satisfaction with WLI (50% in 2023 vs 34% in 2013; p&lt;.001). Fellows across both 2013 and 2023 studies reported higher satisfaction with WLI with advancing fellowship year (1-3). The 2013 analyses found no statistically significant difference in EE, DP or overall burnout between fellows and practicing oncologists. However, in 2023, the prevalence of high EE, DP and burnout was significantly lower in fellows vs both all and early career practicing oncologists (all p&lt;.005, Table). Most (72%) fellows in 2023 felt their training is adequately preparing them to manage burnout in future practice. Conclusions: Oncology fellows in 2023 reported significantly less burnout and higher WLI satisfaction than fellows in 2013 and current practicing oncologists. Most fellows felt prepared to manage burnout in future practice. The higher prevalence of burnout in early career oncologists vs fellows suggests a gap between expectations of practice during training and the actual experience of oncology practice. (1) Shanafelt et al: JCO 32(27), 2014. (2) Shanafelt et al: JCO 32(11), 2014. [Table: see text]

Multilevel factors impacting clinical trial participation: An ASCO State of Cancer Care in America (SOCCA) analysis of oncology fellows’ perceptions of preparedness to conduct clinical research in practice and importance of clinical trial participation.

110 Background: Despite cancer treatment advances from clinical trials, the proportion of those with cancer enrolling on oncology studies remains low, due to system-, patient-, and physician-level facilitators and barriers to participation. To better understand the upstream context impacting oncologists’ consideration of trials, we studied oncology fellows’ perceptions of trial participation and preparedness for clinical research. Methods: All year 1 (Y1), Y2, &amp; Y3 fellows taking the 2023 ASCO Medical Oncology In-Training Exam (ITE) were invited to take a post-ITE optional survey. After consent, fellows answered whether they agree, disagree or are unsure of their preparedness by their fellowship program to address key elements of practice and research and their perceived importance of trial participation after training, ranging from Very Important, Important, Moderately, Slightly, or Not Important, the latter 3 ratings aggregated and labeled as ‘less important.’ Descriptive statistics and Chi-squared tests comparing responses by fellowship year are presented. Results: Overall, 97% of fellows (2184/2231) taking the 2023 ITE participated in this study and are included in this analysis; 32% were in Y1, 37% Y2, 30% Y3. When asked ‘my fellowship training program is adequately preparing me to address the following in my future,’ nearly one in four fellows felt unsure or disagreed that their program prepared them for (a) treating patients outside of an academic medical center (Y1 24%, Y2 24%, Y3 23%) or (b) conducting clinical research (Y1 26%, Y2 26%, Y3 28%). When asked ‘how important is it to you to have the opportunity to participate in clinical trials following completion of your training,’ nearly half of oncology fellows (Y1 49%, Y2 46%, Y3 40%, p.038) felt trial participation was ‘less important.’ When considering future practice setting, two out of 3 fellows (66%) planning a non-academic practice and one of 3 fellows (33%) planning an academic practice rated trial participation as ‘less important’ (p&lt;.0001). When compared to fellows who reporting being unsure or unprepared by their programs to conduct clinical research, those who felt adequately prepared were more likely to rate trial participation as important/very important (57% vs 49%, p=.005). Conclusions: Our findings show that value of clinical trial participation is assigned early in oncologists’ careers during fellowship training. Addressing barriers to clinical trial participation calls for multilevel interventions that include efforts to identify and augment clinical research learning needs of oncology fellows and ensuring broad exposure during fellowship to successful and efficient conduct of cancer clinical trials.