In-training Exam Scores Research Articles

P-960. Improving Resident Education on the Inpatient Infectious Disease Consult Service

Abstract Background Education in infectious disease (ID) is a critical part of an internal medicine curriculum. As the infectious disease consult service is typically high volume and busy, residents and other trainees were often not receiving adequate training in less common infectious conditions or didactic training in common conditions that may be relevant for their future practice or internal medicine board exams. Our Quality Improvement project sought to improve training for residents and medical students through the integration of high-yield lecture services given on the Adult ID consult service.Figure 1:Pre/Post Test Results of Internal Medicine Residents These are the reported correct responses, in terms of percentage of residents who responded correctly to the case-based multiple choice question, to a pre and post test of six specific high-yield topics in Infectious Diseases. Methods During the academic year of 2022-2023, for the Adult ID Consult Service, a series of high-yield lectures were given three times a week from a set of 10 potential lecture topics, as chosen by the volunteer faculty or ID fellow lecturer. Following our first Plan-Do-Study-Act cycle evaluation in July 2023, the lectures were standardized to a fixed rotating schedule of the 6 highest-yield topics. There was distributed a survey with a pre-test before the rotation, and a follow-up survey and post-test following the rotation. Post-intervention Interest in Career in ID among Internal Medicine Residents 15.4% (n=2) of internal medicine residents responded yes they would, and 38.5% (n=5) responded somewhat more likely, to consider a career in Infectious Diseases (ID) following the rotation. Results 44 Internal Medicine Residents took the initial Adult ID survey and pretest, and 13 people took the post-rotation evaluation survey and post-test. 100% of participants felt the rotation fully or somewhat helped prepare for the Adult ID section of Internal Medicine boards. 92.3% (n=12) felt the majority of cases on rotation to be interesting. 15.4% (n=2) responded yes and 38.5% (n=5) responded somewhat more likely to consider ID as a career. 100% (n=13) of post-test participants responded with strongly agree (n=7, 53.8%) or agree (n=6, 46.2%) on enjoying the Adult ID rotation. Additionally, scores on the post-test improved from the pre-test for most residents, along with an improvement in Internal Medicine in-training exam scores from 2021 to 2023. Conclusion A didactic series with a set lesson plan that is delivered in a low-stress way can be helpful for engaging learners, enhance the educational experience of the learner, help supplement preparation for internal medicine board exams, and prepare the learner for their future practice. Additionally, as there is a growing need for infectious diseases fellows, this may improve recruitment efforts. Disclosures Aniruddha Hazra, MD, Gilead Sciences: Advisor/Consultant|Gilead Sciences: Grant/Research Support|ViiV Healthcare: Advisor/Consultant

Development of a neonatal cardiac curriculum for neonatal-perinatal medicine fellowship training.

The topic of neonatal cardiovascular care in neonatal-perinatal medicine (NPM) fellowship training has continued to transform due to increased complexity of patients, development of specialized units, continued Accreditation Council for Graduate Medical Education requirements, and clinical practice variation across centers that care for neonates with congenital heart disease (CHD). We developed a neonatal cardiac curriculum comprised of eight interactive sessions with novel active learning concepts specific to our NPM fellows. A self-assessment survey in comfort in managing infants with CHD and perceived competency in neonatal cardiology topics was performed by all neonatology fellows at baseline and after completion of the curriculum. The American Board of Pediatrics Subspecialty In-training Exam (SITE) scores for fellows were compared to that of the national average. The average comfort score (0-100) of the first-year fellows increased from 33 to 76, and that of the second and third-year fellows increased from 72 to 86, and 75 to 86, respectively. The first-year fellows improved their competency score by 44 points (3 standard deviations), the second-year fellows improved their score by about 26 points (one standard deviation), and there was an overall 9-point increase in the competency score of all fellows (one standard deviation). The average local SITE score was lower than the national average before the initiation of this curriculum, became nearly equal to the national average score at the end of the first year the curriculum was implemented, and has progressively become higher since then. Due to the variable clinical exposure and differing practice models of managing CHD a neonatal cardiac curriculum may be beneficial to NPM trainees.

Validity evidence for a novel multiple-choice question bank in the field of child abuse pediatrics

BackgroundThough child abuse pediatrics has been a board-certified subspecialty for 15 years, there are few formalized board preparation resources available. ObjectiveThe purpose of this project was to establish a multiple-choice question bank with sufficient validity evidence for use in preparation for the child abuse pediatrics board examination. Participants and settingThe question bank was distributed via an electronic child abuse pediatrics mailing list. Participants completing the entire question bank included 27 board-certified child abuse pediatricians (CAPs), 19 board-eligible CAPs, and 18 CAP fellows. MethodsWe used Messick's framework to conduct the validity investigation, which includes five components: content evidence, response process, internal structure, relation to other variables, and consequences. Item analyses included difficulty index, discrimination index, and distractor analysis. We used Cronbach's alpha to estimate internal consistency reliability. We conducted linear regressions of scores on the question bank compared to in-training exam scores and career stage. ResultsEighty-four participants completed part of the question bank, and 64 completed the entire question bank. Of the original 117 questions (“items”), 94 met inclusion criteria. The mean score among board-certified CAPs was 80 %, and among participants reporting passing third-year ITE scores was 81 %. Correlation coefficient of scores on this question bank by career stage was r = 0.94, and by year of fellowship was r = 0.99. Cronbach's alpha for internal consistency reliability was 0.83. ConclusionsThis multiple-choice question bank is the first question bank with a robust validity investigation for use by child abuse pediatrics trainees.

Screening for Conscientiousness at Start of Residency: Early Detection of Professionalism Concerns.

Early identification of residents who may require remediation could help prevent problems for both individual residents and their programs. Our aim in this project was to identify whether residents prone to challenges with professionalism could be identified early. For 3 years we tracked onboarding tasks completed by residents between official match and the first day of residency to develop a tool that would help identify residents who may be at risk for problems with timely chart completion and subsequent remediation. We compared residents' "at risk" scores with in-training exam (ITE) scores to differentiate between professionalism (task completion issues) and medical knowledge. Our at-risk tool successfully predicted timely chart completion rates and need for remediation within our residency program (Fischer's exact P<.001), but was not predictive of ITE scores. Using readily accessible information, we can identify residents at risk for poor task completion performance, who may benefit from extra support and development in the area of organizational skill. Early identification may increase opportunities for early intervention.

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ±63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

Foundations of Emergency Medicine: Impact of a Standardized, Open-access, Core Content Curriculum on In-Training Exam Scores.

Learners frequently benefit from modalities such as small-group, case-based teaching and interactive didactic experiences rather than passive learning methods. These contemporary techniques are features of Foundations of Emergency Medicine (FoEM) curricula, and particularly the Foundations I (F1) course, which targets first-year resident (PGY-1) learners. The American Board of Emergency Medicine administers the in-training exam (ITE) that provides an annual assessment of EM-specific medical knowledge. We sought to assess the effect of F1 implementation on ITE scores. We retrospectively analyzed data from interns at four EM residency programs accredited by the Accreditation Council for Graduate Medical Education. We collected data in 2021. Participating sites were geographically diverse and included three- and four-year training formats. We collected data from interns two years before (control group) and two years after (intervention group) implementation of F1 at each site. Year of F1 implementation ranged from 2015-2018 at participating sites. We abstracted data using a standard form including program, ITE raw score, year of ITE administration, US Medical Licensing Exam Step 1 score, Step 2 Clinical Knowledge (CK) score, and gender. We performed univariable and multivariable linear regression to explore differences between intervention and control groups. We collected data for 180 PGY-1s. Step 1 and Step 2 CK scores were significant predictors of ITE in univariable analyses (both with P < 0.001). After accounting for Step 1 and Step 2 CK scores, we did not find F1 implementation to be a significant predictor of ITE score, P = 0.83. Implementation of F1 curricula did not show significant changes in performance on the ITE after controlling for important variables.

Assessing the impact of adopting a competency-based medical education framework and ACGME-I accreditation on educational outcomes in a family medicine residency program in Abu Dhabi Emirate, United Arab Emirates.

Competency-Based Medical Education (CBME) is now mandated by many graduate and undergraduate accreditation standards. Evaluating CBME is essential for quantifying its impact, finding supporting evidence for the efforts invested in accreditation processes, and determining future steps. The Ambulatory Healthcare Services (AHS) family medicine residency program has been accredited by the Accreditation Council of Graduate Medical Education-International (ACGME-I) since 2013. This study aims to report the Abu Dhabi program's experience in implementing CBME and accreditation. Compare the two residents' cohorts' performance pre-and post-ACGME-I accreditation.Study the bi-annually reported milestones as a graduating residents' performance prognostic tool. All residents in the program from 2008 to 2019 were included. They are called Cohort one-the intake from 2008 to 2012, before the ACGME accreditation, and Cohort two-the intake from 2013 to 2019, after the ACGME accreditation, with the milestones used. The mandatory annual in-training exam was used as an indication of the change in competency between the two cohorts. Among Cohort two ACGME-I, the biannual milestones data were studied to find the correlation between residents' early and graduating milestones. A total of 112 residents were included: 36 in Cohort one and 76 in Cohort two. In Cohort one, before the ACGME accreditation, no significant associations were identified between residents' graduation in-training exam and their early performance indicators, while in Cohort two, there were significant correlations between almost all performance metrics. Early milestones are correlated with the graduation in-training exam score. Linear regression confirmed this relationship after controlling the residents' undergraduate Grade Point Average (GPA). Competency development continues to improve even after residents complete training at Post Graduate Year, PGY4, as residents' achievement in PGY5 continues to improve. Improved achievement of residents after the introduction of the ACGME-I accreditation is evident. Additionally, the correlation between the graduation in-training exam and graduation milestones, with earlier milestones, suggests a possible use of early milestones in predicting outcomes.

Radiology Resident Diagnostic In-Training Exam Scores: Impact of Subspecialty Imaging Volume and Rotation Scheduling

PurposeTo determine the relationship between resident imaging volumes and number of subspecialty rotations with Diagnostic Radiology In-Training (DXIT) subspecialty scores. MethodsDXIT-scaled subspecialty scores from a single large diagnostic radiology training program from 2014 to 2020 were obtained. The cumulative number of imaging studies dictated by each resident and specific rotations were mapped to each subspecialty for each year of training. DXIT subspecialty scores were compared against the total subspecialty imaging volume and the total number of rotations in a subspecialty for each resident year. A total of 52 radiology residents were trained during the study period and included in the dataset. ResultsThere was a positive linear relationship between the number of neuro studies and scaled neuro DXIT scores for R1s (Pearson coefficient: 0.29; p-value: 0.034) and between the number of breast studies and the number of neuro studies with DXIT scores for R2s (Pearson coefficients: 0.50 and 0.45, respectively; p-values: 0.001 and 0.003, respectively). Furthermore, a positive significant linear relationship between the total number of rotations in cardiac, breast, neuro, and thoracic subspecialties and their scaled DXIT scores for R2 residents (Pearson coefficients: 0.34, 0.49, 0.33, and 0.32, respectively; p-value: 0.025, 0.001, 0.03, and 0.036, respectively) and between the total number of nuclear medicine rotations with DXIT scores for R3s (Pearson coefficient: 0.41; p-value: 0.016). ConclusionResident subspecialty imaging volumes and rotations have a variable impact on DXIT scores. Understanding the impact of study volume and the number of subspecialty rotations on resident medical knowledge will help residents and program directors determine how much emphasis to place on these factors during residency.

Association Between In-Training Exam Scores and Clinical Exposure During the COVID-19 Pandemic

BackgroundWe sought to evaluate changes in In-Training Examination (ITE) scores and associations with clinical work during the COVID-19 pandemic. We hypothesized that residents saw a decrease in clinical encounters during the pandemic and that this would be associated with smaller gains in ITE scores. MethodsWe compared ITE score changes with data on patient notes for three classes of pediatric residents at four residency programs: one not exposed to the pandemic during their intern year who entered residency in 2018, one partially exposed to COVID-19 in March of their intern year (2019–2020), and one that was fully exposed to the pandemic, starting residency in June of 2020. ResultsITE scores on average improved from the PGY1 to PGY2 year in the “no covid” and “partial COVID” cohorts. The “full COVID” cohort had little to no improvement, on average. The total number of patient encounters was not associated with a change in ITE scores from PGY1 to PGY2. There was a small but statistically significant association between change in ITE score and number of inpatient H+P notes. ConclusionsA drop in ITE scores occurred in pediatric residents who entered residency during the COVID-19 pandemic. This change was largely unrelated to clinical encounter number changes.

The impact of fellowship-trained medical toxicology faculty on emergency medicine resident in-training examination scores.

The American Board of Emergency Medicine (ABEM) In Training Exam (ITE) gauges residents' medical knowledge and has been shown to correlate with subsequent performance on the ABEM board qualifying examination. It is common for emergency medicine (EM) residencies to employ subspecialty-trained faculty members with the expectation of improved resident education and subspecialty knowledge. We hypothesized that the presence of subspecialty faculty in toxicology would increase residents' scores on the toxicology portion of the ITE. We assessed ABEM ITE scores at our institution from 2013-2022 and compared these to national data. The exposure of interest was the absence or presence of fellowship-trained toxicology faculty. The primary outcome was performance on the toxicology portion of the ITE, and secondary outcome was overall performance on the exam. Residents who had ≥1 toxicology faculty were 37% (95% CI: 1.01-1.87) more likely to surpass the national average for toxicology scores, and those who had ≥2 toxicology faculty were 77% (95% CI: 1.28-2.44) more likely to surpass the national average for toxicology scores on the ABEM ITE. With the presence of ≥2 toxicology faculty, there was also an increase in toxicology score by years in training, with residents being 63% (95% CI: 1.01-2.64), 68% (95% CI: 1.08-2.61), and 92% (95% CI: 1.01-3.63) more likely to surpass the national average for toxicology score in first, second, and third years of residency, respectively. There was no significant relationship between the presence of toxicology faculty and the overall ABEM ITE scores. The presence of fellowship-trained toxicology faculty positively impacted residents' performance on the toxicology portion of the ABEM ITE but did not significantly impact the overall score. With the presence of ≥2 toxicology faculty we noted an improvement in toxicology scores throughout the 3 years of training, indicating that an individual rotation or educational block is probably less important than spaced repetition through a longitudinal curriculum.

Academic Faculty Demonstrate Weak Agreement in Evaluating Orthopaedic Surgery Residents.

As objective metrics fade, subjective elements of orthopaedic surgery applicants carry increasing importance during recruitment. Academic orthopaedic surgeons believe they can select for high-performing orthopaedic residents. However, can they agree? The purpose of this study was (1) to analyze an academic orthopaedic surgery department to determine whether they agree on which residents performed best during residency and; (2) to correlate preresidency and intraresidency factors with postresidency evaluations of resident performance. With Institutional Review Board [IRB] approval, an orthopaedic surgery department completed surveys to evaluate residency performance for 10 years of graduated residents (2012-2022). Faculty determined (1) Post-Residency Class Rank (PRCR)-ranked from the highest (1) to lowest performing resident (5) relative to their class based on faculty perspective of performance-and (2) Rank List Score (RLS)-ranked based off the 5-point AOA SLOR used during recruitment interviews. RLS assessed how likely the department would have graduates in the program again based on residency performance. Free marginal Cohen's kappa statistics assessed faculty inter-rater agreement. Preresidency metrics (United States Medical Licensing Exam [USMLE] 1 and 2 scores, research publications, etc) were correlated with Orthopaedic In-Training Exam (OITE) scores, research productivity, American Board of Orthopaedic Surgery (ABOS) scores, and faculty-derived rankings. Linear regressions with forward variable entry (p < 0.05) were used to determine factors associated with excellent resident performance. Eighteen of 25 faculty members (72%) provided survey responses evaluating 46 residents. Faculty agreed 37% and 38% of the time for PRCR (kappa 0.26) and RLS (kappa 0.23), respectively. Step 2 score was the only preresidency factor significantly associated with PRCR (p = 0.03, r2 = 0.15) and RLS (p = 0.02, r2 = 0.3). The only intraresidency factor significantly correlated with PRCR (p = 0.002, r2 = 0.50) and RLS (p = 0.01, r2 = 0.39) was PGY-4 OITE score. An academic orthopaedic surgery department is able to come to a consensus on evaluations of residency performance relative to peers in the same year of training (PRCR) and an objective standard (RLS). Step 2 and Post-Graduate Year (PGY)-4 OITE scores were the only preresidency and intraresidency factors with significant association to higher postresidency, faculty-derived performance scores. III.

A new dawn in internal medicine education curriculum—a project by the residents, with the residents, for the residents

The academic half-day (AHD) curriculum is an alternative to the noon conference (NC) approach. To date, there is little literature evaluating the transition from NC to AHD in internal medicine residency programs. We investigated the effectiveness of AHD by comparing in-training exam (ITE) and American Board of Internal Medicine (ABIM) certifying exam scores of residents before and after implementation of AHD. In 2019, we transitioned to the AHD curriculum. Averages of three consecutive years of NC (2016–2018) and AHD (2019–2021) were used for statistical testing to determine ITE and ABIM score changes. The class of 2018 experienced both approaches. Cohen’s d effect sizes were calculated to assess the magnitude of change in ITE and ABIM scores between NC (2016–2018) and AHD (2019–2021) cohorts. Residents’ performance significantly improved (P < 0.05) on ABIM scores (513.80 ± 48.34) on average from 2019 to 2021 compared to ABIM scores (452.42 ± 49.72) on average from 2016 to 2018 with a large effect size of 1.27. Similarly, a significant (P = 0.005) improvement in ITE scores was observed from 2019 to 2021 with implementation of AHD compared to preceding NC scores (2016–2018) with a moderate effect size of 0.52. Participating residents in AHD sessions had higher ITE and ABIM scores compared to those in hourly NC didactic sessions with significantly improved resident attendance and overall satisfaction.

Combining desirable difficulty learning strategies to address the amount-to-learn vs. time-to-learn imbalance in residency training

Purpose Residents have limited time and much to learn. Mounting evidence shows that Desirable Difficulty (DD) learning strategies can ease that imbalance, but few studies have specifically studied combinations of these strategies. Methods We tested two different combinations of DD strategies: a double combination of distributed practice and retrieval practice and a triple combination additionally including interleaved practice. We compared residents’ annual In-Training Exam (ITE) scores and graduates’ board certification performance between both DD curricula and a historical baseline. Results Average ITE scores rose from 149.06 in the historical baseline to 160.04 under the combined DD curricula (p < 0.001). Average ITE scores fell from 162.50 under the double combination to 155.11 under the triple combination (p = 0.03). There were no significant changes in graduates’ board performance between any of the curricula. Conclusions These results add to the evidence that DD strategies can enhance residents’ learning. The drop in ITE scores under the triple DD combination may suggest that it pushed learners past beneficial desirable difficulty into detrimental overwhelming difficulty. Further research should apply this framework in larger and more diverse settings to clarify how these DD strategies can be optimally used to enhance residents’ learning.

Hybrid Cross-Discipline, Interactive Curriculum to Nurture the Training Experience for Hematology-Oncology Trainees

Introduction:Hematologic malignancy is a complex science that requires the integration of numerous advanced pathology techniques, including flow cytometry, genomic analysis, and molecular diagnostics, for diagnosis and risk stratification to guide management plans and optimize patient care.Understanding the methodology and proper utilization of new laboratory test methodologies relevant to patient care are required competencies for graduating hematology/oncology trainees, put forth by the Accreditation Council for Graduate Medical Education (ACGME). Therefore, these competencies are heavily tested on specialty board certification exams.We aimed at familiarizing our trainees with distinct laboratory test methodologies, and classifications of hematologic neoplasms from a hematopathologist's perspective. Simplifying such a complex science is a potential way to help trainees conceptualize the proper uses of several diagnostic tests. A shared understanding of clinical and pathologic approaches to hematologic malignancies will lead to improved advanced knowledge in the field, test-taking capabilities and reflect on patient care.We hypothesized that delivering cross-discipline training will improve the level of knowledge for our trainees, and reflect on the hematology in-training-exam scores and subsequently improve the American board of hematology participation and pass rates.Method:Curriculum Description:We developed a cross-discipline curriculum, three-month experience that included rigorous training in advanced hematopathology focused on hematologic malignancies. The curriculum included two hours per week, hybrid style interactive sessions –online/in-person classes–. A self-directed component included pre-class readings and pre- and post-weekly quizzes. The in-person sessions included interactive, case-based seminars guided by cross-discipline faculty members. Instructors for the curriculum included a malignant hematologist and a hematopathologist.The enrollment to the described curriculum made available for all hematology-oncology fellows, internal medicine residents, and advanced practice providers (APPs). This project was accomplished as part of the ASH Medical Educator Institute.Evaluation plan:Trainees were given pre- and post-intervention case-based exams as well as pre- and post-quizzes every week, to assess their knowledge and level of progression. For hematology fellows, trainee's performance on in-training exams (ITEs) was also compared to a historical control group of trainees from the same program.Results:The training included twenty-six participants, eight hematology-oncology fellows, five medical residents, six APPs, and seven faculty. Faculty participants were excluded from the performance evaluation to avoid bias introduction. Pre-class surveys and quizzes were collected to measure baseline knowledge of different hematology subjects and to guide the focus of the training based on common weaknesses amongst a majority of participants.Eighty-six percent of participants elected to attend the training to improve their clinical practice experience and exam scores, the rest intended to network academically. Before each session, participants finished the pre-session quizzes that was compared to the post-session quizzes. We report the persistent positive impact of the sessions on the trainees' weekly performances. We measured the overall impact of the training, comparing the pre-intervention exam with midterm and final exam scores, which also significantly increased compared to pre-intervention scores. (Figures 1 & 2)For hematology Oncology trainees, we also measured the impact of this training on their in-training-exam scores, compared to historical controls from the same fellowship program. For the hematology in-training exam scores, we noticed a significant improvement after the intervention for the year 2021 compared to the year 2020. An increase in the program means score, which reflected on the program overall performance with an increase in our percentile rank by approximately 14 points.Conclusion:Developing a cross-discipline interactive training curriculum is an innovative method to enhance the hematology-oncology trainees' experience. It promoted their level of knowledge and improved exam skills, which is reflected in the annual in-training exam scores. [Display omitted] DisclosuresEllis: Rafael Pharmaceuticals: Consultancy.

Medical School Clinical Knowledge Exam Scores, Not Demographic or Other Factors, Associated With Residency In-Training Exam Performance.

The anesthesiology in-training exam (ITE) is a 200-item multiple-choice assessment completed annually by physician residents. Because all matriculated U.S. Department of Defense (DoD) anesthesiology residents are "hired" by the DoD after residency graduation, it is important to ensure that ITE performance, as a proxy for core competencies achievement, is maximized. Graduated resident program files from 2013 to 2020 were queried for age, sex, matriculant status (medical student vs. other), medical school (Uniformed Services University vs. other), military service (Army vs. Air Force), preresidency military service (yes vs. no), U.S. Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) score, and the American Board of Anesthesiologists ITE Score from the third clinical anesthesia year (CA-3 year). For every 1-point increase in USMLE Step 2 CK true z-score, the CA-3 ITE z-score increased by 0.59 points. Age was not associated with CA-3 ITE z-score in any dataset regression. Categorical covariates of sex, application status, medical school, service, and preresidency military service were not significantly associated with CA-3 ITE z-score (all P >.05), as shown by estimated adjusted marginal means. The estimated adjusted grand mean of CA-3 ITE z-scores was 0.48 (standard error ± 0.14). Resident physicians enter residency with varying degrees of past academic success, and it is important to develop early strategies to support them in acquiring the requisite knowledge base.

Do the selection criteria of internal medicine residency program predict resident performance?

Background: Well-performing physician reflects the success of the residency program in selecting the best candidates for training. This study aimed to evaluate the selection criteria, mainly the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) results and applicants’ status as international or locally trained applicants, used by the medical education department and the internal medicine residency program in Hamad Medical Corporation in Qatar to predict the residents’ performance during their training.Methods: A retrospective chart review was performed for three batches of graduates who started residency training in 2011, 2012, and 2013. Each group completed 4 years of training. The USMLE Step 2 CK status of the applicant, in-training exam (ITE) scores, formative evaluation scores, Arab Board written and clinical exams pass rate, and other indicators were analyzed. Statistical analysis included chi squares and independent t-test to identify associations. Multivariable analyses were conducted using logistic and linear regressions to test for adjusted associations.Results: The study included 118 (81 international/37 locally trained applicants) internal medicine residents. The ITE score correlated positively with the USMLE Step 2 CK score (r = 0.621, r = 0.587, r = 0.576, r = 0.571, p < 0.001) over the 4 years of training and among the international compared with locally trained applicants (p < 0.001). The rate of passing part 1 and 2 written exam of the Arab Board was higher in international than in local applicants, whereas clinical Arab Board exam and formative evaluation were not associated with any criteria.Conclusions: Higher USMLE Step 2 CK score correlated with better performance on ITE but not with other performance indicators, whereas international applicants did better in both ITE and Arab Board written exam than local applicants. These variables may provide reasonable predictors of well-performing physicians.

The critical role of direct observation in entrustment decisions.

BackgroundEntrustment decisions may be retrospective (based on past experiences with a trainee) or real-time (based on direct observation). We investigated judgments of entrustment based on assessor prior knowledge of candidates and based on systematic direct observation, conducted in an objective structured clinical exam (OSCE).Methods:Sixteen faculty examiners provided 287 retrospective and real-time entrustment ratings of 16 cardiology trainees during OSCE stations in 2019 and 2020. Reliability and validity of these ratings were assessed by comparing correlations across stations as a measure of reliability, differences across postgraduate years as an index of construct validity, correlation to standardized in-training exam (ITE) as a measure of criterion validity, and reclassification of entrustment as a measure of consequential validity.Results:Both retrospective and real-time assessments were highly reliable (all intra-class correlations >0.86). Both increased with a year of postgraduate training. Real-time entrustment ratings were significantly correlated with standardized ITE scores; retrospective ratings were not. Real-time ratings explained 37% (2019) and 46% (2020) of variance in examination scores vs. 21% (2019) and 7% (2020) for retrospective ratings. Direct observation resulted in a different level of entrustment compared with retrospective ratings in 44% of cases (p = <0.001).Conclusions:Ratings based on direct observation made unique contributions to entrustment decisions.

A Novel Ambulatory Curriculum for Pulmonary and Critical Care Fellowship Training.

Background: Dedicated ambulatory training during pulmonary and critical care medicine (PCCM) fellowships is often limited. A novel 2-year longitudinal outpatient pulmonary fellowship curriculum was previously developed, piloted, and studied. The exportability and potential impact of this ambulatory curriculum on PCCM fellowship training nationally is not known.Objective: We aim to understand the current state of ambulatory training in PCCM fellowships and the impact of a standardized outpatient curriculum on fellows’ ambulatory knowledge and competency.Methods: Nineteen programs participated in the study from 2017 to 2019. Six programs received the first year of content, seven programs received the entire 2-year curriculum, and seven programs served as a control. Fellows, faculty, and program directors (PDs) completed a series of surveys assessing satisfaction with ambulatory education and the curriculum. Fellows completed a series of medical knowledge inventories, and programs submitted in-training exam scores.Results: A total of 221 fellows (39%) and 17 PDs (89%) completed the precurriculum surveys, and 38 (12%) fellows and 10 (53%) PDs completed postcurriculum surveys. Before curriculum implementation, only 34.4% of fellows rated the quality of their ambulatory education as good or outstanding compared with 57.9% at the end of the study. Eighty-five percent of faculty and 89% of PDs rated the curriculum as good or excellent. Faculty believed that the teaching scripts were easy to use (78.4%), were factually accurate (86.3%), and provided high-yield information (82.1%). The majority of PDs indicated that the curriculum positively impacted patient care (78%) and fulfilled an unmet educational need (100%), and most planned to continue the curriculum after the study (78%). Feedback surrounded the need for updated content based on recently published guidelines and studies.Conclusion: The curriculum is a standardized and feasible way to address a previously unmet need in PCCM fellowship education. PDs rated the curriculum highly and most plan to continue it in the future. Our limited data set suggests that the curriculum was well received by fellows and faculty and positively impacted perceptions of ambulatory education and preparedness for independent practice. Future study with a larger sample of fellows is needed to better understand the generalizability of these findings.

What do internal medicine residents know about rheumatology? A needs assessment for curriculum design.

We sought to determine if analyzing internal medicine (IM) resident performance on the in-training exam (ITE) might reveal curricular needs in rheumatology education beyond those intuitive to learners and educators from their perceptions and experience. We analyzed ITE scores of post-graduate year (PGY) 2 IM residents at our institution from 2010 to 2017. We sorted rheumatology-related educational objectives on the ITEs into 10 disease categories and calculated average scores. We then surveyed our IM residents' self-reported confidence and rheumatology educators' perceptions of IM resident proficiency in the 10 categories on a 10-point Likert scale. Finally, we tallied diagnoses in each category encountered by IM residents in rheumatology clinic. Pearson's correlation coefficients were calculated for the relationship between these data. The ITEs exposed 402 residents to 63 rheumatology-related educational objectives in the 10 categories; 24 objectives were categorized as "other." Survey respondents included 38 residents and 22 educators. There was no correlation between ITE scores in the 10 categories and resident confidence (r = - 0.226, p = 0.530), educator perception (r = - 0.274, p = 0.445), or diagnoses encountered (r = - 0.310, p = 0.383). There was a strong positive correlation between resident confidence and educator perception (r = 0.934, p < 0.001). ITE performance was low in crystalline arthritis and osteoarthritis despite high resident confidence, educator perception, and number of diagnoses seen. Our analysis of IM resident ITE performance identified curricular gaps not apparent in surveys of learner and educator perceptions or an assessment of learner experience. Key Points • In this study, we demonstrate how a systematic analysis of internal medicine resident performance on the in-training exam can identify important curricular gaps in rheumatology education that are not apparent in assessments of learners and educator perceptions or learner experience in rheumatology clinic. • In-training exam performance was low in crystalline arthritis and osteoarthritis despite high resident confidence, educator perception of proficiency, and number of diagnoses seen in these categories. • When rheumatology curricula for internal medicine residents are developed solely on the basis of learner and educator perceptions, common disorders traditionally considered "easy" may be overlooked.

Association Between Family Medicine Residents' Mindsets and In-Training Exam Scores.

In medical practice, a mastery mindset is important for engaging in lifelong learning. The objective of this study was to examine the association between family medicine residents' scores on mindset measures and their performance on in-training examinations (ITE). This was a secondary data analysis of a cohort of family medicine residents. Following ethics approval, residents' ITE scores from each of the 2 years of residency were linked with residents' responses to a mindsets survey that they had taken at the midpoint of residency training. Multiple regression analysis was used to investigate the relationship between residents' mindset scores and their ITE scores. Of 85 residents, 46 (54%) had complete data for the three data collection points. Residents' ITE scores in year 1 were most predictive of their ITE scores in year 2 (β=0.72; P<.001). Mastery mindset scores were negatively associated with residents' performance on the ITE in year 2 (β=-0.29; P=.004). While the observed negative relationship between residents' mastery mindset scores and their ITE performance may be disconcerting, it is not surprising. In clinical settings, residents are individually coached by preceptors and provided with specific, actionable feedback to support their learning. With respect to formative assessments, residents likely require explicit training on how to use their assessment results (ITE scores) to support their self-directed learning. This finding has practical implications for residency programs in using ITEs as formative assessments.