Annual In-Training Examination Research Articles

Use of In-Training Examination Scores as a Fellowship Candidate Evaluation Metric: Time for a Change.

In the field of obstetrics and gynecology (OB/GYN), the Council on Resident Education in Obstetrics and Gynecology (CREOG) administers an annual in-training examination to all OB/GYN residents as a formative educational tool for assessing medical knowledge and promoting self-improvement. Although the CREOG examination is not designed or intended for knowledge certification, many OB/GYN subspecialty fellowship programs request and use CREOG examination scores as a metric to evaluate fellowship candidates. Among the 57 gynecology-based urogynecology fellowship programs, 30 programs (53%) request CREOG examination scores to be submitted by candidates, as of March 2023. Although the use of CREOG examination scores as an evaluation metric may constitute a minor component within the fellowship match process, this practice fundamentally contradicts the intended purpose of the examination as an educational self-assessment. In addition, it introduces the potential for bias in fellowship recruitment, lacks psychometric validity in predicting specialty board examination failure, and shifts the CREOG examination from its original intention as low-stakes self-assessment into a high-stakes examination akin to a certification examination. For these reasons, we call upon the urogynecology community to prioritize the educational mission of the CREOG examination and reconsider the practice of requesting or using CREOG examination scores in the fellowship match progress.

Does QBank participation impact in-training examination performance?

Performance on the annual in-training examination (ITE) for emergency medicine (EM) residents has been shown to correlate with performance on the American Board of Emergency Medicine (ABEM) qualifying examination. As such, significant planning is often committed to ITE preparation, from an individual resident and a residency program perspective. Online specialty-specific question banks (QBanks) represent a popular medium for ITE preparation; however, the impact of QBanks on ITE performance is unclear. We sought to assess the impact of QBank participation on EM resident ITE performance. ITE and QBank performance results were collated over 2 academic years, 2019 and 2020, from a 3-year EM residency program. The QBank was provided as a self-study option in 2019 and incorporated as a mandatory component of the curriculum in 2020. ITE raw scores and percentile rank for training level scores were compared with performance on the QBank, including QBank average performance score as well as number of QBank questions completed. The Pearson correlation coefficient was used to measure association between ITE performance and QBank correlates. Additional descriptive demographics, to include gender, PGY level, and USMLE step 1 and 2scores were also considered. Sixty-two sets (30 residents in 2019, 32 residents in 2020) of ITE performance data and QBank correlates were included. Overall, raw ITE scores and number of QBank questions completed were found to have a significant, positive correlation, (r(60) = 0.34, p < 0.05); correlation was stronger for 2019 (r[28] = 0.39, p < 0.05) compared to 2020 (r[30] =0.25, p = 0.16). Overall, ITE percentile rank for training level scores were also found to have a significant, positive correlation with number of QBank questions completed (r(60) = 0.35, p < 0.05); correlation was again stronger for 2019 (r(28) = 0.42, p < 0.05) compared to 2020 (r(30) = 0.29, p = 0.12). Finally, ITE percentile rank for training level correlated positively with QBank average performance (as a percentage), albeit weakly, and was not found to be significant overall (r[60] = 0.20, p = 0.16); in this instance, 2019 did not show a correlation (r[28] =0.12, p = 0.54); however, 2020 did (r[30] =0.55, p<0.05). Academic year 2020 raw ITE scores also demonstrated a significant correlation with QBank average performance (r[30] = 0.66, p < 0.0001) while 2019 did not (r[28] = 0.08, p = 0.68). Participation and engagement in a QBank are associated with improved EM resident performance on the ABEM ITE. Utilization of a QBank may be an effective mode of ITE preparation for EM residents.

Surveying Obstetrics and Gynecology Residents About Their Residency Applications, Interviews, and Ranking

Residency applications have increased in the last decade, creating growing challenges for applicants and programs. We evaluated factors associated with application and match into obstetrics and gynecology residency. During the annual in-training examination administered to all obstetrics and gynecology residents in the United States, residents were surveyed on the residency application process. Ninety-five percent (5094 of 5347) residents responded to the survey. Thirty-six percent reported applying to 30 or fewer programs, 26.7% applied to more than 31 programs, and 37.1% opted not to answer this question. Forty-nine percent of residents received honors in their obstetrics and gynecology clerkship and 37.1% did not. The majority of residents (88.6%) reported scoring between 200 and 250 on USMLE Step 1. Eighty-six percent matched into one of their top 5 programs. The only factor associated with matching in residents' top 5 programs was receiving honors in their clerkship (OR 1.29; 95% CI 1.08-1.54; P < .005). The only factor associated with matching below the top 5 programs was a couples match (OR 0.56; 95% CI 0.43-0.72; P < .001). In choosing where to apply, residents identified program location and reputation as the most important factors, while for ranking, location and residency culture were the most important. Most obstetrics and gynecology residents reported matching into their top 5 choices. Receiving an honors grade in the clerkship was the only factor associated with matching in applicants' top 5 programs. Location was the most important factor for applying to and ranking of programs.

Evaluation of a national neurosurgical formative examination: the UK experience

Introduction: Formative assessment is a key component in medical education and that it is a helpful process for all involved. Until recently there was no national formative examination for neurosurgical trainees. The Neurosurgery Annual in Training Examination (NAiTE) is an annual online, formative assessment that was introduced in 2014. In this paper, we seek to discuss how well NAiTE relates to established educational practice and principles and its fitness for purpose by discussing its rationale, structure and utility.Methods: A national online examination was introduced in 2014. The NAiTE consists of 100 single best answer multiple choice questions. The examination and questions with were reviewed and the global results presented. The existing literature and educational theory are used to guide subjective assessment of the process.Results: In 2016, 191 candidates participated in the NAiTE, of whom 154 were trainees working in UK neurosurgical units. The mean score for early stage UK trainees (years 1-3) was 52.4%, intermediate (years 4-5) 58.5% and senior (years 6-8) 65.4%. The NAiTE was found to be a reliable (Cronbach-Alpha of 0.89) and valid assessment of trainees with scores approximating those attained in the Intercollegiate Specialty Examination itself.Discussion: Potential areas for improvement are highlighted, including reference to some that have already been implemented.Conclusion: Overall, the examination is a cheap, viable and reliable means of testing trainees and encouraging their onward development and learning as they work towards the Intercollegiate Specialty Examination

Effect of the BASIC Examination on Knowledge Acquisition during Anesthesiology Residency.

The American Board of Anesthesiology recently introduced the BASIC Examination, a component of its new staged examinations for primary certification, typically offered to residents at the end of their first year of clinical anesthesiology training. This analysis tested the hypothesis that the introduction of the BASIC Examination was associated with an acceleration of knowledge acquisition during the residency training period, as measured by increments in annual In-Training Examination scores. In-Training Examination performance was compared longitudinally among four resident cohorts (n = 6,488) before and after the introduction of the staged system using mixed-effects models that accounted for possible covariates. Compared with previous cohorts in the traditional examination system, the first resident cohort in the staged system had a greater improvement in In-Training Examination scores between the first and second years of clinical anesthesiology training (by an estimated 2.0 points in scaled score on a scale of 1 to 50 [95% CI, 1.7 to 2.3]). By their second year, they had achieved a score similar to that of third-year clinical anesthesiology residents in previous cohorts. The second cohort to enter the staged system had a greater improvement of the scores between the clinical base year and the first clinical anesthesiology year, compared with the previous cohorts. These results support the hypothesis that the introduction of the BASIC Examination is associated with accelerated knowledge acquisition in residency training and provides evidence for the value of the new staged system in promoting desired educational outcomes of anesthesiology training.

What Can SCORE Web Portal Usage Analytics Tell Us About How Surgical Residents Learn?

The Surgical Council on Resident Education (SCORE) was established in 2004 with 2 goals: to develop a standardized, competency-based curriculum for general surgery residency training; and to develop a web portal to deliver this content. By 2012, 96% of general surgery residency programs subscribed to the SCORE web portal. Surgical educators have previously described the myriad ways they have incorporated SCORE into their curricula. The aim of this study was to analyze user data to describe how and when residents use SCORE. Using analytic software, we measured SCORE usage from July, 2013 to June, 2016. Data such as IP addresses, geo-tagging coordinates, and operating system platforms were collected. The primary outcome was the median duration of SCORE use. Secondary outcomes were the time of day and the operating system used when accessing SCORE. Descriptive statistics were performed, and a p < 0.05 was deemed statistically significant. There were 42,743 total SCORE subscribers during the study period (75% resident and 25% faculty) with a mean of 14,248 subscribers per year. The overall median duration of SCORE use was 11.9minute/session (interquartile range [IQR]: 6.8). Additionally, there was a significant increase in session length over the 3 academic years; 10.1 (IQR: 6.4), 11.9 (IQR: 7.2), and 13.2minute/session (IQR: 5.4) in 2013 to 2014, 2014 to 2015, and 2015 to 2016, respectively (p < 0.001). SCORE usage was highest in November to February at 21.0minute/session (14.2) compared to July to October and March to June (12.3 [IQR: 3.2] and 9.6minute/session [IQR: 2.2]), respectively (p < 0.001). This seasonal trend continued for all 3 years. We observed an increased number of sessions per day over the 3 years: median of 1500 sessions/d (IQR: 1115) vs 1706 (IQR: 1334) vs 1728 (IQR: 1352), p < 0.001. (Fig.). Most SCORE sessions occurred at night: 38,011 (IQR: 4532) vs 17,529 (IQR: 19,850) during the day (p < 0.001). Windows was the most frequently used operating system at 48.9% (p < 0.001 vs others). SCORE usage has increased significantly over the last 3 years, when measured by number of sessions per day and length of time per session. There are predictable daily, diurnal, and seasonal variations in SCORE usage. The annual in-training examination is a prominent factor stimulating SCORE usage.

Development and implementation of an asynchronous emergency medicine residency curriculum using a web-based platform.

The Residency Review Committee in Emergency Medicine requires residency programs to deliver at least 5hours of weekly didactics. Achieving at least a 70% average attendance rate per resident is required for residency program accreditation, and is used as a benchmark for residency graduation in our program. We developed a web-based, asynchronous curriculum to replace 1hour of synchronous didactics, and hypothesized that the curriculum would be feasible to implement, well received by learners, and improve conference participation. This paper describes the feasibility and learner acceptability of a longitudinal asynchronous curriculum, and describes its impact on postgraduate year-1(PGY-1) resident conference participation and annual in-training examination scores. Using formal curriculum design methods, we developed modules and paired assessment exercises to replace 1hour of weekly didactics. We measured feasibility (development and implementation time and costs) and learner acceptability (measured on an anonymous survey). We compared pre- and post-intervention conference participation and in-service training examination scores using a two sample t test. The asynchronous curriculum proved feasible to develop and implement. PGY-1 resident conference participation improved compared to the pre-intervention year (85.6 vs. 62%; 95% CI 0.295-0.177; p<0.001). We are unable to detect a difference between in-training examination results in either the PGY-1 group or across all residents by the introduction of this intervention. 18/31 (58%) residents completed the post-intervention survey. 83% reported satisfaction with curriculum changes. Strengths of the curriculum included clarity and timeliness of assignments. Weaknesses included technical difficulties with the online platform. Our curriculum is feasible to develop and implement. Despite technical difficulties, residents report high satisfaction with this new curriculum. Among PGY-1 residents there is improved conference participation compared to the prior year.

Enhancement of anesthesiology in-training exam performance with institution of an academic improvement policy

Anesthesiology resident physicians across the United States complete an annual in-training examination (ITE). The ITE evaluates resident knowledge and provides personalized feedback to guide future study in low scoring sections(1). Performance on the ITE correlates with outcomes on the American Board of Anesthesiology (ABA) written board examination(2). Over the last several years, declining ITE scores were observed at the University of North Carolina (UNC). In response to this decline, our department reprioritized the ITE by instituting an academic improvement policy (AIP). The AIP employed both reward for satisfactory achievement and consequence for under-performance to elevate the ITE as a "high stakes" examination. Our hypothesis was that implementation of this AIP would improve ITE scores. ITE scores were compiled from 150 residents in the Department of Anesthesiology at UNC for graduating classes from 2004-2015. Data is presented as the number of residents scoring below the 20th percentile when compared to the national distribution before and after the AIP. In addition, average USMLE Step 1 three-digit scores for each graduating class were compared to average ITE percentile scores of the corresponding graduating class (USMLE does not provide percentile scores). Between 2009 and 2013, the number of residents who scored below the 20th percentile on the ITE increased steadily to a peak of 10 in 2011. After implementation of the AIP in July 2011, there was an 80% decrease in those scoring below the 20th percentile, from 10 to 2 residents (p<0.05). Anesthesiology resident ITE scores improved after implementation of an academic improvement policy.

102: Does Correlation of Faculty Assessment of Emergency Medicine Residents' Medical Knowledge Competency With Performance on the In-Training Examination Improve With Advancement Through the Program?

Faculty assessment of emergency medicine (EM) residents on the medical knowledge (MK) core competency may or may not be predictive of performance on the annual in-training examination. We sought to determine if a greater degree of faculty exposure and experience with EM residents, as determined by PGY level, improved the correlation of the faculty's assessment of MK on quarterly summative evaluations with the score received on the EM in-training examination taken during the same time period.

Morningness-eveningness preferences of emergency medicine residents are skewed toward eveningness.

To determine the morningness-eveningness (ME) distribution of emergency medicine (EM) residents. A voluntary, modified ME questionnaire was administered to all EM residents in the United States at the time of the 1995 American Board of Emergency Medicine's annual In-Training Examination. Seventy-eight percent (2,047/2,614) of the surveys were returned. ME scores ranged from 24 to 76, with a median score of 49 (interquartile range 44, 56). The scores were distributed differently from those of the normal population (p < 0.001), being skewed toward eveningness. There was a correlation (r = 0.13, p < 0.0001) between resident age and ME score, with older residents being more morning-oriented. Males were more morning-oriented than females (p = 0.005), and respondents with children living at home also were significantly more morning-oriented (p < 0.001). Stepwise logistic regression showed that the influence of age, gender, and children was cumulative (r = 0.19) but accounted for only 4% of the observed variability. EM residents are distributed differently from the normal population in terms of their ME preferences, tending slightly toward eveningness. The importance of this distribution in EM residents in unknown. A longitudinal follow-up of this cohort may help to determine the association of ME preference with overall practice satisfaction, tolerance of shift work, and career longevity.

First annual in-training examination in otolaryngology.

ON Saturday, April 3, 1971, the American Academy of Ophthalmology and Otolaryngology will offer the first Annual In-training Examination in Otolaryngology. This examination will be offered free of charge to all residents-in-training and practicing otolaryngologists. The examination is being offered as a service to otolaryngology by the academy through its Office of Continuing Education in Otolaryngology. Since this is otolaryngology's initial venture in the field of in-training examinations, we feel it important that the otolaryngological community be conversant with the details of the examination. The examination and its feedback should be of tremendous importance and value to all residents, program directors, and practicing otolaryngologists. We hope that this description of the examination, its preparation and administration, will provide sufficient understanding of the examination to encourage wide participation. Purpose of the Examination.—The examination, on an annual voluntary basis, will provide an opportunity for residents-in-training to monitor their acquisition of knowledge

First Annual In-Training Examination in Otolaryngology

First Annual In-Training Examination in Otolaryngology