Orthopaedic In-Training Examination Scores Research Articles

Validation of a Surgical Objective Structured Clinical Examination (S-OSCE) Using Convergent, Divergent, And Trainee-Based Assessments of Fidelity

Describe the validation of a surgical objective structured clinical examination (S-OSCE) for the purpose of competency assessment based on the Royal College of Canada's Can-MEDS framework. A surgical OSCE was developed to evaluate the management of common orthopedic surgical problems. The scores derived from this S-OSCE were compared to Ottawa Surgical Competency Operating Room Evaluation (O-SCORE), a validated entrustability assessment, to establish convergent validity. The S-OSCE scores were compared to Orthopedic In-Training Examination (OITE) scores to evaluate divergent validity. Resident evaluations of the clinical encounter with a standardized patient and the operative procedure were scored on a 10-point Likert scale for fidelity. A tertiary level academic teaching hospital. 21 postgraduate year 2 to 5 trainees of a 5-year Canadian orthopedic residency program creating 160 operative case performances for review. There were 5 S-OSCE days, over a 4-year period (2016-2019) encompassing a variety of surgical procedures. Performance on the S-OSCE correlated strongly with the O-SCORE (Pearson correlation coefficient 0.88), and a linear regression analysis correlated moderately with year of training (R² = 0.5345). The Pearson correlation coefficient between the S-OSCE and OITE scores was 0.57. There was a significant increase in the average OITE score after the introduction of the surgical OSCE. Resident fidelity ratings were available from 16 residents encompassing 8 different surgical cases. The average score for the overall simulation (8.0±1.6) was significantly higher than the cadaveric surgical simulation (6.5 ± 0.8) (p < 0.001) CONCLUSIONS: The S-OSCE scores correlate strongly with an established form of assessment demonstrating convergent validity. The correlation between the S-OSCE and OITE scores was less, demonstrating divergent validity. Although residents rank the overall simulation highly, the fidelity of the cadaveric simulation may need improvement. Administration of a surgical OSCE can be used to evaluate preoperative and intraoperative decision making and complement other forms of assessment.

Do Orthopaedic In-Training Examination Scores Predict the Likelihood of Passing the American Board of Orthopaedic Surgery Part I Examination? An Update With 2014 to 2018 Data.

Previous research shows a correlation in performance between the Orthopaedic In-Training Examination (OITE) and the American Board of Orthopaedic Surgery (ABOS) Certifying Examination Part I; however, these studies are not current, with the most recent data from 2009. The purpose of this study was to update the relationship between the OITE and ABOS Part I scores with the five most recent nationwide cohorts of Part I scores and their corresponding OITE scores. The American Academy of Orthopaedic Surgeons provided the results for each resident taking the OITE from 2013 to 2017. The ABOS provided the results for each resident taking the Part I examination from 2014 to 2018. These two datasets were matched at the individual level and analyzed. Between 2014 and 2018, 3,502 first-time test-takers were present for the ABOS Part I Examination. A 96.6% pass rate was noted (3,383 of 3,502). A statistically significant correlation between the OITE score and ABOS Part I score was observed at all levels of training: postgraduate year (PGY) 1 r = 0.380, PGY2 r = 0.463, PGY3 r = 0.498, PGY4 r = 0.504, and PGY5 r = 0.504 (P < 0.001 for all correlations). The OITE scores continue to correlate with the ABOS scores and the pass rate for the ABOS with moderate strength correlation. However, although a correlation of the examinations exists when evaluated as a group, the predictive value of the OITE for passing the Part I examination for any specific individual is far from perfect. Individuals with PGY5 OITE percentile scores less than 10 pass the Part I examination, and individuals with PGY5 OITE percentile scores greater than 90 fail the Part I examination.

Orthopaedic In-training Exam Preparation among Orthopaedic Surgery Residency Programs

The Orthopaedic In-Training Examination (OITE) is given annually to residents to assess their knowledge of clinical orthopaedics and orthopaedic basic science. This study sought to determine what resources orthopaedic residency programs recommend and/or require for residents as preparative tools for the OITE and to understand which resources are most beneficial. An anonymous electronic survey was distributed to program directors of orthopaedic surgery residency programs. It included questions about resources that program directors recommended or required residents to use when preparing for the OITE. The survey was prepared and the results analyzed at two academic medical institutions in Louisiana. The survey was available to respondents in December 2019 and January 2020. The survey was delivered to program directors of 148 ACGME accredited orthopaedic surgery residency programs in the United States. 148 orthopaedic surgery residency program directors received the survey and 44 responded (response rate: 29.7%). Twenty-nine programs (73.2%) reported that practice tests/review of old OITE exams positively correlated with better scores. The most recommended resources for OITE preparation were Orthobullets (35 programs; 85%) and Res Study (AAOS) (27 programs; 67.5%). Programs having formal/required preparation programs had lower mean overall scores on the OITE than programs that did not. Those without a formal/required program scored in the 60th percentile, while those with a formal/required program scored in the 53.3rd percentile, a difference of 6.7 percentile points (p = 0.049). The mean overall percentiles were significantly higher for programs that reported having a threshold/goal OITE score for residents compared to those that didn't. Programs with a threshold/goal score scored in the 60.6th percentile on average, while those without a threshold/goal score scored in the 51.7th percentile on average, a difference of 8.9 percentile points (p = 0.0095). There is substantial variability in the resources that are utilized in preparation for the OITE, with the most commonly recommended resource being Orthobullets. The study method most cited as having a positive impact on scores was practice tests/review of old OITE exams. Having a formal/required program resulted in programs having lower OITE scores. Setting a goal/threshold score correlates positively with an increase in OITE score. Encouraging residents to use practice questions/old OITE tests, setting a goal/threshold score and avoiding formal/required preparation programs may improve resident performance on the OITE.

A Learner-Centered Education Strategy: Optimizing an Interactive and Multimodal Learning Format in Orthopaedic Hip & Knee Arthroplasty

The Accreditation Council for Graduate Medical Education (ACGME) and The American Board of Orthopaedic Surgery proposed The Orthopaedic Surgery Milestone Project. Training residency and fellowship programs have evolved accordingly adjusting academic curriculums. A new comprehensive Learner-Centered Education Curriculum (LCEC) was designed based on critical reviews and interactive collaboration between faculty, residents, and fellows using structured interviews and iterative feedback. We aim to evaluate the results at 4 years after implementation of a new curriculum. The new adult arthroplasty LCEC was implemented in 2015; data collected between 2015 and 2019 was retrospectively reviewed and analyzed. Our primary goal was to evaluate educational, research, and quality successes using objective and quantitative academic quality metrics including annual Orthopedic In-Training Examination scores for the Hip & Knee domain to evaluate the medical knowledge competency. This study was conducted at the Department of Orthopedic Surgery residency and fellowship program of Adult Arthroplasty and Joint Reconstruction Surgery, University of Florida (tertiary care center). Participants include 25 learners (PGY-1 to 5, Clinical fellows) and faculty of an ACGME-accredited orthopedic surgery residency and adult arthroplasty fellowship program. Significant improvements in academic, research and quality metrics were obtained since implementation of the LCEC: Orthopedic In-Training Examination scores for Hip & Knee domain increased from 50th to 87th percentile (P=0.042), annual learner peer-reviewed publications and research awards from none before intervention to 20 and 8 respectively (p < 0.05). Resident and fellows conference attendance improved from 81% to 99% (p = 0.0001). The ACGME resident Likert-evaluations also improved from 4.6 to 4.8. This LCEC, by enhancing an interactive and active academic learning experience, positively influenced fund of knowledge, conceptual thinking, and interest in the specialty and learner attitudes. Significant and consistent improvements in academic, research, and quality metrics were obtained, while maintaining the highest resident evaluation scores in the program.

Orthopedic In-Training Examination: A Performance Review Based on Program- and Resident-Specific Characteristics

The orthopedic in-training examination (OITE) is the most common and objective method used to assess resident knowledge in the United States. As such, residents and programs use a number of strategies to maximize OITE performance. The purpose of this work was to better understand what strategies were being implemented and to determine which program-specific and resident-specific characteristics best correlate with improved scores. A national survey of orthopedic residents and program directors (PDs) was performed to better understand OITE performance and correlate scores with various test preparation strategies. Mayo Clinic, Rochester, MN. The survey was completed by 33 of 48 (68.8%) PDs and 341 of 878 (38.8%) eligible residents. The most commonly used program-wide strategies were as follows: negative consequences for poor performance (72.7%), formal OITE prep program (54.5%), and purchase of OITE test prep material for residents (51.5%). The program-specific characteristics that had the strongest correlation with increased scores were negative consequences for poor performance (p < 0.001), high value placed on the OITE by PD and residents (p < 0.001), excusing residents from clinical duties the evening prior (p < 0.001), having residents take the examination on different days (p = 0.012), and allowing residents to lead a review course (p = 0.047). The resident-specific characteristics that correlated most with score were increased study time leading up to the test (p = 0.031) and attendance at their program's OITE prep program (p = 0.062). Although programs and residents looking to improve knowledge acquisition and OITE scores use a number of techniques, a few distinct strategies correlate with the greatest increases in OITE performance. These may be appropriate methods to consider for those looking to improve their performance in coming years.

Orthopedic Residency: Are Duty Hours Predictive of Performance?

This study examines the relationship between self-recorded resident work hours and Orthopedic In-training Examination (OITE) scores, resident clinical performance, and American Board of Orthopedic Surgery pass rates. The hypothesis of this study is that increasing duty hours would have a positive correlation with clinical and OITE performance. Total duty hours and recorded operating room hours from a single orthopedic residency program were extracted from 2006 to 2012. During the same time span, OITE scores, resident clinical scores from the E-Valuation system, and American Board of Orthopedic Surgery pass rates were collected. The correlation between the variables was assessed using the Pearson correlation coefficient's precision statistic. A large public tertiary academic center in the upper Midwestern United States. A total of 82 orthopedic surgery residents over 7 years. A total of 82 residents were matriculated between 2006 and 2012. The average weekly recorded duty hours were as follows: postgraduate year 2 (PGY2) = 60 hours/week (Standard Deviation (SD) ± 4), PGY3 = 59 hours/week (SD ± 5), PGY4 = 51 hours/week (SD ± 4), PGY5 = 49 hours/week (SD ± 3). There was significant variability in the average number of hours worked among residents (range: 2128-3753h/y) for the full academic year. The OITE scores and the work hours were found to be independent of each other (ρ = 0.017, p = 0.825), and no correlation was found between OITE scores and the resident E-value scores (ρ = 0.071, p = 0.34). Residents spent 36% to 48% of their time in the operating room. Second year residents logging more hours scored higher on faculty evaluation of overall competency (ρ = 0.31, p = 0.035). Faculty assessment of technical skills had a positive correlation with operating room duty hours for PGY5 class (ρ = 0.346, p = 0.025). A large variation in duty hours exists between resident-logged duty hours. No correlation exists between in-training scores and duty hours. There is a positive correlation between senior resident operating room hours and technical skill scores.

Is There an Association Between Study Materials and Scores on the American Board of Orthopaedic Surgeons Part 1 Examination?

BackgroundPrevious studies have shown that certain orthopaedic in-training examination scores can be used to identify which residents may be at risk for failing the American Board of Orthopaedic Surgeons (ABOS) Part 1 examination. However, no studies have examined how study resources may affect residents’ ABOS Part 1 scores. The goal of this study is to determine which review sources or review courses, if any, are associated with improved ABOS Part 1 scores. MethodsA survey was sent to 221 of the 865 examinees who took the ABOS Part 1 examination in 2012. The questions inquired the respondents how well they performed on previous orthopaedic in-training examinations and ABOS Part 1, along with the study sources they most commonly used, review courses they attended, and resources they would recommended if they were to retake ABOS Part 1 examination. ResultsOverall, 118 of the 221 (53%) survey recipients completed the survey. Six (5%) of the respondents failed ABOS Part 1 examination. Orthobullets and the American Academy of Orthopaedic Surgeons self-assessment examinations were recommended as the primary study source significantly more (p < 0.01) than most other resources, but there was no significant association between study source and passing ABOS Part 1 or scoring in a certain percentile on ABOS Part 1. Similarly, there were no associations between attending a review course and either passing or scoring in a certain percentile for ABOS Part 1. Half of the respondents who failed ABOS Part 1 attended multiple review courses. ConclusionsThere does not appear to be an association between improved ABOS Part 1 scores and orthopedic study materials or review courses. Further research into the value of certain educational modalities should be conducted to determine the best ways to educate orthopedic residents and determine the value of some of these commonly used orthopedic review modalities.

Use of an Integrated, Anatomic-Based, Orthopaedic Resident Education Curriculum: A 5-Year Retrospective Review of Its Impact on Orthopaedic In-Training Examination Scores

Experts have called for a comprehensive didactic curriculum in orthopaedic residency training. This study examined the effects of an anatomic-based, integrated conference program on annual Orthopaedic In-Training Examination (OITE) scores at a single orthopaedic residency program. We implemented a new, integrated, anatomic-based curriculum in January 2005. Differences between scores were analyzed by postgraduate year (PGY) of training. OITE scores (percentile ranking and raw scores) of year 1 (PGY-2) through year 4 (PGY-5) residents exposed to the curriculum (2005-2009) were compared to prior PGY-2 through PGY-5 residents (2000-2004) who had experienced the previous unstructured curriculum. To evaluate for cohort effects, United States Medical Licensing Examination (USMLE) Step I scores for these 2 groups were also compared. Eight residents were exposed to the new conference program and 8 to the prior conference program. All residents' percentile rankings improved after exposure to the curriculum, although improvement was not statistically significant for all participants. The most dramatic improvements in OITE scores were seen for PGY-4 and PGY-5 residents, which improved from 65th to 91st percentile (P = .03) and from 66th to 91st percentile (P = .06), respectively. There were no differences between the cohorts in USMLE Step I scores. Initiation of an integrated, anatomic-based, resident conference program had a positive impact on resident performance on the OITE.

Success in Orthopaedic Training: Resident Selection and Predictors of Quality Performance

Multiple studies have attempted to determine which attributes are predictive of success during residency as well as the optimal method of selecting residents who possess these attributes. Factors that are consistently ranked as being important in the selection of candidates into orthopaedic residency programs include performance during orthopaedic rotation, United States Medical Licensing Examination (USMLE) Step 1 score, Alpha Omega Alpha Honor Medical Society membership, medical school class rank, interview performance, and letters of recommendation. No consensus exists regarding the best predictors of resident success, but trends do exist. High USMLE Step 1 scores have been shown to correlate with high Orthopaedic In-Training Examination scores and improved surgical skill ratings during residency, whereas higher numbers of medical school clinical honors grades have been correlated to higher overall resident performance, higher residency interpersonal skills grading, higher resident knowledge grading, and higher surgical skills evaluations. Successful resident performance can be measured by evaluating psychomotor abilities, cognitive skills, and affective domain.

Resident Selection

Selection of the best applicants for orthopaedic residency programs remains a difficult problem. Most quantifiable factors for residency selection evaluate test-taking ability and grades rather than other aspects, such as patient care, professionalism, moral reasoning, and integrity. Four current department members on our resident selection committee ranked four consecutive classes of orthopaedic residents interviewed for residency. We ranked incoming residents in order of best to least qualified and compared those rankings with rank lists by the same faculty on completion of residency. Rankings also were compared with the residents' United States Medical Licensing Examination (USMLE) Part I scores, American Board of Orthopaedic Surgery (ABOS) Part I scores, and fourth-year Orthopaedic-in-Training Examination (OITE) scores. We found fair or poor correlations between the residents' initial rankings, rankings on graduation, and their USMLE, ABOS, and OITE scores. The only relatively strong correlation found was between the OITE and ABOS scores. Despite the faculty's consensus regarding selection criteria, interviewers did not agree in their rankings of residents on graduation. Additional work is necessary to refine the inexact yet important science of selecting residency applicants.

A Quantitative Composite Scoring Tool for Orthopaedic Residency Screening and Selection

The ability to accurately screen and select orthopaedic resident applicants with eventual successful outcomes has been historically difficult. Many preresidency selection variables are subjective in nature and a more standardized objective scoring method seems desirable. A quantitative composite scoring tool (QCST) to be used in a standardized manner to help predict orthopaedic residency performance from application materials was developed. In 64 orthopaedic residents, four predictors (United States Medical Licensing Examination [USMLE] Part I scores, Alpha Omega Alpha status, junior year clinical clerkship honors grades, and the QCST score) were analyzed with respect to four residency outcomes assessments. The outcomes included three standardized assessments, the orthopaedic in-training examination scores (OITE), the American Board of Orthopaedic Surgery (ABOS) written and oral examinations, and an internal outcomes assessment, attainment of satisfactory chief resident associate (CRA) status. Collectively, the QCST score had the strongest association as a predictor for all three standardized outcomes assessments (p < 0.001). Honors grades during junior years clinical clerkships was most strongly associated with satisfactory CRA status (p < 0.001). A composite scoring tool that is an effective predictor of orthopaedic resident outcomes can be developed. Additional work is still required to refine this scoring tool for orthopaedic residency screening and selection.

Orthopaedic In-Training Examination Scores: A Correlation with USMLE Results

Both the United States Medical Licensing Examination and the Orthopaedic In-Training Examination measure factual recall as well as interpretative and problem-solving skills. The former examination is used to a variable degree by postgraduate programs in resident selection. Orthopaedic In-Training Examination scores are one measure of the medical knowledge of residents and are used by all American orthopaedic residency programs on a yearly basis. This investigation was performed to retrospectively review Orthopaedic In-Training Examination scores of orthopaedic residents who took the examination in our program. In addition, we sought to determine whether a relationship existed between performance on the Orthopaedic In-Training Examination and the United States Medical Licensing Examinations taken while in medical school. The records of each orthopaedic resident who took the examination from November 1993 through November 2000 were reviewed. Correlation coefficients and 95% confidence intervals were calculated to assess the relationship, if any, between the Orthopaedic In-Training Examination percentiles and the three-digit scores on the Step-1 and Step-2 United States Medical Licensing Examination. In addition, examination scores were evaluated longitudinally from year-in-training 1 through 4. A significant moderate-sized correlation was found between United States Medical Licensing Examination Step-2 scores and Orthopaedic In-Training Examination score percentiles (p < 0.05); however, with the numbers available, no correlation was seen between United States Medical Licensing Examination Step-1 scores and Orthopaedic In-Training Examination scores. The mean Orthopaedic In-Training Examination scores were in the 66th percentile for year-in-training 1, the 53rd percentile for year 2, the 57th percentile for year 3, and the 50th percentile for year 4. Residents in the laboratory for one year scored in the 88th percentile while in the laboratory (year 0), in the 86th percentile in year 1, and in the 48th percentile in year 4. Although Step-1 United States Medical Licensing Examination scores have been used by our department as a major factor in resident selection historically, our data failed to reveal a significant correlation with performance on the Orthopaedic In-Training Examination. The decrease in Orthopaedic In-Training Examination scores over time for our residents who worked in the laboratory is most likely attributable to multiple factors, including clinical workload hours.

ORTHOPAEDIC IN-TRAINING EXAMINATION SCORES

Background: Both the United States Medical Licensing Examination and the Orthopaedic In-Training Examination measure factual recall as well as interpretative and problem-solving skills. The former examination is used to a variable degree by postgraduate programs in resident selection. Orthopaedic In-Training Examination scores are one measure of the medical knowledge of residents and are used by all American orthopaedic residency programs on a yearly basis. This investigation was performed to retrospectively review Orthopaedic In-Training Examination scores of orthopaedic residents who took the examination in our program. In addition, we sought to determine whether a relationship existed between performance on the Orthopaedic In-Training Examination and the United States Medical Licensing Examinations taken while in medical school. Methods: The records of each orthopaedic resident who took the examination from November 1993 through November 2000 were reviewed. Correlation coefficients and 95% confidence intervals were calculated to assess the relationship, if any, between the Orthopaedic In-Training Examination percentiles and the three-digit scores on the Step-1 and Step-2 United States Medical Licensing Examination. In addition, examination scores were evaluated longitudinally from year-in-training 1 through 4. Results: A significant moderate-sized correlation was found between United States Medical Licensing Examination Step-2 scores and Orthopaedic In-Training Examination score percentiles (p < 0.05); however, with the numbers available, no correlation was seen between United States Medical Licensing Examination Step-1 scores and Orthopaedic In-Training Examination scores. The mean Orthopaedic In-Training Examination scores were in the 66th percentile for year-in-training 1, the 53rd percentile for year 2, the 57th percentile for year 3, and the 50th percentile for year 4. Residents in the laboratory for one year scored in the 88th percentile while in the laboratory (year 0), in the 86th percentile in year 1, and in the 48th percentile in year 4. Conclusion: Although Step-1 United States Medical Licensing Examination scores have been used by our department as a major factor in resident selection historically, our data failed to reveal a significant correlation with performance on the Orthopaedic In-Training Examination. The decrease in Orthopaedic In-Training Examination scores over time for our residents who worked in the laboratory is most likely attributable to multiple factors, including clinical workload hours.

The validity of orthopaedic in-training examination scores.

The validity of orthopaedic in-training examination scores.