Enhancing Internal Medicine Residents’ Royal College Exam Competency Using In-Training Written Exams within a Competency Based Medical Education Framework

Abstract

Background: Canadian residency programs have adopted competency-based medical education, where time-based learning systems are replaced with core competency “milestones” that must be achieved before a student progresses. Assessment tools must be developed to predict performance prior to high-stakes milestones, so interventions can be targeted to improve performance.Objectives: 1. To characterize how well each of three practice written exams predicts passing the Canadian Internal Medicine Royal College (RC) exam. 2. To determine if writing practice exams is perceived to improve performance on the RC exam.Methods: Three 105-question multiple choice question exams were created from a range of internal medicine topics, and offered one month apart to 35 residents. Percentile ranks on each practice exam were compared to the result (pass/fail) on the RC exam. Surveys were completed within 1 month after the RC exam.Results: There were 35 residents invited to participate. Practice exams (PE) 1, 2, and 3 were taken by 33, 26, and 22 residents, for an exam participation rate of 94.3, 74.3, and 62.9%, respectively. Failure on the RC exam could be predicted by percentile ranking <15% on PE1 (OR 19.5, p=0.017) or PE2 (OR 63.0, p=0.006), and by percentile ranking <30% on PE1 (OR 28.8, p=0.003), PE2 (OR 24.0, p=0.010) or PE3 (OR 15.0, p=0.046). The survey was sent out to the 33 participants. Of those, the total number of respondents was 25, with a response rate of 75.5%. Survey takers agreed that practice written exams improved performance on the RC exam (18/25, 88%).Conclusions: Performance in the Canadian Internal Medicine RC Exam can be predicted by performance on any of three practice written exams. This tool can therefore identify trainees for whom additional resources should be invested, to prevent failure of a high-stakes milestone within the competency based medical education framework.RÉSUMÉContexte : Les programmes canadiens de résidence ont choisi de diffuser un enseignement médical axé sur les compétences dans lequel les systèmes d’apprentissage structurés en fonction du temps sont remplacés par des « jalons » liés aux compétences fondamentales que l’étudiant doit atteindre pour aller de l’avant. Il faut élaborer des outils d’évaluation pour prédire la probabilité de résultats escomptés par un étudiant avant que celui-ci ne se présente à certains événements dont les enjeux sont élevés. Ainsi, il devient possible d’intervenir de manière à améliorer les résultats escomptés.Objectifs : 1. Déterminer dans quelle mesure chacun des trois examens de pratique écrits prédit la réussite à l’examen du Collège royal des médecins et chirurgiens du Canada (CRMCC) en médecine interne; 2. Évaluer si le fait de se soumettre à des examens de pratique écrits est perçu comme un élément qui améliore les résultats à l’examen du CRMCC.Méthodologie : Trois examens écrits comportant chacun 105 questions à choix de réponses portant sur un éventail de sujets relatifs à la médecine interne ont été préparés et proposés à 35 résidents à intervalle d’un mois. Les rangs-centiles de chaque examen de pratique ont été comparés avec le résultat obtenu à l’examen du CRMCC (succès/échec). Les sondages ont été effectués dans le mois suivant l’examen du CRMCC.Résultats : Trente-cinq résidents ont été invités aux examens de pratique écrits (EP) 1, 2 et 3. La participation a été respectivement de 33, 26 et 22 résidents, soit de 94,3 %, 74,3 % et 62,9 %. L’échec à l’examen du CRMCC pouvait être prédit par un rang-centile < 15 % à l’EP1 (OR 19,5 et p = 0,017) ou à l’EP2 (OR 63,0 et p = 0,006) et un rang-centile < 30 % à l’EP1 (OR 28,8 et p = 0,003), à l’EP2 (OR 24,0 et p = 0,010) ou à l’EP3 (OR 15,0, et p = 0,046). Le sondage a été envoyé aux 33 participants. Le nombre total de répondants a été de 25, pour un taux de réponse de 75,5 %. La majorité des répondants (18/25, 88 %) sont d’avis que les examens de pratique écrits leur ont permis d’obtenir de meilleurs résultats à l’examen du CRMCC.Conclusions : Les résultats à l’examen du Collège royal des médecins et chirurgiens du Canada (CRMCC) en médecine interne peuvent être prédits par les résultats obtenus à l’un des examens de pratique écrits. Par conséquent, cet outil peut être utilisé dans le cadre de l’enseignement de la médecine axé sur les compétences pour identifier sur qui l’on devrait investir des ressources additionnelles en vue d’éviter un échec à cet événement aux enjeux élevés.Competency-based medical education (CBME) has generated increased attention over the last decade,1–3 and become entrenched within several national medical education frameworks including Canada.4 Proponents of CBME suggest that older medical education models focus on medical knowledge rather than skills, or higher order aspects of practice. 5 Focus on time spent in training can take away from the abilities acquired during that time frame.6 Furthermore, flexible time periods may be more efficient and focused, compared to time-based curriculum.3,6,7 In light of these advantages, the Royal College of Physicians and Surgeons of Canada (RCPSC) has committed to transform medical education to a CBME model for all residency programs by 2017.4While residency programs reorganize toward the CBME model, residents will still be required to perform oral and written exams. It is thus essential that CBME-based programs incorporate assessment tools to predict performance on high-stakes milestones, such as RC exams.We created three written PEs, and evaluated how well each predicted performance on a high-stakes milestone, the RCPSC Internal Medicine exam (RC exam). We also evaluated how well PE were perceived to improve performance on the same high-stakes milestone RC exam.METHODSSetting and ParticipantsThe RC exam contains both written and oral components. All residents sitting both components of the RC exam, within 12 months, who were post-graduate medical residents at Western University (London, Ontario, Canada), were invited to participate. The study was conducted in 2013-2014.Western University Health Sciences Research Ethics Board provided an ethics waiver for this study, since the study was performed as part of the standard operations of an educational program.Intervention: ExamsTwo authors (HS, BT) separately created multiple choice questions (MCQ) reflecting all areas of internal medicine, based on the Objectives of Training of the RC Internal Medicine exam. RC exam questions are not available for purchase, and examinees are forbidden to share RC exam questions. Therefore, PE content and question style was informed by questions purchased for American Board of Internal Medicine (ABIM) course reviews.8,9 MCQ creators had each completed the RC exam within 3 years, and were thus familiar with MCQ and exam format.All authors independently reviewed each PE question to assure quality of content, grammar, spelling, and syntax. Each PE covered all subspecialty areas within internal medicine, including allergy and immunology ( n=4), cardiology (n=13), dermatology (n=2), endocrinology (n=8), gastroenterology (n=10), hematology (n=10), infectious diseases (n=15), nephrology ( n=9), neurology (n=6), oncology (n=4), respiratory and critical care medicine (n=7), rheumatology ( n=14) and statistics (n=3). This topic allocation included 7 questions for JAMA Rational Clinical Exam, and 5 for interpretation of medical images (e. g., chest X-ray, electrocardiogram). PE size (105 questions) and length (3 hours) were chosen to reflect the RC exam.Each PE was offered at two separate times, to assure flexibility within ongoing clinical responsibilities. PE1, PE2, and PE3 were offered approximately 7, 6, and 5 months prior to RC exam, respectively. This timing was chosen so that trainees had sufficient time to improve their performance before the RC exam if a poor PE result was found.Examinees were provided a personalized report for each exam, within 7 days of completing the PE. The personalized report included the examinee’s overall mark, average within each subspecialty, and percentile rank within the entire cohort of examinees. Two separate 1-hour periods were available to review each PE results, with the questions and key, supervised by BT.Intervention: SurveyAll study participants were invited to participate in a survey. The survey assessed how well PE simulated the RC exam, whether the PE were recommended to the next year’s cohort of examinees, and whether the PE improved performance on the RC exam.Outcomes: ExamsEach study participant agreed to provide the RC exam result (“pass” or “fail”) once he or she had received it. Each candidates verbally communicated RC exam result was confirmed online 3 months after the RC exam results were reported (cpso.on.ca).Odds ratios were calculated. The adverse outcome was failure on the RC exam. Exposures evaluated included percentile rank < 15% and <30%. Odds ratios of infinity were prevented by adding 1 adverse outcome to any exposure group without any adverse outcomes; this was performed for 3 exposure groups, but did not impact whether statistical significance was attained. Results are detailed in Table 1.Outcome: SurveySurvey results were on a Likert Scale. The proportion of those respondents who agreed or disagreed were calculated.All data was analyzed using Statistical Package for the Social Sciences (SPSS) version 21.0.RESULTSSetting and ParticipantsThere were 35 residents invited to participate, the total number of possible participants. PE1, PE2, and PE3 were taken by 33, 26, and 22 residents, for an exam participation rate of 94.3, 74.3, and 62.9%, respectively. The majority of invitees took 3 (n=17) or 2 ( n=14), while a minority took 1 (n=2) or 0 (n=2) practice exams.ExamsOf all examinees of the RC exam (n=35), 7 failed. RC exam pass rates were lower when PE1 percentile rank was lower than 15% (40.0 vs. 92.9%, p<0.001) or 30% (44.4 vs. 100%, p<0.004), when PE2 percentile rank was lower than 15% (0.0 vs. 100.0%, p <0.001) or 30% (42.9 vs. 100.0%, p=0.038), and when PE3 percentile rank was lower than 30% (50 vs. 93.75%, p=0.046) (Figure 1). Figure 1. License exam practice pass rate versus percentile rate (PR) on practice exams. Examinees were more likely to fail the RC exam if percentile rank was less than 15% (OR 19.5, p=0.017) or 30% (OR 28.8, p=0.003) in PE1, less than 15% (OR 63.0, p=0.006) or 30% (OR 24.0, p =0.010) in PE2, or less than 30% (OR 15.0, p=0.046) in PE3.SurveyOnly residents who had taken at least 1 practice exam were invited to participate. The survey was sent out to the 33 participants, the total number of possible participants. Of those, the total number of respondents was 25, with a response rate of 75.5%. Most survey respondents agreed that the PEs were an accurate simulation of the written component of the RC examination (20/25, 80%) (Figure 2A). Most survey respondents agreed that the PEs improved performance on the RC written examinations (18/25, 72%) (Figure 2B). Most survey respondents recommended future residents to take the PEs (22/25, 88%) (Figure 2C).DISCUSSIONWe describe the creation of a tool to assess performance on a high-stakes milestone examination, the RC exam. This tool is easy to create, affordable, and is administered on a voluntary basis with high uptake amongst candidates writing the RC exam. The assessment tool has been shown to predict performance well so that resources can be invested in those at risk for failing.There is a possibility that mere participation in the assessment tool itself improves performance on this high-stakes exam. There were insufficient numbers of study participants to determine a correlation between number of exams taken and pass rates. Even still, unwillingness to participate in the study may reflect a general unwillingness to prepare, which means the results would be confounded and correlative rather than causative. One way to look into this is to perform a randomized trial in which half of residents take the assessment tool and the other half doesn’t. Unfortunately, almost all invited residents were anxious to participate, rendering such a possible study impossible. On the other hand, exam takers were able to communicate usefulness of the exam and to provide feedback on how it might be improved for future years.As CBME develops and becomes entrenched, there will continue to be a need to prepare for knowledge based written exams. This exam will continue to be considered a core competency between the stages to transition to practice. Thus, tools are needed to assess exam competence. This study confirms that such tools can and should be developed to assure that trainees are prepared.Ideally, residents with low performance would be identified early enough to intervene to change the outcome. It is uncertain what the ideal time frame is or what the intervention should be. It is reasonable to assume that taking the examination earlier in their training may allow candidates to become aware of their performance and implications thereof and implement earlier changes in learning strategies. For example, in past years, candidates contacted their program directors to ease the clinical workload to allow more study time. Others sought counselling and mentorship from staff, while others were self-directed in their learning and became more motivated to study. On the other hand, poor performance on this formative examination could potentially discourage some residents from studying if they felt their studying was futile. Future research efforts should focus on identifying which intervention is optimal to modify exam performance.The failure rate of 20% on the RC exam the year the study was conducted was unusually high for the program; however, this allowed for a correlation to be established between the PEs and the RC exam. The PEs were able to identify all those who failed the RC exam. However, there were those who scored below the 30th percentile on the PEs and still passed the RC exam. It must be kept in mind that the purpose of these formative exams is to identify those at high risk of failure so they can receive remedial support and improve their chances of passing. Thus, it is possible that through increased remedial support, those candidates who did poorly on the in-training exam managed to pass the RC exam.Only 2 trainees chose to write none of the PEs. While both of these trainees ultimately failed the RC exam, statistical significance could not be established due to the small sample size. It thus remains uncertain whether the act of writing PEs predicts passing the RC exam. However, the study objective was to identify candidates at high risk of failing the RC exam; the next step will be to determine which interventions can improve RC exam result. However, it must be acknowledged that PEs could not only identify candidates at risk of failing RC exam, but also improve their performance. This requires future study before any firm conclusions can be found.This study confirms that formative exams’ results can predict failure on the RC exam. The questions were written by authors who had recently written the RC exam, familiar with its format, and knowledgeable of the current Canadian guidelines, which are a focus of the actual RC exam. Because of confidentiality agreements with the RCPSC, actual RC exam questions can’t be shared, and thus can’t be used as part of the practice exams. However, we attempted to overcome this limitation by having all PE questions reviewed by at least 3 physicians who’d recently successfully completed the RC exam, to assure syntax and format was as similar as possible between PE and RC exam. Furthermore, this limitation does not impact the PEs statistically significant prediction of candidates at risk of failing the RC exam. The study objective was to identify candidates at risk of failing the RC exam, and the PEs are indeed a valid predictor of RC exam performance. There are limitations to this study. Firstly, this was a single centre study. However, Western University has a wide range of subspecialty programs available, and the trainees’ demographics resemble that at other Canadian centers. Secondly, new questions need to be created annually to reflect updated literature and guidelines; this requires ongoing commitment and dedication from staff. These “updated” exams could become more difficult to validate if candidates no longer fail the RC exam. However, if the act of taking the PE predicts passing RC, future research could focus on comparing RC pass rates at programs with and without the PEs. Thirdly, it's entirely possible that the use of questions from old RC exam would be more predictive, but these questions cannot be shared or used for PE due to the confidentiality agreement with the RCPSC. Therefore, creation of independent questions is still required. This is the first study of an assessment tool to predict performance on the Canadian internal medicine examination within the CBME framework. This strategy can easily be replicated and feedback is rapidly provided in a time sensitive manner. This could help trainees direct their preparation and identify knowledge gaps more easily.CONCLUSIONWe report an assessment tool to predict performance on the RC exam that can be a valid and useful form of feedback. This strategy can easily be replicated for other subspecialties or internal medicine programs. Future efforts need to focus on how the results can determine which interventions or learning strategies improve the results of candidates identified to be at risk for failing.DisclaimersThe authors declare they have no competing interest.The authors report no external funding source for this study.The authors declare no previous abstract or poster or research presentation or any online presentation of this study.REFERENCES 1. Caccia N, Nakajima A, Kent N. Competency-based medical education: the wave of the future. J Obstet Gynaecol Can 2015;37:349–53. 2. Carraccio C, Englander R, Gilhooly J, et al. Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. Acad Med 2016 ;92(3):324–30. doi: 10.1097/ACM.0000000000001141. 3. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med 2002;77:361–67. 4. Johnston C. Residents prepare for switch to competency-based medical education. CMAJ2013;185:1029. 5. Talbot M. Monkey see, monkey do: a critique of the competency model in graduate medical education. Med Educ 2004;38:587–92. 6. Long DM. Competency-based residency training: the next advance in graduate medical education. Acad Med 2000;75:1178–83. 7. Bell HS, Kozakowski SM, Winter RO. Competency-based education in family practice. Fam Med 1997;29:701–704.8. Mittman B. Frontrunners 2016: Internal Medicine Q&A Review: Syllabus Companion for Board Review/Practice Questions & Answers for the ABIM Exam. Aliso Viejo, CA: Frontrunners Publishing; 2016.9. Fischer C. Internal Medicine Question Book: Second Edition: Complete Preparation for the American Board of Internal Medicine Exam. New York, NY: Kaplan Publishing; 2009.