Assessment In Competency-based Medical Education Research Articles

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.

International Trends: Application and Latest Developments of Competency-Based Medical Education (CBME)

<p>當代勝任能力導向醫學教育(competency-based medical education, CBME)以可信賴專業活動(entrustable professional activities, EPAs)及里程碑(milestones)為兩項主要推動項目。描述專業工作的EPAs不僅能與描述個人能力的里程碑相輔相成，近年來因其可操作性高，成為國際CBME推動上的關鍵項目。Elaine Van Melle等人提出的CBME核心組成架構，包括成效能力、序列性進展、量身打造的學習經驗、聚焦於能力的指導、計畫性評估5項，可作為落實CBME的參考。在EPAs運用上，需有完整的八大描述項目，選定合適的信賴等級量尺，善用以信賴授權為導向的討論(entrustment-based discussion, EBD)等方式來評估，並參考A-RICH框架(Agency, Reliability, Integrity, Capability, Humility)來進行信賴授權決策。CBME可縱向延伸，自專科住院醫師訓練往醫學生及PGY階段發展，也能整合科技應用進行評估。在不久的將來，時間可變訓練計畫(time-variable training programs)極有可能成真，並會是CBME全面落實的指標。</p> <p> </p><p>The contemporary competency-based medical education (CBME) focuses on individuals’ entrustable professional activities (EPAs) and milestones. The EPAs that describe professional work complement the milestones that describe individual competencies. In recent years, EPAs have become a key component in international CBME due to their high operability. Elaine Van Melle et al. proposed five core components of CBME: outcome competencies, sequenced progression, tailored learning experiences, competency-focused instruction, and programmatic assessment. This core components framework could facilitate the implementation of CBME. EPAs should be written using eight descriptive items, an appropriate entrustment scale, and assessment methods such as entrustment-based discussion; the A-RICH (Agency, Reliability, Integrity, Capability, Humility) framework should be employed to aid entrustment decision making. CBME may be extended vertically from resident doctor training to medical student training and postgraduate year training. Technologies may also be incorporated for assessment in CBME. In the near future, time-variable training programs are likely to be realized and will be an indicator for the full-scale implementation of CBME.</p> <p> </p>

The Assessment Burden in Competency-Based Medical Education: How Programs Are Adapting.

Residents and faculty have described a burden of assessment related to the implementation of competency-based medical education (CBME), which may undermine its benefits. Although this concerning signal has been identified, little has been done to identify adaptations to address this problem. Grounded in an analysis of an early Canadian pan-institutional CBME adopter's experience, this article describes postgraduate programs' adaptations related to the challenges of assessment in CBME. From June 2019-September 2022, 8 residency programs underwent a standardized Rapid Evaluation guided by the Core Components Framework (CCF). Sixty interviews and 18 focus groups were held with invested partners. Transcripts were analyzed abductively using CCF, and ideal implementation was compared with enacted implementation. These findings were then shared back with program leaders, adaptations were subsequently developed, and technical reports were generated for each program. Researchers reviewed the technical reports to identify themes related to the burden of assessment with a subsequent focus on identifying adaptations across programs. Three themes were identified: (1) disparate mental models of assessment processes in CBME, (2) challenges in workplace-based assessment processes, and (3) challenges in performance review and decision making. Theme 1 included entrustment interpretation and lack of shared mindset for performance standards. Adaptations included revising entrustment scales, faculty development, and formalizing resident membership. Theme 2 involved direct observation, timeliness of assessment completion, and feedback quality. Adaptations included alternative assessment strategies beyond entrustable professional activity forms and proactive assessment planning. Theme 3 related to resident data monitoring and competence committee decision making. Adaptations included adding resident representatives to the competence committee and assessment platform enhancements. These adaptations represent responses to the concerning signal of significant burden of assessment within CBME being experienced broadly. The authors hope other programs may learn from their institution's experience and navigate the CBME-related assessment burden their invested partners may be facing.

The influence of clinical coaching teams on quality of entrustable professional activity assessments.

Coaching is an important component of workplace-based assessment in competency-based medical education. Longitudinal coaching relationships have been proposed to enhance the trainee-supervisor relationship and promote high-quality assessment. The objective of this study was to determine the influence of longitudinal coaching relationships on the quality of entrustable professional activity (EPA) assessments. EPAs (n = 174) completed by emergency medicine (EM) supervisors between July 2020 and June 2021 were extracted and divided into two groups; one group consisted of EPAs completed by supervisors when a longitudinal coaching relationship existed (n = 87) and the other group consisted of EPAs completed by the same supervisors when no coaching relationship existed (n = 87). Three physicians were recruited to rate the EPAs using the Quality of Assessment and Learning (QuAL) score, a previously published measure of EPA quality. An analysis of variance was performed to compare mean QuAL scores between the groups. Linear regression analysis was conducted to examine the relationship between trainee performance (EPA rating) and EPA assessment quality (QuAL score). All raters completed the survey. The mean ± SD QuAL score in the coaching relationship group (3.63 ± 0.91) was higher than the no coaching relationship group (3.51 ± 1.10) but the difference was not statistically significant (p = 0.40). Supervisor was a significant predictor of QuAL score (p = 0.012) and supervisor alone accounted for 26% of the variability in QuAL scores (R2 = 0.26). There was no significant relationship between trainee performance and EPA assessment quality. The presence of a longitudinal coaching relationship did not influence the quality of EPA assessments.

SLOE: A step in the right direction

AEM Education and TrainingVolume 7, Issue 3 e10881 CORRESPONDENCE - RESPONSE TO LETTERS TO THE EDITOR SLOE: A step in the right direction Sharon Bord MD, Corresponding Author Sharon Bord MD [email protected] orcid.org/0009-0008-5987-4139 Department of Emergency Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA Correspondence Sharon Bord, Department of Emergency Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA. Email: [email protected]Search for more papers by this authorNicole Dubosh MD, Nicole Dubosh MD orcid.org/0000-0001-8674-5334 Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USASearch for more papers by this authorCullen Hegarty MD, Cullen Hegarty MD orcid.org/0000-0002-7249-8675 Department of Emergency Medicine, University of Minnesota, Minneapolis, Minnesota, USASearch for more papers by this authorDouglas Franzen MD, MEd, Douglas Franzen MD, MEd Department of Emergency Medicine, University of Washington, Seattle, Washington, USASearch for more papers by this author Sharon Bord MD, Corresponding Author Sharon Bord MD [email protected] orcid.org/0009-0008-5987-4139 Department of Emergency Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA Correspondence Sharon Bord, Department of Emergency Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA. Email: [email protected]Search for more papers by this authorNicole Dubosh MD, Nicole Dubosh MD orcid.org/0000-0001-8674-5334 Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USASearch for more papers by this authorCullen Hegarty MD, Cullen Hegarty MD orcid.org/0000-0002-7249-8675 Department of Emergency Medicine, University of Minnesota, Minneapolis, Minnesota, USASearch for more papers by this authorDouglas Franzen MD, MEd, Douglas Franzen MD, MEd Department of Emergency Medicine, University of Washington, Seattle, Washington, USASearch for more papers by this author First published: 22 June 2023 https://doi.org/10.1002/aet2.10881 Supervising Editor: Dr. Susan Promes. Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL No abstract is available for this article. REFERENCES 1Keim SM, Rein JA, Chisholm C, et al. A standardized letter of recommendation for 67 residency application. Acad Emerg Med. 1999; 6(11): 1141- 1146. 2Kukulski P, Schwartz A, Hirshfield LE, Ahn J, Carter K. Racial bias on the emergency medicine standardized letter of evaluation. J Grad med Educ. 2022; 14(5): 542- 548. 3Alvarez A, Mannix A, Davenport D, et al. Ethnic and racial differences in ratings in the medical student standardized letters of evaluation (SLOE). J Grad med Educ. 2022; 14(5): 549- 553. 4Quinn JK, Mongelluzzo J, Addo N, Nip A, Graterol J, Chen E. The standardized letter of evaluation: how we perceive the quiet student. West J Emerg Med. 2023; 24(2): 259- 263. 5Lockyer J, Carraccio C, Chan MK, et al. Core principles of assessment in competency-based medical education. Med Teach. 2017; 39(6): 609- 616. 6Love J, Smith J, Weizberg M, et al. Council of Emergency Medicine Residency Directors' 69 standardized letter of recommendation: the program Director's perspective. Acad Emerg Med. 2014; 21(6): 680- 687. 7Jackson JS, Bond M, Love JN, Hegarty C. Emergency medicine standardized letter of 77 evaluation (SLOE): findings from the new electronic SLOE format. J Grad Med Educ. 2019; 11(2): 182- 186. 8Jung J, Franzen D, Lawson L, et al. The National Clinical Assessment Tool for medical students in the emergency department (NCAT-EM). West J Emerg Med. 2018; 19(1): 66- 74. doi:10.5811/westjem.2017.10.34834 Volume7, Issue3June 2023e10881 ReferencesRelatedInformation

Exploring the Critical Driving Forces and Strategy Adoption Paths of Professional Competency Development for Various Emergency Physicians Based on the Hybrid MCDM Approach.

The implementation of competency-based medical education (CBME) focuses on learners' competency outcomes and performance during their training. Competencies should meet the local demands of the healthcare system and achieve the desired patient-centered outcomes. Continuous professional education for all physicians also emphasizes competency-based training to provide high-quality patient care. In the CBME assessment, trainees are evaluated on applying their knowledge and skills to unpredictable clinical situations. A priority of the training program is essential in building competency development. However, no research has focused on exploring strategies for physician competency development. In this study, we investigate the professional competency state, determine the driving force, and provide emergency physicians' competency development strategies. We use the Decision Making Trial and Evaluation Laboratory (DEMATEL) method to identify the professional competency state and investigate the relationship among the aspects and criteria. Furthermore, the study uses the PCA (principal component analysis) method to reduce the number of components and then identify the weights of the aspects and components using the ANP (analytic network process) approach. Therefore, we can establish the prioritization of competency development of emergency physicians (EPs) with the VIKOR (Vlse kriterijumska Optimizacija I Kompromisno Resenje) approach. Our research demonstrates the priority of competency development of EPs is PL (professional literacy), CS (care services), PK (personal knowledge), and PS (professional skills). The dominant aspect is PL, and the aspect being dominated is PS. The PL affects CS, PK, and PS. Then, the CS affects PK and PS. Ultimately, the PK affects the PS. In conclusion, the strategies to improve the professional competency development of EPs should begin with the improvement from the aspect of PL. After PL, the following aspects that should be improved are CS, PK, and PS. Therefore, this study can help establish competency development strategies for different stakeholders and redefine emergency physicians' competency to reach the desired CBME outcomes by improving advantages and disadvantages.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities.

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice.We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project.

BackgroundCanadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation.MethodsUsing a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations.ResultsBetween July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard.ConclusionsWe identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.

How Clinical Supervisors Conceptualize Procedural Entrustment: An Interview-Based Study of Entrustment Decision Making in Endoscopic Training.

Entrustment is central to assessment in competency-based medical education (CBME). To date, little research has addressed how clinical supervisors conceptualize entrustment, including factors they consider in making entrustment decisions. The aim of this study was to characterize supervisors' decision making related to procedural entrustment, using gastrointestinal endoscopy as a test case. Using methods from constructivist grounded theory, the authors interviewed 29 endoscopy supervisors in the United States and Canada across multiple specialties (adult and pediatric gastroenterology, surgery, and family medicine). Semistructured interviews, conducted between April and November 2019, focused on how supervisors conceptualize procedural entrustment, how they make entrustment decisions, and what factors they consider. Transcripts were analyzed using constant comparison to generate an explanatory framework and themes. Three themes were identified from the analysis of interview transcripts: (1) entrustment occurs in varying degrees and fluctuates over time; (2) entrustment decisions can transfer within and across procedural and nonprocedural contexts; (3a) persistent static factors (e.g., supervisor competence, institutional culture, legal considerations) influence entrustment decisions, as do (3b) fluctuating, situated dynamic factors (e.g., trainee skills, patient acuity, time constraints), which tend to change from one training encounter to the next. In the process of making procedural entrustment decisions, clinical supervisors appear to synthesize multiple dynamic factors against a background of static factors, culminating in a decision of whether to entrust. Entrustment decisions appear to fluctuate over time, and assessors may transfer decisions about specific trainees across settings. Understanding which factors supervisors perceive as influencing their decision making has the potential to inform faculty development, as well as competency committees seeking to aggregate faculty judgments about trainee unsupervised practice. Those leading CBME programs may wish to invest in optimizing the observed static factors, such that these foundational factors are tuned to facilitate trainee learning and achievement of entrustment.

Developing a dashboard for faculty development in competency-based training programs: a design-based research project.

BackgroundCanadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires frequent assessments of entrustable professional activities (EPAs). Faculty struggle to provide helpful feedback and assign appropriate entrustment scores. CBME faculty development initiatives rarely incorporate teaching metrics. Dashboards could be used to visualize faculty assessment data to support faculty development.MethodsUsing a design-based research process, we identified faculty development needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. Data was collected within the emergency medicine residency program at the University of Saskatchewan through interviews with program leaders, faculty development experts, and faculty participating in development sessions. Two investigators thematically analyzed interview transcripts to identify faculty needs that were audited by a third investigator. The needs were described using representative quotes and the dashboard elements designed to address them.ResultsBetween July 1, 2019 and December 11, 2020 we conducted 15 interviews with nine participants (two program leaders, three faculty development experts, and four faculty members). Three needs emerged as themes from the analysis: analysis of assessments, contextualization of assessments, and accessible reporting. We addressed these needs by designing an accessible dashboard to present contextualized quantitative and narrative assessment data for each faculty member.ConclusionsWe identified faculty development needs related to EPA assessments and designed dashboard elements to meet them. The resulting dashboard was used for faculty development sessions. This work will inform the development of CBME assessment dashboards for faculty.

From Utopia Through Dystopia: Charting a Course for Learning Analytics in Competency-Based Medical Education.

The transition to the assessment of entrustable professional activities as part of competency-based medical education (CBME) has substantially increased the number of assessments completed on each trainee. Many CBME programs are having difficulty synthesizing the increased amount of assessment data. Learning analytics are a way of addressing this by systematically drawing inferences from large datasets to support trainee learning, faculty development, and program evaluation. Early work in this field has tended to emphasize the significant potential of analytics in medical education. However, concerns have been raised regarding data security, data ownership, validity, and other issues that could transform these dreams into nightmares. In this paper, the authors explore these contrasting perspectives by alternately describing utopian and dystopian futures for learning analytics within CBME. Seeing learning analytics as an important way to maximize the value of CBME assessment data for organizational development, they argue that their implementation should continue within the guidance of an ethical framework.

Key considerations in planning and designing programmatic assessment in competency-based medical education

Programmatic assessment as a concept is still novel for many in clinical education, and there may be a disconnect between the academics who publish about programmatic assessment and the front-line clinical educators who must put theory into practice. In this paper, we clearly define programmatic assessment and present high-level guidelines about its implementation in competency-based medical education (CBME) programs. The guidelines are informed by literature and by lessons learned from established programmatic assessment approaches. We articulate five steps to consider when implementing programmatic assessment in CBME contexts: articulate the purpose of the program of assessment, determine what must be assessed, choose tools fit for purpose, consider the stakes of assessments, and define processes for interpreting assessment data. In the process, we seek to offer a helpful guide or template for front-line clinical educators. We dispel some myths about programmatic assessment to help training programs as they look to design–or redesign–programs of assessment. In particular, we highlight the notion that programmatic assessment is not ‘one size fits all’; rather, it is a system of assessment that results when shared common principles are considered and applied by individual programs as they plan and design their own bespoke model of programmatic assessment for CBME in their unique context.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO’s conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

Resident Exposure to Aesthetic Surgical and Nonsurgical Procedures During Canadian Residency Program Training.

North American residency programs are transitioning to competency-based medical education (CBME) to standardize training programs, and to ensure competency of residents upon graduation. At the centre of assessment in CBME are specific surgical procedures, or procedural competencies, that trainees must be able to perform. A study previously defined 31 procedural competencies for aesthetic surgery. In this transition period, understanding current educational trends in resident exposure to these aesthetic procedures is necessary. The aim of this study was to characterize aesthetic procedures performed by Canadian plastic surgery residents during training, as well as to describe resident performance confidence levels and degree of resident involvement during those procedures. Case logs were retrieved from all 10 English-language plastic surgery programs. All aesthetic procedures were identified, and coded according to previously defined core procedural competencies (CPCs) in the aesthetic domain of plastic surgery. Data extracted from each log included the procedure, training program, resident academic year, resident procedural role, and personal competence. From July 2004 to June 2014, 6113 aesthetic procedures were logged by 55 graduating residents. Breast augmentation, mastopexy, and abdominoplasty were the most commonly performed CPCs, and residents report high levels of competence and surgical role in these procedures. Facial procedures, in particular rhinoplasty, as well as nonsurgical CPCs are associated with low exposure and personal competence levels. Canadian plastic surgery residents are exposed to most of the core aesthetic procedural competencies, but the range of procedures performed is variable. With the implementation of CBME, consideration should be given to supplementation where gaps may exist in aesthetic case exposure.

Acollective case study of supervision and competence judgments on the inpatient internal medicine ward.

IntroductionWorkplace-based assessment in competency-based medical education employs entrustment-supervision scales to suggest trainee competence. However, clinical supervision involves many factors and entrustment decision-making likely reflects more than trainee competence. We do not fully understand how a supervisor’s impression of trainee competence is reflected in their provision of clinical support. We must better understand this relationship to know whether documenting level of supervision truly reflects trainee competence.MethodsWe undertook a collective case study of supervisor-trainee dyads consisting of attending internal medicine physicians and senior residents working on clinical teaching unit inpatient wards. We conducted field observations of typical daily activities and semi-structured interviews. Data was analysed within each dyad and compared across dyads to identify supervisory behaviours, what triggered the behaviours, and how they related to judgments of trainee competence.ResultsTen attending physician-senior resident dyads participated in the study. We identified eight distinct supervisory behaviours. The behaviours were enacted in response to trainee and non-trainee factors. Supervisory behaviours corresponded with varying assessments of trainee competence, even within a dyad. A change in the attending’s judgment of the resident’s competence did not always correspond with a change in subsequent observable supervisory behaviours.DiscussionThere was no consistent relationship between a trigger for supervision, the judgment of trainee competence, and subsequent supervisory behaviour. This has direct implications for entrustment assessments tying competence to supervisory behaviours, because supervision is complex. Workplace-based assessments that capture narrative data including the rationale for supervisory behaviours may lead to deeper insights than numeric entrustment ratings.Supplementary InformationThe online version of this article (10.1007/s40037-021-00652-1) contains supplementary material, which is available to authorized users.

Implementing Competency-Based Medical Education in Family Medicine: A Narrative Review of Current Trends in Assessment.

The implementation of effective competency-based medical education (CBME) relies on building a coherent and integrated system of assessment across the continuum of training to practice. As such, the developmental progression of competencies must be assessed at all stages of the learning process, including continuing professional development (CPD). Yet, much of the recent discussion revolves mostly around residency programs. The purpose of this review is to synthesize the findings of studies spanning the last 2 decades that examined competency-based assessment methods used in family medicine residency and CPD, and to identify gaps in their current practices. We adopted a modified form of narrative review and searched five online databases and the gray literature for articles published between 2000 and 2020. Data analysis involved mixed methods including quantitative frequency analysis and qualitative thematic analysis. Thirty-seven studies met inclusion criteria. Fourteen were formal evaluation studies that focused on the outcome and impact evaluation of assessment methods. Articles that focused on formative assessment were prevalent. The most common levels of educational outcomes were performance and competence. There were few studies on CBME assessment among practicing family physicians. Thematic analysis of the literature identified several challenges the family medicine educational community faces with CBME assessment. We recommend that those involved in health education systematically evaluate and publish their CBME activities, including assessment-related content and evaluations. The highlighted themes may offer insights into ways in which current CBME assessment practices might be improved to align with efforts to improve health care.

Blended programmatic assessment for competency based curricula.

The uncertainty in all spheres of higher education due to the COVID-19 pandemic has had an unprecedented impact on teaching–learning and assessments in medical colleges across the globe. The conventional ways of assessment are now neither possible nor practical for certifying medical graduates. This has necessitated thoughtful considerations in making adjustments to the assessment system, with most institutions transitioning to online assessments that so far have remained underutilized. Programmatic assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment and at present can be utilized optimally as it ensures the collection of multiple low-stake assessment data which can be aggregated for high-stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance. Though efforts have been made to introduce programmatic assessment in the competency-based undergraduate curriculum, transitioning to online assessment can be a potential opportunity if the basic tenets of programmatic assessment, choice of online assessment tools, strategies, good practices of online assessments and challenges are understood and explored explicitly for designing and implementing online assessments. This paper explores the possibility of introducing online assessment with face-to-face assessment and structuring a blended programmatic assessment in competency-based medical education.

A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education

The coronavirus disease 2019 pandemic resulted in a dramatic change in the Royal College of Physicians and Surgeons of Canada assessment process through elimination of the oral and practical components of the 2020 Anatomical Pathology examination. Our study sought to determine stakeholder opinions and experiences on these changes in the context of the 2019 implementation of competency-based medical education. Surveys were designed for residents and practicing pathologists. In total, 57 residents (estimated response rate 29%) and 185 pathologists (estimated response rate 19%) participated across Canada; 67% of pathologists disagreed with the 2020 Royal College examination changes, compared with 30% for residents (P = <.00001). When asked whether the Royal College examination should be eliminated, 95% of pathologists indicated they would be against this, compared to only 34% of residents (P = <.00001). Perceptions on changes to and importance of different components of assessment in competency-based medical education were similar between pathologists and residents, with participants perceiving assessment practices to have changed fairly little since its implementation, with the exception of more frequent feedback. Analysis of narrative comments identified several common themes around assessment, including the need for objectivity and standardization and the problem of failure-to-fail. However, residents identified numerous elements of their performance that can be assessed only through longitudinal evaluation. Pathologists, on the other hand, tended to view these aspects of performance as laden with bias. Our results will hopefully help guide future innovation in assessment by characterizing different stakeholder perspectives on key issues in medical education.

Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project.

BackgroundCanadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents’ CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents’ needs.MethodsWe utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements.ResultsResident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs.ConclusionsOur design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement. We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels. Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested. There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.