Competency-based Medical Education Programs Research Articles

Dual purposes by design: exploring alignment between residents' and academic advisors' documents in a longitudinal program.

Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose-to help competence committees make decisions-which may be problematic. Using inductive, thematic analysis we analyzed written comments generated by 21 resident-AA dyads in one large internal medicine program who met over a 2year period to determine what residents write when asked to reflect, how this aligns with what the AAs report, and what changes occur over time (total 109 resident self-reflections and 105 AA reports). Residents commented more on their developing autonomy, progress and improvement than AAs, who commented far more on performance measures. Over time, residents' writing shifted away from intrinsic roles, patient care and improvement towards what AAs focused on, including getting EPAs (entrustable professional activities), studying and exams. For EPAs, the emphasis was on getting sufficient numbers rather than reflecting on what residents were learning. Our findings challenge the practice of dual-purposing documents, by questioning the blurring of formative and summative intent, the structure of forms and their multiple conflicting purposes, and assumptions about the advising relationship over time. Our study suggests a need to re-evaluate how reflective documents are used in CBME programs. Further research should explore whether and how documentation can best be used to support resident growth and development.

Advancing anti-oppression and social justice in healthcare through competency-based medical education (CBME)

Competency-based medical education (CBME) focuses on preparing physicians to improve the health of patients and populations. In the context of ongoing health disparities worldwide, medical educators must implement CBME in ways that advance social justice and anti-oppression. In this article, authors describe how CBME can be implemented to promote equity pedagogy, an approach to education in which curricular design, teaching, assessment strategies, and learning environments support learners from diverse groups to be successful. The five core components of CBME programs – outcomes competency framework, progressive sequencing of competencies, learning experiences tailored to learners’ needs, teaching focused on competencies, and programmatic assessment – enable individualization of learning experiences and teaching and encourage learners to partner with their teachers in driving their learning. These educational approaches appreciate each learner’s background, experiences, and strengths. Using an exemplar case study, the authors illustrate how CBME can afford opportunities to enhance anti-oppression and social justice in medical education and promote each learner’s success in meeting the expected outcomes of training. The authors provide recommendations for individuals and institutions implementing CBME to enact equity pedagogy.

Exploring residents' perceptions of competency-based medical education across Canada: A national survey study.

Background: As competency-based medical education (CBME) is implemented across Canada, little is known about residents' perceptions of this model. This study examined how Canadian residents understand CBME and their lived experiences with implementation. Methods: We administered a survey in 2018 with Likert-type and open-ended questions to 375 residents across Canada, of whom 270 were from traditional programs ("pre-CBME") and 105 were in a CBME program. We used the Mann-Whitney test to examine differences across samples, and analyzed qualitative data thematically. Results: Three themes were identified across both groups: program outcome concerns, changes, and emotional responses. In relation to program concerns, both groups were concerned about the administrative burden, challenges with the assessment process, and feedback quality. Only pre-CBME residents were concerned about faculty engagement and buy-in. In terms of changes, both groups discussed a more formalized assessment process with mixed reactions. Residents in the pre-CBME sample reported greater concerns for faculty time constraints, assessment completion, and quality of learning experiences, whilst those in CBME programs reported being more proactive in their learning and greater selfreflection. Residents expressed strong emotional narrative responses including greater stress and frustration in a CBME environment. Conclusion: Findings demonstrate that residents have mixed feelings and experiences regarding CBME. Their positive experiences align with the aim of developing more self-directed learners. However, the concerns suggest the need to address specific shortcomings to increase buy-in, while the emotional responses associated with CBME may require a cultural shift within residency programs to guard against burnout.

Deidentifying Narrative Assessments to Facilitate Data Sharing in Medical Education.

Narrative assessments are commonly incorporated into competency-based medical education programs. However, efforts to share competency-based medical education assessment data among programs to support the evaluation and improvement of assessment systems have been limited in part because of security concerns. Deidentifying assessment data mitigates these concerns, but deidentifying narrative assessments is time-consuming, resource intensive, and error prone. The authors developed and tested a tool to automate the deidentification of narrative assessments and facilitate their review. The authors met throughout 2021 and 2022 to iteratively design, test, and refine the deidentification algorithm and data review interface. Preliminary testing of the prototype deidentification algorithm was performed using narrative assessments from the University of Saskatchewan emergency medicine program. The algorithm's accuracy was assessed by the authors using the review interface designed for this purpose. Formal testing included 2 rounds of deidentification and review by members of the authorship team. Both the algorithm and data review interface were refined during the testing process. Authors from 3 institutions, including 3 emergency medicine programs, an anesthesia program, and a surgical program, participated in formal testing. In the final round of review, 99.4% of the narrative assessments were fully deidentified (names, nicknames, and pronouns removed). The results were comparable for each institution and specialty. The data review interface was improved with feedback obtained after each round of review and found to be intuitive. This innovation has demonstrated viability evidence of an algorithmic approach to the deidentification of assessment narratives while reinforcing that a small number of errors are likely to persist. Future steps include the refinement of both the algorithm to improve its accuracy and the data review interface to support additional data set formats.

Taiwan’s Experience in Promoting Competency-Based Medical Education and Establishing Entrustable Professional Activities for Radiographers

<p>勝任能力導向醫學教育(Competency-Based Medical Education, CBME)是近年來國際醫學教育趨勢，而可信賴專業活動(Entrustable Professional Activities, EPAs)是將CBME精神具體實行的方法之一。中華民國醫事放射師公會全國聯合會為落實醫事放射職類以勝任能力為導向的訓練，提供教師與時俱進的師資培育活動，遂擬定「醫事放射職類CBME訓練計畫三部曲」，透過三階段的方式推廣並落實能力導向醫學教育：「醫事放射學員的能力進展架構」-臺灣醫事放射職類可信賴專業活動的建置、「為勝任能力量身打造的教學與學習經驗」-CBME師資培訓推廣、「計畫性、系統性的評估設計與能力進展的行程」-醫事放射職類CBME訓練計畫全國執行。本會期望輔導各級教學醫院全面推廣台灣醫事放射職類CBME訓練與落實EPAs職場直接觀察教學，並建置執業模式下的能力進展架構與系統化評估，確保醫事放射師專業能力發展和教育品質提升。</p> <p> </p><p>Competency-based medical education (CBME) has become a trend in global medical education. Entrustable professional activities (EPAs) are one of the means to implement CBME. To provide up-to-date teacher training for implementing radiographers’ CBME, the Taiwan Association of Medical Radiation Technologists has organized a Three-Stage Radiographer CBME program. The three stages are as follows: (1) Competency development: establishing EPAs for radiographers. (2) Competency-oriented teaching and learning experience: promoting CBME teacher training. (3) Planned and systematic competency development and assessment: promoting and implementing the CBME teacher training program nationwide. Through the promotion of CBME, teaching based on direct EPA observations, as well as the establishment of a competency development framework and a systematic assessment model for in-service radiographers, the Association expects to improve the competency development of and education quality for radiographers in Taiwan.</p> <p> </p>

Impact and effectiveness of a mandatory competency-based simulation program for pediatric emergency medicine faculty.

Pediatric emergency medicine physicians struggle to maintain their critical procedural and resuscitation skills. Continuing professional development programs incorporating simulation and competency-based standards may help ensure skill maintenance. Using a logic model framework, we sought to evaluate the effectiveness of a mandatory annual competency-based medical education (CBME) simulation program. The CBME program, evaluated from 2016 to 2018, targeted procedural, point-of-care ultrasound (POCUS) and resuscitation skills. Delivery of educational content included a flipped-classroom website, deliberate practice, mastery-based learning, and stop-pause debriefing. Participants' competence was assessed using a 5-point global rating scale (GRS; 3=competent, 5=mastery). Statistical process control charts were used to measure the effect of the CBME program on team performance during in situ simulations (ISS), measured using the Team Emergency Assessment Measure (TEAM) scale. Faculty completed an online program evaluation survey. Forty physicians and 48 registered nurses completed at least one course over 3 years (physician mean ± SD 2.2 ± 0.92). Physicians achieved competence on 430 of 442 stations (97.3%). Mean ± SD GRS scores for procedural, POCUS, and resuscitation stations were 4.34 ± 0.43, 3.96 ± 0.35, and 4.17 ± 0.27, respectively. ISS TEAM scores for "followed standards and guidelines" improved significantly. No signals of special cause variation emerged for the other 11 TEAM items, indicating skills maintenance. Physicians rated CBME training as highly valuable (mean question scores 4.15-4.85/5). Time commitment and scheduling were identified as barriers to participation. Our mandatory simulation-based CBME program had high completion rates and very low station failures. The program was highly rated and faculty improved or maintained their ISS performance across TEAM scale domains.

A Comparison of Resident-Completed and Preceptor-Completed Formative Workplace-Based Assessments in a Competency-Based Medical Education Program.

In competency-based medical education (CBME), should resident self-assessments be included in the array of evidence upon which summative progress decisions are made? We examined the congruence between self-assessments and preceptor assessments of residents using assessment data collected in a 2-year Canadian family medicine residency program that uses programmatic assessment as part of their approach to CBME. This was a retrospective observational cohort study using a learning analytics approach. The data source was archived formative workplace-based assessment forms (fieldnotes) stored in an online portfolio by family medicine residents and preceptors. Data came from three academic teaching sites over 3 academic years (2015-2016, 2016-2017, 2017-2018), and were analyzed in aggregate using nonparametric tests to evaluate differences in progress levels selected both within and between groups. In aggregate, first-year residents' self-reported progress was consistent with that indicated by preceptors. Progress level rating on fieldnotes improved over training in both groups. Second-year residents tended to assign themselves higher ratings on self-entered assessments compared with those assigned by preceptors; however, the effect sizes associated with these findings were small. Although we found differences in the progress level selected between preceptor-entered and resident-entered fieldnotes, small effect sizes suggest these differences may have little practical significance. Reasonable consistency between resident self-assessments and preceptor assessments suggests that benefits of guided self-assessment (eg, support of self-regulated learning, program efficacy monitoring) remain appealing despite potential risks.

Breeding Ground for Paradoxes: Implementing a Competency-Based Medical Education Fellowship Program

Despite its increasing popularity over the past decade as an improved training model, literature evaluating Competency Based Medical Education (CBME) implementation processes is scarce. We used an action research methodology to observe and analyze the implementation processes of a CBME fellowship program in Neonatology wards in hospitals in Israel. Our data revealed that the challenges emerging during the implementation process can be framed as paradoxes and that CBME entails a new category of paradoxes beyond those previously reported in the literature (i.e., role-paradox). In addition, we developed a CBME implementation model that suggests a dynamic process including three paradox-driven circular courses; 1) a direct loop between CBME implementation and the exposure to paradoxes, 2) an indirect loop between exposure to paradoxes and the formation of a paradox mindset, that in turn further enhances CBME implementation, and 3) an indirect loop between the paradox mindset of the individual and the development of a ward level paradox climate. Propositions regarding the suggested model, and the associated individual and organizational level outcomes of the successful implementation of a CBME program are suggested.

Does direct observation influence the quality of workplace-based assessment documentation?

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p=0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

A Systems Thinking approach for the creation of effective competency-based medical education programs

Successful and sustainable implementation of Competency-based Medical Education (CBME) programs is a significant and daunting challenge facing medical education worldwide. Our manuscript endorses for the first time, Systems Thinking as a concept for transforming and redesigning CBME programs employing the full 7-system elements as advocated by the Biomatrix Systems Theory. The majority of internationally recommended actions and processes for such an endeavor are highlighted, each within its system element. New innovative ideas such as having competency-structured clinical training activities as well as re-writing medical textbooks following a novel competency-based roadmap for their disease monographs etc. are also highlighted. Furthermore, the need for innovative partnerships as well as novel medical rotations that may facilitate the creation of “master clinicians” are also stressed.

How Clinical Supervisors Conceptualize Procedural Entrustment: An Interview-Based Study of Entrustment Decision Making in Endoscopic Training.

Entrustment is central to assessment in competency-based medical education (CBME). To date, little research has addressed how clinical supervisors conceptualize entrustment, including factors they consider in making entrustment decisions. The aim of this study was to characterize supervisors' decision making related to procedural entrustment, using gastrointestinal endoscopy as a test case. Using methods from constructivist grounded theory, the authors interviewed 29 endoscopy supervisors in the United States and Canada across multiple specialties (adult and pediatric gastroenterology, surgery, and family medicine). Semistructured interviews, conducted between April and November 2019, focused on how supervisors conceptualize procedural entrustment, how they make entrustment decisions, and what factors they consider. Transcripts were analyzed using constant comparison to generate an explanatory framework and themes. Three themes were identified from the analysis of interview transcripts: (1) entrustment occurs in varying degrees and fluctuates over time; (2) entrustment decisions can transfer within and across procedural and nonprocedural contexts; (3a) persistent static factors (e.g., supervisor competence, institutional culture, legal considerations) influence entrustment decisions, as do (3b) fluctuating, situated dynamic factors (e.g., trainee skills, patient acuity, time constraints), which tend to change from one training encounter to the next. In the process of making procedural entrustment decisions, clinical supervisors appear to synthesize multiple dynamic factors against a background of static factors, culminating in a decision of whether to entrust. Entrustment decisions appear to fluctuate over time, and assessors may transfer decisions about specific trainees across settings. Understanding which factors supervisors perceive as influencing their decision making has the potential to inform faculty development, as well as competency committees seeking to aggregate faculty judgments about trainee unsupervised practice. Those leading CBME programs may wish to invest in optimizing the observed static factors, such that these foundational factors are tuned to facilitate trainee learning and achievement of entrustment.

Effective competency-based medical education requires learning environments that promote a mastery goal orientation: A narrative review

Purpose Competency-based medical education (CBME) emphasizes the need for learners to be central to their own learning and to take an active role in learning. This approach has a dual aim: to encourage learners to actively engage in their own learning, and to push learners to develop learning strategies that will prepare them for lifelong learning. This review paper proposes a theoretical bridge between CBME and lifelong learning and puts forth the argument that in order for CBME programs to produce the physicians truly needed in our society now and in the future, learning environments must be intentionally designed to foster mastery goal orientations and to support the development of adaptive self-regulated learning skills and behaviours. Materials and methods This narrative literature review incorporated results of searches conducted by a subject librarian in PsycInfo and MedLine. Articles were also identified through reference lists of identified papers to capture older key citations. Analysis of the literature used a constructivist epistemological approach to develop an integrative description of the interaction of achievement goal orientation, self-regulated learning, learning environment, and lifelong learning. Results Findings from achievement goal theory research support the assumption that adoption of a mastery goal orientation facilitates the use of adaptive learning behaviours, such as those described in self-regulated learning theory. Adaptive self-regulated learning strategies, in turn, facilitate effective lifelong learning. The authors offer evidence for how learning environments influence goal orientations and self-regulated learning, and propose that CBME programs intentionally plan for such learning environments. Finally, the authors offer specific suggestions and examples for how learning environments can be designed or adjusted to support adoption of a mastery goal orientation and use of self-regulated learning behaviours and strategies to help support development of adaptive lifelong learners.

Challenges of implementing competency-based medical education postgraduate training programs: the issue of context

Introduction: Competency-based medical education (CBME) is being adopted worldwide. The aim of this paper is to discuss the evolution of CBME and address some perceived challenges in CBME curriculum development and implementation in postgraduate (residency) medical education. Methods: This is an opinion paper based on lived experiences and personal beliefs. The authors have professional training in medical education and are actively involved in CBME research, curriculum development and implementation around the world. Results: The issue of local and system-wide context seems to be of particular importance to individuals, programs, institutions, governing bodies and other stakeholders involved in the development and implementation of CBME programs. CBME has evolved differently at different places, and there are concerns regarding the fidelity of implementation. Stakeholders have been dealing with challenging questions in their CBME journeys, which reflect the varied, complex and dynamic nature of health and education systems. Recently, scholars have established core components of any CBME program. Discussion and conclusions: CBME design should benefit from ground-up strategies that consider the local context. It is essential to approach implementation with a quality improvement lens and pay special attention to the fidelity and integrity of the core CBME components.

From Utopia Through Dystopia: Charting a Course for Learning Analytics in Competency-Based Medical Education.

The transition to the assessment of entrustable professional activities as part of competency-based medical education (CBME) has substantially increased the number of assessments completed on each trainee. Many CBME programs are having difficulty synthesizing the increased amount of assessment data. Learning analytics are a way of addressing this by systematically drawing inferences from large datasets to support trainee learning, faculty development, and program evaluation. Early work in this field has tended to emphasize the significant potential of analytics in medical education. However, concerns have been raised regarding data security, data ownership, validity, and other issues that could transform these dreams into nightmares. In this paper, the authors explore these contrasting perspectives by alternately describing utopian and dystopian futures for learning analytics within CBME. Seeing learning analytics as an important way to maximize the value of CBME assessment data for organizational development, they argue that their implementation should continue within the guidance of an ethical framework.

Continued Supervision for the Common Pediatric Subspecialty Entrustable Professional Activities May Be Needed Following Fellowship Graduation.

Entrustable professional activities (EPAs) are one approach to competency-based medical education (CBME), and 7 EPAs have been developed that address content relevant for all pediatric subspecialties. However, it is not known what level of supervision fellowship program directors (FPDs) deem necessary for graduation. The Subspecialty Pediatrics Investigator Network (SPIN) investigated FPD perceptions of the minimum level of supervision required for a trainee to successfully graduate. In 2017, SPIN surveyed all FPDs of accredited fellowships for 14 subspecialties. For each EPA, the minimum supervision level for graduation (ranging from observation only to unsupervised practice) was set such that no more than 20% of FPDs would accept a lower level. The survey response rate was 82% (660/802). The minimum supervision level for graduation varied across the 7 EPAs from 2 (direct) to 4 (indirect for complex cases), with significant differences between EPAs. The percentage of FPDs desiring a lower minimum supervision level ranged from 3% to 17%. Compared with the 4 nonclinical EPAs (quality improvement, management, lead within the profession, scholarship), the 3 clinical EPAs (consultation, handover, lead a team) had higher minimum supervision graduation levels (P < .001), with less likelihood that an FPD would graduate a learner below their minimum level (P < .001). Consensus among FPDs across all pediatric subspecialties demonstrates the potential need for ongoing supervision for graduates in all 7 common pediatric subspecialty EPAs after fellowship. As CBME programs are implemented, processes and infrastructure to support new graduates are important considerations for leaders.

Twelve tips for learners to succeed in a CBME program

The international movement to competency-based medical education (CBME) marks a major transition in medical education that requires a shift in educators’ and learners’ approach to clinical experiences, the way assessment data are collected and integrated, and in learners’ mindsets. Learners entering a CBME curriculum must actively drive their learning experiences and education goals. For some, this expectation may be a significant change from their previous approach to learning in medicine. This paper highlights 12 tips to help learners succeed within a CBME model.

Development and implementation of a novel, mandatory competency-based medical education simulation program for pediatric emergency medicine faculty

BackgroundMaintaining acute care physician competence is critically important. Current maintenance of certification (MOC) programs has started to incorporate simulation-based education (SBE). However, competency expectations have not been defined. This article describes the development of a mandatory annual SBE, competency-based simulation program for technical and resuscitation skills for pediatric emergency medicine (PEM) physicians.MethodsThe competency-based medical education (CBME) program was introduced in 2016. Procedural skill requirements were based on a needs assessment derived from Royal College PEM training guidelines. Resuscitation scenarios were modified versions of pre-existing in-situ mock codes or critical incident cases. All full-time faculty were required to participate annually in both sessions. Delivery of educational content included a flipped classroom website, deliberate practice, and stop-pause debriefing. All stations required competency checklists and global rating scales.ResultsBetween 2016 and 2018, 40 physicians and 48 registered nurses attended these courses. Overall course evaluations in 2018 were 4.92/5 and 4.93/5. Barriers to implementation include the need for many simulation education experts, time commitment, and clinical scheduling during course events.ConclusionWe have developed a mandatory simulation-based, technical, and resuscitation CBME program for PEM faculty. This simulation-based CBME program could be adapted to other acute care disciplines. Further research is required to determine if these skills are enhanced both in a simulated and real environment and if there is an impact on patient outcomes.

The Science of Effective Group Process: Lessons for Clinical Competency Committees.

The Science of Effective Group Process: Lessons for Clinical Competency Committees.

Managing residents in difficulty within CBME residency educational systems: a scoping review

BackgroundBest practices in managing residents in difficulty (RID) in the era of competency-based medical education (CBME) are not well described. This scoping review aimed to inventory the current literature and identify major themes in the articles that address or employ CBME as part of the identification and remediation of residents in difficulty.MethodsArticles published between 2011 to 2017 were included if they were about postgraduate medical education, RID, and offered information to inform the structure and/or processes of CBME. All three reviewers performed a primary screening, followed by a secondary screening of abstracts of the chosen articles, and then a final comprehensive sub-analysis of the 11 articles identified as using a CBME framework.ResultsOf 165 articles initially identified, 92 qualified for secondary screening; the 63 remaining articles underwent full-text abstracting. Ten themes were identified from the content analysis with “identification of RID” (41%) and “defining and classifying deficiencies” (30%) being the most frequent. In the CBME article sub-analysis, the most frequent themes were: need to identify RID (64%), improving assessment tools (45%), and roles and responsibilities of players involved in remediation (27%). Almost half of the CBME articles were published in 2016–2017.ConclusionsAlthough CBME programs have been implemented for many years, articles have only recently begun specifically addressing RID within a competency framework. Much work is needed to describe the sequenced progression, tailored learning experiences, and competency-focused instruction. Finally, future research should focus on the outcomes of remediation in CBME programs.

The embodiment of practice thresholds: from standardization to stabilization in surgical education.

Surgeons practice their own variations on a procedure. Residents experience shifting thresholds between variations that one surgeon holds firmly as principle and another takes more lightly as preference. Such variability has implications for surgical education, but the impact is not well understood. This is a critical problem to investigate as programs seek to define procedures for competency-based medical education (CBME) and improve learning through deliberate practice. Our study analyzes the emergence of procedural variation in an early-adopter CBME program through a situational analysis of tonsillectomy, a foundation level procedure in this otolaryngology, head and neck surgical program.An earlier phase of the study identified frequent variations (n = 12) on tonsillectomy among co-located surgeons who routinely perform this procedure (n = 6). In the phase reported here we interviewed these surgeons (n = 4) and residents at different stages of training (n = 3) about their experiences of these variations to map the relations of contributing social and material actors. Our results show that even a basic procedure resists standardization. This study contributes a sociomaterial grounded theory of surgical practice as an embodied response to conditions materialized by intra-relations of human and more-than-human actors. Shifting root metaphors about practice in surgical education from standardization to stabilization can help residents achieve stable-for-now embodiments of performance as their practice thresholds continue to emerge.