Clinical Competency Committee Members Research Articles

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard.

Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations. Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment. Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants. We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.

Group Decision Making in Competency-Based Medical Education - Clinical Competency Committee

<p>臨床能力委員會(Clinical Competency Committee, CCC)在勝任能力導向醫學教育的總結性評量中扮演重要的角色。委員會根據以評量藍圖執行的職場觀察評量結果，進行系統性的討論，對學員的能力進展做出公正可靠的判斷，回饋給相關利害關係人，並協助學員擬定個人化之補強訓練計畫。進行會議前應有共識會議、師資培育課程、及學員能力進展資料閱讀，事前的學員自我評量能夠促進學員與教師對學員能力進展的認知，並促進學員對於教師回饋的接納理解程度；會議中須建立結構化的決策過程，最終獲得客觀且可信的決策成果；會議後應向學員提供回饋，並協助制訂補強學習計畫或提供訓練協助規劃。藉由完善的CCC程序，可提升臨床醫事人員的訓練品質、教師教學水準、醫療服務品質及專業認同。</p> <p> </p><p>Clinical Competency Committee (CCC) plays a key role in the summative assessment of competency-based medical education (CBME). Specifically, CCC conducts systematic discussions according to the results of workplace observation and assessment derived from assessment blueprints to provide an accountable and reliable judgment on the progression of trainees’ competencies. CCC then informs assessment results to stakeholders and assists trainees in developing personalized remedial learning plans. CCC members should conduct a consensus meeting, complete faculty training courses, and review trainees’ competency development data before calling a CCC meeting . Student self-assessment should also be completed before the CCC meeting to improve trainees’ and CCC members’ understanding of the progress of their competency development and to ensure that trainees agree with and accept teachers’ feedback. During the CCC meeting, a structured decision-making procedure must be constructed for objective and credible decision-making. After the meeting, feedback should be provided to students to assist them in developing remedial learning plans or training assistance plans. A comprehensive group decision-making procedure should be carried out in CCC to improve the quality of clinical staff training, teaching, and medical services as well as the professional identity of clinical staff.</p> <p> </p>

Milestones Next Steps: Results of a Feedback Questionnaire

Milestones Next Steps: Results of a Feedback Questionnaire

Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison.

Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment. EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members. Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively. The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.

Milestones 2.0: An advancement in competency-based assessment for dermatology.

In 2013, Next Accreditation System and Milestones became the competency-based assessment framework required for all specialties accredited by the Accreditation Council for Graduate Medical Education. Dermatology residency programs implemented Milestones 1.0 in the 2013-2014 academic year. The Accreditation Council for Graduate Medical Education committed to review and revise Milestones 1.0 within 3 to 5 years. Subsequently, feedback from key stakeholders influenced the goals for revision, including reducing complexity, enhancing community engagement, and providing additional resources for programs. In 2019, the Dermatology Milestones 2.0 work group streamlined the specialty-specific patient care and medical knowledge subcompetencies. The harmonized milestones allowed for greater uniformity across specialties in systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication and skills. The work group developed a supplemental guide with specialty-specific context to help program directors, clinical competency committee members, and other faculty understand individual milestones. Dermatology Milestones 2.0 reduces the number of subcompetencies from 28 to 21. Milestones 2.0 represents an advancement in competency-based assessment for dermatology. The first year of reporting for Dermatology Milestones 2.0 is 2021.

ACGME Milestones in the Real World: A Qualitative Study Exploring Response Process Evidence.

Since the Accreditation Council for Graduate Medical Education (ACGME) introduced the Milestones in 2013, the body of validity evidence supporting their use has grown, but there is a gap with regard to response process. The purpose of this study is to qualitatively explore validity evidence pertaining to the response process of individual Clinical Competency Committee (CCC) members when assigning Milestone ratings to a resident. Using a constructivist paradigm, we conducted a thematic analysis of semi-structured interviews with 8 Transitional Year (TY) CCC members from 4 programs immediately following a CCC meeting between November and December 2020. Participants were queried about their response process in their application of Milestone assessment. Analysis was iterative, including coding, constant comparison, and theming. Participant interviews identified an absence of formal training and a perception that Milestones are a tool for resident assessment without recognizing their role in program evaluation. In describing their thought process, participants reported comparing averaged assessment data to peers and time in training to generate Milestone ratings. Meaningful narrative comments, when available, differentiated resident performance from peers. When assessment data were absent, participants assumed an average performance. Our study found that the response process used by TY CCC members was not always consistent with the dual purpose of the Milestones to improve educational outcomes at the levels of residents and the program.

Comparing the Quality of Narrative Comments by Rotation Setting

To investigate the effect of rotation setting on trainee-directed narrative comments within a Canadian General Surgery Residency Program. The primary outcome was to use the McMaster Narrative Comment Rating Scale (MNCRS) to evaluate the quality of narrative comments across five domains: valence of language, degree of correction versus reinforcement, specificity, actionability and overall usefulness. As distributed medical education in the postgraduate training context becomes more prevalent, delineating differences in feedback between various sites will be imperative, as it may affect how narrative comments are interpreted by clinical competency committee (CCC) members. A retrospective analysis of 2,469 assessments obtained between July 1, 2014 and May 5, 2019 from the General Surgery Residency Program at the University of British Columbia (UBC) was conducted. Narrative comments were rated using the McMaster Narrative Comment Rating Scale (MNCRS), a validated instrument for evaluating the quality of narrative comments. A repeated measures Analysis of Variance (ANOVA) was conducted to explore the impact of rotation setting, academic, urban tertiary, distributed urban, and distributed rural on the quality of narrative feedback. Overall, the quality of the narrative comments varied substantially between and within rotation settings. Academic sites tended to provide more actionable comments (p = 0.01) and more corrective versus reinforcing comments, compared with other sites (p's < 0.01). Comments produced by the urban tertiary rotation setting were consistently lower in quality across all scale categories compared with other settings (p's < 0.01). The type of rotation setting has a significant effect on the quality of faculty feedback for trainees. Faculty development on the provision of feedback is necessary, regardless of rotation setting, and should appropriately combine rotation-specific needs and overarching program goals to ensure trainees and clinical competence committees receive high quality narrative.

Faculty Development for Milestones and Clinical Competency Committees.

Faculty Development for Milestones and Clinical Competency Committees.

A Validity Framework for Effective Analysis and Interpretation of Milestones Data.

A Validity Framework for Effective Analysis and Interpretation of Milestones Data.

Creating a High-Quality Faculty Orientation and Ongoing Member Development Curriculum for the Clinical Competency Committee.

Creating a High-Quality Faculty Orientation and Ongoing Member Development Curriculum for the Clinical Competency Committee.

Challenges and Solutions in Running Effective Clinical Competency Committees in the International Context.

Challenges and Solutions in Running Effective Clinical Competency Committees in the International Context.

Navigating Academic Law in Competency Decisions.

Navigating Academic Law in Competency Decisions.

The Science of Effective Group Process: Lessons for Clinical Competency Committees.

The Science of Effective Group Process: Lessons for Clinical Competency Committees.

Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended.

Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention. A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively. After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments. Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.

2020 Virtual International Conference on Residency Education.

Introduction: Recent assessment innovations have included the development of resident-sensitive quality measures (RSQMs), which are characterized by: 1) their importance to care quality, and 2) likelihood they are performed by residents. This study explored how clinical competency committee (CCC) members interpret, use, and prioritize RSQM data added to their usual review processes.Methods: In this constructivist grounded theory study, 19 members of the Cincinnati Children’s pediatric residency CCC were purposively and theoretically sampled. Participants were provided a resident assessment portfolio comprised of performance ratings and narrative comments for five rotations, along with RSQM data for one of these rotations. They were asked to make a decision about the resident’s ability to care for patients presenting with common, acute problems (a general pediatric entrustable professional activity). Data collection consisted of: 1) observation and think aloud while participants reviewed performance data, and 2) semi-structured interviews to probe reviews.Results: Five dimensions for how participants view and use RSQMs were identified: 1) Ability to orient to RSQMs: confusing to self-explanatory, 2) Propensity to use RSQMs: reluctant to enthusiastic, 3) RSQM interpretation: requires contextualization to self-evident, 4) RSQMs for assessment decisions: not sticky to sticky, and 5) Expectations for residents: potentially unfair to fair to use RSQMs. The interactions among dimensions generated three RSQM data user profiles, with the first two being most common: eager incorporation, willing incorporation, and disinclined incorporation.Conclusion: Most participants used RSQMs to varying extents, demonstrating willingness to include them as resident assessment data for CCC review.

Feasibility of an Entrustable Professional Activity for Pathology Resident Frozen Section Training

Entrustable professional activities are an intuitive form of workplace-based assessment that can support competency-based medical education. Many entrustable professional activities have been written and published, but few studies describe the feasibility or implementation of entrustable professional activities in graduate medical education. The frozen section entrustable professional activit was introduced into the pathology residency training at the University of Vermont for postgraduate year 1 at the start of their training in frozen section. The feasibility of the entrustable professional activit was evaluated based on 3 criteria: (a) utilization, (b) support of frozen section training, and (c) generating data to support entrustment decision about residents’ readiness to take call. The entrustable professional activit was well utilized and satisfactory to residents, faculty, pathologists’ assistants, and Clinical Competency Committee members. Most members of the Clinical Competency Committee agreed they had sufficient data and noted higher confidence in assessing resident readiness to take call with the addition of entrustable professional activit to the residents’ assessment portfolio. Residents did not endorse it helped them prepare for call; however, the interruption to frozen section training due to the COVID-19 pandemic was a significant contributing factor. The frozen section entrustable professional activit is a feasible addition to pathology resident training based on utilization, support of training, and generation of data to support entrustment decisions for graduated responsibilities. The implementation and integration of the entrustable professional activit into pathology training at our institution is described with discussion of adjustments for future use.

Developing a New Set of ACGME Milestones for Child Neurology Residency

BackgroundThe Educational Milestones developed by the Accreditation Council for Graduate Medical Education (ACGME) are a construct used to evaluate the development of core competencies during residency and fellowship training. The milestones were developed to create a framework for professional development during graduate medical education. The first iteration of milestones for the child neurology residency was implemented in 2015. In the years that followed, the ACGME received and reviewed feedback about the milestones and set out to revise them. MethodsA committee was assembled to review the original milestones and develop a new set of milestones. The group was also encouraged to not only consider the child neurology residency graduate of today but also the graduate of tomorrow, taking into account growing fields such as genetics and technology. ResultsA diverse group of 12 individuals, including 10 child neurologists (all of whom were current or previous program directors or associate program directors), one child neurology resident, and one non-physician program coordinator, were recruited from programs of varying size across the country. ConclusionsThe committee developed a revision to the child neurology milestones. All changes made were with a focus on how the milestones can be useful to trainees, program directors, and clinical competency committee members. Implementation and further feedback should help guide future revisions. These changes should help trainees, clinical competency committee members, and program directors find more meaning from their use.

The role of previously undocumented data in the assessment of medical trainees in clinical competency committees.

IntroductionThe clinical competency committee (CCC) comprises a group of clinical faculty tasked with assessing a medical trainee’s progress from multiple data sources. The use of previously undocumented data, or PUD, during CCC deliberations remains controversial. This study explored the use of previously undocumented data in conjunction with documented data in creating a meaningful assessment in a CCC.MethodsAn instrumental case study of a CCC that uses previously undocumented data was conducted. A single CCC meeting was observed, followed by semi-structured individual interviews with all CCC members (n = 7). Meeting and interview transcripts were analyzed iteratively.ResultsDocumented data were perceived as limited by inaccurate or superficial data, but sometimes served as a starting point for invoking previously undocumented data. Previously undocumented data were introduced as summary impressions, contextualizing factors, personal anecdotes and, rarely, hearsay. The purpose was to raise a potential issue for discussion, enhance and elaborate an impression, or counter an impression. Various mechanisms allowed for the responsible use of previously undocumented data: embedding these data within a structured format; sharing relevant information without commenting beyond one’s scope of experience; clarifying allowable disclosure of personal contextual factors with the trainee pre-meeting; excluding previously undocumented data not widely agreed upon in decision-making; and expecting these data to have been provided as direct feedback to trainees pre-meeting.DiscussionPreviously undocumented data appear to play a vital part of the group conversation in a CCC to create meaningful, developmentally focused trainee assessments that cannot be achieved by documented data alone. Consideration should be given to ensuring the thoughtful incorporation of previously undocumented data as an essential part of the CCC assessment process.Electronic supplementary materialThe online version of this article (10.1007/s40037-020-00624-x) contains supplementary material, which is available to authorized users.

What resources do clinical competency committees (CCCs) require to do their work? A pilot study comparing CCCs across specialties

Purpose Although a growing literature describes how clinical competency committees (CCCs) make decisions about trainees’ clinical performance, little is known about the resources these committees need to perform their work. In this pilot study, we examined key characteristics of CCC processes across generalist and surgical specialties. This study intended to clarify topic areas for further investigation. Methods A cross-sectional web-based survey of CCC chairpersons at two institutions was conducted in 2017. Survey items were designed to describe not only CCC work, including types of assessment data used and time spent discussing learners, but also resource needs such as faculty development, institutional support, and protected time for members. Results The response rate was 59% (16/27). Only 44% offered faculty development to members. There was strong support for the institution to assist with faculty development for CCC members (81.25%), workshops for program coordinators (87.5%) and optimizing residency management software to organize assessment data (81.25%). Most respondents did not receive protected time for their committee work (93.75%). Conclusions Further studies are needed to elucidate whether CCC work varies across specialties and the associated committee resource needs. There may be opportunities for institutions to assist CCCs with resources across specialties.

72. USE OF RESIDENT-SENSITIVE QUALITY MEASURE DATA TO MAKING AN ENTRUSTMENT DECISION IN A PEDIATRIC RESIDENCY

Introduction Recent assessment innovations have included the development of resident-sensitive quality measures (RSQMs), which are characterized by: 1) their importance to care quality, and 2) likelihood they are performed by residents. This study explored how clinical competency committee (CCC) members interpret, use, and prioritize RSQM data added to their usual review processes. Methods In this constructivist grounded theory study, 19 members of the Cincinnati Children's pediatric residency CCC were purposively and theoretically sampled. Participants were provided a resident assessment portfolio comprised of performance ratings and narrative comments for five rotations, along with RSQM data for one of these rotations. They were asked to make a decision about the resident's ability to care for patients presenting with common, acute problems (a general pediatric entrustable professional activity). Data collection consisted of 1) observation and think aloud while participants reviewed performance data, and 2) semi-structured interviews to probe reviews. Results Five dimensions for how participants view and use RSQMs were identified: 1) Ability to orient to RSQMs: confusing to self-explanatory, 2) Propensity to use RSQMs: reluctant to enthusiastic, 3) RSQM interpretation: requires contextualization to self-evident, 4) RSQMs for assessment decisions: not sticky to sticky, and 5) Expectations for residents: potentially unfair to fair to use RSQMs. The interactions among dimensions generated three RSQM data user profiles, with the first two being most common: eager incorporation, willing incorporation, and disinclined incorporation. Conclusion Most participants used RSQMs to varying extents, demonstrating willingness to include them as resident assessment data for CCC review.