Summative Decisions Research Articles

Initial Validity Evidence for the American Board of Emergency Medicine Emergency Medical Services Certification Examination

Objectives The American Board of Emergency Medicine (ABEM) Emergency Medical Services Medicine (EMS) subspecialty was approved by the American Board of Medical Specialties on September 23, 2010. Subspecialty certification in EMS was contingent on two key elements—completing Accreditation Council for Graduate Medical Education (ACGME)-accredited EMS training and passing the subspecialty certification examination developed by ABEM. The first EMS certification examination was offered in October 2013. Meaningful certification requires rigorous assessment. In this instance, the EMS certification examination sought to embrace the tenets of validity, reliability, and fairness. For the purposes of this report, the sources of validity evidence were anchored on the EMS core content, the examination development process, and the association between fellowship training and passing the certification examination. Methods We chose to use validity evidence that included: 1) content validity (based on the EMS core content); 2) response processes (test items required intended cognitive processes); 3) internal structure supported by the internal relationships among items; 4) relations to other variables, specifically the association between examination performance and ACGME-accredited fellowship training; and 5) the consequences of testing. Results There is strong content validity evidence for the EMS examination based on the core content and its detailed development process. The core content and supporting job-task analysis was also used to define the examination blueprint. Internal structure support was evidenced by Cronbach’s coefficient alpha, which ranged from 0.82 to 0.92. Physicians who completed ACGME-accredited EMS fellowship training were more likely to pass the EMS certification examination (chi square, p < 0.0001; Cramér’s, V = 0.24). Finally, there were two sources of consequential validity evidence—use of test results to determine certification and use of the resulting certificate. Conclusions There is substantial and varied validity evidence to support the use of the EMS certifying examination in making summative decisions to award certification in EMS. Of note, there was a statistically significant association between ACGME-accredited fellowship training and passing the examination.

Lessons learned spanning 17 years of experience with three consecutive nationwide competency based medical education training plans.

Curricula for postgraduate medical education have transformed since the introduction of competency based medical education (CBME). Postgraduate training plans offer broader training with different competencies and an outcome-based approach, in addition to the medical technical aspects of training. However, CBME also has its challenges. Over the past years, critical views have been shared on the potential drawbacks of CBME, such as assessment burden and conflicts with practicality in the workplace. Recent studies identified a need for a better understanding of how the evolving concept of CBME has been translated to curriculum design and implemented in the practice of postgraduate training. The aim of this study was to describe the development of CBME translations to curriculum design, based on three consecutive postgraduate training programs spanning 17 years. We performed a document analysis of three consecutive Dutch gynecology and obstetrics training plans that were implemented in 2005, 2013, and 2021. We used template analysis to identify changes over time. Over time, CBME-based curriculum design changed in several domains. Assessment changed from a model with a focus on summative decision to one with an emphasis on formative, low-stakes assessments aimed at supporting learning. The training plans evolved in parallel to evolving educational insights, e.g., by placing increasing emphasis on personal development. The curricula focused on a competency-based concept by introducing training modules and personalized authorization based on feedback rather than on a set duration of internships. There was increasing freedom in personalized training trajectories in the training plans, together with increasing trust towards the resident. The way CBME was translated into training plans has evolved in the course of 17 years of experience with CMBE-based education. The main areas of change were the structure of the training plans, which became increasingly open, the degree to which learning outcomes were mandatory or not, and the way these outcomes were assessed.

Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education

ABSTRACT Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

Changing the Assessment Paradigm: Promoting a Growth Mindset Across the Medical Education Continuum

Competency-based medical education (CBME) forms the basis of assessment in health professions education and aims to develop competent health professionals by utilizing direct observations and formative assessments to inform summative decisions on the progress of learners. Learners performing at a certain standard are deemed competent to progress to the next stage.1,2,3 However, the current evaluation system of emphasizing top summative grades creates a dissonance for learners with aspects of CBME and leads to a fixed mindset that prioritizes evaluation over growth.

Competency-Based Medical Training in Anesthesiology: Has It Delivered on the Promise of Better Education?

Competency-Based Medical Training in Anesthesiology: Has It Delivered on the Promise of Better Education?

The Senior Medical Resident's New Role in Assessment in Internal Medicine.

With the introduction of competency-based medical education, senior residents have taken on a new, formalized role of completing assessments of their junior colleagues. However, no prior studies have explored the role of near-peer assessment within the context of entrustable professional activities (EPAs) and competency-based medical education. This study explored internal medicine residents' perceptions of near-peer feedback and assessment in the context of EPAs. Semistructured interviews were conducted from September 2019 to March 2020 with 16 internal medicine residents (8 first-year residents and 8 second- and third-year residents) at the University of Toronto, Toronto, Ontario, Canada. Interviews were conducted and coded iteratively within a constructivist grounded theory approach until sufficiency was reached. Senior residents noted a tension in their dual roles of coach and assessor when completing EPAs. Senior residents managed the relationship with junior residents to not upset the learner and potentially harm the team dynamic, leading to the documentation of often inflated EPA ratings. Junior residents found senior residents to be credible providers of feedback; however, they were reticent to find senior residents credible as assessors. Although EPAs have formalized moments of feedback, senior residents struggled to include constructive feedback comments, all while knowing the assessment decisions may inform the overall summative decision of their peers. As a result, EPA ratings were often inflated. The utility of having senior residents serve as assessors needs to be reexamined because there is concern that this new role has taken away the benefits of having a senior resident act solely as a coach.

Trusting Judgements: A new approach to progression without a numeric scale.

Medical education has undergone significant reforms since the 1990s when there were multiple regulators with patronage, inequalities and unreliability for postgraduate training. Modernising Medical Careers was implemented from 2005 to ensure all training posts and equivalents were assimilated into programmes with a defined curriculum and national selection. However, this brought its own problems, with a more rigid route, much control and training not linked to workforce planning. In addition, Workplace-based assessments (WBAs), which had been intended as formative tools to support personal development, morphed into summative assessments together with the Annual Review of Competence Progression (ARCP) becoming high stakes rather than a light-touch review of progression as had been intended. Alternative models of competence progression were explored, and initial poll results suggested that from the perspective of trainee/students, 46% would prefer judgements of competence to be decided on team observations/comments. Initial discussion was based on the challenge of “compassionate, collaborative, competent, confident doctors” and the role of WBAs where the current assessment tools were not considered to be the key to improving them. The role of the educator and their relationship with trainees was explored in depth including how to enable practitioners to prioritise education and feedback within busy work environments which requires a change in organisational culture. However, some specialties (psychiatry and general practice) have an expectation that dedicated time will be allocated for trainee and trainer to meet together as a recognition of the importance of high-quality training and celebrating excellent educators. If this could be extended to all specialties, it would provide an opportunity for trainers to review and discuss progress with their trainee, thus ensuring that advancement is maintained and any fall off in improvement is identified early and corrected. However, this requires protected time and the service is often under strain. The flip side of this approach is if there is a breakdown in trainer/trainee relationships, degrees of mistrust, this may cause problems. There could also be a possible power imbalance since feedback should be in both directions. Other key themes included ARCP reform to promote a progressive approach with no surprises rather than summative decisions and a clearer recognition of educational growth within appraisal and revalidation. The way forward should include developmental conversations with a greater coaching and mentoring style rather than tick boxing, more near peer conversations and more focus on programmatic assessment aggregated across a variety of sources. Additional tools may be required to support his approach. It is important to celebrate good training and improve the skills of trainers in developing doctors we want for the future which means going beyond what we do now.

The use of EPA assessments in decision-making: Do supervision ratings correlate with other measures of clinical performance?

Background Entrustable professional activities (EPAs) have been introduced as a framework for teaching and assessment in competency-based educational programs. With growing use, has come a call to examine the validity of EPA assessments. We sought to explore the correlation of EPA assessments with other clinical performance measures to support use of supervision ratings in decisions about medical students’ curricular progression. Methods Spearman rank coefficients were used to determine correlation of supervision ratings from EPA assessments with scores on clerkship evaluations and performance on an end-of-clerkship-year Objective Structured Clinical Examination (CPX). Results Both overall clinical evaluation items score (rho 0.40; n = 166) and CPX patient encounter domain score (rho 0.31; n = 149) showed significant correlation with students’ overall mean EPA supervision rating during the clerkship year. There was significant correlation between mean supervision rating for EPA assessments of history, exam, note, and oral presentation skills with scores for these skills on clerkship evaluations; less so on the CPX. Conclusions Correlation of EPA supervision ratings with commonly used clinical performance measures offers support for their use in undergraduate medical education. Data supporting the validity of EPA assessments promotes stakeholders’ acceptance of their use in summative decisions about students’ readiness for increased patient care responsibility.

Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade.

Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade.

Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane.

Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system. To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time. The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps. Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters.

In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills.

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients.

On the validity of summative entrustment decisions

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Oregon Health & Science University School of Medicine.

Oregon Health & Science University School of Medicine.

Beyond summative decision making: Illuminating the broader roles of competence committees.

Although competence committees (CCs) are most often conceptualised as group decision-making bodies, policy documents forward a more ambitious vision and outline several additional roles for CCs that support the continuous quality improvement of education programmes and promote formative feedback. However, whether these functions are encompassed in the everyday work of CCs is currently unknown. This constructivist grounded theory study elaborates the range of roles taken up by CCs and illuminates the processes through which these roles are actualised. Two investigators observed 27 CC meetings (>80hours) across seven diverse postgraduate programmes at a single Canadian institution. Following each observation, a semi-structured interview was conducted with one CC member. Data collection and analysis unfolded iteratively. In this study, CCs did much more than make summative decisions about progression and entrustment; they identified a variety of problems that arose at the levels of both the individual learner and the programme, and leveraged their knowledge of assessment data to develop solutions. The problem-solving work of CCs was enabled by the in-depth data review, synthesis and analysis work that occurred before scheduled meetings, outside protected academic or administrative time. Although this work often appeared invisible to those outside the committee, the insights gleaned from data review provided committee members with essential information about how their programme of assessment was unfolding in practice. Competence committees may be an untapped resource that can support assessment for learning, local evaluation of competency-based medical education (CBME) implementation and continuous quality improvement for programmes of assessment. However, the ability of CCs to engage in this work is fragile and is currently sustained by the willingness of faculty members to devote their time and energy to it. The resourcing of CCs may have profound implications for translation of the theory of programmatic assessment and CBME into practice.

Using the Family Medicine Certification Longitudinal Assessment to Make Summative Decisions

On January 4, 2019, the American Board of Family Medicine (ABFM) launched the Family Medicine Certification Longitudinal Assessment[1][1] (FMCLA) pilot. Our hope is that FMCLA will provide both summative feedback—assessing whether a candidate has the cognitive expertise to be a board-certified

Exploring Validity Evidence Associated With Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: A Systematic Review.

To collect and examine-using an argument-based validity approach-validity evidence of questionnaire-based tools used to assess physicians' clinical, teaching, and research performance. In October 2016, the authors conducted a systematic search of the literature seeking articles about questionnaire-based tools for assessing physicians' professional performance published from inception to October 2016. They included studies reporting on the validity evidence of tools used to assess physicians' clinical, teaching, and research performance. Using Kane's validity framework, they conducted data extraction based on four inferences in the validity argument: scoring, generalization, extrapolation, and implications. They included 46 articles on 15 tools assessing clinical performance and 72 articles on 38 tools assessing teaching performance. They found no studies on research performance tools. Only 12 of the tools (23%) gathered evidence on all four components of Kane's validity argument. Validity evidence focused mostly on generalization and extrapolation inferences. Scoring evidence showed mixed results. Evidence on implications was generally missing. Based on the argument-based approach to validity, not all questionnaire-based tools seem to support their intended use. Evidence concerning implications of questionnaire-based tools is mostly lacking, thus weakening the argument to use these tools for formative and, especially, for summative assessments of physicians' clinical and teaching performance. More research on implications is needed to strengthen the argument and to provide support for decisions based on these tools, particularly for high-stakes, summative decisions. To meaningfully assess academic physicians in their tripartite role as doctor, teacher, and researcher, additional assessment tools are needed.

Authentic learning tasks and assessment in Physical Education teacher education

Traditionally, physical education teacher education programs (PETE) prepare pre-service PE teachers through various separate courses and by providing authentic teaching-learning situations where learned knowledge and skills can be applied. However, within this fragmented educational approach a lack of transfer of what is learned in university and what is asked during authentic teaching situations can be observed. In order to better prepare students to become effective PE teachers, Fontys University completely redesigned their PETE program by placing authentic learning tasks at the center of the curriculum. As a consequence, in our assessment program we cover various levels of Miller’s pyramid and make high stakes (summative) decisions at the end of a unit only when preceded by and based on multiple low-stake assessment data points. The low stake data-points inform students on their development and also inform teacher educators on the effectiveness of the educational program and/or their scaffolding or guidance.

A Narrative Review of the Evidence Supporting Factors Used by Residency Program Directors to Select Applicants for Interviews.

Residency applicants feel increasing pressure to maximize their chances of successfully matching into the program of their choice, and are applying to more programs than ever before. In this narrative review, we examined the most common and highly rated factors used to select applicants for interviews. We also examined the literature surrounding those factors to illuminate the advantages and disadvantages of using them as differentiating elements in interviewee selection. Using the 2018 NRMP Program Director Survey as a framework, we examined the last 10 years of literature to ascertain how residency directors are using these common factors to grant residency interviews, and whether these factors are predictive of success in residency. Residency program directors identified 12 factors that contribute substantially to the decision to invite applicants for interviews. Although United States Medical Licensing Examination (USMLE) Step 1 is often used as a comparative factor, most studies do not demonstrate its predictive value for resident performance, except in the case of test failure. We also found that structured letters of recommendation from within a specialty carry increased benefit when compared with generic letters. Failing USMLE Step 1 or 2 and unprofessional behavior predicted lower performance in residency. We found that the evidence basis for the factors most commonly used by residency directors is decidedly mixed in terms of predicting success in residency and beyond. Given these limitations, program directors should be skeptical of making summative decisions based on any one factor.

Judging residents\u2019 performance: a qualitative study using grounded theory

BackgroundAlthough program directors judge residents’ performance for summative decisions, little is known about how they do this. This study examined what information program directors use and how they value this information in making a judgment of residents’ performance and what residents think of this process.MethodsSixteen semi-structured interviews were held with residents and program directors from different hospitals in the Netherlands in 2015–2016. Participants were recruited from internal medicine, surgery and radiology. Transcripts were analysed using grounded theory methodology. Concepts and themes were identified by iterative constant comparison.ResultsWhen approaching semi-annual meetings with residents, program directors report primarily gathering information from the following: assessment tools, faculty members and from their own experience with residents. They put more value on faculty’s comments during meetings and in the corridors than on feedback provided in the assessment tools. They are influenced by their own beliefs about learning and education in valuing feedback. Residents are aware that faculty members discuss their performance in meetings, but they believe the assessment tools provide the most important proof to demonstrate their clinical competency.ConclusionsResidents think that feedback in the assessment tools is the most important proof to demonstrate their performance, whereas program directors scarcely use this feedback to form a judgment about residents’ performance. They rely heavily on remarks of faculty in meetings instead. Therefore, residents’ performance may be better judged in group meetings that are organised to enhance optimal information sharing and decision making about residents’ performance.