To the Editor: For Equity in Assessment: A Comment on Bias in the Emergency Medicine Standardized Letter of Evaluation.

Abstract

I read the combined works of Kukulski et al and Alvarez et al with great interest.1,2 As a current medical education fellow with a focus on diversity and inclusion, I responded to their results and conclusions with the knowing nod and weary sigh of any minority-tax-paying junior faculty member who knows the bias is there even if it hasn't been fully elucidated. I applaud both groups of authors for presenting such important work with concise and measured discussion. I also fully appreciate the warning that these initial findings do not necessarily tell us a full story and we have yet to uncover whether this bias translates into disproportionate success at the next level for these students.However, I wonder how we can continue a productive conversation around the standardized letter of evaluation (SLOE) and move toward better equity. It is true that with transition to a pass/fail United States Medical Licensing Examination (USMLE) Step 1 we will see greater emphasis on the SLOE and other instruments like it across specialties. Our field is in a unique position to lead on this front given our time and experience with the tool. Years of data and study suggest that the evidence for response process validity within the SLOE is mixed at best. We traditionally have overinflated and misused norm-referenced rankings, poorly adhered to instructions, and have wide variations in grading practices.3 Additionally, most clinical shift assessments do not easily translate to the SLOE questions, leaving most writers to use gestalt to rate candidates on qualifications and competencies.4 The National Clinical Assessment Tool in Emergency Medicine is an admirable attempt at transitioning to competency-based assessment and early work suggests that it could be a feasible, standardized way to address some of these shortcomings.5Unfortunately, it's taken more than 20 years to identify and describe these characteristics of the SLOE and will likely take several years more before we can expect robust and practice-changing data on new tools. In the meantime, it remains our responsibility to promote utmost equity and carefully reexamine and refine an assessment with stakes as high as the SLOE. National leaders should provide new guidance aimed at mitigating bias which might include published guidelines and expectations for group SLOE committees, standardization of clerkship evaluations themselves, and changes to the current structure of the SLOE to remove peer comparisons.By taking proactive action, we may also exert influence on other fields modeling their own evaluation letters after ours and better promote equity for students applying to all specialties. Many of us have long suspected systemic inequality in our clinical assessment tools, and a failure to act in light of evidence of this inequality is the same as promoting the injustice itself. The looming sea change of a pass/fail USMLE Step 1 might well accelerate and exacerbate the potential effects of this bias. Adapting too slowly could have consequences for an entire generation of residency applicants. The time for intense reflection and revision is now.