A meta-analytic perspective on the valid use of subjective human judgement to make medical school admission decisions

Marie O’Shea,Paul Murphy,Catherine Bruen,Teresa Pawlikowska,Clare Kreiter

doi:10.1080/10872981.2018.1522225

Marie O’Shea, Paul Murphy + Show 3 more

Open Access

https://doi.org/10.1080/10872981.2018.1522225

Copy DOI

Abstract

ABSTRACTWhile medical educators appear to believe that admission to the medical school should be governed, at least in part, by human judgement, there has been no systematic presentation of evidence suggesting it improves selection. From a fair testing perspective, legal, ethical, and psychometric considerations, all dictate that the scientific evidence regarding human judgement in selection should be given consideration. To investigate the validity of using human judgements in admissions, multi-disciplinary meta-analytic research evidence from the wider literature is combined with studies from within medical education to provide evidence regarding the fairness and validity of using interviews and holistic review in medical school admissions. Fourteen studies, 6 of which are meta-analytic studies that summarized 292 individual studies, were included in the final review. Within these studies, a total of 33 studies evaluated the reliability of the traditional interview. These studies reveal that the interview has low to moderate reliability (~.42) which significantly limits its validity. This is confirmed by over 100 studies examining interview validity which collectively show interview scores to be moderately correlated with important outcome variables (corrected value ~.29). Meta-analyses of over 150 studies demonstrate that mechanical/formula-based selection decisions produce better results than decisions made with holistic/clinical methods (human judgement). Three conclusions regarding the use of interviews and holistic review are provided by these meta-analyses. First, it is clear that the traditional interview has low reliability and that this significantly limits its validity. Second, the reliable variance from interview scores appears moderately predictive of outcomes that are relevant to consider in medical school admission. And third, the use of holistic review as a method of incorporating human judgement is not a valid alternative to mechanical/statistical approaches as the evidence clearly indicates that mechanistic methods are more predictive, reliable, cost efficient, and transparent.

Highlights

The most recent survey of admission practices at North American medical schools suggests that admission programmes make substantial use of subjective judgement/assessment [1]
While it is well-known that interviews and subjective holistic review of an applicant plays an influential role, how these subjective judgements impact the integrity of the admission decision is not well understood
Given that selection is governed to a significant degree by the subjective assessment of an applicant, it seems prudent to ask how this impacts the validity of our admission decisions

Summary

Introduction

The most recent survey of admission practices at North American medical schools suggests that admission programmes make substantial use of subjective judgement/assessment [1]. Despite the widespread use of, and support for, subjective human judgement/assessment, there is no evidence-based consensus on how it can best be employed in medical school admission. To utilize human judgement in a way that preserves the predictive value of the measures employed, it is critically important that medical educators understand the formal requirements for the fair and valid use of assessment in high-stakes selection.

Results

Conclusion