Designing Selection Systems for Medicine: The importance of balancing predictive and political validity in high‐stakes selection contexts

Abstract

Selection into medical education and training is a high‐stakes process. A key unanswered issue is the effectiveness of measuring noncognitive predictors via both low‐fidelity and high‐fidelity selection approaches in this high‐stakes context. We review studies investigating the effectiveness of multiple selection instruments in terms of predictive validity, incremental validity, and applicant reactions in both entry‐level and advanced‐level medical selection. Our results show that the situational judgment test (SJT) is the best single predictor of performance, operationalized in multiple ways. In addition, the low‐fidelity SJT has incremental predictive power over cognitively oriented tests, and high‐fidelity assessment center (AC) exercises add incremental validity over the low‐fidelity (and less costly) selection methods. Concerning applicant reactions, results show that overall, the selection system is positively received. However, the method with the highest predictive validity – the SJT – received comparatively lower face validity ratings which may present a ‘justice dilemma’ for employers. Furthermore, various other stakeholders have a political interest in the selection methods used (e.g., government, the regulators and trade unions).