Opening the black box of selection

Sanne Schreurs,Kitty Cleutjens,Carlos F Collares,Jennifer Cleland,Mirjam G A Oude Egbrink

doi:10.1007/s10459-019-09925-1

Sanne Schreurs, Kitty Cleutjens + Show 3 more

Open Access

https://doi.org/10.1007/s10459-019-09925-1

Copy DOI

Abstract

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the ‘black box of selection’). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing’s validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants’ performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants’ results sheds light into the ‘black box of selection’ and can be used to support the construct validity of selection procedures.

Highlights

The purpose of medical school selection is to recruit students who will perform well at medical school as well as in their future career as a doctor (Bandiera et al 2015)
The aim of this study was to investigate the evidence related to the construct validity of our selection procedure, in order to open the ‘black box of selection’
Our specific focus was on content and internal structure, as these shed the most light into this black box

Summary

Introduction

The purpose of medical school selection is to recruit students who will perform well at medical school as well as in their future career as a doctor (Bandiera et al 2015). Frohlich et al 2017; Patterson et al 2018; Prideaux et al 2011; Schreurs et al 2018; Terregino et al 2015): ‘beginning with the end in mind’ To this purpose, the cognitive and (inter)personal competencies or qualities needed throughout the study program and in future work are integrated as constructs into the selection process (Cleland et al 2012; Patterson et al 2016). To date, there has been no systematic consideration of which constructs (e.g. collaboration or empathy) are assessed in medical school selection procedures, and whether this is in line with what was intended from their outcome-based focus (Christian et al 2010; Wilkinson and Wilkinson 2016). Research on construct validity can help the field of selection for medicine move forward in terms of theory and practice

Objectives

Methods

Results

Conclusion