A practical approach to programmatic assessment design

A A Timmerman,J Dijkstra

doi:10.1007/s10459-017-9756-3

Abstract

Assessment of complex tasks integrating several competencies calls for a programmatic design approach. As single instruments do not provide the information required to reach a robust judgment of integral performance, 73 guidelines for programmatic assessment design were developed. When simultaneously applying these interrelated guidelines, it is challenging to keep a clear overview of all assessment activities. The goal of this study was to provide practical support for applying a programmatic approach to assessment design, not bound to any specific educational paradigm. The guidelines were first applied in a postgraduate medical training setting, and a process analysis was conducted. This resulted in the identification of four steps for programmatic assessment design: evaluation, contextualisation, prioritisation and justification. Firstly, the (re)design process starts with sufficiently detailing the assessment environment and formulating the principal purpose. Key stakeholders with sufficient (assessment) expertise need to be involved in the analysis of strengths and weaknesses and identification of developmental needs. Central governance is essential to balance efforts and stakes with the principal purpose and decide on prioritisation of design decisions and selection of relevant guidelines. Finally, justification of assessment design decisions, quality assurance and external accountability close the loop, to ensure sound underpinning and continuous improvement of the assessment programme.

Highlights

Recent discourse about what constitutes good assessment of performance has led experts to state, that assessment requires a programmatic approach consisting of a deliberate and arranged set of assessment activities (Knight 2000; Lew et al 2002; Schuwirth et al 2002; Schuwirth and van der Vleuten 2011a; van der Vleuten et al 2012)
We reflected on the phases and actions performed in the case study: from the evaluation of the Residency Training Programme in General Practice (RTPGP) assessment programme to the formulation of the implementation plan
Our focus was on identifying the core elements that have characterised the evaluation of the RTPGP assessment programme

Summary

Introduction

Recent discourse about what constitutes good assessment of performance has led experts to state, that assessment requires a programmatic approach consisting of a deliberate and arranged set of assessment activities (Knight 2000; Lew et al 2002; Schuwirth et al 2002; Schuwirth and van der Vleuten 2011a; van der Vleuten et al 2012). Research on assessment of (medical) competence focuses increasingly on the top (‘does’) level of ‘Millers pyramid’, mainly in authentic tasks (Miller 1990; Vleuten et al 2010). The promises and resulting advantages of a programmatic approach are summarised in Table 1 (Dijkstra et al 2010; van der Vleuten et al 2012)

Objectives

Methods

Results

Discussion

Conclusion