Graphical Item Maps: providing clearer feedback on professional exam performance.

Mark Beaves,Margaret Wu,Nathan Zoanetti,Euan Wallace,Patrick Griffin

doi:10.15694/mep.2018.0000116.1

Abstract

This article was migrated. The article was marked as recommended. Background: Structured feedback is an important component of learning and assessment and is highly valued by candidates. Unfortunately, item specific feedback is generally not feasible for high stakes professional assessments due to the high cost of item development and the need to maintain stable assessment performance characteristics. In a high stakes assessment of fetal surveillance knowledge, we sought to use graphical item mapping to allow informative candidate feedback without compromising the item bank. Methods: We developed Graphical Item Maps (GIMs) to display individual candidate performance in the Royal Australian and New Zealand College of Obstetricians and Gynaecologists (RANZCOG) Fetal Surveillance Education Program (FSEP) multiple-choice question assessment. GIMs use item and person parameter estimates from Item Response Theory (IRT) models to map the interaction between a test taker and assessment tasks of varying difficulty. Results: It is both feasible and relatively simple to provide GIMs for individual candidate feedback. Operational examples are presented from the RANZCOG FSEP assessment. This paper demonstrates how test takers and educators might use GIMs as a form of assessment feedback. Conclusions: Graphical Item Maps are a useful and insightful assessment feedback tool for clinical practitioners partaking in a high stakes professional education and assessment program. They might be usefully employed in similar healthcare professional assessments to inform directed learning.

Highlights

Assessment programs in undergraduate and postgraduate medical education have been developed to meet a variety of purposes, including the maintenance of professional standards, selection of trainees or students, data collection for program evaluation, and informing supplementary instruction .1,2 Each of these objectives has important roles in ensuring that clinicians possess the appropriate knowledge and skills for safe clinical practice
This work was undertaken with the approval of the Fetal Surveillance Education Program (FSEP) Steering Committee of RANZCOG
A Graphical Item Maps (GIMs) is provided to each candidate within two weeks of completion of the assessment, in accordance with recommendations that feedback be detailed and timely[11]

Summary

Introduction

Assessment programs in undergraduate and postgraduate medical education have been developed to meet a variety of purposes, including the maintenance of professional standards, selection of trainees or students, data collection for program evaluation, and informing supplementary instruction .1,2 Each of these objectives has important roles in ensuring that clinicians possess the appropriate knowledge and skills for safe clinical practice. Assessment programs in undergraduate and postgraduate medical education have been developed to meet a variety of purposes, including the maintenance of professional standards, selection of trainees or students, data collection for program evaluation, and informing supplementary instruction .1,2 Each of these objectives has important roles in ensuring that clinicians possess the appropriate knowledge and skills for safe clinical practice. While the purpose of summative selection and professional licensure examinations are appropriately focused on yielding a pass or fail decision; whether based on minimum standards or candidate ranking; it is clear that examinees value detailed feedback[6,7] This is the case when they have been unsuccessful[7,8]. In a high stakes assessment of fetal surveillance knowledge, we sought to use graphical item mapping to allow informative candidate feedback without compromising the item bank

Methods

Results

Conclusion