Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents\u2019 Test Results After Transitioning from Formative Progress Testing

D. R. Rutgers,A. F. van Raamt,W. van Lankeren,C. Haaring,C. L. J. J. Kruitwagen,O. ten Cate,J. P. J. van Schaik

doi:10.1007/s40670-020-00977-2

Abstract

IntroductionEducational effects of transitioning from formative to summative progress testing are unclear. Our purpose was to investigate whether such transitioning in radiology residency is associated with a change in progress test results.MethodsWe investigated a national cohort of radiology residents (N > 300) who were semi-annually assessed through a mandatory progress test. Until 2014, this test was purely formative for all residents, but in 2014/2015, it was transitioned (as part of a national radiology residency program revision) to include a summative pass requirement for new residents. In 7 posttransitioning tests in 2015–2019, including summatively and formatively tested residents who followed the revised and pre-transitioning residency program, respectively, we assessed residents’ relative test scores and percentage of residents that reached pass standards.ResultsDue to our educational setting, most posttransitioning tests had no residents in the summative condition in postgraduate year 4–5, nor residents in the formative condition in year 0.5–2. Across the 7 tests, relative test scores in postgraduate year 1–3 of the summative resident group and year 3.5–4.5 of the formative group differed significantly (p < 0.01 and p < 0.05, respectively, Kruskal-Wallis test). However, scores fluctuated without consistent time trends and without consistent differences between both resident groups. Percentage of residents reaching the pass standard did not differ significantly across tests or between groups.DiscussionTransitioning from formative to summative progress testing was associated with overall steady test results of the whole resident group in 4 post-transitioning years. We do not exclude that transitioning may have positive educational effects for resident subgroups.

Highlights

Educational effects of transitioning from formative to summative progress testing are unclear
Such a knowledge decrease is a genuine possibility since some research in undergraduates has suggested that education programs with summative progress testing may have lower participants’ test scores than programs with formative progress testing [23]. It is not known from the literature how residents’ knowledge levels develop after shifting to summative progress testing. This gap encouraged us to conduct the present study in which we investigated whether transitioning from formative to summative progress testing in a competency-based radiology residency program was associated with a change in participants’ test scores
In the 7 tests separately, absolute and relative test scores were negatively skewed in tests #2 to #7 and showed a bimodal distribution in test #1

Summary

Introduction

Educational effects of transitioning from formative to summative progress testing are unclear. To decide whether residents master the required competences, programs use summative assessment tools [6, 7]. These can range from workplace evaluations to standardized skills tests and written knowledge assessments. Progress testing is a form of knowledge assessment that has gained ground in medical education in the past two decades [8,9,10,11,12]. It was originally developed in medical schools and later extended to postgraduate medical education [13,14,15]. A key feature of progress testing is the spaced repetition of comprehensive tests which intends to stimulate longterm knowledge retention [16]

Methods

Results

Conclusion