The end of educational science?

Abstract

This has been an interesting week. As well as welcoming in a new (academic) year, I also came across three papers in recent publications that have, in my view, enormous implications for how we go about our trade. If we take the message of these papers to heart, it will no longer be ‘‘business as usual’ in medical educational research. In the September issue of Medical Education, Colliver et al. (2008) reanalyzed studies of effectiveness of PBL derived from a meta-analysis by Gijbels et al. (2005). He found that 10 of 11 studies in the review were ‘‘quasi-experimental’’, and this led to measurable bias. In two studies, assignment to PBL or other was not randomized, resulting in demonstrable baseline differences in favour of PBL. In five others, there was a confounding between intervention and outcome, where PBL students had practice with the outcome format, and one study had both problems (and a few more). Colliver’s conclusions go beyond the old ‘‘PBL-other’’ debate, however; he challenges the whole idea that anything of value can come from systematic reviews directed to the calculation of an effect size, and instead argues that ‘‘the field may be better served, in most cases, by systematic narrative reviews that describe and individually evaluate individual studies and their results rather than reviews that obscure biases and confounds by averaging.’’ Eva, editor of Medical Education, concurred. In his editorial (2008), he states, ‘‘A good educational research literature review, in my opinion, is one that presents a critical synthesis of a variety of literatures, identifies knowledge that is well-established, highlights gaps in understanding, and provides some guidance regarding what remains to be understood.’’ Well, surprisingly, Colliver and I, who have historically found ourselves in protracted debates about many things, are on exactly the same side on this one. In a recent ASME monograph, Eva and I (2008) argued that systematic reviews in education have serious problems. Moreover, we noted a paradox, that ‘‘If one cannot combine the findings in some systematic way as a result of heterogeneity of outcomes to the point of having to describe each study independently, then the only thing separating systematic reviews from critical narrative reviews is the amount of time and resources spent searching for