What is the value of good medical education research?

Abstract

As the saying goes, those who pay the piper call the tune. In most countries where research is publicly funded there are national systems for assessing research outputs. Governments, research councils, charitable bodies and other investors all want value for money. Accordingly, the UK is currently undergoing a research assessment exercise (RAE) in a process which will be familiar to researchers in many countries worldwide.1 Since 1986, the four UK higher education funding bodies have sponsored successive RAEs, the outcomes of which have been used to inform funding allocations and to provide benchmarking for research quality. The current RAE will report its findings towards the end of 2008. That sense of relief is destined to be short-lived Those UK medical education researchers whose work has been submitted for consideration in this exercise2 will therefore breathe a sigh of relief, believing that all the hard work and preparation is over and all they have to do now is await the outcome. Unfortunately, that sense of relief is destined to be short-lived. The Higher Education Funding Councils in the UK are already thinking about the next round of assessment post-2008. A consultation on the proposed new Research Excellence Framework (REF) has begun3, 4 and the message is already clear: the UK government will use metrics to assess all health-related research from 2008, and what the consultation document describes as a ‘lighter-touch’ approach will be developed for those disciplines within the social sciences.3 The UK government will use metrics to assess all health-related research Medical education researchers have reason to mistrust quantitative approaches to assessing their work. Although many medical education editors and researchers would agree that much of our published research, whatever the methodology, would benefit from being more scientifically and theoretically rigorous,5-7 there is a sense of defensiveness amounting almost to indignation among the medical education community regarding how the quality and value of medical education research are assessed. Commentators frequently argue, with some justification, that medical education research is often judged by criteria that may be inappropriate, reductionist or just plain ignorant. Medical education research has not traditionally prospered at the hands of those in positions of influence where the emphasis is ruthlessly practical. Although medical education research is typically considered part of the biomedical sciences family, and it continues to make extensive use of quantitative methods, it is not typical of ‘classical’ medical research. In a climate where the randomised controlled trial is seen as the ‘gold standard’ for research, medical education constantly has to struggle for recognition.8 Indeed, it has been unflatteringly described as ‘the poor relation of medical research’9 because of the ‘soft’ nature of many of its methodological approaches. These factors contribute to low levels of funding for medical education research, which in turn represents a serious barrier to the development of large-scale empirical research projects. Medical education must constantly struggle for recognition So what are medical education researchers to make of the proposed emphasis on bibliometric data in the next RAE? The consultation document raises many practical questions and potential problems, including the issue of the REF’s reliance on a commercial database known to have several significant flaws, its emphasis on English-language publications, its vagueness about how collaborative projects will be assessed, and so on. But before rushing into criticising the detail of this new framework or condemning it out of hand, medical education researchers should consider its general implications very carefully. It appears probable that medical education research in the UK will continue to be viewed as a medical science rather than a social science. Medical education will be classified within one of the three new ‘medical science’ categories within the REF3 and will therefore be subject to the new bibliometric analysis rather than the ‘lighter-touch’ approach proposed for the humanities, social sciences and mathematics. This view of medical education research as more representative of a science than a social science is widely accepted. For example, all the main medical education journals appear in the Science listings in the Thomson ISI Citation Index, rather than in the Social Science categories; published work is listed in the main scientific and medical indices. The harshest critics of medical education research are also its target audience In many ways, medical education gains from this identification with hard science, despite the fact that it poses additional challenges for researchers. After all, medical education research’s harshest critics are also its target audience: clinicians and scientists familiar with the medical science paradigm. These are the funders who assess our grant applications and the journal readers who judge the products of our research and eventually apply them in their practice. It is therefore vital that the medium and the message of medical education research are appropriate for this audience.9-11 Some medical educators respond with something approaching horror to this type of talk. Medical education, they argue, is too complex, too local, too subjective an activity to be reduced to analysis by those interested only in quantitative approaches and outcomes. Indeed, anyone who has wrestled with the practical and theoretical difficulties of undertaking medical education research in a complex and rapidly changing medical school or clinical environment would find it hard to disagree. Yet whatever our individual beliefs about the nature of medical education research, as a community we nevertheless have to accept the challenge that lies before us.2 We have to address the fundamental questions Our primary concern must be to demonstrate the value of medical education research to those who commission and use our work, in ways that they can understand.12 This does not necessarily mean allowing others to set the agenda; but if we do not wish bibliometric analysis alone to have the final say on how our work is evaluated and funded, then we ourselves must develop new measurement indicators for medical education research that are credible and acceptable to our critics. We must address the fundamental questions. Does medical education make a difference? And if so, to whom and why? How do we define and, importantly, measure educational outcomes and impact? We must, as a matter of urgency, research, assess and demonstrate clearly how what we do is important for the improvement of patient care. The gauntlet has been thrown down: it is time for medical education researchers to pick it up.