Assessing the quality of research

Abstract

Over recent years the major medical education journals have published debate about the nature, scope and quality of research in medical education. In general, the tone of the debate has been favourable with wide recognition of the significant contribution made by research to our understanding of student learning, assessment and certification, and to professional development. This debate, which is far from over, has highlighted the importance of evidence to underpin both educational policy and the practical design of teaching programmes. But other voices have raised concerns about the nature of medical education as a discipline with its own specific subject matter and about the nature of much of the research published in leading journals. Doubts have been expressed about the quality of the research and most commentators agree that there is room for improvement. Much, but by no means all, research in medical education is conducted within medical schools and universities within the higher education sector. Some research is also conducted within health services or, as with assessment, within professional institutions such as the National Board of Medical Examiners (North America) or the Royal Colleges in the United Kingdom (UK). In the UK, the Research Assessment Exercise (RAE) is a periodic national peer review process conducted by the Higher Education Funding Councils ‘to provide a stimulus to the improvement of research quality overall’.1 The results of the RAE are used as one of the elements in decision making about how public funds should be distributed for research in universities by the higher education funding councils. The RAE examines research in all the major academic disciplines in higher education. The last exercise covered the five year period ending in December 2001 and the next (RAE 2008) is due to report in 2008 on research carried out in higher education institutions in the UK between January 2002 and October 2007. The nature and format of the 2008 exercise has been developed after a period of wide consultation that followed a review of the 2001 exercise led by Sir Gareth Roberts.2 RAE 2008 will provide quality profiles for research in each higher education institution across all disciplines. The main body of the assessment will take place in 2007–08, with outcomes to be published by the funding councils in December 2008. There will be 15 main panels divided into 67 sub panels responsible for the core work of assessing submissions and making recommendations on ‘quality profiles’ for each submission. Table 1 shows the sub panels and their chairs with most relevance to medical research (further details are available from the RAE website). Guidance for the work of the sub panels has recently been issued and contains details of the quality profiles and definitions of quality levels to be applied by sub panels in considering submitted work. Table 2 shows the definitions of the proposed quality levels ranging from 1* to 4*. One of the conundrums that this national exercise presents to medical education researchers within higher education in the UK is how medical education research will be considered? What criteria would be used to determine if a piece of research in medical education is world leading in terms of its originality, significance and rigour, and who would be qualified to make this judgement? Clearly, if research in medical education is to attract core funding from the major funding councils, indeed if it is to establish a case for itself as a research area to be taken seriously by serious researchers, then those working in the field must aspire to be included in the RAE. A major concern is that RAE 2008 may have come too early for medical education because of its position as a relative newcomer as a field of scholarly endeavour, bringing inevitably low critical mass and relatively poor research funding, and its use of a wide range of methodological and paradigmatic approaches. There is a danger that the resultant judgement on the output of the discipline will be poor and institutions making a submission in the field will be awarded a low star rating with subsequent deleterious effects on future funding and recruitment. In the light of these considerations, some institutions may not report medical education research leading to loss of morale among researchers and eventual problems with continued funding for medical education research posts and units. A more positive approach will be to strengthen focused research activity in the field over the immediate short-term, encouraging returns from those best placed to make them. The future development of research in medical education may then be able to follow a similar path to the one that has been carved out by health services research and has been described by Graham Watt in his commentary3. In the UK, the RAE process poses challenges and threats and raises many questions for medical education research. Whatever action is taken now by higher education institutions will have profound effects on the shape and nature of the discipline in the future.