Measuring the impact of family medicine research: scientific citations or societal impact?

Abstract

The ultimate goal of research in medicine, a spectrum starting with basic biomedical research via clinical trials to implementation research, is to improve the health and the quality of life of individuals and communities. Research should therefore be of the very best quality and assessment of its quality is important. Traditionally, research and researchers are evaluated by means of the ‘scientific impact’ of research output, i.e. publications in biomedical journals. The higher the impact factor of a journal, the better the research scores. Journals know how they can polish up their impact factor and for some this is even a deliberate policy. Increasingly, journal impact factors are also used to evaluate academics nominated for promotion and to allocate research funds. Impact factors are considered as an objective way to judge the quality of research/researchers. However, is the journal impact factor an appropriate and sufficient indicator of quality? There are reasons for doubt. The journal impact factor is calculated by dividing the total number of cited articles in the previous 2 years by the number of ‘citable’ articles in the journal during the same period. It is not clear what makes an article citable and the commercial company that creates and manages the journal impact factor, Thomson Scientific, is not very transparent in the criteria it uses to select journals for inclusion in the database either. The Journal Citation Reports database only covers a fraction of all the available biomedical journals; approximately 5000 compared with over 33 000 that are indexed in Medline. The highest impact factors can be found among the journals in the field of basic medical sciences such as molecular biology and biochemistry. In these highly dynamic research fields, a large proportion of citations are captured in the short term used to calculate the impact factor. In fields with a more durable literature, like epidemiology, public health and also family medicine, citations are usually spread over a longer period of time, resulting in lower impact factors. Moreover, the domain of family medicine is nearly invisible in the database. Only eight family medicine journals are listed in the Journal Citation Reports, all located at the tail of the 105 journals that make up the domain of general and internal medicine. In addition, there is a regional bias since in most domains North American journals have the highest impact factor. This is illustrated by a recent development in the family medicine journal landscape. With a first impact factor of 3.803, the Annals of Family Medicine (launched only in 2003) immediately surpasses all other family medicine journals. We congratulate the journal for this achievement that will be an important boost for family medicine research. However, it also shows that impact factors have no relation to the position of family medicine or research in the region, since family medicine (research) is more developed in some European countries than in the US. So there is a bias of discipline and region, but is it still a good indicator of research quality? Although some argue that a high impact factor is generally linked to better quality of the articles, this is not apparent in all domains. Usually, only a minority of articles account for the majority of citations. In fact, the journal impact factor is nothing more than an index of how often a specific journal has been cited in a selected group of journals. Some have even suggested changing its name to ‘citation rate index’ or ‘journal citation ratio’, since the word ‘impact’ suggests more than what it actually stands for. The journal impact factor has an impact on how research and researchers are valued by their academic peers and on the allocation of research funds, but does it have an impact on what research is ultimately about, improving quality of life for all citizens? It has been suggested to pay attention to the ‘societal impact’ of research as well. Adding to the