A new database of the references on international clinical practice guidelines: a facility for the evaluation of clinical research

Magnus Eriksson,Grant Lewison,Tommy Billhult,Elena Pallari,Annika Billhult

doi:10.1007/s11192-019-03318-2

Abstract

Although there are now several bibliographic databases of research publications, such as Google Scholar, Pubmed, Scopus, and the Web of Science (WoS), and some also include counts of citations, there is at present no similarly comprehensive database of the rapidly growing number of clinical practice guidelines (CPGs), with their references, which sometimes number in the hundreds. CPGs have been shown to be useful for the evaluation of clinical (as opposed to basic) biomedical research, which often suffers from relatively low counts of citations in the serial literature. The objectives were to introduce a new citation database, clinical impact®, and demonstrate how it can be used to evaluate research impact of clinical research publications by exploring the characteristics of CPG citations of two sets of papers, as well as show temporal variation of clinical impact® and the WoS. The paper includes the methodology used to retain the data and also the rationale adopted to achieve data quality. The analysis showed that although CPGs tend preferentially to cite papers from their own country, this is not always the case. It also showed that cited papers tend to have a more clinical research level than uncited papers. An analysis of diachronous citations in both clinical impact® and the WoS showed that although the WoS citations showed a decreasing trend after a peak at 2–3 years after publication, this was less clear for CPG citations and a longer timescale would be needed to evaluate their impact on these documents.

Highlights

The use of counts of citations to research papers has been a standard method for research evaluation for many years (Kostoff 1998; Ingwersen et al 2000; Leydesdorff 2008; Moed 2009)
In order to illustrate the use of the new database for research evaluation, we provide two worked examples of the information that can be provided for a research performer, namely a Scottish hospital, and for a research funder, namely a Swedish collecting charity
Two of the Scottish hospital papers were frequently cited in clinical practice guidelines (CPGs): one in 20 and another in 18 CPGs

Summary

Introduction

The use of counts of citations to research papers has been a standard method for research evaluation for many years (Kostoff 1998; Ingwersen et al 2000; Leydesdorff 2008; Moed 2009). It has been established that the citations to groups of papers, which must strictly be counted in a fixed time window, vary with the parameters of the papers such as the numbers of authors, the numbers of addresses, the numbers of funding acknowledgments, the field of study, and whether the paper is applied (clinical for biomedical research) or basic (Lewison and Dawson 1998; Roe et al 2010; van Eck et al 2013) The last of these factors (Research Level, RL) has a big influence, and (with some exceptions) clinical papers tend to be less well cited than basic ones. It may well have more immediate influence on clinical practice

Methods

Results

Conclusion