Does citation matter? Research citation in policy documents as an indicator of research impact \u2013 an Australian obesity policy case-study

Robyn Newson,Andrew Milat,Adrian Bauman,Lesley King,Lucie Rychetnik

doi:10.1186/s12961-018-0326-9

Robyn Newson, Andrew Milat + Show 3 more

Open Access

https://doi.org/10.1186/s12961-018-0326-9

Copy DOI

Abstract

BackgroundCitation of research in policy documents has been suggested as an indicator of the potential longer-term impacts of research. We investigated the use of research citations in childhood obesity prevention policy documents from New South Wales (NSW), Australia, considering the feasibility and value of using research citation as a proxy measure of research impact.MethodsWe examined childhood obesity policy documents produced between 2000 and 2015, extracting childhood obesity-related references and coding these according to reference type, geographical origin and type of research. A content analysis of the policy documents examined where and how research was cited in the documents and the context of citation for individual research publications.ResultsOver a quarter (28%) of the policy documents (n = 86) were not publicly available, almost two-thirds (63%) contained references, half (47%) cited obesity-related research and over a third (41%) of those containing references used unorthodox referencing styles, making reference extraction laborious. No patterns, in terms of the types of documents more likely to cite research, were observed and the number of obesity research publications cited per document was highly variable. In total, 263 peer-reviewed and 94 non-peer-reviewed obesity research publications were cited. Research was most commonly cited to support a policy argument or choice of solution. However, it was not always possible to determine how or why individual publications were cited or whether the cited research itself had influenced the policy process. Content analysis identified circumstances where research was mentioned or considered, but not directly cited.ConclusionsCitation of research in policy documents in this case did not always provide evidence that the cited research had influenced the policy process, only that it was accessible and relevant to the content of the policy document. Research citation across these public health policy documents varied greatly and is unlikely to be an accurate reflection of actual research use by the policy agencies involved. The links between citation and impact may be more easily drawn in specific policy areas or types of documents (e.g. clinical guidelines), where research appraisal feeds directly into policy recommendations.

Highlights

Citation of research in policy documents has been suggested as an indicator of the potential longer-term impacts of research
The research impact assessment literature calls for metrics as proxy measures for the impacts of research, and the number of research citations in public policy documents has been suggested as one such indicator [2,3,4]
The analysis of citation of research in policy documents performed indicated that research had been accessed and considered, but further qualitative investigation is required to answer the more important questions of why and how the research was used

Summary

Introduction

Citation of research in policy documents has been suggested as an indicator of the potential longer-term impacts of research. Reviewing sources cited in policy documents potentially offers a quantifiable, objective and simple method of measuring the impacts of research. A few studies have examined the feasibility of extracting references from health policy documents [6,7,8,9,10]. These studies suggest that it is possible to use bibliometric techniques to identify the geographical origins of research, categorise the type of research cited (e.g. basic versus clinical research), and identify the funding sources of cited research. Research citation studies have examined a limited range of health policy document types and areas of research, including clinical guidelines, on a range of topics [7,8,9,10,11, 13] and policy documents relevant to nursing home visiting research [14], drug monitoring systems [12], environmental health research [6] and climate change research [15]

Methods

Results

Discussion

Conclusion