Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?

Lutz Bornmann,Robin Haunschild,Werner Marx

doi:10.1007/s11192-016-2115-y

Abstract

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA), societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric—a start-up providing publication level metrics—started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2 % (n = 2341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas “Earth and related environmental sciences” and “Social and economic geography” are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetrics data. Further empirical studies are necessary, because mentions in policy documents are of special interest in the use of altmetrics data for measuring target-oriented the broader impact of research.

Highlights

Academic science emerged at the beginning of the 19th century (Ziman, 1996)
Altmetric—a start-up providing publication level metrics—started to make data for publications available which have been mentioned in policy documents
Altmetric—a start-up providing publication level metrics—started to make data for academic publications available which have been mentioned in policy documents (Liu 2014) in order to uncover the interaction between science and politics

Summary

Introduction

Academic science emerged at the beginning of the 19th century (Ziman, 1996). During academic science, the evaluation of scientific results focuses on their excellence and originality in a self-regulated process—the peer review process (Bornmann 2011; Petit 2004). The post-academic science which begins in the 1980s (Ziman 2000) is characterized by an increasing competition for research funds (national and international) which are mostly project-dependent distributed. Applicants of project proposals are more and more forced to be specific about the expected outcome and its wider economic and societal impact (Ziman 1998): The context of application becomes the interesting topic which decides on funding (besides excellence and originality). The objective of post-academic science ‘‘is not scientific excellence and theory-building as such but rather the production of a result that is relevant and applicable for the users of the research; in other words, the result should be socially relevant, socially robust and innovative’’ In the current UK Research Excellence Framework (REF), and the Excellence in Research for Australia (ERA) societal impact measurements are inherent parts of the national evaluation systems

Methods

Results

Discussion

Conclusion