Judging research quality to support evidence-informed environmental policy

Abstract

In August 2005, PLoS Medicine published an essay by John Ioannidis called: ‘Why most published research findings are false’ [1]. Since then, the paper has been viewed over a million times possibly because of its provocative title, but probably because of growing concerns about the reliability of scientific publications and diminishing confidence in the peer review process to deliver effective quality control. Most recently such concerns have been supported by a number of high profile cases of publication retraction, for example the withdrawal of some statements from articles published in the British Medical Journal regarding the adverse side effects of Statins, a cholesterol-reducing drug [2]. A series of articles in the Lancet has suggested that some $200 billion (estimated to be 80% of the world’s spend on medical research) was wasted on ‘studies that were flawed in their design, redundant, never published or poorly reported’ [3]. Moreover, ‘when a prominent medical journal ran research past other experts in the field, it found that most of the reviewers failed to spot mistakes it had deliberately inserted into papers, even after being told they were being tested’ [4]. In the field of environmental studies too, concerns have been raised about ‘the limited effectiveness of peerreview as a quality-control mechanism’ [5]. The actual and perceived unreliability of scientific reports and papers is particularly problematic for governments’ scientific advisors who operate in the ‘messy’ world of policy making. Here even credible evidence is rarely the only influential factor [6] and policy (and evidence) contention, scientific uncertainty and even ignorance pose significant challenges. The Chief Scientific Advisor for the UK government’s Department for Environment, Food and Rural Affairs has called ‘for an auditing process