Stakeholder views on publication bias in health services research.

Iestyn Williams,Russell Mannion,Richard J Lilford,Yen-Fu Chen,Magdalena Skrybant,Fujian Song,Abimbola A Ayorinde

doi:10.1177/1355819620902185

Abstract

ObjectivesWhile the presence of publication bias in clinical research is well documented, little is known about its role in the reporting of health services research. This paper explores stakeholder perceptions and experiences with regard to the role of publication and related biases in quantitative research relating to the quality, accessibility and organization of health services.MethodsWe present findings from semi-structured interviews with those responsible for the funding, publishing and/or conduct of quantitative health services research, primarily in the UK. Additional data collection includes interviews with health care decision makers as ‘end users’ of health services research, and a focus group with patient and service user representatives. The final sample comprised 24 interviews and eight focus group participants.ResultsMany study participants felt unable to say with any degree of certainty whether publication bias represents a significant problem in quantitative health services research. Participants drew broad contrasts between externally funded and peer reviewed research on the one hand, and end user funded quality improvement projects on the other, with the latter perceived as more vulnerable to selective publication and author over-claiming. Multiple study objectives, and a general acceptance of ‘mess and noise’ in the data and its interpretation was seen to reduce the importance attached to replicable estimates of effect sizes in health services research. The relative absence of external scrutiny, either from manufacturers of interventions or health system decision makers, added to this general sense of ‘low stakes’ of health services research. As a result, while many participants advocated study pre-registration and using protocols to pre-identify outcomes, others saw this as an unwarranted imposition.ConclusionsThis study finds that incentives towards publication and related bias are likely to be present, but not to the same degree as in clinical research. In health services research, these were seen as being offset by other forms of ‘novelty’ bias in the reporting and publishing of research findings.

Highlights

Publication bias occurs when the direction or strength of research findings affects the likelihood of their publication,[1] which can lead to distortions in syntheses of evidence on the effects of interventions or the association between variables
The first issue we explored with interview participants was the extent to which they believed that publication and related bias are a feature of health services research (HSR)
A notable finding of the study is that many study participants felt unable to say with any degree of certainty whether publication bias represents a significant problem in HSR

Summary

Introduction

Publication bias occurs when the direction or strength of research findings affects the likelihood of their publication,[1] which can lead to distortions in syntheses of evidence on the effects of interventions or the association between variables. Preliminary work on publication bias in quantitative HSR found the propensity for publication to be associated with the reporting of statistically significant or positive findings, and about one-quarter of HSR systematic reviews that used statistical methods for detecting potential publication bias finding evidence suggestive of its existence.[5,6] These observations highlight the need for further empirical exploration and theorizing of the factors shaping publication outcomes. Gaining a better understanding of the extent of publication bias and its determinants in HSR is a prerequisite for developing appropriate responses and adapting solutions applied in other research fields.[2]

Objectives

Methods

Results

Conclusion