11 International peer review and decision-making practices, bias, and burden: the funders’ perspective

Abstract

Objectives Health research funding organisations are responsible for allocating funds for research in a fair, transparent and efficient way. Peer review, and often external peer review, is considered vital for the decision-making process. However, it is well established that peer review can be biased and burdensome for the applicant, peer reviewer and funding organisation. There is limited evidence to suggest which approaches have been considered by funding organisations and whether they have been evaluated to determine their applicability and generalisability. The purpose of this work is to contribute to the evidence gap around the value of alternative and/or innovative approaches to decision-making for grant fund allocation from the perspective of funders. The aim of this study was to identify and explore decision-making practices used by UK and international funding organisations for the allocation of funding for health-related research. Method An online survey was distributed to UK and international health and health-related funding organisations (March/April 2019). The survey collected information about current, past and future approaches to decision-making for grant fund allocation. The survey was sent to 63 funders (targeted list collated by National Institute of Health Research [NIHR]). Social media coverage was used to attract additional funding organisations. The survey focused at the level of a research programme rather than the overall organisation. Therefore, an entry could be provided for every research programme in an organisation. The survey had 3 sections:1 characteristics of the funding organisation;2 decision-making approaches currently being used and those used in the past; and3 approaches that may be considered in the future. Participants were encouraged to provide additional information on what worked well, any potential drawbacks and suggestions for improvements. The survey was active for six weeks (ending 17 April 2019). Results Preliminary descriptive statistics and thematic analysis were conducted on the data available at 3 weeks. Data was quality checked before analysis. From the initial analysis, 12 funding organisations responded to the survey providing coverage from charities and research councils in the health sector. The preliminary data shows that funders use triage, external peer review and face-to-face committee meetings. None reported use of open peer review, partial randomisation or sandpits. Key reported benefits of current systems included transparency and fairness, increased quality and relevance of applications, expert opinion, reduced burden through efficient processes, and consistency in assessment. The main drawbacks of current systems included the potential for bias, time and cost burden, and difficulty in securing external peer reviewers. The full analysis and results will be presented along with the recommendations from the study and its contribution to the wider portfolio of work under the NIHR Research on Research in-house programme. Conclusions In the current research environment, funders have a responsibility to demonstrate fair, transparent and efficient decision-making practices, reducing bias and burden to the funder, peer reviewers and the applicant. There is a lack of understanding from the funder’s perspective around what type of approaches work (or not) due to the lack of empirical evidence. This study aims to contribute to and further build on our understanding to provide robust evidence to enhance the decision-making process of grant allocation. Early findings indicate that funders are keen to explore alternative approaches to decision-making that facilitate the process (through reducing bias, time, monetary cost) whilst maintaining transparency, fairness and quality. The findings from this study will be important in the context of determining which approaches may be applicable and generalisable to use within organisations that fund health research.