Evaluation of the National Institute for Health and Care Research’s (NIHR) Global Health Research (GHR) Portfolio - Inception Report

Paula Quigley,Cheri Grace,Seema Kahn,Sarah Hanka,Korina Cox

doi:10.3310/nihropenres.1115201.1

Abstract

Evaluation of the National Institute for Health and Care Research’s (NIHR) Global Health Research (GHR) Portfolio Inception Report October 2022 Contents Executive Summary 1.0 NIHR GHR Portfolio and Introduction to the Evaluation 1.1 Overview of the GHR Portfolio 1.2 Evaluation Objectives and Scope 2.0 Evaluation Approach 2.1 Evaluation Conceptual Framework 2.2 Evaluation Framework 3.0 Evaluation Methodology 3.1 Overview of Research Timetable 3.2 Research Tasks 3.3 Overview of Sampling 3.4 Approach to Data Analysis 3.5 Framework to Assess the Strength of the Evaluation Findings by Strength of Evidence 4.0 Reporting and Dissemination 5.0 Annexes 5.1 Annex 1: Evaluation Framework 5.2 Annex 2: TOC Workshop Summary Tables Table 1 GHR Portfolio Summary Table 2 Evaluation Criteria and Questions Table 3 Number of awards to be evaluated by programme Table 4 Robustness rating for emerging themes/main findings Figures Figure 1 Approach to Contribution Analysis Abbreviation List AAS: African Academy of Sciences AMR: Anti-microbial resistance CA: Contribution Analysis CCF: Central Commissioning Facility CEI: Community Engagement and Involvement CEPI: Coalition for Epidemic Preparedness Innovations DAC: Development Assistance Committee DHSC: Department for Health and Social Care EDCTP: European & Developing Countries Clinical Trials Partnership EF: Evaluation Framework EPSRC: Engineering and Physical Sciences Research Council EQ: Evaluation Question ESRC: Economic and Social Research Council FAF: Financial Assurance Fund FCDO: Foreign, Commonwealth and Development Office FGD: Focus Group Discussion FIND: Foundation for Innovative New Diagnostics GACD: Global Alliance for Chronic Diseases GARDP: Global Antibiotic Research and Development Partnership GCC: Grand Challenges Canada GECO: Global Effort on COVID-19 GFGP: Good Financial Grants Practice GHR: Global Health Research GPSC: Global Patient Safety Collaborative GRSF: Global Road Safety Facility HPSR: Health Policy and Systems Research JGHTI: Joint Global Health Trials Initiative KII: Key Informant Interview LMIC: Low- and Middle-Income Countries MMV: Medicines for Malaria Venture MRC: Medical Research Council NETSCC: NIHR Evaluation, Trials and Studies Coordinating Centre NIHR: National Institute for Health and Care Research ODA: Official Development Assistance OECD: Organisation for Economic Co-operation and Development REA: Rapid Evidence Assessment RIGHT: Research and Innovation for Global Health Transformation RSTMH: Royal Society of Tropical Medicine and Hygiene RW: Research Wave R2HC: Research for Health in Humanitarian Crises SNA: Social Network Analysis SORT-IT: Structured Operational Research and Training Initiative SPARC: Short Placement Award for Research Collaboration TB: Tuberculosis ToC: Theory of Change ToR: Terms of Reference VFM: Value for Money WHO: World Health Organisation Acknowledgements and Disclaimer  Acknowledgments  The lead authors of this report are Paula Quigley, Seema Khan, Cheri Grace, Sarah Hanka and Korina Cox.    Disclaimer  This evaluation is funded by the National Institute for Health Research (NIHR) under its Policy Research Programme (PRP) (Grant Reference Number NIHR203816). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care (DHSC). This report has been prepared by Ecorys for DHSC, for services specified in the Terms of Reference and contract of engagement.  In line with the terms and conditions of the contract, all intellectual property rights in all material (including but not limited to reports, data, designs, whether or not electronically stored) produced by the Supplier or the Supplier's Personnel pursuant to the performance of the Services (‘the Material’) shall be the property of the Supplier. Under the terms of the contract, Ecorys, as the Supplier hereby grants to DHSC a perpetual, world-wide, non-exclusive, irrevocable, royalty-free licence to use all the Material. DHSC will be the final owner of the findings of the evaluation.   Ecorys will store all material related to the evaluation on a secure drive. Data will be managed under the terms and conditions of the contract with DHSC.   There are no conflicts of interest.  Executive Summary The National Institute for Health and Care Research (NIHR)’s Global Health Research (GHR) portfolio was established by the Department for Health and Social Care (DHSC) in 2016 to support the objectives of the UK Aid Strategy and the United Nations’ Sustainable Development Goals. As part of the 2015 spending review, an initial budget of £430 million was allocated for the first phase (2016-2021) of the GHR portfolio to fund applied global health research in low- and middle-income countries (LMICs) eligible to receive Official Development Assistance (ODA). The goal of the initiative is to support high quality applied health research and training to address unmet needs in ODA-eligible countries by generating evidence for the direct benefit of people in these countries. It also aims to strengthen research capacity, develop more equitable partnerships between UK and LMIC research institutes, overcome barriers to health research uptake and ensure a high level of community engagement and involvement (CEI). In December 2021, the DHSC commissioned Ecorys to undertake an evaluation of the first phase of the portfolio (2016/17 to 2020/21). The evaluation’s objectives are to assess the suitability of the design and implementation of the portfolio for achieving its intended results, and to identify key learning to inform development and delivery of the portfolio’s second phase (2021/22 onwards). In addition, the evaluation aims to provide accountability for the GHR portfolio performance to date, determining the Value for Money (VFM) of investments, and assessing whether the portfolio is on track to achieve desired outcomes and long-term impact. The evaluation approach is based on the recognition that there has been notable learning during the first phase of the portfolio implementation and aims to capture this. The evaluation will employ a theory-based approach utilising both quantitative and qualitative methods to assess process, performance and learning based on the GHR portfolio’s Theory of Change (ToC). Specifically, the evaluation will use Contribution Analysis (CA), to build evidence of whether and how the GHR portfolio is contributing to its intended outcomes and impacts. The evaluation framework is structured around three pillars of evaluation criteria[1]: i) relevance and coherence ii) efficiency and effectiveness iii) sustainability and impact. As cross-cutting themes, the evaluation team will assess adaptability and learning across the portfolio, as well as CEI. The evaluation framework is intentionally designed as a comprehensive and overarching framework which will be tailored to the specific context of each stakeholder. Following the inception phase, there will be two research waves – the first from September/October 2022 to April 2023 and the second from May/June to December 2023, followed by a final dissemination phase up to March 2024. The key methods comprise programme and award documentation and data review; interviews with key stakeholders at portfolio, programme and award levels; an online survey to all funded stakeholders; a Social Network Analysis (SNA) to assess interactions and networks; and country visits where feasible in the second year. A purposive sample of programmes and awards for in-depth review is proposed which the evaluation team judges to be reasonably reflective of the overall portfolio and provide the breadth to answer the questions in the evaluation framework. Data sources include primary data, for example, strategic interviews, case study interviews with award holders, observations at country level and survey data; and secondary data, such as portfolio level, delivery mechanism, and award level documentation. All data will be coded according to the evaluation questions ensuring a consistent approach to aid synthesis and minimise bias. The evaluation team will assess and triangulate the evidence from different data sources to build up a picture of the GHR’s contribution to change. An interim report produced in March/April 2023 will collate findings from the portfolio, programme and award assessments from the first research wave and provide key learning points for NIHR and DHSC. This will include any necessary adaptations in the methodology for the second research wave and any proposed recommendations for changes in the ToC. A final evaluation report will be delivered by December 2023. Recommendations will be relevant and realistic, and all evaluation outputs accessible and useful. A range of dissemination products and channels are proposed for the final step. The evaluation team is committed to ensuring that the entire process is responsive and helpful to the DHSC and NIHR, as well as other implementing partners, and will provide opportunities for feedback and interaction as needed. The team will maintain regular communications with key stakeholders throughout as appropriate and aim to be as inclusive as possible, while acknowledging the need to balance engagement with potential overburdening with demands for data and time. The team will ensure an ethical approach to this evaluation and adhere to relevant international guidelines. All data will be securely stored and managed.   This Inception Report presents the proposed approach to this evaluation – detailing the conceptual framework, evaluation questions, methodology for data collection and analysis, reporting and dissemination. 1.0 NIHR GHR Portfolio and Introduction to the Evaluation 1.1 Overview of the GHR Portfolio The Global Health Research (GHR) portfolio was established by the Department of Health and Social Care (DHSC) with the National Institute for Health and Care Research (NIHR), following the publication of the <a href="https://www.nihr.ac.uk/explore-nihr/funding-programmes/global-health.htm">2015 UK Aid Strategy</a>. It aims to directly address the diverse health needs of people in low- and middle-income countries (LMICs) by supporting high quality applied health research and training in areas where there is an unmet need; generating evidence for the direct benefit of people in LMICs; and strengthening LMIC and UK research capabilities and expertise in global health. As part of the 2015 spending review, an initial budget of £430 million was allocated for the first phase (2016-2021) of the GHR portfolio to contribute to improvement in global health outcomes through applied global health research, complementing other Official Development Assistance (ODA) research programmes such as the <a href="https://www.ukri.org/what-we-offer/international-funding/global-challenges-research-fund/">Global Challenges Research Fund</a> and the <a href="https://www.newton-gcrf.org/newton-fund/">Newton Fund</a>. Since its launch in 2016, and as of September 2022, the portfolio has grown to include <a href="https://fundingawards.nihr.ac.uk/search/funder/NIHR%20(ODA)">17 thematic areas</a>, with research activities being conducted in over 50 LMICs across Africa, Asia and Latin America, and through more than 20 delivery partners. In Phase 1, the GHR portfolio funded more than 650 awards which ranged from £5,000 to several million pounds[1]. In addition to being ODA-eligible, all the global health research funded by GHR must be underpinned by the following principles:  <ul> <li>Builds on NIHR’s operating principles of impact, excellence, effectiveness, inclusion and collaboration; and,</li> <li>Strengthens research capability and training through equitable partnerships. </li> </ul> The portfolio is delivered through two main mechanisms: NIHR-delivered programmes and Partnership-delivered programmes. It also includes a range of initiatives directly supporting career development and training and research opportunities which complement the capacity strengthening elements that are embedded in the programmes and are a key principle across the portfolio. The NIHR-delivered programmes are led by NIHR Coordinating Centres, and the Partnerships-delivered programmes are led by various established research funders and partners, including UK partners, international partners and multi-funder initiatives. This division was not pre-determined at the start of the portfolio, but rather a result of the evolution and learning acquired during the initial delivery period which aimed to leverage existing expertise and maximise resources most effectively. There are common thematic areas across programmes, for example non-communicable diseases (including mental health), surgery, injuries and accidents, and comorbidity, health systems research, maternal and neonatal health, and COVID-19. Table 1 below summarises the GHR programmes and delivery partners, in order of disbursement to date for NIHR-delivered programmes, and approximate spend in Phase 1 for partner-delivered programmes. When the portfolio was launched in 2016, DHSC initially prioritised working with partners who were already well established in the global health research space, such as the Medical Research Council (MRC) and the Wellcome Trust. NIHR rapidly built up its internal capacity in managing global health research programmes, growing from a small core team of only three people to a much more extensive team today. Over time NIHR principles that govern all work, whether domestic or global, have also been embedded in the partnerships where feasible providing extra added value of working in partnership together, and extending the influence of NIHR on other funders and activities. More detail behind the story of the evolution of the GHR portfolio will be gathered in the next phase of the evaluation. Given the rapid evolution of the portfolio, this evaluation provides an opportunity for learning, and to inform the second phase and subsequent phases of the portfolio. Table 1       GHR Portfolio Summary NIHR-delivered <ol> <li><a href="https://devtracker.fcdo.gov.uk/projects/GB-GOV-10-NIHR_GHRUG1/summary">GHR Groups</a>, NETSCC, £90m</li> <li><a href="https://devtracker.fcdo.gov.uk/projects/GB-GOV-10-NIHR_GHRUG1/summary">GHR Units</a>, NETSCC, £84m</li> <li><a href="https://www.nihr.ac.uk/explore-nihr/funding-programmes/research-and-innovation-for-global-health-transformation.htm">Research & Innovation for Global Health Transformation (RIGHT</a>), CCF, £54m</li> <li><a href="https://www.nihr.ac.uk/explore-nihr/funding-programmes/global-health-policy-and-systems-research.htm">Global Health Policy and Systems research (HPSR) Development Awards</a>, NETSCC, £17m</li> <li><a href="https://www.nihr.ac.uk/explore-nihr/academy-programmes/research-professorships.htm#:~:text=The%20NIHR%20Global%20Research%20Professorships,public%20health%20or%20care%20research.">Professorships</a>, NIHR Academy, £9.5m</li> <li><a href="https://www.nihr.ac.uk/documents/nihr-global-health-research-financial-management-guidance-for-awards-funded-through-official-development-assistance-oda/29396">Financial Management Guidance for Awards (FAF)</a>, NIHR Academy, N/A</li> <li><a href="https://www.nihr.ac.uk/documents/nihr-ghr-sparc-global-health-research-short-placement-award-for-research-collaboration-where-can-i-go/29738">Global Health Research Short Placement Award for Research Collaboration (SPARC</a>), NIHR Academy, N/A</li> </ol> Partner-delivered <ol> <li>European and Developing Countries Clinical Trials Partnership (EDCTP)[2], EDCTP, £79m</li> <li>Joint Global Health Trials Initiative (JGHTI)[3], MRC, £32m</li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-PDP_CEPI">Coalition for Epidemic Preparedness Innovations (CEPI)</a>, FCDO, £20m</li> <li>Research for Health in Humanitarian Crises (R2HC)[4], ELRHA/Save the Children UK, £8m</li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-DPO">Diagnostics, Prosthetics and Orthotics to Tackle Health Challenges in Developing Countries</a>, EPSRC, £7m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=GB-GOV-10-pdp_mmv#view=act&aid=GB-GOV-10-PDP_MMV">Medicines for Malaria Venture (MMV)</a>, MMV, £6m</li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-AMR_GC">Anti-Microbial Resistance (AMR) in a Global Context</a>, MRC, £6m</li> <li>Research to Improve Adolescent Health in Low- and Middle-Income Country (LMIC)[5], MRC, £6m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=GB-GOV-10-grsf#view=act&aid=GB-GOV-10-GRSF">Global Road Safety Facility (GRSF)</a>, World Bank, £5m</li> <li>Structured Operational Research and Training Initiative on building, sustainable operational research capacity on AMR in LMICs (AMR SORT-IT), WHO, £5m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=GB-GOV-10-pdp_tba#view=act&aid=GB-GOV-10-PDP_TBA">Global Alliance TB Drug Development</a>, TB Alliance, £4.5m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=GB-GOV-10-gardp#view=act&aid=GB-GOV-10-PDP_GARDP_NS">Global Antibiotic Research and Development Partnership’s Neonatal Sepsis Programme (GARDP), GARDP, £4m </a></li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-Wellcome_NIHR_2019">NIHR-Wellcome Global Health Research Partnership</a>, Wellcome, £4m </li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-AMR_B">Antimicrobial Resistance (AMR) Cross-Council Initiative: Behaviour Within and Beyond the Healthcare Setting</a>, ESRC, £3m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=GB-GOV-10-gcc#view=act&aid=GB-GOV-10-GCC">Grand Challenges Canada (GCC) Global Mental Health</a>, GCC, £2.5m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=geco#view=act&aid=GB-GOV-10-GECO">Global Effort on COVID-19 (GECO) Health Research</a>, MRC, £2m</li> <li>Global Alliance for Chronic Diseases (GACD)[6], MRC, £2m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=gpsc#view=act&aid=GB-GOV-10-GPSC">Global Patient Safety Collaborative (GPSC)</a>, WHO, £1.5m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=GB-GOV-10-find#view=act&aid=GB-GOV-10-PDP_FIND">Foundation for Innovative New Diagnostics (FIND)</a>, FIND, £1m</li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-BRG_HIV">Biomedical Resources Grant</a>, Wellcome, £0.9m</li> <li>Royal Society of Tropical Medicine and Hygiene (RSTMH) Small grants scheme[7], RSTMH, £0.75m</li> <li><a href="http://d-portal.org/ctrack.html?text_search=gfgp#view=act&aid=GB-GOV-10-GFGP">Good Financial Grants Practice (GFGP)</a>, AAS, £0.5m</li> <li><a href="http://d-portal.org/ctrack.html#view=act&aid=GB-GOV-10-AGHRB_MNH_call1">Global Maternal and Neonatal Health</a>, MRC, £0.45m</li> </ol> 1.2 Evaluation Objectives and Scope Ecorys has been contracted by DHSC to assess the suitability of the design and implementation of the first phase of the GHR portfolio (2016/17-2020/21) in achieving its intended outcomes and impacts as set out in GHR’s <a href="https://www.nihr.ac.uk/documents/global-health-research-portfolio-theory-of-change/26036">Theory of Change </a>(ToC). The evaluation will contribute to the evidence base on health research impact, as well as building evidence and understanding of a) the extent to which researchers are engaged in effective knowledge mobilisation (and what types of support or interventions encourage the transfer of knowledge), and b) the extent to which community engagement and involvement (CEI) meaningfully and sustainably leads to higher quality research and intended impacts. The ToC has a long-term perspective, acknowledging that it may require 10-25 years for changes in policy, practice and behaviour (outcomes) to contribute towards strengthened health systems and increased individual and community capacity for health promotion and disease prevention (impacts). The evaluation of the first phase of the portfolio takes into account this long-term nature of change and, therefore, will assess evidence that the portfolio is being delivered as expected and whether or not key assumptions are holding true. A core objective of the evaluation is to generate real-time learning to support implementation and inform development and delivery of subsequent phases of the portfolio. The evaluation will include a strategic assessment of VFM at the portfolio level in terms of allocative efficiency, technical efficiency and value/results. Equity, gender equality and social inclusion will be considered key cross cutting principles and integrated and addressed throughout the evaluation. In scope for this evaluation are all GHR portfolio programmes and their awards that started during the Phase 1 timeframe and ideally completed by the second quarter of 2022. However, some programmes have no or very few awards that were completed by Q2 2022 (e.g. Professorships, RIGHT), therefore, the evaluation will also review some active awards to ensure coverage across the entire portfolio. 2.0 Evaluation Approach 2.1 Evaluation Conceptual Framework 2.1.1 GHR Theory of Change The GHR portfolio <a href="https://www.nihr.ac.uk/documents/global-health-research-portfolio-theory-of-change/26036#:~:text=This%20Theory%20of%20Change%20is,%2C%20outputs%2C%20outcomes%20and%20impacts.">Theory of Change</a> (ToC) produced collaboratively by the DHSC GHR Team, NIHR Coordinating Centres and other strategic partners and award holders provides the conceptual framework for this evaluation. The GHR ToC visually represents how the portfolio’s funded activities and outputs are intended to contribute to long-term positive changes in health and health systems. A ToC workshop (see Annex 2) was organised with portfolio stakeholders during inception in order to develop an agreed and improved understanding of all elements of the NIHR GHR ToC, to further refine the evaluation questions and scope. During the workshop, it was agreed that this simplified diagram of expected results from inputs and activities to outputs, outcomes and impact remains broadly valid. However, since it was developed, the initial concept of three activity strands, namely, Programmes and Partnerships with an overlapping strand of People, has evolved over time. The Programmes strand which refers to NIHR-commissioned research calls and the Partnerships strand which refers to collaborations with other funders are the two delivery mechanisms. The People strand refers to training and development of researchers and research support staff. This does not quite capture the capacity strengthening approach that is embedded in all programmes and is a key principle across the portfolio. The evaluation will seek to determine if there is a more appropriate way to reflect the GHR programme delivery in the ToC. It is anticipated that the GHR portfolio’s short-term contribution to producing high quality relevant research outputs, supporting capacity building, facilitating partnerships and networks, and supporting dissemination and knowledge exchange will influence policy making and practice in the medium term. The ToC identifies these anticipated causal links drawing on underpinning theory from literature on health research impact, including the uptake of research evidence to improve the responsiveness of health systems to population needs and improved use of evidence to support capacity for health promotion and disease prevention. Outcomes are dependent on a wide range of external factors, which are captured in the ToC narrative as a set of assumptions. These assumptions have been unpacked in the evaluation framework so that they can be examined and tested. Some of them have been affected by the COVID-19 pandemic and other stresses on the UK economy, for example, the assumption that ‘global health research funders continue funding at present rate’. The evaluation team will review all assumptions during the first year of assessment along with the flow diagram and suggestions for amendments will be made where appropriate. The ToC has been used to inform the Evaluation Framework (EF) (Section 2.2) and will be used to structure a theory-based approach to assessing the portfolio’s outcomes and impact. This approach is appropriate for complex interventions in complex environments. Specifically, the evaluation will use Contribution Analysis (CA), a theory-based evaluation methodology, to build evidence of whether and how the GHR portfolio is contributing to its intended outcomes and impacts. Contribution Analysis is well-suited to meeting the evaluation objectives and answering the evaluation questions, given the time and resources available. Specifically, the choice of this method is informed by the following considerations: <ul> <li>CA assumes there are complex processes at play in achieving any outcome, which is well-suited to a programme with multiple funding and partnering models.  </li> <li>CA is particularly useful when the evaluated programme already has a clearly articulated theory of change, including assumptions linking outputs, outcomes, and impact.  </li> <li>CA provides a methodology for assessing causality in evaluations where a comparison or baseline would not be appropriate or has not been established prior to the start of programming.  </li> <li>CA is useful when evaluations are intended to support internal learning as it supports the development of lessons learned rather than exclusively focusing on showcasing success and accountability</li> </ul> 2.1.2 Approach to Contribution Analysis The CA method and analytical tools provide a way to structure and build up evidence about GHR portfolio activities and their potential contribution to supporting long-term change in health and health systems. The evaluation team will use the ToC to gather evidence and draw conclusions about the relevance and coherence of GHR-supported activities and whether and how different programmes are achieving their intended results, whether these are being implemented in line with key principles that underpin a process of sustainable long-term change (e.g. supporting capacity strengthening, equitable research partnerships), and key factors influencing this. Practically, the CA methodology facilitates the structuring of evidence during each of the evaluation phases outlined below. First, the evaluation will collate and collect evidence about observed changes related to the portfolio’s activities in the short term, exploring the processes and causal pathways through which these have occurred and identify emerging evidence of progress towards influence on policy making and practice (i.e. expected medium term outcomes). Second, the evaluation will assess the role played by the GHR funded activity and wider influence linked to partnership working, and the role of other factors in hindering or enabling positive change. This systematic approach will provide a foundation for the evaluation to report on evidence of the GHR portfolio’s contribution to outcomes and impact under the first phase and generate learning to inform future phases. Figure 1       Approach to Contribution Analysis Inception Phase <ul> <li>Work closely with DHSC and NIHR to finalise Evaluation Questions (EQs) and articulate a testable framework for analysis against the GHR ToC.</li> </ul> Research Waves 1 + 2 (Fieldwork) <ul> <li>Interviews/focus groups will focus on gathering and mapping evidence to verify or challenge the evidence and causal pathways identified through the ToC and inception phase.</li> <li>Analysis of existing evidence will continue through ongoing review of (new) documentation and interview data, producing a contribution st

Full Text