Abstract
A major movement in global health and development in the past 10 years has been the enthusiastic adoption by many of randomized controlled trials (RCTs) from the field of medicine to represent the most rigorous method to evaluate a program's causal impact [1]–[4]. More recently, this movement has brought about a conceptual debate in global health and development about the proper role for RCTs in informing policy, with increasing efforts to “mind the gap” [5] between the evidence generated by RCTs (which focus on internal consistency) and the larger policy questions at the level of communities or populations (which require, among other things, generalizability) [4],[6]–[10]. The field of medicine that developed the RCT also developed the concept of “evidence-based” medicine that aims to improve health policy decision making by encouraging policymakers to base their policies on the best available evidence. Large international policy-making bodies appear set on applying a similar concept to global health and health systems research [4],[11]. In order to be evidence-based, decisions about global health interventions must consider the available evidence in terms of its quantity, quality, and relevance. Rather than use implicit judgment or other ad hoc methods, in evidence-based medicine it is now advocated and common practice to use a formal framework for considering the evidence as part of a systematic review, the advantages of which include increased transparency and better decision-making. Formal frameworks for evaluating evidence about community-level public health interventions have been proposed and advocated for similar reasons [12]–[17]. These frameworks differ in the degree to which they weight the importance of data from RCTs as compared to data from other study designs, the magnitude of potential benefits and harms, the role of context and implementation, and other factors. At present, there are no commonly accepted guidelines within global public health for how to evaluate evidence, and there is scant evidence to guide policymakers when selecting a framework to use for assessing a body of evidence about a global health intervention. We sought to assess how summary conclusions about the evidence for interventions or programs currently in use or proposed for wide adoption could be influenced by the choice of framework. Consistent results across frameworks would increase policymakers' confidence in using and applying evidence frameworks, and may thereby help to narrow the gap between the questions asked by global health researchers and policymakers. Inconsistent results would call for a re-examination of current frameworks in terms of the domains they assess and the ways in which they are applied.
Highlights
A major movement in global health and development in the past 10 years has been the enthusiastic adoption by many of randomized controlled trials (RCTs) from the field of medicine to represent the most rigorous method to evaluate a program’s causal impact [1,2,3,4]
We find that assessing the same body of evidence using existing public health frameworks yields somewhat to markedly different conclusions depending on the framework applied
Had policymakers used the Grading of Recommendations Assessment (GRADE) or Highest Attainable STandard of Evidence (HASTE) framework, they would have concluded that the three interventions varied from ‘‘insufficient’’ or ‘‘low quality’’ to ‘‘strong’’ and ‘‘high quality.’’ Had six different policymakers been considering the same evidence on household water chlorination to reduce diarrheal outcomes and each used a different framework, they could have reached differing conclusions about the strength of support that ranged from grade ‘‘C’’ to grade ‘‘B’’ to ‘‘possible’’ to ‘‘moderate quality’’ to ‘‘strong.’’ Actual policy decisions will include other factors, such as feasibility, financial resources, and health systems capacity, but the current push for ‘‘evidence-based’’ decision-making makes the adoption of an evidence framework likely, and, the rating of evidence would likely be one important factor in decision-making
Summary
A major movement in global health and development in the past 10 years has been the enthusiastic adoption by many of randomized controlled trials (RCTs) from the field of medicine to represent the most rigorous method to evaluate a program’s causal impact [1,2,3,4]. For interventions involving community or lay health workers, we chose the outcome ‘‘reduce morbidity in children under 5 years old compared to usual care’’ as it seemed both to be an outcome very important to communities and to have enough studies to make a meta-analysis meaningful With this intervention the various frameworks again generally rate the evidence as being of low or moderate quality with the exception of USCPSTF, which assigns the highest grade of ‘‘strong.’’ HASTE, on the other hand, would rate this same body of evidence as grade three ‘‘insufficient,’’ and GRADE assigns it a ‘‘low quality of evidence.’’. ‘‘This provisional framework provides a practical and transparent method for deriving grades of recommendation for public health interventions, based on a synthesis of all relevant supporting evidence from research.’’. Grades can be moved down depending on factors such as risk of bias or inconsistency, or up in light of a large measured effect or evidence of a dose-response
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
