Do Cochrane systematic reviews meet WHO needs?

Abstract

Several international organizations, including the World Health Organization (WHO), have implemented approaches for creating treatment recommendations based on explicit assessments of the evidence base 1. As these approaches require systematic and critical appraisal of the best available evidence, the use of evidence already aggregated and synthesized in the form of systematic reviews of randomized trials has become increasingly common. Therefore, producers of high-quality systematic reviews, such as the Cochrane Collaboration, have progressively acquired a prominent role in informing international guidelines, as exemplified in this issue of Addiction by Davoli and collaborators, who used Cochrane reviews to develop a set of WHO guidelines for the psychosocially assisted pharmacological treatment of opioid dependence 2. This case study represents an interesting example of how Cochrane systematic reviews may effectively inform WHO recommendations. However, there are challenging issues. First, Cochrane reviews are not designed and conducted with the primary purpose of meeting the needs of WHO or other international organizations. This implies that for some questions, and for several outcomes that are key for guideline developers, no data are available from Cochrane. Davoli and collaborators reported that they had to undertake a new systematic review of observational studies, as the evidence from trial data was inadequate, and that for several outcomes, judged to be of critical relevance for the guideline production process, no data were available from Cochrane reviews. A similar mismatch between what Cochrane can offer, and what WHO actually needs, was found recently in the field of mental and neurological disorders 3. Secondly, Cochrane systematic reviews, in addition to standard western databases, do not search comprehensively other databases storing reports of studies conducted in low- and middle-income countries (LAMICs) and published in non-western languages. Also, the grey literature is not searched comprehensively enough to meet the needs of WHO, which is a major issue, as it is often the case that randomized studies conducted in low-resource settings are not published or are disseminated only through local channels. If Cochrane reviews systematically miss a proportion of evidence from LAMICs, then their relevance in informing WHO recommendations, which are focused especially on the needs of LAMICs, cannot be expected to be very high. This is highlighted clearly by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology 4, as this system forces guideline developers to rate the extent to which the question being addressed by the guideline panel is different from the available evidence regarding, for example, the target population. Thirdly, most Cochrane reviews include randomized trials only. Although this is quite reasonable in principle, as randomized evidence is more reliable than observational evidence, there are questions being addressed by guideline developers that cannot be answered by randomized trials only 5. For example, the unintended consequences of pharmacological treatments, including side effects, which are given high value by WHO, are addressed in an unsatisfactory way by Cochrane reviews. Cochrane reviews usually include proxy measure of side effects, such as acceptability rates (dropouts), but there are individual side effects that are particularly relevant, especially in the long term, which are not captured at all. Similarly, WHO may be particularly interested in providing advice on how health systems should be developed to offer optimal care at sustainable costs 6, but trials are very sparse on these issues (and may be unsuitable to answer these system-level questions). By contrast, there may be observational epidemiological data which might be used to inform decisions, but these are almost never included in Cochrane and non-Cochrane systematic reviews. The Cochrane Collaboration, which has been having a tremendous impact in developing and optimizing the methodology of systematic reviews of trial data, should try to make a similar effort in developing better methodologies for aggregating and synthesizing the evidence from observational studies. This would represent a major step forward for different stakeholders, including WHO. Fourthly, as wisely pointed out by Davoli and collaborators, according to the GRADE methodology treatment recommendations are informed by the evidence base, its quality and by factors related to values, preferences and feasibility considerations. It should be noted that attributing value not only to the evidence base, but also to other issues, may have negative and positive consequences. A potential negative consequence is that recommendations may be related only loosely to the background evidence. Davoli and collaborators reported that five of eight strong recommendations were based on low or very low quality of evidence. This is consistent with a recent survey of WHO recommendations that found that strong recommendations based on low or very low confidence estimates are made very frequently in WHO guidelines 7. These findings raise questions concerning the extent to which guideline developers take into consideration uncertainties in the evidence when they consider the strength of recommendations. Conversely, however, it should be noted that value-based decisions may be unavoidable in specific circumstances. In reviewing the evidence for WHO recommendations for mental disorders 8, for example, it was found that for strategies aimed at improving community attitudes towards people with mental, neurological and substance use conditions, the evidence base was very poor and indirect. Nevertheless, because of strong values and the importance of improving community attitudes, panel members made a recommendation to consider the planning and implementation of activities such as anti-stigma campaigns. It should be emphasized that the added value of GRADE in these circumstances is that it is required to report transparently that some recommendations are based on strong values and weak evidence. This adaptation and contextualization of the evidence has been highlighted recently as a crucial aspect when putting the principles of evidence-based medicines into practice, and has been called real evidence-based medicine 9. In the field of mental, neurological and substance use disorders the greatest benefit in the next decade will derive from providing better care based on current knowledge 10. In order to convert this forecast into action, Cochrane reviews should be increasingly designed in partnership with selected stakeholders 11, giving priority to those who have been implementing activities and programmes for scaling-up care for disorders of high priority, especially for the LAMICs. None. As Director of a WHO Collaborating Centre, Corrado Barbui has been providing methodological support to the WHO in the development of evidence-based recommendations using the GRADE approach. Corrado Barbui is also one of the Editors of the Cochrane Collaboration Depression, Anxiety and Neurosis (CCDAN) group.