Reliability of evidence-review methods in restoration ecology.

Abstract

In restoration science, evidence reviews play a crucial role in summarizing research findings in practice and policy. However, if unreliable or inappropriate methods are used to review evidence, decisions based on these reviews may not accurately reflect the available evidence base. To assess the current value of restoration reviews, we examined a sample of meta-analyses and narrative syntheses (n = 91) with the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT), which uses detailed criteria to assesses the method of policy-relevant evidence synthesis according to elements important for objectivity, transparency, and comprehensiveness. Overall, reviews scored low based on this standard: median score 16 out of 39, modal score 15, and mean 16.6. Meta-analyses scored higher than narrative syntheses (median 17 vs. 5, respectively), although there were some outlier narrative syntheses that had high scores, suggesting that quantitative synthesis does not solely reflect the reliability of a review. In general, criteria spanning the more fundamental review stages (i.e., searching for studies and including studies) received low scores for both synthesis types. Conversely, criteria comprising the later stages of the review (i.e., critical appraisal, data extraction, and data synthesis) were generally well described in meta-analyses; thus, these criteria achieved the highest individual CEESAT scores. We argue that restoration ecology is well positioned to advance so-called evidence-based restoration, but review authors should elucidate their conceptual understanding of evidence syntheses and recognize that conducting reliable reviews demands the same methodological rigor and reporting standards used in primary research. Given the potential of evidence reviews to inform management, policy, and research, it is of vital importance that the overall methodological reliability of restoration reviews be improved.