How GREAT is GRADE? Taking stock of GRADE for evidence review in environmental epidemiology

Abstract

It has been proposed to apply the GRADE approach to the evaluation of environmental hazards using the same criteria already used for clinical evaluation, i.e. anchored to the principle that RCT is the gold standard. GRADE defines four levels of evidence quality: very low, low, moderate, and high. Evidence is given a starting quality level (high for RCTs, lower for observational studies) that may decrease (risk of bias, inconsistency, indirectness, and publication bias) or increase (large magnitude of the effect, all plausible confounding would reduce the demonstrated effect or increase the effect if no effect was observed, a dose-response relationship). GRADE recommends against making modifications to the approach, although the literature on different attitudes to applying GRADE is growing. The proposed application of GRADE to environmental risks presents conceptual and operational difficulties. In clinical medicine, the aim is mostly to establish the efficacy of a treatment, whereas in the assessment of environmental exposures the harmfulness of an exposure is of concern. Two different perspectives. In the first case, the main interest is to minimize the number of false positives when the null hypothesis is true (i.e. the treatment is ineffective) with the aim of preventing an ineffective treatment to be used. In the second case, it is necessary to minimize the number of false negatives when the environmental exposure is actually harmful, because it is essential to protect the population (even at the cost of accepting false positive results). GRADE works with a system of downgrading and upgrading of evidence. However, some crucial aspects are not considered: 1. number of studies that constitute the body of the evidence; 2. consistency of the effects across populations that differ in factors such as time, location, and/or exposure assessment; 3. consistency of the effects across studies with different design features. Finally, the assessment of the risk of bias is particularly critical as currently there is no gold standard for such evaluation. In sum, it is obvious that the immediate application of GRADE to environmental issues is not straightforward.