Assessing Risk of Bias in Human Epidemiologic Studies Using Four Tools: Different Conclusions from Different Tools

Abstract

Background: Systematic reviews (SR) are increasingly prevalent in environmental health due to their improved ability to synthesize large bodies of evidence, while minimizing bias. Different SR methods have been developed by the U.S. NTP’s Office of Health Assessment and Translation (OHAT), U.S. EPA’s Integrated Risk Information System (IRIS), and by the U.S. EPA under the Toxic Substances Control Act (TSCA), including how they assess risk of bias (ROB). The objective of this study was to compare the performance of three tools (OHAT, IRIS, TSCA) in assessing ROB, in epidemiology studies in order to identify best practices. Methods: Two reviewers independently applied the OHAT, IRIS and TSCA ROB tools to epidemiologic studies used in a previous SR on polybrominated diphenyl ethers and Intelligence Quotient and/or Attention Deficit Hyperactivity Disorder. This SR was chosen because it has been deemed good quality by the National Academy of Sciences. We measured the time to apply each tool and interrater reliability. Results: The time to complete the ROB assessments using the different tools varied widely (mean = 20, 32, and 40 minutes per study for the OHAT, IRIS, and TSCA tools, respectively). Interrater reliability was moderate for all tools (Kappa values= 0.56, 0.58, and 0.54, respectively). Based on the ROB assessments the overall quality of the body of evidence was similar for the OHAT and IRIS tools. All studies were rated ‘unacceptable’ using the TSCA tool because of an ‘unacceptable’ rating in only one metric not related to ROB, including ‘statistical power’ and ‘study design’. Therefore, no studies were retained to make a judgement about the quality of evidence. Conclusions: Using the TSCA tool may significantly reduce the available evidence to assess the harms of hazardous environmental exposures due to only one ‘unacceptable’ criterion. We recommend using the OHAT method when assessing ROB in SRs.