Abstract
BackgroundAs systematic reviews’ limited coverage of the medical literature necessitates decision-making based on unsystematic review, we investigated a possible advantage of systematic review (aside from dataset size and systematic analysis): does systematic review avoid potential bias in sampling primary studies from high impact factor journals? If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than RCTs reported in lower-impact journals, readers who focus on higher-impact journals for their rapid literature reviews may introduce bias which could be mitigated by complete, systematic sampling.MethodsWe randomly sampled Cochrane Library (20 July 2005) treatment reviews that measured mortality as a binary outcome, published in English or French, with at least five RCTs with one or more deaths. Our domain-based assessment of risk of bias included funding source, randomness of allocation sequence, blinding, and allocation concealment. The primary analysis employed logistic regression by a generalized linear model with a generalized estimating equation to estimate the association between various factors and publication in a journal with a high journal impact factor (JIF).ResultsFrom the 29 included systematic reviews, 189 RCTs contributed data. However, in the primary analyses comparing RCT results within meta-analyses, there was no statistically significant association: unadjusted odds of greater than 50% mortality protection in high-JIF (> 5) journals were 1.4 (95% CI 0.42, 4.4) and adjusted, 2.5 (95% CI 0.6, 10). Elements of study quality were weakly, inconsistently, and not statistically significantly correlated with journal impact factor.ConclusionsJournal impact factor may have little to no association with study results, or methodological quality, but the evidence is very uncertain.
Highlights
As systematic reviews’ limited coverage of the medical literature necessitates decision-making based on unsystematic review, we investigated a possible advantage of systematic review: does systematic review avoid potential bias in sampling primary studies from high impact factor journals? If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than Randomized controlled trial (RCT) reported in lower-impact journals, readers who focus on higher-impact journals for their rapid literature reviews may introduce bias which could be mitigated by complete, systematic sampling
Does systematic review avoid bias in selecting a sample to read up on the field of medicine [12, 13]? If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than RCTs reported in lower-impact journals, unsystematic inclusion of higher- over lower-impact journals from their rapid literature reviews may introduce bias, whereas the systematic review’s complete sample frame would protect against biased reading
If there is no significant relationship between journal impact factor (JIF) and effect size estimates, there would be no evidence to support systematic sampling of all studies to avoid bias in the selection of RCTs: a lack of association would support a greater trust in the primary data underpinning lay and rapid literature reviews
Summary
As systematic reviews’ limited coverage of the medical literature necessitates decision-making based on unsystematic review, we investigated a possible advantage of systematic review (aside from dataset size and systematic analysis): does systematic review avoid potential bias in sampling primary studies from high impact factor journals? If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than RCTs reported in lower-impact journals, readers who focus on higher-impact journals for their rapid literature reviews may introduce bias which could be mitigated by complete, systematic sampling. If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than RCTs reported in lower-impact journals, unsystematic inclusion of higher- over lower-impact journals from their rapid literature reviews may introduce bias, whereas the systematic review’s complete sample frame would protect against biased reading. If there is no significant relationship between journal impact factor (JIF) and effect size estimates, there would be no evidence to support systematic sampling of all studies to avoid bias in the selection of RCTs: a lack of association would support a greater trust in the primary data underpinning lay and rapid literature reviews (albeit with the caveats mentioned above in terms of data interpretation). The primary purpose of this study was to determine whether clinical trials’ effect sizes are associated with JIF
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
