Reliability Of Systematic Reviews Research Articles

Glossary for systematic reviews and meta-analyses.

A systematic review aims to answer a focussed research question through a structured review of the evidence, using a predefined methodology, which often includes a meta-analysis. A meta-analysis is a statistical method used to combine the effect estimates from the individual studies included in a systematic review. Systematic reviews and meta-analyses are positioned at the highest level in the hierarchy of clinical evidence. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was introduced in 2009 to help authors improve the quality and reliability of systematic reviews and meta-analyses. Recently, the volume of systematic reviews and meta-analyses in the field of Endodontology has increased; however, the quality of the published manuscripts has been reported to be sub-optimal, which does not take account of the systematic reviews that were rejected because of more obvious deficiencies. The aim of this paper is to present a comprehensive glossary of terminology commonly used in systematic reviews and meta-analyses in an attempt to provide easily understood definitions and explanations to assist authors when reporting systematic reviews and meta-analyses and to allow those wishing to read them to become better informed.

A case study of the organisation of research knowledge by systematic reviews within the mental health care field

Our aim was to explore how systematic reviews of the effects of complex interventions within the field of organisation of mental health care organised research knowledge. We looked up references included in 14 mental health care reviews in the database Epistemonikos to examine overlap between reviews. We classified topic concordance between study reports and reviews as similar, narrower, broader or other topic to indicate how well the research knowledge was organised. We examined 182 comparisons between the theme of study reports and reviews. In 100 (55%) of the 182 comparisons, the review had a broader theme, 18 (10%) narrower, 34 (19%) similar and in 30 (16%), the review's theme was classified as 'other'. The content analysis indicated that there existed (inconsistent) overlapping of study reports between different topics. Within the field organisation of mental health care, there exists some unclear and inconsistent organisation of current knowledge. This may diminish the validity and reliability of systematic reviews. It is important that review authors take care in defining the review question precisely, conduct thorough literature searches, consider each study's hypothesis, disciplinary tradition and context, and if necessary, examine which other reviews have included a study in question.

Core outcome domains for clinical trials in non-specific low back pain.

Inconsistent reporting of outcomes in clinical trials of patients with non-specific low back pain (NSLBP) hinders comparison of findings and the reliability of systematic reviews. A core outcome set (COS) can address this issue as it defines a minimum set of outcomes that should be reported in all clinical trials. In 1998, Deyo et al. recommended a standardized set of outcomes for LBP clinical research. The aim of this study was to update these recommendations by determining which outcome domains should be included in a COS for clinical trials in NSLBP. An International Steering Committee established the methodology to develop this COS. The OMERACT Filter 2.0 framework was used to draw a list of potential core domains that were presented in a Delphi study. Researchers, care providers and patients were invited to participate in three Delphi rounds and were asked to judge which domains were core. A priori criteria for consensus were established before each round and were analysed together with arguments provided by panellists on importance, overlap, aggregation and/or addition of potential core domains. The Steering Committee discussed the final results and made final decisions. A set of 280 experts was invited to participate in the Delphi; response rates in the three rounds were 52, 50 and 45%. Of 41 potential core domains presented in the first round, 13 had sufficient support to be presented for rating in the third round. Overall consensus was reached for the inclusion of three domains in this COS: 'physical functioning', 'pain intensity' and 'health-related quality of life'. Consensus on 'physical functioning' and 'pain intensity' was consistent across all stakeholders, 'health-related quality of life' was not supported by the patients, and all the other domains were not supported by two or more groups of stakeholders. Weighting all possible argumentations, the Steering Committee decided to include in the COS the three domains that reached overall consensus and the domain 'number of deaths'. The following outcome domains were included in this updated COS: 'physical functioning', 'pain intensity', 'health-related quality of life' and 'number of deaths'. The next step for the development of this COS will be to determine which measurement instruments best measure these domains.

Can a core outcome set improve the quality of systematic reviews? – a survey of the Co-ordinating Editors of Cochrane review groups

BackgroundMissing outcome data or the inconsistent reporting of outcome data in clinical research can affect the quality of evidence within a systematic review. A potential solution is an agreed standardized set of outcomes known as a core outcome set (COS) to be measured in all studies for a specific condition. We investigated the amount of missing patient data for primary outcomes in Cochrane systematic reviews, and surveyed the Co-ordinating Editors of Cochrane Review Groups (CRGs) on issues related to the standardization of outcomes in their CRG’s reviews. These groups are responsible for the more than 7,000 protocols and full versions of Cochrane Reviews that are currently available, and the several hundred new reviews published each year, presenting the world’s largest collection of standardized systematic reviews in health care.MethodsUsing an unselected cohort of Cochrane Reviews, we calculated and presented the percentage of missing patient data for the primary outcome measure chosen for each review published by each CRG. We also surveyed the CRG Co-ordinating Editors to see what their policies are with regards to outcome selection and outcomes to include in the Summary of Finding (SoF) tables in their Cochrane Reviews. They were also asked to list the main advantages and challenges of standardizing outcomes across all reviews within their CRG.ResultsIn one fifth of the 283 reviews in the sample, more than 50% of the patient data for the primary outcome was missing. Responses to the survey were received from 90% of Co-ordinating Editors. Thirty-six percent of CRGs have a centralized policy regarding which outcomes to include in the SoF table and 73% of Co-ordinating Editors thought that a COS for effectiveness trials should be used routinely for a SoF table.ConclusionsThe reliability of systematic reviews, in particular meta-analyses they contain, can be improved if more attention is paid to missing outcome data. The availability of COSs for specific health conditions might help with this and the concept has support from the majority of Co-ordinating Editors in CRGs.

Are studies reporting significant results more likely to be published?

Our objective was to assess the hypothesis that there are variations of the proportion of articles reporting a significant effect, with a higher percentage of those articles published in journals with impact factors. The contents of 5 orthodontic journals (American Journal of Orthodontics and Dentofacial Orthopedics, Angle Orthodontist, European Journal of Orthodontics, Journal of Orthodontics, and Orthodontics and Craniofacial Research), published between 2004 and 2008, were hand-searched. Articles with statistical analysis of data were included in the study and classified into 4 categories: behavior and psychology, biomaterials and biomechanics, diagnostic procedures and treatment, and craniofacial growth, morphology, and genetics. In total, 2622 articles were examined, with 1785 included in the analysis. Univariate and multivariate logistic regression analyses were applied with statistical significance as the dependent variable, and whether the journal had an impact factor, the subject, and the year were the independent predictors. A higher percentage of articles showed significant results relative to those without significant associations (on average, 88% vs 12%) for those journals. Overall, these journals published significantly more studies with significant results, ranging from 75% to 90% (P = 0.02). Multivariate modeling showed that journals with impact factors had a 100% increased probability of publishing a statistically significant result compared with journals with no impact factor (odds ratio [OR], 1.99; 95% CI, 1.19-3.31). Compared with articles on biomaterials and biomechanics, all other subject categories showed lower probabilities of significant results. Nonsignificant findings in behavior and psychology and diagnosis and treatment were 1.8 (OR, 1.75; 95% CI, 1.51-2.67) and 3.5 (OR, 3.50; 95% CI, 2.27-5.37) times more likely to be published, respectively. Journals seem to prefer reporting significant results; this might be because of authors' perceptions of the importance of their findings and editors' and reviewers' preferences for significant results. The implication of this factor in the reliability of systematic reviews is discussed.

Editor's Summary and Q&A

<h3>Introduction</h3> Our objective was to assess the hypothesis that there are variations of the proportion of articles reporting a significant effect, with a higher percentage of those articles published in journals with impact factors. <h3>Methods</h3> The contents of 5 orthodontic journals (<i>American Journal of Orthodontics and Dentofacial Orthopedics, Angle Orthodontist, European Journal of Orthodontics, Journal of Orthodontics, and Orthodontics and Craniofacial Research</i>), published between 2004 and 2008, were hand-searched. Articles with statistical analysis of data were included in the study and classified into 4 categories: behavior and psychology, biomaterials and biomechanics, diagnostic procedures and treatment, and craniofacial growth, morphology, and genetics. In total, 2622 articles were examined, with 1785 included in the analysis. Univariate and multivariate logistic regression analyses were applied with statistical significance as the dependent variable, and whether the journal had an impact factor, the subject, and the year were the independent predictors. <h3>Results</h3> A higher percentage of articles showed significant results relative to those without significant associations (on average, 88% vs 12%) for those journals. Overall, these journals published significantly more studies with significant results, ranging from 75% to 90% (<i>P</i> = 0.02). Multivariate modeling showed that journals with impact factors had a 100% increased probability of publishing a statistically significant result compared with journals with no impact factor (odds ratio [OR], 1.99; 95% CI, 1.19-3.31). Compared with articles on biomaterials and biomechanics, all other subject categories showed lower probabilities of significant results. Nonsignificant findings in behavior and psychology and diagnosis and treatment were 1.8 (OR, 1.75; 95% CI, 1.51-2.67) and 3.5 (OR, 3.50; 95% CI, 2.27-5.37) times more likely to be published, respectively. <h3>Conclusions</h3> Journals seem to prefer reporting significant results; this might be because of authors' perceptions of the importance of their findings and editors' and reviewers' preferences for significant results. The implication of this factor in the reliability of systematic reviews is discussed.