Errors In Meta-analysis Research Articles

Meta-analysis of correlation coefficients: A cautionary tale on treating measurement error.

A scale to measure a psychological construct is subject to measurement error. When meta-analyzing correlations obtained from scale scores, many researchers recommend correcting for measurement error. I considered three caveats when correcting for measurement error in meta-analysis of correlations: (a) the distribution of true scores can be non-normal, resulting in violation of the normality assumption for raw correlations and Fisher's z transformed correlations; (b) coefficient alpha is often used as the reliability, but correlations corrected for measurement error using alpha can be inaccurate when some assumptions of alpha (e.g., tau-equivalence) are violated; and (c) item scores are often ordinal, making the disattenuation formula potentially problematic. Via three simulation studies, I examined the performance of two meta-analysis approaches-with raw correlations and z scores. In terms of estimation accuracy and coverage probability of the mean correlation, results showed that (a) considering the true-score distribution alone, estimation of the mean correlation was slightly worse when true scores of the constructs were skewed rather than normal; (b) when the tau-equivalence assumption was violated and coefficient alpha was used for correcting measurement error, the mean correlation estimates can be biased and coverage probabilities can be low; and (c) discretization of continuous items can result in biased estimates and undercoverage of the mean correlations even when tau-equivalence was satisfied. With more categories and/or items on a scale, results can improve whether tau-equivalence was met or not. Based on these findings, I gave recommendations for conducting meta-analyses of correlations. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

Background : Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective : To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods : We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion : The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol.

Background: Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective: To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods: We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion: The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

Bias caused by sampling error in meta-analysis with small sample sizes.

BackgroundMeta-analyses frequently include studies with small sample sizes. Researchers usually fail to account for sampling error in the reported within-study variances; they model the observed study-specific effect sizes with the within-study variances and treat these sample variances as if they were the true variances. However, this sampling error may be influential when sample sizes are small. This article illustrates that the sampling error may lead to substantial bias in meta-analysis results.MethodsWe conducted extensive simulation studies to assess the bias caused by sampling error. Meta-analyses with continuous and binary outcomes were simulated with various ranges of sample size and extents of heterogeneity. We evaluated the bias and the confidence interval coverage for five commonly-used effect sizes (i.e., the mean difference, standardized mean difference, odds ratio, risk ratio, and risk difference).ResultsSampling error did not cause noticeable bias when the effect size was the mean difference, but the standardized mean difference, odds ratio, risk ratio, and risk difference suffered from this bias to different extents. The bias in the estimated overall odds ratio and risk ratio was noticeable even when each individual study had more than 50 samples under some settings. Also, Hedges’ g, which is a bias-corrected estimate of the standardized mean difference within studies, might lead to larger bias than Cohen’s d in meta-analysis results.ConclusionsCautions are needed to perform meta-analyses with small sample sizes. The reported within-study variances may not be simply treated as the true variances, and their sampling error should be fully considered in such meta-analyses.

Patient-controlled analgesia with remifentanil versus alternative parenteral methods for pain management in labour.

Patient-controlled analgesia with remifentanil versus alternative parenteral methods for pain management in labour.

BinomiRare: A robust test of the association of a rare variant with a disease for pooled analysis and meta-analysis, with application to the HCHS/SOL.

Most regression-based tests of the association between a low-count variant and a binary outcome do not protect type 1 error, especially when tests are rejected based on a very low significance threshold. Noted exception is the Firth test. However, it was recently shown that in meta-analyzing multiple studies all asymptotic, regression-based tests, including the Firth, may not control type 1 error in some settings, and the Firth test may suffer a substantial loss of power. The problem is exacerbated when the case-control proportions differ between studies. I propose the BinomiRare exact test that circumvents the calibration problems of regression-based estimators. It quantifies the strength of association between the variant and the disease outcome based on the departure of the number of diseased individuals carrying the variant from the expected distribution of disease probability, under the null hypothesis of no association between the disease outcome and the rare variant. I provide a meta-analytic strategy to combine tests across multiple cohorts, which requires that each cohort provides the disease probabilities of all carriers of the variant in question, and the number of diseased individuals among the carriers. I show that BinomiRare controls type 1 error in meta-analysis even when the case-control proportions differ between the studies, and does not lose power compared to pooled analysis. I demonstrate the test in studying the association of rare variants with asthma in the Hispanic Community Health Study/Study of Latinos.

A Simplification and Implementation of Random-effects Meta-analyses Based on the Exact Distribution of Cochran’s Q

The random-effects (RE) model is the standard choice for meta-analysis in the presence of heterogeneity, and the standard RE method is the DerSimonian and Laird (DSL) approach, where the degree of heterogeneity is estimated using a moment-estimator. The DSL approach does not take into account the variability of the estimated heterogeneity variance in the estimation of Cochran's Q. Biggerstaff and Jackson derived the exact cumulative distribution function (CDF) of Q to account for the variability of τ^². The first objective is to show that the explicit numerical computation of the density function of Cochran's Q is not required. The second objective is to develop an R package with the possibility to easily calculate the classical RE method and the new exact RE method. The novel approach was validated in extensive simulation studies. The different approaches used in the simulation studies, including the exact weights RE meta-analysis, the I² and τ² estimates together with their confidence intervals were implemented in the R package metaxa. The comparison with the classical DSL method showed that the exact weights RE meta-analysis kept the nominal type I error level better and that it had greater power in case of many small studies and a single large study. The Hedges RE approach had inflated type I error levels. Another advantage of the exact weights RE meta-analysis is that an exact confidence interval for τ² is readily available. The exact weights RE approach had greater power in case of few studies, while the restricted maximum likelihood (REML) approach was superior in case of a large number of studies. Differences between the exact weights RE meta-analysis and the DSL approach were observed in the re-analysis of real data sets. Application of the exact weights RE meta-analysis, REML, and the DSL approach to real data sets showed that conclusions between these methods differed. The simplification does not require the calculation of the density of Cochran's Q, but only the calculation of the cumulative distribution function, while the previous approach required the computation of both the density and the cumulative distribution function. It thus reduces computation time, improves numerical stability, and reduces the approximation error in meta-analysis. The different approaches, including the exact weights RE meta-analysis, the I² and τ² estimates together with their confidence intervals are available in the R package metaxa, which can be used in applications.

Conclusive meta-analyses on antenatal magnesium may be inconclusive! Are we underestimating the risk of random error?

Results from meta-analyses significantly influence clinical practice. Both simulation and empirical studies have demonstrated that the risk of random error (i.e. spurious chance findings) in meta-analyses is much higher than previously anticipated. Hence, authors and users of systematic reviews and meta-analyses have a responsibility to carefully consider the risk of random errors to avoid misleading conclusions. Trial sequential analysis is a useful meta-analytic method for gauging the risk of random error in meta-analysis and the amount of additional evidence required to reach firm conclusions about the investigated intervention effect(s). We outline the rationale for conducting trial sequential analysis including some examples of the meta-analysis on antenatal magnesium for women at risk of preterm birth.

Humans and models: converging ‘truths’

INTERNATIONAL JOURNAL OF EPIDEMIOLOGY rejected. To inspire his followers, Morris had attached his two exceedingly long and aggressive reviews of our paper (12 858 words and 5291 words, respect- ively), calling for critical letters in abundance to the IJE editors. Breaking unwritten confidentiality and courtesy rules of the peer-review process, Morris dis- tributed his slandering criticism of our study to people working for the same cause. Rather than resorting to such selective distribution among friends, Morris should make both reviews freely available on the internet by posting them in their entirety on his pro circumcision homepage (www.circinfo.net). Alternatively, interested readers should feel free to request them from me at the e-mail address above. Despite poorly founded criticisms and attempts at obstruction our findings suggest that male circumci- sion may be associated with hitherto unappreciated negative sexual consequences in a non-trivial propor- tion of men and women. Further carefully conducted studies are needed. References Morris BJ. Why circumcision is a biomedical imperative for the 21st century. Bioessays 2007;29:1147–58. Perneger TV. What’s wrong with Bonferroni adjustments. Br Med J 1998;316:1236–38. Barros AJ, Hirakata VN. Alternatives for logistic regres- sion in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Methodol 2003;3:21. Krieger JN, Mehta SD, Bailey RC et al. Adult male cir- cumcision: effects on sexual function and sexual satisfac- tion in Kisumu, Kenya. J Sex Med 2008;5:2610–22. Kigozi G, Watya S, Polis CB et al. The effect of male cir- cumcision on sexual satisfaction and function, results from a randomized trial of male circumcision for human immunodeficiency virus prevention, Rakai, Uganda. BJU Int 2008;101:65–70. Circleaks.http://circleaks.org/index.php?title¼Brian_Morr is (8 August 2011, date last accessed). Circleaks.http://circleaks.org/index.php?title¼Jake_H._ Waskett (8 August 2011, date last accessed). Morris BJ. Renin, genes, and beyond: 40 years of molecu- lar discoveries in the hypertension field. Hypertension YouTube.http://www.youtube.com/v/7yDvL4hNny4 August 2011, date last accessed). Morris BJ, Wodak A. Circumcision survey misleading. Aust N Z J Public Health 2010;34:636–37. Waskett JH, Morris BJ, Weiss HA. Errors in meta-analysis by Van Howe. Int J STD AIDS 2009;20:216–18. Waskett JH, Morris BJ. Fine-touch pressure thresholds in the adult penis. BJU Int 2007;99:1551–52. doi:10.1093/ije/dyr181 Advance Access publication 28 November 2011 s The Author 2011; all rights reserved. Humans and models: converging ‘truths’ From TIM A BRUCKNER 1 * and CLAIRE MARGERISON-ZILKO 2 Program in Public Health & Department of Planning, Policy, and Design, University of California, Irvine, CA, USA and Population Research Center & Center for Social Work Research, University of Texas, Austin, TX, USA *Corresponding author. Program in Public Health & Department of Planning, Policy, and Design, University of California, 202 Social Ecology I, Irvine, CA, 92697, USA. E-mail: tim.bruckner@uci.edu We thank Jay Kaufman for his thoughtful commen- tary 1 regarding our recent manuscript in which we reported a positive relation between acute income gains and accidental deaths among Cherokee Indians in rural North Carolina. 2 Although we agree with many of Kaufman’s points, we would like to respond to a key question that holds relevance to most analyses using time series data: how should epi- demiologists approximate the counterfactual value of a population exposed at a specific point in time? One approach to deriving counterfactual values in time involves using a model-based framework. In our analysis of the Cherokee response to acute and large cash disbursements from a local Casino, we em- ployed a Poisson regression with a conventional log-linear functional form. 3 To control for confound- ing by temporal patterns in accidental deaths (e.g. seasonality), we included indicator variables for cal- endar months and years. Identification of an effect of the Casino payments on accidental deaths, therefore, relies on a systematic deviation—in the 20 exposed months of the Casino disbursements—above expected values derived from the specific underlying (multi- plicative) functional form of accidental deaths. The analyst, of course, could impose different model- based assumptions regarding temporality of accidental deaths (e.g. additivity), yet Kaufman notes the prob- lem of insufficient statistical power to test such as- sumptions. In other words, the analyst has limited ability to detect what we call ‘wrong model bias’ in the functional form of a time series. An alternative approximation of the counterfactual value, which Kaufman suggests, involves using a comparison series of accidental deaths among

Trial sequential analyses of meta-analyses of complications in laparoscopic vs. small-incision cholecystectomy: more randomized patients are needed

Objective Conclusions based on meta-analyses of randomized trials carry a status of “truth.” Methodological components may identify trials with systematic errors (“bias”). Trial sequential analysis (TSA) evaluates random errors in meta-analysis. We analyzed meta-analyses on laparoscopic vs. small-incision cholecystectomy regarding different outcome measures for the occurrence of type I errors. Study Design and Setting Using TSA, we calculated the required information size (IS) and the trial sequential monitoring boundaries regarding complications in our Cochrane review with meta-analyses of cholecystectomy. For each outcome, we calculated a low risk of bias heterogeneity-adjusted IS. As a sensitivity analysis, we calculated an a priori heterogeneity-adjusted IS. Results According to the trial sequential analyses based on a low risk of bias heterogeneity-adjusted IS definitive evidence may be reached by conducting one more randomized trial. Information may be required on 582 and 119 additional randomized patients to evaluate the effect on severe complications and serious adverse events (SAEs), respectively. Conclusion Our results provide incentives to conduct a new trial with a low risk of bias focusing on a new composite outcome measure of SAEs to obtain conclusive evidence on which operative method to recommend.

Errors in meta-analysis by Van Howe

Errors in meta-analysis by Van Howe

Revisiting youden's index as a useful measure of the misclassification error in meta-analysis of diagnostic studies

The paper considers meta-analysis of diagnostic studies that use a continuous score for classification of study participants into healthy or diseased groups. Classification is often done on the basis of a threshold or cut-off value, which might vary between studies. Consequently, conventional meta-analysis methodology focusing solely on separate analysis of sensitivity and specificity might be confounded by a potentially unknown variation of the cut-off value. To cope with this phenomena it is suggested to use, instead, an overall estimate of the misclassification error previously suggested and used as Youden's index and; furthermore, it is argued that this index is less prone to between-study variation of cut-off values. A simple Mantel-Haenszel estimator as a summary measure of the overall misclassification error is suggested, which adjusts for a potential study effect. The measure of the misclassification error based on Youden's index is advantageous in that it easily allows an extension to a likelihood approach, which is then able to cope with unobserved heterogeneity via a nonparametric mixture model. All methods are illustrated at hand of an example on a diagnostic meta-analysis on duplex doppler ultrasound, with angiography as the standard for stroke prevention.

Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies

Miller and Rosenstein [1] conclude that “evidence from placebo analgesia experiments strongly supports the reality of the placebo effect.” Our main concern is whether there is reliable evidence for clinically relevant effects of placebo interventions, not whether a subclinical effect exists. In addition, the meta-analysis on which Miller and Rosenstein base their arguments is seriously flawed.