Abstract
An important question for clinicians appraising a meta‐analysis is: are the findings likely to be valid in their own practice—does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity—where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple (‘leave‐one‐out’) cross‐validation technique, we demonstrate how we may test meta‐analysis estimates for statistical validity using a new validation statistic, Vn, and derive its distribution.We compare this with the usual approach of investigating heterogeneity in meta‐analyses and demonstrate the link between statistical validity and homogeneity. Using a simulation study, the properties of Vn and the Q statistic are compared for univariate random effects meta‐analysis and a tailored meta‐regression model, where information from the setting (included as model covariates) is used to calibrate the summary estimate to the setting of application. Their properties are found to be similar when there are 50 studies or more, but for fewer studies Vn has greater power but a higher type 1 error rate than Q. The power and type 1 error rate of Vn are also shown to depend on the within‐study variance, between‐study variance, study sample size, and the number of studies in the meta‐analysis. Finally, we apply Vn to two published meta‐analyses and conclude that it usefully augments standard methods when deciding upon the likely validity of summary meta‐analysis estimates in clinical practice. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.
Highlights
The capacity to aggregate multiple studies and provide a summary estimate for translation into practice was one of the motivations that drove the development of meta-analysis
In this article we propose a general method for assessing the statistical validity of meta-analysis results when applied in clinical practice
The predominant question we aim to address is when should we apply a summary meta-analysis estimate to an independent setting? if μma is the parameter for the true summary effect of interest as estimated by the meta-analysis analysis model and μsetting is the parameter for the true effect in an independent setting of interest, does μma = μsetting? When the two are equal, we propose that the summary estimate from the meta-analysis model can be described as having statistical validity
Summary
The capacity to aggregate multiple studies and provide a summary estimate for translation into practice was one of the motivations that drove the development of meta-analysis In this regard, it has achieved undoubted success; the blight of heterogeneity, which so often affects meta-analyses, can potentially affect the applicability of results in individual clinical settings, such as individual practices, hospitals, regions, or even countries. Assessing whether meta-analysis results translate into practice should be the concern of all reviewers and statisticians producing summary results from a body of evidence, whether it is for the purpose of diagnosis, treatment, prognosis, or otherwise With this in mind, in this article we propose a general method for assessing the statistical validity of meta-analysis results when applied in clinical practice.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
