Publication Bias In Meta-analysis Research Articles

Approaches for identifying and managing publication bias in meta-analysis

The consequences of publication bias in meta-analysis pose significant risks, potentially leading to erroneous conclusions within the meta-analytic framework. The objective of this article was to explore the methodologies for identifying publication bias and approaches for mitigating its effects. The techniques employed to detect publication bias can generally be distinguished into two major categories: graphical and statistical methodologies. Graphical approaches utilize techniques such as funnel plots and meta-plots, which visually depict the distribution of effect sizes and standard errors across studies. Statistical methods encompass various computations, including Fail-Safe N, rank correlation, Egger regression, tests for excess significance (TES), and selection models tailored for evaluating publication bias through quantitative analyses. The combination of these methods is recommended for a more comprehensive assessment, rather than relying on individual approaches. Methods for addressing publication bias include the trim and fill (T&F) method, Publication Error and True Effect Size Estimation (PET-PEESE) method, and the Weight-Function Model, each offering unique strategies for adjusting effect size estimates. The selection of these methods should consider the specific characteristics of the meta-analysis under consideration, ensuring the most appropriate approach is employed. Publication bias poses a significant risk in the field of meta-analysis, and selecting methods for its identification and mitigation requires comprehensive consideration

Sensitivity analysis for the interactive effects of internal bias and publication bias in meta-analyses.

Meta-analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst-case" form of publication bias. Robust estimation methods accommodate non-normal effects, small meta-analyses, and clustered estimates. As we illustrate by re-analyzing published meta-analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).

DTAmetasa: An R shiny application for meta-analysis of diagnostic test accuracy and sensitivity analysis of publication bias.

Meta-analysis of diagnostic test accuracy (DTA) is a powerful statistical method for synthesizing and evaluating the diagnostic capacity of medical tests and has been extensively used by clinical physicians and healthcare decision-makers. However, publication bias (PB) threatens the validity of meta-analysis of DTA. Some statistical methods have been developed to deal with PB in meta-analysis of DTA, but implementing these methods requires high-level statistical knowledge and programming skill. To assist non-technical users in running most routines in meta-analysis of DTA and handling with PB, we developed an interactive application, DTAmetasa. DTAmetasa is developed as a web-based graphical user interface based on the R shiny framework. It allows users to upload data and conduct meta-analysis of DTA by "point and click" operations. Moreover, DTAmetasa provides the sensitivity analysis of PB and presents the graphical results to evaluate the magnitude of the PB under various publication mechanisms. In this study, we introduce the functionalities of DTAmetasa and use the real-world meta-analysis to show its capacity for dealing with PB.

Publication bias in meta-analyses of the therapeutic efficacy of remdesivir interventions for patients with COVID-19

Purpose No comprehensive statistical assessment of publication bias has been conducted in remdesivir-based intervention research for COVID-19 patients. This study aims to examine all meta-analyses of the efficacy of remdesivir interventions in COVID-19 patients and perform a statistical assessment of publication bias. Design/methodology/approach This is an analytic study conducted to assess the impact of publication bias on the results of meta-analyses of remdesivir-based interventions in patients infected with COVID-19. All English full-text meta-analyses published in peer-reviewed journals in 2019–2021 were included. A computerized search of PubMed and Web of Science electronic databases was performed on December 24, 2021. The trim-and-fill method calculated the number of missing studies and the adjusted cumulative effect sizes. Findings The final analysis comprised 21 studies with 88 outcomes. The investigation revealed missing studies in 46 outcomes (52%). Seventy-six missing studies were replaced in the outcomes using the trim-and-fill procedure. The adjusted recalculated effect sizes of the 27 outcomes increased by an average of 0.04. In comparison, the adjusted effect size of 18 outcomes fell by an average of 0.036. Only 14 out of 46 outcomes with publication bias were subjected to a gray literature search (30%). To discover related research, no gray literature search was conducted in most outcomes with publication bias (n = 32; 70%). In conclusion, the reported effect estimates regarding the effect of remdesivir in COVID-19 patients are only slightly affected by publication bias and can be considered authentic. Health-care decision-makers in COVID-19 should consider current research results when making clinical decisions. Research limitations/implications Most health decisions are based on the effect sizes revealed in meta-analyses. When deciding on remdesivir-based treatment for COVID-19 patients, therefore, the outcomes of this investigation may be of paramount importance to health policymakers, leading to better treatment strategies. Practical implications According to the results, no major publication bias and missing studies were detected on average. Therefore, the calculated effect sizes of remdesivir-based interventions on meta-analyses can be used as authentic and unbiased benchmarks by health-care decision-makers in treating patients with COVID-19. Originality/value This is the first study to examine the effect of publication bias and gray literature searches on the results of meta-analyses of treatment with COVID-19 (remdesivir).

An artificial neural network (ANN) model for publication bias: a machine learning-based study on PubMed meta-analyses

PurposeNo study has investigated the effects of different parameters on publication bias in meta-analyses using a machine learning approach. Therefore, this study aims to evaluate the impact of various factors on publication bias in meta-analyses.Design/methodology/approachAn electronic questionnaire was created according to some factors extracted from the Cochrane Handbook and AMSTAR-2 tool to identify factors affecting publication bias. Twelve experts were consulted to determine their opinion on the importance of each factor. Each component was evaluated based on its content validity ratio (CVR). In total, 616 meta-analyses comprising 1893 outcomes from PubMed that assessed the presence of publication bias in their reported outcomes were randomly selected to extract their data. The multilayer perceptron (MLP) technique was used in IBM SPSS Modeler 18.0 to construct a prediction model. 70, 15 and 15% of the data were used for the model's training, testing and validation partitions.FindingsThere was a publication bias in 968 (51.14%) outcomes. The established model had an accuracy rate of 86.1%, and all pre-selected nine variables were included in the model. The results showed that the number of databases searched was the most important predictive variable (0.26), followed by the number of searches in the grey literature (0.24), search in Medline (0.17) and advanced search with numerous operators (0.13).Practical implicationsThe results of this study can help clinical researchers minimize publication bias in their studies, leading to improved evidence-based medicine.Originality/valueTo the best of the author’s knowledge, this is the first study to model publication bias using machine learning.

Adjusting for publication bias in meta-analysis via inverse probability weighting using clinical trial registries.

Publication bias is a major concern in conducting systematic reviews and meta-analyses. Various sensitivity analysis or bias-correction methods have been developed based on selection models, and they have some advantages over the widely used trim-and-fill bias-correction method. However, likelihood methods based on selection models may have difficulty in obtaining precise estimates and reasonable confidence intervals, or require a rather complicated sensitivity analysis process. Herein, we develop a simple publication bias adjustment method by utilizing the information on conducted but still unpublished trials from clinical trial registries. We introduce an estimating equationfor parameter estimation in the selection function by regarding the publication bias issue as a missing data problem under the missing not at random assumption. With the estimated selection function, we introduce the inverse probability weighting (IPW) method to estimate the overall mean across studies. Furthermore, the IPW versions of heterogeneity measures such as the between-study variance and the I2 measure are proposed. We propose methods to construct confidence intervals based on asymptotic normal approximation as well as on parametric bootstrap. Through numerical experiments, we observed that the estimators successfully eliminated bias, and the confidence intervals had empirical coverage probabilities close to the nominal level. On the other hand, the confidence interval based on asymptotic normal approximation is much wider in some scenarios than the bootstrap confidence interval. Therefore, the latter is recommended for practicaluse.

Let's decide what would be convincing, conduct randomized trials with rigorous comparison conditions, and report tests of moderation and publication bias in meta-analyses.

We appreciate Jacobson and colleagues' thoughtful commentary on our meta-review of mobile phone-based interventions for mental health. In this response, we address two issues raised: requiring low to moderate heterogeneity (I2<50%) and requiring no evidence of publication bias for evidence to be classified as "convincing." While we agree these represent a high bar, we disagree that these requirements are destined to fail. Other effect sizes reported in the literature, including effect sizes related to mental health interventions and effect sizes related to mobile health (mHealth) interventions (although not their combination) have met requirements for convincing evidence. Jacobson and colleagues argue that features of the mHealth interventions may produce heterogeneity when meta-analyses combine across intervention types. However, several of the effect sizes we reviewed were based on relatively homogeneous portions of the literature and many of the effect sizes we reviewed showed low to moderate heterogeneity. Ideally, future meta-analyses will examine intervention features as moderators of treatment effects. While an absence of publication bias may be a stringent criterion, all but two of the 34 effect sizes we reviewed did not report formal tests of publication bias. Clearly there is a need to reach consensus on how the strength of evidence for mHealth interventions can be evaluated. From our perspective, convincing evidence will ultimately come from large-scale randomized controlled trials employing rigorous comparison conditions along with meta-analyses that do not combine across control condition types, that examine theoretically important moderators, and report formal tests of publication bias. It is this kind of evidence that the public, clinicians, and the scientific community may need to encourage adoptions of mHealth interventions for mental health treatment and prevention.

Associations Between Wearing Masks and Respiratory Viral Infections: A Meta-Analysis and Systematic Review.

BackgroundRespiratory viral infections (RVIs) are a major health concern, and some previous studies have shown that wearing masks was effective in preventing RVIs, while others failed to show such effect. Therefore, a systematic review and meta-analysis was conducted to investigate the effectiveness of wearing masks.MethodsPubMed, ScienceDirect, Web of Science, the Cochrane Library, EMBASE, MEDLINE, China National Knowledge Infrastructure (CNKI), and Chinese Scientific Journal Database (VIP database) were searched for studies evaluating the effectiveness of wearing masks. The risk ratio (RR) was used to measure the effectiveness of wearing masks in preventing RVIs for randomized controlled trials (RCTs) and cohort studies, and the odds ratio (OR) was used for case-control studies. Forest plots were used to visually assess pooled estimates and corresponding 95% CIs. The I2 test was used to examine the heterogeneity, and subgroup analysis was used to explore the possible explanations for heterogeneity or compare the results between subgroups. Sensitivity analysis was conducted to assess robustness of the synthesized results. Begg's test and Egger's test were used to assess the publications bias.ResultsThirty-one studies (13,329 participants) were eligible for meta-analyses. Overall, the results showed that wearing masks was effective in preventing RVIs. The sensitivity analysis showed that the results of those meta-analyses were robust and reliable. There was no significant publication bias in meta-analysis of case-control studies and most subgroup analyses.ConclusionsWearing masks might be effective in preventing RVIs. To reduce their RVI risk, people should wear masks when they go out in public.Systematic Review Registrationhttps://www.crd.york.ac.uk/PROSPERO/, identifier: CRD42021296092.

Antiepileptic drugs for the primary and secondary prevention of seizures after stroke.

Seizures after stroke are an important clinical problem and may result in poor outcomes. The indications of antiepileptic drugs (AEDs) for seizure prophylaxis after stroke remain unclear. This is an updated version of the Cochrane Review previously published in 2014. To assess the effects of AEDs for the primary and secondary prevention of seizures after stroke. For primary prevention, we aimed to assess whether AEDs reduce the likelihood of seizures in people who have a stroke but do not have a seizure. For secondary prevention, we aimed to assess whether AEDs reduce the likelihood of further seizures in people who have a stroke and at least one post-stroke seizure. We searched the following databases on 9 March 2021: Cochrane Register of Studies (CRS Web), MEDLINE (Ovid, 1946 to March 08, 2021). CRS Web includes randomised or quasi-randomised controlled trials from PubMed, Embase, ClinicalTrials.gov, the World Health Organisation International Clinical Trials Registry Platform (ICTRP), the Cochrane Central Register of Controlled Trials (CENTRAL), and the Specialised Registers of Cochrane Review Groups including Epilepsy and Stroke. We also checked the reference lists of articles retrieved from these searches. We selected randomised and quasi-randomised controlled studies that recruited participants with a clinical diagnosis of stroke, either ischaemic or haemorrhagic. We excluded studies that only recruited participants with subarachnoid haemorrhage, subdural haemorrhage, extradural haemorrhage, or other non-stroke diagnoses such as tumour- or infection-related infarction or haemorrhage. We also excluded studies that recruited only participants who had undergone neurosurgery. We included participants of all ages suffering any seizure type who were assigned to AEDs or placebo groups. In accordance with standard methodological procedures expected by The Cochrane Collaboration, two review authors independently assessed trials for inclusion before evaluating trial risk of bias and extracting relevant data. The primary outcome assessed was the proportion of participants who experienced seizures in the follow-up period. We presented results as summary risk ratios (RRs) with 95% confidence intervals (CIs) for dichotomous outcomes and mean differences (MDs) with 95% CIs for continuous outcomes. Where we had sufficient data, we calculated random-effects (Mantel-Haenszel) meta-analyses for dichotomous outcomes; otherwise, we reported results narratively. We used the I2 statistic to analyse statistical heterogeneity. We planned to use funnel plots to assess publication bias in meta-analyses with at least 10 included studies. We used the GRADE approach to assess the certainty of the evidence. Two studies with a total of 856 subjects were included. AEDs were not shown to be effective in primary prophylaxis of post-stroke seizure (RR 0.65, 95% CI 0.34 to 1.26; 2 studies, 856 participants; moderate-certainty evidence). The first study was a randomised double-blind study comparing valproic acid with placebo for primary seizure prevention up to one year after stroke. The study included 72 adults with intracerebral haemorrhage. There was no difference in the risk of post-stroke seizures (RR 0.88, 95% CI 0.35 to 2.16) or of death (RR 1.20, 95% CI 0.40 to 3.58). The second study was a substudy on the use of diazepam in acute stroke. It was a randomised double-blind study, comparing a three-day diazepam treatment versus placebo for primary seizure prevention up to three months after stroke in 784 adults with acute stroke. There was no evidence of a difference in the risk of post-stroke seizures for all stroke or subgroups of haemorrhagic or ischaemic stroke (RR for all stroke 0.47, 95% CI 0.18 to 1.22). In a subgroup analysis of anterior circulation cortical infarcts, primary prophylaxis with diazepam was associated with a reduced risk of post-stroke seizures (RR 0.21, 95% CI 0.05 to 0.95). Risks of mortality did not differ between the diazepam and the placebo group at two weeks (RR 0.84, 95% CI 0.56 to 1.26) and three months follow-up (RR 0.95, 95% CI 0.72 to 1.26). We assessed both studies to be at a low overall risk of bias. Using the GRADE approach, we assessed the overall certainty of the evidence as low to moderate. There is insufficient evidence to support the routine use of AEDs on the primary and secondary prevention of seizures after stroke. Further well-conducted studies are warranted for this important clinical problem.

Publication bias: A “bird's-eye view” of meta-analytic practice in criminology and criminal justice

ObjectivesThe aims of the current study were 2-folded. This study examined the recent meta-analyses in the criminology and criminal justice (CCJ) field to capture a snapshot of the current state of publication bias practices and provided practical guidelines and recommendations on how to address the issues of publication bias in meta-analytic reviews. MethodsThe content analysis reviewed 64 meta-analyses published in top-tier journals in the CCJ field in 2019–2020. The narrative review of previous simulation studies in medical science and psychology fields was performed to synthesize practical guidelines on publication bias in meta-analyses. ResultsRecent CCJ meta-analytic studies have at least partially addressed the issue of publication bias by employing systematic search and statistical methods. However, the current state of CCJ meta-analyses does not meet the expectation required in medical science and psychology that all meta-analytic reviews report the range of effect size estimates across multiple publication bias detection and correction tests. The statistical methods commonly used for assessing publication bias are applied without testing and interpreting assumptions about the missing studies. ConclusionsThere is a need to continue monitoring the quality of meta-analyses to gain a comprehensive picture of how bias leaves a potential imprint in CCJ research.

Publication bias in meta-analysis

Publication bias in meta-analysis

Using clinical trial registries to inform Copas selection model for publication bias in meta-analysis.

Prospective registration of study protocols in clinical trial registries is a useful way to minimize the risk of publication bias in meta-analysis, and several clinical trial registries are available nowadays. However, they are mainly used as a tool for searching studies and information submitted to the registries has not been utilized as efficiently as it could. In addressing publication bias in meta-analyses, sensitivity analysis with the Copas selection model is a more objective alternative to widely-used graphical methods such as the funnel-plot and the trim-and-fill method. Despite its ability to quantify the potential impact of publication bias, the Copas selection model relies on sensitivity analyses, in which some parameters are varied across a certain range. This may result in some difficulty in interpreting the results. In this paper, we propose an alternative inference procedure for the Copas selection model by utilizing information from clinical trial registries. Our method provides a simple and accurate way to estimate all unknown parameters of the Copas selection model. A simulation study revealed that our proposed method resulted in smaller biases and more accurate confidence intervals than existing methods. Furthermore, three published meta-analyses were re-analyzed to demonstrate how to implement the proposed method in practice.

Tests for publication bias are unreliable in case of heteroscedasticity

Regression based methods for the detection of publication bias in meta-analysis have been extensively evaluated in literature. When dealing with continuous outcomes, specific hidden factors (e.g., heteroscedasticity) may interfere with the test statistics. In this paper we investigate the influence of residual heteroscedasticity on the performance of four tests for publication bias: the Egger test, the Begg-Mazumdar test and two tests based on weighted regression. In the presence of heteroscedasticity, the Egger test and the weighted regression tests highly inflate the Type I error rate, while the Begg-Mazumdar test deflates the Type I error rate. Although all three tests already have low statistical power, heteroscedasticity typically reduces it further. Our results in combination with earlier discussions on publication bias tests lead us to conclude that application of these tests on continuous treatment effects is not warranted.

Inhaled nitric oxide and acute kidney injury risk: a meta-analysis of randomized controlled trials

Purpose There are conflicting results as to the effect of inhaled nitric oxide (iNO) therapy on the risk of acute kidney injury (AKI). The aim of this study was to perform a meta-analysis to assess the updated data. Methods We systematically searched Web of Science, the Cochrane Library, Wanfang, and PubMed for relevant randomized control trials between database inception and 9/07/2020. Relative risks (RRs) with 95% confidence intervals (CIs) predicting the risk of AKI were extracted to obtain summary estimates using fixed-effects models. The Trim and Fill method was used to evaluate the sensitivity of the results and adjust for publication bias in meta-analysis. Results 15 randomized controlled studies from 14 articles involving 1853 patients were included in the study. Analyzing the eligible studies we found: (1) iNO therapy significantly increased the risk of AKI in acute respiratory distress syndrome patients (RR 1.55, 95% CI 1.15–2.10, p = 0.004; I 2 for heterogeneity 0%; P het = 0.649). (2) The use of iNO was associated with reduced AKI risk in patients undergoing cardiac surgery (RR 0.80, 95% CI 0.64–0.99, p = 0.037; I 2 for heterogeneity 0%; P het = 0.528). (3) For organ transplantation recipients, there was no effect of iNO administration on the risk of AKI (RR 0.50, 95% CI 0.16–1.56, p = 0.233; I 2 for heterogeneity 0%; P het = 0.842). The Trim and Fill analysis showed that the overall effect of this meta-analysis was stable. Conclusions The effect of iNO on AKI risk might be disease-specific. Future RCTs with larger patient populations should aim to validate our findings.

Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers.

Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in PLoS One, top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each, including only the peer-reviewed studies. Across all meta-analyses, we estimated that "statistically significant" results in the expected direction were only 1.17 times more likely to be published than "nonsignificant" results or those in the unexpected direction (95% CI: [0.93, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in PLoS One, 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than threefold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication bias estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures ("statistical significance" with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, we conclude that the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results.

Publication Bias in Meta-Analyses of Posttraumatic Stress Disorder Interventions

Meta-analyses are susceptible to publication bias, the selective publication of studies with statistically significant results. If publication bias is present in psychotherapy research, the efficacy of interventions will likely be overestimated. This study has two aims: (1) investigate whether the application of publication bias methods is warranted in psychotherapy research on posttraumatic stress disorder (PTSD) and (2) investigate the degree and impact of publication bias in meta-analyses of the efficacy of psychotherapeutic treatment for PTSD. A comprehensive literature search was conducted and 26 meta-analyses were eligible for bias assessment. A Monte-Carlo simulation study closely resembling characteristics of the included meta-analyses revealed that statistical power of publication bias tests was generally low. Our results showed that publication bias tests had low statistical power and yielded imprecise estimates corrected for publication bias due to characteristics of the data. We recommend to assess publication bias using multiple publication bias methods, but only include methods that show acceptable performance in a method performance check that researchers first have to conduct themselves.

Sensitivity analysis for publication bias in meta-analyses.

SummaryWe propose sensitivity analyses for publication bias in meta‐analyses. We consider a publication process such that ‘statistically significant’ results are more likely to be published than negative or “non‐significant” results by an unknown ratio, η. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse probability weighting and robust estimation that accommodates non‐normal population effects, small meta‐analyses, and clustering, we develop sensitivity analyses that enable statements such as ‘For publication bias to shift the observed point estimate to the null, “significant” results would need to be at least 30 fold more likely to be published than negative or “non‐significant” results’. Comparable statements can be made regarding shifting to a chosen non‐null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of η across disciplines. We show that a worst‐case meta‐analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta‐analysis of only the negative and ‘non‐significant’ studies; this method sometimes indicates that no amount of such publication bias could ‘explain away’ the results. We illustrate the proposed methods by using real meta‐analyses and provide an R package: PublicationBias.

Hybrid test for publication bias in meta-analysis.

Publication bias frequently appears in meta-analyses when the included studies' results (e.g., p-values) influence the studies' publication processes. Some unfavorable studies may be suppressed from publication, so the meta-analytic results may be biased toward an artificially favorable direction. Many statistical tests have been proposed to detect publication bias in recent two decades. However, they often make dramatically different assumptions about the cause of publication bias; therefore, they are usually powerful only in certain cases that support their particular assumptions, while their powers may be fairly low in many other cases. Although several simulation studies have been carried out to compare different tests' powers under various situations, it is typically infeasible to justify the exact mechanism of publication bias in a real-world meta-analysis and thus select the corresponding optimal publication bias test. We introduce a hybrid test for publication bias by synthesizing various tests and incorporating their benefits, so that it maintains relatively high powers across various mechanisms of publication bias. The superior performance of the proposed hybrid test is illustrated using simulation studies and three real-world meta-analyses with different effect sizes. It is compared with many existing methods, including the commonly used regression and rank tests, and the trim-and-fill method.

Not Published Is Not Perished: Addressing Publication Bias in Meta-Analytic Studies in Communication

AbstractPublication bias has been recognized as a threat to the validity of meta-analytic findings and scientific knowledge in general. Given the recent rise in meta-analytic research in communication, how well publication bias concerns are addressed by communication meta-analysts merits attention. In this essay, after a brief overview of publication bias and some major methods of assessment, we provide a systematic review of meta-analyses published in six major communication journals between 2005 and 2018. The review focuses on two aspects of addressing publication bias in meta-analyses: (a) reducing the potential impact of bias via an inclusive literature search; and (b) empirically assessing the extent and impact of bias in meta-analytic findings. Our review shows that the current practices in communication meta-analyses are inadequate in both aspects. We offer recommendations on ways of improving practices in meta-analyses, as well as in research and publication processes, to better safeguard knowledge claims.

Addressing Publication Bias in Meta-Analysis

Abstract. Meta-analyses are an indispensable research synthesis tool for characterizing bodies of literature and advancing theories. One important open question concerns the inclusion of unpublished data into meta-analyses. Finding such studies can be effortful, but their exclusion potentially leads to consequential biases like overestimation of a literature’s mean effect. We address two questions about unpublished data using MetaLab, a collection of community-augmented meta-analyses focused on developmental psychology. First, we assess to what extent MetaLab datasets include gray literature, and by what search strategies they are unearthed. We find that an average of 11% of datapoints are from unpublished literature; standard search strategies like database searches, complemented with individualized approaches like including authors’ own data, contribute the majority of this literature. Second, we analyze the effect of including versus excluding unpublished literature on estimates of effect size and publication bias, and find this decision does not affect outcomes. We discuss lessons learned and implications.