Average Statistical Power Research Articles

Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability

ABSTRACTTwo factors that decrease the replicability of studies in the scientific literature are publication bias and studies with underpowered desgins. One way to ensure that studies have adequate statistical power to detect the effect size of interest is by conducting a-priori power analyses. Yet, a previous editorial published in the Journal of Sports Sciences reported a median sample size of 19 and the scarce usage of a-priori power analyses. We meta-analysed 89 studies from the same journal to assess the presence and extent of publication bias, as well as the average statistical power, by conducting a z-curve analysis. In a larger sample of 174 studies, we also examined a) the usage, reporting practices and reproducibility of a-priori power analyses; and b) the prevalence of reporting practices of t-statistic or F-ratio, degrees of freedom, exact p-values, effect sizes and confidence intervals. Our results indicate that there was some indication of publication bias and the average observed power was low (53% for significant and non-significant findings and 61% for only significant findings). Finally, the usage and reporting practices of a-priori power analyses as well as statistical results including test statistics, effect sizes and confidence intervals were suboptimal.

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: a meta-analysis

ABSTRACT Enhanced expectancies and autonomy-support through self-controlled practice conditions form the motivation pillar of OPTIMAL theory [Wulf, G., & Lewthwaite, R. (2016). Optimizing performance through intrinsic motivation and attention for learning: The OPTIMAL theory of motor learning. Psychonomic Bulletin & Review, 23(5), 1382–1414. https://doi.org/10.3758/s13423-015-0999-9]. The influence of these practice variables on motor learning was recently evaluated in two separate meta-analyses. Both meta-analyses found that the published literature suggested a moderate and significant benefit on motor learning; however, evidence for reporting bias was found in both literatures. Although multiple bias-corrected estimates were reported in the self-controlled meta-analysis, there was no principled way to prefer one over the other. In the enhanced expectancies meta-analysis, the trim-and-fill-technique failed to correct the estimated effects. Here, we addressed these limitations by reanalyzing the data from both meta-analyses using robust Bayesian meta-analysis methods. Our reanalysis revealed that reporting bias substantially exaggerated the benefits of these practice variables in the original meta-analyses. The true effects appear small, uncertain, and potentially null. We found the estimated average statistical power among all studies from the original meta-analyses was 6% (95% confidence interval [5%, 13%]). These results provide compelling and converging evidence that strongly suggests the available literature is insufficient to support the motivation pillar of OPTIMAL theory. Our results highlight the need for adequately powered experimental designs if motor learning scientists want to make evidence-based recommendations.

On the predicted replicability of two decades of experimental research on system justification: A Z‐curve analysis

AbstractWe examine the predicted replicability of experimental research on system justification theory (SJT) by conducting a z‐curve analysis. Z‐curve is a meta‐analytic technique similar to p‐curve, but which performs better under conditions of heterogeneity. It estimates the predicted replication rate, average power, false discovery risk, and file drawer ratio (FDR) of a sample of studies. The z‐curve based on 116 papers and 232 unique samples suggests that the experimental SJT literature is likely to show low rates of replicability, as indicated by an overall average statistical power of 16%. Moderator analyses suggest that this may be driven in part by publication pressures, that the replicability of research in this area has improved since 2015, and that studies using system threat manipulations show particularly low estimated replication rates (ERR). Implications for the replicability and validity of the experimental SJT literature are discussed, and recommendations to increase the rigor of research are put forth.

Hybrid of Restricted and Penalized Maximum Likelihood Method for Efficient Genome-Wide Association Study.

In genome-wide association studies, linear mixed models (LMMs) have been widely used to explore the molecular mechanism of complex traits. However, typical association approaches suffer from several important drawbacks: estimation of variance components in LMMs with large scale individuals is computationally slow; single-locus model is unsatisfactory to handle complex confounding and causes loss of statistical power. To address these issues, we propose an efficient two-stage method based on hybrid of restricted and penalized maximum likelihood, named HRePML. Firstly, we performed restricted maximum likelihood (REML) on single-locus LMM to remove unrelated markers, where spectral decomposition on covariance matrix was used to fast estimate variance components. Secondly, we carried out penalized maximum likelihood (PML) on multi-locus LMM for markers with reasonably large effects. To validate the effectiveness of HRePML, we conducted a series of simulation studies and real data analyses. As a result, our method always had the highest average statistical power compared with multi-locus mixed-model (MLMM), fixed and random model circulating probability unification (FarmCPU), and genome-wide efficient mixed model association (GEMMA). More importantly, HRePML can provide higher accuracy estimation of marker effects. HRePML also identifies 41 previous reported genes associated with development traits in Arabidopsis, which is more than was detected by the other methods.

A hybrid method for fast detection of spatial disease clusters in irregular shapes

Detection of spatial disease clusters in irregular shapes has generated considerable interest among public health researchers and policymakers. The existing methods have varying issues such as enormous computing workloads, peculiar cluster shapes, and high subjectivity of parameters. To support fast detection of irregularly shaped clusters, we are proposing a hybrid method combining Tango’s restricted likelihood ratio as the test statistic and Assuncao et al.’s dynamic Minimum Spanning Tree method as the search strategy. We discuss the advantages and the implementation of the hybrid method, and systematically compare its performance with other three well-known scan-based cluster detection methods, including Tango’s method, Assuncao et al.’s method, and Kulldorff’s circular spatial scan statistic method. Using simulated data of six cluster models combining two disease incidence levels and three true cluster shapes, the performance of the methods is evaluated in terms of statistical power, geographic accuracy, and computational intensity. The experimental results indicate that our hybrid method with 0.2 as the screening level value has the third highest average statistical power and the best average geographic accuracy among the four methods with all of the tested parameters. The four methods are then applied to the county-level lung cancer incidence data of Georgia from 1998 to 2005, and all find a significant cluster in northwestern Georgia but varying in shape and size.

Comparative Assessment of the Different American College of Rheumatology/European League Against Rheumatism Remission Definitions for Rheumatoid Arthritis for Their Use as Clinical Trial End Points.

The American College of Rheumatology (ACR) and the European League Against Rheumatism (EULAR) have defined remission using Boolean- or index-based criteria (i.e., a Simplified Disease Activity Index [SDAI] score of ≤3.3). We undertook this study to compare definitions of remission to inform choice of end points for future rheumatoid arthritis (RA) clinical trials, and we also included in our comparison the remission criterion of a score of ≤2.8 on the Clinical Disease Activity Index (CDAI). We performed post hoc analyses on clinical remission rates using data from 2 infliximab trials (the ASPIRE [Active-Controlled Study of Patients Receiving Infliximab for the Treatment of Rheumatoid Arthritis of Early Onset] and ATTRACT [Anti-Tumor Necrosis Factor Trial in Rheumatoid Arthritis with Concomitant Therapy] trials) and 1 golimumab trial (the GO-FORWARD trial). We investigated stringency of the different remission definitions, their power to discriminate between active drug and comparator, and aspects of their internal and external validity. We also investigated population determinants of discriminatory power for a particular remission end point. In patients with early RA (the ASPIRE trial), ACR/EULAR Boolean, CDAI, and SDAI remission rates at 6-7 months were 4-6% for methotrexate (MTX) alone versus 11-14% for infliximab plus MTX. In patients with MTX-refractory active RA (the ATTRACT and GO-FORWARD trials), remission rates were ≤1% for comparator (add-on of placebo) versus 4-6% for add-on of infliximab in the ATTRACT trial and ≤3% for comparator (add-on of placebo) versus 11-13% for add-on of golimumab in the GO-FORWARD trial. Existing remission cut points of different measures were generally comparable, with the Boolean criteria being somewhat more stringent than the SDAI and CDAI criteria. Remission rates were similar across definitions, as was average statistical power (CDAI, 55.6%; Boolean, 59.9%; SDAI, 62.6%). Remission is an ambitious primary end point for RA clinical trials, to be reserved for selected scenarios based on power considerations. The ACR/EULAR definitions are interchangeable, with slightly higher stringency of Boolean criteria over index-based criteria.

Low statistical power in biomedical science: a review of three human research domains.

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

Metaresearch for Evaluating Reproducibility in Ecology and Evolution.

Recent replication projects in other disciplines have uncovered disturbingly low levels of reproducibility, suggesting that those research literatures may contain unverifiable claims. The conditions contributing to irreproducibility in other disciplines are also present in ecology. These include a large discrepancy between the proportion of “positive” or “significant” results and the average statistical power of empirical research, incomplete reporting of sampling stopping rules and results, journal policies that discourage replication studies, and a prevailing publish-or-perish research culture that encourages questionable research practices. We argue that these conditions constitute sufficient reason to systematically evaluate the reproducibility of the evidence base in ecology and evolution. In some cases, the direct replication of ecological research is difficult because of strong temporal and spatial dependencies, so here, we propose metaresearch projects that will provide proxy measures of reproducibility.

The Importance of Sample Size for Reproducibility of tDCS Effects.

The Importance of Sample Size for Reproducibility of tDCS Effects.

Underappreciated problems of low replication in ecological field studies.

The cost and difficulty of manipulative field studies makes low statistical power a pervasive issue throughout most ecological subdisciplines. Ecologists are already aware that small sample sizes increase the probability of committing Type II errors. In this article, we address a relatively unknown problem with low power: underpowered studies must overestimate small effect sizes in order to achieve statistical significance. First, we describe how low replication coupled with weak effect sizes leads to Type M errors, or exaggerated effect sizes. We then conduct a meta-analysis to determine the average statistical power and Type M error rate for manipulative field experiments that address important questions related to global change; global warming, biodiversity loss, and drought. Finally, we provide recommendations for avoiding Type M errors and constraining estimates of effect size from underpowered studies.

The experimental design of postmortem studies: the effect size and statistical power.

PurposeThe aim is of this study was to show the poor statistical power of postmortem studies. Further, this study aimed to find an estimate of the effect size for postmortem studies in order to show the importance of this parameter. This can be an aid in performing power analysis to determine a minimal sample size.MethodsGPower was used to perform calculations on sample size, effect size, and statistical power. The minimal significance (α) and statistical power (1 − β) were set at 0.05 and 0.80 respectively. Calculations were performed for two groups (Student’s t-distribution) and multiple groups (one-way ANOVA; F-distribution).ResultsIn this study, an average effect size of 0.46 was found (n = 22; SD = 0.30). Using this value to calculate the statistical power of another group of postmortem studies (n = 5) revealed that the average statistical power of these studies was poor (1 − β < 0.80).ConclusionThe probability of a type-II error in postmortem studies is considerable. In order to enhance statistical power of postmortem studies, power analysis should be performed in which the effect size found in this study can be used as a guideline.

A meta-analysis of syntactic priming in language production

Abstract We performed an exhaustive meta-analysis of 73 peer-reviewed journal articles on syntactic priming from the seminal Bock (1986) paper through 2013. Extracting the effect size for each experiment and condition, where the effect size is the log odds ratio of the frequency of the primed structure X to the frequency of the unprimed structure Y, we found a robust effect of syntactic priming with an average weighted odds ratio of 1.67 when there is no lexical overlap and 3.26 when there is. That is, a construction X which occurs 50% of the time in the absence of priming would occur 63% if primed without lexical repetition and 77% of the time if primed with lexical repetition. The syntactic priming effect is robust across several different construction types and languages, and we found strong effects of lexical overlap on the size of the priming effect as well as interactions between lexical repetition and temporal lag and between lexical repetition and whether the priming occurred within or across languages. We also analyzed the distribution of p-values across experiments in order to estimate the average statistical power of experiments in our sample and to assess publication bias. Analyzing a subset of experiments in which the primary result of interest is whether a particular structure showed a priming effect, we did not find evidence of major p-hacking and the studies appear to have acceptable statistical power: 82%. However, analyzing a subset of experiments that focus not just on whether syntactic priming exists but on how syntactic priming is moderated by other variables (such as repetition of words in prime and target, the location of the testing room, and the memory of the speaker), we found that such studies are, on average, underpowered with estimated average power of 53%. Using a subset of 45 papers from our sample for which we received raw data, we estimated subject and item variation and give recommendations for appropriate sample size for future syntactic priming studies.

An Assessment of the Magnitude of Effect Sizes

This study compiles information from more than 250 meta-analyses conducted over the past 30 years to assess the magnitude of reported effect sizes in the organizational behavior (OB)/human resources (HR) literatures. Our analysis revealed an average uncorrected effect of r = .227 and an average corrected effect of ρ = .278 ( SDρ = .140). Based on the distribution of effect sizes we report, Cohen’s effect size benchmarks are not appropriate for use in OB/HR research as they overestimate the actual breakpoints between small, medium, and large effects. We also assessed the average statistical power reported in meta-analytic conclusions and found substantial evidence that the majority of primary studies in the management literature are statistically underpowered. Finally, we investigated the impact of the file drawer problem in meta-analyses and our findings indicate that the file drawer problem is not a significant concern for meta-analysts. We conclude by discussing various implications of this study for OB/HR researchers.

Supply chain management research: Key elements of study design and statistical testing

Over the past three decades, supply chain management (SCM) has evolved from its origins as a nascent field of study to encompass construct definition, identification of the field's central issues, and establishment of its conceptual boundaries. At this point, a sufficient body of empirical SCM research has been put forward to allow for quantitative assessment of the field. Therefore, we examine three key elements of study design to assess what has happened, what is currently happening, and where we should be heading as a field. To do so, following a pattern of reviews in similar disciplines, we begin with an examination of effect sizes of the relationships under investigation. Results show that effect sizes in SCM research have marginally increased over time and that sub-domains within SCM that receive the most scholarly attention also have higher effect sizes. We also conduct a post hoc analysis of statistical power and empirically examine a range of factors and study contexts that could influence power. Findings suggest that average statistical power in SCM research exceeds the statistical power of most related disciplines and is particularly high in several unique contexts. Lastly, we find that measurement reliability and the use of control variables have increased over time, possibly suggesting the field has matured, instilling a degree of confidence in its research. Overall, our results show that SCM research is becoming more empirically rigorous, but we also uncover key areas that warrant improvement. We describe implications of our review for the design of future SCM empirical studies.

Bias and power in group-based epidemiologic studies of low-back pain exposure and outcome--effects of study size and exposure measurement efforts.

Exposure-outcome studies, for instance on work-related low-back pain (LBP), often classify workers into groups for which exposures are estimated from measurements on a sample of workers within or outside the specific study. The present study investigated the influence on bias and power in exposure-outcome associations of the sizes of the total study population and the sample used to estimate exposures. At baseline, lifting, trunk flexion, and trunk rotation were observed for 371 of 1131 workers allocated to 19 a-priori defined occupational groups. LBP (dichotomous) was reported by all workers during 3 years of follow-up. All three exposures were associated with LBP in this parent study (P < 0.01). All 21 combinations of n = 10, 20, 30 workers per group with an outcome, and k = 1, 2, 3, 5, 10, 15, 20 workers actually being observed were investigated using bootstrapping, repeating each combination 10000 times. Odds ratios (OR) with P values were determined for each of these virtual studies. Average OR and statistical power (P < 0.05 and P < 0.01) was determined from the bootstrap distributions at each (n, k) combination. For lifting and flexed trunk, studies including n ≥ 20 workers, with k ≥ 5 observed, led to an almost unbiased OR and a power >0.80 (P level = 0.05). A similar performance required n ≥ 30 workers for rotated trunk. Small numbers of observed workers (k) resulted in biased OR, while power was, in general, more sensitive to the total number of workers (n). In epidemiologic studies using a group-based exposure assessment strategy, statistical performance may be sufficient if outcome is obtained from a reasonably large number of workers, even if exposure is estimated from only few workers per group.

Is there an excess of significant findings in published studies of psychotherapy for depression?

Many studies have examined the efficacy of psychotherapy for major depressive disorder (MDD) but publication bias against null results may exist in this literature. However, to date, the presence of an excess of significant findings in this literature has not been explicitly tested. We used a database of 1344 articles on the psychological treatment of depression, identified through systematic search in PubMed, PsycINFO, EMBASE and the Cochrane database of randomized trials. From these we identified 149 studies eligible for inclusion that provided 212 comparisons. We tested for an excess of significant findings using the method developed by Ioannidis and Trikalinos (2007), and compared the distribution of p values in this literature with the distribution in the antidepressant literature, where publication bias is known to be operating. The average statistical power to detect the effect size indicated by the meta-analysis was 49%. A total of 123 comparisons (58%) reported a statistically significant difference between treatment and control groups, but on the basis of the average power observed, we would only have expected 104 (i.e. 49%) to do so. There was therefore evidence of an excess of significance in this literature (p = 0.010). Similar results were obtained when these analyses were restricted to studies including a cognitive behavioural therapy (CBT) arm. Finally, the distribution of p values for psychotherapy studies resembled that for published antidepressant studies, where publication bias against null results has already been established. The small average size of individual psychotherapy studies is only sufficient to detect large effects. Our results indicate an excess of significant findings relative to what would be expected, given the average statistical power of studies of psychotherapy for major depression.

Power failure: why small sample size undermines the reliability of neuroscience

A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.

Mild stimulation of stomach 36 acupuncture point by organic nanoscale SP6 patch improves cellular physiologic functional status of different organs

Acupuncture works by stimulating peripheral sensory nerves and their endings causing an increase in cutaneous blood flow and microcirculation, as well as release of neurotransmitters, neuropeptides, and hormones. SP6 Patch is a nanoscale nontransdermal device that mildly stimulates Stomach 36 (zusanli) and other acupuncture points. As stimulation of these points has been indicated to have an effect on hypothalamic function, it is of great research interest to investigate the effect of SP6 Patch on the physiology of organs that are affected by hypothalamic regulation. Bioelectrical tissue impedance data indicative of cellular physiologic organ function, using an Electro Interstitial Scan (EIS) system, were acquired from hypothalamus, pancreas, liver, intestines, kidneys, thyroid and adrenal glands in 10 (1 male, 9 females) volunteers while wearing the SP6 Patch daily for 1 week. EIS testing was performed at baseline with no patch, 30 min after wearing the patch, and after wearing the patch 12 h/day for 1 week. Subjects were instructed to keep well hydrated during the study period. All subjects served as their own control. The hypothesis was: The SP6 Patch worn 12 h/day on the Stomach 36 acupuncture point for 1 week, may significantly improve cellular physiologic functional status of different organs measured by EIS. All tested organs achieved significant improvement in their functional physiologic status after wearing the SP6 Pach 12 h/day for 1 week compared to baseline with an overall average statistical power > 89%. Based upon these results the hypothesis was accepted as true.

The Power and Effects of Entrepreneurship Research

This study summarizes and analyzes average statistical power and effect sizes in empirical entrepreneurship research. Results show that statistical power was higher than expected, and was particularly high in studies employing archival measures. Statistical power has also increased over time. Effect sizes were higher than expected, a finding that remained consistent for different levels of analysis and across multiple subdomains. We discuss these findings, compare them to related disciplines, and draw implications for the design of future studies.

Sample size determination for the false discovery rate.

There is not a widely applicable method to determine the sample size for experiments basing statistical significance on the false discovery rate (FDR). We propose and develop the anticipated FDR (aFDR) as a conceptual tool for determining sample size. We derive mathematical expressions for the aFDR and anticipated average statistical power. These expressions are used to develop a general algorithm to determine sample size. We provide specific details on how to implement the algorithm for a k-group (k > or = 2) comparisons. The algorithm performs well for k-group comparisons in a series of traditional simulations and in a real-data simulation conducted by resampling from a large, publicly available dataset. Documented S-plus and R code libraries are freely available from www.stjuderesearch.org/depts/biostats.