Power Of Statistical Tests Research Articles

Simulation methodologies to determine statistical power in laboratory animal research studies.

Null hypothesis significance testing is a statistical tool commonly employed throughout laboratory animal research. When experimental results are reported, the reproducibility of the results is of utmost importance. Establishing standard, robust, and adequately powered statistical methodology in the analysis of laboratory animal data is critical to ensure reproducible and valid results. Simulation studies are a reliable method for assessing the power of statistical tests, however, biologists may not be familiar with simulation studies for power despite their efficacy and accessibility. Through an example of simulated Harlan Sprague-Dawley (HSD) rat organ weight data, we highlight the importance of conducting power analyses in laboratory animal research. Using simulations to determine statistical power prior to an experiment is a financially and ethically sound way to validate statistical tests and to help ensure reproducibility of findings in line with the 4R principles of animal welfare.

Some non parametric tests for umbrella alternatives in experimental design: A simulation study

Testing the null hypothesis against the umbrella alternative arises in various practical situations. This article has studied some non parametric tests that may be used for umbrella alternatives and their performance in different symmetric and asymmetric distributions, including long-tailed, short-tailed, and a mixture of two distributions. We have considered viz., Mack and Wolfe test, Hettmansperger and Norton test, Shi test, SU test, and Modified Mack and Wolfe test. The simulation approach assesses the empirical level and power of test statistics. The study shows that the SU and Shi tests have performed almost the same in Normal, Cauchy, Contaminated normal, and mixture distributions. However, the overall performance of the SU test was found to be better than that of other selected tests.

Poor statistical power in population-based association study of gene interaction

BackgroundStatistical epistasis, or “gene–gene interaction” in genetic association studies, means the nonadditive effects between the polymorphic sites on two different genes affecting the same phenotype. In the genetic association analysis of complex traits, nevertheless, the researchers haven’t found enough clues of statistical epistasis so far.MethodsWe developed a statistical model where the statistical epistasis was presented as an extra linkage disequilibrium between the polymorphic sites of different risk genes. The power of statistical test for identifying the gene–gene interaction was calculated and then compared in different hypothesis scenarios.ResultsOur results show the statistical power increases with the increasing of interaction coefficient, relative risk, and linkage disequilibrium with genetic markers. However, the power of interaction discovery is much lower than that of regular single-site association test. When rigorous criteria were employed in statistical tests, the identification of gene–gene interaction became a very difficult task. Since the criterion of significance was given to be p-value ≤ 5.0 × 10–8, the same as that of many genome-wide association studies, there is little chance to identify the gene–gene interaction in all kind of circumstances.ConclusionsThe lack of epistasis tends to be an inevitable result caused by the statistical principles of methods in the genetic association studies and therefore is the inherent characteristic of the research itself.

Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis

Abstract Conjoint analysis is a popular experimental design used to measure multidimensional preferences. Many researchers focus on estimating the average marginal effects of each factor while averaging over the other factors. Although this allows for straightforward design-based estimation, the results critically depend on the ways in which factors interact with one another. An alternative model-based approach can compute various quantities of interest, but requires correct model specifications, a challenging task for conjoint analysis with many factors. We propose a new hypothesis testing approach based on the conditional randomization test (CRT) to answer the most fundamental question of conjoint analysis: Does a factor of interest matter in any way given the other factors? Although it only provides a formal test of these binary questions, the CRT is solely based on the randomization of factors, and hence requires no modeling assumption. This means that the CRT can provide a powerful and assumption-free statistical test by enabling the use of any test statistic, including those based on complex machine learning algorithms. We also show how to test commonly used regularity assumptions. Finally, we apply the proposed methodology to conjoint analysis of immigration preferences. An open-source software package is available for implementing the proposed methodology. The proposed methodology is implemented via an open-source software R package CRTConjoint, available through the Comprehensive R Archive Network https://cran.r-project.org/web/packages/CRTConjoint/index.html.

Spatial-extent inference for testing variance components in reliability and heritability studies

Abstract Clusterwise inference is a popular approach in neuroimaging to increase sensitivity, but most existing methods are currently restricted to the General Linear Model (GLM) for testing mean parameters. Statistical methods for testing variance components, which are critical in neuroimaging studies that involve estimation of narrow-sense heritability or test-retest reliability, are underdeveloped due to methodological and computational challenges, which would potentially lead to low power. We propose a fast and powerful test for variance components called CLEAN-V (CLEAN for testing Variance components). CLEAN-V models the global spatial dependence structure of imaging data and computes a locally powerful variance component test statistic by data-adaptively pooling neighborhood information. Correction for multiple comparisons is achieved by permutations to control family-wise error rate (FWER). Through analysis of task-functional magnetic resonance imaging (fMRI) data from the Human Connectome Project across five tasks and comprehensive data-driven simulations, we show that CLEAN-V outperforms existing methods in detecting test-retest reliability and narrow-sense heritability with significantly improved power, with the detected areas aligning with activation maps. The computational efficiency of CLEAN-V also speaks of its practical utility, and it is available as an R package.

Empirical likelihood method for detecting change points in network autoregressive models

<p>The network autoregressive model is a super high-dimensional time series model that can fully explain social relationships. This model can fully reflect the complex relationships in reality. Therefore, it plays a vital role in detecting the inflection point problem of this network autoregressive model for economics and finance. In this paper, we proposed the change-point problem of detecting network autoregressive models using empirical likelihood statistics based on the expected error term of the switching rule being 0, using the empirical likelihood method. Moreover, the asymptotic null distribution of the proposed empirical likelihood statistic was investigated. Simulation studies based on different settings were considered, and the results showed that the power of test statistics is significant. In the end, the Chinese stock market was investigated to demonstrate the significance of the proposed method.</p>

LMdist: Local Manifold distance accurately measures beta diversity in ecological gradients.

Differentiating ecosystems poses a complex, high-dimensional problem constrained by capturing relevant variation across species profiles. Researchers use pairwise distances and subsequent dimensionality reduction to highlight variation in a few dimensions. Despite popularity in analysis of ecological data, these low-dimensional visualizations can contain geometric abnormalities such as "arch" and "horseshoe" effects, potentially obscuring the impact of environmental gradients. These abnormalities appear in ordination but are in fact a product of oversaturated large pairwise distances. We present Local Manifold distance (LMdist), an unsupervised algorithm which adjusts pairwise beta diversity measures to better represent true ecological distances between samples. Beta diversity measures can have a bounded dynamic range in depicting long environmental gradients with high species turnover. Using a graph structure, LMdist projects pairwise distances onto a manifold and traverses the manifold surface to adjust pairwise distances at the upper end of the beta diversity measure's dynamic range. This allows for values beyond the range of the original measure. Not all datasets will have oversaturated pairwise distances, nor will capture variation that resembles a manifold, so LMdist adjusts only those pairwise values which may be undervalued in the presence of a sampled gradient. The adjusted distances serve as input for ordination and statistical testing. We demonstrate on real and simulated data that LMdist effectively recovers distances along known gradients and along complex manifolds such as the Swiss roll dataset. LMdist enables more powerful statistical tests for gradient effects and reveals variation orthogonal to the gradient. Available on GitHub at https://github.com/knights-lab/LMdist.

Tests of independence and randomness for arbitrary data using copula-based covariances

In this article, we study tests of independence for data with arbitrary distributions in the non-serial case, i.e., for independent and identically distributed random vectors, as well as in the serial case, i.e., for time series. These tests are derived from copula-based covariances and their multivariate extensions using Möbius transforms. We find the asymptotic distributions of these statistics under the null hypothesis of independence or randomness, as well as under contiguous alternatives. This enables us to find out locally most powerful test statistics for some alternatives, whatever the margins. Numerical experiments are performed for Wald’s type combinations of these statistics to assess the finite sample performance.

Samples in many cell-based experiments are matched/paired but taking this into account does not always increase power of statistical tests for differences in means.

Power of statistical tests for differences in means is the probability of obtaining a statistically significant p value when means differ. When samples in experimental replicates come from a single cell culture, they are matched or paired because they share between-trials biological variability. This can cause positive correlation between values from conditions in a replicate. Correlation can also be caused in otherwise independent samples by shared technical variability. However, correlation is reduced by noise that affects samples individually. I investigated how to maximize power in experiments with two conditions over a range of correlations. Normalizing data to control increases the rate of false positives, if Student's t test is used. Paired t tests, theoretically the correct test for matched samples, have higher power than Student's t test when correlation is high, but lower power when correlation is low. Testing correlation to select a test for differences in mean can affect the subsequent rate of false positives. Ultimately, components of experimental variability must be considered to choose the most powerful two sample test for differences in mean. This contrasts with experiments with more than two conditions, where random-block ANOVA, a matched samples test, can be used as a default.

A Credibility Crisis in Psychology?

The interest in the global result obtained by B. Nosek’s team increased significantly, not only among psychologists, after an article presenting the results of a large-scale international replication of psychological empirical research had been published in Science (cf. Open Science Collaboration, 2015). While 97% of the original research yielded statistically significant results (p <. 05), only 36% of the results were significant in the replication. The author of the present article postulates that this result laid the ground for unjustified generalizations about the methodological weaknesses of psychology as an empirical science. Psychology is an empirical science, but it also has its peculiarities due to the specificity of the subject matter and the method (e.g. Orne, 1962, 1973; Rosenthal, 1966/2009; Rosenzweig, 1933). Equally importantly, psychology is not practiced in social or cultural isolation. Finally, psychological research is bound by rigorous ethical standards/constraints, and psychologists (as well as researchers in other fields) who publish the results of empirical research to be analyzed statistically are constrained by the editorial practices of scientific journals. Journals have an interest only in papers that present statistically significant results (where “p < .05”!), which leads to the so-called file-drawer effect (Rosenthal, 1979). As strongly emphasized by the author, the debate cannot be limited to the statistical significance of psychological research (in particular the power of statistical test which has emerged as a popular trend in recent years). In this article, the author discusses (and presents his point of view) the following problems: 1) the methodological specificity of psychology as an empirical science, 2) the triad of statistical significance (the problematic criterion of “p < .05”), effect size, and the power of a statistical test, 3) the socio-cultural context of psychological research, 4) researchers' failure to follow methodological and ethical guidelines, and 5) possible precautions and remedies.

Analyzing the Impact of Sample Size and Population Distribution on Statistical Test Power

Analyzing the Impact of Sample Size and Population Distribution on Statistical Test Power

Power of testing for exposure effects under incomplete mediation.

Mediation analysis studies situations where an exposure may affect an outcome both directly and indirectly through intervening variables called mediators. It is frequently of interest to test for the effect of the exposure on the outcome, and the standard approach is simply to regress the latter on the former. However, it seems plausible that a more powerful test statistic could be achieved by also incorporating the mediators. This would be useful in cases where the exposure effect size might be small, which for example is common in genomics applications. Previous work has shown that this is indeed possible under complete mediation, where there is no direct effect. In most applications, however, the direct effect is likely nonzero. In this paper we study linear mediation models and find that under certain conditions, power gain is still possible under this incomplete mediation setting for testing the null hypothesis that there is neither a direct nor an indirect effect. We study a class of procedures that can achieve this performance and develop their application to both low- and high-dimensional mediators. We then illustrate their performances in simulations as well as in an analysis using DNA methylation mediators to study the effect of cigarette smoking on gene expression.

Gene filtering strategies for machine learning guided biomarker discovery using neonatal sepsis RNA-seq data.

Machine learning (ML) algorithms are powerful tools that are increasingly being used for sepsis biomarker discovery in RNA-Seq data. RNA-Seq datasets contain multiple sources and types of noise (operator, technical and non-systematic) that may bias ML classification. Normalisation and independent gene filtering approaches described in RNA-Seq workflows account for some of this variability and are typically only targeted at differential expression analysis rather than ML applications. Pre-processing normalisation steps significantly reduce the number of variables in the data and thereby increase the power of statistical testing, but can potentially discard valuable and insightful classification features. A systematic assessment of applying transcript level filtering on the robustness and stability of ML based RNA-seq classification remains to be fully explored. In this report we examine the impact of filtering out low count transcripts and those with influential outliers read counts on downstream ML analysis for sepsis biomarker discovery using elastic net regularised logistic regression, L1-reguarlised support vector machines and random forests. We demonstrate that applying a systematic objective strategy for removal of uninformative and potentially biasing biomarkers representing up to 60% of transcripts in different sample size datasets, including two illustrative neonatal sepsis cohorts, leads to substantial improvements in classification performance, higher stability of the resulting gene signatures, and better agreement with previously reported sepsis biomarkers. We also demonstrate that the performance uplift from gene filtering depends on the ML classifier chosen, with L1-regularlised support vector machines showing the greatest performance improvements with our experimental data.

Outcome measures of phase III anticancer drug trials in China.

Outcome measures of phase III anticancer drug trials in China.

A Replication of “The Effect of the Conservation Reserve Program on Rural Economies: Deriving a Statistical Verdict from a Null Finding” (American Journal of Agricultural Economics, 2019)

Abstract This study replicates the paper “Brown, J. P., Lambert, D. M., & Wojan, T. R. (2019). The effect of the conservation reserve program on rural economies: deriving a statistical verdict from a null finding. American Journal of Agricultural Economics, 101(2), 528–540” and their procedure for calculating the so-called ex post power of statistical tests of significance for regression coefficients. There appears no generally accepted method for calculating ex post power, and Brown, Lambert, and Wojan (BLW) provided a bootstrapping method that can be applied after the parameter of interest is estimated. They recommend researchers to use this procedure to investigate whether a statistically insignificant finding is likely to be due to a low power property of the significance test. This study makes two main contributions. First, it verifies whether the data and code that BLW provided are reliable to reproduce their results. Second, it constructs Monte Carlo experiments to assess the performance of BLW’s method. The results indicate that their method produces ex post power estimates that are relatively close to the true power values. Mean power estimates are generally unbiased, and 95% of the estimates lie within +/− 5% points of the true power. In conclusion, my replication provides further evidence of the reliability of BLW’s method.

Reporting and interpreting non-significant results in animal cognition research.

How statistically non-significant results are reported and interpreted following null hypothesis significance testing is often criticized. This issue is important for animal cognition research because studies in the field are often underpowered to detect theoretically meaningful effect sizes, i.e., often produce non-significant p-values even when the null hypothesis is incorrect. Thus, we manually extracted and classified how researchers report and interpret non-significant p-values and examined the p-value distribution of these non-significant results across published articles in animal cognition and related fields. We found a large amount of heterogeneity in how researchers report statistically non-significant p-values in the result sections of articles, and how they interpret them in the titles and abstracts. Reporting of the non-significant results as "No Effect" was common in the titles (84%), abstracts (64%), and results sections (41%) of papers, whereas reporting of the results as "Non-Significant" was less common in the titles (0%) and abstracts (26%), but was present in the results (52%). Discussions of effect sizes were rare (<5% of articles). A p-value distribution analysis was consistent with research being performed with low power of statistical tests to detect effect sizes of interest. These findings suggest that researchers in animal cognition should pay close attention to the evidence used to support claims of absence of effects in the literature, and-in their own work-report statistically non-significant results clearly and formally correct, as well as use more formal methods of assessing evidence against theoretical predictions.

Quantitative analyses of cytokine profiles reveal hormone-mediated modulation of cytokine profiles in recurrent spontaneous miscarriage

PurposeCytokines play important roles in pregnancy complications. Some hormones such as estrogen, progesterone, and dydrogesterone have been shown to alter cytokine profiles. Understanding how cytokine profiles are affected by these hormones is therefore an important step towards immunomodulatory therapies for pregnancy complications. We analyse previously published data on the effects of estrogen, progesterone, and dydrogesterone on cytokine balances in women having recurrent spontaneous miscarriages. Materials and methodsLevels of eight cytokines (IFN-γ, IL-2, IL-6, IL-10, IL-13, IL-17, IL-23, TNF-α) from n = 22 women presenting unexplained recurrent spontaneous miscarriages were studied. Cytokine values were recorded after in vitro exposure of peripheral blood cells to estrogen, progesterone, and dydrogesterone. We expand on earlier analysis of the dataset by employing different statistical techniques including effect sizes for individual cytokine values, a more powerful statistical test, and adjusting p-values for multiple comparisons. We employ multivariate analysis methods, including to determine the relative magnitude of the effects of the hormone therapies on cytokines. A new statistical method is introduced based on pairwise distances able to accommodate complex relations in cytokine profiles. ResultsWe report several statistically significant differences in individual cytokine values between the control group and each hormone treated group, with estrogen affecting the fewest cytokines, and progesterone and dydrogesterone both affecting seven out of eight cytokines. Exposure to estrogen produces no large effects sizes however, while IFN-γ and IL-17 show large effect sizes for both progesterone and dydrogesterone, among other cytokines. Our new method for identifying which collections (i.e. subsets) of cytokines best distinguish contrasting groups identifies IFN-γ, IL-10 and IL-23 as especially noteworthy for both progesterone and dydrogesterone treatments. ConclusionsWhile some statistically significant differences in cytokine levels after exposure to estrogen are found, these have small effect sizes and are unlikely to be clinically relevant. Progesterone and dydrogesterone both induce statistically significant and large effect-size differences in cytokine levels, hence therapy with these two progestogens is more likely to be clinically relevant. Univariate and multivariate methods for identifying cytokine importances provide insight into which groups of cytokines are most affected and in what ways by therapies.

Learning to increase the power of conditional randomization tests

The model-X conditional randomization test is a generic framework for conditional independence testing, unlocking new possibilities to discover features that are conditionally associated with a response of interest while controlling type-I error rates. An appealing advantage of this test is that it can work with any machine learning model to design powerful test statistics. In turn, the common practice in the model-X literature is to form a test statistic using machine learning models, trained to maximize predictive accuracy with the hope to attain a test with good power. However, the ideal goal here is to drive the model (during training) to maximize the power of the test, not merely the predictive accuracy. In this paper, we bridge this gap by introducing, for the first time, novel model-fitting schemes that are designed to explicitly improve the power of model-X tests. This is done by introducing a new cost function that aims at maximizing the test statistic used to measure violations of conditional independence. Using synthetic and real data sets, we demonstrate that the combination of our proposed loss function with various base predictive models (lasso, elastic net, and deep neural networks) consistently increases the number of correct discoveries obtained, while maintaining type-I error rates under control.

Sex proportion as a covariate increases the statistical test power in growth performance based experiments using as-hatched broilers.

The availability of sexed day-old broiler chicks is becoming an issue as feather sexing is no longer possible. This has great implications for broiler researchers as the use of randomly distributed mixed-sex birds may result in a greater between-pen variation and thus less statistical power than the use of single-sex birds. The objective of this study was to evaluate the effect of including sex proportion as a covariate in an analysis of covariance (ANCOVA) on the statistical power compared to analysis of variance (ANOVA) where sex was not considered. The statistical parameters examined include mean square error (MSE), the F-statistic, model fit, model significance and observed power. A total of 4 separate experiments that used mixed-sex broilers with unequal numbers of male and female birds per pen were conducted during which performance of the birds was measured. The male % in each pen was recorded during each experiment and corrected for mortality. The performance results were analysed by ANOVA and the statistical parameters were then compared to ANCOVA where sex proportion was included as a covariate. The results showed that a set of assumptions first needed to be met to run ANCOVA. In addition, if the ANOVA results show a high level of model significance and power, then ANCOVA may not be necessary. In other circumstances where the assumptions are met and model significance and observed power are low, the inclusion of sex proportion as a covariate in the analysis will help to reduce MSE, increase the F-statistic value and improve the model significance, model fit and observed power. Therefore, it is suggested that sex proportion should be considered as a covariate in ANCOVA to improve statistical power in nutritional experiments when male and female broilers are unequally and randomly distributed in pens.

Disturbance and Invasive Plant Occurrence along High-Elevation Boundaries Surrounding California Protected Areas

Protected areas rarely are large enough to encompass the ecological processes that affect them. Accordingly, some authors call for cooperative management of multiple jurisdictions within protected area-centered ecosystems (PACEs) to sustain landscape-scale processes and maximize connectivity and ecosystem service flows. However, over time, differing land use and disturbance histories between adjoining jurisdictions may lead to divergence in plant community composition or structure along administrative boundaries, thus reducing landscape connectivity within PACEs. We tested this hypothesis by measuring evidence of human disturbance and occurrence of nonnative plant species along boundaries between protected areas and multiple-use lands in the California PACEs surrounding Lassen Volcanic and Sequoia & Kings Canyon National Parks. Invasive plants were found in about half of sampled units at Sequoia & Kings Canyon, but occurrence was rare at Lassen Volcanic and above 2000 m in both PACEs. We found no significant differences in invasive plant occurrence or human disturbance along boundaries between adjoining jurisdictions, and correlations between disturbance and invasive plant detections were not significant except in U.S. Forest Service non-wilderness lands. The overall low rate of detection in this study limited the power of statistical tests, and may be largely due to the high average elevation of sampling locations. Cooperative invasive plant management already occurs at times within these PACEs, and its continued use may be useful for maintaining a relatively invader-free environment under future conditions.