Use Of Power Analysis Research Articles

Uncertainty limits the use of power analysis.

The calculation of statistical power has been taken up as a simple yet informative tool to assist in designing an experiment, particularly in justifying sample size. A difficulty with using power for this purpose is that the classical power formula does not incorporate sources of uncertainty (e.g., sampling variability) that can impact the computed power value, leading to a false sense of precision and confidence in design choices. We use simulations to demonstrate the consequences of adding two common sources of uncertainty to the calculation of power. Sampling variability in the estimated effect size (Cohen's d) can introduce a large amount of uncertainty (e.g., sometimes producing rather flat distributions) in power and sample-size determination. The addition of random fluctuations in the population effect size can cause values of its estimates to take on a sign opposite the population value, making calculated power values meaningless. These results suggest that calculated power values or use of such values to justify sample size add little to planning a study. As a result, researchers should put little confidence in power-based choices when planning future studies. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Supplemental Material for Uncertainty Limits the Use of Power Analysis

Supplemental Material for Uncertainty Limits the Use of Power Analysis

Standing on the shoulders of bias: lack of transparency and reporting of critical rigor characteristics in pain research.

Rigorous experimental design with transparent reporting in biomedical science reduces risk of bias and allows for scientists to judge the quality of the research. Basic factors of rigor such as blinding, randomization, power analysis, and inclusion of both sexes impact the reproducibility by reducing experimental bias. We designed a systematic study to analyze basic factors of rigor, inclusion of sex, and whether data were analyzed or disaggregated by sex over the past 10 years in the journal PAIN . Studies that included humans reported randomization in 81%, blinding in 48%, and the use of a power analysis calculation in 27% over the past 10 years. Studies that included mice reported randomization in 35%, blinding in 70%, and the use of a power analysis in 9%. Studies that included rats reported randomization in 38%, blinding in 63%, and the use of power analysis in 12%. This study also found that human studies consistently included both sexes over the past decade, but less than 20% of data were disaggregated or analyzed for sex differences. Although mouse and rat studies predominately used males only, there has been a slight increase in inclusion of both sexes over the past few years. Justification for single-sex studies was below 50% in both human and rodent data. In both human and animal studies, transparency in reporting of experimental design and inclusion of both sexes should be considered standard practice and will result in improved quality and reproducibility of published research.

Application of power analysis to determine the optimal reporting time frame for use in statewide trauma system quality reporting

BackgroundMeaningful reporting of quality metrics relies on detecting a statistical difference when a true difference in performance exists. Larger cohorts and longer time frames can produce higher rates of statistical differences. However, older data are less relevant when attempting to enact change in the clinical setting. The selection of time frames must reflect a balance between being too small (type II errors) and too long (stale data). We explored the use of power analysis to optimize time frame selection for trauma quality reporting. MethodsUsing data from 22 Level III trauma centers, we tested for differences in 4 outcomes within 4 cohorts of patients. With bootstrapping, we calculated the power for rejecting the null hypothesis that no difference exists amongst the centers for different time frames. From the entire sample for each site, we simulated randomly generated datasets. Each simulated dataset was tested for whether a difference was observed from the average. Power was calculated as the percentage of simulated datasets where a difference was observed. This process was repeated for each outcome. ResultsThe power calculations for the 4 cohorts revealed that the optimal time frame for Level III trauma centers to assess whether a single site’s outcomes are different from the overall average was 2 years based on an 80% cutoff. ConclusionPower analysis with simulated datasets allows testing of different time frames to assess outcome differences. This type of analysis allows selection of an optimal time frame for benchmarking of Level III trauma center data.

Use of power analysis to determine the number of samples needed to assess water quality in lakes and flowing waters.

The Clean Water Act requires states to develop methods for assessing water quality. Assessment methods serve as decision-making procedures for including waterbodies on the Section 303(d) List of Impaired Waters. We used 17 years of ambient water quality data to explore statistical analyses for assessment methods that represent New York's waterbodies. Power analyses were performed to determine how many samples are needed to evaluate exceedances of water quality criteria using one-sample t-tests in lakes and flowing waters. Results suggest six samples for lakes and eight samples for flowing waters are needed to obtain at least 80% power, which is fewer samples than most other types of statistical assessment methodologies. This smaller number was possible because the power analysis was applied to the actual variability found in monitoring data to calculate the effect size as opposed to more conservative statistical estimates based on random data. Water quality criteria can have different analysis requirements such as single samples or means above the threshold, so we compared how many impairments would occur in the dataset if the six or eight samples were assessed as two single exceedances or a mean above the water quality criteria. Because the power analysis gives no indication of the time frame of when samples should be collected, the intra- and interannual variability of the data was assessed to determine whether sampling over a growing season in one year or sampling over multiple years is more representative of the water quality status. Results demonstrated that data collected over the growing season captured more variability in water quality data than data collected over multiple years in both waterbody types. With the prevalence of regulatory agencies having large, historical datasets rising, it would be possible for other agencies to apply these types of analyses to their assessment methodologies. Integr Environ Assess Manag 2022;18:1621-1628. © 2021 SETAC.

Logical Experimental Design and Execution in the Biomedical Sciences

Lack of reproducibility has been highlighted as a significant problem in biomedical research. The present unit is devoted to describing ways to help ensure that research findings can be replicated by others, with a focus on the design and execution of laboratory experiments. Essential components for this include clearly defining the question being asked, using available information or information from pilot studies to aid in the design the experiment, and choosing manipulations under a logical framework based on Mill's "methods of knowing" to build confidence in putative causal links. Final experimental design requires systematic attention to detail, including the choice of controls, sample selection, blinding to avoid bias, and the use of power analysis to determine the sample size. Execution of the experiment is done with care to ensure that the independent variables are controlled and the measurements of the dependent variables are accurate. While there are always differences among laboratories with respect to technical expertise, equipment, and suppliers, execution of the steps itemized in this unit will ensure well-designed and well-executed experiments to answer any question in biomedical research. © 2017 by John Wiley & Sons, Inc.

Power analysis for generalized linear mixed models in ecology and evolution.

‘Will my study answer my research question?’ is the most fundamental question a researcher can ask when designing a study, yet when phrased in statistical terms – ‘What is the power of my study?’ or ‘How precise will my parameter estimate be?’ – few researchers in ecology and evolution (EE) try to answer it, despite the detrimental consequences of performing under- or over-powered research. We suggest that this reluctance is due in large part to the unsuitability of simple methods of power analysis (broadly defined as any attempt to quantify prospectively the ‘informativeness’ of a study) for the complex models commonly used in EE research. With the aim of encouraging the use of power analysis, we present simulation from generalized linear mixed models (GLMMs) as a flexible and accessible approach to power analysis that can account for random effects, overdispersion and diverse response distributions.We illustrate the benefits of simulation-based power analysis in two research scenarios: estimating the precision of a survey to estimate tick burdens on grouse chicks and estimating the power of a trial to compare the efficacy of insecticide-treated nets in malaria mosquito control. We provide a freely available R function, sim.glmm, for simulating from GLMMs.Analysis of simulated data revealed that the effects of accounting for realistic levels of random effects and overdispersion on power and precision estimates were substantial, with correspondingly severe implications for study design in the form of up to fivefold increases in sampling effort. We also show the utility of simulations for identifying scenarios where GLMM-fitting methods can perform poorly.These results illustrate the inadequacy of standard analytical power analysis methods and the flexibility of simulation-based power analysis for GLMMs. The wider use of these methods should contribute to improving the quality of study design in EE.

A convenient method and numerical tables for sample size determination in longitudinal-experimental research using multilevel models

Recent years have shown increased awareness of the importance of sample size determination in experimental research. Yet effective and convenient methods for sample size determination, especially in longitudinal experimental design, are still under development, and application of power analysis in applied research remains limited. This article presents a convenient method for sample size determination in longitudinal experimental research using a multilevel model. A fundamental idea of this method is transformation of model parameters (level 1 error variance [σ(2)], level 2 error variances [τ 00, τ 11] and its covariance [τ 01, τ 10], and a parameter representing experimental effect [δ]) into indices (reliability of measurement at the first time point [ρ 1], effect size at the last time point [Δ T ], proportion of variance of outcomes between the first and the last time points [k], and level 2 error correlation [r]) that are intuitively understandable and easily specified. To foster more convenient use of power analysis, numerical tables are constructed that refer to ANOVA results to investigate the influence on statistical power by respective indices.

Corticosterone responses in birds: Individual variation and repeatability in Adelie penguins ( Pygoscelisadeliae) and other species, and the use of power analysis to determine sample sizes

Plasma corticosterone concentrations increase when birds experience a stressor, and in this study we quantified variation in corticosterone responses for the first time in a species of free-living bird. Adelie penguins ( Pygoscelis adeliae) nesting at Cape Bird on Ross Island in Antarctica were sampled on three occasions. Penguins with relatively low or high corticosterone responses on the first occasion had consistently low or high responses, as previously found for great tits and chickens. A model for birds is proposed in which birds with low corticosterone responses and proactive personalities are likely to be more successful (have greater fitness) in constant or predictable conditions, whilst birds with reactive personalities and high corticosterone responses will be more successful in changing or unpredictable conditions. There is thus no linear relationship between the size of a corticosterone response and fitness. Whilst the absolute magnitude of corticosterone responses varies markedly across species of birds, coefficients of variation are similar. Individual corticosterone responses are generally repeatable, with significant statistical repeatabilities for 30 min corticosterone concentrations and integrated corticosterone concentrations in the Adelie penguin, great tit and chicken. Coefficients of variation in corticosterone responses between birds and power analyses were used to provide a rule of thumb for determining differences between groups of birds in mean corticosterone concentrations to enable statistical analyses to have acceptable levels of statistical power for given sample sizes. It is suggested that power analyses and this rule of thumb be adopted in future investigations of corticosterone responses in birds.

A Critical Discussion of Null Hypothesis Significance Testing and Statistical Power Analysis within Psychological Research

The uses of null hypothesis significance testing (NHST) and statistical power analysis within psychological research are critically discussed. The article looks at the problems of relying solely on NHST when dealing with small and large sample sizes. The use of power-analysis in estimating the potential error introduced by small and large samples is advocated. Power analysis is not recommended as a replacement to NHST but as an additional source of information about the phenomena under investigation. Moreover, the importance of conceptual analysis in relation to statistical analysis of a hypothesis is emphasised.

Feeding and dementia: a systematic literature review

This paper reports a systematic review of the literature on interventions to promote oral nutritional intake of older people with dementia and feeding difficulty between 1993 and 2003. Older people with dementia commonly experience difficulty with feeding, especially in the later stages of the condition. This topic and related nursing care was reviewed in 1993 and the conclusion was that there was little research into interventions that nurses could use to alleviate feeding difficulty. A systematic review of the literature was carried out using the CINAHL, Medline, EMBASE and Cochrane databases and the search terms 'feeding', 'eating' and 'dementia' combined as follows: '(feeding or eating) and (dementia)'. A second search was carried out combining the search terms 'mealtimes' and 'dementia' as follows: 'mealtimes and dementia'. The literature search was carried out on 1 December 2003 and papers were included in the review if retrieved by 31 December 2003. English language papers only were retrieved. Sixty-seven papers were retrieved, of which 13 addressed interventions aimed at helping older people with dementia to feed. All studies reported positive outcomes but only one randomized controlled trial was reported. Music was the most common intervention but there were no standardized interventions or outcomes across the studies and none reported the use of power analysis to decide on sample size. There were problems in some studies with confounding variables. Further research is needed into interventions aimed at how nurses can help older people with dementia to feed. There are some promising lines of enquiry, with music being one of these, but future studies need to use adequate samples and to use power calculations and account adequately for confounding variables. There is also a need to standardize interventions and outcomes across such studies to facilitate meta-analysis.

Statistical Power, Sample Sizes, and the Software to Calculate Them Easily

Many authors have advocated the use of power analysis for designing more sensitive and efficient ecological studies. However, few manuscripts report using power analysis for improving the study design, perhaps because of the wide range of tests routinely used by most ecologists and the limitations of most power analysis software. UnifyPow, an excellent freeware macro that runs within SAS, easily and accurately calculates power or sample sizes for a wide range of statistical tests for a broad range of statistical models. Despite its potential contribution to the field, UnifyPow is virtually unused by ecologists. Perhaps this is because UnifyPow was developed by medical statisticians or because there is no manual, although there is extensive online documentation. I briefly review the flexibility and breadth of UnifyPow; describe how it works; and, using ecological examples, demonstrate how to use it to calculate power for an extremely wide range of statistical tests.

Assessment of ecological effects due to forest harvesting: approaches and statistical issues

SummaryTimber harvesting exemplifies many aspects of experimental manipulation in applied ecology. Evaluation of the research approaches used to assess ecological effects of harvesting thus has relevance to a diverse range of ecological questions that address large‐scale and human impacts on the environment.We measured the frequency of approaches, and also design and statistical issues, in assessments of harvesting effects on three response variables (tree regeneration, vertebrates and water) in two major native forests of south‐east Australia. Our evaluation was based on 124 documents that contain unique, field‐based research and that represent the complete written record (published and unpublished) of relevant research prior to mid‐2002.The dominant approach was contrasts between manipulated stands, nearly always involving mature and old‐growth stands as the reference or control. In studies of tree regeneration, focus on wood‐production objectives was apparent in the relatively frequent use of stocking standards, and in the use of between‐ and within‐stand contrasts without a control condition. Comparatively greater use of models in vertebrate and water studies suggested more frequent integration of generic principles and, in water studies, reflected strong collective knowledge based on the repeated use of BACI‐type designs (i.e. before–after, control–impact) or long‐term data sets.Statistical issues such as poor reporting of experimental designs and of sample statistics were common. Explicit hypotheses were rare, and there was no evidence of definition of critical effect sizes, use of power analysis or use of non‐classical statistical approaches (e.g. Bayesian inference).Lack of treatment replication was widespread in between‐stand contrasts and less frequent in within‐stand contrasts. Inappropriate interpretation of results from non‐replicated experimental designs remains an ongoing issue.Synthesis and applications. The amount of information obtainable from stand‐level manipulations of forest would substantially increase through greater attention to data quality and experimental protocols. Studies require broader experimental contexts, including their potential place in meta‐replication frameworks. Increasingly, constraints of large‐scale manipulations necessitate the integration of data and the acknowledgement of uncertainty in formalized systems (e.g. Bayesian methods and adaptive management frameworks). Our evaluation suggests the alternative is a perpetuation of short‐term, isolated studies that repeat errors and have limited potential to refine knowledge on the ecological effects of timber harvesting and other similarly scaled disturbances.

Investigating the spatial dynamics of bovine tuberculosis in badger populations: evaluating an individual-based simulation model

We describe an individual-based spatially-explicit model designed to investigate the dynamics of badger populations and TB epidemiology in a real landscape. We develop a methodology for evaluating the sensitivity of the model to its input parameters through the use of power analysis, partial correlation coefficients and binary logistic regression. This novel approach to sensitivity analysis provides a formal statement of confidence in our findings based on statistical power, and a solution for analysing sparse data sets of disease prevalence. The sensitivity analysis revealed that the simulated badger population size after 20 years was most dependent on five parameters affecting female recruitment (probability of breeding, mortality of adult females in the first half of the year, mortality of juvenile females in the second half of the year and mortality of female cubs in the both halves of the year). The simulated prevalence of TB was most affected by the population size, the rate at which infectious badgers transmit the disease to other members of their social group, and the rate at which the disease is spread outside of the social group. The spatial and temporal predictions of the model were tested against badger demography and TB prevalence data derived from the field. When validated in space, the model generated population sizes and disease incidence that were consistent with the observed field population. We conclude that modelling TB dynamics must include spatial and temporal heterogeneity in life history parameters, social behaviour and the landscape. Based on parameter sensitivity and data availability, we suggest priorities for future empirical research on badgers and bovine tuberculosis.

The need for, and use of power analysis in tests of the role of species richness in ecosystem processes

OikosVolume 96, Issue 2 p. 383-385 The need for, and use of power analysis in tests of the role of species richness in ecosystem processes Graham H. R. Osler, Graham H. R. Osler Agriculture and Agri-food Canada, Lethbridge Research Centre, PO Box 3000, Lethbridge, AB, Canada T1J 4B1 ([email protected])Search for more papers by this author Graham H. R. Osler, Graham H. R. Osler Agriculture and Agri-food Canada, Lethbridge Research Centre, PO Box 3000, Lethbridge, AB, Canada T1J 4B1 ([email protected])Search for more papers by this author First published: 01 May 2002 https://doi.org/10.1034/j.1600-0706.2002.960220.xCitations: 2Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Citing Literature Volume96, Issue2February 2002Pages 383-385 RelatedInformation

Effectiveness of spinal manipulative therapy in the treatment of mechanical thoracic spine pain: A pilot randomized clinical trial

Background: To date, no substantiated studies have been performed to investigate the efficacy of spinal manipulative therapy on thoracic spinal syndromes. Objective: To investigate the effectiveness of spinal manipulative therapy in the treatment of mechanical thoracic spine pain. Study Design: A single-blind, randomized, comparative, controlled pilot study. Setting: Technikon Natal Chiropractic Clinic in Durban, South Africa. Participants: Thirty subjects selected from the general population (from 16 to 60 years old) were randomly divided into two different treatment groups of 15 each. Methods: The objective measurements collected were the thoracic spine ranges of motion with the BROM II goniometer and pain threshold with an algometer. The subjective information required completion of the Oswestry Back Pain Disability Index, short-form McGill Pain Questionnaire, and Numerical Pain Rating Scale-101 Questionnaire by the patient. These 3 forms and objective measurements were collected before the first and final treatment and again at the 1-month follow-up consultation. The data gathered were then statistically analyzed with use of a 95% confidence level. The nonparametric Mann-Whitney U test and the Wilcoxon signed-rank test were used to compare intergroup and intragroup data, respectively. This was conducted at the α =.05 level of confidence. Further assessment of the data was conducted by use of power analysis. Interventions: The treatment group received thoracic spinal manipulation. The placebo group received nonfunctional ultrasound application only. The research project was carried out so that both groups received 6 treatments over a period of 2 to 3 weeks. A 1-month follow-up appointment was scheduled after the final treatment to assess the relative long-term benefits of the two different treatments. Results: Statistically significant results (P ≤.025) were noted for the percentage of pain experienced (Numerical Pain Rating Scale) and for right and left lateral flexion during intergroup comparison after the final treatment. The final treatment results were maintained at the 1-month follow-up consultation; however, there were no further statistically significant results. It was noted that the power was weak, so the probability of committing type II error (falsely accepting the null hypothesis) for the other measurements was high. The intragroup analysis showed statistically significant improvements in the group that received spinal manipulative therapy in both subjective and objective measurements between the first to final treatment and the first treatment to the 1-month follow-up consultation. The placebo group analysis showed a statistically significant improvement in sensory pain only (subjective measurement) between the first treatment and the final treatment. Conclusions: This pilot study suggests that spinal manipulative therapy has greater benefits than placebo treatment. The sample size was small, therefore the findings of this trial study should not be considered conclusive but rather should be used as a foundation for planning future studies. In further studies a larger sample size will be necessary to identify subtle changes in measurement parameters and to add to the validity of the results. (J Manipulative Physiol Ther 2001;24:394-401)

Evaluation of the Greenland AMAP programme 1994–1995, by use of power analysis (illustrated by selected heavy metals and POPs)

The levels of PCBs, HCB, HCHs, DDTs, Cd, Pb, Hg and Se, and especially the variability in biota obtained during Phase 1 of the Greenland AMAP-programme have been used to illustrate the ability of the programme to detect differences in contaminant levels over time. The statistical power of t-tests of contaminant levels are illustrated according to various scenarios of magnitude of change, significance level and sample size. The statistical power of various time series of contaminant levels to detect linear trends in mean log-concentrations, including a random between-year variation component, is illustrated. We conclude that the ability to detect differences is rather poor for many combinations of contaminants and media, and that long time series are needed before temporal trends are likely to be detected.

Use of power analysis to develop detectable significance criteria for sea urchin toxicity tests

When sufficient data are available, the statistical power of a test can be determined using power analysis procedures. The term “detectable significance” has been coined to refer to this criterion based on power analysis and past performance of a test. This power analysis procedure has been performed with sea urchin ( Arbacia punctulata) fertilization and embryological development data from sediment porewater toxicity tests. Data from 3100 and 2295 tests for the fertilization and embryological development tests, respectively, were used to calculate the criteria and regression equations describing the power curves. Using Dunnett’s test, a minimum significant difference (MSD) ( β=0.05) of 15.5% and 19% for the fertilization test, and 16.4% and 20.6% for the embryological development test, for α≤0.05 and α≤0.01, respectively, were determined. The use of this second criterion reduces type I (false positive) errors and helps to establish a critical level of difference based on the past performance of the test.

Use of power analysis to develop detectable significance criteria for sea urchin toxicity tests

Abstract When sufficient data are available, the statistical power of a test can be determined using power analysis procedures. The term “detectable significance” has been coined to refer to this criterion based on power analysis and past performance of a test. This power analysis procedure has been performed with sea urchin (Arbacia punctulata) fertilization and embryological development data from sediment porewater toxicity tests. Data from 3100 and 2295 tests for the fertilization and embryological development tests, respectively, were used to calculate the criteria and regression equations describing the power curves. Using Dunnett's test, a minimum significant difference (MSD) (β = 0.05) of 15.5% and 19% for the fertilization test, and 16.4% and 20.6% for the embryological development test, for α ≤ 0.05 and α ≤ 0.01, respectively, were determined. The use of this second criterion reduces type I (false positive) errors and helps to establish a critical level of difference based on the past performance of the test.

The effect of predation by wintering cormorants Phalacrocorax carbo on grayling Thymallus thymallus and trout (Salmonidae) populations: two case studies from Swiss rivers. Comment

Summary Using the same data on the predation effect of cormorants Phalacrocorax carbo on the grayling Thymallus thymallus population located in the river Rhine below Lake Constance, Staub et al. (1992) and Suter (1995) reached contradictory conclusions (additional and compensatory mortality, respectively). This comment clarifies the reasons for the differences. The biomass of older fish (age classes 4 and 5–7) was not judged to be decreasing by Suter because of the use of inappropriate (parametric) inferential statistics and no use of power analysis. More powerful (non‐parametric) inferential tests show a significant decrease in biomass. Staub et al. compared fish numbers per length class in the angling catch (autumn) and in the spawning catch (spring). Differences were thought to be the effect of cormorant predation during winter. Suter converted fish numbers per length class into biomass per age class for the same comparisons. However, this conversion considered neither the length increases of grayling during winter (c. 1 cm) nor the dominance of faster growing males (> 1.5 cm longer than females of the same age) in the spawning catch. Thus, changes of the age pyramid during winter have been masked.