Simulation-based Power Analysis Research Articles

Simulation-Based Power Analyses for the Smallest Effect Size of Interest: A Confidence-Interval Approach for Minimum-Effect and Equivalence Testing

Effect sizes are often used in psychology because they are crucial when determining the required sample size of a study and when interpreting the implications of a result. Recently, researchers have been encouraged to contextualize their effect sizes and determine what the smallest effect size is that yields theoretical or practical implications, also known as the “smallest effect size of interest” (SESOI). Having a SESOI will allow researchers to have more specific hypotheses, such as whether their findings are truly meaningful (i.e., minimum-effect testing) or whether no meaningful effect exists (i.e., equivalence testing). These types of hypotheses should be reflected in power analyses to accurately determine the required sample size. Through a confidence-interval-focused approach and simulations, I show how to conduct power analyses for minimum-effect and equivalence testing. Moreover, I show that conducting a power analysis for the SESOI might result in inconclusive results. This confidence-interval-focused simulation-based power analysis can be easily adopted to different types of research areas and designs. Last, I provide recommendations on how to conduct such simulation-based power analyses.

Environmental, behavioral, and design-related factors affect accuracy and precision of beluga abundance estimates from aerial surveys

Abundance estimation of wildlife populations is frequently derived from systematic survey data. Accuracy and precision of estimates, however, depend on the number of replicate surveys, and on adjustments made for animals unavailable to (availability bias), or available but undetected (perception bias) by observers. This study offers a comprehensive analysis of the relative influence of methodological, environmental and behavioral factors on availability bias estimates from photographic and visual aerial surveys of a small cetacean with a highly clumped distribution, the beluga (Delphinapterus leucas). It also estimates the effect of the number of surveys on accuracy and precision of abundance estimates, using 28 replicate visual surveys flown within a 16—29 day window depending on survey year. Availability bias was estimated using detailed dive data from 27 beluga from the St. Lawrence Estuary, Canada, and applied to systematic visual and photographic aerial surveys of this population, flown using various survey platforms. Dive and surface interval durations varied among individuals, and averaged (weighted) 176.6 s (weighted s.e. = 12.6 s) and 51.6 s (weighted s.e. = 4.5 s), respectively. Dive time and instantaneous availability, but not surface time, were affected by local turbidity, seafloor depth, whale behavior (i.e., whether beluga were likely in transit or not), and latent processes that were habitat-specific. Overall, adjustments of availability for these effects remained minor compared to effects from survey design (photographic or visual) and type of platform, and observer search patterns. For instance, mean availability varied from 0.33—0.38 among photographic surveys depending on sightings distribution across the study area, but exceeded 0.40 for all visual surveys. Availability also varied considerably depending on whether observers searched within 0-90° (0.42—0.60) or 170° (0.70—0.80). Simulation-based power analysis indicates a large benefit associated with conducting more than 1 or 2 survey reps, but a declining benefit of conducting > 5—10 survey reps. An increase in sample size from 2, to 5, and 10 reps decreased the CV from 30, to 19 and 13%, respectively, and increased the probability of the abundance estimate being within 15% of true abundance from 0.42, to 0.59 and 0.69 in species like beluga.

Sample size planning for complex study designs: A tutorial for the mlpwr package.

A common challenge in designing empirical studies is determining an appropriate sample size. When more complex models are used, estimates of power can only be obtained using Monte Carlo simulations. In this tutorial, we introduce the R package mlpwr to perform simulation-based power analysis based on surrogate modeling. Surrogate modeling is a powerful tool in guiding the search for study design parameters that imply a desired power or meet a cost threshold (e.g., in terms of monetary cost). mlpwr can be used to search for the optimal allocation when there are multiple design parameters, e.g., when balancing the number of participants and the number of groups in multilevel modeling. At the same time, the approach can take into account the cost of each design parameter, and aims to find a cost-efficient design. We introduce the basic functionality of the package, which can be applied to a wide range of statistical models and study designs. Additionally, we provide two examples based on empirical studies for illustration: one for sample size planning when using an item response theory model, and one for assigning the number of participants and the number of countries for a study using multilevel modeling.

Tutorial: Power Analyses for Interaction Effects in Cross-Sectional Regressions

Interaction analyses (also termed “moderation” analyses or “moderated multiple regression”) are a form of linear regression analysis designed to test whether the association between two variables changes when conditioned on a third variable. It can be challenging to perform a power analysis for interactions with existing software, particularly when variables are correlated and continuous. Moreover, although power is affected by main effects, their correlation, and variable reliability, it can be unclear how to incorporate these effects into a power analysis. The R package InteractionPoweR and associated Shiny apps allow researchers with minimal or no programming experience to perform analytic and simulation-based power analyses for interactions. At minimum, these analyses require the Pearson’s correlation between variables and sample size, and additional parameters, including reliability and the number of discrete levels that a variable takes (e.g., binary or Likert scale), can optionally be specified. In this tutorial, we demonstrate how to perform power analyses using our package and give examples of how power can be affected by main effects, correlations between main effects, reliability, and variable distributions. We also include a brief discussion of how researchers may select an appropriate interaction effect size when performing a power analysis.

Evaluation of a smartphone application for self-help for patients with social anxiety disorder: a randomized controlled study—SMASH

BackgroundThere is growing evidence that Internet-based cognitive behavioral therapy (ICBT) is as effective as a stand-alone treatment and helps facilitating access to treatment. Given the complexity of the treatment, we argue that the effect of ICBT could be even greater if guided by a therapist, as this could increase treatment adherence. We modified an established and well-evaluated treatment approach and developed a mobile application for treating social anxiety disorder (SAD). In the present study, we compare the efficacy of app use alone (APP) with video-based, therapist-guided app use (TG-APP) and with a wait-list control group (WLC) in terms of symptom reduction, and various secondary outcomes such as increase in quality of life or decrease of general psychological distress.Methods/designA within-between interaction design with randomization to one of three conditions will be used. In the APP condition, patients receive only the app without any additional contact with therapists, while in the TG-APP condition, therapists provide 8 sessions of video-based treatment in addition to using the app. The study will be conducted in two university outpatient treatment centers with reliably diagnosed SAD patients. The primary outcome will be defined as change in SAD symptoms, as measured by the Liebowitz Social Anxiety Scale (expert rating). Furthermore, a wide range of self-reports and clinician ratings for other symptoms (depression, general psychopathology) or quality of life will be used. A simulation-based power analysis for a 3 × 2 interaction effect (group × time) on the primary outcome in a linear mixed model resulted in a total sample size of N = 165.DiscussionThe present study will be one of the first to examine the additional benefit of therapist-guided video sessions regarding the use of an app treating SAD. Study results are pivotal to future treatment application in SAD.

Silicon-correlated Simulation Methodology of EM Side-channel Leakage Analysis

Cryptography hardware is vulnerable to side-channel (SC) attacks on power supply current flow and electromagnetic (EM) emission. This article proposes simulation-based power and EM side-channel leakage analysis (SCLA) techniques on a cryptographic integrated circuit (IC) chip in system level assembly. SCLA measures SC leakage metrics including T-score, SC leakage score, and the number of measurement traces to disclosure, leveraged by a secure system-on-chip design flow toward SC attack resiliency and SC leakage sign off. Power SCLA features the tracking of security sensitive registers within cryptographic logic paths and the automatic assignments of probe points on associated physical power nets. Power supply current traces are efficiently simulated for the large set of input payloads, with direct vector-based and vector-less random switching controls. EM SCLA evaluates magnetic fields created by every piece of metal wiring in metal stacks where power supply current of cryptographic processing flows. The EM emission and EM SCLA from the backside Si surface of an IC chip in flip-chip packaging are experimentally examined with a 0.13 μm test chip. The proposed simulation-based SCLA exhibits the SC leakage metrics of on-chip location and direction dependency as accurately as in the measurements.

PowerLAPIM: An application to conduct power analysis for linear and quadratic longitudinal actor–partner interdependence models in intensive longitudinal dyadic designs

The longitudinal actor–partner interdependence model (L-APIM) is used to study actor and partner effects, both linear and curvilinear, in dyadic intensive longitudinal data. A burning question is how to conduct power analyses for different L-APIM variants. In this paper, we introduce an accessible power analysis application, called PowerLAPIM, and provide a hands-on tutorial for conducting simulation-based power analyses for 32 L-APIM variants. With PowerLAPIM, we target the number of dyads needed, but not the number of repeated measurements for both partners (which is often fixed in longitudinal studies). PowerLAPIM allows to study moderation of linear and quadratic actor and partner effects by incorporating time-varying covariates or a categorical dyad-level predictor to test group differences. We also provide the functionality to account for serial dependency in the outcome variable by including autoregressive effects. Building on existing study that can yield estimates and thus plausible values of relevant model parameters, we illustrate how to perform a power analysis for a future study. In this illustration, we also demonstrate how to run a sensitivity analysis, to assess the impact of uncertainty about the model parameters, and of changes in the number of repeated measurements.

An empirically based power primer for laboratory aggression research.

Recent reviews suggest that, like much of the psychological literature, research studies using laboratory aggression paradigms tend to be underpowered to reliably locate commonly observed effect sizes (e.g., r = ~.10-.20, Cohen's d = ~0.20-0.40). In an effort to counter this trend, we provide a "power primer" that laboratory aggression researchers can use as a resource when planning studies using this methodology. Using simulation-based power analyses and effect size estimates derived from recent literature reviews, we provide sample size recommendations based on type of research question (e.g., main effect vs. two-way vs. three-way interactions) and correlations among predictors. Results highlight the large number of participants that must be recruited to reach acceptable (~80%) power, especially for tests of interactions where the recommended sample sizes far exceed those typically employed in this literature. These discrepancies are so substantial that we urge laboratory aggression researchers to consider a moratorium on tests of three-way interactions. Although our results use estimates from the laboratory aggression literature, we believe they are generalizable to other lines of research using behavioral tasks, as well as psychological science more broadly. We close by offering a series of best practice recommendations and reiterating long-standing calls for attention to statistical power as a basic element of study planning.

Editor's Spotlight/Take 5: How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis.

Editor's Spotlight/Take 5: How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis.

How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis.

How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis.

Amending survey design to improve statistical inferences: Monitoring recruitment of juvenile reef fish in the eastern Gulf of Mexico

Many species of interest to management and conservation remain data-limited, and the data that are available are often unable to produce statistically reliable population trends. We examined 10 years of juvenile reef fish catch data from two gear-specific, fishery-independent surveys in the eastern Gulf of Mexico to assess our ability 1) to characterize population trends for various reef species sampled using haul seines or otter trawls and 2) to amend our survey, as logistics would allow, to improve those characterizations. Species richness, catch-per-unit-effort, and length-frequency distributions were generally similar between gear types, suggesting a single survey (i.e., one gear type) may be sufficient. Simulation-based power analyses for the reef fish species indicated that overall, otter trawl data provided greater power for detecting trends over a 10-year period (41–75 % probability to detect a 50 % change in abundance) than haul seine data (27–52 %). Likewise, otter trawl data provided greater power for detecting trends from one year to the next (22–53 % probability to detect a 50 % change in abundance in one year) than haul seine data (15–33 %). Simplifying data collection from two surveys to a single, trawl-only survey (and approximately doubling the number of trawl tows) was more efficient, more effective, and more powerful (64–91 % for the amended design with more tows) in detecting abundance trends. Furthermore, by increasing sample size but retaining all other design elements, the data collected during the trawl survey before and after the change remain comparable; the time series was not interrupted. These changes increase our confidence in estimating population trends, predicting productivity, and informing management and conservation decisions.

Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R

Mixed-effects models are a powerful tool for modeling fixed and random effects simultaneously, but do not offer a feasible analytic solution for estimating the probability that a test correctly rejects the null hypothesis. Being able to estimate this probability, however, is critical for sample size planning, as power is closely linked to the reliability and replicability of empirical findings. A flexible and very intuitive alternative to analytic power solutions are simulation-based power analyses. Although various tools for conducting simulation-based power analyses for mixed-effects models are available, there is lack of guidance on how to appropriately use them. In this tutorial, we discuss how to estimate power for mixed-effects models in different use cases: first, how to use models that were fit on available (e.g. published) data to determine sample size; second, how to determine the number of stimuli required for sufficient power; and finally, how to conduct sample size planning without available data. Our examples cover both linear and generalized linear models and we provide code and resources for performing simulation-based power analyses on openly accessible data sets. The present work therefore helps researchers to navigate sound research design when using mixed-effects models, by summarizing resources, collating available knowledge, providing solutions and tools, and applying them to real-world problems in sample sizing planning when sophisticated analysis procedures like mixed-effects models are outlined as inferential procedures.

Considerations of variability and power for long-term monitoring of stream fish assemblages

Little attention has been given to optimizing statistical power for monitoring stream fish assemblages. We explored the relationship between temporal variability and statistical power using 34 metrics from fish community data collected annually at six sites over 10 years via electrofishing. Metric variability differed by the life stage and group of species considered, use of abundance or mass data, and data standardization technique. Lower variability was associated with community data, abundance data, and time-based standardizations, while greater variability was associated with young-of-the-year data, mass data, and area-based standardizations. Simulation-based power analysis indicated metric choice, and to a lesser degree, monitoring design (annual, biennial, endpoints, or haphazard sampling) influenced power to detect change. Across a fixed number of surveys (N = 60), endpoints sampling performed best. The N needed to detect change was heavily dependent upon metric choice for all monitoring designs, with the most biologically specific metrics requiring greater N. Large savings in effort and resource expenditure can be obtained utilizing biologically relevant metrics that are robust to temporal noise within an appropriate sampling design.

Selection of the Number of Participants in Intensive Longitudinal Studies: A User-Friendly Shiny App and Tutorial for Performing Power Analysis in Multilevel Regression Models That Account for Temporal Dependencies

In recent years, the popularity of procedures for collecting intensive longitudinal data, such as the experience-sampling method, has increased greatly. The data collected using such designs allow researchers to study the dynamics of psychological functioning and how these dynamics differ across individuals. To this end, the data are often modeled with multilevel regression models. An important question that arises when researchers design intensive longitudinal studies is how to determine the number of participants needed to test specific hypotheses regarding the parameters of these models with sufficient power. Power calculations for intensive longitudinal studies are challenging because of the hierarchical data structure in which repeated observations are nested within the individuals and because of the serial dependence that is typically present in these data. We therefore present a user-friendly application and step-by-step tutorial for performing simulation-based power analyses for a set of models that are popular in intensive longitudinal research. Because many studies use the same sampling protocol (i.e., a fixed number of at least approximately equidistant observations) within individuals, we assume that this protocol is fixed and focus on the number of participants. All included models explicitly account for the temporal dependencies in the data by assuming serially correlated errors or including autoregressive effects.

Simulation-Based Power Analysis for Factorial Analysis of Variance Designs

Researchers often rely on analysis of variance (ANOVA) when they report results of experiments. To ensure that a study is adequately powered to yield informative results with an ANOVA, researchers can perform an a priori power analysis. However, power analysis for factorial ANOVA designs is often a challenge. Current software solutions do not allow power analyses for complex designs with several within-participants factors. Moreover, power analyses often need [Formula: see text] or Cohen’s f as input, but these effect sizes are not intuitive and do not generalize to different experimental designs. We have created the R package Superpower and online Shiny apps to enable researchers without extensive programming experience to perform simulation-based power analysis for ANOVA designs of up to three within- or between-participants factors. Predicted effects are entered by specifying means, standard deviations, and, for within-participants factors, the correlations. The simulation provides the statistical power for all ANOVA main effects, interactions, and individual comparisons. The software can plot power across a range of sample sizes, can control for multiple comparisons, and can compute power when the homogeneity or sphericity assumption is violated. This Tutorial demonstrates how to perform a priori power analysis to design informative studies for main effects, interactions, and individual comparisons and highlights important factors that determine the statistical power for factorial ANOVA designs.

The ACA Medicaid Expansions and Opioid Mortality: Is There a Link?

The Medicaid expansions made addiction treatment more accessible but they also made it less costly to obtain the prescription opioids that can trigger an addiction. We investigated the association between the Medicaid expansions and drug-related deaths. We add to the literature by explicitly accounting for the properties of illicit drug markets and by conducting a simulation-based power analysis to assess whether a plausible change in drug-related mortality could be detected with our data. We identify three main challenges in isolating the effect of the Medicaid expansions on drug-related mortality that cannot be sufficiently addressed with current data: (a) nonparallel preexpansion trends in drug-related mortality, (b) the contemporaneous surge in the supply of illicitly manufactured fentanyl, and (c) lack of statistical power. We argue that more comprehensive data are needed to answer this question.

The Peril of Power: A Tutorial on Using Simulation to Better Understand When and How We Can Estimate Mediating Effects.

Mediation analyses are valuable for examining mechanisms underlying an association, investigating possible explanations for nonintuitive results, or identifying interventions that can improve health in the context of nonmanipulable exposures. However, designing a study for the purpose of answering a mediation-related research question remains challenging because sample size and power calculations for mediation analyses are typically not conducted or are crude approximations. Consequently, many studies are probably conducted without first establishing that they have the statistical power required to detect a meaningful effect, potentially resulting in wasted resources. In an effort to advance more accurate power calculations for estimating direct and indirect effects, we present a tutorial demonstrating how to conduct a flexible, simulation-based power analysis. In this tutorial, we compare power to estimate direct and indirect effects across various estimators (the Baron and Kenny estimator (J Pers Soc Psychol. 1986;51(6):1173-1182), inverse odds ratio weighting, and targeted maximum likelihood estimation) using various data structures designed to mimic important features of real data. We include step-by-step commented R code (R Foundation for Statistical Computing, Vienna, Austria) in an effort to lower implementation barriers to ultimately improving power assessment in mediation studies.

Assessing water quality changes in agricultural drainages: Examples from oxbow lake tributaries in northwestern Mississippi, United States, and simulation-based power analyses

Hydrology (streamflow and stage) and water quality (suspended sediment, total nitrogen [N], ammonia [NH₃], total Kjeldahl N, nitrate plus nitrite [NO₃ + NO₂], and total phosphorus [TP]) were monitored in two small agricultural drainages in northwestern Mississippi to document changes in water quality that coincided with the implementation of best management practices (BMPs) in upstream drainages. Using an event-based data set and bootstrapping techniques, we tested for difference and equivalence in median event concentration and differences in concentration-streamflow (C-Q) relationships between an early and late period at each site, where most of the major BMP implementation occurred during the early period. Results for Bee Lake Tributary were inconclusive. Using 95% confidence intervals, none of the constituents were statistically different or equivalent for median event concentrations between the periods, indicating a lack of evidence to determine whether water quality had changed or stayed the same, and only TP had a significantly higher C-Q intercept during the late period. At Lake Washington Tributary, more than half the constituents had a significantly different median, slope, or intercept between periods, indicating a 35% or more decrease in event concentration following a period of intense BMP implementation. These mixed results could be due to a variety of differences between the sites including the type and timing of BMP implementation, production practices, and crop types. We also used the monitoring data to generate synthetic data and performed a simulation-based power analysis to explore the ability to detect change under 25 scenarios of sampled event counts and hypothetical percentage changes. The simulation-based power analysis indicated that natural variability in event concentration and flow hindered our ability to detect change. The results from this study can be used to guide future decisions pertaining to monitoring efforts in small agricultural drainage basins in the Mississippi Alluvial Plain.

On the Importance of Power Analyses for Cognitive Modeling

The high prevalence of underpowered empirical studies has been identified as a centerpiece of the current crisis in psychological research. Accordingly, the need for proper analyses of statistical power and sample size determination before data collection has been emphasized repeatedly. In this commentary, we argue that—contrary to the opinions expressed in this special issue’s target article—cognitive modeling research will similarly depend on the implementation of power analyses and the use of appropriate sample sizes if it aspires robustness. In particular, the increased desire to include cognitive modeling results in clinical and brain research raises the demand for assessing and ensuring the reliability of parameter estimates and model predictions. We discuss the specific complexity of estimating statistical power for modeling studies and suggest simulation-based power analyses as a solution to this challenge.

Statistical power of dynamic occupancy models to identify temporal change: Informing the North American Bat Monitoring Program

Dynamic occupancy models provide a flexible framework for estimating and mapping species occupancy patterns over space and time for large-scale monitoring programs (e.g., the North American Bat Monitoring Program (NABat), the Amphibian Research and Monitoring Initiative). Challenges for designing surveys using the dynamic occupancy modeling framework include defining appropriate derived trend parameters, and providing usable tools for researchers to conduct project-specific sample size investigations. We present a simulation-based power analysis framework for dynamic occupancy models that allows for the incorporation of the underlying environmental space (i.e., as covariates) within a specific study region to inform sample size estimation. We investigate two definitions of temporal trend: (1) a gradual, sustained (linear or nonlinear) change over a period of many years, and (2) an abrupt increase or decrease between two time periods. We draw upon pilot data collected following NABat protocols to inform assumed data generating values in a demonstration of our approach. Due to the complicated parameter structure of dynamic occupancy models, we emphasize the importance of visualizing simulated changes over time based on different parameter settings prior to conducting a power analysis. Our simulations revealed that the linearity of short-term trends (five years in our investigation) conferred higher power with lower sample size than longer trends where occupancy probabilities approached zero (ten years in our investigation). We provide an example of how to use our tools to conduct customized investigations using questions posed by NABat, and in doing so, we shed light on general guidelines that can be applied to programs monitoring species occupancy for other taxa. Importantly, we created an R package to execute our approach for informing program-, species-, and study-specific investigations aimed at identifying changes in species occupancy.