Minimum Detectable Effect Research Articles

Power calculation for causal inference in social science: sample size and minimum detectable effect determination

ABSTRACTThis paper presents the statistical concepts used in power calculations for experimental design. It provides detailed definitions of parameters used to perform power calculations, useful rules of thumb and different approaches that can be used when performing power calculations. The authors draw from real-world examples to calculate statistical power for individual and cluster randomised controlled trials. This paper provides formulae for sample size determination and minimum detectable effect (MDE) associated with a given statistical power. The paper is accompanied by the sample size and MDE calculator©, a free online tool that allows users to work with the formulae presented in Section 4.

An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials.

Prior research has investigated design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Less is known about parameters important for assessing differential impacts. This article develops a statistical framework for designing CRTs to assess differences in impact among student subgroups and presents initial estimates of critical parameters. Effect sizes and minimum detectable effect sizes for average and differential impacts are calculated before and after conditioning on effects of covariates using results from several CRTs. Relative sensitivities to detect average and differential impacts are also examined. Student outcomes from six CRTs are analyzed. Achievement in math, science, reading, and writing. The ratio of between-cluster variation in the slope of the moderator divided by total variance-the "moderator gap variance ratio"-is important for designing studies to detect differences in impact between student subgroups. This quantity is the analogue of the intraclass correlation coefficient. Typical values were .02 for gender and .04 for socioeconomic status. For studies considered, in many cases estimates of differential impact were larger than of average impact, and after conditioning on effects of covariates, similar power was achieved for detecting average and differential impacts of the same size. Measuring differential impacts is important for addressing questions of equity, generalizability, and guiding interpretation of subgroup impact findings. Adequate power for doing this is in some cases reachable with CRTs designed to measure average impacts. Continuing collection of parameters for assessing differential impacts is the next step.

Progress in the past decade: an examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences

ABSTRACT This article examines the statistical precision of cluster randomized trials (CRTs) funded by the Institute of Education Sciences (IES). Specifically, it compares the total number of clusters randomized and the minimum detectable effect size (MDES) of two sets of studies, those funded in the early years of IES (2002–2004) and those funded in the recent years (2011–2013). Overall, the average precision in terms of MDES of studies in the recent cohort was more than double that of the early cohort (i.e. 0.48 compared to 0.23). The findings suggest a consistent and substantial increase in the precision of CRTs funded by IES in the past decade which is a critical step towards designing studies that have the potential to yield high-quality evidence about the effectiveness of educational interventions.

Power estimation and sample size determination for replication studies of genome-wide association studies.

BackgroundReplication study is a commonly used verification method to filter out false positives in genome-wide association studies (GWAS). If an association can be confirmed in a replication study, it will have a high confidence to be true positive. To design a replication study, traditional approaches calculate power by treating replication study as another independent primary study. These approaches do not use the information given by primary study. Besides, they need to specify a minimum detectable effect size, which may be subjective. One may think to replace the minimum effect size with the observed effect sizes in the power calculation. However, this approach will make the designed replication study underpowered since we are only interested in the positive associations from the primary study and the problem of the “winner’s curse” will occur.ResultsAn Empirical Bayes (EB) based method is proposed to estimate the power of replication study for each association. The corresponding credible interval is estimated in the proposed approach. Simulation experiments show that our method is better than other plug-in based estimators in terms of overcoming the winner’s curse and providing higher estimation accuracy. The coverage probability of given credible interval is well-calibrated in the simulation experiments. Weighted average method is used to estimate the average power of all underlying true associations. This is used to determine the sample size of replication study. Sample sizes are estimated on 6 diseases from Wellcome Trust Case Control Consortium (WTCCC) using our method. They are higher than sample sizes estimated by plugging observed effect sizes in power calculation.ConclusionsOur new method can objectively determine replication study’s sample size by using information extracted from primary study. Also the winner’s curse is alleviated. Thus, it is a better choice when designing replication studies of GWAS. The R-package is available at: http://bioinformatics.ust.hk/RPower.html.

Ethnic Quotas and the Distribution of Public Benefits in India: A Replication and Reanalysis of Dunning and Nilekani (2013)

In contrast to previous research, Dunning and Nilekani (2013 APSR) find that household level data do not support the view that the distribution of targeted public benefits to scheduled castes and scheduled tribes has increased as a result of quotas in Indian village councils. Using a natural experiment created by rotating reserved council presidencies for these historically disadvantaged groups, the authors contend that the marginal effects of quotas are not significantly different from zero. Although a strong design, a transparent estimation strategy, and fully replicable results make the study compelling, we combine an intuitive presentation of power calculations and equivalence tests (Hartman and Hidalgo 2011), to show that the study is able to reject a null hypothesis of no effect only under an alternative hypothesis of a large difference in means. The study also lacks the power to reject a null of effect sizes previously documented in the literature on reservations in Indian village councils. Through the utilization of minimum detectable effects and equivalence testing, we provide a principled approach for evaluating null results in other settings.

Developing robust field survey protocols in landscape ecology: a case study on birds, plants and butterflies

Sustainable land management requires scientists to provide reliable data on diversity distribution patterns. Resource restrictions limit the affordable sampling effort, both with respect to number of survey sites and amount of effort per site. We compared different levels of survey effort in a case study in Central Romania, varying the number of repeats per site and number of survey sites. Target taxa were plants, birds and butterflies. For plants, we surveyed three 10 m2 plots and ten plots of 1 m2 at each site. For birds, we used point counts and for butterflies Pollard walks, in both cases with four repeats. We fitted hierarchical community models to estimate true species richness per site. Estimates of true species richness per site strongly correlated with observed species richness. However, hierarchical community models yielded unrealistically high estimates of true species richness per site, hence we used observed richness for further analyses. For each species group, we compared diversity indices from subsets of the dataset with the full dataset. Findings obtained with a reduced survey effort reflected well those obtained with full effort. Moreover, we conducted a power analysis to assess how the number of survey sites affected the minimum detectable effect of landscape heterogeneity on species richness, and found there was an exponential decrease in the minimum detectable effect with increasing number of sites. In combination, our findings suggest that assessing broad diversity patterns in abundant and readily detectable organisms may be possible with relatively low survey effort per site. Our study demonstrates the utility of conducting pilot studies prior to designing large-scale studies on diversity distribution patterns.

An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Background: Prior research has examined design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Little is known about parameters for assessing differential impacts for student subgroups. Objectives: This article develops the framework for optimally designing CRTs to detect differential impacts of programs by student subgroups and presents initial estimates of the critical parameters.Research Design: Estimates of intraclass correlation coefficients (ICCs) for the slope of the student-level moderator and Minimum Detectable Effect Sizes (MDESs) for average and differential impacts are calculated before and after conditioning on effects of covariates.Subjects: Data from six cluster randomized trials are analyzed with student outcomes in grades three through ten. Measures: Achievement in math, science, reading and writing is assessed with CRTs conducted in six states. Results: In figuring MDES for differential impact, the “ICC for the slope” – the between-cluster variation in the slope of the moderator divided by total variance – replaces the usual ICC. Estimates of median ICCs for the slope were .02 for gender and .04 for socioeconomic. Lower values of ICC for slope, compared to the usual ICC, contend with inflated influence of within-cluster variance in determining power to detect differential impact. For studies considered, after conditioning on covariates there was similar power to detect average and differential impacts (for socioeconomic status and gender) of the same size; and estimates of differential impact were often larger than of average impacts. Conclusions: Measuring differential impacts in CRTs is important for addressing questions of equity, generalizability, and to guide interpretation of subgroup impact findings. Adequate power to detect these effects appears reachable with CRTs designed to measure average impact. Ongoing collection of parameters for assessing differential impacts is the next step.

Functional neuroimaging abnormalities in idiopathic generalized epilepsy.

Magnetic resonance imaging (MRI) techniques have been used to quantitatively assess focal and network abnormalities. Idiopathic generalized epilepsy (IGE) is characterized by bilateral synchronous spike–wave discharges on electroencephalography (EEG) but normal clinical MRI. Dysfunctions involving the neocortex, particularly the prefrontal cortex, and thalamus likely contribute to seizure activity. To identify possible morphometric and functional differences in the brains of IGE patients and normal controls, we employed measures of thalamic volumes, cortical thickness, gray–white blurring, fractional anisotropy (FA) measures from diffusion tensor imaging (DTI) and fractional amplitude of low frequency fluctuations (fALFF) in thalamic subregions from resting state functional MRI. Data from 27 patients with IGE and 27 age- and sex-matched controls showed similar thalamic volumes, cortical thickness and gray–white contrast. There were no differences in FA values on DTI in tracts connecting the thalamus and prefrontal cortex. Functional analysis revealed decreased fALFF in the prefrontal cortex (PFC) subregion of the thalamus in patients with IGE. We provide minimum detectable effect sizes for each measure used in the study. Our analysis indicates that fMRI-based methods are more sensitive than quantitative structural techniques for characterizing brain abnormalities in IGE.

Strategies for Improving Power in School-Randomized Studies of Professional Development.

Group-randomized designs are well suited for studies of professional development because they can accommodate programs that are delivered to intact groups (e.g., schools), the collaborative nature of professional development, and extant teacher/school assignments. Though group designs may be theoretically favorable, prior evidence has suggested that they may be challenging to conduct in professional development studies because well-powered designs will typically require large sample sizes or expect large effect sizes. Using teacher knowledge outcomes in mathematics, we investigated when and the extent to which there is evidence that covariance adjustment on a pretest, teacher certification, or demographic covariates can reduce the sample size necessary to achieve reasonable power. Our analyses drew on multilevel models and outcomes in five different content areas for over 4,000 teachers and 2,000 schools. Using these estimates, we assessed the minimum detectable effect sizes for several school-randomized designs with and without covariance adjustment. The analyses suggested that teachers' knowledge is substantially clustered within schools in each of the five content areas and that covariance adjustment for a pretest or, to a lesser extent, teacher certification, has the potential to transform designs that are unreasonably large for professional development studies into viable studies.

Clinically important deterioration in patients undergoing lumbar spine surgery: a choice of evaluation methods using the Oswestry Disability Index, 36-Item Short Form Health Survey, and pain scales

Health-related quality of life (HRQOL) measures have become the mainstay for outcome appraisal in spine surgery. Clinically meaningful interpretation of HRQOL improvement has centered on the minimum clinically important difference (MCID). The purpose of this study was to calculate clinically important deterioration (CIDET) thresholds and determine a CIDET value for each HRQOL measure for patients undergoing lumbar fusion. Seven hundred twenty-two patients (248 males, 127 smokers, mean age 60.8 years) were identified with complete preoperative and 1-year postoperative HRQOLs including the Oswestry Disability Index (ODI), 36-Item Short Form Health Survey (SF-36), and numeric rating scales (0-10) for back and leg pain following primary, instrumented, posterior lumbar fusion. Anchor-based and distribution-based methods were used to calculate CIDET for each HRQOL. Anchor-based methods included change score, change difference, and receiver operating characteristic curve analysis. The Health Transition Item, an independent item of the SF-36, was used as the external anchor. Patients who responded "somewhat worse" and "much worse" were combined and compared with patients responding "about the same." Distribution-based methods were minimum detectable change and effect size. Diagnoses included spondylolisthesis (n = 332), scoliosis (n = 54), instability (n = 37), disc pathology (n = 146), and stenosis (n = 153). There was a statistically significant change (p < 0.0001) for each HRQOL measure from preoperatively to 1-year postoperatively. Only 107 patients (15%) reported being "somewhat worse" (n = 81) or "much worse" (n = 26). Calculation methods yielded a range of CIDET values for ODI (0.17-9.06), SF-36 physical component summary (-0.32 to 4.43), back pain (0.02-1.50), and leg pain (0.02-1.50). A threshold for clinical deterioration was difficult to identify. This may be due to the small number of patients reporting being worse after surgery and the variability across methods to determine CIDET thresholds. Overall, it appears that patients may interpret the absence of change as deterioration.

SAS Code for Calculating Intraclass Correlation Coefficients and Effect Size Benchmarks for Site-Randomized Education Experiments

When evaluators plan site-randomized experiments, they must conduct the appropriate statistical power analyses. These analyses are most likely to be valid when they are based on data from the jurisdictions in which the studies are to be conducted. In this method note, we provide software code, in the form of a SAS macro, for producing statistical power analyses using such data. The macro is designed to estimate statistical parameters for multiple school subjects and multiple grades in a single computer run. Among other convenient features, the macro calculates intraclass correlation coefficient confidence intervals, which are essential for planning studies conservatively. We also provide a SAS macro for calculating benchmarks–again, using locally-based–data for interpreting the feasibility of achieving the minimum detectable effect sizes that are calculated in statistical power analyses. The benchmarks are potentially useful for interpreting the meaning of the effect sizes that are achieved in completed studies, as well. We describe the macros, describe how we verified their accuracy, show how they can be useful to education evaluators, and give examples of their use with statewide educational assessment data. It is our hope that the macros will help evaluators use local data when conducting group-randomized studies.

PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies

This paper and the accompanying tool are intended to complement existing supports for conducting power analysis tools by offering a tool based on the framework of Minimum Detectable Effect Sizes (MDES) formulae that can be used in determining sample size requirements and in estimating minimum detectable effect sizes for a range of individual- and group-random assignment design studies and for common quasi-experimentaldesign studies. The paper and accompanying tool cover computation of minimum detectable effect sizes under the following study designs: individual random assignment designs, hierarchical random assignment designs (2-4 levels),block random assignment designs (2-4 levels), regression discontinuity designs (6 types), and short interrupted time-series designs. In each case, the discussion and accompanying tool consider the key factors associatedwith statistical power and minimum detectable effect sizes, including the level at which treatment occurs and the statistical models (e.g., fixed effect and random effect) used in the analysis. The tool also includes amodule that estimates for one and two level random assignment design studies the minimum sample sizes required in order for studies to attain user-defined minimum detectable effect sizes.

Short‐Term Impacts of a 4‐Lane Highway on American Black Bears in Eastern North Carolina

Short‐Term Impacts of a 4‐Lane Highway on American Black Bears in Eastern North Carolina

Multicenter Evaluation of Infant Lung Function Tests as Cystic Fibrosis Clinical Trial Endpoints

The conducting of clinical trials in infants with cystic fibrosis (CF) has been hindered by lack of sensitive outcome measures. To evaluate safety, feasibility, and ability to detect abnormalities in lung function of serial pulmonary function tests (PFTs) in infants with CF. Multicenter observational study using a commercial device, rigorous training, ongoing quality control, and over-reading of data by an independent panel. Raised volume rapid thoracoabdominal compression technique and plethysmography were performed at enrollment and at 6 and 12 months, with an additional 1-month reproducibility visit. A total of 342 procedures were performed in 100 infants with CF at 10 centers. FRC measurements were acceptable at a higher proportion of study visits (89%) than raised volume (72%) or fractional lung volume (68%) measurements. Average Z scores for many parameters differed significantly from historical control values. Mean (95% confidence interval) Z scores were: -0.52 (-0.78 to -0.25) for forced expiratory flow at 75% (FEF₇₅) for FVC; 1.92 (1.39-2.45) for FRC; 1.22 (0.68-1.76) for residual volume; 0.87 (0.60-1.13) for FRC/total lung capacity; and 0.66 (0.27-1.06) for residual volume/total lung capacity. For future multicenter clinical trials using infant PFTs as primary endpoints, minimum detectable treatment effects are presented for several sample sizes. In this 10-center study, key PFT measures were significantly different in infants with CF than in historical control subjects. However, infant PFTs do not yet appear ready as primary efficacy endpoints for multicenter clinical trials, particularly at inexperienced sites, based on acceptability rates, variability, and potentially large sample sizes required to detect reasonable treatment effects.

Testing for qualitative interactions between stages in an adaptive study

I consider the underlying structure for a test of qualitative interaction of a treatment when assessing heterogeneity between stages in an adaptive trial. Since decisions about the clinical utility of a drug are based on the balance of risks and benefits, a quantitative interaction in treatment efficacy across different groups could lead to qualitatively different decisions. Thus, the difference between quantitative and qualitative interactions is not a true dichotomy. I show that the standard tests for qualitative interactions (Gail and Simon,Biometrics 1985; 41:361-372; Piantadosi and Gail, Statist. Med. 1993; 12:1239-1248) are very conservative in this application. Theoretical calculations in a simpler situation confirm that the published criteria are very conservative, which may help explain why the tests are known to have very low power to detect interaction. I introduce the concept of 'minimum detectable effect', which is the smallest effect that a study could identify as statistically significant. I propose that important heterogeneity between stages in an adaptive trial be identified when two criteria are met. First, at least one individual stage must be below the overall study mean by at least the minimum detectable effect. Second, using an appropriate critical value based on simulations, there must be statistically significant heterogeneity between the stages.

An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences

This article examines the power analyses for the first wave of group-randomized trials funded by the Institute of Education Sciences. Specifically, it assesses the precision and technical accuracy of the studies. The authors identified the appropriate experimental design and estimated the minimum detectable standardized effect size (MDES) for each study under plausible assumptions about intra-class correlations, covariate-outcome correlations, and explanatory effects of blocking. The MDESs ranged from .18 to .40 for studies funded by the National Center for Education Evaluation and Regional Assistance (NCEE) and from .20 to 1.0 for studies funded by the National Center for Education Research (NCER). They evaluated the technical accuracy by comparing the estimated MDES to the MDES stated in the proposal. Studies funded by the NCEE tended to be very accurate: estimated MDESs and stated MDESs were similar, whereas studies funded by the NCER were less accurate. However, the technical accuracy of the NCER studies improved over time.

Generating and using evidence to guide public policy and practice: Lessons from the Campbell Test-bed Project

This paper offers lessons from a three-year Test-bed project that tested systematic review practices developed by the Campbell Collaboration and the Cochrane Collaboration. Under the Test-bed project 14 systematic reviews were completed of interventions in crime prevention, social welfare, and education. (References to the products of these test-bed reviews are included in the reference list, preceded by an asterisk). Building on this experience, the authors recommend structuring future reviews around well-defined review topics more explicitly focused on particular interventions, and constraining literature search to evaluations of relevant interventions. Reviewers should analyze and report findings from RCTs separately from non-RCT studies and report on impact estimates in natural units, instead of relying solely on effect size metrics. Further, reviewers should report intent-to-treat estimates as the causally valid outcomes from RCTs. Analyses of impacts for treated sub-groups should be reported as non-experimental findings. More attention should be given to the minimum detectable effect a study can support, as well as any information on the possible costs and benefits of the intervention. Pooling results from studies of disparate interventions, populations, and contexts is not recommended. Meta-analysis should be reserved for homogeneous clusters of interventions studies. Forest plots are helpful for presenting study findings and confidence limits. However, simple bar charts preserve important information on the base levels for the outcomes. Finally reviewers should define a priori the minimum data set or required elements that allow study inclusion, and use this information systematically in making decisions about what evidence to admit into the review.

Toxicity of uranium and copper individually, and in combination, to a tropical freshwater macrophyte ( Lemna aequinoctialis)

Copper (Cu) and uranium (U) are of potential ecotoxicological concern to tropical freshwater biota in northern Australia, as a result of mining activities. Few data are available on the toxicity of U, and no data are available on the toxic interaction of Cu and U, to freshwater biota. This study determined the toxicity of Cu and U individually, and in combination, to a tropical freshwater macrophyte, Lemna aequinoctialis (duckweed), in a synthetic soft water (27 °C; pH, 6.5; hardness, 40 mg CaCO 3 l −1, alkalinity, 16 mg CaCO 3 l −1), typical of many fresh surface waters in coastal northern Australia. The growth rate of L. aequinoctialis decreased with increasing Cu or U concentrations, with the concentration of Cu inhibiting growth by 50% (EC50) being 16 ± 1.0 μg l −1, with a minimum detectable effect concentration (MDEC) of 3.2 μg l −1. The concentration of U inhibiting growth by 50% (EC50) was 758 ± 35 μg l −1 with a MDEC of 112 μg l −1. The EC50 value for the exposure of L. aequinoctialis to equitoxic mixtures of Cu and U was significantly ( P ⩽ 0.05) higher than one toxic unit (1.35; 95% confidence interval, 1.18–1.52), indicating that the combined effects of Cu and U are less than additive (antagonistic). Therefore, inhibition of the growth rate of L. aequinoctialis was reduced when Cu and U were present in equitoxic mixtures, relative to individual metal exposures. Since non-additive (e.g. antagonistic) interactions of metal mixtures cannot be predicted using current mixture models, these results have important potential implications for the protection of freshwater ecosystems through the derivation of national water quality guidelines.

Multivariate Analysis of Paired Watershed Data to Evaluate Agricultural Best Management Practice Effects on Stream Water Phosphorus

Quantification of the effects of management programs on water quality is critical to agencies responsible for water resource protection. This research documents reductions in stream water phosphorus (P) loads resulting from agricultural best management practices (BMPs) implemented as part of an effort to control eutrophication of Cannonsville Reservoir, a drinking water supply for New York City. Dairy farms in the upstate New York reservoir basin were the target of BMPs designed to reduce P losses. A paired watershed study was established on one of these farms in 1993 to evaluate changes in P loading attributable to implementation of BMPs that included manure management, rotational grazing, and improved infrastructure. Intensive stream water monitoring provided data to calculate P loads from the 160-ha farm watershed for all runoff events during a two-year pre-treatment period and a four-year post-treatment period. Statistical control for inter-annual climatic variability was provided by matched P loads from a nearby 86-ha forested watershed, and by several event flow variables measured at the farm. A sophisticated multivariate analysis of covariance (ANCOVA) provided estimates of both seasonal and overall load reductions. Statistical power and the minimum detectable treatment effect (MDTE) were also calculated. The results demonstrated overall event load reductions of 43% for total dissolved phosphorus (TDP) and 29% for particulate phosphorus (PP). Changes in farm management practices and physical infrastructure clearly produced decreases in event P losses measurable at the small watershed scale.

Thermoluminescence dosimetric properties of a new thin beta detector(LiF:Mg, Cu, P; GR-200F) in comparison with highly sensitiveAl2O3:C beta dosimeters

There is an increasing need for efficient beta detectors to fulfil ICRU recommendationsfor new quantities especially in the field of medical physics and retrospectivedosimetry. The thermoluminescence properties of thin LiF:Mg, Cu, P(GR-200F) tapes produced in 1998 by Sange Company, People’s Republicof China, are investigated and compared with those of highly sensitive thinAl2O3:C beta detectors as regards their applicability in the detection of low energy photons andbeta particles. The radiation dose response, minimum detectable dose, reproducibility ofmeasurements and effect of residual signal at low dose are assessed for the possible low levelbeta dosimetry use. The radiation dose response and photon and beta detection efficienciesare tested under practical laboratory conditions. The effects of indoor fluorescent light andresidual signal after the first read-out are investigated with a view to optimising handlingconditions such as post-irradiation and pre-heating treatments for routine dosimetry. Thephoton energy responses of the detectors are investigated using 150 keV filtered x-rays and60Co γ-rays.