Underlying Effect Size Research Articles

Sensitivity analysis with iterative outlier detection for systematic reviews and meta-analyses.

Meta-analysis is a widely used tool for synthesizing results from multiple studies. The collected studies are deemed heterogeneous when they do not share a common underlying effect size; thus, the factors attributable to the heterogeneity need to be carefully considered. A critical problem in meta-analyses and systematic reviews is that outlying studies are frequently included, which can lead to invalid conclusions and affect the robustness of decision-making. Outliers may be caused by several factors such as study selection criteria, low study quality, small-study effects, and so on. Although outlier detection is well-studied in the statistical community, limited attention has been paid to meta-analysis. The conventional outlier detection method in meta-analysis is based on a leave-one-study-out procedure. However, when calculating a potentially outlying study's deviation, other outliers could substantially impact its result. This article proposes an iterative method to detect potential outliers, which reduces such an impact that could confound the detection. Furthermore, we adopt bagging to provide valid inference for sensitivity analyses of excluding outliers. Based on simulation studies, the proposed iterative method yields smaller bias and heterogeneity after performing a sensitivity analysis to remove the identified outliers. It also provides higher accuracy on outlier detection. Two case studies are used to illustrate the proposed method's real-world performance.

On the importance of modeling the invisible world of underlying effect sizes

The headline findings from the Open Science Collaboration (2015)―namely, that 36% of original experiments replicated at p < .05, with the overall replication effect sizes being half as large as the original effects―cannot be meaningfully interpreted without a formal model. A simple model-based approach might ask: what would the state of original science be and what would replication results show if original experiments tested true effects half the time (prior odds = 1), true effects had a medium effect size (Cohen’s δ = 0.50), and power to detect true effects was 50%? Assuming no questionable research practices, 91% of p < .05 findings in the original literature would be true positives. However, only 58% of original p < .05 findings would be expected to replicate using the Open Science Collaboration approach, and the replication effects overall would be only ~60% as large as the original effects. A minor variant of this model yields an expected replication rate of only 45%, with overall replication effect sizes dropping by half. If the state of original science is as grim as a non-model-based (i.e., intuitive) interpretation of the Open Science Collaboration data suggests, should it be this easy to largely account for those findings using a model in which 91% of statistically significant findings in the original science literature are true positives? Claims that the findings reported by the Open Science Collaboration indicate a replication crisis should not be based solely on intuition but should instead be accompanied by a specific model that supports that interpretation.

Sample size planning for multiple contrast tests.

Sample size calculations for two (independent) samples are well established and applied in (pre-)clinical research. When planning several samples, which is common in, for example, preclinical studies, sample size planning tools based on analysis of variance methods are available. Since the underlying effect sizes of these methods are often hard to interpret and to provide for the sample size planning, we employ multiple contrast test procedures for sample size computations in both parametric (under normality assumption) and nonparametric designs using Steel-type tests. Since the exact distributions of the test statistics are unknown under the alternative and variance heterogeneity, we use approximate solutions. Furthermore, since no closed formula for the sample size is available, we use numerical approximations for their computation. Extensive simulation studies are finally conducted to assess the quality of the approximations. It turns out that the methods are accurate in the sense that the multiple contrast test procedures reach the target power to detect the alternative of interest with the sample size computed. The developed procedures are a valuable tool to plan (pre-)clinical trials with several samples and are easily accessible in publicly availablesoftware.

Fixed-effect Versus Random-effects Models for Meta-analyses: Fixed-effect Models

A fixed-effect model considers a common underlying effect size for all the studies included in a meta-analysis. In the face of appreciable between-study heterogeneity, a fixed-effect model is a valuable tool when precision is the priority.

Theoretical false positive psychology.

A fundamental goal of scientific research is to generate true positives (i.e., authentic discoveries). Statistically, a true positive is a significant finding for which the underlying effect size (δ) is greater than 0, whereas a false positive is a significant finding for which δ equals 0. However, the null hypothesis of no difference (δ = 0) may never be strictly true because innumerable nuisance factors can introduce small effects for theoretically uninteresting reasons. If δ never equals zero, then with sufficient power, every experiment would yield a significant result. Yet running studies with higher power by increasing sample size (N) is one of the most widely agreed upon reforms to increase replicability. Moreover, and perhaps not surprisingly, the idea that psychology should attach greater value to small effect sizes is gaining currency. Increasing N without limit makes sense for purely measurement-focused research, where the magnitude of δ itself is of interest, but it makes less sense for theory-focused research, where the truth status of the theory under investigation is of interest. Increasing power to enhance replicability will increase true positives at the level of the effect size (statistical true positives) while increasing false positives at the level of theory (theoretical false positives). With too much power, the cumulative foundation of psychological science would consist largely of nuisance effects masquerading as theoretically important discoveries. Positive predictive value at the level of theory is maximized by using an optimal N, one that is neither too small nor too large.

Sleep deprivation and memory: Meta-analytic reviews of studies on sleep deprivation before and after learning.

Research suggests that sleep deprivation both before and after encoding has a detrimental effect on memory for newly learned material. However, there is as yet no quantitative analyses of the size of these effects. We conducted two meta-analyses of studies published between 1970 and 2020 that investigated effects of total, acute sleep deprivation on memory (i.e., at least one full night of sleep deprivation): one for deprivation occurring before learning and one for deprivation occurring after learning. The impact of sleep deprivation after learning on memory was associated with Hedges’ g = 0.277, 95% CI [0.177, 0.377]. Whether testing took place immediately after deprivation or after recovery sleep moderated the effect, with significantly larger effects observed in immediate tests. Procedural memory tasks also showed significantly larger effects than declarative memory tasks. The impact of sleep deprivation before learning was associated with Hedges’ g = 0.621, 95% CI [0.473, 0.769]. Egger’s tests for funnel plot asymmetry suggested significant publication bias in both meta-analyses. Statistical power was very low in most of the analyzed studies. Highly powered, preregistered replications are needed to estimate the underlying effect sizes more precisely.

Effect-Size Estimation Using Semiparametric Hierarchical Mixture Models in Disease-Association Studies with Neuroimaging Data.

In disease-association studies using neuroimaging data, evaluating the biological or clinical significance of individual associations requires not only detection of disease-associated areas of the brain but also estimation of the magnitudes of the associations or effect sizes for individual brain areas. In this paper, we propose a model-based framework for voxel-based inferences under spatial dependency in neuroimaging data. Specifically, we employ hierarchical mixture models with a hidden Markov random field structure to incorporate the spatial dependency between voxels. A nonparametric specification is proposed for the effect size distribution to flexibly estimate the underlying effect size distribution. Simulation experiments demonstrate that compared with a naive estimation method, the proposed methods can substantially reduce the selection bias in the effect size estimates of the selected voxels with the greatest observed associations. An application to neuroimaging data from an Alzheimer's disease study is provided.

Could the ambient higher temperature decrease the transmissibility of COVID-19 in China?

BackgroundExisting literatures demonstrated that meteorological factors could be of importance in affecting the spread patterns of the respiratory infectious diseases. However, how ambient temperature may influence the transmissibility of COVID-19 remains unclear. ObjectivesWe explore the association between ambient temperature and transmissibility of COVID-19 in different regions across China. MethodsThe surveillance data on COVID-19 and meteorological factors were collected from 28 provincial level regions in China, and estimated the instantaneous reproductive number (Rt). The generalized additive model was used to assess the relationship between mean temperature and Rt. ResultsThere were 12,745 COVID-19 cases collected in the study areas. We report the associated effect of temperature on Rt is likely to be negative but not of statistical significance, which holds for most Chinese regions. ConclusionsWe found little statistical evidence for that the higher temperature may reduce the transmissibility of COVID-19. Since intensive control measures against the COVID-19 epidemics were implemented in China, we acknowledge this may impact the underlying effect size estimation, and thus cautiousness should be taken when interpreting our findings.

The orchard plot: Cultivating a forest plot for use in ecology, evolution, and beyond.

"Classic" forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution, meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a "forest-like plot," showing point estimates (with 95% confidence intervals [CIs]) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the "orchard plot." Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also include 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

The use and abuse of meta-analysis.

The use and abuse of meta-analysis.

Science is not a signal detection problem

The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, because continuously distributed effects selected using a P < 0.05 filter must be inflated, the fact that they are smaller when replicated (reflecting regression to the mean) is no reason to sound the alarm. Considered from this perspective, recent replication efforts suggest that most published P < 0.05 scientific findings are “true” (i.e., in the correct direction), with observed effect sizes that are inflated to varying degrees. We propose that original science is a screening process, one that adopts NHST logic as a useful fiction for selecting true effects that are potentially large enough to be of interest to other scientists. Unlike original science, replication science seeks to precisely measure the underlying effect size associated with an experimental protocol via large-N direct replication, without regard for statistical significance. Registered reports are well suited to (often resource-intensive) direct replications, which should focus on influential findings and be published regardless of outcome. Conceptual replications play an important but separate role in validating theories. However, because they are part of NHST-based original science, conceptual replications cannot serve as the field’s self-correction mechanism. Only direct replications can do that.

Genome-wide Genetic Contribution to Diet Response in an Epidemiological Dataset (P15-018-19)

ObjectivesMany gene-diet interactions have been uncovered for obesity and other cardiometabolic risk factors, but truly personalized nutritional recommendations will require the incorporation of an individual’s full genome in predicting response to diet. Statistical genetics studies typically require thousands of individuals, limiting the ability of dietary intervention trials to answer these genome-wide nutrigenetic questions. We sought to explore a novel approach for identifying the genetic architecture of the diet-body mass index (BMI) relationship using an epidemiological dataset. MethodsAs a mathematical correlation is defined as the expected product of two standardized variables, it may be possible to estimate the genetic signal describing an underlying diet-BMI correlation by predicting their product. Statistical simulations were performed to assess the ability of this method to pick up pre-specified effects of genotype on diet response. In white women from the longitudinal Women’s Health Initiative (WHI) dataset, the product of log-transformed fat-to-carbohydrate ratio (F: C) and body mass index (BMI) (both variables standardized) was calculated both cross-sectionally at baseline (n = 9357) and with respect to longitudinal changes in these variables before follow-up (n = 1333). Plink and GCTA tools were used to estimate the genotype-based heritability of these products, as well as that of the change in BMI in response to a separate intervention in WHI focused partially on fat reduction. ResultsSimulations demonstrated that the method is sensitive to changes in the underlying effect sizes, but is able to detect underlying statistical correlations as intended. Genetic heritability estimates using cross-sectional data were negligible, while those using longitudinal data approached statistical significance (variance explained = 14%, P = 0.07). BMI changes in the dietary modification trial showed non-significant heritability (v.e. = 4%), which was insufficient to validate any genetic correlation with the longitudinal results. ConclusionsWhile cross-sectional data may contain too much noise, this method shows promise for the detection of genome-wide contributions to diet response in longitudinal data, and should be investigated further in larger datasets and with alternative phenotypes. Funding SourcesThis study was supported by the NHLBI T32 training grant.

Evaluation of the cardioprotective potential of extracellular vesicles \u2013 a systematic review and meta-analysis

Cardiovascular diseases are the main cause of death worldwide, demanding new treatments and interventions. Recently, extracellular vesicles (EVs) came in focus as important carriers of protective molecules such as miRNAs and proteins which might contribute to e.g. improved cardiac function after myocardial infarction. EVs can be secreted from almost every cell type in the human body and can be transferred via the bloodstream in almost every compartment. To provide an all-encompassing overview of studies investigating these beneficial properties of EVs we performed a systematic review/meta-analysis of studies investigating the cardioprotective characteristics of EVs. Forty-three studies were investigated and catalogued according to the EV source. We provide an in-depth analysis of the purification method, size of the EVs, the conducted experiments to investigate the beneficial properties of EVs as well as the major effector molecule encapsulated in EVs mediating protection. This study provides evidence that EVs from different cell types and body fluids provide cardioprotection in different in vivo and in vitro studies. A meta-analysis was performed to estimate the underlying effect size. In conclusion, we demonstrated that EVs from different sources might serve as a promising tool for treating cardiovascular diseases in the future.

Do we really need confidence intervals in the new statistics?

ABSTRACTThis paper compares the use of confidence intervals (CIs) and a sensitivity analysis called the number needed to disturb (NNTD), in the analysis of research findings expressed as ‘effect’ sizes. Using 1,000 simulations of randomised trials with up to 1,000 cases in each, the paper shows that both approaches are very similar in outcomes, and each one is highly predictable from the other. CIs are supposed to be a measure of likelihood or uncertainty in the results, showing a range of possible effect sizes that could have been produced by random sampling variation alone. NNTD is supposed to be a measure of the robustness of the effect size to any variation, including that produced by missing data. Given that they are largely equivalent and interchangeable under the conditions tested here, the paper suggests that both are really measures of robustness. It concludes that NNTD is to be preferred because it requires many fewer assumptions, is more tolerant of missing data, is easier to explain, and directly addresses the key question of whether the underlying effect size is zero or not.

Complainant emotional expressions and perceived credibility: Exploring the role of perceivers’ facial mimicry and empathy

PurposeThis research investigated the roles of perceivers’ facial mimicry and empathy in the emotional victim effect (EVE) – the finding that complainants tend to appear more credible when exhibiting (vs. not exhibiting) negative emotional displays during their statements. Because facial mimicry plays a key role in empathic responding, it was hypothesized that inhibiting and facilitating perceivers’ mimicry would attenuate and amplify the EVE, respectively.MethodsParticipants (N = 362) in an experiment were instructed to mimic or not to mimic facial expressions (controls received no mimicry instructions) while watching a statement by an emotional or non‐emotional rape complainant. Participants rated the complainant's believability and the extent to which they experienced cognitive and affective empathy.ResultsThe perceived believability of the complainant was not affected by complainant emotions, thus failing to replicate the EVE. However, the inhibition of mimicry unexpectedly reduced the perceived believability of the complainant, apparently by decreasing participants’ cognitive empathy.ConclusionsThe current findings suggest that mimicry inhibition may increase scepticism in the context of credibility assessment. This has important implications for decision‐making in legal settings and for research on the process of credibility attribution. Moreover, the failure to replicate the EVE adds to the cumulative evidence on the underlying effect size for the phenomenon.

Accounting for measurement error in human life history trade-offs using structural equation modeling.

Revealing causal effects from correlative data is very challenging and a contemporary problem in human life history research owing to the lack of experimental approach. Problems with causal inference arising from measurement error in independent variables, whether related either to inaccurate measurement technique or validity of measurements, seem not well-known in this field. The aim of this study is to show how structural equation modeling (SEM) with latent variables can be applied to account for measurement error in independent variables when the researcher has recorded several indicators of a hypothesized latent construct. As a simple example of this approach, measurement error in lifetime allocation of resources to reproduction in Finnish preindustrial women is modelled in the context of the survival cost of reproduction. In humans, lifetime energetic resources allocated in reproduction are almost impossible to quantify with precision and, thus, typically used measures of lifetime reproductive effort (e.g., lifetime reproductive success and parity) are likely to be plagued by measurement error. These results are contrasted with those obtained from a traditional regression approach where the single best proxy of lifetime reproductive effort available in the data is used for inference. As expected, the inability to account for measurement error in women's lifetime reproductive effort resulted in the underestimation of its underlying effect size on post-reproductive survival. This article emphasizes the advantages that the SEM framework can provide in handling measurement error via multiple-indicator latent variables in human life history studies.

Meta-analysis of GWAS of over 16,000 individuals with autism spectrum disorder highlights a novel locus at 10q24.32 and a significant overlap with schizophrenia

BackgroundOver the past decade genome-wide association studies (GWAS) have been applied to aid in the understanding of the biology of traits. The success of this approach is governed by the underlying effect sizes carried by the true risk variants and the corresponding statistical power to observe such effects given the study design and sample size under investigation. Previous ASD GWAS have identified genome-wide significant (GWS) risk loci; however, these studies were of only of low statistical power to identify GWS loci at the lower effect sizes (odds ratio (OR) <1.15).MethodsWe conducted a large-scale coordinated international collaboration to combine independent genotyping data to improve the statistical power and aid in robust discovery of GWS loci. This study uses genome-wide genotyping data from a discovery sample (7387 ASD cases and 8567 controls) followed by meta-analysis of summary statistics from two replication sets (7783 ASD cases and 11359 controls; and 1369 ASD cases and 137308 controls).ResultsWe observe a GWS locus at 10q24.32 that overlaps several genes including PITX3, which encodes a transcription factor identified as playing a role in neuronal differentiation and CUEDC2 previously reported to be associated with social skills in an independent population cohort. We also observe overlap with regions previously implicated in schizophrenia which was further supported by a strong genetic correlation between these disorders (Rg = 0.23; P = 9 × 10−6). We further combined these Psychiatric Genomics Consortium (PGC) ASD GWAS data with the recent PGC schizophrenia GWAS to identify additional regions which may be important in a common neurodevelopmental phenotype and identified 12 novel GWS loci. These include loci previously implicated in ASD such as FOXP1 at 3p13, ATP2B2 at 3p25.3, and a ‘neurodevelopmental hub’ on chromosome 8p11.23.ConclusionsThis study is an important step in the ongoing endeavour to identify the loci which underpin the common variant signal in ASD. In addition to novel GWS loci, we have identified a significant genetic correlation with schizophrenia and association of ASD with several neurodevelopmental-related genes such as EXT1, ASTN2, MACROD2, and HDAC4.

Multi-subgroup gene screening using semi-parametric hierarchical mixture models and the optimal discovery procedure: Application to a randomized clinical trial in multiple myeloma.

This article proposes an efficient approach to screening genes associated with a phenotypic variable of interest in genomic studies with subgroups. In order to capture and detect various association profiles across subgroups, we flexibly estimate the underlying effect size distribution across subgroups using a semi-parametric hierarchical mixture model for subgroup-specific summary statistics from independent subgroups. We then perform gene ranking and selection using an optimal discovery procedure based on the fitted model with control of false discovery rate. Efficiency of the proposed approach, compared with that based on standard regression models with covariates representing subgroups, is demonstrated through application to a randomized clinical trial with microarray gene expression data in multiple myeloma, and through a simulation experiment.

The albatross plot: A novel graphical tool for presenting results of diversely reported studies in a systematic review.

Meta‐analyses combine the results of multiple studies of a common question. Approaches based on effect size estimates from each study are generally regarded as the most informative. However, these methods can only be used if comparable effect sizes can be computed from each study, and this may not be the case due to variation in how the studies were done or limitations in how their results were reported. Other methods, such as vote counting, are then used to summarize the results of these studies, but most of these methods are limited in that they do not provide any indication of the magnitude of effect.We propose a novel plot, the albatross plot, which requires only a 1‐sided P value and a total sample size from each study (or equivalently a 2‐sided P value, direction of effect and total sample size). The plot allows an approximate examination of underlying effect sizes and the potential to identify sources of heterogeneity across studies. This is achieved by drawing contours showing the range of effect sizes that might lead to each P value for given sample sizes, under simple study designs. We provide examples of albatross plots using data from previous meta‐analyses, allowing for comparison of results, and an example from when a meta‐analysis was not possible.

Bayesian evaluation of effect size after replicating an original study.

The vast majority of published results in the literature is statistically significant, which raises concerns about their reliability. The Reproducibility Project Psychology (RPP) and Experimental Economics Replication Project (EE-RP) both replicated a large number of published studies in psychology and economics. The original study and replication were statistically significant in 36.1% in RPP and 68.8% in EE-RP suggesting many null effects among the replicated studies. However, evidence in favor of the null hypothesis cannot be examined with null hypothesis significance testing. We developed a Bayesian meta-analysis method called snapshot hybrid that is easy to use and understand and quantifies the amount of evidence in favor of a zero, small, medium and large effect. The method computes posterior model probabilities for a zero, small, medium, and large effect and adjusts for publication bias by taking into account that the original study is statistically significant. We first analytically approximate the methods performance, and demonstrate the necessity to control for the original study’s significance to enable the accumulation of evidence for a true zero effect. Then we applied the method to the data of RPP and EE-RP, showing that the underlying effect sizes of the included studies in EE-RP are generally larger than in RPP, but that the sample sizes of especially the included studies in RPP are often too small to draw definite conclusions about the true effect size. We also illustrate how snapshot hybrid can be used to determine the required sample size of the replication akin to power analysis in null hypothesis significance testing and present an easy to use web application (https://rvanaert.shinyapps.io/snapshot/) and R code for applying the method.