Meta-analyses In Ecology Research Articles

Computationally reproducing results from meta-analyses in ecology and evolutionary biology using shared code and data.

Many journals in ecology and evolutionary biology encourage or require authors to make their data and code available alongside articles. In this study we investigated how often this data and code could be used together, when both were available, to computationally reproduce results published in articles. We surveyed the data and code sharing practices of 177 meta-analyses published in ecology and evolutionary biology journals published between 2015-17: 60% of articles shared data only, 1% shared code only, and 15% shared both data and code. In each of the articles which had shared both (n = 26), we selected a target result and attempted to reproduce it. Using the shared data and code files, we successfully reproduced the targeted results in 27-73% of the 26 articles, depending on the stringency of the criteria applied for a successful reproduction. The results from this sample of meta-analyses in the 2015-17 literature can provide a benchmark for future meta-research studies gauging the computational reproducibility of published research in ecology and evolutionary biology.

Decline effects are rare in ecology.

The scientific evidence base on any given topic changes over time as more studies are published. Currently, there is widespread concern about nonrandom, directional changes over time in the scientific evidence base associated with many topics. In particular, if studies finding large effects (e.g., large differences between treatment and control means) tend to get published quickly, while small effects tend to get published slowly, the net result will be a decrease over time in the estimated magnitude of the mean effect size, known as a "decline effect." If decline effects are common, then the published scientific literature will provide a biased and misleading guide to management decisions, and to the allocation of future research effort. We compiled data from 466 meta-analyses in ecology to look for evidence of decline effects. We found that decline effects are rare. Only ~5% of ecological meta-analyses truly exhibit a directional change in mean effect size over time arising for some reason other than random chance, usually but not always in the direction of decline. Most apparent directional changes in mean effect size over time are attributable to regression to the mean, consistent with primary studies being published in random order with respect to the effect sizes they report. Our results are good news: decline effects are the exception to the rule in ecology. Identifying and rectifying rare cases of true decline effects remains an important task, but ecologists should not overgeneralize from anecdotal reports of decline effects.

Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension.

ABSTRACTSince the early 1990s, ecologists and evolutionary biologists have aggregated primary research using meta‐analytic methods to understand ecological and evolutionary phenomena. Meta‐analyses can resolve long‐standing disputes, dispel spurious claims, and generate new research questions. At their worst, however, meta‐analysis publications are wolves in sheep's clothing: subjective with biased conclusions, hidden under coats of objective authority. Conclusions can be rendered unreliable by inappropriate statistical methods, problems with the methods used to select primary research, or problems within the primary research itself. Because of these risks, meta‐analyses are increasingly conducted as part of systematic reviews, which use structured, transparent, and reproducible methods to collate and summarise evidence. For readers to determine whether the conclusions from a systematic review or meta‐analysis should be trusted – and to be able to build upon the review – authors need to report what they did, why they did it, and what they found. Complete, transparent, and reproducible reporting is measured by ‘reporting quality’. To assess perceptions and standards of reporting quality of systematic reviews and meta‐analyses published in ecology and evolutionary biology, we surveyed 208 researchers with relevant experience (as authors, reviewers, or editors), and conducted detailed evaluations of 102 systematic review and meta‐analysis papers published between 2010 and 2019. Reporting quality was far below optimal and approximately normally distributed. Measured reporting quality was lower than what the community perceived, particularly for the systematic review methods required to measure trustworthiness. The minority of assessed papers that referenced a guideline (~16%) showed substantially higher reporting quality than average, and surveyed researchers showed interest in using a reporting guideline to improve reporting quality. The leading guideline for improving reporting quality of systematic reviews is the Preferred Reporting Items for Systematic reviews and Meta‐Analyses (PRISMA) statement. Here we unveil an extension of PRISMA to serve the meta‐analysis community in ecology and evolutionary biology: PRISMA‐EcoEvo (version 1.0). PRISMA‐EcoEvo is a checklist of 27 main items that, when applicable, should be reported in systematic review and meta‐analysis publications summarising primary research in ecology and evolutionary biology. In this explanation and elaboration document, we provide guidance for authors, reviewers, and editors, with explanations for each item on the checklist, including supplementary examples from published papers. Authors can consult this PRISMA‐EcoEvo guideline both in the planning and writing stages of a systematic review and meta‐analysis, to increase reporting quality of submitted manuscripts. Reviewers and editors can use the checklist to assess reporting quality in the manuscripts they review. Overall, PRISMA‐EcoEvo is a resource for the ecology and evolutionary biology community to facilitate transparent and comprehensively reported systematic reviews and meta‐analyses.

A Beginner's Guide to Conducting Reproducible Research

A Beginner's Guide to Conducting Reproducible Research

Estimation in meta-analyses of response ratios

BackgroundFor outcomes that studies report as the means in the treatment and control groups, some medical applications and nearly half of meta-analyses in ecology express the effect as the ratio of means (RoM), also called the response ratio (RR), analyzed in the logarithmic scale as the log-response-ratio, LRR.MethodsIn random-effects meta-analysis of LRR, with normal and lognormal data, we studied the performance of estimators of the between-study variance, τ2, (measured by bias and coverage) in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect in the log scale, λ. We obtained additional empirical evidence from two examples.ResultsThe results of our extensive simulations showed several challenges in using LRR as an effect measure. Point estimators of τ2 had considerable bias or were unreliable, and interval estimators of τ2 seldom had the intended 95% coverage for small to moderate-sized samples (n<40). Results for estimating λ differed between lognormal and normal data.ConclusionsFor lognormal data, we can recommend only SSW, a weighted average in which a study’s weight is proportional to its effective sample size, (when n≥40) and its companion interval (when n≥10). Normal data posed greater challenges. When the means were far enough from 0 (more than one standard deviation, 4 in our simulations), SSW was practically unbiased, and its companion interval was the only option.

The orchard plot: Cultivating a forest plot for use in ecology, evolution, and beyond.

"Classic" forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution, meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a "forest-like plot," showing point estimates (with 95% confidence intervals [CIs]) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the "orchard plot." Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also include 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Advancing meta-analysis beyond simple parameter estimation.

Advancing meta-analysis beyond simple parameter estimation.

Yes! There are resilient generalizations (or "laws") in ecology.

ABSTRACT It is often argued that ecological communities admit of no useful generalizations or "laws" because these systems are especially prone to contingent historical events. Detractors respond that this argument assumes an overly stringent definition of laws of nature. Under a more relaxed conception, it is argued that ecological laws emerge at the level of communities and elsewhere. A brief review of this debate reveals an issue with deep philosophical roots that is unlikely to be resolved by a better understanding of generalizations in ecology. We therefore propose a strategy for transforming the conceptual question about the nature of ecological laws into a set of empirically tractable hypotheses about the relative re- silience of ecological generalizations across three dimensions: taxonomy, habitat type, and scale. These hypotheses are tested using a survey of 240 meta-analyses in ecology. Our central finding is that generalizations in community ecology are just as prevalent and as resilient as those in population or ecosystem ecology. These findings should help to establish community ecology as a generality-seeking science as opposed to a science of case studies. It also supports the capacity for ecologists, working at any of the three levels, to inform matters of public policy.

Gauging the impact of meta-analysis on ecology

Meta-analyses are an increasingly used set of statistical tools that allow for data from multiple studies to be drawn together allowing broader, more generalizable conclusions. The extent to which the increase in the number of meta-analyses in ecology, relative to other types of papers, has influenced how questions are asked and the current state of knowledge has not been assessed before. Here, we gauge the impact of meta-analyses in ecology quantitatively and qualitatively. For the quantitative assessment, we conducted an analysis of 240 published meta-analyses to examine trends in ecological meta-analyses. Our examination shows that publication rates of meta-analyses in ecology have increased through time, and that more recent meta-analyses have been more comprehensive, including more studies and a greater temporal range of studies. Meta-analyses in ecology are the result of larger collaborations with meta-analyses being authored by larger teams than other studies, and those funded by collaborative centers have even larger collaborations. These larger collaborations result in a larger scope and scale of the analyses—by using more papers, datasets, species and years of data. Qualitatively, we discuss three examples: the strength of competition, the nature of how biodiversity affects ecosystem function, and the response of species to global climate change, where meta-analyses supplied the critical evaluation of accepted ecological explanations. As scientific criticism and controversy mount, the true power of meta-analyses is to serve as the capstone evidence supporting the validity of an explanation and to possibly herald the shift to other potential explanations.

Advancing Synthetic Ecology: A Database System to Facilitate Complex Ecological Meta‐Analyses

Advancing Synthetic Ecology: A Database System to Facilitate Complex Ecological Meta-Analyses

How much variance can be explained by ecologists and evolutionary biologists?

The average amount of variance explained by the main factor of interest in ecological and evolutionary studies is an important quantity because it allows evaluation of the general strength of research findings. It also has important implications for the planning of studies. Theoretically we should be able to explain 100% of the variance in data, but randomness and noise may reduce this amount considerably in biological studies. We performed a meta-analysis using data from 43 published meta-analyses in ecology and evolution with 93 estimates of mean effect size using Pearson's r and 136 estimates using Hedges' d or g. This revealed that (depending on the exact analysis) the mean amount of variance (r 2) explained was 2.51-5.42%. The various 95% confidence intervals fell between 1.99 and 7.05%. There was a strongly positive relationship between the fail-safe number (the number of null results needed to nullify an effect) and the coefficient of determination (r 2) or effect size. Analysis at the level of individual tests of null hypotheses showed that the amount of variance key factors explained differed among fields with the largest amount in physiological ecology, lower amounts in ecology and the lowest in evolutionary studies. In all fields though, the hypothesized relationship (e.g. main effect of a fixed treatment) explained little of the variation in the trait of interest. Our finding has important implications for the interpretation of scientific studies. Across studies, the average effect size reported is between Pearson r=0.180 and 0.193 and Hedges' d=0.631 and 0.721. Thus the average sample sizes needed to conclude that a particular relationship is absent with a power of 80% and α=0.05 (two-tailed) are considerably larger than usually recorded in studies of evolution and ecology. For example, to detect r=0.193, the required sample size is 207.