When limited availability of data meets with a thorough meta-analysis: a comment on Moore et al.

Abstract

On the basis of a systematic literature survey, Moore et al. (2016) have reviewed correlative evidence for the relationship between the expression of sexual signals and stress response. They have identified 38 published studies on 26 species reporting 118 effect sizes for the focal relationship that were entered in a phylogenetic metaanalysis. The analysis failed to derive a general support for the hypothesis that males (or females) with more elaborate traits would experience different stress regimes at the metabolic level than conspecifics with smaller traits. Although Moore et al. (2016) elegantly applied a modern statistical approach, I suggest that the inferences that can be made from their study are limited due to the scarcity of data that are currently available to test the underlying hypothesis. Below, I point to 3 issues to consider for the interpretations of the meta-analytic results. First, the available sample sizes correspond to a statistical power that allow identifying significant associations for very strong effects only. Although Moore et al. (2016) relied on more than 100 effect sizes, these were largely nonindependent. The variation in the predictor variables typically resided on the betweenstudy or between-species levels; thus, the degrees of freedom that were associated with most of the meta-regression tests were determined by the number of studies/species (and not by the number of effect sizes). As a result, most of the effect size estimates were derived with a considerably broad confidence interval (see Table 2 in Moore et al. 2016), which incurs the risk that a biologically meaningful association with smaller effect size remains undetected in a null hypothesis–testing framework. However, for biological reasons, it is expected that most of the relationships would fall into weak to intermediate effect size ranges (sensu Cohen 1988). This is because the focal traits (both physiological measure of stress and sexual traits) are measured with error, and the causal relationship between them, if one exists, is indirect (i.e., mediated by immune suppression or body condition) and involves other physiological/behavioral components (i.e., testosterone levels, escape behaviors). These confounding factors likely set up an upper limit for the strength of the correlations, well below the magnitude of large effect sizes that can be detected between the studied traits. The underlying dataset of Moore et al. (2016) does not have an appropriate statistical power to cope with the biologically expected range of effect sizes. Second, the relationship between the focal traits is complex, may involve nonlinear associations, and be mediated by different mechanistic routes that are difficult to handle statistically given the available sample sizes (and degrees of freedom). For example, as the authors also mention, completely different rules might be applied for the relationship between call rate and glucocorticoid levels in a toad species than for the relationship between repertoire size and levels of heat-shock proteins in a bird species. Furthermore, it would be desirable to distinguish between mechanisms due to baseline stress levels and stress response and to consider confounding effects that are mediated by the ecology of species (Moore et al. 2016). To investigate such complex associations, statistical models that include higher level of interactions between the fixed effects and consider additional potentially confounding variables would be needed. Currently, the limited sample size precludes modeling complex biological mechanisms. Third, at low sample sizes, the problems imposed by influential data points are more emphasized. A careful inspection of the single significant result that has been found in the study (the difference in mean effect size across different types of sexually selected traits) was caused by a single species. This species is the human, which has a distinguished phylogenetic position and the only species that falls in the “opposite sex preference” category. It is therefore likely that the exclusion of this species/category would change the results, and such sensitivity would limit considerably what can be generalized from the current data. In summary, Moore et al. (2016) investigate a timely question in behavioral ecology by summarizing currently available evidence in a thorough meta-analysis. However, the data that have been accumulated on the topic to date are so limited that they undermine any conclusions that can be made from their systematic review. When more data become available, the approach of Moore et al. (2016) will be a straightforward way to follow.