Abstract
Based on findings from six experiments, Dallas, Liu, and Ubel (2019) conclude that placing calorie labels to the left of menu items influences consumers to choose lower calorie food options. Contrary to previously reported findings, they suggest that calorie labels can influence food choices, but only when placed to the left because they are in this case read first. If true, these findings have important implications for the design of menus and may help address the obesity pandemic. However, an analysis of the reported results indicates that they seem too good to be true. We show that if the effect sizes in Dallas et al. (2019) are representative of the populations, a replication of the six studies (with the same sample sizes) has a probability of only 0.014 of producing uniformly significant outcomes. Such a low success rate suggests that the original findings might be the result of questionable research practices or publication bias. We therefore caution readers and policy makers to be skeptical about the results and conclusions reported by Dallas et al. (2019).
Highlights
Contrary to previously reported findings, they suggest that calorie labels can influence food choices, but only when placed to the left because they are in this case read first
When conducting six independent studies, each with a power of 0.5, one should expect only about half of the studies to produce significant results. It would be very rare for all six studies to produce significant results, namely 0.56 ≈ 0.016. When such excess success is observed in a publication, readers should suspect that the experiments were carried out using questionable research practices (John et al, 2012; Simmons et al, 2011) or that some experiments with nonsignificant results were run but not reported
While there are other methods that aim to detect publication bias or questionable research practices, the Test for Excess Success (TES) analysis is currently the only approach that deals with multiple hypothesis tests from a single sample; something that is relevant for the findings reported in Dallas et al (2019)
Summary
We show that if the effect sizes in Dallas et al (2019) are representative of the populations, a replication of the six studies (with the same sample sizes) has a probability of only 0.014 of producing uniformly significant outcomes Such a low success rate suggests that the original findings might be the result of questionable research practices or publication bias. While there are other methods (see for example Renkewitz & Keiner, 2019) that aim to detect publication bias or questionable research practices, the TES analysis is currently the only approach that deals with multiple hypothesis tests from a single sample; something that is relevant for the findings reported in Dallas et al (2019).
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
