Real data examples in statistical methods papers: Tremendously valuable, and also tremendously misvalued

Abstract

When a statistical methods paper is submitted to a journal for publication, examples in which the method is applied to real data are highly encouraged by many journals and in some cases are explicitly demanded. In this commentary, we argue that real data examples serve several useful purposes. However, we also argue that in many cases, particularly in the fields of genetics and genomics, there is an implicit or explicit expectation for examples to support purposes for which they are ill-suited and furthermore that these inappropriate expectations have negative consequences for the field. We conclude by noting that real data examples can be tremendously valuable and should continue to be used where appropriate, but that the demands for, expectations of, and conclusions drawn from them need to be scaled back.