Beware study design measurement errors.

Abstract

I concur with Leek and Peng (“What is the question?” Perspectives, 20 March, p. [1314][1]) that careful alignment of statistical methodology to research questions will reduce the most common errors in data analysis. The data analysis flowchart they present is an elegant decision tree that should guide not only research scientists but also peer reviewers and journal editors. However, minimizing errors in data analysis is a necessary criterion but not sufficient. Research results can be reproducible yet incorrect. As scientists, peer reviewers, and journal editors, we all need to be fully cognizant of the basic tenets of “measurement theory”: precision, accuracy, reliability, and validity of data. This means that we also need to be cognizant of bias, which can be minimized most effectively during the study design phase. Measurement errors can constitute both random (variance) and nonrandom (bias) components. Increasing sample size in a study will reduce variance but will have no effect on bias, which can be much more insidious in research undertakings. Warren S. Sarle of the SAS Institute made an important distinction: “Mathematical statistics is concerned with the connection between inference and data. Measurement theory is concerned with the connection between data and reality. Both statistical theory and measurement theory are necessary to make inferences about reality” ([ 1 ][2]). An excellent framework for anticipating and addressing bias in experimental and observational studies is presented by David F. Ransohoff ([ 2 ][3]). I fully concur with Leek and Peng that “data analytic education” should be a key component of research training, but would like to additionally emphasize that fundamentals of study designs and measurement theory should also be covered in such research training. 1. [↵][4]1. W. S. Sarle , Measurement theory: Frequently asked questions ( ). 2. [↵][5]1. D. F. Ransohoff , Nat. Rev. Cancer 5, 142 (2005). [OpenUrl][6][CrossRef][7][PubMed][8][Web of Science][9] [1]: pending:yes [2]: #ref-1 [3]: #ref-2 [4]: #xref-ref-1-1 View reference 1 in text [5]: #xref-ref-2-1 View reference 2 in text [6]: {openurl}?query=rft.jtitle%253DNature%2Breviews.%2BCancer%26rft.stitle%253DNat%2BRev%2BCancer%26rft.aulast%253DRansohoff%26rft.auinit1%253DD.%2BF.%26rft.volume%253D5%26rft.issue%253D2%26rft.spage%253D142%26rft.epage%253D149%26rft.atitle%253DBias%2Bas%2Ba%2Bthreat%2Bto%2Bthe%2Bvalidity%2Bof%2Bcancer%2Bmolecular-marker%2Bresearch.%26rft_id%253Dinfo%253Adoi%252F10.1038%252Fnrc1550%26rft_id%253Dinfo%253Apmid%252F15685197%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [7]: /lookup/external-ref?access_num=10.1038/nrc1550&link_type=DOI [8]: /lookup/external-ref?access_num=15685197&link_type=MED&atom=%2Fsci%2F348%2F6234%2F512.atom [9]: /lookup/external-ref?access_num=000226721000015&link_type=ISI