Last word on viewpoint: justifying small-n research in scientifically amazing settings: challenging the notion that only "big-n" studies are worthwhile.

Abstract

ViewpointLast Word on Viewpoint: Justifying small-n research in scientifically amazing settings: Challenging the notion that only “big-n” studies are worthwhileRobert J. Ploutz-Snyder, James Fiedler, and Alan H. FeivesonRobert J. Ploutz-SnyderUniversities Space Research Association, NASA Johnson Space Center, Houston, Texas, James FiedlerUniversities Space Research Association, NASA Johnson Space Center, Houston, Texas, and Alan H. FeivesonUniversities Space Research Association, NASA Johnson Space Center, Houston, TexasPublished Online:01 May 2014https://doi.org/10.1152/japplphysiol.00214.2014MoreSectionsPDF (26 KB)Download PDF ToolsExport citationAdd to favoritesGet permissionsTrack citations ShareShare onFacebookTwitterLinkedInEmailWeChat to the editor: We thank Drs. Bacchetti and Shelhamer (see Ref. 1) for their comments, and we are glad to see they share our opinion that traditional methods and defaults for justifying sample size are overused and underscrutinized (2). We have become increasingly convinced of this problem in our work at NASA, but as Bacchetti suggests, all researchers, regardless of discipline or setting, should abandon the habit of blindly using traditional methods to determine sample size and should instead carefully consider, for each study, the benefits and drawbacks for varying sample size.Dr. Bacchetti urges us to consider feasibility and cost considerations in sample size determination, along with other factors, and of course we agree. Unfortunately, in our work at NASA, we have not yet been successful in justifying sample size based on feasibility and costs or other less tangible metrics. For example, when proposing research on the International Space Station, it is incredibly difficult to estimate the cost for subject participation in a study. Should we include the cost to train and launch astronauts? Or just the cost that it takes for them to participate in a given experiment? We think the latter, but even estimating the cost for an astronaut to participate in a given study (for example, cost per minute of participation) is incredibly difficult. So although we agree with Dr. Bacchetti that cost and feasibility should be considered, we find it difficult to rely solely on those factors for sample size justification sections in grant proposals. We suspect that other researchers may have similar problems, because in addition to the difficulty in estimating true costs and feasibility numbers, most grant reviewers expect to see traditional methods for determining sample size and in our experience will tend to reject methods that are too unfamiliar. In addition, constraints (real or PI perceived) on the amount of text that should be devoted to sample size considerations sometimes restrict the novelty or depth of consideration for sample size justification.Ultimately, we believe that the sample size justification challenge should be a multipronged and thoughtful process, not something that is performed as a perfunctory requirement for grant reviewers using canned procedures or assumptions. Most PIs and many funding agencies fail to commit the page space to this topic, and, for better or worse, the reviewer community has the expectation that a traditional “power analysis” is the holy grail for sample size justification. In response to this expectation, we think that augmenting current sample size determination methods with newer methods and ideas, particularly those that more fully appreciate what can be gained by collecting even small amounts of data in highly novel areas of inquiry, is the right answer. We have provided a few options here, and, in the short term, we hope that this is useful to those who are considering writing or reviewing grants. In the long term, we hope to generate thoughtful discourse among researchers and statisticians alike on how to improve our understanding of what is gained or lost by varying sample size.DISCLOSURESNo conflicts of interest, financial or otherwise, are declared by the author(s).AUTHOR CONTRIBUTIONSAuthor contributions: R.J.P.-S. drafted manuscript; R.J.P.-S. edited and revised manuscript; R.J.P.-S. approved final version of manuscript.REFERENCES1. Bacchetti P, Shelhamer MJ. Commentaries on Viewpoint: Justifying small-n research in scientifically amazing settings: Challenging the notion that only “big-n” studies are worthwhile. J Appl Physiol; doi:10.1152/japplphysiol.00200.2014.Link | ISI | Google Scholar2. Ploutz-Snyder RJ, Fiedler J, Feiveson AH. Viewpoint: Justifying small-n research in scientifically amazing settings: Challenging the notion that only “big-n” studies are worthwhile. J Appl Physiol; doi:10.1152/japplphysiol.01335.2013.Link | ISI | Google ScholarAUTHOR NOTESAddress for reprint requests and other correspondence: R. J. Ploutz-Snyder, Universities Space Research Association, NASA Johnson Space Center, 2101 NASA Parkway, Mail Code SK3, Houston, TX 77058 (e-mail: Robert.[email protected]gov). Download PDF Previous Back to Top Next FiguresReferencesRelatedInformation More from this issue > Volume 116Issue 9May 2014Pages 1254-1254 Copyright & PermissionsCopyright © 2014 the American Physiological Societyhttps://doi.org/10.1152/japplphysiol.00214.2014PubMed24789885History Published online 1 May 2014 Published in print 1 May 2014 Metrics