Survey design, sampling, and significance testing: Key issues

Abstract

Papers submitted to the journals of academic librarianship sometimes include errors in research design or statistical methods that limit the extent to which the results can be generalized beyond the sample of study participants. This guide discusses how to avoid common problems associated with survey design, sampling, and significance testing (hypothesis testing). The first section, on surveys, covers generalization, random error, investigator bias, response bias, survey design, and alternative methods of gathering data. The second section, on sampling, covers the need for sampling, the importance of defining the population and the cases, sampling strategies, sampling bias (selection bias, volunteer bias, survivorship bias, and nonresponse bias), the need to compare sample and population characteristics, and sample size. The third section, on significance testing, covers the need for significance tests, their inability to account for bias, levels of measurement, types of tests (independent-samples, paired-samples, and one-sample tests), t-tests and nonparametric alternatives, p values and alpha levels, hypothesis testing, and the importance of magnitude or effect size. The paper concludes with guidance on non-statistical methods of generalizing from limited data.