Experimental design heuristics for scientific discovery: the use of “baseline” and “known standard” controls

Abstract

What type of heuristics do scientists use when they design experiments? In this paper, we analysed the ways biological scientists designed complex experiments at their weekly laboratory meetings. In two studies, we found that one of the key components of experimental design is that specific types of control conditions are used in the service of different goals that are important in scientific discovery. “Baseline” control conditions are identical to the experimental manipulation, except that a key feature, such as a reagent, is absent from the control condition and present in the experimental condition. “Known standard” control conditions involve performing the experimental technique on materials where the expected result is already well known; if the expected result is obtained, the scientist can have confidence that the procedure is working. In Study 1, which analysed transcripts of real-world biology laboratory meetings, we found that scientists used baseline controls when testing hypotheses and known standard controls when focusing on possible error. In Study 2, undergraduate science students were asked to address the goals of hypothesis testing and dealing with potential error as they designed experiments. Like the real-world scientists, science majors proposed baseline controls to test hypotheses and known standard controls to deal with potential error. We argue that baseline control conditions play an important role in hypothesis testing: unexpected results obtained on baseline control conditions can alert scientists that their hypotheses are incorrect, and hence should encourage the scientists to reformulate their hypotheses. We further argue that use of known standard controls is a heuristic that enables scientists to solve an important problem in real-world science: when to trust their data. Both of these heuristics can be incorporated into experimental design programs, thus making it more likely that scientific discoveries will be made.