Optimal design in psychological research.

Abstract

Psychologists often do not consider the optimality of their research designs. However, increasing costs of using inefficient designs requires psychologists to adopt more efficient designs and to use more powerful analysis strategies. Common designs with many factor levels and equal allocations of observations are often inefficient for the specific questions most psychologists want to answer. Happenstance allocations determined by random sampling are usually even more inefficient and some common analysis strategies can exacerbate the inefficiency. By selecting treatment levels and allocating observations optimally, psychologists can greatly increase the efficiency and statistical power of their research designs. A few heuristic design principles can produce much more efficient designs than are often used. Experimental researchers outside psychology often carefully consider the efficiency of their research designs. For example, the high costs of conducting large-scale experiments with industrial processes has motivated the search for designs that are optimally efficient. As a consequence, a substantial literature on optimal design has developed outside psychology. In contrast, psychologists have not been as constrained by costs, so their research designs have been based on tradition and computational ease. Most psychologists are unaware of the literature on optimal research design; this topic receives little or no attention in popular textbooks on methods and statistics in psychology. Experimental design textbooks offer little if any advice on how many levels of the independent variables to use or on how to allocate observations across those levels to obtain optimal efficiency. When advice is offered, it is usually based on heuristics derived from experience rather than statistical principles. The purpose of this article is to review the basic concepts of optimal design and to illustrate how a few