Bias in the evaluation of research methods

Abstract

There is a substantial body of evidence that judgement and decision-making are partly controlled by unconscious mechanisms which take the form of preferred forms of order, heuristic processes and mental accounting. These automatic mechanisms are likely to be involved in any judgements, including those made about theories and methods in science. Examples are presented where bias has affected the acceptance of theories and methods, and the mechanisms that may create such bias are described. Limitations in the design of controlled experiments in consumer behaviour are reviewed; these designs restrict generalization, and therefore the value of such work in advancing understanding. Despite this, such controlled experiments are highly regarded in consumer behaviour and social psychology; one reason for this high regard may be the effect of automatic mechanisms in biasing judgement about controlled experiments. By contrast, inferences from surveys seem less susceptible to such bias. If controlled experimental findings are overvalued in comparison to other methods, the direction of research will suffer and scientific advance will be impeded.