The Influence of Prior Beliefs on Scientific Judgments of Evidence Quality

Abstract

This paper is concerned with the influence of scientists′ prior beliefs on their judgments of evidence quality. A laboratory experiment using advanced graduate students in the sciences (study 1) and an experimental survey of practicing scientists on opposite sides of a controversial issue (study 2) revealed agreement effects. Research reports that agreed with scientists′ prior beliefs were judged to be of higher quality than those that disagreed. In study 1, a prior belief strength × agreement interaction was found, indicating that the agreement effect was larger among scientists who held strong prior beliefs. In both studies, the agreement effect was larger for general, evaluative judgments (e.g., relevance, methodological quality, results clarity) than for more specific, analytical judgments (e.g., adequacy of randomization procedures). A Bayesian analysis indicates that the pattern of agreement effects found in these studies may be normatively defensible, although arguments against implementing a Bayesian approach to scientific judgment are also advanced.