How hard is hard science, how soft is soft science? The empirical cumulativeness of research.

Abstract

Research results in the social and behavioral sciences are often conceded to be less replicable than research results in the physical sciences. However, direct empirical comparisons of the cumulativeness of research in the social and physical sciences have not been made to date. This article notes the parallels between methods used in the quantitative synthesis of research in the social and in the physical sciences. Essentially identical methods are used to test the consistency of research results in physics and in psychology. These methods can be used to compare the consistency of replicated research results in physics and in the social sciences. The methodology is illustrated with 13 exemplary reviews from each domain. The exemplary comparison suggests that the results of physical experiments may not be strikingly more consistent than those of social or behavioral experiments. The data suggest that even the results of physical experiments may not be cumulative in the absolute sense by statistical criteria. It is argued that the study of the actual cumulativeness found in physical data could inform social scientists about what to expect from replicated experiments under good conditions. Psychologists and other social scientists have often compared their fields to the natural (the hard) sciences with a tinge of dismay. Those of us in the social and behavioral sciences know intuitively that there is something softer and less cumulative about our research results than about those of the physical sciences. It is easy to chronicle the differences between soft and hard sciences that might lead to less cumulative research results in the soft sciences. One such chronicle is provided by Meehl (1978), who listed 20 such differences and went on to argue that reliance on tests of statistical significance also contributes to the poorer cumulativeness of research results in the social sciences. Other distinguished researchers have cited the pervasive presence of interactions (Cronbach, 1975) or historical influences (Gergen, 1973, 1982) as reasons not to expect a cumulative social science. Still others (Kruskal, 1978, 1981) have cited the low quality of data in the social sciences as a barrier to truly cumulative social inquiry. These pessimistic views have been accompanied by a tendency to reconceptualize the philosophy of inquiry into a format that implies less ambitious aspirations for social knowledge (e.g., Cronbach, 1975; Gergen, 1982). Cumulativeness in the scientific enterprise can mean at least two things. In the broadest sense scientific results are cumulative if empirical laws and theoretical structures build on one another so that later developments extend and unify earlier work. This idea might be called conceptual or theoretical cumulativeness. The assessment of theoretical cumulativeness must be rather subjective. A narrower and less subjective indicator of cumulativeness is the degree of agreement among replicated experiments or the degree to which related experimental results fit into a simple pattern that makes conceptual sense. This idea might be called empirical cumulativeness. The purpose of this article is to suggest that it may be possible to compare at least the empirical cumulativeness of psychological research with that of research in the physical sciences. An exemplary comparison suggests that the differences may be less striking than previously imagined. The mechanism for this comparison is derived from recent developments in methods for the quantitative synthesis of research in the social sciences. Some of the methods used in meta-analysis are analogous to methods used in the quantitative synthesis of research in the physical sciences. In particular, physicists and psychologists use analogous methods for assessing the consistency of research results, a fact that makes possible comparisons among quantitative reviews in physics and in psychology. One such comparison is reported in this article. This comparison was not chosen in a way that guarantees it to be representative of either social science research or physical science research. However, some effort was exerted to prevent the comparison from obviously favoring one domain or the other, and additional examples are provided to suggest that the case for the empirical cumulativeness of physical science could have been made to look far worse. More data would obviously be needed to support strong conclusions. It seems, however, that the obvious conclusion that the results of physical science experiments are more cumulative than those of social science experiments does not have much empirical sup-