We show that the community of accounting researchers has not fully appreciated the sensitivity of research conclusions to (necessarily) subjective research design choices, and that this failure has led to the subsequent overgeneralization of early evidence. Our analysis is based on a case study that examines the effects of two basic research design choices, and subsequent researchers' appreciation of the impact of those choices. To enhance our study's accessibility to a broad cross-section of the accounting research community, our case study focuses on an article that is covered in most accounting doctoral programs — the recipient of the second American Accounting Association “Seminal Contribution to Accounting Literature Award” — Beaver's 1968 article on the information content of annual earnings announcements [ Beaver, W. (1968). The information content of annual earnings announcements. Empirical research in accounting; selected studies, 1968. Supplement to Journal of Accounting Research, 6, 67–92.] Our analysis reveals that the research design choices in Beaver's study significantly affected the results. Beaver clearly explained these choices, and was careful in drawing inferences. Nevertheless, subsequent researchers interpreted Beaver's evidence too broadly. Our new empirical evidence suggests that the information content inference is much more fragile than is generally suspected. We also present citation analyses showing that, consistent with the Kuhnian view that adherents of a paradigm tend to ignore (later) anomalous evidence, subsequent researchers largely overlooked post-Beaver evidence that was inconsistent with the paradigm's interpretation of Beaver's results. The paper concludes with a brief consideration of how cognitive biases of individuals, combined with biases inherent in the review process and the academy in general, foster an environment where the placement of the first research bricks affect the whole wall. While the case study focuses on an archival empirical financial accounting study, consideration of the effects of subjective research design choices and the documented deficiencies in the interpretation of research are relevant to empirical accounting researchers in general and to critical theorists, as well. While accounting scholars ‘think we understand’ the impact of research design choices, the evidence presented here suggests that we have collectively failed to apply this understanding in practice. Such failure delays the acquisition of knowledge. This study provides a salient demonstration indicating that we must constantly remain on guard to recognize the subjective nature of research design choices, to consider the likely effects of those choices, and to be cautious in drawing generalizations.
Read full abstract