Revealing an exchange between authors and reviewers about statistical significance

Abstract

s editors, we are privy to exchanges between authors and reviewers during the review process. To preserve confidentiality in the review process, these exchanges are not revealed beyond the editorial review process, even though they are often quite enlightening and often produce interesting and useful clarifications of issues in the field. However, recently we were led to make an exception, which prompts this editorial. During the review of the article in this issue authored by Barbara M. Taylor, P. David Pearson, Debra S. Peterson, and Michael C. Rodriguez, there was an exchange of views between the authors and three members of RRQ’s editorial review board, who were asked to respond to a concern raised by one of the original reviewers. At our request, the authors and the reviewers have graciously agreed to share their exchange. Their respective contributions to the exchange appear subsequent to this editorial, although each has been slightly revised and edited for publication. To introduce and explain this exchange, we wish to provide readers with background about how it emerged in the context of the review process. During the initial review of the manuscript, one of the reviewers questioned the use of accepting as statistically significant a p value of .07. When submitting a subsequent revision, the authors argued that the circumstances of their research and particularly their statistical procedures justified an alpha above the customary value of .05. Realizing that this may be an important issue given new views about statistical analyses (see New Directions for Research, Reading Research Quarterly, Volume 39, No. 1) and sensing that we did not possess sufficient statistical expertise to make a final decision, we sought advice from three statistical experts who serve on RRQ’s editorial review board. We asked them specifically to address this issue in the context of the authors’ manuscript. They did, and we shared their anonymous responses with the authors. A