AbstractThe purpose of the present investigation was to assess the agreement between two independent reviews of each of 278 manuscripts submitted to the Journal of Educational Psychology on an overall recommendation and on specific rating items. The single-reviewer reliability for the overall recommendation, .30, was comparable with the .34 in a previous study with the same journal and with an average value of .27 found in other research. Factor analyses and multitrait-multimethod analyses provided support for four dimensions: Research Methods, Relevance to Readers, Writing Style/Presentation Clarity, and Significance/Importance. However, idiosyncratic response biases (e.g., tendencies toward leniency or harshness in responses) were larger for the specific rating items that defined these dimensions than for the overall recommendation that was better anchored in terms of behavioral consequences. Consequently, agreement between different reviewers on each separate dimension, on the unweighted sum of these d...
Read full abstract