Open up: a survey on open and non-anonymized peer reviewing

Lonni Besançon,Niklas Rönnberg,Jonas Löwgren,Matthew Cooper,Jonathan P Tennant

doi:10.1186/s41073-020-00094-z

Abstract

BackgroundOur aim is to highlight the benefits and limitations of open and non-anonymized peer review. Our argument is based on the literature and on responses to a survey on the reviewing process of alt.chi, a more or less open review track within the so-called Computer Human Interaction (CHI) conference, the predominant conference in the field of human-computer interaction. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is now being considered and used in other fields.MethodsWe ran an online survey with 30 responses from alt.chi authors and reviewers, collecting quantitative data using multiple-choice questions and Likert scales. Qualitative data were collected using open questions.ResultsOur main quantitative result is that respondents are more positive to open and non-anonymous reviewing for alt.chi than for other parts of the CHI conference. The qualitative data specifically highlight the benefits of open and transparent academic discussions. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.ConclusionWhile the benefits are quite clear and the system is generally well-liked by alt.chi participants, they remain reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation.

Highlights

Pre-publication peer review of scientific articles is generally considered to be an essential part of ensuring the quality of scholarly research communications [1,2,3]
Respondents had submitted an average of 1.9 papers (SD = 1.8) through the open reviewing process of alt.chi, while only two authors had submitted to a juried version of alt.chi
We have conducted an initial investigation on the perception of open reviewing within the only venue that has an open reviewing process in the human-computer interaction community

Summary

Introduction

Pre-publication peer review of scientific articles is generally considered to be an essential part of ensuring the quality of scholarly research communications [1,2,3]. It can take many forms from single-round peer review, typical of conferences, to multiple-stage peer reviewing, more common in scholarly journals. Classical single/double-blind reviewing is held in high regard within scientific communities and is often considered as the gold standard for assessing the validity of research communications [1,2,3, 8,9,10,11]. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is being considered and used in other fields

Objectives

Results

Conclusion