The Role of Peer Review in Raising Quality of Scientific Sociological Knowledge (The Case of <i>Sociological Studies</i>)

Peer review on the Internet: A better class of conversation

JID Innovations and Peer Review

Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemic

A Tool for Reviewers

Dispirited Away: The Peer Review Process

While the term Open Science may immediately bring to mind Open Access publishing, this movement also encompasses Open Data, Open Source, Open Methodology, Open Educational Resources and Open Peer Review. Equine Veterinary Journal uses a hybrid publishing model whereby we offer authors an Open Access publishing option. Currently, around 10% of articles in Equine Veterinary Journal are published in this way. With respect to Open Data, Equine Veterinary Journal encourages authors to make data accessible to other researchers in the belief that the more data are made openly available, the more efficient the scientific process becomes, to the benefit of society 1. The barriers to data sharing include fears of exploitation and loss of data on which the scientists who have collected such data are relying for future publications. Authors also rightly recognise that making data available to others takes time and effort for which they are given little or no career reward or recognition. With clinical data, there is an additional concern that patient confidentiality must be maintained above all else, and owners agreeing to participate in a specific study for which they have been provided with details of the aims and methods, do not necessarily agree to use of such data in future. Appreciating these barriers, Equine Veterinary Journal encourages authors to use public repositories as much as possible but we have also chosen to allow our authors to make use of the option of providing data upon reasonable request, a vehicle which is particularly useful for clinical datasets and allows researchers to evaluate and maintain control of how the data are used in future projects. Science has relied on peer review for almost three centuries 2. Peer review has two distinct functions; firstly, technical evaluation of methods, analysis and interpretation of results and secondly, in assisting editors in evaluating the impact of the work in their community. Arguably, the first function is more transparent and authors are provided with detailed reports highlighting and justifying any concerns or areas of weakness identified by the peer reviewers. In assessing scope and impact in the journal’s target readership, editors rely on their own judgement and perception of what the target readers are likely to be interested in and find useful supplemented by the expertise and judgement of peer reviewers who will typically be selected on the basis of their specific areas of interest and achievement. In this respect, editors rely heavily on the journal’s stated mission, which for Equine Veterinary Journal is to publish and promote high quality research focused on equids that improves clinical practice and informs veterinary science. There is no doubt that our focus increasingly is on clinical relevance but assessment of impact remains, and likely always will be, a subjective assessment. Equine Veterinary Journal is fortunate that our team of editors collectively encompass a wide range of expertise relevant to equine clinical practice. Researchers generally acknowledge that peer review leads to an improvement in quality of publications 3 and arguably, peer review is of equal importance to our readers who often rely on the peer review process to check for appropriate and robust methodology and factual accuracy 4. Five principles for better peer review have been identified as underpinning best practice: Content Integrity, Content Ethics, Fairness, Usefulness and Timeliness 5. Integrity includes ensuring the content is accurate, complete and reproducible while ethics in the publication context relates to the duty of editors to ensure that authors declare that work has not been published previously and individuals are acknowledged appropriately for their contribution. Fairness in peer review relates to ensuring that each manuscript is judged on its own merits without bias. Equine Veterinary Journal’s double-masked peer review process is one of several mechanisms which seeks to ensure this 2. Editors and reviewers are expected to declare any potential conflicts of interest in general and in relation to individual manuscripts and journals should have clearly stated policies regarding authorship attributions details of which are set out in our Author Guidelines. Fairness on the part of the peer reviewers requires them to write thorough reviews in the first round of peer review. Reviewers should use subsequent rounds of revision to ensure these comments are addressed but they should not introduce new comments without good reason in further revision/review rounds. An example where it might be appropriate to raise new concerns regarding a revision is where aspects of the methods have been so poorly reported in the first manuscript version that a major methodological flaw does not become evident until reporting of the methods is revised. Editors must take all reasonable steps not to involve additional individuals in the peer review process at the revision stage, although regrettably this is sometimes unavoidable where peer reviewers become unavailable due to changes in their circumstances. Peer reviewers should also be mindful that the process is confidential—while this is not the case in all areas of science, Equine Veterinary Journal discourages peer reviewers from seeking assistance from others when completing their reports. Rather, if a peer reviewer realises that aspects of a manuscript are out of their personal expertise, they should contact the editor handling the paper as soon as possible. Generally, the reviewer will be expected to complete a report on the areas they are competent to deal with but early notification gives the editor the opportunity to invite additional reviewers as required. Some individuals may want to outsource peer review to a younger colleague, seeing this as a way to provide mentorship in the peer reviewing process. While there certainly is merit in widening the pool of reviewers and introducing early career scientists to peer reviewing, Equine Veterinary Journal prefers that the appointment of peer reviewers remains in the hands of the editors. We will often couple less experienced reviewers with one or more senior individuals and we share all peer review reports so that reviewers can assess their own performance against others. Our reviewer invitations are accompanied by an option to nominate others, we formally ask our Editorial Consultant Board for nominations annually and individuals who would like to get involved in peer review are encouraged to contact the editor to volunteer. We are in the fortunate position that through Wiley and our partnership with Publons, we now have access to a wide range of online resources to support novice (and experienced) peer reviewers in addition to our own reviewer guidelines which are linked to every manuscript file. Timeliness, or lack thereof, is one of researchers’ main criticisms of peer review 6. The speed of publication of veterinary research is similar to that of medical and general science high-impact journals and data available in a preprint posted online in 2018 suggests the interval from submission to acceptance is similar across most journals 7. Comparing ourselves, subjectively, with other veterinary journals, suggests Equine Veterinary Journal performs fairly well in terms of time from acceptance to publication (median 38 days, IQR 32–48 days) but we have room for improvement in our submission-to-acceptance time (median 184 days, IQR 144–262 days), albeit that we appear to be pretty much in the centre of the pack. Authors may feel reassured to learn that Equine Veterinary Journal’s editors review timelines on a quarterly basis and we also quantify the number of revisions requests associated with each manuscript to ensure repeat review rounds are minimised. This is an area in which we can claim some progress in recent years, although again with continued room for improvement and arguably having started from a fairly low point. At the end of 2013, 73% of our articles were accepted with two or less revisions, this is now over 80% with a target of pushing this ever upward to 100%. This emphasis on detail, quality, and timeliness places considerable demands on the peer reviewers, a group of volunteers who largely perform this chore for altruistic reasons, motivated mainly by duty, a desire to pay back for support from a system they have been beneficiaries of and to see equine welfare improve by the advancement of knowledge of disease and its management. The individuals who have contributed to peer review over the last year are listed in this issue of Equine Veterinary Journal and I thank these individuals for their generous support of the journal. Open Peer Review is an exciting new initiative, which has arisen out of the Open Science movement and seeks to enhance science in two ways. Transparency adds to the usefulness of peer review 5. Publishing review reports is argued to improve accountability without negatively impacting review quality. Open Peer Review also seeks to demystify the process and showcase the work of the editors and peer reviewers. Currently, there is no single definition of Open Peer Review or Open review 8. These terms can be taken to mean that the identities of authors and peer reviewers are known to each other but readers are not made aware of the reviewer’s identity or comments, a process which is not in place at Equine Veterinary Journal, rather we do the opposite and mask both identities 2. Alternatively, open peer review can be defined as a system whereby reviewer and editor reports are published alongside published articles. Recently, Equine Veterinary Journal has joined a Transparent Peer Review pilot which Wiley has recently launched across 50 of their journals in partnership with Publons. The process has two parallel but separate components: Open Report and Open Reviewer Identity. Open Report ultimately leads to online publication of all reviewer and editor reports together with the authors’ responses online. There is no change to the peer review mechanism and the published reports are hosted by Publons and accessed via a link within the digital Version of Record of each published paper. As each report can be regarded as an online publication in its own right, reviewers can choose to use these to build up a verified portfolio of their contributions to science through peer review. Open Report is voluntary: authors are asked at the time of submission, and with every subsequent submission, whether they want to join the Transparent Peer Review system. There is no obligation to do so and authors are not disadvantaged if they decline initially or withdraw at the revision stage. Potential peer reviewers are made aware at the time they are invited to review a manuscript that the Transparent Peer Review system is in operation. Unfortunately, currently at least, if they do not support the initiative at all, their only option is to decline the invitation. However, importantly, Open Review Identity, the process of revealing their identity along with the published reports is separate—in other words, their report can be included regardless of whether they choose to reveal their identity or not. To gain the credit for certified peer review, reviewers must have a Publons account but this has general benefits independent of the Transparent Peer Review system. Work performed within Wiley based on an earlier pilot with 5 journals and around 2000 papers has shown that there is good uptake from authors: early signs within Equine Veterinary Journal suggests around 80% of our authors are agreeing to join. The feedback from reviewers in Wiley’s early pilot is positive and Transparent Peer Review drives peer review quality at both editor and reviewer levels 8. There are some potential downsides to our Transparent Peer Review system: a clear weakness is that the reports are only published for papers which are accepted, an insurmountable problem with the system Equine Veterinary Journal is piloting which requires an accepted and published paper to exist before reports can be linked to it. There is no real expectation that published peer review reports will be widely read. Their existence serves to maintain an accessible audit trial to enhance transparency and accountability but individual readers may find them useful where they are undertaking detailed analysis of a study. There is also concern emerging from Wiley’s earlier pilot that operation of Transparent Peer Review drives editors to evaluate initial submissions more critically. This, in turn, decreases the number of manuscripts which are selected to proceed to the peer review process. Arguably rejection at the early, Internal Editorial Assessment stage, although disappointing, can be of benefit to authors. If the work they have submitted has a low likelihood of success in a given journal, often for reasons of relevance to the readership, it is useful to learn that as soon as possible. Early rejection with constructive advice can help authors find a more appropriate journal in a timely fashion, and in some instances, authors will be given advice on how to expand their work or improve their manuscript before resubmitting to Equine Veterinary Journal. It remains to be seen whether Equine Veterinary Journal will adopt Transparent Peer Review permanently. Over the next 6 months, we will be monitoring uptake rates from authors, reviewer behaviour patterns and feedback from stakeholders to ensure that this is widely regarded as a positive policy with no negative impact on either publication times or manuscript success rates. We encourage our readers, authors and reviewers to provide comments on this initiative in Open Science.

Web 2.0 and academic debate: Social media challenges traditions in scientific publishing.

“Thinking With” When Peer Reviewing: Introduction to the <i>PoLAR Online</i> Emergent Conversation on Peer Review

Reviewer Blinding in Peer Review

Scientific journals have been called the ‘minutes of science’ [1]. Born out of the exchange of letters on scientific topics and results, publication is a way of documenting what was done and, particularly in the case of open-access journals, sharing the outcome. Journal publications are considered authoritative and are generally used to inform the work of others, be it further research or, in the case of biomedicine, in treatment decisions for patients. This makes some kind of quality control all the more important. The ‘gold standard’ for this quality control is peer review. Described as a form of self-regulation by qualified members of a profession, it was first introduced by Henry Oldenburg, the founding editor of Philosophical Transactions of the Royal Society, in 1665 as a means of vetting contributions to the Royal Society of London and has persisted in various forms ever since [2]. Peer review is the evaluation of a piece of work by two or more people of similar competence to the authors; but, assuming the reviewing process excludes all those involved in the direct research itself, the persons most qualified to judge the validity of a submitted research paper are precisely those who are the scientist’s closest competitors. This means that the review process can become adversarial, with referees seeming to see it as their responsibility to insist on time-consuming additions and revisions [3,4]. Moreover, under traditional, closed peer-review policies, the identity of the reviewer is withheld from the author, presenting them with a greater opportunity to act arbitrarily. It was in an effort to combat this bias that some journals introduced double-blind peer-review, whereby the author’s name was also concealed from the referee. However, research is a small world and maintaining that blinding often proved impossible. So how did peer review, with these intrinsic issues and biases, become the judicial system of the intellectual world? Simply put, peer review is to science what democracy was to Churchill – ‘the worst form of government, except all those other forms that have been tried from time to time.’ It has served science well, with a widely-held view that, while it may not be perfect, it is nonetheless far better than anything else we have been able to devise. Indeed, the fundamental idea of peer-review seems sound; the issues lie more with the execution. Under closed systems, such as that currently enforced by the Journal of Negative Results in BioMedicine, there is a lack of transparency of the peer-review process and a lack of availability of evaluative information about published articles to the public. Therefore, as of February 2014, the Journal of Negative Results in BioMedicine will adopt an open peer-review policy. Articles already published, or those manuscripts currently submitted, will not be affected by this change. However, for all manuscripts submitted during or after February 2014, authors will see the reviewers’ names and, if the article is published, the reading public will also see who reviewed the article and how the authors responded. This will be available as part of the pre-publication history of the published article. The peer review process will therefore be completely open and transparent, with the peer reviews being part of the record. Research into the effect of open peer review suggests numerous benefits, in particular accountability, fairness and crediting reviewers for their efforts [5-7]. Furthermore, in a recent study, Kowalczuk et al. revealed that reviewer reports operating under an open peer-review system were of overall higher quality than those under a closed system, with higher scores on questions relating to feedback on the methods (11% higher), constructiveness of the comments (5% higher), and the amount of evidence provided to substantiate the comments (9% higher) [8]. Despite this, we recognise that there are also negatives. Some (junior) reviewers may feel uncomfortable signing a critical report, especially when recommending rejection [9]. This reluctance also means that more potential referees may need to be invited to review a manuscript openly than under a closed peer-review system (Parkin EC et al. unpublished observations) [9-11]. Reviewing an article is no easy task and many of us will have faced the situation where it feels we have put more thought into our review of the article than the authors did in designing the study and writing the manuscript. The move towards an open peer-review policy will give credit where it is due, but moreover will provide valuable information to those reading the article, sharing the referees’ critique of the manuscript and presenting all the necessary information for them to make an objective evaluation for themselves.

“Publication is the hard currency of science. It is the primary yardstick for establishing priority of discovery, making the status of a publication a critical factor in resolving priority disputes or intellectual property claims.” (International Working Group, 2000) Nurse authors, especially those just starting out with publishing their work, may be unsure about which type of publication it is best to aim for. The purpose of this article is to assist in making this choice. The objective is to show what makes a publication scholarly, what an editor looks for when deciding to publish an article, and lastly the peer review process. This will help nurse authors to decide whether a peer-reviewed publication is the appropriate place for their articles, and the process used by editors when deciding whether to accept articles. ‘Scholarly publishing is a communicative practice within a community’ and ‘an activity engaged in by scholars who primarily want their reports to be widely read and credited by their target audience’ (Kling and McKim 1999 p.890). Scholarly publication is a social process, not an activity undertaken in isolation, as often other scholars review, test and critique the information presented. As Borgman (2000 p. 412) explained, ‘research in all fields is incomplete until it is validated through review processes and shared with others‘. Scholarly publication is most often divided into three stages: first, informal communication, which now occurs largely through the use of electronic media (Gabbay, 2000); second, the initial public dissemination of research through conferences and proceedings, and finally, formal publication i.e. in journals or books, chapters in books, articles, on-line formal and public proceedings, most of which are peer-reviewed (Graham, 2000). These formal publications are often the result of extensive review of literature (meta-analysis), personal clinical experiences (case studies) or rigorous research. As they seek opportunities to publish their work, scholars may find it difficult to determine which source is the best especially in terms of what is scholarly and what is not (Rozenzweiz, 1999). There are three key ways to identify a scholarly publication. The first question to ask is: Who does the writing? In a scholarly publication the writing of the manuscript is done by a scholar/researcher in the field. They have the education and credentials that assure the reader that they have the knowledge to write on the topic. The scholar/researcher is reporting on the results of work for which they are responsible. In more popular publications some of the writing may be done by a staff writer, who is most often a professional reporter or a journalist. They interview others and report information learned. They have not done the research or the review or experienced the clinical case study, but instead report on work done by someone other than themselves. Some of the writing may also be done by nurses, but in a more reader-friendly and less academic style that will appeal to busy practitioners who do not have the time or confidence to read scholarly journals. However, these popular publications often publish information that is based on research or other types of evidence and can be used to develop or improve practice. Examples of such popular nursing publications are Nursing Times in the UK, Journal of the American Academy of Nurse Practitioners in the USA and Australian Electronic Journal of Nursing Education (AEJNE) in Australia. The second question to ask is: Who are the readers? Scholarly publications are intended to be read by other scholars interested in the same field or discipline. Nurse educators also draw on these kinds of publications for their teaching, and recommend them on reading lists for their students. More popular publications are intended to be read by general readers. The articles often omit features that a researcher would want to know, such as statistics or probability information, and focus more on the ‘take home message’ or practical implications of the topic. The third question to ask is: Is the publication peer reviewed? Manuscripts submitted to scholarly publications are sent for review to other scholars or researchers who have expertise in the topic discussed. These reviewers critique the manuscript and make recommendations on whether or not it should be published. This process is called peer review and is extremely important in determining whether the manuscript is noteworthy. It will determine if the manuscript is appropriate for their publication and if it meets the rigor expected of articles published. As mentioned above, peer review is the key to determining whether a manuscript is scholarly or not. Before a manuscript can be published in most scientific journals, it must be reviewed by colleagues (“peers”) who are experts in the topic written about in the manuscript. Peer review is a method of providing a system of determining quality of material to be published and assuring readers that the information they read is reliable and valid – worth their attention and time. Why is peer review so important? In this process, each article undergoes “blind” or anonymous review by a group of “peers” (scholars with similar expertise and education). These “peers” are the experts, scholars and colleagues of those who themselves write articles and books, and are often university professors. Most peer reviewers have extensive publishing experience on the research or topic on which the article or book focuses. “Blind” peer review is the most reliable and valid form of peer review process because the reviewers do not know the authors' names or university affiliations. This protects the reviewers' ability to give constructive criticism without fear of reprisal from an offended author (Scarry 1991). Authors benefit from the blind review in that, if a manuscript is poorly written, their careers are not jeopardized because the reviewers would have no means of identifying who the authors are. After the peer reviewers have completed their reviews, they are forwarded to the author with recommendations that the manuscript is ready for publication as it stands (very rarely); the manuscript is suitable for publication if certain modifications can be made, questions answered and any errors corrected; or the manuscript is not suitable for publication and has been rejected. The manuscript may be suitable for publication but does not fit the particular focus of the journal, and the reviewers or editor may suggest submitting it to a different journal. Only in the first two cases will the manuscript reach print, and in the second it will probably be returned to the reviewers for another round of scrutiny before actual acceptance for publication. This is what makes peer-reviewed scholarly publications so valuable. Choosing the most appropriate scholarly publication is vital to an author's publishing success. As well as the points discussed in this article, an early step is to check out possible journals by looking at hard copies in an academic library or by going to journals' websites. These websites will also take you to each journal's author guidelines, which need to be applied as closely as possible to improve the articles chances of success. Jacqueline Rhoads PhD, ACNP-BC, ANP-C, CCRN is a Professor, Director, Primary Care Nurse Practitioner Program, Louisiana State University Health Science Center, School of Nursing, New Orleans, USA.

The JAMA and NEJM Rulings and Their Impact on the Sanctity of Confidential Peer Review

Open peer review (OPR) is a cornerstone of the emergent Open Science agenda. Yet to date no large-scale survey of attitudes towards OPR amongst academic editors, authors, reviewers and publishers has been undertaken. This paper presents the findings of an online survey, conducted for the OpenAIRE2020 project during September and October 2016, that sought to bridge this information gap in order to aid the development of appropriate OPR approaches by providing evidence about attitudes towards and levels of experience with OPR. The results of this cross-disciplinary survey, which received 3,062 full responses, show the majority (60.3%) of respondents to be believe that OPR as a general concept should be mainstream scholarly practice (although attitudes to individual traits varied, and open identities peer review was not generally favoured). Respondents were also in favour of other areas of Open Science, like Open Access (88.2%) and Open Data (80.3%). Among respondents we observed high levels of experience with OPR, with three out of four (76.2%) reporting having taken part in an OPR process as author, reviewer or editor. There were also high levels of support for most of the traits of OPR, particularly open interaction, open reports and final-version commenting. Respondents were against opening reviewer identities to authors, however, with more than half believing it would make peer review worse. Overall satisfaction with the peer review system used by scholarly journals seems to strongly vary across disciplines. Taken together, these findings are very encouraging for OPR’s prospects for moving mainstream but indicate that due care must be taken to avoid a “one-size fits all” solution and to tailor such systems to differing (especially disciplinary) contexts. OPR is an evolving phenomenon and hence future studies are to be encouraged, especially to further explore differences between disciplines and monitor the evolution of attitudes.

Reviewers and the review process

Objective: F1000Research operates an author-driven, open, and post-publication peer review model. The identity of the reviewer and the peer review report including its recommendation are made public immediately after submission by the reviewer. This study aimed to identify any potential influence of the first published peer review on the recommendation of the second reviewer, as measured by the agreement between the 2 recommendations and the time between them. Design: Bringing together a dataset of articles published between July 2012 and February 2017 and associated open peer review reports, we analyzed the agreement among reviewers depending on the time between reports. Only articles presenting original research or methods were included. Articles where the time gap between the 2 reviewer reports was longer than a year (365 days) were excluded. The recommendations (“approved,” “approved with reservations,” or “not approved”) of the first 2 reviewers were recorded, as well as the published date of the reports. Cohen’s κ was used to measure inter-rater reliability, and its change with time between reports was used to assess potential bias. In the absence of survey data on whether the second reviewer had read a previous report before submitting their recommendation, reports published within the same day are considered the control group. Results: The analyzed dataset contained 1,133 articles and 2,266 reviewer reports, ie the first 2 reviewer reports of each article. The median (interquartile range) time between the first 2 peer reviews was 18 (6-52) days. In aggregate, the breakdown of the peer review decision (“approved,” “approved with reservations,” or “not approved”) across the dataset was virtually identical between the 2 reviewers (724 [63.9%], 355 [31.3%], and 54 decisions [4.8%] vs 705 [62.2%], 372 [32.8%], and 56 [4.9%] decisions, respectively). However, comparing the recommendations made for each article individually, the Cohen’s κ was 0.330 (compared with 0.282 for the control group), indicating only a fair agreement between the reviewers. Moreover, the Cohen’s κ changed minimally with the length of time between the peer review publication dates. Conclusions: Our analysis of the F1000Research open peer reviews found that the agreement between reviewers did not change substantially with the time gap between peer reviews. The second reviewer does not seem to be systematically influenced by the ability to see the recommendation of an earlier reviewer. This is an important finding and something to continue to monitor as the momentum and acceptance of open peer review models, and open science more broadly, continues to grow.