Improve Peer Review Research Articles

A structured, journal-led peer-review mentoring program enhances peer review training

BackgroundPeer review is essential to the advancement of knowledge. However, training on how to conduct peer review is limited, unorganized, and not well studied. Thus, we sought to determine if a structured mentored peer-review program improved peer review training as measured by multiple quantitative and qualitative assessments.MethodsThis pre-post intervention study enrolled 55 mentees across 5 cohorts from 2020 to 2023. Each cohort completed pre-program evaluations, participated in 2 mentored reviews, and completed post-program evaluations over 6 months. Mentors and mentees completed pre-program demographic and review experience questionnaires. Outcome measures included (1) total and sub-scores on the modified Review Quality Index (mRQI) applied to the same pre-selected research manuscript reviewed by mentees both pre and post intervention, (2) mentee self-perceived comfort with and understanding of the review process using a custom questionnaire, and (3) mentor satisfaction surveys. Pre- and post-program measures were compared using the Wilcoxon signed-rank test.ResultsPost-program total modified RQI score (median (IQR) = 31 (26.3–35.8)) was higher than pre-program total score (26.6 (19.7–29.7)) for the 42 mentees who completed both pre- and post-program reviews. Mentees reported improved perception of review (median (IQR) pre = 4 (3–4), post = 5 (4–5)) and editorial processes (pre = 3 (2–4), post = 4 (4–5)) as well as self-perceived confidence in completing an independent review of both scientific (median (IQR) pre = 2 (2–3), post = 4 (4–4)) and non-scientific (pre = 3 (2–4), post = 4 (4–5)) manuscripts following program participation. p < 0.0001 for all scores noted. Mentors reported high scores for enjoyment (median (range) 5/5 (3–5)) and interest in repeat participation (5/5 (2–5)).ConclusionsA 6-month structured mentored-review program including 2 mentored reviews improves peer review training as measured by the modified RQI as well as participant self-perceived understanding of publication science with high mentor satisfaction.

A harm reduction approach to improving peer review by acknowledging its imperfections

This candid perspective written by scholars from diverse disciplinary backgrounds is intended to advance conversations about the realities of peer review and its inherent limitations. Trust in a process or institution is built slowly and can be destroyed quickly. Trust in the peer review process for scholarly outputs (i.e., journal articles) is being eroded by high-profile scandals, exaggerated news stories, exposés, corrections, retractions, and anecdotes about poor practices. Diminished trust in the peer review process has real-world consequences and threatens the uptake of critical scientific advances. The literature on “crises of trust” tells us that rebuilding diminished trust takes time and requires frank admission and discussion of problems, creative thinking that addresses rather than dismisses criticisms, and planning and enacting short- and long-term reforms to address the root causes of problems. This article takes steps in this direction by presenting eight peer review reality checks and summarizing efforts to address their weaknesses using a harm reduction approach, though we recognize that reforms take time and some problems may never be fully rectified. While some forms of harm reduction will require structural and procedural changes, we emphasize the vital role that training editors, reviewers, and authors has in harm reduction. Additionally, consumers of science need training about how the peer review process works and how to critically evaluate research findings. No amount of self-policing, transparency, or reform to peer review will eliminate all bad actors, unscrupulous publishers, perverse incentives that reward cutting corners, intentional deception, or bias. However, the scientific community can act to minimize the harms from these activities, while simultaneously (re)building the peer review process. A peer review system is needed, even if it is imperfect.

Peer review’s irremediable flaws: Scientists’ perspectives on grant evaluation in Germany

Abstract Peer review has developed over time to become the established procedure for assessing and assuring the scientific quality of research. Nevertheless, the procedure has also been variously criticized as conservative, biased, and unfair, among other things. Do scientists regard all these flaws as equally problematic? Do they have the same opinions on which problems are so serious that other selection procedures ought to be considered? The answers to these questions hints at what should be modified in peer review processes as a priority objective. The authors of this paper use survey data to examine how members of the scientific community weight different shortcomings of peer review processes. Which of those processes’ problems do they consider less relevant? Which problems, on the other hand, do they judge to be beyond remedy? Our investigation shows that certain defects of peer review processes are indeed deemed irreparable: (1) legitimate quandaries in the process of fine-tuning the choice between equally eligible research proposals and in the selection of daring ideas; and (2) illegitimate problems due to networks. Science-policy measures to improve peer review processes should therefore clarify the distinction between field-specific remediable and irremediable flaws than is currently the case.

Utilising WeChat to improve peer review communication in China

Western editorial management systems and processes are built around the English language and Western technology experience. China comprised 17% of the global published output in 2021, yet there is a mismatch between STM workflows and processes that have not been designed with Chinese users in mind. Journal submission systems use email as the primary form of communication, which is culturally at odds with the Chinese digitally mobile community. China has a unique social media ecosystem with influential home-grown channels of communication. Communication via Chinese super apps, like WeChat, is more than simply building a social media strategy in China — it’s about publisher inclusivity and publishers communicating more effectively with their author communities. In this article, Charlesworth Gateway <https://cwbot.com.cn/> (Gateway) shares their experience in creating a software solution for publishers and their Chinese authors. Gateway is a plug-in tool that allows publishers to integrate their submission systems with their WeChat channel, either via a mini-program or direct within their WeChat channel. Gateway has been developed to improve the Chinese author communication and publishing experience by sending automated unique short notifications in Chinese to the author throughout the submission process via this dominant social media app. This article will explore how early adopters of the solution have seen positive benefits to their workflows and author communication strategies. Through a process of engaging with authors and clients to refine the product over time, Gateway has created a configurable service that supports a publisher to create a more personal, branded experience for authors in China.

Towards improving peer review: Crowd-sourced insights from Twitter

Peer review is an essential part of academic publishing, yet many authors, reviewers, and editors have reportedly encountered problems with the review process. Some scholars view peer-review as a necessary process for the advancement of science, while other scholars argue that for many publishers and journals, both authors and reviewers are being exploited. The aim of this commentary is two-fold. First, to provide a narrative review of current perspectives and available research on the peer-review process to date, and second, to summarise potential solutions elicited from scholars on Twitter. A review of the literature identified several problems with peer-review including publication delays, an over reliance on a narrow pool of reviewers, threats to anonymity, perceived exploitation, as well as overworked editors. Recommendations to redress these issues that emerged from scholars on Twitter suggested publishers, journals, their editors and associate editors, universities, individual academics and their communities all have a role to play towards creating an equitable and fair system. This commentary aims to ignite conversations about improving the peer-review process.

Alice through the Looking-glass: Can We Improve Peer Review?

Peer review is an essential component for the evaluation of material submitted to medical and scientific journals. This process usually involves single blind, double blind or open peer review. In the present Editorial, we discuss the advantages and disadvantages of these peer review options. Our comments are influenced by our experience as Editors-in-Chief. We also outline new developments in the field of peer review.

What works for peer review and decision-making in research funding: a realist synthesis

IntroductionAllocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom.MethodsRealist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding.ResultsWe analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels.ConclusionsReports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts.

Towards improving peer review: Crowd-sourced insights from Twitter

Peer review is an essential part of academic publishing, yet many authors, reviewers, and editors have reportedly encountered problems with the review process. Some scholars view peer-review as a necessary process for the advancement of science, while other scholars argue that for many publishers and journals, both authors and reviewers are being exploited. The aim of this commentary is two-fold. First, to provide a narrative review of current perspectives and available research on the peer-review process to date, and second, to summarise potential solutions elicited from scholars on Twitter. A review of the literature identified several problems with peer-review including publication delays, an over reliance on a narrow pool of reviewers, threats to anonymity, perceived exploitation, as well as overworked editors. Recommendations to redress these issues that emerged from scholars on Twitter suggested publishers, journals, their editors and associate editors, universities, individual academics and their communities all have a role to play towards creating an equitable and fair system. This commentary aims to ignite conversations about improving the peer-review process.

Publishing your scholarship: a survey of pearls from top reviewers

ABSTRACT Experts have described ways to improve peer review quality. Perspectives from expert reviewers are largely absent in the health professions education literature. To gather guidance from expert reviewers, to aid authors striving to publish and reviewers aiming to perform their task effectively. This study surveyed the Journal of Graduate Medical Education (JGME) ‘Top Reviewers’ from 2017, 2018, and 2019. ‘Top Reviewers’ perform four or more reviews per year, with high average ratings. Top reviewers were sent an 11-item survey in February 2020. The survey included three demographic questions and eight open-ended, free-text questions about the concepts reviewers most often target in their reviews. We calculated descriptive statistics and performed a thematic analysis of open-ended responses. Of 62 eligible top reviewers, 44 (71%) responded to the survey. Only eight (18.2%) and seven (15.9%) respondents reported having ‘stock phrases’ or a reviewer template used for reviewer feedback to authors, respectively. The what (research question, methods), how (presentation, writing), and why (relevance, impact) were the resulting themes summarizing how reviewers categorized and responded to common problems. For ‘really good papers’ reviewers found the what acceptable and focused on how and why. For ‘really bad’ papers, reviewers focused on big picture feedback, such as the value of the study. Top reviewers from a single health professions education journal appear to have similar approaches to conducting reviews. While most do not use stock phrases or templates, they share similar strategies to differentiate ‘good’ vs. ‘bad’ papers through the what, why, and how of a manuscript.

Recommendations for improving peer review of qualitative research in conservation journals.

Wildlife and conservation journals must do more to ensure qualitative research contributions are reviewed appropriately and fairly. This article is protected by copyright. All rights reserved.

Improving peer review of systematic reviews by involving librarians and information specialists: protocol for a randomized controlled trial

BackgroundProblems continue to exist with the reporting quality and risk of bias in search methods and strategies in systematic reviews and related review types. Peer reviewers who are not familiar with what is required to transparently and fully report a search may not be prepared to review the search components of systematic reviews, nor may they know what is likely to introduce bias into a search. Librarians and information specialists, who have expertise in searching, may offer specialized knowledge that would help improve systematic review search reporting and lessen risk of bias, but they are underutilized as methodological peer reviewers.MethodsThis study will evaluate the effect of adding librarians and information specialists as methodological peer reviewers on the quality of search reporting and risk of bias in systematic review searches. The study will be a pragmatic randomized controlled trial using 150 systematic review manuscripts submitted to BMJ and BMJ Open as the unit of randomization. Manuscripts that report on completed systematic reviews and related review types and have been sent for peer review are eligible. For each manuscript randomized to the intervention, a librarian/information specialist will be invited as an additional peer reviewer using standard practices for each journal. First revision manuscripts will be assessed in duplicate for reporting quality and risk of bias, using adherence to 4 items from PRISMA-S and assessors’ judgements on 4 signaling questions from ROBIS Domain 2, respectively. Identifying information from the manuscripts will be removed prior to assessment.DiscussionThe primary outcomes for this study are quality of reporting as indicated by differences in the proportion of adequately reported searches in first revision manuscripts between intervention and control groups and risk of bias as indicated by differences in the proportions of first revision manuscripts with high, low, and unclear bias. If the intervention demonstrates an effect on search reporting or bias, this may indicate a need for journal editors to work with librarians and information specialists as methodological peer reviewers.Trial registrationOpen Science Framework. Registered on June 17, 2021, at https://doi.org/10.17605/OSF.IO/W4CK2.

Peer review experiences of academic chemists in Ph.D. granting institutions in the United States

ABSTRACT Academic chemists at Ph.D. granting institutions in the United States were surveyed on the time and effort they spend on peer reviews and how they rate themselves as reviewers. Thirty percent of the respondents reviewed 16 or more papers yearly. This seemingly high number is consistent with the number of papers some scientists publish, and the rough estimate of two to three reviews is obtained per manuscript submission. Approximately 30% of the respondents reported that they spent two hours or less per review; that 60% rate themselves as strong or very strong reviewers; that the youngest reviewers are more likely to be compulsive in their reviewing; and that respondents who spend more time on reviews complete fewer reviews per year. Sixty percent of the respondents categorized themselves as strong or very strong reviewers, suggesting that most scientists see reviewing papers as an essential component of their professional responsibilities. These ratings suggest an opportunity to improve peer review quality. Good citizenship within the scientific community suggests that each scientist should review ca. two to three times as many papers each year as they submit, and that reviewers need to see reviewing as “providing to others what authors hope reviewers will provide to them.”

How I Review a Paper to Help Improve Citability and Impact

How I Review a Paper to Help Improve Citability and Impact

Bibliometrically Disciplined Peer Review: on Using Indicators in Research Evaluation

Evaluation of research uses peer review and bibliometrics, and the debate about their balance in research evaluation continues. Both approaches have supporters, and both approaches are criticized. In this paper, we describe an interesting case in which the use of bibliometrics in a panel-based evaluation of a mid-sized university was systematically tried out. The case suggests a useful way in which bibliometric indicators can be used to inform and improve peer review and panel-based evaluation. We call this ‘disciplined peer review’, and disciplined is used here in a constructive way: Bibliometrically disciplined peer review is more likely to avoid the subjectivity that often influences the outcomes of the peer and panel review-based evaluation.

The analysis of living systems can generate both knowledge and illusions.

Life relies on phenomena that range from changes in molecules that occur within nanoseconds to changes in populations that occur over millions of years. Researchers have developed a vast range of experimental techniques to analyze living systems, but a given technique usually only works over a limited range of length or time scales. Therefore, gaining a full understanding of a living system usually requires the integration of information obtained at multiple different scales by two or more techniques. This approach has undoubtedly led to a much better understanding of living systems but, equally, the staggering complexity of these systems, the sophistication and limitations of the techniques available in modern biology, and the need to use two or more techniques, can lead to persistent illusions of knowledge. Here, in an effort to make better use of the experimental techniques we have at our disposal, I propose a broad classification of techniques into six complementary approaches: perturbation, visualization, substitution, characterization, reconstitution, and simulation. Such a taxonomy might also help increase the reproducibility of inferences and improve peer review.

Skin in the Game: Personal Accountability and Journal Peer Review.

Two preregistered studies explored the likelihood paper reviewers would request clarification from authors regarding potential questionable research practices (QRPs). Study 1 participants were instructed to imagine reviewing a journal manuscript as either a coauthor or peer reviewer and rate the extent to which they would request clarification from the author when encountering potential QRPs. Participants reported greater likelihood of requesting clarification when assigned to the coauthor relative to the peer reviewer role. Study 2 participants were assigned to either an anonymous or open-review condition and rated the extent to which they would seek clarification from an author regarding potential QRPs. Men (but not women) in the open review condition reported greater likelihood of seeking clarification about potential QRPs than men in the blind review condition. Results provide tentative evidence that motivational factors influence the peer review process, and suggestions are made for improving peer review practices.

Medicolegal Sidebar: Peer Review of Professional Competence.

Medicolegal Sidebar: Peer Review of Professional Competence.

Universal Principled Review: A Community-Driven Method to Improve Peer Review

Despite being a staple of our science, the process of pre-publication peer review has few agreed-upon standards defining its goals or ideal execution. As a community of reviewers and authors, we assembled an evaluation format and associated specific standards for the process as we think it should be practiced. We propose that we apply, debate, and ultimately extend these to improve the transparency of our criticism and the speed with which quality data and ideas become public.

A Study of the Practices and Responsibilities of Scholarly Peer Review in Rhetoric and Composition

This article presents findings of an interview study with twenty rhetoric and composition scholars. Findings focus on the responsibilities of reviewers, editors, and writers in scholarly peer review. The authors make several recommendations for improving peer review practices and call for a field-wide discussion of and research about the topic.

What does better peer review look like? Underlying principles and recommendations for better practice

We conducted a literature review of best practice in peer review. Following this research, we identified five principles for better peer review: Content Integrity, Content Ethics, Fairness, Usefulness, and Timeliness. For each of these principles, we have developed a set of recommendations to improve peer review standards. In this article, we describe the role of peer review and how our five principles support that goal. This article is intended to continue the conversation about improving peer review standards and provide guidance to journal teams looking to improve their standards. It is accompanied by a detailed checklist, which could be used by journal teams to assess their current peer review standards.