Peer Review Practices Research Articles

Research Note—Making Peer Review Evidence-Based: It’s Time to Open the “Black Box”

ABSTRACT Peer review serves an essential role in the cultivation, validation, and dissemination of social work scholarship. Nevertheless, the current peer review system has been described as unreliable, biased, ineffective, and unaccountable, among numerous other issues. That said, peer review is still commonly viewed as the best possible system of knowledge governance, given the relevant alternatives. In this research note, I scrutinize this assumption. Although peer review can sometimes be effective, it is not therefore a rigorous or even dependable system. Indeed, the practice of peer review in social work is overwhelmingly closed and opaque, and assurances of its rigor are speculative at best. After highlighting common criticisms of traditional peer review, I articulate a research agenda on “open peer review”.

A Study on the Correlation between English Majors’ Critical Thinking Disposition and Performance Outcomes in Peer Review of Translation

This study took 61 English Majors MA as the research participants. Questionnaire data, students’ translation drafts, peer feedback and peer scoring were analyzed to reveal the level of students’ critical thinking disposition (CTD) and the correlation between their CTD and their performance outcomes in respect of peer feedback quality and peer scoring accuracy. The results show that English majors have a positive medium level CTD. There is a moderate positive correlation between analyticity and peer feedback quality, and a low positive correlation between inquisitiveness and peer feedback quality. A low positive correlation is also detected between truth-seeking and peer feedback quality. In addition, a moderate negative correlation is found between analyticity and peer scoring accuracy. A low negative correlation is detected between cognitive maturity and peer scoring accuracy, and there is a low negative correlation between self-confidence and peer scoring accuracy. In view of this, this study suggests that in future peer review practices, special instructions are required for the dimensions of CTD, in the hope of promoting the efficacy of peer review in learning.

Peer Review in Radiation Oncology: Where Does the Middle East, North Africa, and Türkiye Region Stand?

This study aims to assess the status of radiation oncology peer review procedures across the Middle East, North Africa, and Türkiye (MENAT) region. A cross-sectional electronic survey was conducted among radiotherapy centers in the MENAT region in March 2024. It assessed peer review practices, departmental demographics, perceived importance of peer review, and potential barriers. Data from 177 radiation oncology centers revealed varying peer review implementation across the MENAT region. Egypt had the highest participation (16.4%) among all responders. Most centers (31%) treated 500-1,000 cases annually. The majority (77.4%) implemented peer review, with varying levels between countries and across different centers. Advanced radiotherapy techniques significantly correlated with implementation of peer review (P < .05). Peer review meetings were mostly scheduled on a weekly basis (46%) and organized by radiation oncologists (84.7%). Target volume contouring (89%) and radiotherapy prescription (82%) were frequently peer-reviewed. Respondents with peer review at their institutions significantly valued peer review for education, adherence to guidelines, improving planning protocols, and reducing variation in practice institutions without peer review (P < .05). The most frequently reported barriers to peer review were having a high number of patients (56%) and shortage of time (54%). Peer review is essential for improving the quality of practice in radiation oncology. Despite some discrepancies, numerous obstacles, and challenges in implementation, it is instrumental in the improvement of patient care in most centers throughout the region. Raising awareness among radiation oncologists about the importance of peer review is paramount to lead to better outcomes.

Impact of peer review type and editor gender on women authorships in mathematics

Abstract Aiming to explain and help reduce measured gender gaps in mathematics publications—a discipline where single anonymized peer review practices and men editors are still the norm—we study changes in authorships straddling a 2015 switch in peer review type and editor gender in the American Mathematical Monthly, one of the oldest mathematics journals in the United States. Our results show a significant increase in women authorships after the journal’s switch to a woman editor and double anonymized reviews, an increase deemed exceptional relative to growth trends in comparable journals operating under field standards (men editors along with single anonymized reviews). We leverage literature, data-based observations and our own findings to argue for the likely effects of editor gender and peer review type separately and together as it concerns both women and newcomer authorships. Our study, the first to our knowledge on editorial and peer review changes in mathematics, contributes to a small body of literature on the impact of equity practices in peer review in science and mathematics. It also serves to introduce work, methods, and open problems on measuring and reducing the global gender gap in science and mathematics.

How can meta-research be used to evaluate and improve the quality of research in the field of traditional, complementary, and integrative medicine?

The field of traditional, complementary, and integrative medicine (TCIM) has garnered increasing attention due to its holistic approach to health and well-being. While the quantity of published research about TCIM has increased exponentially, critics have argued that the field faces challenges related to methodological rigour, reproducibility, and overall quality. This article proposes meta-research as one approach to evaluating and improving the quality of TCIM research. Meta-research, also known as research about research, can be defined as “the study of research itself: its methods, reporting, reproducibility, evaluation, and incentives”. By systematically evaluating methodological rigour, identifying biases, and promoting transparency, meta-research can enhance the reliability and credibility of TCIM research. Specific topics of interest that are discussed in this article include the following: 1) study design and research methodology, 2) reporting of research, 3) research ethics, integrity, and misconduct, 4) replicability and reproducibility, 5) peer review and journal editorial practices, 6) research funding: grants and awards, and 7) hiring, promotion, and tenure. For each topic, we provide case examples to illustrate meta-research applications in TCIM. We argue that meta-research initiatives can contribute to maintaining public trust, safeguarding research integrity, and advancing evidence based TCIM practice, while challenges include navigating methodological complexities, biases, and disparities in funding and academic recognition. Future directions involve tailored research methodologies, interdisciplinary collaboration, policy implications, and capacity building in meta-research.

How the readability of manuscript before journal submission advantages peer review process: Evidence from biomedical scientific publications

The practice of uploading preprints of scientific manuscripts prior to journal submission has become increasingly popular. As such, it is essential to understand the impact of the preprint version of a manuscript on the peer review process to facilitate the development of open peer review practices. In the current research, we analyze a dataset comprising 1,078 biomedical papers published in Nature Communications and eLife in 2019, along with their manuscript information posted on preprint servers and their peer review histories. Our investigation focuses on the relationship between the readability of manuscript before journal submission, as represented by preprints, and the sentimental features expressed by reviewers. Based on empirical analysis utilizing a linear regression model, it has been found that reviewers are inclined to express positive sentiments towards preprints characterized by technical language, as indicated by low value on the readability indices. Additional subgroup analysis suggests that this positive effect is more pronounced in papers with lower social and scientific impact, as indicated by online attention scores and scholarly views after publication, respectively. Overall, results of our analysis reveals that the utilization of technical language characterized by lower readability level in academic papers does not seem to hinder the peer review process in biomedical science, which has significant implications for the open peer review practice.

A proposed guide to reducing bias and improving assessments of decolonization in global health research

IntroductionThe movement to decolonize global health encompasses efforts to dismantle historically inequitable structures and processes in global health research, education, and practice. However, despite increasing literature on the decolonization of global health, gaps between action and knowledge exist in assessments of knowledge production. In this Perspective, we will outline potential biases in current approaches to assessing knowledge production and propose a systems-focused guide to improve the interrogation of knowledge production in this field.MethodsWe leverage the “Inner Setting” and “Outer Setting” domains of the Consolidated Framework for Implementation Research (CFIR), a well-established, commonly-used implementation science framework to critically assess the status quo of decolonization and to develop criteria to help guide decolonization efforts in academic contexts. We defined the Inner Setting as academic and research institutions leading and participating in global health research collaborations, and the Outer Setting as the funding, editorial, and peer review policies and practices that influence knowledge production in global health. Research institutions in the Inner organizational domain continually interact with the Outer policy domains. We categorize the levels at which decolonization may occur and where action should be focused as follows: (1) North–South, (2) South–South, (3) Local South, and (4) Local North. Using CFIR domains and the levels of action for decolonization, we propose a multi-level guide to improve on the standardization, granularity, and accuracy of decolonization assessments in global health research.Conclusion and expected impactThe proposed guide is informed by our global health research expertise and experiences as African scientists with extensive exposure in both global North and global South research contexts. We expect that the proposed guide will help to identify and address the biases identified and will lead to better knowledge-driven action in the process of decolonizing global health research.

How Long Does Contouring Really Take? Results of the Royal College of Radiologists Contouring Surveys

AimsThe success and safety of modern radiotherapy relies on accurate contouring. Understanding the time taken to complete radiotherapy contours is critical to informing workforce planning and, in the context of a workforce shortfall, advocating for investment in technology and multi-professional skills mix. We aimed to quantify the time taken to delineate target volumes for radical radiotherapy. Materials and methodsThe Royal College of Radiologists circulated two electronic surveys via email to all clinical oncology consultants in the UK. The individual case survey requested anonymous data regarding the next five patients contoured for radical radiotherapy. The second survey collected data on respondents' usual practice in radiotherapy contouring. ResultsThe median time to contour one radiotherapy case was 85 minutes (IQR = 50–131 minutes). Marked variability between and within tumour sites was evident: paediatric cancers took the most time (median = 210 minutes, IQR = 87.5 minutes), followed by head and neck and gynaecological cancers (median = 120 minutes, IQR = 71 and 72.5 minutes respectively). Breast cancer contouring required the least time (median = 43 minutes, IQR = 60 minutes). Radiotherapy technique, inclusion of nodes and 4D CT planning were associated with longer contouring times. A non-medical professional was involved in contouring in 65% of cases, but clinical oncology consultants were involved in target volume delineation in 90% of cases, and OARs in 74%. Peer review took place in 46% of cases with 56% of consultants reporting no time for peer review in their job plan. ConclusionContouring for radical radiotherapy is complex and time-consuming, and despite increasing involvement of non-medical professionals, clinical oncology consultants remain the primary practitioners. Peer review practice is variable and time is often a limiting factor. Many factors influence the time required for contouring, and departments should take these factors and the need for peer-review into account when developing job plans.

Peer Review in the Age of Generative AI

Rapid advances in artificial intelligence (AI), including recent generative forms, are significantly impacting our lives and work. A key aspect of our work as IS researchers is the publishing of research articles, for which peer review serves as the primary means of quality control. While there have been debates about whether and to what extent AI can replace researchers in various domains, including IS, we lack an in-depth understanding of how AI can impact the peer review process. Considering the high volume of submissions and limited reviewer resources, there is a pressing need to use AI to augment the review process. At the same time, advances in AI have been accompanied by concerns about biases introduced by AI tools and the ethics of using them, among other issues such as hallucinations. Thus, critical issues to understand are: how can AI augment and potentially automate the review process, what are the pitfalls in doing so, and what er the implications for IS research and peer review practice. I will offer my views on these issues in this opinion piece.

An Empirical Study on Roles and Their Information Correlations for Interdisciplinary Peer Reviewer

Peer review has always been known as the gatekeeper for the academic quality of papers. Traditional peer review is mostly carried out in a singular discipline, while peer review for interdisciplinary scientific research must go across the boundaries of multiple disciplines, and there is a real opposition between disciplines ‘differentiation’ and ‘de-differentiation.’ Due to this opposition, the practice of interdisciplinary peer review has more research value. Through an empirical study using Publons data, this article introduces the role theory from sociology as a theoretical support, studies role characteristic information of interdisciplinary peer reviewers, extracts their roles, measures the gaps between the roles according to their correlations, and proposes countermeasures to narrow the gaps through role construction to improve the quality of peer review in general. Specifically, this study adopts the empirical analysis method to obtain the behavioural characteristics data of the reviewers in the top 1 per cent in Cross-Field of the Global Peer Review Awards in 2018 and 2019 from the Publons platform and uses the exploratory factor analysis method to extract the roles of the reviewers. Using structural equation modelling to fit the role information association model, the authors compare and analyse the data models of 2018 and 2019 by using the comparative analysis method. The results found are as follows: The ‘researcher’ role, ‘reviewer’ role, and ‘editor’ role are all positively correlated, and the correlations are getting higher. It indicates that the correlations between the three are getting stronger, which will make transitioning between roles less difficult.

Need for Open-Science Policies

Publishing, sharing, and disseminating their research results are among scientists’ most important objectives. This process, which consists of stages full of intensive labor and care, is very laborious and may not always result in a positive outcome. A review of the scientific literature indicates that some non-academic authors publish more than academics. On the other hand, the presence of a widespread reproducibility problem even in highly cited publications negatively affects the reliability of scientific findings. Open-science policies and public peer review practices are needed to overcome these problems.

In the eye of beholder? The notions of quality in the humanities

Abstract In this article we attempt to reconstruct the tacit and implicit notions of quality in the humanities. This reconstruction is based on a series of semi-structured qualitative interviews with 33 humanities scholars. Applying Max Weber’s theory of authority, we argue the quality notions have two different sources—external and internal. External sources correspond to the Weber’s types of authority: traditional authorities (academic tradition, professors, PhD advisors), rational-legal authorities (research administrators, policy makers) and charismatic authorities (‘the great minds’, ‘the founding fathers’ in a given academic field). Internal sources providing the quality notion do not fit into Weberian classification. These sources are based on the personal experience of a humanities researcher’s evaluation practices, which cannot be reduced to either type of authority above. Combining the interview data and Max Weber’s theory of authority, we try to demonstrate the existence of four different and sometimes incompatible notions of quality in the humanities: administrative; individual; semi-administrative, semi-individual; moderate individual. These notions are interpreted as ideal types, which serve as a regulative ideas rather than objective representations of research evaluation reality. The manuscript is important to Research Evaluation for the following reasons: first, it reconstructs the different types of notions of quality, which are crucial in better understanding peer review and other qualitative research evaluation practices; second, it provides better understanding of individual evaluator’s premises; third, it provides opportunity to have a glimpse beyond dominant administrative quality notions and criteria as usually the perspectives of the humanities researchers are neglected.

Actitudes de los editores de revistas académicas brasileñas hacia la revisión por pares abierta (open peer review). Una encuesta

Open review is one of the components of open science that scientific journals are incorporating in their editorial processes. In contrast to open access to literature and research data, open review still raises many doubts and concerns among the involved parties (editors, reviewers and authors). This article aims to analyze the perception of editors of Brazilian academic journals on open peer review. To achieve this, a questionnaire was sent to 3,208 editors, out of which 351 responded to the full set of 42 questions. The editors expressed satisfaction with the current model of scientific communication, the double-blind model, and disagreed with any type of identification of reviewers. However, they perceived an advantage in open review, as it allows for mutual interaction between authors and reviewers with the goal of improving the quality of content. As barriers, they pointed out conflicts of interest and rivalries that open review might generate and the difficulty in finding reviewers willing to accept this review model. The overall conclusion points to a conservative profile among editors regarding the introduction of open peer review practices.

Evaluation of Peer Review Practice in Radiation Oncology Treatment Planning: Single Institution Experience

Pretreatment Peer review meeting at our radiation oncology department is a weekly meeting, where the initial radiation treatment (RT) planning process for every patient is discussed including: Computed Tomography (CT) simulation and treatment planning. In this study, we are aiming to evaluate our peer review process and to report the compliance to the suggested modifications. All RT plans for all patients indicated for radiotherapy accepted by the primary radiation oncology team then presented in our weekly peer review meeting for final approval. A minimum of two radiation oncologists reviewed each plan and their suggestions documented then suggested modifications are classified as changes in: field arrangement, clinical target volume (CTV), dose and fractionation. Compliance to suggested modification also recorded as: applied and not applied with or without documented justification. Atotal of 1779 radiation treatment plans were discussed in our regular weekly peer review meetings for a period of 3 years. 67.5% of them were treated with radical intent while 32.5% treated with palliative intent. The most common technique used was three-dimensional conformal (3DCRT) followed by intensity modulated RT (IMRT) (70.6% and 29.4%, respectively). The most frequently reviewed sites were: breast (38%), Gastrointestinal (11.9%) and head and neck (9.3%).1592 plans (89.48%) were labeled as: approved, 162 (9.1%) labeled as: modified and only 25 (1.4%) plans labeled as: hold. The frequency of modifications was significantly greater in breast (39.5%) than Gastrointestinal (17.9%) followed by gynecological cancers (14.2%) (P value <0.001). Modifications were more predominant with radical plans (64.2%) than palliative ones (35.8%) (P value <0.001) and statistically greater using 3DCRT (64.2%) than IMRT (35.8%) (P value <0.001). CTV recorded as the most frequent variable suggested for modifications (34.6%), followed by dose and fractionation (24.7%) (P value = 0.035). 72.2% of recommended modifications were applied by the consultant of primary team and 22.2% of non-applied modifications were justified and documented. There was 100% compliance rate to all plans labeled as hold. The practiceof peer review in radiation oncology specialty is very challenging. Its role in quality assurance of the radiotherapy process is highly accepted by radiation oncologists at our radiation oncology department. It is reflected in their high compliance rate to the suggested modifications. We recommend its implementation routinely at any radiation oncology department to enhance the quality of RT plans.

The academic status of reviewers predicts their language use

Peer review plays an essential role in scientific research, but the influence of reviewers' academic status is often overlooked during this process. By accessing peer review reports, in this study we empirically investigate this effect. Specifically, we analyzed 2,580 peer review histories from eLife submissions between 2016 and 2021 to examine the relationship between reviewers’ academic status and their language usage in the first round of peer review. We focused on two types of language features: emotional features (e.g., positivity and subjectivity) and linguistic features (e.g., number of long words and complex words). Our findings revealed no significant reviewer bias of academic status, such that the reviewers’ comments with emotional features were not significantly associated with reviewers’ awareness of being more prestigious than the last corresponding author of the manuscripts. More accomplished reviewers, however, were more likely to use longer and more complex words. Additionally, the results of linguistic features remained robust in the group where the last author served as the last corresponding author. Overall, our findings suggest that the quality of peer review remains the primary consideration for reviewers when evaluating submissions. These results have significant implications for open peer review practices and the fair assessment of the peer review process.

The importance of a good review(er) for educational technology research

The process of peer review has been central to academic publishing in educational technology for at least 50 years. In this editorial we discuss what makes a good review as well as a good reviewer for AJET. This includes an overview of the peer review process and the identification of key features of a good review. We discuss the selection and appointment of reviewers with reference specifically to AJET, and how decisions are made when assigning reviewers to articles. Current challenges facing peer review both broadly in academia and specifically in our field involve the intersection of increasing demand for reviews (due to an expansion of the number of journals), limited opportunities for professional development, and decreasing time available for service to the academy and community for researchers. We conclude with a discussion of the future of peer review practices and how these relate to future directions for AJET.

Comment on the article: “Peer review practices in academic medicine: how the example of orthopaedic surgery may help shift the paradigm?”

Comment on the article: “Peer review practices in academic medicine: how the example of orthopaedic surgery may help shift the paradigm?”

Peer review practices in academic medicine: how the example of orthopaedic surgery may help shift the paradigm?

PurposeTo establish the current peer-reviewed practices in the discipline of orthopaedic surgery and correlate these to the journal’s impact factor. Unfortunately, this is not receiving much attention and a critical literature gap in various disciplines; thus, determining the current practices in the discipline of orthopaedic surgery could provide valid insight that may be potentially applicable to other academic medicine disciplines as well.MethodsOrthopaedic surgery journals belonging to the Journal Citation Reports were queried, and the following was extracted: impact factor (IF) and blinding practices: single (SBPR), double (DBPR), triple (TBPR), quadruple (QBPR), and open (OPR) blinding review process and possibility of author-suggested reviewer (ASR) and non-preferred reviewer (NPR) options.ResultsOf the 82 journals, four were excluded as they allowed submission by invitation only. In the remaining, blinding was as follows: SBPR nine (11.5%), DBPR 52 (66.7%), TBPR two (2.6%), QBPR zero (0%), and OPR three (3.8%), and in 12 (15.4%), this was unclear. ASR and NPR options were offered by 34 (43.6%) and 27 (34.6%) journals respectively, whereas ASR was mandatory in eight (10.2%). No correlation between IF and any other parameter was found.ConclusionThe rules of the “game” are unclear/not disclosed in a significant number of cases, and the SBPR system, along with the ASR (mandatory sometimes) and NPR, is still extensively used with questionable integrity and fairness. Several recommendations are provided to mitigate potentially compromising practices, along with future directions to address the scarcity of research in this critical aspect of science.

An overview of innovations in the external peer review of journal manuscripts.

Background: There are currently numerous innovations in peer review and quality assurance in scholarly publishing. The Research on Research Institute conducted a programme of co-produced projects investigating these innovations. This literature review was part of one such project 'Experiments in peer review' which created an inventory and framework of peer review innovations. The aim of this literature review was to aid the development of the inventory by identifying innovations in the external peer review of journal manuscripts reported in the scholarly literature and by providing a summary of the different approaches. This did not include interventions in editorial processes. Methods: This review of reviews is based on data identified from Web of Science and Scopus limited from 2010 to 2021. A total of 291 records were screened, with six review articles chosen for the focus of the literature review. Items were selected that described approaches to innovating peer review or illustrated examples. Results: The overview of innovations are drawn from six review articles. The innovations are divided into three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review with sub-categories of results presented in tabular form and summarised. A summary of all innovations found is also presented. Conclusions: From a simple synthesis of the review authors' conclusions, three key messages are presented: observations on current practice; authors' views on the implications of innovations in peer review; and calls for action in peer review research and practice.

A national audit of performance standards for blood cultures in Aotearoa New Zealand: opportunities for improvement.

To audit key quality indicators for blood culture (BC) practices across Aotearoa New Zealand to facilitate national BC practice peer review and promote BC quality improvement interventions. Microbiology laboratories providing diagnostic services to district health board (DHB) hospitals were invited to participate. Practice was compared against published BC recommendations. Laboratories were required to submit data for BC positivity and contamination rates, BC bottle fill volume and the proportion of BC received as a single set. Laboratories serving 15 of the 20 DHBs participated in the audit. Nine DHBs (60%) demonstrated a positivity rate within the target range of 8% to 15%. Eight DHBs (53%) reported a contamination rate lower than the accepted 3%, but seven (47%) DHBs exceeded this target and two reported a contamination rate greater than 5%. Mean BC bottle fill volumes were generally greater than the target of 8mL, but this volume was not reached by three DHBs and a further three were unable to provide fill volume data. No DHB met the audit standard for single-set BCs representing <20%, and for six DHBs single-set BC comprised more than half of all samples. No DHB failed all audit targets. This audit demonstrates wide variation in BC performance across New Zealand. In most instances an inadequate volume of blood is being collected, lowering the chance of culturing a pathogen. A significant opportunity for improvement exists; clinical services and laboratories are encouraged to work together to implement targeted quality improvement processes to correct deficiencies in practice.