Is the Present Peer-review System Sustainable?

Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 % of submitted manuscripts remained unpublished. The mean acceptance rate was 21 % after external peer review and rejection rate 32 % after in-house review; 15 % publications resulted from the first submission, 47 % the second submission and 20 % the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review.

Purpose: To demonstrate the efficacy of a peer review system (PRS) for remote quality assurance in radiation oncology. The PRS is a Web‐based system which allows radiation oncologists to share anonymous treatment plans (imaging, structure sets, and dose distributions) with peer reviewers for their feedback. Methods: We have developed a DICOM‐RT compliant PRS that can store patient images and treatment planning data. The PRS infrastructure consisting of three components: Gateway, Web Server and Web Viewer, which is developed with current Web technology, utilizing .NET framework and Silverlight plug‐in for web interface. The patient data can be visualized through a Web browser and distributed across multiple locations by the local area network and the Internet. The submitter uses the Gateway to anomymize patient data, chooses a reviewer from the master list, adds annotations and messages and submits patient data for review via the PRS Web Server. Upon receiving the data from the submitter, the reviewer is notified via email and prompted to review the submitted treatment plan from the Web Viewer simply by clicking on the embedded URL. The PRS Web Server has built‐in functionality to keep track of reviewer comments communication between submitter and reviewers. Results: The system was evaluated at two facilities with different treatment planning systems (Varian Eclipse v.8.9 and Philips Pinnacle v.9.0). The CT images and related DICOM‐RT objects for several different patients from each facility were successfully submitted and reviewed using the prototype PRS. Conclusions: The PRS has been successfully pilot‐tested for remote peer‐review of treatment planning data. The PRS allows radiation oncology service providers access to treatment planning data from any location and an ability to review it in an interactive and collaborative manner. The broader implementation of the PRS in clinical settings will improve the quality and safety of radiation therapy.

Biology Direct, an online open access journal published by BioMed Central, is celebrating its 7th anniversary. Biology Direct started as an experiment, perhaps a daring one, on a new system of open peer review, under which the signed reviews and the author responses are published as an integral part of the final version of each article. The goals of the journal were set high: we strived to establish a new system of peer review that we hoped would avoid the all too obvious pitfalls of anonymous peer review. In addition, we expected that Biology Direct would generate productive scientific debate that would substantially add to the content of an article, in particular by alerting readers to potential problems with the reviewed work as well as additional relevant data and ideas [1, 2].

The classical view of peer review is that it is our primary process for assessing and judging whether research results should be published in a scholarly journal. However, the increased pressure to publish and technological developments are transforming peer review such that it is becoming a system that judges where work is published rather than whether the research is publishable (a ‘where rather than if’ process). Ecology is a field in which publication numbers puts a particular pressure on the review system. In this forum piece, I summarize the issues with the current publication system and discuss how technology is changing it, while suggesting solutions for important prior and ongoing issues with the peer review system. The view explored here is that technological developments (e.g. ease of creating journals, internet sites, storage, data generation, sharing of data and analytical code) will not eliminate peer review per se but will allow for a new set of parameters in which ethics and the optimal use of public funding will play a vital role in the evolution of the review process.SynthesisThe number of papers and journals in Ecology has increased dramatically in the past decade. I present a critical overview of our review system and proposes that pressure to publish and technological developments have transformed peer review into a system that decides “where rather than if” papers are publishable. While reviewing the current pressures and factors playing a vital role in the evolution of the review and publication systems, I propose potential solutions to deal with current and future challenges to the peer review and publication systems.

Patterns and practices in fisheries assessment peer review systems

As a vital learning activity in second language (L2) writing classrooms, peer feedback plays a crucial role in improving students' writing skills. However, student reviewers face challenges in providing impactful feedback on peers' essays. Low‐quality peer reviews emerge as a persistent problem, adversely affecting the learning effect of peer feedback. To enhance students' peer feedback provision, this study introduces EvaluMate, an AI‐supported peer review system, which incorporates a chatbot named Eva, designed to evaluate and provide feedback on student reviewers' comments on peers' essays. Forty‐four Chinese undergraduate students engaged with EvaluMate, utilising its features to generate feedback on peers' English argumentative essays. Chat log data capturing the students' interactions with the chatbot were collected, including the comments they wrote on peer essays and the feedback offered by the chatbot on their comments. The results indicate that the integration of AI supervision improved the quality of students' peer reviews. Students employed various strategies during their comment revision in response to AI feedback, such as introducing new points, adding details, and providing illustrative examples, which helped improve their comment quality. These findings shed light on the benefits of AI‐supported peer review systems in empowering students to provide more valuable feedback on peers' written work. Practitioner notes What is already known about this topic Scholars have extensively investigated diverse pedagogical strategies to enhance students' peer feedback provision skills in second language (L2) writing classrooms. Artificial intelligence (AI) technologies have been utilised to monitor and evaluate the peer feedback generated by student reviewers. AI‐enabled peer feedback evaluation tools have demonstrated the ability to provide valid assessments of student reviewers' peer feedback. What this paper adds In the context of L2 writing, there is a lack of bespoke AI‐enabled peer feedback evaluation tools. To address this gap, we have developed an AI‐supported peer review system, EvaluMate, which incorporates a large language model‐based chatbot named Eva. Eva is designed to provide feedback on L2 students' comments on their peers' writing. While previous studies have primarily focused on assessing the validity of AI‐enabled peer feedback evaluation tools, little is known about how students incorporate AI support into improving their peer review comments. To bridge this gap, our study examines not only whether using the system (EvaluMate) can enhance the quality of L2 students' peer review comments but also how students respond to Eva's feedback when revising their comments. Implications for practice and/or policy The development of the AI‐supported peer review system (EvaluMate) introduces an innovative pedagogical approach for L2 writing teachers to train and enhance their students' peer feedback provision skills. Integrating AI supervision into L2 students' peer feedback generation improves the quality of comments provided by student reviewers on their peers' writing. Students employ various strategies when revising their comments in response to Eva's feedback, and these strategies result in varying degrees of improvement in comment quality. L2 writing teachers can teach effective revision strategies to their students.

Editorial Policy

Challenges faced in the peer review system in open access journals

Editorial Policy

This article discusses the peer review system in the context of countries with small scientific communities. It is argued that the traditional peer review system is inadequate and probably dangerous for such countries. The reasoning is that in scientifically small countries personal relations and politics dominate the scene and objective impartial evaluation is not possible. The novel peer review systems adopted in the Netherlands and in South Africa are outlined and suggestions for improvements are made.SUMMARY AND SUGGESTIONS FOR IMPROVEMENTThe thrust of this article is the argument that the traditional peer‐review system is inadequate and probably dangerous for scientifically small countries. Although there is no hard empirical evidence to support the thesis that the peer‐review system is functioning better in scientifically large countries, a priori reasoning and circumstantial evidence indicate that the system is inefffective in countries with small scientific communities. The reasoning is that in small countries personal relations and politics dominate the scene and objective, impartial evaluation is not possible. As a result science policy can be misguided and societal loss can occur as valuable research effort is spent on ineffective peer‐evaluations.A variety of measures can be used to compensate for the ineffectiveness of the peer‐review system. The prime goal, however, is not only to increase the effectiveness of the system, but also to minimize societal costs due to misallocation of resources.The first step towards improvement of the system is to monitor it. Referees' reports and advice can be kept in files and properly analysed periodically. In this way biases concerning researchers, institutes, scientific subjects and approaches can be identified and appropriately remedied. Refereeing the referees could elimate intentional biases and improve the quality control system by anticipation. Referees knowing that their reports would be checked would take care to prepare proper evaluations.Increasing the number of referees is another way to increase reliability. Scientists from other disciplines can and should be incorporated in the evaluation juries. This way relevance of research and cross‐fertilisation of ideas can be incorporated in the system, partly ‘disinterested’ referees will be included in the juries (i.e. referees and applicants would not compete for the same research funds) and the lack of scientists in scientifically small countries will be alleviated. Care should be taken, however, not to over‐burden the system with unnecessary evaluation effort. In requiring researchers with demonstrated ability to go through the process of making formal applications for funding every year, valuable research time is wasted in preparing and evaluating proposals which, after all, will go through due to the status of the applicant. Attempting to regulate the expansion or contraction of the forest and not of the trees may provide a better solution. Evaluating researchers and disciplines instead of individual projects and funding them accordingly may save valuable research time and increase research productivity and effectiveness.Finally, one of the most promising areas for improvement in the decision making process is the incorporation of quantitative techniques in the tool‐kit of evaluation criteria. Publication and citation analyses can reveal the past performance (which is the only indicator of success for future research) of individuals and research groups. In a sense, bibliometric anaiysis can be seen as an expansion in time and panel size of the conventional peer reviews. Instead of asking a limited number of referees about a scientist's work, we count the citations that the particular scientist received by the whole scientific community and over an extended time period. Provided that care is taken to exclude self‐citations, negative citations, compare the same with the same, citation analysis can identify the pecking order in the scientific community. As bibliometric measures correlate well with non‐bibliometric measures (Table 2) the use of such criteria can ameliorate personal biases, improve objective evaluations and reduce time spent on evaluation purposes considerably.

Tight budgets complicate Canadian health research reforms

Editorial Policy and Style Information

BackgroundPeer review is the major method used by biomedical journals for making the decision of publishing an article. This cross-sectional survey assesses views concerning the review system of biomedical journals among academics globally.MethodsA total of 28,009 biomedical academics from high-ranking universities listed by the 2009 Times Higher Education Quacquarelli Symonds (THE-QS) World University Rankings were contacted by email between March 2010 and August 2010. 1,340 completed an online survey which focused on their academic background, negative experiences and views on biomedical journal peer review and the results were compared among basic scientists, clinicians and clinician scientists.ResultsFewer than half of the respondents agreed that the peer review systems of biomedical journals were fair (48.4%), scientific (47.5%), or transparent (25.1%). Nevertheless, 58.2% of the respondents agreed that authors should remain anonymous and 64.4% agreed that reviewers should not be disclosed. Most, (67.7%) agreed to the establishment of an appeal system. The proportion of native English-speaking respondents who agreed that the “peer review system is fair” was significantly higher than for non-native respondents (p = 0.02). Similarly, the proportion of clinicians stating that the “peer review system is fair” was significantly higher than that for basic scientists and clinician-scientists (p = 0.004). For females, (β = −0.1, p = 0.03), the frequency of encountering personal attacks in reviewers’ comments (β = −0.1, p = 0.002) and the frequency of imposition of unnecessary references by reviewers (β = −0.06, p = 0.04) were independently and inversely associated with agreement that “the peer review system is fair”.ConclusionAcademics are divided on the issue of whether the biomedical journal peer review system is fair, scientific and transparent. A majority of academics agreed with the double-blind peer review and to the establishment of an appeal system. Female academics, experience of personal attacks and imposition of unnecessary references by reviewers were related to disagreement about fairness of the peer review system of biomedical journals.

Editorial Policy

Editorial Policy