Peer Review Model Research Articles

From peer review to collaborative peer enquiry: Action research for school improvement and leadership development

School peer reviews are increasingly part of the evaluation and school improvement landscape for school leaders and teachers in a number of countries. This article describes the growth of peer review, particularly in England, and its emergence elsewhere (for example, Australia, across Europe and in Chile). While these approaches provide a useful form of professional and moral accountability, this article identifies ways in which they could go further to empower practitioners through the use of an enquiry approach, combining formal academic knowledge with practitioner knowledge and school-based data. The term collaborative peer enquiry (CPE) is suggested as a way to explore this potential. The article sets out a typology of action research as a form of professional learning (type 1), practical philosophy (type 2) or as a form of critical social science (type 3). Four examples are given of different peer review models, two of them CPE approaches, and these are analysed using the above typology. A distinction is made between some peer review models that mimic external inspections and err towards self-policing, and others that encourage open enquiry and learning. In particular, the CPE models show the potential as forms of type 2 and type 3 action research. The role of peer review and CPE in the accountability system, in leadership development, and challenges for these models are explored in the discussion.

Grade Language Heterogeneity in Simulation Models of Peer Review

Grade Language Heterogeneity in Simulation Models of Peer Review

Peer review: single-blind, double-blind, or all the way-blind?

A scholarly peer review is the process whereby referees scrutinize research work or a manuscript within their field of expertise and decide on its acceptability for publication in a journal or scientific proceeding. Ideally, peer review is impartial. Among the many models of peer review, the single blind is currently the most adopted model in scientific journals. The double-blind model has been claimed to decrease bias, despite some difficulty in implementation.

Transferring German Evaluation Policy to China: A Prospective Evaluation of Peer Review in TVET

Comparative research into technical and vocational education and training (TVET) and policy transfer focuses primarily on the system level. This study was designed to extend the focus of research work to policy transfer at the institutional level. This article presents a project that investigated the German concept of peer review in TVET as a potential solution for Chinese TVET schools. The survey investigated the extent to which there is potential to transfer the evaluation concept of peer review in TVET to China. The study used the concept of prospective evaluation. The findings showed that not every feature of the peer review model could be transferred to the Chinese context. The peer review concept needs to be adapted. The supporting and inhibiting factors derived from this study enable us to draw lessons about how to support policy transfer at the institutional level.

Constructive and collegial peer-review as a necessary precursor to data-driven environmental policy

As academic output increases, editors and reviewers are inundated with manuscripts, a phenomenon that has resulted in extended wait times, and, in many cases, suboptimal reviews. This is particularly an issue if data-driven environmental policy is the goal, as peer-review is an important step in this process. Numerous systemic fixes have been suggested to improve peer-review; however, we question the value of system-wide change if peer-reviewers continue to engage in behaviour that is not collegial, or in-keeping with the sentiment behind rigorous peer-review. We provide nine best practice statements that compliment proposed system-wide changes with the goal to improve the process of peer-review by helping to develop reviewer codes of conduct. These statements focus only upon reviewer conduct, not issues such as conflicts of interest or confidentiality. It is our hope that data driven improvements to peer-review models, and reviewer codes of conduct will make the peer-review process more efficient and collegial. Such improvements are necessary to promote novel and innovative academic research, as well as to enhance the diversity of voices present in the academic literature.

Xing Xu

Xing Xu

A scoping review of simulation models of peer review

Peer review is a process used in the selection of manuscripts for journal publication and proposals for research grant funding. Though widely used, peer review is not without flaws and critics. Performing large-scale experiments to evaluate and test correctives and alternatives is difficult, if not impossible. Thus, many researchers have turned to simulation studies to overcome these difficulties. In the last 10 years this field of research has grown significantly but with only limited attempts to integrate disparate models or build on previous work. Thus, the resulting body of literature consists of a large variety of models, hinging on incompatible assumptions, which have not been compared, and whose predictions have rarely been empirically tested. This scoping review is an attempt to understand the current state of simulation studies of peer review. Based on 46 articles identified through literature searching, we develop a proposed taxonomy of model features that include model type (e.g. formal models vs. ABMs or other) and the type of modeled peer review system (e.g. peer review in grants vs. in journals or other). We classify the models by their features (including some core assumptions) to help distinguish between the modeling approaches. Finally, we summarize the models’ findings around six general themes: decision-making, matching submissions/reviewers, editorial strategies; reviewer behaviors, comparisons of alternative peer review systems, and the identification and addressing of biases. We conclude with some open challenges and promising avenues for future modeling work.

Library Publishers as Educators: Crafting Curriculum for Undergraduate Research Journals

INTRODUCTION Library publishing programs continue to play an increasingly important role in educating their constituents. In particular, library publishers that support undergraduate student journals often provide guidance to students on both mechanical and conceptual issues related to publishing. This article presents a case study for developing a one-credit-hour course to support an undergraduate student journal publication, the Indiana University Journal of Undergraduate Research (IUJUR), at Indiana University Bloomington. DESCRIPTION OF COURSE The course is offered every fall as a mechanism for onboarding about thirty new undergraduate editors. The course was developed and taught by a librarian and an undergraduate student in consultation with IU’s Office of the Vice Provost for Undergraduate Education. Course curriculum touches on topics that scholarly communication and information literacy librarians alike can adapt for a variety of educational contexts, including authentic activities for understanding peer review models and applying publishing innovations. ASSESSMENT The article details both the formative and summative assessment strategies the instructor utilized to gauge student understanding of key publishing concepts. The summative assessment utilizes pre- and post-tests and extends previous library literature to evaluate students’ actual understanding of publishing concepts in addition to their perceived understanding and confidence. LIMITATIONS AND NEXT STEPS The course curriculum will continue to grow and change in order to accommodate students’ misconceptions and interests.

Chasing Zero Harm in Radiation Oncology: Using Pre-treatment Peer Review.

Purpose: The Joint Commission has encouraged the healthcare industry to become “High Reliability Organizations” by “Chasing Zero Harm” in patient care. In radiation oncology, the time point of quality checks determines whether errors are prevented or only mitigated. Thus, to “chase zero” in radiation oncology, peer review has to be implemented prior to treatment initiation. A multidisciplinary group consensus peer review (GCPR) model is used pre-treatment at our institution and has been successful in our efforts to “chase zero harm” in patient care.Methods: With the GCPR model, policy-defined complex cases go through a treatment planning conference, which includes physicians, residents, physicists, and dosimetrists. Three major plan aspects are reviewed: target volumes, target and normal tissue dose coverage, and dose distributions. During the review, any team member can ask questions and afterwards a group consensus is taken regarding plan approval.Results: The GCPR model has been implemented through a commitment to peer review and creative conference scheduling. Automated analysis software is used to depict color-coded results for department approved target coverage and dose constraints. About 8% of plans required re-planning while about 23% required minor changes. The mean time for review of each plan was 8 min.Conclusions: Catching errors prior to treatment is the only way to “chase zero” in radiation oncology. Various types of errors may exist in treatment plans and our GCPR model succeeds in preventing many errors of all shapes and sizes in target definition, dose prescriptions, and treatment plans from ever reaching the patients.

Registered Reports, Replication, and the Canadian Journal of School Psychology: Improving the Evidence in Evidence-Based School Psychology

The Canadian Journal of School Psychology (CJSP) is offering scholars the opportunity to register research reports and make research protocols publicly available to promote replication, transparency, credibility, and utility for clinical practice. The purpose of this article is to outline the challenges regarding replication, reproducibility, and evidence-based practices, as well as describe the submission protocol and criteria for acceptance of registered reports. Advances and criticisms of the registered reports model are discussed. Although CJSP will accept submissions through the traditional peer-review model, registered reports and support of replication studies have the objective of promoting high-quality research to improve the research foundation for evidence-based practices in the profession of school psychology.

A conceptual peer review model for arXiv and other preprint databases

A global survey conducted by arXiv in 2016 showed that 58% of arXiv users thought arXiv should have a peer review system. The current opinion is that arXiv should adopt the Community Peer Review model. This paper evaluates and identifies two weak points of Community Peer Review and proposes a new peer review model – Self‐Organizing Peer Review. We propose a model in which automated methods of matching reviewers to articles and ranking both users and articles can be implemented. In addition, we suggest a strategic plan to increase recognition of articles in preprint databases within academic circles so that second generation preprint databases can achieve faster and cheaper publication.

Opening and Reusing Transparent Peer Reviews with Automatic Article Annotation

An increasing number of scientific publications are created in open and transparent peer review models: a submission is published first, and then reviewers are invited, or a submission is reviewed in a closed environment but then these reviews are published with the final article, or combinations of these. Reasons for open peer review include giving better credit to reviewers, and enabling readers to better appraise the quality of a publication. In most cases, the full, unstructured text of an open review is published next to the full, unstructured text of the article reviewed. This approach prevents human readers from getting a quick impression of the quality of parts of an article, and it does not easily support secondary exploitation, e.g., for scientometrics on reviews. While document formats have been proposed for publishing structured articles including reviews, integrated tool support for entire open peer review workflows resulting in such documents is still scarce. We present AR-Annotator, the Automatic Article and Review Annotator which employs a semantic information model of an article and its reviews, using semantic markup and unique identifiers for all entities of interest. The fine-grained article structure is not only exposed to authors and reviewers but also preserved in the published version. We publish articles and their reviews in a Linked Data representation and thus maximise their reusability by third party applications. We demonstrate this reusability by running quality-related queries against the structured representation of articles and their reviews.

Science peer review for the 21st century: Assessing scientific consensus for decision-making while managing conflict of interests, reviewer and process bias

Science peer review plays an important role in the advancement and acceptance of scientific information, particularly when used to support decision-making. A model for science peer review is proposed here using a large, multi-tiered case study to engage a broader segment of the scientific community to support decision making on science matters, and to incorporate many of the design advantages of the two common forms of peer review (journal peer review, science advisory panels). This peer review consisted of a two-tiered structure consisting of seven panels (five review panels in Tier 1, two review panels in Tier 2), which focused on safety data for a modified risk tobacco product (MRTP). Experts from all over the world were invited to apply to one or more positions on seven peer review panels. 66 peer reviewers were selected from available applicants using objective metrics of their expertise, and for some panels based upon a consideration of panel diversity with respect to demographic parameters (e.g., geographic region, sector of employment, years of experience). All peer reviewers participated anonymously in which a third-party auditor was used to provide independent verification of their expertise. Peer reviewers were provided electronic links to all review material which included access to publications, reports, omics data, and histopathology slides, with topic-specific panels focusing on topic-specific components of the review package. Peer reviews consisted either of single-round, or multi-round (e.g., modified Delphi) format. Peer reviewer responses to the charge questions were collected via an online survey system, and were assembled into a database. Responses in the database were subject to analyses to assess the degree of favorability (i.e., supportive of the review material), degree of consensus, reproducibility of replicate panels, hidden sources of bias, and outlier response patterns. Conclusions: By careful consideration of science peer review design elements we have shown that: 1) panel participation can be broadened to include scientists who would otherwise not participate; 2) panel diversity can be managed in an unbiased manner without adverse impacts to panel expertise; 3) results obtained from independent concurrent panels are shown to be reproducible; and 4) there are benefits of collecting input from expert panels via a structured format (i.e., survey) to support characterization of consensus, identification of hidden sources of bias, and identification of potential outlier participants.

Registered Reports: Genre Evolution and the Research Article

The research article is a staple genre in the economy of scientific research, and although research articles have received considerable treatment in genre scholarship, little attention has been given to the important development of Registered Reports. Registered Reports are an emerging, hybrid genre that proceeds through a two-stage model of peer review. This article charts the emergence of Registered Reports and explores how this new form intervenes in the evolution of the research article genre by replacing the central topoi of novelty with methodological rigor. Specifically, I investigate this discursive and publishing phenomenon by describing current conversations about challenges in replicating research studies, the rhetorical exigence those conversations create, and how Registered Reports respond to this exigence. Then, to better understand this emerging form, I present an empirical study of the genre itself by closely examining four articles published under the Registered Report model from the journal Royal Society Open Science and then investigating the genre hybridity by examining 32 protocols (Stage 1 Registered Reports) and 77 completed (Stage 2 Registered Reports) from a range of journals in the life and psychological sciences. Findings from this study suggest Registered Reports mark a notable intervention in the research article genre for life and psychological sciences, centering the reporting of science in serious methodological debates.

Peer Review of Scholarly Work.

Peer Review of Scholarly Work.

Proposing a Transactional Model of eHealth Literacy: Concept Analysis.

BackgroundElectronic health (eHealth) literacy was conceptualized in 2006 as the ability of internet users to locate, evaluate, and act upon web-based health information. Now, advances in eHealth technology have cultivated transactional opportunities for patients to access, share, and monitor health information. However, empirical evidence shows that existing models and measures of eHealth literacy have limited theoretical underpinnings that reflect the transactional capabilities of eHealth. This paper describes a conceptual model based on the Transactional Model of Communication (TMC), in which eHealth literacy is described as an intrapersonal skillset hypothesized as being dynamic; reciprocal; and shaped by social, relational, and cultural contexts.ObjectiveThe objective of our study was to systematically examine eHealth literacy definitions, models, and measures to propose a refined conceptual and operational definition based on the TMC.MethodsWalker and Avant’s concept analysis method was used to guide the systematic review of eHealth literacy definitions (n=10), rating scales (n=6), models (n=4), and peer-reviewed model applications (n=16). Subsequent cluster analyses showed salient themes across definitions. Dimensions, antecedents, and consequences reflected in models and measures were extracted and deductively analyzed based on codes consistent with the TMC.ResultsSystematic review evidence revealed incongruity between operational eHealth literacy included in definitions compared with literacies included within models and measures. Theoretical underpinnings of eHealth literacy also remain dismal. Despite the transactional capabilities of eHealth, the role of “communication” in eHealth literacy remains underdeveloped and does not account for physical and cognitive processing abilities necessary for multiway transactions.ConclusionsThe Transactional Model of eHealth Literacy and a corresponding definition are proposed. In this novel model, eHealth literacy comprises a hierarchical intrapersonal skillset that mediates the reciprocal effect of contextual factors (ie, user oriented and task oriented) on patient engagement in health care. More specifically, the intrapersonal skillset counteracts the negative effect of “noise” (or impediments) produced by social and relational contexts. Cutting across health and technology literacies, the intrapersonal skillset of eHealth literacy is operationalized through four literacies that correspond with discrete operative skills: (1) functional (ie, locate and understand); (2) communicative (ie, exchange); (3) critical (ie, evaluate); and (4) translational (ie, apply).

Open Peer Review for Evaluating Academic Legal Publications: The “Antidote” to an “Ill” Blind Peer Review?

Numerous peer-reviewed law journals, in particular Latin-American ones, stand out for their general use of blind methods to assess the substantive quality of academic articles. Nevertheless, this traditional peer review model is considered ill (i.e. poor-quality) across disciplines due to different technical and social issues. The cause seems to be the anonymity between the actors involved. Then, open peer review models have emerged as an antidote to combat the main symptoms that the blind model presents, namely, lack of transparency during the process and absence of reviewers’ accountability. However, thus far law journals have not adopted an open system yet. Moreover, neither theoretical nor empirical research has been conducted about its potential use. This article presents an in-depth analysis of the pros and cons of both the traditional and the open identities review models in Science, Technology, Engineering and Mathematics (STEM) disciplines – in which the latter has been broadly applied. This paper compares the experience of STEM disciplines in relation to both of the methods and use the insights from this comparison to examine the legal discipline. The author specifically argues that unmasking the identities of authors and evaluators may be a suitable way to counteract only some of the flaws that the blind model has entailed in the evaluation of academic legal publications, therefore a mixed method could be a fairer alternative for current blind-peer-reviewed law journals. The author concludes by providing recommendations for further research.

The state of the art in peer review.

Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly changing environments, within what is widely considered as the ‘open research’ or ‘open science’ movement. Here, we summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and communications infrastructure and suggest why uptake of new models of peer review appears to have been so low compared to what is often viewed as the ‘traditional’ method of peer review. Finally, we offer some insight into the potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture.

Challenges to open peer review

PurposeThe purpose of this paper is to assess what the challenges to open peer review (OPR) are, relative to traditional peer review (TPR).Design/methodology/approachBy examining select issues within peer review, more broadly, and challenges within TPR, the effectiveness of OPR is questioned.FindingsAlthough OPR brings an aspect of transparency, by partially eliminating biases, fear of reprisals and of professional blow-back, either by authors who may be criticized or by competitors, limits the expansion of this peer review model, or its adoption as an industry-wide standard.Originality/valueOpen Science 2.0 boasts of greater openness and transparency and OPR is touted as one tool to achieve this. However, that potential is limited. This limitation needs to be recognized.

Uptake and outcome of manuscripts in Nature journals by review model and author characteristics

BackgroundDouble-blind peer review has been proposed as a possible solution to avoid implicit referee bias in academic publishing. The aims of this study are to analyse the demographics of corresponding authors choosing double-blind peer review and to identify differences in the editorial outcome of manuscripts depending on their review model.MethodsData includes 128,454 manuscripts received between March 2015 and February 2017 by 25 Nature-branded journals. We investigated the uptake of double-blind review in relation to journal tier, as well as gender, country, and institutional prestige of the corresponding author. We then studied the manuscripts’ editorial outcome in relation to review model and author’s characteristics. The gender (male, female, or NA) of the corresponding authors was determined from their first name using a third-party service (Gender API). The prestige of the corresponding author’s institutions was measured from the data of the Global Research Identifier Database (GRID) by dividing institutions in three prestige groups with reference to the 2016 Times Higher Education (THE) ranking. We employed descriptive statistics for data exploration, and we tested our hypotheses using Pearson’s chi-square and binomial tests. We also performed logistic regression modelling with author update, out-to-review, and acceptance as response, and journal tier, author gender, author country, and institution as predictors.ResultsAuthor uptake for double-blind submissions was 12% (12,631 out of 106,373). We found a small but significant association between journal tier and review type (p value < 0.001, Cramer’s V = 0.054, df = 2). We had gender information for 50,533 corresponding authors and found no statistically significant difference in the distribution of peer review model between males and females (p value = 0.6179). We had 58,920 records with normalised institutions and a THE rank, and we found that corresponding authors from the less prestigious institutions are more likely to choose double-blind review (p value < 0.001, df = 2, Cramer’s V = 0.106). In the ten countries with the highest number of submissions, we found a large significant association between country and review type (p value < 0.001, df = 10, Cramer’s V = 0.189). The outcome both at first decision and post review is significantly more negative (i.e. a higher likelihood for rejection) for double-blind than single-blind papers (p value < 0.001, df = 1, Cramer’s V = 0.112 for first decision; p value < 0.001; df = 1, Cramer’s V = 0.082 for post-review decision).ConclusionsThe proportion of authors that choose double-blind review is higher when they submit to more prestigious journals, they are affiliated with less prestigious institutions, or they are from specific countries; the double-blind option is also linked to less successful editorial outcomes.