Peer Review Outcomes Research Articles

Economics Peer-Review: Problems, Recent Developments, and Reform Proposals

This article contributes to the debate on reforming the peer-review process in the economics profession. It examines the current state of peer-review in economics, surveys the relevant literature, and identifies several problems and solutions. Problems to be discussed are referee overreach and excessive revisions, strategic refereeing and conflicts of interest, prestige bias and other discrimination, and the noisy outcome of peer-review. It recommends several solutions for reform. First, enforce referee guidelines that referee reports must explicitly separate their suggestions into essential and optional, with 3 essential maximum. Second, let authors award the best referee report. Third, adopt conflict of interest policies for referees and punish non-disclosure. Fourth, use double-blind refereeing. Fifth, make better use of prior reports from other journals. Sixth, pay referees for prompt reports. A discussion of the role of editors highlights additional issues that deserve a debate in the profession.

Peer Review Is Primary: Presentations, Publications, Promotions, and Practice.

Peer review is primarily thought of as the process used to determine whether manuscripts are published in medical or other academic journals. While a publication may be one outcome of peer review, this article shares a model of 4 Ps to remind faculty of some important additional applications of peer review. The 4 Ps are publication, presentation, promotion, and practice. The medical literature offers few reasons why faculty should get involved in peer review. In this article, we define peer review, illustrate the role of peer review in four important processes, describe how the volume of material to review has changed over time, and share how participation in these processes promotes career advancement. Understanding the peer review process and its benefits can encourage professionals to participate in peer review in any of the four Ps as they recognize the benefits to their discipline and their career.

Evaluation of the National Cancer Institute (NCI) Pathway to Independence Award (K99/R00) Program

The National Cancer Institute (NCI) K99/R00 award is intended to help postdoctoral scholars transition in a timely manner to research independence and to foster their development of an impactful cancer research program that is competitive for subsequent independent funding. Here we analyzed factors that impact peer review outcomes and evaluated whether NCI K99/R00 awardees have achieved the goals of the K99/R00 funding mechanism. Our analysis of the K99/R00 review criterion scores demonstrates that while all review criterion scores are positively correlated with the overall impact score, the Research Plan criterion is the strongest predictor of the overall impact score and funding outcomes. In addition, our analysis shows the NCI K99/R00 award facilitated the successful transition of postdoctoral scholars to research independence and enhanced the likelihood of K99/R00 awardees to secure subsequent R01-equivalent NIH grant support although not in an accelerated fashion as originally intended. An NCI K99/R00 award was not determined to be a prerequisite to obtain a faculty position, but for some awardees, it was an asset in that transition. Our results suggest that the NCI K99/R00 award is an important component for training and retention of the next generation of independent cancer researchers and to increasing the percentage of women and promoting the diversity of the cancer research workforce.

Enhancing quality assurance in radiotherapy for gynaecological cancers: implementation of an on-demand peer review process.

Ensuring high-quality radiotherapy requires peer-reviewing target volumes. The Royal College of Radiologists recommends peer review specifically for individual target volumes in cases of gynaecological cancers. This study presents the outcomes of implementing an on-demand peer review system for gynaecological cancers within our institute. The peer review process was planned for gynaecological cancer cases intended for curative radiotherapy. After junior clinical oncologists (COs) completed the segmentation, two senior COs specializing in gynaecological cancers conducted the peer review. All peer review outcomes were recorded prospectively. The audit process compliance, the proportion of patients requiring major and minor modifications in target volumes, the direction of changes, and the factors influencing these changes were reported. A total of 230 patients were eligible, and out of these, 204 (88.3%) patients underwent at least one peer review. Among the patients, 108 required major modifications in their target volumes. P-charts revealed a stabilization in the need for major modifications at the end of three months, indicating that 38.2% and 28% of patients still required major modifications for the nodal and primary CTV, respectively. Multivariable analysis demonstrated that major modifications were associated with the use of extended field radiotherapy and radical radiation in non-cervical primary cases. An on-demand peer review system was feasible and resulted in clinically meaningful, major modifications in the target volumes for 53% of patients. Gynaecological cancers require ongoing peer review to ensure quality of care in radiotherapy. A flexible on-demand system not only ensures that patient treatment start is not delayed but also has an important educational role for junior trainees.

Double‐blind peer review affects reviewer ratings and editor decisions at an ecology journal

Abstract There is substantial evidence that systemic biases influence the scholarly peer review process. Many scholars have advocated for double‐blind peer review (also known as double‐anonymous review) to reduce these biases. However, the effectiveness of double‐blind peer review in eliminating biases is uncertain because few randomized trials have manipulated blinding of author identities for journal submissions and those that have are generally small or provide few insights on how it influences reviewer biases. In 2019, Functional Ecology began a large, randomized trial, using real manuscript submissions, to evaluate the various consequences of shifting to double‐blind peer review. Research papers submitted to the journal were randomly assigned to be reviewed with author identities blinded to reviewers (double‐blind review) or with authors identified to reviewers (single‐blind review). In this paper, we explore the effect of blinding on the outcomes of peer review, examining reviewer ratings and editorial decisions, and ask whether author gender and/or location mediate the effects of review type. Double‐blind review reduced the average success of manuscripts in peer review; papers reviewed with author identities blinded received on average lower ratings from reviewers and were less likely to be invited for revision or resubmission. However, the effect of review treatment varied with the author's location. Papers with first authors residing in countries with a higher human development index (HDI) and/or higher average English proficiency fared much better than those from countries with a lower HDI and lower English proficiency, but only when author identities were known to reviewers; outcomes were similar between demographic groups when author identities were not known to reviewers. Blinding author identities had no effect on gender differences in reviewer ratings or editor decisions. Our data provide strong evidence that authors from higher income and/or English‐speaking countries receive significant benefits (a large positive bias) to being identified to reviewers during the peer review process and that anonymizing author‐identities (e.g. double‐blind review) reduces this bias, making the peer review process more equitable. We suggest that offering optional blinding of author identities, as some journals allow, is unlikely to substantially reduce the biases that exist because authors from higher‐income and English‐speaking countries are the least likely to choose to be reviewed with their identity anonymized. Read the free Plain Language Summary for this article on the Journal blog.

Improving Communication of Peer-Review Conference Outcomes: A Practical Experience

The aim of this work was to describe the design and implementation of a more robust workflow for communicating outcomes from a peer-review chart rounds conference. We also provide information regarding cycle times, plan revisions, and other key metrics that we have observed since initial implementation. A multidisciplinary team of stakeholders including physicians, physicists, and dosimetrists developed a revised peer-review workflow that addressed key needs to improve the prior process. Consensus terminology was developed to reduce ambiguity regarding the priority of peer-review outcomes and to clarify expectations of the treating physician in response to peer-review outcomes. A custom workflow software tool was developed to facilitate both upstream and downstream processes from the chart rounds conference. The peer-review outcomes of the chart rounds conference and resulting plan changes for the first 18 months of implementation were summarized. In the first 18 months after implementation of the revised processes, 2294 plans were reviewed, and feedback priority levels assigned. Across all cases with feedback, the median time for the treating attending physician to acknowledge conference comments was 1 day and was within 7 calendar days for 89.1% of cases. Conference feedback was acknowledged within 1 day for 74 of 115 (64.3%) cases with level 2 comments and for 18 of 21 (85.7%) cases with level 3 comments (P=.054). Contours were modified in 13 of 116 (11%) cases receiving level 2 feedback and 10 of 21 (48%) cases receiving level 3 feedback (P < .001). The treatment plan was revised in 18 of 116 (16%) cases receiving level 2 feedback and 13 of 21 (61%) cases receiving level 3 feedback (P < .001). We successfully implemented a workflow to improve upstream and downstream processes for a chart rounds conference. Standardizing how peer-review outcomes were communicated and recording physician responses allow for improved ability to monitor conference activities.

Relations Between Participation in Peer Review and Learning Outcomes in Online Learning

Relations Between Participation in Peer Review and Learning Outcomes in Online Learning

AFTER REVIEW AND WAYS TO GET INVOLVED

Abstract After the review meeting, the SRO prepares the Summary Statement, which is the primary outcome of peer review. Program staff and Advisory Councils at the Institutes use this official document to make decisions about which applications to fund. How can you best prepare for the next steps for your application? Understand the qualities of a competitive NIH application, talk to Program staff, and take advantage of the opportunity to become a peer reviewer yourself. From the field, apply the tips shared by applicants and reviewers about what they do and what they look for in a well-written application. Receiving the summary statement can be daunting and discouraging; but by de-mystifying the process and getting involved, you can help position your application towards maximum success.

Multifactor Citation Analysis over Five Years: A Case Study of SIGMETRICS Papers

Performance evaluation is a broad discipline within computer science, combining deep technical work in experimentation, simulation, and modeling. The field’s subjects encompass all aspects of computer systems, including computer architecture, networking, energy efficiency, and machine learning. This wide methodological and topical focus can make it difficult to discern what attracts the community’s attention and how this attention evolves over time. As a first attempt to quantify and qualify this attention, using the proxy metric of paper citations, this study looks at the premier conference in the field, SIGMETRICS. We analyze citation frequencies at monthly intervals over a five-year period and examine possible associations with myriad other factors, such as time since publication, comparable conferences, peer review, self-citations, author demographics, and textual properties of the papers. We found that in several ways, SIGMETRICS is distinctive not only in its scope, but also in its citation phenomena: papers generally exhibit a strongly linear rate of citation growth over time, few if any uncited papers, a large gamut of topics of interest, and a possible disconnect between peer-review outcomes and eventual citations. The two most-cited papers in the dataset also exhibit larger author teams, higher than typical self-citations, and distinctive citation growth curves. These two papers, sharing some coauthors and a research focus, could either signal the area where SIGMETRICS had the most research impact, or they could represent outliers; their omission from the analysis reduces some of the otherwise distinctive observed metrics to nonsignificant levels.

Peer review quality assurance in stereotactic body radiotherapy planning: the impact of case volume

Abstract Purpose: Peer review is an essential quality assurance component of radiation therapy planning. A growing body of literature has demonstrated substantial rates of suggested plan changes resulting from peer review. There remains a paucity of data on the impact of peer review rounds for stereotactic body radiation therapy (SBRT). We therefore aim to evaluate the outcomes of peer review in this specific patient cohort. Methods and materials: We conducted a retrospective review of all SBRT cases that underwent peer review from July 2015 to June 2018 at a single institution. Weekly peer review rounds are grouped according to cancer subsite and attended by radiation oncologists, medical physicists and medical radiation technologists. We prospectively compiled ‘learning moments’, defined as cases with suggested changes or where an educational discussion occurred beyond routine management, and critical errors, defined as errors which could alter clinical outcomes, recorded prospectively during peer review. Plan changes implemented after peer review were documented. Results: Nine hundred thirty-four SBRT cases were included. The most common treatment sites were lung (518, 55%), liver (196, 21%) and spine (119, 13%). Learning moments were identified in 161 cases (17%) and translated into plan changes in 28 cases (3%). Two critical errors (0.2%) were identified: an inadequate planning target volume margin and an incorrect image set used for contouring. There was a statistically significantly higher rate of learning moments for lower-volume SBRT sites (defined as ≤30 cases/year) versus higher-volume SBRT sites (29% vs 16%, respectively; p = 0.001). Conclusions: Peer review for SBRT cases revealed a low rate of critical errors, but did result in implemented plan changes in 3% of cases, and either educational discussion or suggestions of plan changes in 17% of cases. All SBRT sites appear to benefit from peer review, though lower-volume sites may require particular attention.

Is There a Relationship Between Facility Peer Review Findings and Quality in the Veterans Health Administration?

The aim of clinical peer review (PR) is to improve facility health care quality. However, prior authors have shown that PR may be biased, have rater reliability concerns, or be used for punitive reasons. It is important to determine whether facility PR processes are related to objective facility quality of care. We collected proportion of PR findings that "most experienced and competent clinicians may have managed the case differently" or "most experienced and competent clinicians would have managed the case differently" as an objective measure of facility PR processes and outcomes. We correlated these with facility quality metrics for 2019. PR findings were not associated with facility quality metrics but were strongly associated with previous year findings. This study describes a potentially new source of bias in PR and demonstrates that objective facility outcomes are not related to individual PR findings.

What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports

• We used machine learning and qualitative text analysis to study grant peer review. • Cluster analysis revealed that content of review reports matched the predefined evaluation criteria. • Outcome of grant peer review is more influenced by proposals' weaknesses than strengths. • Features of review reports were consistent among disciplinary evaluation panels. The evaluation of grant proposals is an essential aspect of competitive research funding. Funding bodies and agencies rely in many instances on external peer reviewers for grant assessment. Most of the research available is about quantitative aspects of this assessment, and there is little evidence from qualitative studies. We used a combination of machine learning and qualitative analysis methods to analyse the reviewers' comments in evaluation reports from 3667 grant applications to the Initial Training Networks (ITN) of the Marie Curie Actions under the Seventh Framework Programme (FP7). Our results show that the reviewers' comments for each evaluation criterion were aligned with the Action's prespecified criteria and that the evaluation outcome was more influenced by the proposals’ weaknesses than by their strengths.

Peer reviewers\u2019 dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

Independent evaluations of grant applications by subject experts are an important part of the peer-review system. However, little is known about the real-time experiences of peer reviewers or experts who perform reviews of a grant application independently. This study sought to gain insight into this stage of the grant evaluation process by observing how experts conduct an independent review in near real time. Using the think aloud approach and Critical Decision Method of interviewing, in-depth interviews were conducted with 16 peer reviewers from a range of roles and disciplines within the medical humanities and social sciences. Participants were asked to think aloud while reviewing applications to different grant schemes from a single prestigious funder. The analysis shows reviewers encountered five dilemmas during the evaluation process. These dilemmas were related to whether or not one should (1) accept an invitation to review, (2) rely exclusively on the information presented in the application, (3) pay attention to institutional prestige, (4) offer comments about aspects that are not directly related to academics’ area of expertise, and (5) to take risks and overlook shortcomings rather than err on the side of caution. In order to decide on the appropriate course of action, reviewers often engaged in a series of deliberations and trade-offs—varying in length and complexity. However, their interpretation of what was ‘right’ was influenced by their values, preferences and experiences, but also by relevant norms and their understanding of the funder’s guidelines and priorities. As a result, the way reviewers approached the identified dilemmas was idiosyncratic and sometimes diametrically opposed to other reviewers’ views, which could lead to variation in peer-review outcomes. The dilemmas we have uncovered suggest that peer reviewers engage in thoughtful considerations during the peer-review process. We should, therefore, be wary of reducing the absence of consensus as resulting from biased, instinctive thinking. Rather, these findings highlight the diversity of values, priorities and habits and ways of working each reviewer brings to the fore when reviewing the applicants and their project proposals and call for further reflection on, and study of, this “invisible work” to better understand and continue to improve the peer-reviewing process.

Journal and disciplinary variations in academic open peer review anonymity, outcomes, and length

Understanding more about variations in peer review is essential to ensure that editors and reviewers harness it effectively in existing and new formats, including for mega-journals and when published online. This article analyzes open reviews from the MDPI suite of journals to identify commonalities and differences from a simplistic quantitative perspective, focusing on reviewer anonymity, review length, and review outcomes. The sample contained 45,385 first round open reviews from published standard journal articles in 288 MDPI journals classified into one or more Scopus disciplinary areas (Health Sciences; Life Sciences; Physical Sciences; Social Sciences). The eight main findings include substantial differences between journals and disciplines in review lengths, reviewer anonymity, review outcomes, and the use of attachments. In particular, Physical Sciences journal reviews tended to be stricter and were more likely to be anonymous. Life Sciences and Social Sciences reviews were the longest overall. Signed reviews tend to be 15% longer (perhaps to be more careful or polite) but gave similar decisions to anonymous reviews. Finally, reviews with major revision outcomes tended to be 68% longer than reviews with for minor revision outcomes, except in a few journals. In conclusion, signing reviews does not seem to threaten the validity of peer review outcomes and authors, editors, and reviewers of multidisciplinary articles should be aware of substantial field differences in what constitutes an appropriate review.

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. Redaction reduced, but did not eliminate, reviewers' ability to correctly guess features of identity. The primary, preregistered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not preregistered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen's d of 0.20-0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity. Funding was provided by the NIH.

Surgical research journals - Under review: An assessment of diversity among editorial boards and outcomes of peer review

BackgroundThe COVID-19 pandemic exposed racism as a public health crisis embedded in structural processes. Editors of surgical research journals pledged their commitment to improve structure and process through increasing diversity in the peer review and editorial process; however, little benchmarking data are available. MethodsA survey of editorial board members from high impact surgical research journals captured self-identified demographics. Analysis of manuscript submissions from 2016 to 2020 compared acceptance for diversity, equity, and inclusion (DEI)-focused manuscripts to overall rates. Results25.6% of respondents were female, 2.9% Black, and 3.3% Hispanic. There was variation in the diversity among journals and in the proportion of DEI submissions they attract, but no clear correlation between DEI acceptance rates and board diversity. ConclusionsDiversity among board members reflects underrepresentation of minorities seen among surgeons nationally. Recruitment and retention of younger individuals, representing more diverse backgrounds, may be a strategy for change. DEI publication rates may benefit from calls for increasing DEI scholarship more so than changes to the peer review process.

Establishing Interrater Reliability Within the Advanced Practice Nurse Peer Review Process.

The Advanced Practice Nurse (APN) Council refined the APN peer review to an objective, data-driven process. The purpose of the study was to assess the interrater reliability of APN peer reviews using the APN Rubric based on Hamric, Spross & Hanson's Model of Advanced Practice Nursing. A quantitative single-site study with a convenience sample of 80 APN Portfolios. Analysis of six core competencies (direct clinical practice, leadership, consultation/collaboration, coaching/guiding, research, and ethical decision-making) within the APN Rubric demonstrated substantial and near perfect agreement levels in the APN peer review process. The application of APN core competencies within the peer review process demonstrated high consistency, thereby increasing the significance and objectivity of peer review outcomes.

Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome

AbstractWe analyzed a data set of scientific manuscripts that were submitted to various conferences in artificial intelligence. We performed a combination of semantic, lexical, and psycholinguistic analyses of the full text of the manuscripts and compared them with the outcome of the peer review process. We found that accepted manuscripts scored lower than rejected manuscripts on two indicators of readability, and that they also used more scientific and artificial intelligence jargon. We also found that accepted manuscripts were written with words that are less frequent, that are acquired at an older age, and that are more abstract than rejected manuscripts. The analysis of references included in the manuscripts revealed that the subset of accepted submissions were more likely to cite the same publications. This finding was echoed by pairwise comparisons of the word content of the manuscripts (i.e., an indicator of semantic similarity), which were more similar in the subset of accepted manuscripts. Finally, we predicted the peer review outcome of manuscripts with their word content, with words related to machine learning and neural networks positively related to acceptance, whereas words related to logic, symbolic processing, and knowledge-based systems negatively related to acceptance.

Associations of topic-specific peer review outcomes and institute and center award rates with funding disparities at the National Institutes of Health.

A previous report found an association of topic choice with race-based funding disparities among R01 applications submitted to the National Institutes of Health ('NIH') between 2011 and 2015. Applications submitted by African American or Black ('AAB') Principal Investigators ('PIs') skewed toward a small number of topics that were less likely to be funded (or 'awarded'). It was suggested that lower award rates may be related to topic-related biases of peer reviewers. However, the report did not account for differential funding ecologies among NIH Institutes and Centers ('ICs'). In a re-analysis, we find that 10% of 148 topics account for 50% of applications submitted by AAB PIs. These applications on 'AAB Preferred' topics were funded at lower rates, but peer review outcomes were similar. The lower rate of funding for these topics was primarily due to their assignment to ICs with lower award rates, not to peer-reviewer preferences.

Research Conducted in Women Was Deemed More Impactful but Less Publishable than the Same Research Conducted in Men.

Background: Female scientists, who are more likely than their male counterparts to study women and report findings by sex/gender, fare worse in the article peer review process. It is unknown whether the gender of research participants influences the recommendation to publish an article describing the study. Materials and Methods: Reviewers were randomly assigned to evaluate one of three versions of an article abstract describing a clinical study conducted in men, women, or individuals. Reviewers assessed the study's scientific rigor, its level of contribution to medical science, and whether they would recommend publishing the full article. Responses were analyzed with logistic regression controlling for reviewer background variables, including sex and experience level. Results: There was no significant difference in perceived research rigor by abstract condition; contribution to medical science was perceived to be greater for research conducted in women than men (odds ratio = 1.7; p = 0.030). Nevertheless, reviewers were almost twice as likely to recommend publication for research conducted in men than the same research conducted in women (predicted probability 0.606 vs. 0.322; p = 0.000). Conclusions: These results are consistent with abundant data from multiple sources showing a lower societal value placed on women than men. Because female investigators are more likely than male investigators to study women, our findings suggest a previously unrecognized bias that could contribute to gender asymmetries in the publication outcomes of peer review. This pro-male publication bias could be an additional barrier to leadership attainment for women in academic medicine and the advancement of women's health.