Grant Peer Review Research Articles

Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability.

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement, and implement the computations in the R package IRR2FPR.

Balancing best practice and reality in behavioral intervention development: A survey of principal investigators funded by the National Institutes of Health.

Preliminary studies play a prominent role in the development of large-scale behavioral interventions. Though recommendations exist to guide the execution and interpretation of preliminary studies, these assume optimal scenarios which may clash with realities faced by researchers. The purpose of this study was to explore how principal investigators (PIs) balance expectations when conducting preliminary studies. We surveyed PIs funded by the National Institutes of Health to conduct preliminary behavioral interventions between 2000 and 2020. Four hundred thirty-one PIs (19% response rate) completed the survey (November 2021 to January 2022, 72% female, mean 21 years post-terminal degree). Most PIs were aware of translational models and believed preliminary studies should precede larger trials but also believed a single preliminary study provided sufficient evidence to scale. When asked about the relative importance of preliminary efficacy (i.e. changes in outcomes) and feasibility (i.e. recruitment, acceptance/adherence) responses varied. Preliminary studies were perceived as necessary to successfully compete for research funding, but among PIs who had peer-reviewed federal-level grants applications (n = 343 [80%]), responses varied about what should be presented to secure funding. Confusion surrounding the definition of a successful, informative preliminary study poses a significant challenge when developing behavior interventions. This may be due to a mismatch between expectations surrounding preliminary studies and the realities of the research enterprise in which they are conducted. To improve the quality of preliminary studies and advance the field of behavioral interventions, additional funding opportunities, more transparent criteria in grant reviews, and additional training for grant reviewers are suggested.

Applying Findable, Accessible, Interoperable, and Reusable Principles to Biospecimens and Biobanks.

The importance of stimulating greater sharing of data for use and reuse in health research is widely recognized. To this end, the findable, accessible, interoperable, and reusable (FAIR) principles for data have been developed and widely accepted in the research community. Research biospecimens are a resource that leads to much of this health research data but are also a form of data. Therefore, the FAIR principles should apply to biospecimens. Nevertheless, there is a widespread problem of not sharing biospecimen resources that is clearly visible within the research arena. The impacts of this are likely to include diversion of precious research funds into compiling duplicate biospecimen cohorts, detraction from research productivity as researchers compete for and create duplicate resources, and deterrence of attempts to assess research reproducibility. This article explores some of the barriers that may limit availability of FAIR biospecimens. These barriers relate to the type of biospecimen collections and the characteristics of the custodians that influence their intention and interest in sharing. Barriers also relate to the ethical, legal, and social issues concerning collections, the research context of the collections, and cost and expertise involved in repurposing collections to enable sharing. Several solutions to increase sharing are identified. Some have recently been implemented, including enhancing biospecimen locators with tools to guide researchers and facilitating transfer of research collections to centralized biobank infrastructures at the conclusion of projects. New proposed solutions include improving search capabilities within publication databases, and introduction of evidence-based justifications for all new collections into peer-reviewed grant competition processes. It is recognized that there are both scientific factors and practical reasons that can impose limits to sharing biospecimens. However, funding availability, productivity, and progress in health research all stand to benefit from improved sharing of research biospecimen collections.

How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review

Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees’ overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most—but not all—of the referees’ judgments. Future studies should therefore examine how referees’ judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.

Obtrusive Plagiarism and Data Falsification in Biological Sciences: Trends and Remedies

Scientific misconduct includes plagiarism, falsification and fabrication. The intentional presentation of other’s work as own work without any authority is referred as plagiarism. Plagiarism also contains other’s images, structure and designed elements. It affects both published and unpublished materials obtained through evaluation of dissertations, peer review process or grant proposals. Fabrication is another research misconduct in which some fake activities are committed such as making up data, results and reporting them. If there is no any experimentation, it is also called fabrication. The Study shows that 60% of articles published in predatory journals are not found to be cited over the five-year period from the date of publication. Clear ethical standards should be made for assuring the researchers whether their work break certain codes or not. The investigation of organizations must be transparent, fair and prompt into research irregularities.

Reviewer training for improving grant and journal peer review.

Funders and scientific journals use peer review to decide which projects to fund or articles to publish. Reviewer training is an intervention to improve the quality of peer review. However, studies on the effects of such training yield inconsistent results, and there are no up-to-date systematic reviews addressing this question. To evaluate the effect of peer reviewer training on the quality of grant and journal peer review. We used standard, extensive Cochrane search methods. The latest search date was 27 April 2022. We included randomized controlled trials (RCTs; including cluster-RCTs) that evaluated peer review with training interventions versus usual processes, no training interventions, or other interventions to improve the quality of peer review. We used standard Cochrane methods. Our primary outcomes were 1. completeness of reporting and 2. peer review detection of errors. Our secondary outcomes were 1. bibliometric scores, 2. stakeholders' assessment of peer review quality, 3. inter-reviewer agreement, 4. process-centred outcomes, 5. peer reviewer satisfaction, and 6. completion rate and speed of funded projects. We used the first version of the Cochrane risk of bias tool to assess the risk of bias, and we used GRADE to assess the certainty of evidence. We included 10 RCTs with a total of 1213 units of analysis. The unit of analysis was the individual reviewer in seven studies (722 reviewers in total), and the reviewed manuscript in three studies (491 manuscripts in total). In eight RCTs, participants were journal peer reviewers. In two studies, the participants were grant peer reviewers. The training interventions can be broadly divided into dialogue-based interventions (interactive workshop, face-to-face training, mentoring) and one-way communication (written information, video course, checklist, written feedback). Most studies were small. We found moderate-certainty evidence that emails reminding peer reviewers to check items of reporting checklists, compared with standard journal practice, have little or no effect on the completeness of reporting, measured as the proportion of items (from 0.00 to 1.00) that were adequately reported (mean difference (MD) 0.02, 95% confidence interval (CI) -0.02 to 0.06; 2 RCTs, 421 manuscripts). There was low-certainty evidence that reviewer training, compared with standard journal practice, slightly improves peer reviewer ability to detect errors (MD 0.55, 95% CI 0.20 to 0.90; 1 RCT, 418 reviewers). We found low-certainty evidence that reviewer training, compared with standard journal practice, has little or no effect on stakeholders' assessment of review quality in journal peer review (standardized mean difference (SMD) 0.13 standard deviations (SDs), 95% CI -0.07 to 0.33; 1 RCT, 418 reviewers), or change in stakeholders' assessment of review quality in journal peer review (SMD -0.15 SDs, 95% CI -0.39 to 0.10; 5 RCTs, 258 reviewers). We found very low-certainty evidence that a video course, compared with no video course, has little or no effect on inter-reviewer agreement in grant peer review (MD 0.14 points, 95% CI -0.07 to 0.35; 1 RCT, 75 reviewers). There was low-certainty evidence that structured individual feedback on scoring, compared with general information on scoring, has little or no effect on the change in inter-reviewer agreement in grant peer review (MD 0.18 points, 95% CI -0.14 to 0.50; 1 RCT, 41 reviewers, low-certainty evidence). Evidence from 10 RCTs suggests that training peer reviewers may lead to little or no improvement in the quality of peer review. There is a need for studies with more participants and a broader spectrum of valid and reliable outcome measures. Studies evaluating stakeholders' assessments of the quality of peer review should ensure that these instruments have sufficient levels of validity and reliability.

Impact of Facility Type on Survival in Chronic Myelomonocytic Leukemia: A Propensity Score Matched, National Cancer Database Analysis

Introduction Chronic myelomonocytic leukemia (CMML) is an MDS/MPN overlap syndrome with a heterogenous clinical presentation and limited therapeutic options. Given its rarity, we hypothesize that there may be disparate outcomes based on the type of treatment facility (academic versus community) similar to what has been observed in other myeloid malignancies such as acute myeloid leukemia (VR Bhatt AJH 2017). To that end, we performed an analysis of the National Cancer Database (NCDB) to assess the impact of academic status of the facility on overall survival (OS) in patients with CMML. Methods We identified patients in the NCDB by the ICD-O-3 code for CMML (9945); we excluded patients who did not receive treatment and those who received all or part of their treatment outside of the reporting facility. Importantly, baseline clinical characteristics, laboratory/mutational data, and treatment details are generally not available in the NCDB. Academic centers were defined as those with a peer-reviewed NCI-designated cancer center support grant or university medical school that provide postgraduate medical education in at least four program areas, as defined by the Commission on Cancer Accreditation program. The primary outcome was OS. Propensity score matching (PSM) was performed and after backwards selection, the following variables were used for matching: age, race, median income, urban/rural, education, Charlson comorbidity index (CCI). Cox proportional hazard multivariable regression was employed to identify independent predictors of survival in the PSM population. Results A total of 6290 patients were identified. Baseline characteristics based on facility type are shown in Table 1. Patients treated at an academic center were younger, with a higher prevalence of Black race and lower comorbidity burden. Academic centers also had a patient population with a significantly higher median household income, more private insurance, and higher education level as compared to non-academic centers. The median follow-up time was 14.6 months (95% CI 0.4-89.7). The median OS for the entire cohort was 59 months (95% CI 56.7-62.0) and the median survival was significantly longer in patients treated at an academic center (60.6 months [95% CI 57.5-64.8]) as compared to those treated at a non-academic center (57.3 months [95% CI 53.4-61.8]). In the PSM analysis, a cohort of 1884 patients treated at an academic center were matched to 1895 patients treated at a non-academic center. In an unadjusted analysis, there was no significant difference in median OS (p=0.3) between the academic and non-academic groups. However, in a multivariable analysis of the PSM populations that accounted for age, race, CCI, insurance, distance to center, and median income, patients treated at an academic center had a significantly improved survival as compared to those treated at a non-academic center (HR 0.82, 95% CI 0.71-0.94, p = 0.005, Figure 1). Discussion We found significant differences in the characteristics of CMML patients treated at academic versus non-academic centers. After PSM and multivariable adjustment, survival was significantly improved with treatment at an academic center versus non-academic. Although limited in the depth of information about CMML prognostic features and treatments, this relatively large analysis of CMML patients suggests that management of this rare disease at an academic center may be associated with improved survival outcomes. Future prospective studies to ascertain the underlying causes for survival discrepancy in CMML at academic versus non-academic centers may help identify patients that should be triaged to an academic facility.

Questionable research practices in competitive grant funding: A survey.

There has been a surge of interest in research integrity over the last decade, with a wide range of studies investigating the prevalence of questionable research practices (QRPs). However, nearly all these studies focus on research design, data collection and analysis, and hardly any empirical research has been done on the occurrence of QRPs in the context of research funding. To fill this gap, we conducted a cross-sectional pre-registered survey of applicants, reviewers and panel members from the Research Foundation-Flanders (FWO), one of the main funding agencies in Belgium. We developed a bespoke survey and further refined it through feedback from experienced researchers and a pilot study. We asked how often respondents had engaged in a series of QRPs over the last ten years. A total of 1748 emails were sent, inviting recipients to participate in the survey, complemented by featuring the survey in the FWO newsletter. This resulted in 704 complete responses. Our results indicate that such QRPs are remarkably prevalent. Of the 496 participants who answered both the applicant and reviewer track, more than 60% responded that they engaged regularly in at least one of such practices, and around 40% indicated that they engaged at least occasionally in half of the QRPs queried. Only 12% reported not to have engaged in any of the QRPs. Contrary to our hypotheses, male respondents did not self-report to engage in the QRPs more often than female respondents, nor was there an association between the prevalence of QRPs and self-reported success rate in grant funding. Furthermore, half of the respondents indicated that they doubted the reliability of the grant peer review process more often than not. These results suggest that preventive action is needed, and provide new reasons to reconsider the practice of allocating research money through grant peer review.

Atrial Fibrillation Recurrence in Patients With Transient New-Onset Atrial Fibrillation Detected During Hospitalization for Noncardiac Surgery or Medical Illness : A Matched Cohort Study.

Atrial fibrillation (AF) is often detected for the first time in patients who are hospitalized for another reason. Long-term risks for AF recurrence in these patients are unclear. To estimate risk for AF recurrence in patients with new-onset AF during a hospitalization for noncardiac surgery or medical illness compared with a matched population without AF. Matched cohort study. (ClinicalTrials.gov: NCT03221777). Three academic hospitals in Hamilton, Ontario, Canada. The study enrolled patients hospitalized for noncardiac surgery or medical illness who had transient new-onset AF. For each participant, an age- and sex-matched control participant with no history of AF from the same hospital ward was recruited. All participants left the hospital in sinus rhythm. 14-day electrocardiographic (ECG) monitor at 1 and 6 months and telephone assessment at 1, 6, and 12 months. The primary outcome was AF lasting at least 30 seconds on the monitor or captured by ECG 12-lead during routine care at 12 months. Among 139 participants with transient new-onset AF (70 patients with medical illness and 69 surgical patients) and 139 matched control participants, the mean age was 71 years (SD, 10), the mean CHA2DS2-VASc score was 3.0 (SD, 1.5), and 59% were male. The median duration of AF during the index hospitalization was 15.8 hours (IQR, 6.4 to 49.6 hours). After 1 year, recurrent AF was detected in 33.1% (95% CI, 25.3% to 40.9%) of participants in the transient new-onset AF group and 5.0% (CI, 1.4% to 8.7%) of matched control participants; after adjustment for the number of ECG monitors worn and for baseline clinical differences, the adjusted relative risk was 6.6 (CI, 3.2 to 13.7). After exclusion of participants who had electrical or pharmacologic cardioversion during the index hospitalization (n = 40) and their matched control participants and limiting to AF events detected by the patch ECG monitor, recurrent AF was detected in 32.3% (CI, 23.1% to 41.5%) of participants with transient new-onset AF and 3.0% (CI, 0% to 6.4%) of matched control participants. Generalizability is limited, and the study was underpowered to evaluate subgroups and clinical predictors. Among patients who have transient new-onset AF during a hospitalization for noncardiac surgery or medical illness, approximately 1 in 3 will have recurrent AF within 1 year. Peer-reviewed grants.

Gender Equality Training for Students in Higher Education: Protocol for a Scoping Review.

The principles of gender equality are integral to the goals, targets, and indicators of all sustainable development goals. Higher education institutes can be powerful agents for promoting gender equality, diversity, and inclusion not only in the higher education context but also in society as a whole. To address and overcome gender inequality in the higher education environment, experts posit that change needs to occur from day 1 of the student's academic experience. To this end, training is required. A preliminary review of the literature indicates that multiple gender equality-based training programs or initiatives for students have been designed and evaluated in second and third-level education settings. Examples of educational activities undertaken include delivery of didactic teaching, participation in a face-to-face collaboration project, site visits, case studies, and coaching. Yet, our initial search indicated that, to date, a comprehensive review collating the available evidence on gender equality training for third-level students has not yet been carried out. Our review seeks to identify and explore the existing literature on gender equality training interventions for third-level students, with a particular emphasis on training content, methodology, and outcome evaluation. This scoping review will be structured using the Arskey and O'Malley's 5-stage framework and will consider empirical research and other relevant published works that address gender equality training. Systematic searches will be carried out in 6 research databases and the gray literature using key search terms. Inclusion and exclusion criteria have been defined, and a data charting tool created to methodically extract information from selected literature. The free web software Rayyan will be used for primary screening where each reference will be screened in duplicate first by title, then abstract, and finally by full text. This review forms part of the LIBRA (Balance) study and has received peer-reviewed grant funding from the Irish Higher Education Authority. LIBRA aims to use simulation-based education to develop a gender equality leadership training program for student leaders in higher education. The findings will be summarized in tabular form, and a narrative synthesis produced to inform curriculum development. This review seeks to inform curriculum design by reporting on the gender equality-enabling skills and leadership skills necessary to foster gender equality. This paper should inform recommendations for training and catalyze future research in this rapidly evolving area. DERR1-10.2196/44584.

Distrust in grant peer review—reasons and remedies

Abstract With the increasing reliance on competitive grants to fund research, we see a review system under pressure. While peer review has long been perceived as the cornerstone of self-governance in science, researchers have expressed distrust in the peer review procedures of funding agencies. This paper draws on literature pointing out ability, benevolence, and integrity as important for trustworthiness and explores the conditions under which researchers have confidence in grant review. Based on rich survey material, we find that researchers trust grant reviewers far less than they trust journal peer reviewers or their colleagues’ ability to assess their research. Yet, scholars who have success with grant proposals or serve on grant review panels appear to have more trust in grant reviewers. We conclude that transparency and reviewers with field competencies are crucial for trust in grant review and discuss how this can be ensured.

Creating a Community of Nurse Educator Scholars: The QSEN Academic Task Force Success.

Engaging in scholarship is a critical component of being an effective nurse educator and can have a significant impact on the nursing profession as a whole. Nurse educators are expected to teach and simultaneously engage in scholarship. Barriers include heavy workloads and lack of resources including faculty mentors. Evidence supports that organized work groups support success. Members of the Quality and Safety Education for Nurses (QSEN) Academic Task force were queried about scholarly activity as a result of engagement with fellow task force members. Scholarly outcomes included numerous peer-reviewed publications, teaching tips, presentations, and grants. Participating in national work groups can serve as a platform for fostering a community of scholars to make meaningful contributions to the profession.

Disclosing and Managing Non-Financial Conflicts of Interest in Scientific Publications.

In the last decade, there has been increased recognition of the importance of disclosing and managing non-financial conflicts of interests to safeguard the objectivity, integrity, and trustworthiness of scientific research. While funding agencies and academic institutions have had policies for addressing non-financial interests in grant peer review and research oversight since the 1990s, scientific journals have been only recently begun to develop such policies. An impediment to the formulation of effective journal policies is that non-financial interests can be difficult to recognize and define. Journals can overcome this problem by providing guidance concerning the types of non-financial interests that should be disclosed, including direct research interests, direct professional interests, expert testimony, involvement in litigation, holding a leadership position in a non-governmental organization, providing technical or scientific advice to a non-governmental organization, and personal or professional relationships. The guidance should apply to authors, editors, and reviewers.

Autism research capacity building in Northern Mexico: Preliminary evaluation of an ongoing process.

To inform improvement of care and public policy, quality research is required. Conducting research projects requires skills and infrastructure. Research capacity building is the process by which individuals and organizations develop greater ability to conduct useful research. However, in the scientific community, research quality and productivity measures are often focused on long-term products, such as publications or grants awarded. Those measures are not helpful when measuring initial stages of research, a common situation in low- and middle-income countries. We used a six-principle framework designed by J. Cooke to assess the progress of an autism research collaborative in northern Mexico. We established 64 criteria across the six principles, and rated them as Not Started, In Progress, or Achieved, based on narratives and process documentation collected for this purpose. Ratings were established for the Years 2018, 2020, and 2022. The principles with the highest Achieved rates were "Linkages, collaborations, and partnerships" (90%), and "Infrastructure" (Principle 6, 70%). Cooke's research capacity building framework is a useful way to monitor progress of research capacity building and contribute to harmonious development of relevant principles at different levels.

Three years of quality assurance data assessing the performance of over 4000 grant peer review contributions to the Canadian Institutes of Health Research Project Grant Competition

The Canadian Institutes of Health Research (CIHR) commenced a Quality Assurance Program in 2019 to monitor the quality of peer review in its Project Grant Competition Peer Review Committees. Our primary aim was to describe the performance of CIHR grant peer reviewers, based on the assessments made by CIHR peer review leaders during the first 3 years of the Research Quality Assurance Program. All Peer Review Committee Chairs and (or) Scientific Officers who led peer review for CIHR in 2019, 2020, and 2021 completed Reviewer Quality Feedback forms immediately following Peer Review Committee meetings. The form assessed Performance, Future potential, Review quality, Participation, and Responsiveness. We summarised and descriptively synthesised data from assessments conducted after each of the four grant competitions. The performance of peer reviewers on 4438 occasions was rated by Chairs and Scientific Officers. Approximately one in three peer reviewers submitted outstanding reviews or discussed additional applications and one in 10 demonstrated potential as a future Peer Review Committee leader. At most, one in 20 peer reviewers was considered to have not performed adequately with respect to review quality, participation, or responsiveness. There is a need for more research on the processes involved in allocating research grant funding.

Dental teaching faculty involvement in research activities.

This study aimed to examine personal and professional factors in determining the research activity and research culture among dental teaching faculty members in Hyderabad City, Telangana. Two dental schools with ongoing dental undergraduate and postgraduate programmes were included in the study. The Research and Development (R&D) Culture Index Questionnaire comprising 16 items was employed for this study. Also, self-reported research activity undertaken in the past twelve months was recorded. The association of demographic variables and professionally related variables with the perception of R&D culture and participation in research activities was assessed using the Chi-square test. A P value was set at 0.05. Of the one-hundred and fifteen faculty members who participated in the study, seventy-four (64.4%) worked in the government sector and forty-one (35.6%) in privately funded dental college. The mean age of the study population was 34.11 + 7.13 years. The majority of them presented a paper or poster at the conference (66.1%; 76) followed by publishing paper in peer-review journal (42.6%; 49) and grant application for research (6.1%; 7), and authoring of a textbook was undertaken by only 2.6% of the study population. Males (61.1%; P = 0.04) and those employed in private dental college (65.8%; P = 0.02) had significantly high levels of overall perception than their respective counterparts. Our study concludes that this study population has a positive perception (50.32 + 6.95) towards research with the majority of them having done a paper or poster publication (66.1%) in the last one year.

PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments

In peer selection a group of agents must choose a subset of themselves, as winners for, e.g., peer-reviewed grants or prizes. We take a Condorcet view of this aggregation problem, assuming that there is an objective ground-truth ordering over the agents. We study agents that have a noisy perception of this ground truth and give assessments that, even when truthful, can be inaccurate. Our goal is to select the best set of agents according to the underlying ground truth by looking at the potentially unreliable assessments of the peers. Besides being potentially unreliable, we also allow agents to be self-interested, attempting to influence the outcome of the decision in their favour. Hence, we are focused on tackling the problem of impartial (or strategyproof) peer selection – how do we prevent agents from manipulating their reviews while still selecting the most deserving individuals, all in the presence of noisy evaluations? We propose a novel impartial peer selection algorithm, PeerNomination, that aims to fulfil the above desiderata. We provide a comprehensive theoretical analysis of the recall of PeerNomination and prove various properties, including impartiality and monotonicity. We also provide empirical results based on computer simulations to show its effectiveness compared to the state-of-the-art impartial peer selection algorithms. We then investigate the robustness of PeerNomination to various levels of noise in the reviews. In order to maintain good performance under such conditions, we extend PeerNomination by using weights for reviewers which, informally, capture some notion of reliability of the reviewer. We show, theoretically, that the new algorithm preserves strategyproofness and, empirically, that the weights help identify the noisy reviewers and hence to increase selection performance.1

The Necessity of Commensuration Bias in Grant Peer Review

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it is in many cases impossible to avoid commensuration bias.

Integrating Systematic Reviews into Supportive Care Trial Design: The Rethinking Clinical Trials (REaCT) Program.

To review the successes and challenges of integrating systematic reviews (SRs) into the Rethinking Clinical Trials (REaCT) Program. All REaCT program SRs were evaluated and descriptive summaries presented. Twenty-two SRs have been performed evaluating standard of care interventions for the management of: breast cancer (n = 15), all tumour sites (n = 4), breast and prostate cancers (n = 2), and prostate cancer (n = 1). The majority of SRs were related to supportive care (n = 14) and survivorship (n = 5) interventions and most (19/22, 86%) confirmed the existence of uncertainty relating to the clinical question addressed in the SR. Most SRs (15/22, 68%) provided specific recommendations for future studies and results were incorporated into peer-reviewed grant applications (n = 6) and clinical trial design (n = 12). In 12/22 of the SRs, the first author was a trainee. All SRs followed PRISMA guidelines. SRs are important for identifying and confirming clinical equipoise and designing trials. SRs provide an excellent opportunity for trainees to participate in research.

Towards theorizing peer review

Abstract Academic peer review is seriously undertheorized because peer review studies focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe the main characteristics of the peer review literature, which focuses mainly on journal and grant peer review. Based on these characteristics, I then argue why theory is useful in research on peer review, and I present some theoretical efforts on peer review. I conclude by encouraging peer review researchers to be more theoretically engaged and outline activities that theoretical work on peer review could involve. This invitation to theory-building complements recent roadmaps and calls that have emphasized that we need to have better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.