Peer Review Of Grant Proposals Research Articles

Gender and other potential biases in peer review: cross-sectional analysis of 38 250 external peer review reports

ObjectivesTo examine whether the gender of applicants and peer reviewers and other factors influence peer review of grant proposals submitted to a national funding agency.SettingSwiss National Science Foundation (SNSF).DesignCross-sectional analysis...

Peer Review Evaluation Process of Marie Curie Actions under EU's Seventh Framework Programme for Research.

We analysed the peer review of grant proposals under Marie Curie Actions, a major EU research funding instrument, which involves two steps: an independent assessment (Individual Evaluation Report, IER) performed remotely by 3 raters, and a consensus opinion reached during a meeting by the same raters (Consensus Report, CR). For 24,897 proposals evaluated from 2007 to 2013, the association between average IER and CR scores was very high across different panels, grant calls and years. Median average deviation (AD) index, used as a measure of inter-rater agreement, was 5.4 points on a 0-100 scale (interquartile range 3.4-8.3), overall, demonstrating a good general agreement among raters. For proposals where one rater disagreed with the other two raters (n=1424; 5.7%), or where all 3 raters disagreed (n=2075; 8.3%), the average IER and CR scores were still highly associated. Disagreement was more frequent for proposals from Economics/Social Sciences and Humanities panels. Greater disagreement was observed for proposals with lower average IER scores. CR scores for proposals with initial disagreement were also significantly lower. Proposals with a large absolute difference between the average IER and CR scores (≥10 points; n=368, 1.5%) generally had lower CR scores. An inter-correlation matrix of individual raters' scores of evaluation criteria of proposals indicated that these scores were, in general, a reflection of raters’ overall scores. Our analysis demonstrated a good internal consistency and general high agreement among raters. Consensus meetings appear to be relevant for particular panels and subsets of proposals with large differences among raters’ scores.

Blue skies, impacts, and peer review

This paper describes the results of a survey regarding the incorporation of societal impacts considerations into the peer review of grant proposals submitted to public science funding bodies. The survey investigated perceptions regarding the use of scientific peers to judge not only the intrinsic scientific value of proposed research, but also its instrumental value to society. Members of the scientific community have expressed – some more stridently than others – resistance to the use of such societal impact considerations. We sought to understand why. Results of the survey suggest that such resistance may be due to a lack of desire rather than a lack of confidence where judging impacts is concerned. In other words, it may be less that scientists feel unable to judge broader societal impacts and more that they are unwilling to do so.

Around the World

In science news around the world this week, a Senate bill would preserve the U.S. helium supply for research, researchers have issued the first-ever international principles governing peer review of grant proposals, the European Parliament rebuked three regulatory agencies, Norway opened the world's largest carbon capture and storage test facility, and a new Twitter feed run by the Shanghai U.S. consulate is charting the city's air quality.

POINT: Statistical analysis in NIH peer review— identifying innovation

In this essay/hypothesis, I assess the statistical technology used in the NIH peer review of research proposals. In the NIH system of proposal review, sample sizes are inappropriately small, samples are highly selective or nonrandom which makes the review particularly susceptible to bias—the samples are not independently derived, the scoring precision is unrealistically high, and the arithmetic mean is the only statistical value considered. I propose a complementary peer review system that addresses these issues. Moreover, I hypothesize that innovative grant applications can be assessed in a system with robust statistical procedures by measures of dispersion such as variance and/or kurtosis. The identification of innovation has been a longterm deficiency at NIH. The peer review of grant proposals is an essential part of the management of biomedical research. Consequently, it is reasonable to consider how statistics are used in this peer review process. First, NIH and most funding agencies assess grant applications by requesting reviews from just two or three qualified persons. The number of samples is constrained by the large size of the grant applications and by writing proposals narrowly for extreme experts in narrowly defined fields. Second, peer review does not use random sampling of the relevant population. Instead experts are chosen based on their perceived expertise and qualifications. Unfortunately, this sampling scheme is particularly subject to bias because the extreme experts have a vested interest in the current paradigms. Third, the review process used by NIH and others involves discussions among reviewers. These discussions are meant to provide consensus, not to assess the distribution of sensibilities among the relevant population of scientists. The scores provided in this system are not independently derived, but independence of the observations is a valuable attribute for statistical analysis. Fourth, NIH uses a 41-grade scoring system which suggests an unrealistic degree of precision. A high-precision score is needed to distinguish among the many applications in a situation with only two or three dependent samples being collected. Finally, peer review of grant applications at NIH and most funding agencies uses a single statistic, the arithmetic mean. The use of this single statistic may be reasonable for a normal distribution of parametrically related observations; yet, scores given in peer review are not necessarily normally distributed and are definitely not parametrically related. Although funding agencies use statistical techniques that are constrained and although they largely ignore the distribution of reviews among the relevant peer group, their system has been successful in identifying excellent grant proposals. What NIH and other funding agencies have specifically struggled with is the identification of innovative ideas. Innovation and excellence are easily conflated; nevertheless, their distinction is essential in establishing the most potent funding policies. Excellence is the property that indicates a high degree of certainty that the proposed work will enhance our scientific understanding or capabilities. The trade-off is the inverse relationship between the degree of certainty and the degree of the enhancement. Excellence usually provides us with small steps forward. In contrast, innovation is the property that indicates novelty which carries with it a low degree of certainty of success. Although innovative projects lack certainty, the potential for huge enhancements in our scientific understanding or

Preparing the senior or graduating student for graduate research

AbstractSenior undergraduates in the honors or graduation year with an intention to further their career in science would soon face the real world of scientific research as a junior scientist. It is important to acquaint these students with and adequately prepare them for the key aspects of a scientist's professional life. These include technical capacities indispensable for continuous and intense appraisal of data and knowledge as well as awareness that would ensure proper professional conduct within the laboratory. We have designed and conducted a module called “Scientific Critique” in the fourth year for Biochemistry majors. The module emphasizes scientific literature review and critique. It also incorporates techniques in preparing research grant proposals and an exercise in peer review of grant proposals as well as lectures and presentations on scientific integrity and research ethics. Positive response and feedback from students are indicative of an appreciation for the module contents, and we believe it of value to include such contents in any senior undergraduate life sciences curriculum.

A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings

SummaryThe peer review of grant proposals is very important to academics from all disciplines. Although there is limited research on the reliability of assessments for grant proposals, previously reported single-rater reliabilities have been disappointingly low (between 0.17 and 0.37). We found that the single-rater reliability of the overall assessor rating for Australian Research Council grants was 0.21 for social science and humanities (2870 ratings, 1928 assessors and 687 proposals) and 0.19 for science (7153 ratings, 4295 assessors and 1644 proposals). We used a multilevel, cross-classification approach (level 1, assessor and proposal cross-classification; level 2, field of study), taking into account that 34% of the assessors evaluated more than one proposal. Researcher-nominated assessors (those chosen by the authors of the research proposal) gave higher ratings than panel-nominated assessors chosen by the Australian Research Council, and proposals from more prestigious universities received higher ratings. In the social sciences and humanities, the status of Australian universities had significantly more effect on Australian assessors than on overseas assessors. In science, ratings were higher when assessors rated fewer proposals and apparently had a more limited frame of reference for making such ratings and when researchers were professors rather than non-professors. Particularly, the methodology of this large scale study is applicable to other forms of peer review (publications, job interviews, awarding of prizes and election to prestigious societies) where peer review is employed as a selection process.

Investigating Misconduct in Science: The National Science Foundation Model

Reports in the popular press,(1) as well as recent studies by organizations,(2) make clear that the problem of misconduct in science(3) is real and persistent. integrity is not always ensured by science's self-correcting mechanisms.(4) The effects of misconduct go beyond distortion of data. Misconduct can also undermine the proper and fair allocation of federal resources when it involves peer review of grant proposals, a process that depends on honesty and integrity. The public faith in science that is necessary to ensure sufficient funding for the progress of science depends on honest applications for federal funding and honest use of public funds. Therefore, both the government and the academic community need effective procedures for dealing with misconduct in science. Despite the need for effective procedures for establishing whether misconduct in science has occurred, considerable controversy has arisen over which procedures should be used. Most universities have only recently put in place procedures for dealing with misconduct in science and have not yet fully tested them.(5) According to an American Association for the Advancement of Science survey, scientists lack confidence in university handling of misconduct cases.(6) This article presents our study of the principles underlying NSF's procedures for handling misconduct in science, as well as the results of NSF's experience in handling allegations of misconduct. By presenting the rationale for NSF's procedures, we hope not only to persuade the community of the applicability of these procedures to misconduct in science cases but also to stimulate comment and criticism. The Scientific Dialogue Model Many early cases of misconduct in science have been handled according to procedures developed at the Public Health Service (PHS) and referred to as the scientific model.(7) This model emulates the peer review process by assembling a panel of scientists who investigate allegations and reach a consensus on whether misconduct has occurred.(8) Although the familiarity of the procedures initially appealed to some scientists, the community itself has criticized handling of cases under the dialogue model for not meeting standards of confidentiality and fairness.(9) Applying methods developed for the evaluation of manuscripts, grant proposals, or tenure qualifications to the determination of ethically acceptable conduct leaves unfulfilled expectations. When proceedings may lead to sanctions, both accused scientists and observers expect certain protections. Nonetheless, some scientists resisted application of traditional American investigative and adjudicative procedures, which offer such protections, to misconduct in science cases. Perhaps this was due to concerns that issues would get lost in such a setting.(10) However, theory and practice in the handling of misconduct cases has brought out the importance of procedural fairness. PHS has shown an increased awareness of procedural protections in handling misconduct in science cases by introducing the opportunity for a hearing.(11) NSF's Model for Handling Misconduct in Science Cases Since 1987 NSF has based its policies and procedures for handling misconduct in science cases on well-established administrative, civil, and criminal methods of investigation and adjudication.(12) NSF believes that traditional investigative and adjudicative procedures are appropriate when the issue is culpability for misconduct. These traditional procedures are preferable to alternatives adapted from the process of peer review. Peer reviewers necessarily assume that authors are truthful. Because the central issue to be resolved in a misconduct case is whether an individual has been truthful, a system like peer review is inappropriate because it lacks the investigative, adjudicatory, and due process mechanisms to evaluate that issue accurately and fairly. …

Priority Disputes and Peer Review: Academic Self-Regulation

The Review o f Higher Education Fall 1993, Volume 17, No. 1, pp. 95-104 Copyright © 1993 Association for the Study of Higher Education All Rights Reserved (ISSN 0162-5748) Review Essay Priority Disputes and Peer Review: Academic Self-Regulation John M. Braxton Daryl E. Chubin and Edward J. Hackett, Peerless Science: Peer Review and U.S. Science Policy (Albany: State University of New York Press, 1990), 267 pp. Susan E. Cozzens, Social Control and Multiple Discovery in Science: The Opiate Receptor Case (Albany: State University of New York, 1989), 236 pp. Society expects professions to be self-regulating and, in exchange, gives them autonomy over the conduct of their professional roles (Goode 1969). For the academic profession, research is one of its core activities (Parsons and Platt 1973). Because the goal of research is the advancement of knowledge, the fundamental aims of regulating research role perfor­ mance are to assure not only that the research makes a contribution to knowledge but also that it is conducted according to the technical stan­ dards, ethical principles, and scientific norms of an academic discipline. Assuring that research is conducted in accordance with cognitive and normative standards is presumed to be essential to the advancement of knowledge (Merton 1973). Responsibility for self-regulation resides on several levels. Melissa Anderson and Karen Seashore Louis (1991) have delineated three such levels: the individual academic professional, the academic department, and the academic discipline. Individual academic professionals are reJohn M. Braxton is Associate Professor in the Department of Educational Leadership, Peabody College, Vanderbilt University, Nashville, Tennessee. 96 T h e R e v i e w o f H i g h e r E d u c a t i o n F a l l 1993 sponsible for the competent and ethical pursuit of their programs of research. At the department level, these same academic professionals evaluate the scholarship of their colleagues as the basis for decisions about promotion, tenure, and salary increases. At the level of the aca­ demic discipline, academics are responsible for the full spectrum of scholarship including the peer review of grant proposals and manu­ scripts submitted to academic journals, the allocation of recognition for contributions, and the enforcement of standards for research and codes of conduct. An opportunity to acquire a fine-grained knowledge and understand­ ing of self-regulation at the level of the academic discipline is provided by two recent books written by sociologists of science. These books are Daryl E. Chubin and Edward J. Hackett’s Peerless Science: Peer Review and the U.S. Science Policy and Susan E. Cozzens’s Social Control and Mul­ tiple Discovery in Science: The Opiate Receptor Case. Both of these volumes are part of the Science, Technology, and Society series of the State Uni­ versity of New York Press. Although both volumes address self-regulation in the scientific re­ search community, each examines a different facet of this process. As the titles of the books imply, peer review is Chubin and Hackett’s con­ cern, while Cozzens focuses on the role of third parties in disputes over recognition for scientific discoveries. An analysis of peer review provides us with a window on profes­ sional self-regulation, since peer review is predicated on the assumption that both the significance and quality of research are best appraised by scientists in the same field or subspecialty (Anderson and Louis 1991). In other words, expert knowledge and skill are the bases for peer review. If peer review is improperly discharged, then the advancement of knowl­ edge suffers. In turn, the public trust in the scientific community declines. We gain another window on professional self-regulation through Cozzens’s interest in how third parties exercise social control in disputes among scientists over recognition for discoveries made at about the same time. The ways in which third parties react to the credit-seeking behavior of disputants determine how self-regulation is executed. How the aca­ demic community allocates recognition for research contributions is im­ portant in understanding self-regulation, for recognition is the “coin of the realm” of scholarship. Collegial recognition certifies an individual’s role in the advancement of knowledge in an academic discipline (Storer 1973...

The Predictive Ability of Peer Review of Grant Proposals: The Case of Ecology and the US National Science Foundation

Analyses of temporal correlations in measures of scientific productivity and quality suggest that individuals who have produced good science in the recent past will continue to do so in the near future. However, previous work has shown little correlation between measures of the quality of previous publications by a scientist and the ratings received by that individual's research proposals. This suggests that evaluating previously published work may be a better means of predicting the quality of future work than is evaluating research proposals. Possible explanations for this finding are explored, using the Ecology Panel of the US National Science Foundation as an example.

The Freedom of Information Act and Accountability in University Research

In general, information on government research grants is covered by FOIA only if it is in possession of a government agency. Increasingly, citizens and policy-makers look to science to help solve pressing national and international problems relating to such matters as health, environmental quality, and defense. It's no surprise, then, that federal government is primary funder of university research in United States.1 With federal money comes an interest in policy implications of research, and scientists are regularly asked to contribute to development of governmental policy. In recent years, however, several widely publicized cases of fraud have raised questions about accountability in research, especially federally funded research.2 For scientists, promotions and better jobs depend to a great extent on publications in scholarly journals. The pressure to produce high numbers of publications can tempt researchers to fabricate findings.3 Peer review of grant proposals and of manuscripts submitted to scholarly journals is of little help because peer review is designed to assess quality of a piece of research, not to uncover fraud.4 Fraud is most likely when scrutiny of research process is least.5 Many scienlists zealously guard their data and lab notebooks6; for most part, scientists are accountable only to other scientists. However, as Justice Brandeis once said, sunshine is best of disinfectants. If members of public (including journalists and scientists) had a greater right of access to information about federally funded research, fraud would be less likely. Increased openness leads to increased accountability. In addition to inhibiting fraud, a freer flow of information about science and technology might have other benefits. For example, many believe that increased scrutiny by press and public could help avert incidents such as accident at Three-Mile Island nuclear power plant7 and destruction of space shuttle Challenger.8 This article examines extent to which Freedom of Information Act,9 designed to increase accountability among public institutions, enables access to information about federally funded university research. The topic clear relevance to journalists who cover science or higher education. But it also relates to much broader debate about access to government information generally,10 as well as about control of science information.11 Scientists often are unwilling to communicate to public via news media.12 And when scientists do talk to reporters, they often attempt to exert tight control over nature of communication.13 Although journalists who cover science rarely take an adversarial stance toward their sources,14 journalists and scientists sometimes differ over access to information.15 In such cases, a reporter may turn to FOIA to try to obtain information that a scientist been unwilling to provide. Background United States citizens have no constitutional right of access to information about government-funded research. Although information-gathering been linked to ability to exercise First Amendment rights,16 Supreme Court has never intimated a First Amendment guarantee of a right of access to all sources of information within governmental control.17 In absence of a clear First Amendment right, court said that determining what kinds of information must be available to public is clearly a legislative task.18 Accordingly, public access to governmental information hinges on what former Supreme Court Justice Potter Stewart once called the tug and pull of political forces in American Society.19 In 1966 those political forces produced FOIA, which created a strong legal presumption that information about federal activity should be available to public. A series of amendments in 1974 strengthened act, which been principal legal instrument used by citizens seeking access to information about federally funded university research. …