Applying Psychometric Theory and Research to Developing a Continuously Distributed Approach to Making Research Funding Decisions

Abstract

Scientists and funding agencies have long struggled with identifying optimal methods for allocatingscarce research dollars to pools of research funding proposals. Much of the research has focused onthe unreliability of the subjective peer review ratings used to estimate the scholarly merit of fundingproposals. Yet, psychologists are uniquely positioned to aid in improving the validity of the fundingdecisions based on subjective peer review evaluations. Importantly, no federal funding agency currentlyimplements a decision-making method that goes beyond a traditional dichotomous system for allocatingsupport (i.e., “fund” vs. “not fund” a proposal). However, a robust line of psychometric research indicatesthe superiority of continuous, relative to dichotomous, rating systems for reliably and validly estimatingthe kinds of constructs of interest in peer review evaluations (i.e., domains relevant to estimating thescholarly merit of research proposals). In this paper, we illustrate how applying psychometric researchand theory informs the development of a continuously distributed approach to making research fundingdecisions. This approach allocates a specific degree of support to proposals, commensurate with thecontinuously distributed peer review ratings used at major funding agencies to estimate the scholarlymerit of proposals. Importantly, we illustrate a hypothetical use of the continuously distributed fundingapproach that—relative to a dichotomous funding approach and using the same amount of fundingresources—allocates funding dollars to 260% more funding proposals. We also discuss the strengths andlimitations of a continuously distributed funding approach, as well as its implications for scientists andfunding agencies.Keywords: peer review, psychometrics, research funding, methodology