Incorporating costs and benefits to the evaluation of uncertain research results: Applications to cancer research funding

Abstract

Abstract Correctness is a key aspiration of the scientific process, yet recent studies suggest that many high-profile findings may be difficult to replicate or require considerable evidence for verification. Proposals to fix these issues typically ask for tighter statistical controls (e.g., stricter p-value thresholds or higher statistical power). However, these approaches often overlook the importance of contemplating research outcomes’ potential costs and benefits. Here, we develop a framework grounded in Bayesian decision theory that seamlessly integrates cost-benefit analysis into evaluating research programs with potentially uncertain results. We derive minimally acceptable pre-study odds and positive predictive values for cost and benefit levels. We show that tolerance to inaccurate results changes dramatically due to uncertainties posed by research. We also show that reducing uncertainties (e.g., by recruiting more subjects) may have limited effects on the expected benefit of continuing specific research programs. We apply our framework to several types of cancer research and their funding. Our analysis shows that highly exploratory research designs are easily justifiable due to their potential benefits, even when probabilistic models suggest otherwise. We discuss how the cost and benefit of research could and should always be part of the toolkit used by scientists, institutions, or funding agencies.