The Role of Expert Judgment in Statistical Inference and Evidence-Based Decision-Making

Naomi C. Brownstein,Thomas A. Louis,Jane Pendergast,Anthony O’Hagan

doi:10.1080/00031305.2018.1529623

Abstract

ABSTRACTThis article resulted from our participation in the session on the “role of expert opinion and judgment in statistical inference” at the October 2017 ASA Symposium on Statistical Inference. We present a strong, unified statement on roles of expert judgment in statistics with processes for obtaining input, whether from a Bayesian or frequentist perspective. Topics include the role of subjectivity in the cycle of scientific inference and decisions, followed by a clinical trial and a greenhouse gas emissions case study that illustrate the role of judgments and the importance of basing them on objective information and a comprehensive uncertainty assessment. We close with a call for increased proactivity and involvement of statisticians in study conceptualization, design, conduct, analysis, and communication.

Highlights

As participants in the October 2017 Symposium on Statistical Inference (SSI), organized and sponsored by the American Statistical Association (ASA), we were challenged to host a session and write a paper inspired by the question, “Do expert opinion and judgment have a role in statistical inference and evidence-based decision-making?” While we work from different perspectives and in different statistical paradigms, there was a resounding “yes!” among us, with ample common ground in our thinking related to this infrequently discussed and often under-appreciated component of statistical and scientific practice
We examine the roles of expert judgment in each of the four stages presented in Figure 1, paying particular attention to integration of statistical and content expertise
Results of the UK Carbon Flux Study The content experts were certainly interested in knowing how much uncertainty in the total Net Biosphere Production (NBP) would be induced by uncertainty in the inputs

Summary

Introduction

As participants in the October 2017 Symposium on Statistical Inference (SSI), organized and sponsored by the American Statistical Association (ASA), we were challenged to host a session and write a paper inspired by the question, “Do expert opinion and judgment have a role in statistical inference and evidence-based decision-making?” While we work from different perspectives and in different statistical paradigms (both frequentist and Bayesian), there was a resounding “yes!” among us, with ample common ground in our thinking related to this infrequently discussed and often under-appreciated component of statistical and scientific practice. Frequentist methods require some components of subjectivity in the choice of a model and of estimators, test statistics, etc. These choices are subjective in the sense that every expert builds knowledge and judgment into their own personal framework of understanding, which is not likely to be identical to that of anyone else. We consider the roles of expert judgment in scientific practice. Our article is organized where we share our thoughts on when and how expert judgment has a legitimate and necessary role in scientific inquiry. We include in this stage the conduct of the study, resulting in some form of data

The Cycles of Inference and Decision in Science

Question

Interpret

Inform

Science and Subjectivity

The Question Stage

The Study Stage

The Interpret Stage

The Inform Stage

But Is It Science?

Expert Judgment in Randomized Clinical Trials

Background