Comment: Expert Elicitation for Reliable System Design

Abstract

This article successfully identifies and addresses some of the most important challenges in the use of elicitation as part of engineered systems analysis. The authors make two key advances to the field. The first is that they perform an exhaustive synthesis of the probability elicitation literature relevant to the engineered systems context. The second advance to the field offered by this paper is that it reframes the elicitation literature around the limits, possibilities and actual constraints posed by systems engineering practice. This latter point is no small contribution—the authors have successfully opened a much needed discussion as to why system elicitation differs fundamentally from cultural ethnography methods, and why risk estimation and “systems ethnography” (elicitation of system dependencies and evolving uncertainties) are only partially informed by methods developed for identified and stable single distribution elicitation. I like the overview of the systems engineering (SE) life cycle and the link made to reliability through the r = r(d, p,u,m, c) relationship. In a follow-up paper or discussion it would be interesting to learn more about the types of systems the authors have studied. In thinking about how to elicit system information over a complete range of engineering efforts, it quickly becomes apparent how hard it is to characterize the elicitation effort (and why this article is such a notable exception to the general lack of disciplined study of this qualitative field). Because the elicitation problem varies greatly depending on the specific form of a technical system, as well as local analytic and decision-making realities, perhaps