Expert Judgement in Risk and Decision Analysis

About the Authors

About the Authors

About the Authors

About the Authors

About the Authors

About the Authors

In January, major cities on the East Coast of the United States braced for large snowfalls as Winter Storm Juno approached. Fortunately, the actual snowfall was much less than had been confidently predicted in many cities. In hindsight, it seemed clear that substantial costs of closing businesses and preparing for a blizzard could have been avoided if forecasts been more accurate. What happened? Why were so many forecasts both confident and wrong? Bob Winkler, who discussed aggregation of expert judgments in the January issue of Risk Analysis, describes how over-weighting of the more extreme forecasts among many different modelbased forecasts that were available contributed to overestimation of snowfall and of confidence in the forecast. How best to characterize uncertainties in model-based risk predictions when multiple moreor-less plausible models give very different results is a crucial challenge for the field, with possible applications ranging from dose-response modeling to climate change forecasting.

David J. Caswell (“ Analysis of National Strategies to Counter a Country's Nuclear Weapons Program ”) is an officer in the U.S. Air Force and a research affiliate with the Center for International Security and Cooperation at Stanford University. David has served in various positions ranging from operational simulation development to operations analysis for national intelligence. He currently serves as an operations analyst in support of regional air and space employment in the Pacific. David received his Ph.D. in management science and engineering at Stanford University. His current research continues to apply computer science and operations research methods for gaining insights for nuclear policy and other international security issues. Address: http://www.stanford.edu/group/ERRG/davidc1.htm ; e-mail: david.caswell33@gmail.com . Kjell Hausken (“ Governments' and Terrorists' Defense and Attack in a T-Period Game ”) has since 1999 been a professor of economics and societal safety at the University of Stavanger, Norway. His research fields are strategic interaction, risk analysis, reliability, conflict, and terrorism. He holds a Ph.D. (thesis: “Dynamic Multilevel Game Theory”) from the University of Chicago (1990–1994), and was a postdoc at the Max Planck Institute for the Studies of Societies (Cologne) from 1995 to 1998 and a visiting scholar at Yale School of Management from 1989 to 1990. He holds a doctorate program degree in administration from the Norwegian School of Economics and Business Administration, and an M.Sc. degree in electrical engineering from the Norwegian Institute of Technology. He completed military service at the Norwegian Defence Research Establishment, has published 110 articles, and is on the editorial board for Theory and Decision and Defence and Peace Economics. Address: Faculty of Social Sciences, University of Stavanger, N-4036 Stavanger, Norway; e-mail: kjell.hausken@uis.no . Ronald A. Howard (“ Analysis of National Strategies to Counter a Country's Nuclear Weapons Program ”) is a professor of management science and engineering in the School of Engineering at Stanford University. Professor Howard directs teaching and research in the Decision Analysis Program of the department, and is the director of the Decisions and Ethics Center, which examines the efficacy and ethics of social arrangements. He defined the profession of decision analysis in 1964 and has supervised more than 80 doctoral theses in decision analysis and related areas. His experience includes dozens of decision analysis projects that range over virtually all fields of application, from investment planning to research strategy, and from hurricane seeding to nuclear waste isolation. He has been a consultant to several companies and was a founding director and chairman of Strategic Decisions Group. He is president of the Decision Education Foundation, which he and colleagues founded to teach decision skills to young people. He has written four books, dozens of technical papers, and provided editorial service to seven technical journals. His society affiliations have included the Institute of Electrical and Electronics Engineers (Fellow); The Institute of Management Sciences, which he served as president, and the Institute for Operations Research and the Management Sciences (INFORMS) (Fellow). Continuing research interests are improving the quality of decisions, life-and-death decision making, and the creation of a coercion-free society. In 1986 he received the Frank P. Ramsey Medal “for Distinguished Contributions in Decision Analysis” from the Decision Analysis Special Interest Group of the Operations Research Society of America (the predecessor to the Decision Analysis Society of INFORMS). In 1998 he received from INFORMS the first award for the Teaching of Operations Research/Management Science Practice. In 1999 he was elected to the National Academy of Engineering. Address: Management Science and Engineering, Huang Engineering Center, 475 Via Ortega, Stanford University, Stanford, CA 94305-4121; e-mail: rhoward@stanford.edu . Joseph B. (“Jay”) Kadane (“ Partial-Kelly Strategies and Expected Utility: Small Edge Asymptotics ”) is Leonard J. Savage University Professor of Statistics and Social Sciences, Emeritus, at Carnegie Mellon University. He received a B.S. in mathematics from Harvard and a Ph.D. in statistics from Stanford. He was recently elected to the American Academy of Arts and Sciences. His theoretical interests center on subjective Bayesian theory. His current applied interests include Internet security, medicine, law, physics, marketing, and air pollution. He serves as an expert witness in legal cases. His most recent book is Principles of Uncertainty, which is scheduled to be released in May 2011 by Chapman and Hall and will be available free on the Web for any noncommercial purpose. Address: Department of Statistics, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213; e-mail: kadane@stat.cmu.edu . Konstantinos V. Katsikopoulos (“ Psychological Heuristics for Making Inferences: Definition, Performance, and the Emerging Theory and Practice ”) holds a Ph.D. in industrial engineering and operations research from the University of Massachusetts Amherst and is currently a senior research scientist at the Center for Adaptive Behavior and Cognition of the Max Planck Institute for Human Development. He has been a visiting assistant professor of operations research at the Naval Postgraduate School and of systems engineering at the Massachusetts Institute of Technology. He has made contributions to the theory of bounded rationality and its applications to decisions “in the wild” in fields such as engineering design and medicine. Address: Max Planck Institute for Human Development, Lentzeallee 94, 14195 Berlin, Germany; e-mail: katsikop@mpib-berlin.mpg.de . L. Robin Keller (“ Investment and Defense Strategies, Heuristics, and Games: From the Editor … ”) is a professor of operations and decision technologies in the Merage School of Business at the University of California, Irvine. She received her Ph.D. and M.B.A. in management science and her B.A. in mathematics from the University of California, Los Angeles. She has served as a program director for the Decision, Risk, and Management Science Program of the U.S. National Science Foundation (NSF). Her research is on decision analysis and risk analysis for business and policy decisions and has been funded by NSF and the U.S. Environmental Protection Agency. Her research interests cover multiple attribute decision making, riskiness, fairness, probability judgments, ambiguity of probabilities or outcomes, risk analysis (for terrorism, environmental, health, and safety risks), time preferences, problem structuring, cross-cultural decisions, and medical decision making. She is currently Editor-in-Chief of Decision Analysis, published by the Institute for Operations Research and the Management Sciences (INFORMS). She is a Fellow of INFORMS and has held numerous roles in INFORMS, including board member and chair of the INFORMS Decision Analysis Society. She is a recipient of the George F. Kimball Medal from INFORMS. She has served as the decision analyst on three National Academy of Sciences committees. Address: The Paul Merage School of Business, University of California, Irvine, Irvine, CA 92697-3125; e-mail: lrkeller@uci.edu . Jeryl L. Mumpower (“ Playing Squash Against Ralph Keeney: Should Weaker Players Always Prefer Shorter Games? ”) is Director of the Master of Public Service and Administration Program at the Bush School of Government and Public Service at Texas A&M University, where he holds the Joe R. and Teresa Lozano Long Chair in Business and Public Policy. Previously he was at the Nelson A. Rockefeller College of Public Affairs and Policy, State University of New York at Albany, where he was a professor of public administration, public policy, public health, and information science and served in a variety of University-level administrative positions. His previous experience includes six years as a program director and policy analyst at the National Science Foundation. Mumpower received his B.A. from the College of William and Mary and his Ph.D. in social and quantitative psychology from the University of Colorado, Boulder. He is author or editor of nine books and more than 50 book chapters and articles. His research has addressed basic and applied topics in negotiation and bargaining, environmental policy, individual and group decision-making processes, the use of scientific expertise in public policy making, and risk analysis and management. Address: Bush School of Government and Public Service, Texas A&M University, 1092 Allen Building, 4220 TAMU, College Station, TX 77843-4220; e-mail: jmumpower@bushschool.tamu.edu . M. Elisabeth Paté-Cornell (“ Analysis of National Strategies to Counter a Country's Nuclear Weapons Program ”) is the Burt and Deedee McMurtry Professor and Chair, Department of Management Science and Engineering at Stanford University. Her specialty is engineering risk analysis with application to complex systems (including space systems and medical systems). Her research has focused on explicit consideration of human and organizational factors in the analysis of failure risks and, recently, on the use of game theory in risk analysis. Applications in the last few years have included counterterrorism and nuclear counterproliferation problems. She is a member of the National Academy of Engineering and of several boards (Aerospace, Draper, InQtel, etc.). She was a member of the President's Intelligence Advisory Board until December 2008. She holds an engineer degree (Applied Math/CS) from the Institut Polytechnique de Grenoble (France), and an M.S. in Operations Research and a Ph.D. in Eng

About the Authors

Decision analysis and risk analysis have grown up around a set of organizing questions: what might go wrong, how likely is it to do so, how bad might the consequences be, what should be done to maximize expected utility and minimize expected loss or regret, and how large are the remaining risks? In probabilistic causal models capable of representing unpredictable and novel events, probabilities for what will happen, and even what is possible, cannot necessarily be determined in advance. Standard decision and risk analysis questions become inherently unanswerable ("undecidable") for realistically complex causal systems with "open-world" uncertainties about what exists, what can happen, what other agents know, and how they will act. Recent artificial intelligence (AI) techniques enable agents (e.g., robots, drone swarms, and automatic controllers) to learn, plan, and act effectively despite open-world uncertainties in a host of practical applications, from robotics and autonomous vehicles to industrial engineering, transportation and logistics automation, and industrial process control. This article offers an AI/machine learning perspective on recent ideas for making decision and risk analysis (even) more useful. It reviews undecidability results and recent principles and methods for enabling intelligent agents to learn what works and how to complete useful tasks, adjust plans as needed, and achieve multiple goals safely and reasonably efficiently when possible, despite open-world uncertainties and unpredictable events. In the near future, these principles could contribute to the formulation and effective implementation of more effective plans and policies in business, regulation, and public policy, as well as in engineering, disaster management, and military and civil defense operations. They can extend traditional decision and risk analysis to deal more successfully with open-world novelty and unpredictable events in large-scale real-world planning, policymaking, and risk management.

Decision support models help structure and inform complex choices under uncertainty. Two classic models are risk analysis and decision analysis. Risk analysis is understood here as risk characterization, and in some cases, the identification and benefit assessment of some risk management options. It is based on systems analysis and probability, and it excludes the actual decision phase, which requires the preferences, e.g., the utility function, of the decision maker(s). Risk analysis and decision analysis have some similarities and are often complementary. To model uncertainties, both rely on probability, generally a subjective Bayesian degree of belief. A decision analysis can include a risk analysis component, and the design of a risk management plan may require decision analysis support. The challenge for risk analysts is to characterize potential failure problems before decision options have been identified, and when there is no single decision maker, or group of decision makers, who can provide preference functions and degrees of belief. Yet, a correct and complete model of uncertainties in the probabilistic risk analysis phase is important if the results are to be used later for decision support, especially when the number of systems involved and the duration of their operations is unknown. In this paper, we explore some of the challenges inherent to probabilistic risk analysis that should be of interest to the decision analysts who intend to use risk analysis results.

About the Authors

Contributors

For many decision and risk analysis problems, probabilistic modelling of uncertainties provides key information for decision-makers. A common challenge is lacking relevant historical data to quantify the models used in decision and risk analyses. Therefore, experts are often sought to assess uncertainties in cases of incomplete or non-existing historical data. As experts might be prone to cognitive fallacies, a structured approach to expert judgement elicitation is encouraged with the aim to mitigate such fallacies. Further, it enhances the assessment’s transparency. An area, in which the assessment and modelling of uncertainties are particularly challenging due to incomplete or non-existing historical data is adversarial risk analysis (ARA). In contrast to more traditional application areas of decision and risk modelling, in ARA intelligent adversaries add more complexity to assessing uncertainties given that their behaviour and motivations can be versatile so that they adapt and react to decision-makers’ actions, including actions based on traditional risk assessments. This often inhibits the availability of historical data. This additional complexity is also shown by the challenges that machine learning methods face when informing adversarial risk assessments. As such, using expert judgements for assessing adversarial risk (at least supplementary) often provides a more robust decision. In this chapter, we discuss the importance of structured expert judgement for ARA and present an application of the Classical Model as a structured way for eliciting uncertainty from experts on geo-political adversarial risks. We elicit the frequency of terrorist attacks and strikes, riots and civil commotions (SR & CCs), including insurgencies and civil wars, in various global regions of interest. Assessing such uncertainties is of particular interest for insurance underwriting.

. This chapter is concerned with the aggregation of probability distributions in decision and risk analysis. Experts often provide valuable information regarding important uncertainties in decision and risk analyses because of the limited availability of hard data to use in those analyses. Multiple experts are often consulted in order to obtain as much information as possible, leading to the problem of how to combine or aggregate their information. Information may also be obtained from other sources such as forecasting techniques or scientific models. Because uncertainties are typically represented in terms of probability distributions, we consider expert and other information in terms of probability distributions. We discuss a variety of models that lead to specific combination methods. The output of these methods is a combined probability distribution , which can be viewed as representing a summary of the current state of information regarding the uncertainty of interest. After presenting the models and methods, we discuss empirical evidence on the performance of the methods. In the conclusion, we highlight important conceptual and practical issues to be considered when designing a combination process for use in practice. Introduction Expert judgments can provide useful information for forecasting, making decisions, and assessing risks. Such judgments have been used informally for many years. In recent years, the use of formal methods to combine expert judgments has become increasingly commonplace. Cooke (1991) reviews many of the developments over the years as attempts have been made to use expert judgments in various settings.