Teaching Bayesian Statistics To Intelligence Analysts: Lessons Learned

Abstract

The Community must develop and integrate into regular use new toolsthat can assist analysts in filtering and correlating the vast quantities of information that threaten to overwhelm the analytic process...—Commission the Intelligence Capabilities of the United StatesRegarding Weapons of Mass Destruction (The WMD Report)1Unlike the other social sciences and, particularly, the physical sciences, where scientists get to choose the questions they wish to answer and experiments are carefully designed to confirm or negate hypotheses, intelligence analysis requires analysts to deal with thedemands of decision makers and estimate the intentions of foreign actors, criminals or business competitors in an environment filled with uncertainty and even deliberate This article is available in Journal of Strategic Security: http://scholarcommons.usf.edu/jss/vol2/iss1/3 39 Teaching Bayesian Statistics To Intelligence Analysts: Lessons Learned Kristan J. Wheaton Jennifer Lee Hemangini Deshmukh Mercyhurst College The Community must develop and integrate into regular use new tools that can assist analysts in filtering and correlating the vast quantities of information that threaten to overwhelm the analytic process... —Commission the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction (The WMD Report)1 From the 9/11 Commission Report to the WMD Report to the Intelligence Reform and Terrorism Prevention Act of 2004, there has been renewed emphasis an improvement in intelligence analysis and the tools and methods that analysts use to create their estimates. Congress, frustrated with recent intelligence failures that were linked, at least partially, to poor analysis, decided to legislate certain aspects of the analytic process by mandating the use of method commonly called Red Teaming in the 2004 Act: Not later than 180 days after the effective date of this Act, the Director of National Intelligence shall establish process and assign an individual or entity the responsibility for ensuring that, as appropriate, elements of the intelligence community conduct alternative analysis (commonly referred to as ''red-team analysis'') of the information and conclusions in intelligence products. —Intelligence Reform and Terrorism Prevention Act of 20042 This and other methods used routinely by intelligence analysts suffer from number of problems. Unlike the other social sciences and, particularly, the physical sciences, where scientists get to choose the questions they wish to answer and experiments are carefully designed to confirm or negate hypotheses, intelligence analysis requires analysts to deal with the demands of decision makers and estimate the intentions of foreign actors, criminals or business competitors in an environment filled with uncertainty and even deliberate deception. For these reasons, those unfamiliar with the challenges of this discipline often criticize intelligence analysis methods: Wheaton et al.: Teaching Bayesian Statistics To Intelligence Analysts: Lessons Le Produced by The Berkeley Electronic Press, 2009 Journal of Strategic Security 40 It's time to require national security analysts to assign numerical probabilities to their professional estimates and assessments as both matter of rigor and of record. Policymakers can't weigh the risks associated with their decisions if they can't see how confident analysts are in the evidence and conclusions used to justify those decisions. The notion of imposing intelligence accountability without intelligent counting—without numbers—is fool's errand. —Michael Schrage, Senior Advisor To MIT's Security Studies Program3 Scientists base their criticisms deep understanding of the power of statistics. Traditional statistics—the kind of statistics that one commonly studies in undergraduate or graduate programs—is based largely normal distributions, structured data sets, linear regression analysis, null hypothesis testing and the like. It produces stunning results and is responsible for many of the advances in the hard and soft sciences. This type of analysis, the other hand, is normally considered inappropriate for intelligence analysis. The data collected for intelligence analysis are largely unstructured, often incomplete or deceptive, and rarely capable of interpretation by these scientifically acceptable methods. Even the most quantitatively oriented analysts acknowledge that there are some, perhaps many, problems that do not lend themselves to numerical analysis.4 One response to this issue is to claim that intelligence analysis is mere guesswork, unscientific and prone to massive failures. Another response is to look for alternative structured methods, methods that have basis in science but allow for the uncertainty of the traditional intelligence data set. One of the methods often recommended to analysts by the scientific community is to apply the statistical theory of the 19th century clergyman and mathematician, Thomas Bayes. Schrage recommends Bayes Theorem and calls it powerful tool to weigh evidence.5 Bruce Blair of the Center For Defense Information also advocates Bayes to the intelligence community and calls it rigorous approach.6 Bayes, non-conformist Minister and Fellow of the Royal Society, is largely remembered today for his work non-traditional statistical problems.7 Specifically, the Bayesian Method depends on taking some expression of your beliefs about an unknown quantity before the data was available and modifying them [the beliefs] in light of the [new] data.8 Such an approach appears to have immediate applicability to intelligence problems. Previous analyses, or an analyst's own experience, often proJournal of Strategic Security, Vol. 2 No. 1 http://scholarcommons.usf.edu/jss/vol2/iss1/3 DOI: http://dx.doi.org/10.5038/1944-0472.2.1.3 Teaching Bayesian Statistics To Intelligence Analysts: Lessons Learned 41 vide an intuitive idea of the range of probabilities inherent in particular problem. While Bayes has its detractors, the method provides the math that would allow analysts to update their current beliefs in logically consistent and scientifically defensible way. The difficulty, as always, lies in the details. People do not appear to be natural Bayesians, i.e. they do not seem to follow Bayesian reasoning when making decision.9 Furthermore, Bayes seems difficult to teach. It is generally considered to be advanced statistics and, given the problem that many people (including intelligence analysts) have with traditional elementary probabilistic and statistical techniques, such solution seems to require expertise not currently resident in the intelligence community or available only through expensive software solutions. The intelligence community has toyed with Bayes before. In 1978, Nicolas Schweitzer reported an experiment using Bayesian analysis.10 At the time, according to Schweitzer, the technique was already used extensively in imagery analysis and was thought (by no less than the then Director of Central Intelligence, William Colby) to have potential in political analysis. The experiment, in fact, confirmed the utility of Bayes in less technical analysis and despite several misgivings, including, among others, problems with data, problems over time and the potential for manipulation, Schweitzer pronounced the technique a useful adjunct to traditional analysis.11 Despite this endorsement, Schrage's modern critique tracks well with one of the author's own experience: Most analysts have never heard of Thomas Bayes, much less possess the ability to apply his theories to non-technical analytic problems. This paper reports efforts to change that. Can Bayes be simplified such that it can be taught? More importantly, can Bayes be taught in such way that entry-level analysts can use the method at any time and under any circumstances? This paper outlines series of studies and experiments over the last year that try to do just that and, based those experiments, suggests new lines of research that might well provide the answers to these problems. Addressing the needs of junior analysts has never been as important as it is now. Due primarily to hiring freeze in the 1990's, analysts inside the intelligence community tend to be either quite senior (and near retirement) or quite junior (hired within the last five years). For example, if Congress allows the FBI to hire all of the analysts it needs to replace those retiring in 2007, approximately one third of its analysts would have less Wheaton et al.: Teaching Bayesian Statistics To Intelligence Analysts: Lessons Le Produced by The Berkeley Electronic Press, 2009 Journal of Strategic Security 42 than two years experience.12 If Bayes is potential solution, then figuring out how to teach it to young analysts is critical concern. A Brief Introduction To Bayes Formally, Bayes Theorem13 looks like this: