Assessing the Value of Structured Analytic Techniques in the U.S. Intelligence Community

Abstract

Abstract : The Intelligence Community (IC) is strongly emphasizing the use of structured analytic techniques (SATs) to promote rigorous analysis, lessen the risk of intelligence failure, and make analysts reasoning more transparent to consumers. So far, however, the IC has made little effort to assess whether SATs in general or specific SATs are improving the quality of analysis. One primarily qualitative method of evaluating these techniques would be periodic in-depth reviews of IC production on a variety of topics to ascertain how frequently SATs are used, whether they advance the attainment of IC analytic quality standards, and which specific SATs are most effective. RANDs preliminary SAT review of a limited sample of Central Intelligence Agency (CIA), Defense Intelligence Agency (DIA), and National Intelligence Council (NIC) publications finds that the minority of them employing SATs addressed a broader range of potential outcomes and implications than did other analyses, although the logic behind specific techniques was not always transparent. More-comprehensive SAT reviews, along with interviews of analysts, managers, and consumers, could determine how effectively IC agencies are employing SATs and compile lessons learned on best practices and pitfalls of their use in intelligence analysis. In addition, quantitative research could examine the extent to which the use or nonuse of SATs correlates with the quality of analytic output as measured by the Office of the Director of National Intelligences (ODNIs) Office of Analytic Integrity and Standards and by individual agencies, while controlled experiments could test the use of SATs under conditions similar to those prevailing in the IC.