Analytic Tradecraft Research Articles

Cyber Threat Intelligence meets the Analytic Tradecraft

Cyber Threat Intelligence meets the Analytic Tradecraft

Towards Trustworthy AI-Enabled Decision Support Systems: Validation of the Multisource AI Scorecard Table (MAST)

The Multisource AI Scorecard Table (MAST) is a checklist tool to inform the design and evaluation of trustworthy AI systems based on the U.S. Intelligence Community’s analytic tradecraft standards. In this study, we investigate whether MAST can be used to differentiate between high and low trustworthy AI-enabled decision support systems (AI-DSSs). Evaluating trust in AI-DSSs poses challenges to researchers and practitioners. These challenges include identifying the components, capabilities, and potential of these systems, many of which are based on the complex deep learning algorithms that drive DSS performance and preclude complete manual inspection. Using MAST, we developed two interactive AI-DSS testbeds. One emulated an identity-verification task in security screening, and another emulated a text-summarization system to aid in an investigative task. Each testbed had one version designed to reach low MAST ratings, and another designed to reach high MAST ratings. We hypothesized that MAST ratings would be positively related to the trust ratings of these systems. A total of 177 subject-matter experts were recruited to interact with and evaluate these systems. Results generally show higher MAST ratings for the high-MAST compared to the low-MAST groups, and that measures of trust perception are highly correlated with the MAST ratings. We conclude that MAST can be a useful tool for designing and evaluating systems that will engender trust perceptions, including for AI-DSS that may be used to support visual screening or text summarization tasks. However, higher MAST ratings may not translate to higher joint performance, and the connection between MAST and appropriate trust or trustworthiness remains an open question.

Structured Analytic Techniques: A Pragmatic Approach

Cognitive scientists question the effectiveness of structured analytic techniques (SATs), concluding there is no evidence they reduce biases. In some cases, SAT usage might even add new biases. Yet the U.S. Intelligence Community remains committed to promoting these techniques to improve analytic tradecraft, although some evidence suggests many analysts still do not use them. Practitioners themselves often see value in SATs but doubt they substitute for expertise and intuition. The IC needs a better understanding of how SATs improve analysis, and particularly, how they help generate insight, the key value-added for intelligence products. This article sizes up the current debate and offers practical suggestions on how SATs might best be used in light of current research.

The occasional maverick of analytic tradecraft

Roughly a decade ago, in a commissioned report I wrote for the US National Research Council’s Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Secur...

Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis.

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products.Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments.Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability.Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.

ODNI as an analytic ombudsman: is Intelligence Community Directive 203 up to the task?

ABSTRACT In the wake of 9/11 and the war in Iraq, the Office of the Director of National Intelligence adopted Intelligence Community Directive (ICD) 203 – a list of analytic tradecraft standards – and appointed an ombudsman charged with monitoring their implementation. In this paper, we identify three assumptions behind ICD203: (1) tradecraft standards can be employed consistently; (2) tradecraft standards sufficiently capture the key elements of good reasoning; and (3) good reasoning leads to more accurate judgments. We then report on two controlled experiments that uncover operational constraints in the reliable application of the ICD203 criteria for the assessment of intelligence products.

Computational social science and intelligence analysis

Computational Social Science (CSS) is an emerging, interdisciplinary approach to the study of social systems. This chapter provides readers with an introduction to CSS, and discusses why examining the behavior of individuals and groups in social systems from an algorithmic perspective provides new and exciting analytic opportunities for the Intelligence Community and analytic tradecraft. Through the use of artificial societies, commonly referred to as Agent-Based Models (ABMs), intelligence analysts can improve strategic intelligence assessments by capitalizing on the scientific and tradecraft merits of computational simulation.

The US Government and the Italian coup manqué of 1964: the unintended consequences of intelligence hierarchies

In the summer of 1964, Italian security forces and the President of the Republic attempted to remove the US-backed Italian center-left government. The attempt did not succeed, but the threat to do so was used to curtail the government’s reformist program. This article shows that the State Department and the CIA misunderstood the plans of the Italian President and security officers, dismissing the possibility of a forceful removal of the center-left, despite having a long-standing hierarchical relationship with Italian intelligence. US officials failed because of poor analytic tradecraft and because of two unintended consequences of international intelligence hierarchies: an excessive reliance on liaison over penetrations and the increased freedom of maneuver of Italian intelligence when faced with multiple, competing principals.

Choosing an SAT with SAnTA: A Recommender System for Informal Workplace Learning

Current intelligence community analytic standards encourage the use of structured analytic techniques; these are taught in training programs and supported by analytic tradecraft cells. Yet resources are scarce, timely personalized help is not available to all when it is most needed, and conditions for the improper selection and misapplication of these techniques prevail. The Structured Analytic Technique Advisor (SAnTA) is an electronic interactive job aid that recommends structured analytic techniques to analysts, based on the current state of their analysis and on the synthesized expertise of tradecraft staff. SAnTA is undergoing formative evaluation and iterative development. When released, SAnTA will provide individualized support to a large number of analysts every day. This article is available in Journal of Strategic Security: http://scholarcommons.usf.edu/jss/vol8/iss5/8

Analytic Tradecraft and the Intelligence Community: Enduring Value, Intermittent Emphasis

The emphasis and visibility afforded analytic tradecraft in the Intelligence Community's analytic production has fluctuated throughout its existence. The explanation for this intermittent emphasis on tradecraft may lie in changes in consumer preferences and collection means, the role played by individual tradecraft advocates, and the lack of an intelligence failure matching the severity of 9/11 and the absence of weapons of mass destruction in Iraq. Both failures served as a forceful reminder that while strong analytic tradecraft does not guarantee ‘getting a judgment right’, it increases the likelihood that the assessments produced are transparent, relevant, and rigorous.

Assessing Uncertainty in Intelligence

This article addresses the challenge of managing uncertainty when producing estimative intelligence. Much of the theory and practice of estimative intelligence aims to eliminate or reduce uncertainty, but this is often impossible or infeasible. This article instead argues that the goal of estimative intelligence should be to assess uncertainty. By drawing on a body of nearly 400 declassified National Intelligence Estimates as well as prominent texts on analytic tradecraft, this article argues that current tradecraft methods attempt to eliminate uncertainty in ways that can impede the accuracy, clarity, and utility of estimative intelligence. By contrast, a focus on assessing uncertainty suggests solutions to these problems and provides a promising analytic framework for thinking about estimative intelligence in general.

Assessing Uncertainty in Intelligence

This article addresses the challenge of managing uncertainty when producing estimative intelligence. Much of the theory and practice of estimative intelligence aims to eliminate or reduce uncertainty, but this is often impossible or infeasible. This article instead argues that the goal of estimative intelligence should be to assess uncertainty. By drawing on a body of nearly 400 declassified National Intelligence Estimates as well as prominent texts on analytic tradecraft, this article argues that current tradecraft methods attempt to eliminate uncertainty in ways that can impede the accuracy, clarity, and utility of estimative intelligence. By contrast, a focus on assessing uncertainty suggests solutions to these problems and provides a promising analytic framework for thinking about estimative intelligence in general.

Intelligence reform: The logic of information sharing

A cornerstone of US intelligence reform is ‘information sharing’ as a means of adapting to contemporary security challenges. It was a central recommendation of the 9/11 Commission, reflected in the wide-ranging ‘Information Sharing Environment’ mandated by the Intelligence Reform and Terrorism Prevention Act of 2004. Yet the underlying logic of information sharing for intelligence reform has received little attention. Drawing on information and communications theory, this paper critiques the logic by highlighting problems of sense-making and interpretation overlooked amid the scholarly enthusiasm for an intelligence ‘culture of sharing’. With their impersonal, technical, and highly bureaucratic approach, today's reforms may favor the flow of information and its sheer volume at the expense of the context and analytic tradecraft that render it meaningful, actionable intelligence. For effective information sharing, the paper suggests reformers pay more attention to the socio-technical environment of analysis when interpreting ambiguous, uncertain information.

Effective Intelligence Analysis: The Human Dimension

ABSTRACT This paper reports a two stage investigation into the role that personality (NEO Five Factor Inventory – NEO), cognitive (Watson‐Glaser critical thinking appraisal – WGCTA), and interpersonal traits play in the production of accurate and well constructed intelligence advice. The first stage consisted of the creation of a multiple choice intelligence assessment device that assesses the essential elements of good analytic tradecraft. The second stage reports on the psychological study of a sample of trainee analysts to assess the contribution of the intellectual, personality and interpersonal traits of analysts to good analytic practice. The findings indicate that analysts need empathic skills over and above analytic skills in order to carry out the tasks identified as integral to good intelligence analysis.

Why secret intelligence fails

Acknowledgements Preface List of Abbreviations Chapter One - The Uncertainty Principle What Is Intelligence? Utility of Intelligence The Intelligence Cycle The Plan Chapter Two - Intelligence, American Style Historical Reflections The Intelligence Community The Ineffectual DCI The New Counterterrorism Focus Chapter Three - Pitfalls of American Style Intelligence The Intelligence Ethos Bureaucratic Pathologies Communication Problems Inflection Points Chapter Four - The Foreign Experience The British Model The Israeli Model The South African Model The French Model Points of Comparison Chapter Five - Requirements And Priorities Shoddy, Hyped, and False Intelligence How Is It Done? The Policy Environment Intelligence Community Dynamics The Analyst/Collector Environment Inflection Points Chapter Six - Perils of Intelligence Collection Collection Disciplines ORCON The Classification System An Ounce of Prevention Inflection Points Chapter Seven - Analytic Snafus The World of the Analyst The Estimative Process Politicized Intelligence Cognitive Issues Inflection Points Chapter Eight - Getting Intelligence to the Right People Producing Intelligence Disseminating Intelligence Using Intelligence Inflection Points Chapter Nine - Contributing Factors Counterintelligence Covert Action Inflection Points Chapter Ten - Toward Smarter Intelligence The Tragedy of Unrealistic Expectations Provide Strong leadership With Unity of Command Refine the Fusion Center Concept Integrate Domestic with Foreign Intelligence Promote Total Information Awareness Improve Analytic Tradecraft The Future of Warning? Glossary