Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis


Authors
Alexandru Marcoci
University of North Carolina, Chapel Hill
Ariel Kruger
University of Melbourne
Fiona Fidler
University of Melbourne
1 more
Abstract
Background. The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the US Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Method. Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results. Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusions. Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.
Keywords intelligence analysis  intelligence failures  intelligence reform  IRTPA  ICD203  tradecraft standards  inter-rater reliability
Categories (categorize this paper)
Reprint years 2019
DOI 10.3389/fpsyg.2018.02634
Options
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

Our Archive


Upload a copy of this paper     Check publisher's policy     Papers currently archived: 46,405
External links

Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library

References found in this work BETA

Add more references

Citations of this work BETA

No citations found.

Add more citations

Similar books and articles

Natural Problems and Artificial Intelligence.Tracy B. Henley - 1990 - Behavior and Philosophy 18 (2):43-55.
Consciousness, Intentionality, and Intelligence: Some Foundational Issues for Artificial Intelligence.Murat Aydede & Guven Guzeldere - 2000 - Journal of Experimental and Theoretical Artificial Intelligence 12 (3):263-277.
A Viable Model and Self-Report Measure of Spiritual Intelligence.David B. King & Teresa L. DeCicco - 2009 - International Journal of Transpersonal Studies 28 (1):68-85.
The Case for Integrated Intelligence.Marcus Anthony - 2008 - World Futures 64 (4):233 – 253.
Mandevillian Intelligence.Paul R. Smart - 2018 - Synthese 195 (9):4169-4200.

Analytics

Added to PP index
2018-12-09

Total views
22 ( #423,789 of 2,286,206 )

Recent downloads (6 months)
9 ( #106,813 of 2,286,206 )

How can I increase my downloads?

Downloads

My notes

Sign in to use this feature