Spelling and “Analytic Tradecraft”
03.06.2008 - 11:16 AMThe CIA and U.S. intelligence have gotten a lot of things wrong in recent years, at great cost to our national well-being. A significant part of the problem lies in “analysis,” where data is supposed to be interpreted but is all too often misinterpreted.
Gregory F. Treverton and C. Bryan Gabbard have written a new study of “analytic tradecraft,” published by RAND, that takes up the nature of the problem and looks at some of the solutions being put in place.
Some of the approaches to improving analysis they point to are technological. For example, there is a program called GENOA -II, designed to help intelligence analysts work better in groups. Among other things, it attempts to “automate team processes,” develop “cognitive aids that allow humans and machines to ‘think together’ in real-time about complicated problems,” and find ways to “overcome the biases and limitations of the human cognitive system.”
This sounds great. But count me deeply skeptical. Here’s why.
No technological solution can be better than the people running it. Consider a very simple “cognitive aid” like a computer spell-check program. These things have been around for a long time and everyone uses them. Treverton and Gabbard are smart men, who have every interest in producing a highly professional study. Treverton has handled all of Europe for the National Security Council and served as vice chair of the National Intelligence Council, overseeing the writing of America’s National Intelligence Estimates (NIEs). Gabbard is also an accomplished person, with a wealth of experience under his belt. But even so, and even with RAND editors poring over their study before it was released, the spell-check program was not fail-safe.
The Treverton-Gabbard study has:
“intellience” and “intellence” instead of intelligence;
“builiding” instead of building;
“proceess” instead of process;
“solftware” instead of software;
“uniue” instead of unique;
“syehtsis” instead of synthesis;
“coopertive” instead of cooperative;
“poential” instead of potential.
Why should there be nine such mistakes when the technology is in place to produce, almost effortlessly, a zero-error rate? The United States is not going fall victim to a surprise attack because of some typos in a RAND study. But we will fall victim to another surprise attack if don’t focus on the fact that the problem facing our intelligence community is not technology but severe shortcomings in the selection of analysts themselves.
See the case of Michael Scheuer, the kooky head of the CIA’s Osama bin Laden desk in the 1990’s, for one set of illustrations. See the case of Richard Immerman, the radical professor now in charge of analytic “integrity and standards” for the Intelligence Community, for another set of illustrations.
How many more illustrations do we need?
| »Back to Connecting the Dots | »Back to Commentary |
March 6th, 2008 at 1:05 PM
Damn! I KNEW it meant catastrophe when CENTCOM was spelling it “Usama bin Laden” and CIA was spelling it “Osama bin Ladin.” Probably caused the whole 9/11 shebang.
(Actually, we referred to bin Laden at CENTCOM as “Waldo,” as in “Where’s Waldo?”)
The GENOA-II brief is an amusing walk down memory lane. There actually is something to using technology to achieve better analysis. GENOA’s evolution included positive outcomes from collaborative, cross-agency analysis schemes (e.g., routine networked collaboration), and from data-mining/data-sifting programs that (as with ABLE SENTRY) revealed connections and correlations in terrorist networks that humans, unaided, would not find in a lifetime. There’s some real there there.
The problem with guys like Scheuer and Immerman isn’t analysis, it’s political leanings and spin. I keep using the 2007 Iran NIE to illustrate this, because it works so well. There WAS a discrete, definable event in 2003 that amounted to Iran pulling back from the more detectable elements of its nuclear weaponization program. Receiving the report of this in 2006 helped analysts make sense of a LACK of indicators along these lines after 2003. (There is never anyone waving a big red flag that reads, unambiguously, “OK, we’ve stopped — or STARTED — weaponizing now!” Until an actual detonation, analysts have to interpret collateral indicators, or — more difficult — the absence of them.)
The context of this 2003 event could be presented in different ways. The way I would present it would be as follows: we know from IAEA inspections and our own intelligence how much farther Iran has progressed in uranium enrichment since 2003. Progress has been significant — even dramatic. We know from our own intelligence that Iran has not ceased to develop and test ballistic missiles (the delivery systems for nuclear warheads). We also know that Iran could have resumed some portion of its weaponization effort — making a functioning warhead — since 2003. If Iran moves slowly enough, and carries the cost of doing much of the work underground, a significant segment of this effort can remain undetected. We also know, from Iran’s posture with IAEA and the EU-3, that Iran has every intention of exploiting the negotiation process to gain time — as we have seen with its persistence in perfecting uranium enrichment through the series of negotiations and inspections. We can thus reasonably suspect that some weaponization activity has resumed, and that it warrants the greatest risks we can take, to gain better intelligence on it. We can still expect Iran to be capable of detonating a nuke as early as 2009, and have a functioning weapon by 2015. We can, finally, conclude that in suspending its weaponization effort in 2003 (and simulating a more cooperative stance with IAEA that fall), Iran was most likely reacting to the US invasion of Iraq.
(Let us also recognize that in spite of all we are certain that we know about Iran’s uranium enrichment and ballistic missile programs, if Iran had long enough to clean out its storage and R&D sites, we could be made guilty of an intelligence failure for our current certainty that any of the activity is related to WEAPONS. North Korea and Iraq have taught that lesson well.)
As we know, of course, the 2007 NIE presented a different context for the 2003 data point. But this is not because analysts don’t know about all the elements of the context I just outlined. Analysts haven’t failed to recognize those indicators. This is NOT the “failure” they are accused of perpetrating before 9/11. Instead, even though anyone with Google can unearth all the context I have outlined, the IC leadership chose to either ignore or barely mention these factors, in making the 2003 data point public. The only ANALYTICAL difference between my summary and the NIE’s is the conclusion about WHY Iran suspended weaponization in 2003. Since we actually don’t know why, and we’re all making deductions about that, no one can be taken to the woodshed for having an analytical difference of opinion.
In only a single sense can the NIE be denounced for literally getting a point of substance wrong, and that is its assertion that Iran was reacting, in 2003, to diplomatic pressure. We don’t know for sure what Iran was reacting to, but John Bolton pointed out, shortly after the NIE’s executive summary was released, that almost no diplomatic pressure was being applied to Iran in the relevant time frame. The diplomatic PRESSURE (threats of sanctions) was applied afterward. Bolton piece here:
http://www.washingtonpost.com/wp-dyn/content/article/2007/12/05/AR2007120502234.html
Our greatest problem with intelligence today is not that we are literally missing indicators or failing to recognize their potential import, although that does happen (and prudent, well-directed steps to minimize it are always appropriate). Our problem is that valid and essentially adequate intelligence findings are being spun, for politics’ sake, from inside the IC. The event that has made that not only more possible but inevitable is the 2005 IC reorganization, which gave us a DNI office staffed by Bush’s old enemies from State and CIA.
March 6th, 2008 at 1:11 PM
I’m remembering a report that Linda Tripp was in an $80K+ job in the Pentagon despite the fact that her resume basically was empty of any real credentials. Of course in the case of Tripp there may have been logic to her selection because she was crafty enough to plan an operation that almost brought down a President. In fact, its a pity no one writes operas any more because there is a good one on the subject of long plotted revenge in the Tripp story.
March 6th, 2008 at 2:41 PM
I’m remembering a report that Linda Tripp was in an $80K+ job
Along with the over qualified Monica at the Pentagon too.
March 6th, 2008 at 8:50 PM
http://www.haaretz.com/hasen/spages/961337.html
Gabe,
Thought you’d like this. You are mentioned in the article.
March 6th, 2008 at 8:51 PM
She was a clerical assistant, with clearances from the time she worked with the Special Forces.
Monica on the other hand, had a security clearance, because of her ahem special talents.
March 8th, 2008 at 7:33 AM
All,
Let’s talk about projection and see if we can find two mistakes in the following passage:
“The United States is not going fall victim to a surprise attack because of some typos in a RAND study. But we will fall victim to another surprise attack if don’t focus on the fact that the problem facing our intelligence community is not technology but severe shortcomings in the selection of analysts themselves.”
Answers:
“The United States is not going [sic. ‘to’] fall victim to a surprise attack because of some typos in a RAND study. But we will fall victim to another surprise attack if [sic. ‘we’] don’t focus on the fact that the problem facing our intelligence community is not technology but severe shortcomings in the selection of analysts themselves.”
The Lesson:
Luke 6:42, “Or how can you say to your brother, ‘Brother, let me take out the speck that is in your eye,’ when you yourself do not see the log that is in your own eye? You hypocrite, first take the log out of your own eye, and then you will see clearly to take out the speck that is in your brother’s eye.”