a citizen’s journal by Thomas Nephew

On the limits of economic analysis in occupation policy evaluation

Posted by Thomas Nephew on September 29th, 2007

On the excellent “Nation Building” site, Aziz Poonawalla points to a post by economist and “Freakonomics” author Steven Levitt discussing MIT economist Michael Greenstone’s analysis of Bush’s so-called “surge” in Iraq (“Is the ‘Surge’ Working? Some New Facts,” 9/14/07, Dept. of Economics working paper).

Levitt and Poonawalla rightly praise Greenstone’s efforts to carefully analyze the “surge” using the data at his disposal. To Levitt, the most interesting part is Greenstone’s analysis of bond markets, which indicate waning investor confidence in Iraq’s future as a political entity. Greenstone says his findings suggest that “to date, the Surge is failing to pave the way toward a stable Iraq and may in fact be undermining it.” But Levitt leads with more upbeat news:

On at least one dimension, there is strong evidence of progress. The data clearly suggest that deaths of civilians in Baghdad have fallen…*

In Greenstone’s words:

…Specifically, the upward trend in fatalities in the pre-period is evident. Most notably, there is a sharp downward break in trend after the initiation of the Surge. The figure provides compelling evidence of a decline in civilian fatalities after the Surge‚Äôs initiation.

Overall, the civilian fatality results demonstrate that the Surge is associated with an important decline in civilian fatalities.11 It is certainly possible that some of these results are due to the reported increases in migration out of Baghdad and Iraq more generally, but the sharpness of the results makes it unlikely that that this is the entire story. In many respects, this result confirms the popular theory of domestic policing that putting more cops on the beat reduces the crime rate.

Strictly speaking, there aren’t any false sentences in this excerpt. The data at least arguably show a “sharp” decline; it happened around the time of the “Surge”; some other stuff happened, too, but the author doesn’t think it explains the sharp decline; this reminds one of community policing. The question is, how much significance to attach to the “sharpness of the results,” how much to “other stuff”, and whether all the “other stuff” is accounted for.

The rabbit I’ll pull out of the hat now will be familiar to those who’ve recently read this blog: “Progress is just another word for nothing left to kill” (9/14/07). Follow the link or scroll down; by comparing maps provided under oath** to Congress by General Jones and General Betray Us pardon me, Petraeus, it was plain that the latter’s presentation concealed a very significant fact in the Baghdad story: the progress of ethnic cleansing and separation in Baghdad neighborhoods. While it’s not definitive, it certainly suggests that an alternative explanation for the bulk of the decline in civilian casualties in Baghdad was simply that there were fewer and fewer neighborhoods that “needed” sectarian scaring and cleansing.

What about that sharp decline? First off, looking at the figure (#3, for all of Iraq), it doesn’t look all that sharp to me. More importantly, Greenstone’s methodology seems designed to capture a pre-post treatment effect — that is, it rather assumes what it sets out to demonstrate. You could probably fit all kinds of other curves to the data — for example, ones based on models that factor in the number of Iraqis with bull’s eyes on their backs: that is, those still living in ethnically mixed neighborhoods or on the borders of segregated neighborhoods.

But leave that aside, and stipulate there was some kind of sharp “change in trend” in casualties. coinciding with the “surge.” Well: so what? No one ever suggested that dropping thousands of additional U.S. troops in Baghdad and around the country wouldn’t have some effect. Take some smooth curve over time, subtract some constant from it after time “S”, and the course of that trend line will reflect a sudden change — whether or not the actual amount subtracted by the surge per se is a significant part of the story.

There’ll always be another factor to consider, and I couldn’t have done what Greenstone did even without factoring in Baghdad or Iraqi sectarian geography. To repeat, Greenstone doesn’t say anything factually wrong in the above. I just think he oversells the significance of his casualty data observations. They’re consistent with a “surge” effect — but they don’t require it, and Greenstone may overestimate it given other factors.

Aziz’s post, on the other hand, is titled “Is the Surge Working? Ask the Data, Not the Politicians.” And that actually is wrong.

I know what he meant: look at the facts, don’t listen to bloviations from Capitol Hill. But as the above discussion seeks to demonstrate, facts are funny things, and new ones come along all the time. Deciding which facts about an issue like Iraq are important and which ones aren’t is more than half the battle — and even a well spoken, thorough economist may not make the right choices. That’s actually what I elect politicians for — with indifferent success, to be sure.

Fundamentally, any worthwhile analysis would have to consult the data and the politicians — particularly the ones living in the middle of the data. The point of the ‘surge’ was ostensibly to quickly provide some time for Iraqi politicians to achieve some stability, come to some agreements, settle differences. The fact that all of them left town in August instead was the plainest indication possible that the surge was having, if anything, the opposite political effect to the one intended.

And that might be about all the analysis you really needed about the “surge.”

* I’ve shortened the quote in order to focus on the first part of it. The rest of the quote is “…, and there is no evidence that the crackdown in Baghdad has shifted violence to the rest of the country.” I have no opinion on that at this time.
** I mention this because of that pesky “the truth, the whole truth, and nothing but the truth” phrase. Jones provided the accurate maps of Baghdad ethnic composition, Petraeus did not, resulting in a significantly misleading account of the ‘surge’ — or escalation, to call it by name rather than euphemism.

5 Responses to “On the limits of economic analysis in occupation policy evaluation”

  1. Nell Says:

    “No evidence that the crackdown in Baghdad has shifted violence to the rest of the country.”
    I have an opinion on that. It’s informed by an atrocity that seems to have passed out of the public’s mind entirely: the massive bombings in Yezidi communities in northern Iraq (Kurdestan to the ruling party), the single biggest killing of civilians in one day since the invasion began.
    I don’t see how anyone can agree to the quoted statement unless they’re willing to remove that day as an “outlier”. (Even if so, I think there’s additional evidence that fighters moved out to the provinces north of Baghdad during the surge. But needless to say, I’m not prepared to cross stats with the article right now, even if I thought there were some point in doing so.)

  2. Nell Says:

    Damn, now they’ve got me saying ‘surge’. Escalation!
    Good post.

  3. Thomas Nephew Says:

    Yes, the Yezidi attacks were almost certainly ignored by the Petraeus graphic. I don’t know whether this guy did, but he took a lot of his numbers from the IraqBodyCount tally, and I don’t think they ignored those atrocities. I’ll see if I can figure it out.

  4. narciso Says:

    Of course they weren’t ignored, That’s what drove the death toll
    over a thousand. You think Shia
    only had lived in Karrada & Saddam
    (now Sadr City) like Kurds had not lived in Mosul pre-Anfal or the
    people of the Marshlands like the

  5. Thomas Nephew Says:

    “Of course they weren’t ignored”… by whom — Petraeus or IraqBodyCount? I assume you mean IBC.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> -- (comment rules)