A couple of days ago Jim Manzi posted a long and technical critique of my hypothesis that gasoline lead is strongly linked to the rise and fall of violent crime that we've experienced over the past half century. (Detailed in "Criminal Element" in our current issue.) It's the kind of critique that probably ought to be addressed by an expert, but unfortunately there don't seem to be any in my living room at the moment. Just me. So I'm going to respond myself, and hopefully others may respond in their own way later on.
A quick note: I spoke to Manzi while I was preparing my article on the lead-crime hypothesis, and I've also read Uncontrolled, his excellent book about the inherent problems with econometric analysis (review here). So I'm not surprised that he has some pushback. Nonetheless, I think he pushes back too much.
The rest of this is likely to get long and a little wonky, and it doesn't contain any fascinating new factlets about lead that I left out of my magazine piece. For that reason, I'm going to put it below the fold. However, if you make it all the way to the end, there's an irony to our disagreement that you might find amusing. Click the link for more.
Let's start off with Manzi's basic critique. Suppose I gathered up the records of a thousand school children and discovered that heavier kids scored better on standardized math tests. As a result, I recommended to parents that they feed their kids lots of rich, fatty food if they want them to grow up to be scientists. What would you think of my advice?
If you're smart, you'd think I'm an idiot. As kids get older, they weigh more. They also do better on math tests. I haven't discovered a link between weight and math ability. All I've discovered is the obvious fact that older kids know more math.
The usual way to handle this is to control for age. That is, I need to find out if kids of the same age show the same relationship, namely that heavier ones are better at math. Suppose I did that, and it turned out they are. Am I vindicated?
Not quite. It's possible, for example, that kids who like math are more sedentary than kids who don't. That makes them heavier. The chain of causation doesn't go from weight to math, it goes from fondness for math to scores on math tests. But fondness for math also makes you heavier.
So can we control for that too? Sure. But Manzi's point is that we simply never know if we've controlled for everything. In some settings, like tests of new pharmaceuticals, we can limit the number of variables enough that we can probably control for everything important. But in complex, real-life settings, there's just too much going on. No matter how much we control for, we simply don't know how everything interacts or whether there's some hidden variable that we haven't thought about.
This is at the heart of his critique of the lead-crime hypothesis. Crime is about as complex as social problems come, and trying to tease out the causes using historical data and econometric analysis is very, very hard. I don't think anyone would argue with him on that score.
Still, a generic critique of population-based econometric studies isn't enough. We need to look at the actual studies themselves. So Manzi takes a look at what he calls my "key econometric source," a paper by Jessica Reyes that takes a clever approach to controlling for unknown factors in the lead-crime hypothesis. It turns out that, for fairly random reasons, some states adopted unleaded gasoline faster than others. So if the lead-crime theory is right, you'd expect those states to also see a faster decline in violent crime rates. And that's what she found.
Manzi has two problems with this. First, Reyes didn't find a correlation in every possible subset of data. For example, lead didn't have an effect on property crime, only violent crime. Nor did Reyes find an association with murder rates. Manzi calls this "extraordinarily counter-intuitive," which I find odd. It actually makes perfect sense that a higher propensity for violence would increase the rate of violent crime, but might not have much effect on property crime. As for murder, other studies have found a relationship, but in any case, it's certainly possible that other factors could swamp the effect of lead. The absolute number of murders is low, and small, localized effects could easily move the numbers enough to make the relationship fail a standard test of statistical significance.
Second, Manzi simply doesn't accept that the adoption rate of unleaded gasoline might be truly random:
Which states have what consumer preferences for mix of car types — think states with lots of American pick-ups versus states with lots of Toyota sedans — is very likely correlated with differences in political economy that in turn will affect changes in crime rates over decades in material ways. Climate — for example, who chooses to live in the Sunbelt versus the Upper Midwest — is similarly a confounding factor. Number of gasoline pumps at stations is highly related to land costs, road networks, land-use regulation, political strength of dealer lobbies, and other factors highly related to political economy. Age and stock of cars, ditto.
....[But even if none of this stuff matters], what this model hasn’t accounted for is that the evolution of political economy over time during these decades could systematically vary between late- and early-phase-out states. This could easily be the case, if the evolution of, say, the political economy of Sunbelt states versus Rustbelt states evolved systematically differently over a time frame of many decades, and Sunbelt states tended to phase out leaded gasoline either earlier or later than Rustbelt states.
There are several things to say about this. First, as human beings we have to make decisions even if our methods aren't perfect and we don't have as much data as we'd like. You can always make up a list of possible hidden variables, as Manzi does above, and at some point we have to make a judgment about how important they're likely to be. To my ear, Manzi is stretching here. Reyes has shown a very unusual correlation, and I think the odds are good that state-level adoption of unleaded gasoline really is random enough to be a useful variable.
But that's far from all. If Reyes' paper was the only evidence for the link between lead and violent crime, I'd agree with Manzi. It's not enough. But it's far from the only evidence. We have striking evidence at the national level, of course. We have evidence at the city level. We have evidence that merely living in a housing project near an expressway is associated with more crime. And most important, we have evidence at the international level. As long as the data is all from the United States, you can argue, as Manzi does, that there might be some systematic effect of culture or political economy that's hidden in the background and affecting the results. But it's a lot harder to say that when you find the same results in Vancouver, Montreal, and Toronto. And it's way harder to say that when you find the same results in Britain, France, and Australia over different time periods (because different countries banned leaded gasoline at different times). Sure, it's still possible that there's some systematic hidden variable affecting these results, but that would be a helluva thing, wouldn't it? You're talking about some aspect of culture, or policing tactics, or drug use, or automobile preference, or whatever, that affects country after country around the world.
And there's more. From a Bayesian perspective, our priors should be pretty high in favor of this hypothesis already. Far from being an exotic, hard-to-believe explanation for the rise and fall of violent crime, the truth is that lead is actually an explanation that makes perfect sense. After all, we have multiple prospective studies that associate lead with arrest rates for violent crime in individuals. We have MRI studies showing that lead affects the brain in ways likely to increase aggression levels. We have copious historical evidence of the effect of high doses of lead on workers: for years people said it made them "dumb and mean." We have medical studies showing that prisoners convicted of violent crimes have higher lead levels in their teeth than similar populations. We have studies linking lead exposure to juvenile delinquency. Dose-response effects litter the literature. And much, much more.
In retrospect, if I were writing my article over again I'd begin with this evidence. I chose to begin with the population studies mainly for narrative purposes, but I think that was a mistake because it led a fair number of readers, like Manzi, to believe that the Reyes paper was the linchpin of my argument. But it's not. It's just one confirming piece in an ocean of evidence.
AND NOW FOR THE AMUSING IRONY. I promised you something if you made it to the end, so here it is. Manzi's key argument is that econometric studies are inherently flawed because you never know if there's some hidden variable you haven't considered. And guess what? That's exactly my argument too. The criminology community has spent years conducting econometric studies trying to link the rise and fall of crime to poverty, policing tactics, drugs, guns, demographics, the state of the economy, and more. These efforts, to be charitable, have been frustrating. These factors may influence crime rates, but they simply don't seem to be sufficient to explain a huge rise and then a huge fall in crime over the course of 50 years.
Why not? Well, my argument is that they all missed a hidden variable: gasoline lead. That was affecting crime all along, and exposure to gasoline lead is associated with all sorts of things: city size (big cities are more affected than small ones), race (African-Americans were generally exposed to more lead than whites), incarceration rates (probably a symptom more than a cause), and poverty. That's why it was missed for so long.
So in one sense, the lead-crime hypothesis is a vindication of Manzi's skepticism about the utility of econometric analysis of complex social issues. It's not perfect, and in a messy world no one should expect lead to correlate nicely with every subset of crime in every locality, but it sure looks to me like the best explanation we have.