Statistical Zombies

Seth Michaels wants me to republish an old post setting out the top ten mistakes that infest day-to-day reporting of numerical and statistical information. Well, why not? Let's call them statistical zombies so that I can get in on the zombie craze. Here they are:

  1. What’s the real income? Money comparisons over time should almost always be reported in real inflation-adjusted terms or else they're worthless. In nearly all cases, they should be reported in per capita terms as well.
  2. What’s the survey error? Statistical sampling error in opinion polls is trivial compared to the error from other sources. Things such as question wording, question order, interviewer bias, and non-response rates, not to mention Bayesian reasons for suspecting that even the standard mathematical confidence interval is misleading, give most polls an accuracy of probably no more than ±15%. Example: a couple of years ago a poll asked respondents if they had voted in the last election. 72% said yes, even though the reality was that voter turnout in that election had been only 51%. Most polls and studies are careful to document the statistical sampling error, but who cares about a 3% sampling error when there might be 21 points of error from other causes?
  3. Does A really cause B or might there be another explanation? If A and B are correlated, A might indeed cause B, but it’s also possible that it's just a coincidence or — even more likely — that some third source is causing both A and B. This problem is especially rampant in social science studies where virtually everything is related to everything else and even well designed multivariate analysis is extremely difficult.
  4. Is it the first study? Even putting aside other errors, 95% confidence means there’s a 5% chance that the result is wrong. We only believe that smoking causes cancer because there have been hundreds of confirming studies. Always be cautious about accepting the first study on any subject.
  5. Maybe it really was just a freak chance. “That can’t be a coincidence” is usually the result of not understanding how many rare things are nonetheless likely to happen once or twice in a population of 300 million. In a large country, there will always be some cities, or some groups, or some people, that are way above average for, say, cancer. The flip side of this is that something that seems dangerous might not really be. 100 kidnappings a year might seem like a lot, but in reality those are odds of one in three million. That's less likely than the odds of two people randomly picking out the same word from an encyclopedia.
  6. Compared to what? A 5% rise might be good or might be bad depending on whether everything else is growing at 0% or 10%. Which is it?
  7. Is there contradictory data? Two types of publication bias are involved here: researchers often don’t publish null results, and newspapers don’t bother reporting them when they are published.
  8. Statistically speaking, why did the headline number go up (or down)? Did everyone’s income go up 5%, or was it just that Bill Gates’ income went up 1000%? Distribution is as important as central tendencies. Check for mean vs. median. The value of statistics is to summarize a large mass of data, but it’s important not to summarize too much.
  9. Was the sample large and unbiased? For example, the original gay gene study used only about 40 people, and that was simply all the data they had. What’s worse, even if you do have a large sample it’s still difficult to ensure that it’s unbiased. Chapter 29 of Dana Milbank’s book Smashmouth is a pretty good down-and-dirty introduction to the delicate and tricky decisions that election pollsters have to make under deadline pressure to try and get accurate results.
  10. Does all the data point a little too cleanly to a single cause? Life is messy. A single report can often produce masses of data and should probably be viewed with suspicion if it claims that every bit of its data can be explained by a single cause — especially if it's a cause that the researcher is already known to favor.

Plus I'll add another pet peeve that's numerical but not strictly statistical in nature: the reluctance to simply report percentages as percentages. This leads to sentences like this: "Although 40% of voters support spending reductions, nearly a third don't want defense spending cut and two in ten decline to identify any reductions at all." This is almost unreadable. Why not just provide the percentages in parallel form? I really don't think readers are so stupid that they can't handle it.

And via Seth, here's another approach to identifying bad everyday statistics, but with examples! Take a look and see if you can spot the errors.

Pakistani President Asif Ali Zardari and Interior Minister Rehman Malik would like global anti-graft watchdog Transparency International to leave their broken government alone. Or else.

On Sunday, Asian News International quoted Malik in a story accusing TI Pakistan, Transparency International's local affiliate, of acting like a "detective agency." Malik also made a not-so-veiled threat to kick the group out of the country. This comes just weeks after TI released its 2010 Corruption Perceptions Index. Pakistan's score of 2.3 out of 10 didn't exactly endear the group to Zardari and co.; it's barely better than Libya's (2.2), and only a slightly higher than the Democratic Republic of Congo's (2), and places Pakistan as the 34th most corrupt country out of the 178 surveyed.

Zardari and Malik's problems with TI stem from the five-year, $7.5 billion aid package the United States began delivering to Pakistan last year. At least half of that money, the Wall Street Journal reports, will be funneled directly through government ministries, rather than foreign aid groups and contractors. That's a heavy chunk of change to leave floating around the brackish bureaucratic backwaters of Pakistan. The US, justifiably skittish about the arrangement, signed an agreement with TI Pakistan in September that is intended to help ensure that the money gets where it's supposed to go.

But by early November, the transparency advocates were already running into problems. That month, TI Pakistan wrote to Zardari and US Ambassador to Pakistan Cameron Munter to complain about threats against the group's local chairman, Syed Adil Gilani, and his staff. TI Pakistan urged Zardari to guarantee the rule of law and freedom and safety to all TI staffers based in Pakistan. From the Journal

Gilani . . . said he received death threats recently from "high-level" government officials urging him to stop his organization's anti-graft investigations, including plans for [a] graft hotline. He declined to name the source of the threats. "They don't want TI Pakistan to monitor" the US aid flows into the country, Mr. Gilani said in an interview.

For Pakistan, more "detectives" means more scrutiny. It's unclear whether Zardari knew that that $7.5 billion would come with such a sizable asterik. TI's next move: to circumvent the Zardari administration altogether. In a letter to Pakistani Supreme Court Chief Justice Iftikhar Muhammad Chaudhry, the group again expressed its concerns about the government's plans, arguing that it "should be protected from illegal or extrajudicial acts to fetter its anti-corruption work." In a press release on the matter, TI cites reports of government officials calling on all ministries, divisions, and departments to sever contact with TI Pakistan.

Packages like the one TI oversees are the sugar that make the bitter pill of war a little easier to swallow. Without such aid, it becomes harder for the Pakistani government to justify helping the US, and to sell its citizens on the idea that having thousands of Americans in the region serves a common interest. But without at the least the appearance of oversight, it becomes almostly politically impossible for the Obama administration to continue feeding the aid stream. Giving TI the job of keeping everything above board is the crucial variable in the Pakistan equation. Without it, the United States' uneasy alliance with the country could fall into jeopardy.

Congressional Republicans aren't likely to get very far in their attempt to repeal the entire health care law. But there are growing signs that the courts could end up doing the lifting for them. A federal district judge in Virginia has promised to rule on the constitutionality of the bill's requirement that individuals purchase health insurance (the so-called "individual mandate") by the end of the year. The New York Times explains why the Obama administration is so concerned:

Lawyers on both sides expect the issue eventually to be decided by the Supreme Court. But the appellate path to that decision could take two years. In the meantime, any district court judge who rules against the law would have to decide whether to block enforcement of one or more of its provisions, potentially creating bureaucratic chaos.

The administration is already facing enormous hurdles in trying to put the federal health law into effect, receiving pressure from industry lobbyists who want to water down the new regulations and states that don't want to comply. Unfavorable court rulings will only complicate matters. Moreover, there are major parts of the health law that simply won't work unless the individual mandate is enacted. And finally, there's the concern that court rulings against the law—even if they don't ultimately hold up—could add to public animus against federal health reform:

"Any ruling against the act creates another P.R. problem for the Democrats, who need to resell the law to insured Americans," said Jonathan Oberlander, a University of North Carolina political scientist, who wrote in The New England Journal of Medicine last week that such a ruling "could add to health care reform's legitimacy problem."

Most of the biggest changes under the law, including the individual mandate and the insurance exchanges, won't be fully enacted until 2014. But the more obstacles the Democrats face in putting the law into effect, the harder it's going to be to convince the public that the earliest reforms are actually helping ordinary Americans—and that the bigger reforms down the road are worth the investment. Though their GOP antagonists in Congress are more bark than bite, the Affordable Care Act still faces a real threat. The Democratic leadership needs to ramp up its own offensive to sell health reform to the public now if they want to preserve the law's future integrity.

So what do I actually think about the WikiLeaks dump of U.S. embassy cables? A bunch of contradictory things, it turns out. Here's sort of a stream of consciousness of what was going through my mind yesterday as I read about this:

  • This isn't a whistleblower case. In fact, surprisingly little official lying of any kind has been revealed so far. Rather, it's an action aimed very generally at weakening American influence and exposing American intentions.
  • To the extent that this is done by, say, Australians and Germans, there's nothing unpatriotic or even wrong about this. If foreign nationals who oppose American hegemony get a chance to lob a stink bomb at America, why shouldn't they?
  • At the same time, the American citizen who leaked this stuff — Pfc Bradley Manning, apparently — should do serious time as long as the government can convince a jury of his peers that they've nailed the right guy. No government in the world can, or should, tolerate this kind of massive security breach from one of its own.
  • Governments have lots of legitimate reasons for wanting to keep communications confidential. This is not some kind of weird pathology exclusive to nation states, either. You keep secrets. I keep secrets. Companies keep secrets, families keep secrets, labor unions keep secrets, nonprofits keep secrets, and your neighbors keep secrets. There's always the risk of this stuff going too far, and there's always the risk of your secrets getting spilled. But this is all part of the human condition, not a sign of depravity in the State Department. 
  • In general, this kind of indiscriminate data dump is a bad thing. This particular dump, for example, could conceivably hurt chances of ratifying the START treaty, strain relations with the UN, strain relations with Russia, make an attack on Iran a little bit likelier, and reduce even the meager leverage we currently have over Hamid Karzai. More broadly, it could hobble American efforts at replacing saber rattling with genuine diplomacy, which is really in nobody's interest. 
  • And yet....even secrets that are justifiable in the short term can often end up being toxic in the long term. Routine secrecy quickly becomes a crutch, as it plainly has in the United States, and an occasional informational enema like this can have a salutary effect. Governments might have a legitimate need for secrecy, but they should also be keenly aware that there's a risk to doing business like this. I wouldn't want this kind of mass disclosure to become a regular occurrence, and I do think the leaker should pay for his crime, but at the same time I can't honestly say that I'm entirely sorry this happened.

I suppose this is all very unsatisfactory, but those were the thoughts buzzing through my brain yesterday. They still are. Perhaps they'll gel into something more definite as I give it more thought and the ramifications become more clear.

The Future of the Euro

With financial contagion from Greece and Ireland starting to spread to Portugal and Spain, the eventual breakup of the euro is becoming at least conceivable. But is it likely? Barry Eichengreen is probably the guy who's thought the hardest about this, and back in 2007 he concluded that it was extremely unlikely regardless of circumstances. The folks over at Five Books recently asked him for his latest thinking on the subject:

What’s your view now?

Rarely does an academic have the privilege of a real-time test of his hypothesis. With the benefit of that test, I would now say that I was both right and wrong. I was right in that, yes, if the Greek government were to announce tomorrow that it had decided to reintroduce the drachma, it would precipitate the mother of all financial crises. Everyone would know that its intention was to depreciate the new drachma, so in the first minute everyone would rush to get their money out of the country, out of its banks, and out of its bond market. The result would be the biggest bank run and financial crisis the world has ever seen. This danger is a formidable deterrent to even contemplating going down this road. So I think the argument I made in 2007, that attempting to exit the euro area would be the equivalent of burning down your own house in order to find a way out, was exactly right.

In what way were you wrong?

I was wrong in that, as Paul Krugman observed earlier this year, if the house is burning down anyway, then the normal advice not to play with matches loses much of its force. If there’s a run on your banking system anyway, then the deterrent to action no longer applies. If there’s a run on your banking system and you have to close down your banks and financial markets anyway, you may want to take that opportunity to reintroduce your own currency. I still don’t think that things will be allowed to get to this point, but I no longer attach a zero probability to a country’s exiting the euro — just a close to zero probability. Never say never, but I still believe that the euro is an example of a path-dependent historical process that is unlikely to be reversed.

I think he's probably right. But I'm also not entirely sure of that. So far there's been no sign of a serious run on any euro-area banking system, but there have been signs of a "bank walk," for lack of a better term. And that could lead to a run on one bank which, in turn, could lead to a run on so many banks that the European monetary authorities can't stop it. Megan McArdle's warning that pundits "have started seeing Creditanstalt everywhere" is a good one, but this is still a scary process we're going through. I'm not sure exactly what nonzero probability Eichengreen would attach to a euro breakup, but even five or ten percent is pretty serious. Buckle up.

As always, Paul Krugman has more. So does Tyler Cowen, who is pessimistic. "In a nutshell," he says, "we're watching the most pitched, highest-stakes, most determined battle between politics and finance which has been staged. I am expecting finance to win."

People who do not need more evidence of Sarah Palin's lack of seriousness should not read further.

As the WikiLeaks controversy continues, Palin could not resist the urge to tweet her thoughts about the affair. On Monday morning, she sent this message to her 317,000 Twitter followers:

Inexplicable: I recently won in court to stop my book "America by Heart" from being leaked,but US Govt can't stop Wikileaks' treasonous act?

Inexplicable? Does she not understand the difference between apples and nuclear reactors? The two instances she links have little in common. In the case of her book, she managed to get a judge to order Gawker to take down a post showing portions of her book after the website had put them up. And the judge in this case was following precedent established when The Nation magazine was successfully sued by Harper & Row in the 1970s after publishing excerpts of former President Gerald Ford's memoirs before the book was released. The Supreme Court, deciding the case in favor of the publisher, said media outlets could not, under a claim of fair use, publish a significant portion of a copyrighted book (accepting the argument that this could weaken the commercial value of the book). Palin's lawyers took advantage of this ruling, in demanding that Gawker not show the actual pages of her book.

Stopping a media leak involving government information before the fact is not the same. The grand-daddy legal decision on this front comes out of the famous Pentagon Papers case, when the Supreme Court ruled that the government could not block newspapers from publishing the secret Pentagon history of the Vietnam war leaked by Daniel Ellsberg to The New York Times and other papers. The guiding principle here: the government does not have the right to impose prior restraint on the media.

This latest WikiLeaks episode could cause some, including Palin, to argue that in these post-9/11 days the prior restraint rule is a luxury that cannot be afforded. But that's where the law stands. With her tweet tying this important and historical issue to her own (less consequential) book, Palin demonstrates that for her simplistic analysis is the best analysis and that the best way to understand anything is to view that topic from Planet Sarah.

The much anticipated release of The Anthology of Rap has gotten off to a bit of a rocky start. The project, which features a foreword by Henry Louis Gates Jr., has drawn criticism for the abundance of transcription errors—and in hip-hop, the difference between the right word and the almost-right word is the difference between, well, George Wallace and Gerald Wallace (an actual mistake in the book). That's a shame because, errors aside, it's an awesome compilation: 920 pages of some of the baddest, phattest, flyest tracks ever dropped.

And that, invariably, means plenty of preposterous pop-culture references. Unless you rolled with Junior M.A.F.I.A. back in the day or hail from Queensbridge, you're probably not included in this book. But plenty of totally random people (and things! and historical events!) are. So what exactly shakes like Smucker's grape jelly? What's the best way to hijack a space shuttle? And what does Nas really think about Calvin Coolidge? We've got you covered. Here's our unofficial, abridged political and pop-culture Anthology of Rap index.

Attila the Hun

  • is not LL Cool J, p. 216


  • mad phatness of, p. 312

Bicentennial Celebration

  • boringness of, p. 419
  • similarities to Arsenio Hall Show, p. 419

 Booth, John Wilkes

  • as second-grade icon, p. 358

Bush, George H.W.

  • as real estate agent, p. 424
  • worth less than a bird in the hand, p. 424

Bush, George W.

  • a natural ass, p. 699
  • did 9/11, p. 751
  • not a Seminole Indian, p. 715
  • treatment of blacks relative to treatment of trash, p. 699

Clinton, Bill

  • hanging in strip club, p. 681
  • slanging crack, p. 485
  • smoking up, p. 681

Coolidge, Calvin

  • suspected black ancestry of, p. 469

Dahmer, Jeffery

  • consumption of dogs, p. 485
  • consumption of mail-order brides (hypothesized), p. 647


  • as matter of life and death, p. 180

Falwell, Jerry

  • as funky dope musician, p. 136


  • as serial killer, p. 389

Goodyear Blimp

  • says "Ice Cube's a pimp," p. 426

McCain, John


  • as metaphor, p. 361


  • hijacking of space shuttle with a .38, p. 389

Nixon, Richard

  • wickedness relative to Ghostface Killah, p. 548

Nobel Peace Prize

  • awarded to Canibus, p. 352
  • passed up by Canibus, p. 352

Obama, Barack

  • as pen pal, p. 760
  • complexion of, p. 717
  • merits of putting on $5,000 bill, p. 717

Peloponnesian War

  • lack of respect for science or intellect, p. 159

Picasso, Pablo

  • similarities to Kool Moe Dee, p. 204


  • complexion of relative to a Maybach, p. 438
  • dead, p. 427
  • dead, with sprinkles, p. 428
  • fuckin' dead, p. 428
  • efforts to combat Chinese expansion, p. 755
  • Eric B. is, p. 170
  • future employment as, p 391
  • impeachment of, p. 255
  • is black, p. 715
  • is half-white, p. 438
  • never did shit for me, p. 716

Reagan, Nancy

  • sexual intimacy with, p. 298

Reagan, Ronald

  • employment of Too $hort as economic advisor, p. 298
  • selling cocaine in the White House, p. 298


  • not fuckin' broke, p. 424
  • should be disabled, p. 341


  • getting high, p. 424


  • shafting like, p. 223

Sharon, Ariel

  • odds of showing compassion relative to odds that Immortal Technique will stop blastin', p. 655


  • shaking jelly like, p. 530

Steve, Scuba

  • respiratory struggles of, p. 635

Swayze, Patrick

  • is Method Man, p. 539


  • as artificially flavored equivalent to Ol' Dirty Bastard's rhymes, p. 545

Truman, Harry S.

  • dapper as a rapper, p. 135

Tyson, Mike

  • Knocked-out like, p. 61
  • Knock you out like, p. 215


  • decision to spend dough on hoes instead of, p. 717

Click here for more Music Monday features from Mother Jones.

The red-hot R&B revivalist Janelle Monáe takes cyborgs, and her career, pretty seriously. She's been compared to, among others, Michael Jackson, David Bowie, and Prince. Which is an indication both of her talent—Sean "Puffy" Combs called her one of the most important signings of his career—and her wide range of influences, from Stevie Wonder to Salvador Dalí to Fritz Lang's silent film Metropolis (hence the cyborgs).

Monáe grew up doing musical theater in Kansas City and helped create the Atlanta-based Wondaland Arts Society, which released her first EP, Metropolis: The Chase Suite. Her next album continued the story of Cindi Mayweather, a time-traveling ArchAndroid sent to free the citizens of Metropolis from their dystopian world. Her songs might be viewed as a metaphor for race and gender discrimination—but Monáe also wants you to take them literally: She insists that they're a preemptive call for the equality of cyborgs. Even her usual outfit—saddle shoes, pompadour, tailored tux—is not so much an artistic statement as a practical move; tuxedos are a timeless look, she's said, and so a natural choice for a time-traveling robot. Most recently, her song "Without a Fight" appeared on the For Colored Girls soundtrack. We recently spoke with the cyberfunkstress about tuxedo politics, her journey to fame, and her favorite holiday music.

Mother Jones: Did you hear that Bradford Cox of Deerhunter is calling you the next David Bowie?

Janelle Monáe: No. I'm sure he meant well, but that isn't my goal. I like David Bowie though. I guess if I got a sex change, that could work out.

U.S. Airmen with the Air Force Honor Guard at Bolling Air Force Base in Washington, D.C., perform during a Veterans Day parade in Las Vegas, Nev., Nov. 11, 2010. (U.S. Air Force photo by Tech. Sgt. Michael R. Holzworth/Released)


Bombing Iran, Part 2

David Frum asks a question (in a series of 140-character chunks):

What do we learn from Wikileaks re Iran?

1) Many more govts than you might think back a US military strike.

2) It's now public knowledge that Iran and North Korea are exchanging deadly military technology.

3) Whole world can see that US has gone every extra mile to reach a negotiated outcome with Iran

4) Nevertheless Iran has pursued nuclear ambitions all-out.

Seems to me the combined effect of this information would be to make US military action more politically acceptable....Both inside US and outside.


Also for discussion: all governments have a legitimate need for a certain amount of secrecy. In particular, embassy officials need to be able to report candidly to their superiors about what's going on in their sphere of responsibility. So what's the most likely consequence of the WikiLeaks document dump? That governments around the world realize the error of their ways and become more open about their dealings with the rest of the world? Or that governments around the world — and in particular the United States government — clamp down hard on classified information and restrict its distribution even more than they have in the past?