Page 1 of 2

An Ode to Fact-Checking

Mike Daisey, Fareed Zakaria, Jonah Lehrer, Niall Ferguson: Amid the truth-mangling epidemic, a shout-out to what it takes to get to the facts of the matter.

| Wed Dec. 21, 2011 6:00 AM EST

The first rule of fact-checking is that everything you read and hear is wrong.

Were you to be hired as a fact-checker, as I was in 2007, at Mother Jones—or at the other remaining bastions of fact-checking, mostly a handful of magazines known for their reporting—you would be taught that information couldn't be trusted. It is, rather, presumed fallacious until proved otherwise. Statistics and news clips must be subjected to intense tests of verification. Don't even think the word "Wikipedia." In my first meeting, among new coworkers of startling cynicism and genius, the announcement that the source of some fact was a book set off a mighty wave of scoffing and eye rolling around the conference table.

In true fact-checking, literally every word of every factual statement must be traced to a primary source, whether a document or the corroborated accounts of independent experts or witnesses. "Primary source" means that if the story you're fact-checking says some soldier was the 44th Canadian soldier to die in Afghanistan since 2002, you're calling the Canadian Army. Once, I heard one of our fact-checkers call a bar in Mexico to ask, in Spanish, whether its floor was metal, per William T. Vollmann's assertion.

Advertise on

For the record, it was. But often, it's not. People just get things wrong. They read them wrong, or remember them wrong or the way they want to, or the information they read right was wrong in the first place. You hear the same fact a thousand times, but if you track down its origins, you find out all the repeaters are using the same source, and source zero was just guessing, or citing a highly questionable source or study. Or misciting a highly questionable source or study. Or confusing the details, so that by now, everybody's under the erroneous impression that a shot of espresso contains more caffeine than a cup of coffee.

So I wasn't a week into my job before I, too, had undergone the completely life-changing installation of an irrevocable—and warranted—skepticism of everything I heard or read. So it's life-ruining, also. Which is why I did something book writers rarely do—which is subject their manuscripts to a Mother Jones-style fact-check—and that they rarely do for really good reason—which is that holy fucking shit is it hard.

Once, I heard a fact-checker call a bar in Mexico to ask, in Spanish, whether its floor was metal, per William T. Vollmann's assertion. 

Enter former MoJo research editor Leigh Ferrara, a fact-checking and multitasking wizard, and the hardest-working and most charming person you could hope to be stuck in a studio apartment with for 12-hour fact-review marathons. The book, For Us Surrender is Out of the Question, and the MoJo piece of the same name, was about Burmese refugees who snuck back into their country to document human rights atrocities. And, you know, the whole history of Burma. The manuscript I gave Leigh had 1,240 footnotes, plus piles of sources noted haphazardly within the text, plus a bunch of sentences with no sourcing at all. So: I'd read and subsequently written that Burma had the fourth-highest child mortality rate in the world, and Leigh had to figure out whether that was true or not. It's not. Burma was 36th on that list, actually, which Leigh tracked down in UNICEF's "State of the World's Children 2009." That took care of 11 words, out of more than 100,000. And that was a pretty easy one. (BTW, it is now 44th.)

Buy the book here.For historical details, we unearthed and paged through colonialists' reports and missionaries' diaries, or cross-checked information with other historical accounts we'd made sure weren't all using the same one original source—as is often the case—and/or vetted the minutiae and main ideas with scholars and specialists. We tracked down witnesses to and experts on subjects way outside the spotlight of popular scrutiny. We then evaluated those sources, trying to determine if they were reliable and where they were getting their information. One of Leigh's experts helpfully eliminated a handful of questions from her long list of outstanding facts; then she realized the source of his expertise was a book we'd already determined to be mistake-tastic.

Further complicating the fact-checking process was the inconvenience that there's often no such thing as "fact." Another figure I'd cited was that trade between China and Burma was up to $2.6 billion in 2008, from $630 million in 2001. That turned out to be "true" (shout-out to the Wall Street Journal for its totally solid Burma info. Ditto The Washington Post. And The Irrawaddy, the Burmese exile paper, does work that, in addition to filling a critical reporting void, is incredibly reliable.), based on data Leigh uncovered in the United Nations Commodity Trade Statistics Database. Mexicali's 13 Negro bar either does or does not have a metal floor, but the trouble with this trade statistic is perfectly summed up immediately following it in the WSJ article where it appeared: "Analysts say the official numbers vastly understate the full extent of China's investments in Myanmar." Mmmm, analysts do say that. Knowing that—however meticulously we sourced our facts, and though every estimate we were working so hard to find and confirm was at least the best possible estimate in existence—"true" is often still kind of a relative concept…it's demoralizing.

We both had dark moments while trying to keep a million little pieces needing verification up in the air. I had nightmares about working on the final edits and not being able to write any of the words I wanted because I didn't have sourcing for them and there was no time for further fact-checking. Leigh started inadvertently holding her breath when opening emails from sources, because they might say that she was out of luck, or an idiot, because the assertion she was asking them to confirm—which I'd pulled from non-fact-checked books or articles—was absurd. We each went through a period of extreme temporomandibular pain, at which point we realized we'd started clenching our teeth furiously. We took turns psyching each other up, holding up opposite sides of motivational conversation:

"Why are we doing this?"

"It's fine. We're doing a great job. People are going to give us trophies when they realize how thorough we've been."

"Yeah, because anyone will ever even notice that we did this, and they totally give trophies for fact-checking."

"You're doing a great job. It'll all be worth it when we get the trophies."

Page 1 of 2