One Last Encore for the Great “Statistical Tie” Fallacy

For indispensable reporting on the coronavirus crisis and more, subscribe to Mother Jones' newsletters.

If Ed Kilgore can steal an old post of mine about the whole “statistical tie” fallacy, well, I can too. I was going to do it anyway because Nicholas Beaudrot told me to, so here it is:

The idea of a “statistical tie” is based on the theory that (a) statistical results are credible only if they are at least 95% certain to be accurate, and (b) any lead less than a poll’s margin of error is less than 95% certain.

There are two problems with this: first, 95% is not some kind of magic cutoff point, and second, the idea that the MOE represents 95% certainty is wrong anyway. A poll’s MOE does represent a 95% confidence interval for each individual’s percentage, but it doesn’t represent a 95% confidence for the difference between the two, and that’s what we’re really interested in.

In fact, what we’re really interested in is the probability that the difference is greater than zero — in other words, that one candidate is genuinely ahead of the other. But this probability isn’t a cutoff, it’s a continuum: the bigger the lead, the more likely that someone is ahead and that the result isn’t just a polling fluke. So instead of lazily reporting any result within the MOE as a “tie,” which is statistically wrong anyway, it would be more informative to just go ahead and tell us how probable it is that a candidate is really ahead. Here’s a table that gives you the answer to within a point or two:

Pretty handy, no? Most national polls have an MOE of about 3%, so you can usually just use that row. NBC, for example, puts Obama ahead of Romney right now by 49-46%. So what are the odds that Obama is really ahead, and this isn’t just a statistical fluke? Answer: 84%.

But now for the bad news: As fun as it is to haul this thing out every few years, it’s obsolete. If you want to know who’s ahead, there are now loads of sites that aggregate multiple polls in various ways to provide estimates with far less margin of error than any single poll. If Pollster or RCP says that Obama is ahead by three points, then the odds are that he really is ahead by three points. There’s still plenty of room for various kinds of error in these poll-of-polls averages, but pure sample error isn’t really one of them any more.

For the record, as of today Pollster has Obama ahead by 4.3%; RCP has Obama ahead by 4.0%; Sam Wang’s meta-margin has Obama ahead by 5.06%; and Nate Silver has Obama ahead by 3.9%. I think it’s pretty safe to say that, at this moment in time, Obama is comfortably ahead.

Thank you!

We didn't know what to expect when we told you we needed to raise $400,000 before our fiscal year closed on June 30, and we're thrilled to report that our incredible community of readers contributed some $415,000 to help us keep charging as hard as we can during this crazy year.

You just sent an incredible message: that quality journalism doesn't have to answer to advertisers, billionaires, or hedge funds; that newsrooms can eke out an existence thanks primarily to the generosity of its readers. That's so powerful. Especially during what's been called a "media extinction event" when those looking to make a profit from the news pull back, the Mother Jones community steps in.

The months and years ahead won't be easy. Far from it. But there's no one we'd rather face the big challenges with than you, our committed and passionate readers, and our team of fearless reporters who show up every day.

Thank you!

We didn't know what to expect when we told you we needed to raise $400,000 before our fiscal year closed on June 30, and we're thrilled to report that our incredible community of readers contributed some $415,000 to help us keep charging as hard as we can during this crazy year.

You just sent an incredible message: that quality journalism doesn't have to answer to advertisers, billionaires, or hedge funds; that newsrooms can eke out an existence thanks primarily to the generosity of its readers. That's so powerful. Especially during what's been called a "media extinction event" when those looking to make a profit from the news pull back, the Mother Jones community steps in.

The months and years ahead won't be easy. Far from it. But there's no one we'd rather face the big challenges with than you, our committed and passionate readers, and our team of fearless reporters who show up every day.

We Recommend

Latest

Sign up for our newsletters

Subscribe and we'll send Mother Jones straight to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate

We have a new comment system! We are now using Coral, from Vox Media, for comments on all new articles. We'd love your feedback.