There’s a Bigger Difference Between 0 and 1 Than You Think

A few weeks ago I wrote about a study showing that student evaluations of teachers could differ substantially if a scale of 1-6 was used instead of 1-10. Today, I learned that the boffins at Pew Research just finished a similar study. They wanted to test whether responses about political ideology changed depending on whether people were asked to describe themselves on a 0-6 scale vs. a 1-7 scale. Indeed they did:

In all three countries, the 1-7 scale produced higher results for the ideology defined by a low number (which happened to be lefties in this case). The differences weren’t small, either. Take a look at France, where the percentage of lefties increases from 35 percent to 49 percent. That’s a huge difference, and all because the ranking system changed by one digit.

This is a great example of how fragile polling can be. We obsess over things like the margin of error, which might be responsible for a point or two of error, when there are other things that can introduce errors of a dozen points or more. Of course, it’s not really correct to talk about “errors” here since we don’t know what the true value is or whether there even is one. What a study like this shows is probably that (a) lots of people are kind of fuzzy about where they stand, and (b) lots of people aren’t very good at math.

You can see the same sort of thing with question wording. This is especially evident in polls about abortion, where seemingly small changes in question wording can produce differences of ten or twenty points. This is why you should always be skeptical when you see a headline shouting about a big shift in abortion sentiment. If the change really is big, it’s almost certainly because of question wording, not because the American public has really shifted its views significantly.

Anyway, this is a good example of just how hard opinion polling is. Who would have guessed that a tiny change in scale would produce wild results like this?

HERE ARE THE FACTS:

Our fall fundraising drive is off to a rough start, and we very much need to raise $250,000 in the next couple of weeks. If you value the journalism you get from Mother Jones, please help us do it with a donation today.

As we wrote over the summer, traffic has been down at Mother Jones and a lot of sites with many people thinking news is less important now that Donald Trump is no longer president. But if you're reading this, you're not one of those people, and we're hoping we can rally support from folks like you who really get why our reporting matters right now. And that's how it's always worked: For 45 years now, a relatively small group of readers (compared to everyone we reach) who pitch in from time to time has allowed Mother Jones to do the type of journalism the moment demands and keep it free for everyone else.

Please pitch in with a donation during our fall fundraising drive if you can. We can't afford to come up short, and there's still a long way to go by November 5.

payment methods

ONE MORE QUICK THING:

Our fall fundraising drive is off to a rough start, and we very much need to raise $250,000 in the next couple of weeks. If you value the journalism you get from Mother Jones, please help us do it with a donation today.

As we wrote over the summer, traffic has been down at Mother Jones and a lot of sites with many people thinking news is less important now that Donald Trump is no longer president. But if you're reading this, you're not one of those people, and we're hoping we can rally support from folks like you who really get why our reporting matters right now. And that's how it's always worked: For 45 years now, a relatively small group of readers (compared to everyone we reach) who pitch in from time to time has allowed Mother Jones to do the type of journalism the moment demands and keep it free for everyone else.

Please pitch in with a donation during our fall fundraising drive if you can. We can't afford to come up short, and there's still a long way to go by November 5.

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate