- ‹ previous
- 152 of 2823
- next ›
Poll Flippery Explained!
Suppose you conduct an opinion poll and get answer X on a particular question. If you follow up with a question like "But what if....." then X is likely to change. But how much? Is there some minimum amount of change you'll get no matter what followup question you ask?
I asked that question a couple of weeks ago, and Dave Munger of Cognitive Daily decided to investigate. The result was a cheap-and-cheerful nonscientific online poll that gauged whether some people would change their minds no matter what the followup question was. I've been sworn to secrecy until now, but here are the results:
While it is true that someone changed their answer for each question, in some cases, very few people did. Consider the responses to the question "Should the United States withdraw all troops from Afghanistan?"....While 35 percent of respondents said they'd change their answer if the US kept one base in Afghanistan to address only the terrorist threat, only 4 percent said they'd change their answer to the original question if the US also closed the prison at Guantanamo Bay.
Aside from one genuinely out-of-the-blue question, that seems to have been the baseline: you can get 4% of your respondents to change their minds no matter what the followup is. That's actually pretty low.
But there's more! Who changes their minds more, liberals or conservatives? Click to link to find out.
"When the facts change, I change my mind. What do you do, sir?"
So Brett Favre is a Keynesian? Or, worse yet, a liberal?
Henry
I tried to make sense out of
I tried to make sense out of this but couldn't. A proper investigation, it seems to me, would try to ask follow-up questions that had no material bearing on the substance of the first question -- and, also, that were not independently of interest to a significant portion of the sample (not an easy thing to establish).
But as far as I can see, the explanation of the methodology at Cognitive Daily gives no clue as to how these follow-up questions were chosen. One example is given with no further explanation.
So I'm not sure exactly what, if anything, was demonstrated by this exercise. Perhaps a statistical study of the results of a large number of published polls using follow-up questions might come closer to giving an answer of some sort.
Still, this Dave Munger is surely a good sport.
We did ask questions with no material bearing
We did ask follow-up questions with no material bearing on the original question. For one question (about whether lotteries should be banned), we followed up with "Suppose J.K. Rowling had never written the Harry Potter books. Would your opinion change?" In this case, only 1 percent of respondents flip-flopped.
We asked some follow-ups that might lead respondents in one direction, some that might lead them another direction, and some (like the above) that bore no relation to the original question.
The really cool part is that
The really cool part is that fully 1 in 100 would flip based on a non-sequitur about a fiction writer that had nothing to do with anything.
This actually goes a long way towards explaining the behaviour of some of my neighbors.
But if you ask a complete
But if you ask a complete non-sequitur, then the subjects smell a rat -- so they will play according to their temperaments. I probably would too.
If Rowling never wrote the
If Rowling never wrote the Harry Potter books I'd %$#$ing enlist in the army myself and request assignment in Afghanistan.
Apparently even more would
Apparently even more would enlist just to prevent George Lucas from filming an additional three Star Wars sequels
That's what Aristotle called
That's what Aristotle called the non-sequel non-sequitur.
I just lie.
I just lie.
Define non sequitor
What bothers me isn't what's said, that the project didn't ask silly enough follow-ups. That'd only prove, as it seems to, that people aren't stupider than sea slugs. It still doesn't get at some key problems with public opinion research that trace either to polling or to the public's lack of information.
Thus, in the health-care debate, people constantly show contradictory responses, and it'd appear they're sufficiently uninformed that they might be manipulated. And similarly bad (partisan) polling can manipulate results not just with leading questions, but with non sequitors that MAY be interpreted as relevant. This story itself seems more about the silliness of research in the humanities than about the silliness of the public or of polls.