How to Mislead With Statistics: GMO Crops Edition


A few days ago the New York Times ran a big front-page story about GMO crops. Roughly speaking, there were two takeaways: GMO crops don’t improve yields and don’t cut down on pesticide use. So why bother?

That sounded interesting, but I decided not to write about it. The story was pretty shallow in its use of statistics. It assumed that you can compare different countries without controlling for anything (different soils, climates, crops, etc.). And it seemed to suggest that American farmers must be idiots, because they keep buying GMO seeds even though they’re worthless. Put together, this was a little too much for my spidey sense, and it would have taken a long time to educate myself enough to really figure out what was going on. So I passed.

Yesterday, however, Nathanael Johnson of Grist took a close look at the Times piece and concluded that it was pretty misleading. You can read Johnson’s piece if you want all the gory details, but it contained a pretty interesting example of how to mislead with statistics, which I figured I should pass along. The Times story says that in the US, despite the wide use of GMO seeds, use of insecticides has fallen but the use of herbicides has gone up. In France, by contrast, where they don’t use GMO seeds, “use of insecticides and fungicides has fallen by a far greater percentage — 65 percent — and herbicide use has decreased as well, by 36 percent.” Using data from Andrew Kniss, here’s what this looks like in chart form:

It does indeed look like France is doing pretty well. But what if you look at the raw numbers, rather than percentages? Here’s a chart from Kniss:

In the case of herbicides, France started at a high level, and has only recently caught up to the US. In the case of insecticides, France started out way, way higher than the US and is still way higher despite their recent reductions. Looked at this way, it’s clear that France hasn’t actually done that well. What’s more, I had to cherry pick starting and ending years in the top set of charts in order to make France look as good as it does. Plus there’s this, from Kniss:

It is true that France has been reducing pesticide use, but France still uses more pesticides per arable hectare than we do in the USA. In the case of fungicide & insecticides, a LOT more. But a relatively tiny proportion of these differences are likely due to GMOs; pesticide use depends on climate, pest species, crop species, economics, availability, tillage practices, crop rotations, and countless other factors….Given all of these confounding factors, I wonder why France was singled out by Mr. Hakim as the only comparison to compare pesticide use trends. Pesticide use across Europe varies quite a bit, and trends in most EU countries are increasing, France is the exception in this respect, not the rule. In the early 1990’s, France was using more herbicides compared to almost every other country, so it shouldn’t be too surprising that pesticide use decreased as formation of the EU began to standardize pesticide regulations after 1993.

So what’s the best comparison to make: percentage change or raw volume? In this case, presenting only the percentage change leaves you with a very misleading impression. Presenting raw volumes, however, makes everything pretty clear. You can see that France started out far higher than the US and that its use of herbicides and insecticides has been decreasing. So why present only percentages?

POSTSCRIPT: If you click on the chart pack in the Times story, you will actually find charts showing raw volume of pesticide use in the US and France. However, they’re shown in two different charts, using different units, and broken up into different categories. If you were deliberately trying to make a comparison nearly impossible, this is how you’d do it.

$500,000 MATCHING GIFT

In 2014, before Donald Trump announced his run for president, we knew we had to do something different to address the fundamental challenge facing journalism: how hard-hitting reporting that can hold the powerful accountable can survive as the bottom falls out of the news business.

Being a nonprofit, we started planning The Moment for Mother Jones: A special campaign to raise $25 million for key investments to make Mother Jones the strongest watchdog it can be. Five years later, readers have stepped up and contributed an astonishing $23 million in gifts and future pledges. This is an incredible statement from the Mother Jones community in the face of the huge threats—both economic and political—against the free press.

Read more about The Moment and see what we've been able to accomplish thanks to readers' incredible generosity so far, and please join them today. Your gift will be matched dollar for dollar, up to $500,000 total, during this critical moment for journalism.

We Recommend

Latest

Sign up for our newsletters

Subscribe and we'll send Mother Jones straight to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate

We have a new comment system! We are now using Coral, from Vox Media, for comments on all new articles. We'd love your feedback.