Is it Science or Is It Bullshit? (Cookie Crumbling Edition)

For indispensable reporting on the coronavirus crisis and more, subscribe to Mother Jones' newsletters.

Sarah Kliff reports on a new study showing that portion size matters. Here’s the nickel version: If you offer kids a bowl full of cookies, they eat more than if you offer them a bowl full of smaller cookies.

Well, that sounded interesting. But I was curious about the details. So I found the original dissertation, which is ungated, and browsed through it. First there was this:

Recruitment was performed in an elementary school attended by one of the experimenter’s children. School director and legal guardians (i.e. mother/father) gave all written informed consent and both 1st and [] 6th grade classes participated voluntarily. The experimental protocol was approved by the Ethical Committee of the Faculty of Psychological Sciences of the Université Libre de Bruxelles.

OK, that has nothing to do with the research. It just cracks me up. They had to get IRB approval to feed some kids whole cookies and other kids half-size cookies! This is modern science, red in tooth and claw.

Moving on, there’s a detailed description of the cookies (several layers of wafers filled with milk chocolate topping) and the number of kids (85 total, about half of them girls). The experiment took place at 2:45 pm (the children’s “usual snack time”). Here’s the experimental design:

Kids were called up in alphabetical order, reported pre-study hunger (4-point scale labeled ‘not at all’, ‘a little’, ‘fairly’ and ‘a lot’) and were randomly assigned to a room and table. Children were seated in front of square tables. Individual plates were used in order to stick to maximize external validity, as all children bring their own afternoon snack. For first graders, nine tables were filled with four and one with three children. For sixth graders, ten tables were filled with four and two with three children. Children were told they could eat as much or as little as desired and were informed they would be given a refill if they liked to. They were allowed to talk but not to share their food. Experimenters ensured that the foods were not shared and if not consumed, were left on the table.

Let’s just stop right here. There’s a grand total of 85 kids in this experiment, about 20 in each experimental group (half in each grade level, half of each getting whole cookies and half getting small cookies). It was, apparently, run once. It was preceded by having the kids report their “pre-study hunger.” Kids were trooped over to a table and given instructions so specific that even a six-year-old would know something was up. Eight kids were excluded from the analysis due to an unexplained algorithm based on “presence of food allergies, overweight, weight problems, dieting behavior, food intake control in order to gain / lose weight and not hungry.” On the remaining data, “One-sample Kolmogorov-Smirnov was used to examine normality of data distributions and chi-square tests to analyze the independence of the sampling distributions for the habit to eat cookies.” The study apparently made no effort to see if reduced cookie eating had any effect on total caloric intake for the rest of the day. Nor did it follow up to see if the children’s behavior remained the same if they were offered the different cookie choices repeatedly over a period of weeks or months.

Look: portion control is probably important. No argument there. But this study is ridiculous. I have no idea how stuff like this gets published. The small sample size and one-time nature of the experiment should be enough to get it laughed out of town immediately, and the rest of the problems are just icing on the cake (so to speak). Unless someone can convince me that I’m being unfair here, I’d say that on the “Is it science or is it bullshit” scale, this one qualifies pretty firmly as bullshit.

UPDATE: For what it’s worth, no one in comments has convinced that this study is non-ridiculous. After reading a few of the defenses, however, I’m starting to have some doubts about the modern scientific enterprise.

THE BIG PICTURE

You expect the big picture, and it's our job at Mother Jones to give it to you. And right now, so many of the troubles we face are the making not of a virus, but of the quest for profit, political or economic (and not just from the man in the White House who could have offered leadership and comfort but instead gave us bleach).

In "News Is Just Like Waste Management," we unpack what the coronavirus crisis has meant for journalism, including Mother Jones’, and how we can rise to the challenge. If you're able to, this is a critical moment to support our nonprofit journalism with a donation: We've scoured our budget and made the cuts we can without impairing our mission, and we hope to raise $400,000 from our community of online readers to help keep our big reporting projects going because this extraordinary pandemic-plus-election year is no time to pull back.

THE BIG PICTURE

You expect the big picture, and it's our job at Mother Jones to give it to you. And right now, so many of the troubles we face are the making not of a virus, but of the quest for profit, political or economic (and not just from the man in the White House who could have offered leadership and comfort but instead gave us bleach).

In "News Is Just Like Waste Management," we unpack what the coronavirus crisis has meant for journalism, including Mother Jones’, and how we can rise to the challenge. If you're able to, this is a critical moment to support our nonprofit journalism with a donation: We've scoured our budget and made the cuts we can without impairing our mission, and we hope to raise $400,000 from our community of online readers to help keep our big reporting projects going because this extraordinary pandemic-plus-election year is no time to pull back.

We Recommend

Latest

Sign up for our newsletters

Subscribe and we'll send Mother Jones straight to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate

We have a new comment system! We are now using Coral, from Vox Media, for comments on all new articles. We'd love your feedback.