Is it Science or Is It Bullshit? (Cookie Crumbling Edition)

Facts matter: Sign up for the free Mother Jones Daily newsletter. Support our nonprofit reporting. Subscribe to our print magazine.

Sarah Kliff reports on a new study showing that portion size matters. Here’s the nickel version: If you offer kids a bowl full of cookies, they eat more than if you offer them a bowl full of smaller cookies.

Well, that sounded interesting. But I was curious about the details. So I found the original dissertation, which is ungated, and browsed through it. First there was this:

Recruitment was performed in an elementary school attended by one of the experimenter’s children. School director and legal guardians (i.e. mother/father) gave all written informed consent and both 1st and [] 6th grade classes participated voluntarily. The experimental protocol was approved by the Ethical Committee of the Faculty of Psychological Sciences of the Université Libre de Bruxelles.

OK, that has nothing to do with the research. It just cracks me up. They had to get IRB approval to feed some kids whole cookies and other kids half-size cookies! This is modern science, red in tooth and claw.

Moving on, there’s a detailed description of the cookies (several layers of wafers filled with milk chocolate topping) and the number of kids (85 total, about half of them girls). The experiment took place at 2:45 pm (the children’s “usual snack time”). Here’s the experimental design:

Kids were called up in alphabetical order, reported pre-study hunger (4-point scale labeled ‘not at all’, ‘a little’, ‘fairly’ and ‘a lot’) and were randomly assigned to a room and table. Children were seated in front of square tables. Individual plates were used in order to stick to maximize external validity, as all children bring their own afternoon snack. For first graders, nine tables were filled with four and one with three children. For sixth graders, ten tables were filled with four and two with three children. Children were told they could eat as much or as little as desired and were informed they would be given a refill if they liked to. They were allowed to talk but not to share their food. Experimenters ensured that the foods were not shared and if not consumed, were left on the table.

Let’s just stop right here. There’s a grand total of 85 kids in this experiment, about 20 in each experimental group (half in each grade level, half of each getting whole cookies and half getting small cookies). It was, apparently, run once. It was preceded by having the kids report their “pre-study hunger.” Kids were trooped over to a table and given instructions so specific that even a six-year-old would know something was up. Eight kids were excluded from the analysis due to an unexplained algorithm based on “presence of food allergies, overweight, weight problems, dieting behavior, food intake control in order to gain / lose weight and not hungry.” On the remaining data, “One-sample Kolmogorov-Smirnov was used to examine normality of data distributions and chi-square tests to analyze the independence of the sampling distributions for the habit to eat cookies.” The study apparently made no effort to see if reduced cookie eating had any effect on total caloric intake for the rest of the day. Nor did it follow up to see if the children’s behavior remained the same if they were offered the different cookie choices repeatedly over a period of weeks or months.

Look: portion control is probably important. No argument there. But this study is ridiculous. I have no idea how stuff like this gets published. The small sample size and one-time nature of the experiment should be enough to get it laughed out of town immediately, and the rest of the problems are just icing on the cake (so to speak). Unless someone can convince me that I’m being unfair here, I’d say that on the “Is it science or is it bullshit” scale, this one qualifies pretty firmly as bullshit.

UPDATE: For what it’s worth, no one in comments has convinced that this study is non-ridiculous. After reading a few of the defenses, however, I’m starting to have some doubts about the modern scientific enterprise.

THE TRUTH...

is the first thing despots go after. An unwavering commitment to it is probably what draws you to Mother Jones' journalism. And as we're seeing in the US and the world around, authoritarians seek to poison the discourse and the way we relate to each other because they can't stand people coming together around a shared sense of the truth—it's a huge threat to them.

Which is also a pretty great way to describe Mother Jones' mission: People coming together around the truth to hold power accountable.

And right now, we need to raise about $400,000 from our online readers over the next two months to hit our annual goal and make good on that mission. Read more about the information war we find ourselves in and how people-powered, independent reporting can and must rise to the challenge—and please support our team's truth-telling journalism with a donation if you can right now.

payment methods

THE TRUTH...

is the first thing despots go after. An unwavering commitment to it is probably what draws you to Mother Jones' journalism. And as we're seeing in the US and the world around, authoritarians seek to poison the discourse and the way we relate to each other because they can't stand people coming together around a shared sense of the truth—it's a huge threat to them.

Which is also a pretty great way to describe Mother Jones' mission: People coming together around the truth to hold power accountable.

And right now, we need to raise about $400,000 from our online readers over the next two months to hit our annual goal and make good on that mission. Read more about the information war we find ourselves in and how people-powered, independent reporting can and must rise to the challenge—and please support our team's truth-telling journalism with a donation if you can right now.

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate