Is it Science or Is It Bullshit? (Cookie Crumbling Edition)

Fight disinformation: Sign up for the free Mother Jones Daily newsletter and follow the news that matters.

Sarah Kliff reports on a new study showing that portion size matters. Here’s the nickel version: If you offer kids a bowl full of cookies, they eat more than if you offer them a bowl full of smaller cookies.

Well, that sounded interesting. But I was curious about the details. So I found the original dissertation, which is ungated, and browsed through it. First there was this:

Recruitment was performed in an elementary school attended by one of the experimenter’s children. School director and legal guardians (i.e. mother/father) gave all written informed consent and both 1st and [] 6th grade classes participated voluntarily. The experimental protocol was approved by the Ethical Committee of the Faculty of Psychological Sciences of the Université Libre de Bruxelles.

OK, that has nothing to do with the research. It just cracks me up. They had to get IRB approval to feed some kids whole cookies and other kids half-size cookies! This is modern science, red in tooth and claw.

Moving on, there’s a detailed description of the cookies (several layers of wafers filled with milk chocolate topping) and the number of kids (85 total, about half of them girls). The experiment took place at 2:45 pm (the children’s “usual snack time”). Here’s the experimental design:

Kids were called up in alphabetical order, reported pre-study hunger (4-point scale labeled ‘not at all’, ‘a little’, ‘fairly’ and ‘a lot’) and were randomly assigned to a room and table. Children were seated in front of square tables. Individual plates were used in order to stick to maximize external validity, as all children bring their own afternoon snack. For first graders, nine tables were filled with four and one with three children. For sixth graders, ten tables were filled with four and two with three children. Children were told they could eat as much or as little as desired and were informed they would be given a refill if they liked to. They were allowed to talk but not to share their food. Experimenters ensured that the foods were not shared and if not consumed, were left on the table.

Let’s just stop right here. There’s a grand total of 85 kids in this experiment, about 20 in each experimental group (half in each grade level, half of each getting whole cookies and half getting small cookies). It was, apparently, run once. It was preceded by having the kids report their “pre-study hunger.” Kids were trooped over to a table and given instructions so specific that even a six-year-old would know something was up. Eight kids were excluded from the analysis due to an unexplained algorithm based on “presence of food allergies, overweight, weight problems, dieting behavior, food intake control in order to gain / lose weight and not hungry.” On the remaining data, “One-sample Kolmogorov-Smirnov was used to examine normality of data distributions and chi-square tests to analyze the independence of the sampling distributions for the habit to eat cookies.” The study apparently made no effort to see if reduced cookie eating had any effect on total caloric intake for the rest of the day. Nor did it follow up to see if the children’s behavior remained the same if they were offered the different cookie choices repeatedly over a period of weeks or months.

Look: portion control is probably important. No argument there. But this study is ridiculous. I have no idea how stuff like this gets published. The small sample size and one-time nature of the experiment should be enough to get it laughed out of town immediately, and the rest of the problems are just icing on the cake (so to speak). Unless someone can convince me that I’m being unfair here, I’d say that on the “Is it science or is it bullshit” scale, this one qualifies pretty firmly as bullshit.

UPDATE: For what it’s worth, no one in comments has convinced that this study is non-ridiculous. After reading a few of the defenses, however, I’m starting to have some doubts about the modern scientific enterprise.

AN IMPORTANT UPDATE

We’re falling behind our online fundraising goals and we can’t sustain coming up short on donations month after month. Perhaps you’ve heard? It is impossibly hard in the news business right now, with layoffs intensifying and fancy new startups and funding going kaput.

The crisis facing journalism and democracy isn’t going away anytime soon. And neither is Mother Jones, our readers, or our unique way of doing in-depth reporting that exists to bring about change.

Which is exactly why, despite the challenges we face, we just took a big gulp and joined forces with the Center for Investigative Reporting, a team of ace journalists who create the amazing podcast and public radio show Reveal.

If you can part with even just a few bucks, please help us pick up the pace of donations. We simply can’t afford to keep falling behind on our fundraising targets month after month.

Editor-in-Chief Clara Jeffery said it well to our team recently, and that team 100 percent includes readers like you who make it all possible: “This is a year to prove that we can pull off this merger, grow our audiences and impact, attract more funding and keep growing. More broadly, it’s a year when the very future of both journalism and democracy is on the line. We have to go for every important story, every reader/listener/viewer, and leave it all on the field. I’m very proud of all the hard work that’s gotten us to this moment, and confident that we can meet it.”

Let’s do this. If you can right now, please support Mother Jones and investigative journalism with an urgently needed donation today.

payment methods

AN IMPORTANT UPDATE

We’re falling behind our online fundraising goals and we can’t sustain coming up short on donations month after month. Perhaps you’ve heard? It is impossibly hard in the news business right now, with layoffs intensifying and fancy new startups and funding going kaput.

The crisis facing journalism and democracy isn’t going away anytime soon. And neither is Mother Jones, our readers, or our unique way of doing in-depth reporting that exists to bring about change.

Which is exactly why, despite the challenges we face, we just took a big gulp and joined forces with the Center for Investigative Reporting, a team of ace journalists who create the amazing podcast and public radio show Reveal.

If you can part with even just a few bucks, please help us pick up the pace of donations. We simply can’t afford to keep falling behind on our fundraising targets month after month.

Editor-in-Chief Clara Jeffery said it well to our team recently, and that team 100 percent includes readers like you who make it all possible: “This is a year to prove that we can pull off this merger, grow our audiences and impact, attract more funding and keep growing. More broadly, it’s a year when the very future of both journalism and democracy is on the line. We have to go for every important story, every reader/listener/viewer, and leave it all on the field. I’m very proud of all the hard work that’s gotten us to this moment, and confident that we can meet it.”

Let’s do this. If you can right now, please support Mother Jones and investigative journalism with an urgently needed donation today.

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate