Is it Science or Is It Bullshit? (Cookie Crumbling Edition)

Let our journalists help you make sense of the noise: Subscribe to the Mother Jones Daily newsletter and get a recap of news that matters.

Sarah Kliff reports on a new study showing that portion size matters. Here’s the nickel version: If you offer kids a bowl full of cookies, they eat more than if you offer them a bowl full of smaller cookies.

Well, that sounded interesting. But I was curious about the details. So I found the original dissertation, which is ungated, and browsed through it. First there was this:

Recruitment was performed in an elementary school attended by one of the experimenter’s children. School director and legal guardians (i.e. mother/father) gave all written informed consent and both 1st and [] 6th grade classes participated voluntarily. The experimental protocol was approved by the Ethical Committee of the Faculty of Psychological Sciences of the Université Libre de Bruxelles.

OK, that has nothing to do with the research. It just cracks me up. They had to get IRB approval to feed some kids whole cookies and other kids half-size cookies! This is modern science, red in tooth and claw.

Moving on, there’s a detailed description of the cookies (several layers of wafers filled with milk chocolate topping) and the number of kids (85 total, about half of them girls). The experiment took place at 2:45 pm (the children’s “usual snack time”). Here’s the experimental design:

Kids were called up in alphabetical order, reported pre-study hunger (4-point scale labeled ‘not at all’, ‘a little’, ‘fairly’ and ‘a lot’) and were randomly assigned to a room and table. Children were seated in front of square tables. Individual plates were used in order to stick to maximize external validity, as all children bring their own afternoon snack. For first graders, nine tables were filled with four and one with three children. For sixth graders, ten tables were filled with four and two with three children. Children were told they could eat as much or as little as desired and were informed they would be given a refill if they liked to. They were allowed to talk but not to share their food. Experimenters ensured that the foods were not shared and if not consumed, were left on the table.

Let’s just stop right here. There’s a grand total of 85 kids in this experiment, about 20 in each experimental group (half in each grade level, half of each getting whole cookies and half getting small cookies). It was, apparently, run once. It was preceded by having the kids report their “pre-study hunger.” Kids were trooped over to a table and given instructions so specific that even a six-year-old would know something was up. Eight kids were excluded from the analysis due to an unexplained algorithm based on “presence of food allergies, overweight, weight problems, dieting behavior, food intake control in order to gain / lose weight and not hungry.” On the remaining data, “One-sample Kolmogorov-Smirnov was used to examine normality of data distributions and chi-square tests to analyze the independence of the sampling distributions for the habit to eat cookies.” The study apparently made no effort to see if reduced cookie eating had any effect on total caloric intake for the rest of the day. Nor did it follow up to see if the children’s behavior remained the same if they were offered the different cookie choices repeatedly over a period of weeks or months.

Look: portion control is probably important. No argument there. But this study is ridiculous. I have no idea how stuff like this gets published. The small sample size and one-time nature of the experiment should be enough to get it laughed out of town immediately, and the rest of the problems are just icing on the cake (so to speak). Unless someone can convince me that I’m being unfair here, I’d say that on the “Is it science or is it bullshit” scale, this one qualifies pretty firmly as bullshit.

UPDATE: For what it’s worth, no one in comments has convinced that this study is non-ridiculous. After reading a few of the defenses, however, I’m starting to have some doubts about the modern scientific enterprise.

IT'S NOT THAT WE'RE SCREWED WITHOUT TRUMP:

"It's that we're screwed with or without him if we can't show the public that what we do matters for the long term," writes Mother Jones CEO Monika Bauerlein as she kicks off our drive to raise $350,000 in donations from readers by July 17.

This is a big one for us. It's our first time asking for an outpouring of support since screams of FAKE NEWS and so much of what Trump stood for made everything we do so visceral. Like most newsrooms, we face incredibly hard budget realities, and it's unnerving needing to raise big money when traffic is down.

So, as we ask you to consider supporting our team's journalism, we thought we'd slow down and check in about where Mother Jones is and where we're going after the chaotic last several years. This comparatively slow moment is also an urgent one for Mother Jones: You can read more in "Slow News Is Good News," and if you're able to, please support our team's hard-hitting journalism and help us reach our big $350,000 goal with a donation today.

payment methods

IT'S NOT THAT WE'RE SCREWED WITHOUT TRUMP:

"It's that we're screwed with or without him if we can't show the public that what we do matters for the long term," writes Mother Jones CEO Monika Bauerlein as she kicks off our drive to raise $350,000 in donations from readers by July 17.

This is a big one for us. So, as we ask you to consider supporting our team's journalism, we thought we'd slow down and check in about where Mother Jones is and where we're going after the chaotic last several years. This comparatively slow moment is also an urgent one for Mother Jones: You can read more in "Slow News Is Good News," and if you're able to, please support our team's hard-hitting journalism and help us reach our big $350,000 goal with a donation today.

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate