Publication Bias Is Boring. You Should Care About It Anyway.

Fight disinformation: Sign up for the free Mother Jones Daily newsletter and follow the news that matters.


You all know about publication bias, don’t you? Sure you do. It’s the tendency to publish research that has bold, affirmative results and ignore research that concludes there’s nothing going on. This can happen two ways. First, it can be the researchers themselves who do it. In some cases that’s fine: the data just doesn’t amount to anything, so there’s nothing to write up. In other cases, it’s less fine: the data contradicts previous results, so you decide not to write it up.

The second way it can happen is via journal editors. Generally speaking, they prefer papers with exciting new results. This, of course, makes things even worse for researchers. They’re not much interested in writing up negative results in the first place, and they’re even less interested if they know there’s virtually no chance of getting them published in a good journal.

Why bring this up? Because yesterday Brian Resnick wrote about a truly remarkable case of publication bias. It revolves around research showing that the hormone oxytocin increases trust between humans. At first, multiple lines of research suggested it could. But then Anthony Lane, one of the primary researchers in the field, began getting negative results:

After 2010, fewer and fewer of their lab’s experiments yielded data that confirmed the oxytocin could reliably increase levels of trust….The doubts crested in 2014 when Lane and his colleagues couldn’t replicate their own envelope study….The lab was able to publish this negative finding, but Lane felt a larger problem was lurking. Labs are judged on the strength of their published work. And Lane’s published portfolio on oxytocin just wasn’t representative of their work anymore. They still had five papers showing promise for oxytocin, and only one casting doubt.

In a new paper published March in the journal Neuroendocrinology, Lane and his colleagues go through their “file drawer” of studies, and conclude the whole of their work yields an inconclusive result on the power of oxytocin spray to change behavior. They looked at 25 different tests their lab conducted. Only six of the 25 tests yielded significant results. In aggregate, the difference between the oxytocin sniffers in their studies and placebo groups “was not reliably different than zero,” the paper found.

When they tried to submit their null findings, they “were rejected time and time again,” the paper reports.

So there you go. Not just a boring “no result” study from an unknown researcher. Nor a trivial correction to previous work. This is a well-known researcher evaluating his entire output on a subject and concluding that he’s been wrong for years. And yet he couldn’t get it published.

Lots of null results don’t deserve to be published. But lots of them do: otherwise readers are left with an impression that the evidence is far more positive than it really is. That’s the case here: the overall impression Lane and others had left is that oxytocin works. It increases trust levels. Without a corrective, that’s what people would go on believing.

Everyone knows about this problem. Everyone agrees—in theory—that it needs to be addressed. And yet nothing ever seems to happen. Science deserves better.

WE'LL BE BLUNT.

We have a considerable $390,000 gap in our online fundraising budget that we have to close by June 30. There is no wiggle room, we've already cut everything we can, and we urgently need more readers to pitch in—especially from this specific blurb you're reading right now.

We'll also be quite transparent and level-headed with you about this.

In "News Never Pays," our fearless CEO, Monika Bauerlein, connects the dots on several concerning media trends that, taken together, expose the fallacy behind the tragic state of journalism right now: That the marketplace will take care of providing the free and independent press citizens in a democracy need, and the Next New Thing to invest millions in will fix the problem. Bottom line: Journalism that serves the people needs the support of the people. That's the Next New Thing.

And it's what MoJo and our community of readers have been doing for 47 years now.

But staying afloat is harder than ever.

In "This Is Not a Crisis. It's The New Normal," we explain, as matter-of-factly as we can, what exactly our finances look like, why this moment is particularly urgent, and how we can best communicate that without screaming OMG PLEASE HELP over and over. We also touch on our history and how our nonprofit model makes Mother Jones different than most of the news out there: Letting us go deep, focus on underreported beats, and bring unique perspectives to the day's news.

You're here for reporting like that, not fundraising, but one cannot exist without the other, and it's vitally important that we hit our intimidating $390,000 number in online donations by June 30.

And we hope you might consider pitching in before moving on to whatever it is you're about to do next. It's going to be a nail-biter, and we really need to see donations from this specific ask coming in strong if we're going to get there.

payment methods

WE'LL BE BLUNT.

We have a considerable $390,000 gap in our online fundraising budget that we have to close by June 30. There is no wiggle room, we've already cut everything we can, and we urgently need more readers to pitch in—especially from this specific blurb you're reading right now.

We'll also be quite transparent and level-headed with you about this.

In "News Never Pays," our fearless CEO, Monika Bauerlein, connects the dots on several concerning media trends that, taken together, expose the fallacy behind the tragic state of journalism right now: That the marketplace will take care of providing the free and independent press citizens in a democracy need, and the Next New Thing to invest millions in will fix the problem. Bottom line: Journalism that serves the people needs the support of the people. That's the Next New Thing.

And it's what MoJo and our community of readers have been doing for 47 years now.

But staying afloat is harder than ever.

In "This Is Not a Crisis. It's The New Normal," we explain, as matter-of-factly as we can, what exactly our finances look like, why this moment is particularly urgent, and how we can best communicate that without screaming OMG PLEASE HELP over and over. We also touch on our history and how our nonprofit model makes Mother Jones different than most of the news out there: Letting us go deep, focus on underreported beats, and bring unique perspectives to the day's news.

You're here for reporting like that, not fundraising, but one cannot exist without the other, and it's vitally important that we hit our intimidating $390,000 number in online donations by June 30.

And we hope you might consider pitching in before moving on to whatever it is you're about to do next. It's going to be a nail-biter, and we really need to see donations from this specific ask coming in strong if we're going to get there.

payment methods

We Recommend

Latest

Sign up for our free newsletter

Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox.

Get our award-winning magazine

Save big on a full year of investigations, ideas, and insights.

Subscribe

Support our journalism

Help Mother Jones' reporters dig deep with a tax-deductible donation.

Donate