Kevin Drum

I've been spending a bit of time this weekend trying to understand better what the real issues are with the Oregon Medicaid study that was released on Thursday and shortly afterward exploded across the blogosphere. Unfortunately, I've come to the conclusion that it's next to impossible to explain it in a way that would be understandable to most people. My readers, however, are not "most people," so I figure I'll take a crack at it anyway. The damage may have already been done from the many misinterpretations of the Oregon study that have been published over the past few days, but who knows? Maybe this will help anyway.

First, to refresh your memory: In 2008, Oregon expanded Medicaid coverage but didn't have enough money to cover everyone. So they ran a lottery. If you lost, you got nothing. If you won, you were offered the chance to sign up for Medicaid. This provided a unique opportunity to study the effect of Medicaid coverage, because Oregon provided two groups of people who were essentially identical except for the fact that one group (the control group) didn't have access to Medicaid, while the other group (the treatment group) did.

There are several things to say about the Oregon study, but I think the most important one is this: not that the study didn't find statistically significant improvements in various measures of health, but that the study couldn't have found statistically significant improvements. It was impossible from the beginning.

Here's why. The first thing the researchers should have done, before the study was even conducted, was estimate what a clinically significant result would be. For example, based on past experience, they might have decided that if access to Medicaid produced a 20 percent reduction in the share of the population with elevated levels of glycated hemoglobin (a common marker for diabetes), that would be a pretty successful intervention.

Then the researchers would move on to step two: suppose they found the clinically significant reduction they were hoping for? Is their study designed in such a way that a clinically significant result would also be statistically significant? Obviously it should be.

Let's do the math. In the Oregon study, 5.1 percent of the people in the control group had elevated GH levels. Now let's take a look at the treatment group. It started out with about 6,000 people who were offered Medicaid. Of that, 1,500 actually signed up. If you figure that 5.1 percent of them started out with elevated GH levels, that's about 80 people. A 20 percent reduction would be 16 people.

So here's the question: if the researchers ended up finding the result they hoped for (i.e., a reduction of 16 people with elevated GH levels), is there any chance that this result would be statistically significant? I can't say for sure without access to more data, but the answer is almost certainly no. It's just too small a number. Ditto for the other markers they looked at. In other words, even if they got the results they were hoping for, they were almost foreordained not to be statistically significant. And if they're not statistically significant, that means the headline result is "no effect."

The problem is that, for all practical purposes, the game was rigged ahead of time to produce this result. That's not the fault of the researchers. They were working with the Oregon Medicaid lottery, and they couldn't change the size of the sample group. What they had was 1,500 people, of whom about 5.1 percent started with elevated GH levels. There was no way to change that.

Given that, they probably shouldn't even have reported results. They should have simply reported that their test design was too underpowered to demonstrate statistically significant results under any plausible conditions. But they didn't do that. Instead, they reported their point estimates with some really big confidence intervals and left it at that, opening up a Pandora's Box of bad interpretations in the press.

Knowing all this, what's a fair thing to say about the results of this study?

• One fair thing would be to simply say that it's inconclusive, full stop. It tells us nothing about the effect of Medicaid access on diabetes, cholesterol levels, or blood pressure maintenance. I'm fine with that interpretation.
• Another fair thing would be to say that the results were positive, but the study was simply too small to tell us if the results are real.
• Or there's a third fair thing you could say: From a Bayesian perspective, the Oregon results should slightly increase our belief that access to Medicaid produces positive results for diabetes, cholesterol levels, and blood pressure maintenance. It shouldn't increase our belief much, but if you toss the positive point estimates into the stew of everything we already know, they add slightly to our prior belief that Medicaid is effective.
• But you can't say that the results are disappointing, at least not without a lot of caveats. At a minimum, the bare fact that the results aren't statistically significant certainly can't be described as a disappointment. That was baked into the cake from the beginning. This study was never likely to find significant results in the first place.

So that's that. You can't honestly say that the study shows that Medicaid "seemed to have little or no impact on common medical conditions like hypertension and diabetes." That just isn't what it showed.

POSTSCRIPT: That said, there are a few other things worth saying about this study too. For example, the researchers apparently didn't have estimates of clinical significance in mind before they conducted the study. That's odd, and it would be nice if they confirmed whether or not this is true. Also: the subjects of the study were an unusually healthy group, with pretty low levels of the chronic problems that were being measured. That makes substantial improvements even less likely than usual. And finally: on the metrics that had bigger sample sizes and could provide more reliable results (depression, financial security, self-reported health, etc.), the results of the study were uniformly positive and statistically significant.

• Print
• Email
• Tweet

Maria Bustillos writes in Aeon that when she drives from Los Angeles to San Francisco, she always takes I-5, the most direct route:

This is the way I always go. Not only because it’s fastest, but because, despite the conventional wisdom, I find it the most beautiful of all. California natives always call it just ‘The Five’: not Highway Five or Interstate Five or I-5.

Hoo boy. As soon as I read that I knew she was going to get some angry responses. Sure enough, here's the very first comment:

"The" in front of a highway number is NOT a CA native expression! It is rather, a SoCal thing based largely on CHP and LAFD radio chatter. It has become widespread all over the west into Arizona, Nevada and even Oregon, but is roundly discouraged in The Bay Area and in Norcal.

I'm not sure about Oregon, but this is definitely a Southern California thing that has spread throughout the southwest. It goes north to about San Luis Obispo or Monterey, but stops there. It's also, oddly enough, common in the Buffalo/Toronto area, but apparently nowhere else in the United States.

However, I don't think there's any evidence that this habit is based on CHP or LAFD chatter. Unfortunately, it's not clear what it is based on, since none of the explanations I've ever heard have really been convincing. The full story is here: Part 1, Part 2, Part 3.

• Print
• Email
• Tweet

Greg Sargent explores the immigration debate on the right, and explains why Marco Rubio and others are making the arguments they're making:

They are subtly making the case to their base that a defeat for immigration reform is actually a hidden victory for Obama....The idea is that if we don’t pass the Gang of Eight plan, Obama wins. This case is being made on several levels. On the one hand, this notion of leaving the issue “entirely in the hands of Obama” is a partly a suggestion that the President just may use his executive powers to solve the undocumented immigrant problem himself if we don’t pass the Senate plan — just as he did with the DREAMers.

....There’s a key nuance here. As I understand the thinking, GOP base voters are turned off by the political argument that we must reform immigration because if we don’t, Obama will be able to screw Republicans over politically with Latinos....That’s why the argument can’t be openly stated as: If we embrace reform, Obama loses. It has to be carefully calibrated in the manner Rubio has adopted: Not doing anything opens the door for a far greater victory for Obama later. He will be able to do for the undocumented what he did for the DREAMers — while not securing the border — a twofer for Obama.

Obama is playing his part in this dance, too. He and the White House frequently take care to say — not in these exact words, but this is the message — that while he supports the Senate compromise, it’s far from the liberal dream legislation he’d like.

Interesting. Basically, Sargent is suggesting that Republican base voters respond more strongly to the suggestion of a crafty Obama victory if they do nothing than they do to the possibility of an Obama loss if they do something. Partly that's because they've been trained to think of Obama as a cunning grifter constantly putting one over on them, and partly because they're not really convinced that passing immigration reform will help Republicans.

I'm not sure if I buy this or not. But I might!

• Print
• Email
• Tweet

Tyler Cowen links to a couple of writers today who say the recent study of the Oregon Medicaid experiment was bad news for Medicaid fans because it showed that Medicaid coverage had no effect on most of the medical conditions that were studied. His summary:

Do read the rest of those posts for a more complete picture of the results, but many commentators are overlooking these rather simple upshots.

It's disappointing to keep reading this stuff, because it's flatly not true. There are three main takeaways from the study:

1. There were positive results on some measures (depression, financial security, rates of diagnosis).
2. There were also positive results on all of the other measures studied (blood pressure, diabetes, cholesterol).
3. But the size of the study was too small to determine if the positive results in #2 were real. This says nothing at all about Medicaid. It just says that, unfortunately, the experiment was too small to be definitive.

In addition, the study says nothing at all about some more fundamental questions. Is Medicaid well run? Does it deliver better performance per dollar than some other programs? Are there useful ways it could be reformed? Even if the results from #2 turn out to be real and significant, are they worth the cost? For that matter, is healthcare in general worth the price we pay in America? Those are all great questions that we should spend a lot of time investigating, but this particular study simply says nothing one way or the other about any of them. Strictly in terms of how effective Medicaid is, Austin Frakt and Aaron Carroll have it right when they tell us how much this study should change our thinking: "Not that much."

Speaking for myself, I'd also repeat—over and over and over until at least one person responds to this—that the study looked only at a few easily measurable chronic conditions. These are important, but they're not the bulk of what medical care is about. Most of it is about routine preventive care and acute problems: broken bones, infections, flu shots, immunizations, etc. etc. I'm genuinely puzzled by the fact that virtually no one seems to acknowledge this.

But please, please, please: don't say the Medicaid study "showed no effects." It ain't true. It showed positive effects, but it was too small and underpowered to tell us for sure if these positive effects were real. That's it.

• Print
• Email
• Tweet