In oral arguments about the Voting Rights Act last week, Chief Justice John Roberts pressed attorneys on which state had a better record of turning out African-American voters. Was it Massachusetts or Mississippi? The answer appears to be Mississippi, but NPR’s 
Nina Totenberg suggests this might be a statistical illusion caused by small sample sizes in a Census survey:
The number of black citizens eligible to vote in Massachusetts is 236,000, while it is 721,000 in Mississippi, more than three times that number. Therefore, according to Census officials, when looking at the estimated turnout rate in Massachusetts, the voting percentage for African-Americans at first blush is estimated at 39.3 percent. But the margin of error is 11.5 percentage points, meaning that the black voter turnout actually could be as high as 50.8 percent (or, conversely, as low as 27.8 percent).
Now, look at Mississippi, where black turnout is listed at 48.7 percent. But because of the large size of the African-American population that was sampled, the margin of error is only 5.4 percentage points.
For Roberts, Mississippi clearly tops Massachusetts. But given the small sample size, it’s equally plausible, statistically speaking, that Massachusetts has an African-American voter turnout rate that’s 7.5% better than Mississippi’s. Totenberg added, “Bottom line, as Census officials told me, these numbers are simply not reliable for state-by-state comparisons because of the high margins of error in some states.”
I have to take Roberts’ side here. It’s true that the sample sizes in the Census surveys are small, which means their results shouldn’t be considered authoritative. Still, it’s really not “equally plausible” that Massachusetts might be doing better than Mississippi. I don’t have the data to plug into a two-sample z-test to figure this out, but my horseback guess is that there’s maybe a 5 percent chance that the Census results are merely due to sample error and Massachusetts is truly doing better than Mississippi. The most probable true result, by far, is that Mississippi really does have a higher percentage turnout among its black population than Massachusetts does.
Totenberg’s premise is similar to the “statistical tie” beloved of horserace analysts during presidential campaigns. But as we all know, it doesn’t really work there either.
