A new study being released today, by some of the country’s top health economists, aims to estimate the effects of not having health insurance — and the effects are large.

The researchers used a lottery that the state of Oregon conducted in 2008 to determine who would become eligible to apply for a limited number of Medicaid slots. The researchers compared the health outcomes of those who won the lottery (many of whom then received insurance) and those who did not (who were more likely to remain uninsured).

The researchers have followed the subjects for only a year so far, so the paper has some clear limitations. But it nonetheless suggests that having health insurance substantially improves health. Expanding insurance does not save society money — as some advocates of preventive medicine have claimed — but it does appear to make people mentally and physically healthier.

This is from New York Times writer/blogger David Leonhardt’s column today.

Sounds like a good paper, but I’m sure the limitations are not just limited to following the subjects only a year! But without reading the paper and just going on the summary in Leonhardt’s post, here’s one other problem I see:

[W]e find that insurance is associated with improvements across the board in our measures of self-reported physical and mental health, averaging two-tenths of a standard deviation improvement. These results appear to reflect improvements in mental health and also at least partly a general sense of improved well being; they may also reflect improvements in objective, physical health, but this is more difficult to determine with the data we now have available.

That seems really small!

It seems I pick on Leonhardt a lot, so I’ll throw in another example. Take the 4th of July-makes-you-a-Republican paper–surely you’ve heard of this, right? See this post at the Monkey Cage.

Here’s the abstract:

This paper investigates the role of Fourth of July celebrations in shaping political views and behavior in the United States. We study the impact of Fourth of July during childhood on partisanship and participation later in life. Our method uses daily precipitation data from 1920-1990 to proxy for exogenous variation in participation on Fourth of July as a child. The estimates imply that days without rain on Fourth of July in childhood increase the likelihood of identifying with the Republicans as an adult, voting for the Republican but not the Democratic candidate, and voter turnout. Our findings are significant: one Fourth of July without rain before age 18 increases the likelihood of identifying as a Republican at age 40 by 2 percent, the share of people voting for the Republican candidate at age 40 by 4 percent, and the share of people turning out to vote at age 40 by 0.9 percent. The evidence is consistent with childhood experience having foundational effects less susceptible to adult political influence. It also suggests that there is political congruence between patriotism promoted on Fourth of July and Republican beliefs, as well as Fourth of July transmitting a non-partisan civic duty to vote.

I commented on the original post that 2% seemed kind of small–though it’s true that this is the effect per rainy day. Like everyone else on the interwebz, I haven’t actually read this paper–just the abstract and skimmed the tables.

I think both of these examples are really impressive pieces of research–though I’m more fond of the second one. But they highlight another problem with press coverage of research in causal inference: the risk of failing to distinguish between “statistical” and “substantive” significance.

Turns out others have already posted about this problem elsewhere on the internet. See here, for example.