“Surprising studies attract more scrutiny than unsurprising studies because of human nature, study finds”

Since (but not because) I linked to the hurricane study in my last post, many people have torn apart the study on various blogs. Andrew Gelman and Jeremy Freese have both been very insightful (and brutal). I didn’t read the study before posting, just glanced at the abstract and found it interesting. Aside from the observational data, the authors used a survey experiment to support their argument.

Now many have pointed out the results are not robust to small changes in the analysis — for example, dropping a couple hurricanes here and there. And survey experiments, well who believes those nowadays, given that that’s where p-hacking originated?

It’s interesting to watch the paper being criticized, but aside from whether it’s useful to do so (it probably isn’t), it’s also not very fair to single this paper out. Open any journal from any year, pull out a random article, and try to replicate it. I can almost guarantee you won’t be able to. Even if “replication data” is posted, all that does (in my experience) is show you more clearly how fragile the results are: download the data, make a few arbitrary changes, and see the results go away. In fact, it’s so guaranteed that replication assignments like this are common in graduate social science training.

So why target this harmless hurricane study? Is it because it’s making strong recommendations for policy — erm, how we name hurricanes? — and behavior — don’t judge a hurricane by it’s gender! — ? That can’t be the case, because social science journals are filled with policy recommendations that have no clear relationship with the strength of the statistical argument.

I think Andrew Gelman mentioned being particularly irked that the study was getting so much press attention. Or, that authors of studies like these have taken to putting out press releases along with the published paper. I don’t see that as a bad thing. Shouldn’t we be thanking these authors for shining a light on how science works? Or are we saying there should be another layer of peer review that deems whether a study merits a press release as well as a publication? Do we really think such a panel would be any more likely to catch mistakes?

I don’t want to come off as being in favor of shoddy work. I just think the only thing unusual about the hurricane study is that it got so much attention. I would guess the level of quality (by which I guess we mean robustness) is probably around the median, if not above, in the social science canon. And I guess I’m also just puzzled as to the point of lambasting these authors in particular. Aren’t we just falling victim to the same type of novelty bias that makes these studies newsworthy?

Great new paper on Citizens United

What effect has Citizens United v. FEC had on independent spending in American politics? Previous attempts to answer this question have focused solely on federal elections where there is no baseline for comparing changes in spending behavior. We overcome this limitation by examining the effects of Citizens United as a natural experiment on the states. Before Citizens United about half of the states banned corporate independent expenditures and thus were “treated” by the Supreme Court’s decision, which invalidated these state laws. We rely on recently released state-level data to compare spending in “treated” states to spending in the “control” states that have never banned corporate or union independent expenditures. We find that while independent expenditures increased in both treated and control states between 2006 and 2010, the increase was more than twice as large in the treated states and nearly all of the new money was funneled through nonprofit organizations and political committees where weak disclosure laws and practices protected the anonymity of the spenders. Finally, we observe that the increase in spending after Citizens United was not the product of fewer, larger expenditures as many scholars and pundits predicted, and we note that people were just as likely to make smaller expenditures (less than $400) after Citizens United as they were before. This finding is particularly striking because it cuts against the conventional wisdom of spending behavior and also challenges the logic of those who disagree with the most controversial element of the Citizens United decision – the rejection of political equality as a valid state interest.

Never did I think I’d be reading a diff-in-diff in a law journal. And then they had a quantile regression too! And all with graphs! (And not to mention — the substantive conclusions are pretty interesting too.)

The paper is here. Via (who else?) Rick Hasen.

Is this the best they can come up with?

Via Rick Hasen, this NY Times article purports to document corruption in New York state government.

I’m a little slow sometimes, and I tend to read articles fast as a result of my past habit of trying to read as many news articles as possible every morning. But I didn’t get the corruption from this article. Let’s go through it slowly for both our benefits (meaning me and the spambots).

Count 1: the article opens with a “bald expression of transactional expectations” revealed by internal communications from a “politically connected organization.” Persons were complaining that they had given a lot to candidates, but had not gotten enough grant money in return.

My reaction: one-sided expectations do not a contract make. Also, doesn’t the fact that they are complaining about not getting their money’s work tell us that things aren’t as bad as they might be? Bear in mind that the article describes this as “among the most striking examples of possible misconduct” revealed by the state commission investigating corruption.

ok, reading, reading, bla bla bla money in politics is bad, bla bla ethics reforms needed. Finally another instance of alleged corruption:

Count 2:

In one case, they were able to find a company that lobbied vigorously for passage of legislation, and then, after its passage, gave both major political parties large contributions directed through what the report described as “shadowy corporate affiliates with generic names that do not readily appear to have anything to do with the company.”

My reaction: Sounds potentially bad, but unfortunately we can’t infer anything from this. We have no idea whether, as the article seems to imply, the lobbying consisted of making a deal to pass the legislation in exchange for contributions. Also as the article notes, this is all perfectly legal.

Count 3:

The commission also expressed interest in how money influenced a tax abatement program for real estate developers, a wage law exemption for a large retailer and other “custom-tailored laws” that a particularly powerful lobbyist won for a wide-ranging group of high-paying clients.

My reaction: More innuendo, but there’s no information here. The commission is “interested” in these things, but the article has no evidence that money actually influenced these outcomes.

Count 4:

The commission found repeated evidence that money influenced governmental action.

In one investigation, a lobbyist negotiating with a prospective client provided the client with a “fair projection of expenses” that included not only the lobbyist’s fees, but also expensive “political contributions” that the client would have to make to politicians, including the chairmen of the legislative committees that had jurisdiction over a certain bill before the Legislature.

My reaction: OK, this may be crossing the line. But again, an expression of transactional expectations doesn’t convince me that corruption is happening.

Does this mean I don’t think there is any corruption in American politics? No, but I just think this article misses the mark. Incidentally, Rick Hasen posted another article that I think is relatively clear-cut, involving a corrupt county in Kentucky.



Imbens and Manski on validity

Guido Imbens recently reviewed a new book by Charles Manski in the Economic Journal. Manski then responded. The issue with both articles may be found here. I read this a few days ago, but for some reason was recently reminded of this particular back and forth on internal and external validity.

One issue Manski raises is the relative importance of internal versus external validity.
He credits Campbell (Campbell and Stanley, 1963; Campbell, 1984) with the claim
‘that studies should be judged primarily by their internal validity and only secondarily
by their external validity’ (Manski, p. 36). Manski takes issue with that by claiming that
‘from the perspective of policy choice, it makes no sense to value one type of validity
above the other’ (Manski, p. 37). Deaton (2010) makes a similar point in the context of
his criticism of the use of randomised experiments in development economics.
I strongly disagree with the Manski and Deaton view. Without strong internal validity
studies have little to contribute to policy debates, whereas studies with very limited
external validity often are, and in my view should be, taken seriously in such

Manski, in his reply, writes:

Imbens and I diverge sharply on the subject of internal and external validity. To
characterise my perspective, he quotes the opening sentence of a paragraph in my new
book. I think it is important to quote the paragraph in full (Manski, 2013, p. 37):

‘From the perspective of policy choice, it makes no sense to value one type of validity
above the other. What matters is the informativeness of a study for policy making,
which depends jointly on internal and external validity. Hence, research should
strive to measure both types of validity’.

The second and third sentences explain the rationale for the first.

Manski is one of my favorite economists, so no surprise that I agree with his view more than I do with Imbens. “What matters is the informativeness of a study for policy making…” I might change “policy making” to “decision-making,” given that not all social science is (or ought to be) aimed at guiding elites. But otherwise I think this a great way to evaluate research.


Is “explanation” science?

Don’t remember where I found this, but here it is: a letter to the editor of the NY Times from economist Eric Maskin on whether economics is a science:

I disagree with Alex Rosenberg and Tyler Curtain’s characterization of science in general and economics in particular. They claim that a scientific discipline is to be judged primarily on its predictions, and on that basis, they suggest, economics doesn’t qualify as a science.

Prediction is certainly a valuable goal in science, but not the only one. Explanation is also important, and there are plenty of sciences that do a lot of explaining and not much predicting. Seismology, for example, has taught us why earthquakes occur, but doesn’t tell Californians when they’ll be hit by “the big one.”

And through meteorology we know essentially how hurricanes form, even though we can’t say where the next storm will arise.

I’ve commented before that I don’t “get” the difference between explanation and prediction.  Maskin, a quite reputable economist, seems to make a hard distinction. Not knowing anything about seismology or meteorology, I can only speculate: is the idea that “explaining” earthquakes and hurricanes is a precursor to an ultimate goal of predicting them, someday?

Indeed another letter-writer David Berman makes a similar point at the above link. Berman’s letter begins:

Mr. Maskin’s distinction between prediction and explanation as different but equally valuable goals of science is a false one — the two are inextricably linked. The ability to predict is both what makes our explanations useful, and what confirms that our explanations are correct.

Mr. Maskin’s choice of meteorology as example is a good one. It’s true that in the long run “we can’t say where the next storm will arise,” but we are now very good at forecasting the short-term futures of tropical storms, so good that we can predict landfalls within miles and within hours, and give populations ample time to protect themselves (if people would only listen).

And associate professor of philosophy “SAMUEL RUHMKORFF” makes a similar point at the above link: “In fact, economists make predictions, and their status as scientists should be judged by their willingness to revise their theories when these predictions do not bear out.” I like that — judged based on willingness to revise theory, not whether the predictions are “right”. And reader “JOHN DOUARD” also makes this point.

Maskin also has a response to these letter-writers at the end of the page at the above link, but he seems to avoid the point about whether the explanation-prediction distinction is a valid one.