Since (but not because) I linked to the hurricane study in my last post, many people have torn apart the study on various blogs. Andrew Gelman and Jeremy Freese have both been very insightful (and brutal). I didn’t read the study before posting, just glanced at the abstract and found it interesting. Aside from the observational data, the authors used a survey experiment to support their argument.
Now many have pointed out the results are not robust to small changes in the analysis — for example, dropping a couple hurricanes here and there. And survey experiments, well who believes those nowadays, given that that’s where p-hacking originated?
It’s interesting to watch the paper being criticized, but aside from whether it’s useful to do so (it probably isn’t), it’s also not very fair to single this paper out. Open any journal from any year, pull out a random article, and try to replicate it. I can almost guarantee you won’t be able to. Even if “replication data” is posted, all that does (in my experience) is show you more clearly how fragile the results are: download the data, make a few arbitrary changes, and see the results go away. In fact, it’s so guaranteed that replication assignments like this are common in graduate social science training.
So why target this harmless hurricane study? Is it because it’s making strong recommendations for policy — erm, how we name hurricanes? — and behavior — don’t judge a hurricane by it’s gender! — ? That can’t be the case, because social science journals are filled with policy recommendations that have no clear relationship with the strength of the statistical argument.
I think Andrew Gelman mentioned being particularly irked that the study was getting so much press attention. Or, that authors of studies like these have taken to putting out press releases along with the published paper. I don’t see that as a bad thing. Shouldn’t we be thanking these authors for shining a light on how science works? Or are we saying there should be another layer of peer review that deems whether a study merits a press release as well as a publication? Do we really think such a panel would be any more likely to catch mistakes?
I don’t want to come off as being in favor of shoddy work. I just think the only thing unusual about the hurricane study is that it got so much attention. I would guess the level of quality (by which I guess we mean robustness) is probably around the median, if not above, in the social science canon. And I guess I’m also just puzzled as to the point of lambasting these authors in particular. Aren’t we just falling victim to the same type of novelty bias that makes these studies newsworthy?