With a caveat that this is all filtered through an NPR show, I found this interview on the Higgs Boson `quasi-discovery’ relevant to current discussions of what’s supposedly wrong with social science.

I transcribed parts of the interview, which took place between WAMU host Celeste Headlee and Scientific American associate editor John Matson.

Headlee: Scientists needed to find out, whether or not the standard theory held water: why does matter hold mass? And they may have gotten their proof. I say may because we don’t know exactly what they’ve found yet, other than that it’s a new subatomic particle. Remember last year scientists at CERN claimed they had discovered particles that were faster than the speed of light. You remember that? Then they had to retract that claim. So the burden of proof when you discover something new…the standard is pretty high. The folks at today’s announcement needed to be certain if they had something…

Tell me about their certainty here. What standard did they use?

Matson: Sure. So the standard, uh sort of physics measure is sigma, or standard deviations. So they say if they have 3 sigma evidence that’s good for evidence of a new factor a new particle. 5 sigma is a discovery. And 5 sigma relates to a 1 in 3 and a 1/2 million chance that it’s just a statistical fluke, you know you’re just seeing some noise that looks like something real. And in this case they made it to that, they made it to 5 sigma, so this is certainly a very strong effect that they’re seeing.

Headlee: [...] So it’s like a 1 in 3.5 million chance it’s _not_ a new particle, right?

Matson: Well assuming that they’ve gotten everything correct, and that is where the faster than light particle finding comes in. That was a very high sigma effect, but wrong in other ways. So there’s always a chance there could be something funny going on, but in this case with this particle that looks like it could be the Higgs, that’s probably not going to happen, some mundane explanation, because there are two different experiments that are seeing what looks like the same thing, it’s similar to what’s been predicted for decades, so everything sort of rings true here, whereas with the faster than light neutrino thing last year, that sort of came out of no where, there was really no other experiment that supported that, and it went against decades and decades of scientific findings that said this shouldn’t be possible.

Here’s what caught my attention in particular.

• Arbitrary standards of significance. In social science we have p-values, and apparently in physics they have sigmas, but they both contain the same information: how likely is it that what we’ve found is a result of random chance? Matson states that 5 sigma is the standard for a “discovery” in physics, where a discovery is not quite going the whole way. And he says that the researchers just barely made it to 5. But in fact, a researcher is quoted in the piece as saying (apparently at a press conference), “We conclude by saying that we have observed a new boson with a mass of 125.3, plus or minus .6 GV, at 4.9 standard deviations.”
• Significance is only meaningful when combined with assumptions. A precisely estimated point estimate is meaningless if it is biased. As Matson is quick to point out, the significance comes from a null hypothesis that is based on model assumptions: it’s a 1 in 3.5 million chance they are wrong, “assuming they’ve gotten everything correct.”
• Fragility of results. I like how the story keeps circling back to the “faster than light neutrino” finding and retraction from last year. It’s instructive how they use it as a baseline: we had a huge sigma there, but it was later retracted, so how do we avoid being fooled again? The two pieces of evidence they give are (1) replication–this finding comes from not one but two experiments, and (2) theory–the previous finding contradicted decades of other findings, but this one is consistent with decades of theory. While it makes sense to believe things that are consistent with other pieces of evidence–whether another experiment, other findings, or other theory–you can’t help but worry that this type of thinking causes us to reject useful information as well.
• Theory testing. The researchers are motivated by the desire to test (implications of) theories. Some say that social science shouldn’t bother with this, but instead focus on “thinking deeply about what prompts human beings to behave the way they do.” Substitute “particles” for “human beings”, and this sort of advice would have prevented such an announcement. I also see an affinity with the CERN director quoted in the story as saying “To know that our maths, our equations, all our Greek symbols tell us some deep truth about everything and what everything is made of, and to have that verified with the discovery of the Higgs – that is one of the great, great moments in science.”