Tag Archives: physics envy

Prediction or Explanation?

Leave a reply

A couple weeks ago political scientist Jacqueline Stevens attacked her own discipline on the opinion pages of the New York Times. Her critique, as portrayed in the piece’s title, was that political scientists are “lousy forecasters.”

Many political scientists have responded to Stevens’ piece, from different angles and levels of rage, but one theme I’ve noticed is people disputing the assumption that political science’s goal is to make forecasts. For example, this recent letter to the editor of the Times by a professor emeritus of political science at the University of Iowa:

Forecasting is a very specialized field in political science, limited to predicting election outcomes, and the record in that field is impressive. But that is not what most political scientists do.

Political science, like most social sciences, seeks explanation rather than prediction. Its aim is to explain puzzling phenomena by relating them to phenomena that are well understood. Much of science consists of trying to resolve puzzles in that way.

I actually think prediction is what the majority of political scientists do. The distinction is that Stevens focuses on a couple of extraordinary historical cases, whereas most political science predictions involve more specific data sets. For example, theory might say that when we apply experimental treatment X, Y will increase or decrease. Or, someone might claim that Republicans give more to charity than Democrats, which is a prediction about what a data set of party identification and giving behavior would reveal.

It’s hard for me, in contrast, to think of a political science example of explanation that is divorced from prediction. I suppose the idea is to find a historical case–let’s take Stevens’ case of the end of the Cold War–and try to predict it retrospectively. But then why wouldn’t the prediction there apply to any future situations–why wouldn’t it also be a forecast? If the idea is that every historical case has to be considered on its own and that as a result their explanations tells us nothing about the future, I don’t know if that is science at all.

Physics Envy

Leave a reply

With a caveat that this is all filtered through an NPR show, I found this interview on the Higgs Boson `quasi-discovery’ relevant to current discussions of what’s supposedly wrong with social science.

I transcribed parts of the interview, which took place between WAMU host Celeste Headlee and Scientific American associate editor John Matson.

Headlee: Scientists needed to find out, whether or not the standard theory held water: why does matter hold mass? And they may have gotten their proof. I say may because we don’t know exactly what they’ve found yet, other than that it’s a new subatomic particle. Remember last year scientists at CERN claimed they had discovered particles that were faster than the speed of light. You remember that? Then they had to retract that claim. So the burden of proof when you discover something new…the standard is pretty high. The folks at today’s announcement needed to be certain if they had something…

Tell me about their certainty here. What standard did they use?

Matson: Sure. So the standard, uh sort of physics measure is sigma, or standard deviations. So they say if they have 3 sigma evidence that’s good for evidence of a new factor a new particle. 5 sigma is a discovery. And 5 sigma relates to a 1 in 3 and a 1/2 million chance that it’s just a statistical fluke, you know you’re just seeing some noise that looks like something real. And in this case they made it to that, they made it to 5 sigma, so this is certainly a very strong effect that they’re seeing.

Headlee: [...] So it’s like a 1 in 3.5 million chance it’s _not_ a new particle, right?

Matson: Well assuming that they’ve gotten everything correct, and that is where the faster than light particle finding comes in. That was a very high sigma effect, but wrong in other ways. So there’s always a chance there could be something funny going on, but in this case with this particle that looks like it could be the Higgs, that’s probably not going to happen, some mundane explanation, because there are two different experiments that are seeing what looks like the same thing, it’s similar to what’s been predicted for decades, so everything sort of rings true here, whereas with the faster than light neutrino thing last year, that sort of came out of no where, there was really no other experiment that supported that, and it went against decades and decades of scientific findings that said this shouldn’t be possible.

Here’s what caught my attention in particular.

  • Arbitrary standards of significance. In social science we have p-values, and apparently in physics they have sigmas, but they both contain the same information: how likely is it that what we’ve found is a result of random chance? Matson states that 5 sigma is the standard for a “discovery” in physics, where a discovery is not quite going the whole way. And he says that the researchers just barely made it to 5. But in fact, a researcher is quoted in the piece as saying (apparently at a press conference), “We conclude by saying that we have observed a new boson with a mass of 125.3, plus or minus .6 GV, at 4.9 standard deviations.”
  • Significance is only meaningful when combined with assumptions. A precisely estimated point estimate is meaningless if it is biased. As Matson is quick to point out, the significance comes from a null hypothesis that is based on model assumptions: it’s a 1 in 3.5 million chance they are wrong, “assuming they’ve gotten everything correct.”
  • Fragility of results. I like how the story keeps circling back to the “faster than light neutrino” finding and retraction from last year. It’s instructive how they use it as a baseline: we had a huge sigma there, but it was later retracted, so how do we avoid being fooled again? The two pieces of evidence they give are (1) replication–this finding comes from not one but two experiments, and (2) theory–the previous finding contradicted decades of other findings, but this one is consistent with decades of theory. While it makes sense to believe things that are consistent with other pieces of evidence–whether another experiment, other findings, or other theory–you can’t help but worry that this type of thinking causes us to reject useful information as well.
  • Theory testing. The researchers are motivated by the desire to test (implications of) theories. Some say that social science shouldn’t bother with this, but instead focus on “thinking deeply about what prompts human beings to behave the way they do.” Substitute “particles” for “human beings”, and this sort of advice would have prevented such an announcement. I also see an affinity with the CERN director quoted in the story as saying “To know that our maths, our equations, all our Greek symbols tell us some deep truth about everything and what everything is made of, and to have that verified with the discovery of the Higgs – that is one of the great, great moments in science.”