This course aims to provide a firm grounding in the foundations of probability and statistics. Specific topics include:

1. Describing data (types of data, data visualization, descriptive statistics)
2. Statistical inference (probability, probability distributions, sampling theory, hypothesis testing, confidence intervals, pitfalls of p-values)
3. Specific statistical tests (ttest, ANOVA, linear correlation, non-parametric tests, relative risks, Chi-square test, exact tests, linear regression, logistic regression, survival analysis; how to choose the right statistical test)

The course focuses on real examples from the medical literature and popular press. Each week starts with “teasers,” such as: Should I be worried about lead in lipstick? Should I play the lottery when the jackpot reaches half-a-billion dollars? Does eating red meat increase my risk of being in a traffic accident? We will work our way back from the news coverage to the original study and then to the underlying data. In the process, students will learn how to read, interpret, and critically evaluate the statistics in medical studies.

The course also prepares students to be able to analyze their own data, guiding them on how to choose the correct statistical test and how to avoid common statistical pitfalls. Optional modules cover advanced math topics and basic data analysis in R.

I think it sounds like a great way to teach introductory statistics, mainly because of the “teasers” idea.

The reason is that, as part of my dissertation research, I’m studying local elections in Virginia, and have been requesting that county election offices provide me with election results dating back to the early 1990s. A few counties outright refused to help me, citing the state law that says they don’t have to. The majority just did not respond. The state election office even decided to get in touch with me, apparently after some localities contacted them about my request, reminding me that their offices are under no obligation to help me. (Naturally, I tried to tailor my requests to be as polite as possible, and made clear that I was willing to reimburse local offices for any fees associated with my request.)

Of course I knew about this clause of the state law going into my request — probably I learned about it from the Virginia Coalition for Open Government web site, which was involved with the case and was on the side of the plaintiffs who argued the residency restriction was unconstitutional. All is not lost for me, anyway — I have since been in contact with a researcher in Virginia and we’re planning on completing the requests soon.

I have two comments about the On the Media report about the decision. First, I found it really strange that Justice Alito, writing for the Court, seems to believe that “Requiring noncitizens to conduct a few minutes’ of Internet research in lieu of using a relatively cumbersome state FOIA process cannot be said to impose any significant burden.” Assuming this is not horribly taken out of context, it’s just wrong — if government information was readily available on the Internet, then no one would be doing FOIA requests in the first place. Indeed, in my experience, the results of FOIA requests are often paper records — if you’re lucky, the person processing the request will scan them for you. So they often aren’t even digitized, let alone on a web site.

My second comment relates to a point raised by Mark Caramanica of Reporters Committee for Freedom of the Press. He points out that these type of residency requirements threaten evaluations of government policy — how well are different states doing on certain benchmarks:

For example, USA Today did a story where they compared how the No Child Left Behind initiative was impacting teacher behavior in a variety of jurisdictions and pulled records dealing whether teachers were disciplined for potentially coaching students and, you know, trying to cut corners on the standardized tests, and so forth, to boost up their scores. That was a case where, you know, you need access to Virginia records, obviously, to put Virginia in the mix.

ProPublica did a story where they investigated dialysis centers all across the country, their effectiveness and safety records, and so forth. To get that national perspective, you need Virginia. And, you know, it leaves a hole in the entire story and the complete picture, but it also shortchanges citizens of Virginia because they don’t get the benefit of knowing where their state stacks up on certain issues.

(Of course, this type of cross-state comparison is more commonly done by social scientists, including political scientists who study states but also, maybe even more often, by economists. While a large number of press organizations, including the New York Times, filed an amicus brief on the side of the plaintiffs in this case, did any social science organizations? Apparently not. SCOTUSblog has all the amica briefs for the case, and I don’t see any.)

Maybe it’s not surprising that the most insular branch, not especially friendly to sharing information in its own domain, decided this way. On the other hand, a famous Justice once spoke eloquently of the potential of states as generators of good public policy. With this decision, the Court has made this potential just a little bit harder to realize.

I want to be careful not to exaggerate. For one, already, workarounds are apparently in the works. For another, VA is apparently one of 8 states with such a clause. But I’ve found some states don’t specify residency — as a researcher, I worry that the next time I make a request to a state with an ambiguous law, they might decide to refuse me on residency grounds.

The case of Even Start stands out because it is so rare. At a time when the federal budget is increasingly squeezed — and lawmakers are wrestling with tough choices on what to cut or to keep — the government does very, very little to find out which programs produce the best results for the money spent on them.

There are several reasons for that, most wrapped up in politics. There’s no natural ideological constituency for program evaluations. Lawmakers who champion social programs often fear that attempts to measure them will be only thinly disguised excuses to kill the programs. Fiscal hawks don’t often love the idea of spending more money on evaluations.

The frame of the article is that recently some economists are calling for a more institutionalized process for evaluating government programs. One proposal is to start by including evaluations in decisions for federal transfers to state and local governments:

[Harvard economist and former Obama budget official] Jeffrey Liebman’s most specific proposal would change the flow of federal money to state and local governments. He would require that 1 percent of such grants be set aside for programs whose effectiveness has been proven through randomized or other rigorous research methods. Over time, the requirement would rise to 5 percent.

Given there is “no natural ideological constituency” for this sort of thing, how do the economists expect policy makers to get on board? Apparently there are some assumptions about the wisdom of voters and the electoral incentive.

But if Washington ever hopes to provide the services voters say they want, at the tax rates voters say they’re willing to pay, economists say the government will need to ramp up its efforts to figure out which programs work and which ones don’t, and shift resources accordingly.

[...]

That’s especially bad at a time when Washington is debating tax increases and spending cuts to reduce the federal deficit, said Jeffrey Liebman, a Harvard economist and also a former Obama budget official. “It’s imperative to be able to show that the things [voters’] tax dollars are being spent on work, and that we’re trying to improve performance and do it in a data-driven way,” Liebman said. “That’s just good stewardship.”

Child sponsorship is a leading form of direct aid from wealthy country households to children in developing countries. Over 9 million children are supported through international sponsorship organizations. Using data from six countries, we estimate impacts on several outcomes from sponsorship through Compassion International, a leading child sponsorship organization. To identify program effects, we utilize an age-eligibility rule implemented when programs began in new villages. We find large, statistically significant impacts on years of schooling; primary, secondary, and tertiary school completion; and the probability and quality of employment. Early evidence suggests that these impacts are due, in part, to increases in children’s aspirations.

The view that small businesses create the most jobs remains appealing to policymakers and small business advocates. Using data from the Census Bureau Business Dynamics Statistics and Longitudinal Business Database, we explore the many issues at the core of this ongoing debate. We find that the relationship between firm size and employment growth is sensitive to these issues. However, our main finding is that once we control for firm age, there is no systematic relationship between firm size and growth. Our findings highlight the important role of business start-ups and young businesses in U.S. job creation.

But apparently there’s been lots of popular commentary about whether this claim is true as well.

Political science produces highly policy-relevant research, but politicians ignore it in favor of their own (or their supporters’) biases. I give examples from such fields as anti-immigrant politics, political business cycles and the politics of redistribution. The sole area in which politicians do attend closely to scholarly research is where it assists their own efforts at electoral success (e.g. effect and duration of political advertising). But politicians equally ignore the expertise of climatologists, physicists, biologists, economists and even spies, where that expertise contradicts their own preferred policies. All of this points more to a problem of democratic politics than of political (or any other) science.

I’m realizing that we probably left out an important third explanation: the relationship is an artifact, either a technical error or, perhaps, willful manipulation of the data. (Well, we did always tell students to always look at their own data, but we probably could have emphasized this more in how they evaluate research by others.) Recent events make me think that technical errors are probably more common than I previously thought.

The particular event I have in mind is the Reinhart-Rogoff fiasco. The (perhaps overblown — the 2010s are still young after all) money quote from one observer of the situation:

one of the core empirical points providing the intellectual foundation for the global move to austerity in the early 2010s was based on someone accidentally not updating a row formula in Excel.

But this follow-up on Andrew Gelman’s blog, which details all the seemingly common and easy-to-make mistakes that occur even using more sophisticated statistical software. I have to say I think point #5 at that post is the best, and again is actually just another version of the advice we gave to our undergraduates: always show your data!

Anyway, see this post by another UMass Amherst economist for what seems like a terrific explanation of why reverse causality is actually the more serious problem for Rogoff-Reinhart.

I have spent the last 20+ years of my life doing academic research and popular writing on economics.  I’ve been lucky, and my work has gotten a lot of exposure.  I certainly have had a lot of fun along the way.

But, I think I can honestly say that no government has ever changed a law or a public policy as a result of my work.  Sometimes politicians cite my research in pushing an agenda but having talked to these politicians, it is clear they had the agenda first, and then they went looking for research – any research – that would support their position.  When I’ve taken unpopular stances (like saying children’s car seats don’t work well), there has never been even a sliver of political movement on the issue.

I found this depressing, because while it’s slightly tongue-in-cheek, I take Levitt to be serious. From what I know, Levitt is one of the most publicly engaged economists, co-author of the best-selling Freakonomics and many interesting and technically sophisticated papers on important public policy questions, and now a frequent blogger and contributor to NPR. So if anyone would be expected to have an influence on public policy, it would be Levitt. It’s also my impression that economics as a field is the most influential in policy circles. So taking Levitt as the “most-likely” case of an academic producing research that affects policy, we might infer then that his lack of success is generalizable to all academics.

So why is non-influence depressing? For one, I think many social scientists, particularly political scientists, are driven by a (perhaps naive) desire that their research can make the world a better place. And even if you aren’t, basically every paper in political science, no matter how obscure the subject matter or how technical the methodology, ends with a series of implications for public policy. This is for good reason, because it’s hard to be interested in a paper if these implications are lacking. But if no one is ever going to take these implications seriously, then what is the point of them?

There is another interpretation of Levitt’s comment, but it doesn’t seem any less pessimistic to me. Note Levitt doesn’t say no one is listening to him, but that those who do “had the agenda first, and then they went looking for research — any research — that would support their position.” Which brings us to Reinhart and Rogoff (some background, and a terrific replication, here). These famous Harvard economists published a paper in the top economics journal in 2010 arguing that (1) there was a relationship between national debt and economic growth such that high debt led to slower growth, and (2) that there was a certain threshold of debt above which things got really bad for countries in terms of their growth. Then recently, three UMass Amherst economists tried to replicate the analysis and found they couldn’t, finally realizing that the original paper’s conclusions were driven by technical errors (more on that in another post).

Sociologist Peter Frase finds the ensuing discussion in the “economics blogosphere” to be missing the point: the problem is not the technical errors per se, he seems to be saying, but something similar to what Levitt pointed out: the function of research to satisfy pre-existing political arguments:

The reaction of the left-wing peanut gallery (at least to judge by my Twitter feed) has been to ridicule liberals for caring about this at all. Obsessing over the analytical missteps in this paper reeks of the preoccupation with having correct and empirically supported arguments, while ignoring the importance of power and ideology. For while this new critique of Reinhart-Rogoff just now became possible because they finally made their original data available, plenty of people pointed out earlier that the whole analysis rested on shaky conceptual foundations. It used a correlation to assert that high debt to GDP ratios lead to slower growth, ignoring the much more plausible theory that the causal order was the opposite, with slow growth leading to increasing debt loads. If the political elite in Washington failed to heed these criticisms, it wasn’t because they were unaware of them, but because the claim that debt leads to slow growth fit a deficit hysteria that was already entrenched. In other words, Reinhart-Rogoff was being used as rhetorical cover for a pre-existing position, not as an actual empirical aid to decision-making.

Frase’s post is focused on railing against “wonks” — policy-focused journalists / bloggers who specialize in translating (or in his view, sometimes simply parroting) policy-relevant social science research to readers. But his general point about “power and ideology” is interesting. I think what he’s saying is that the poor quality of the underlying research was discounted because it served an ideological purpose. Presumably, high-quality research that made the opposite argument would be ignored, under Frase’s implicit model of policy advice.

Of course, there could be other possible avenues of “making a difference.” One is communicating research results to organizations outside of government. Unfortunately, the type of confirmation biases Levitt and Frase discuss also appear to operate there as well, as another Freakonomics post recently described.

Note: Many of the links in this article were found via this Monkey Cage post by John Sides.

I still Rick Hasen’s Election Law Blog, however, which is always chalk full of interesting policy and causal questions. In the wake of the 2012 election, and the president’s apparently off-handed comment about long lines at the polling places, there is apparently some very tiny rumblings about nationalizing election administration in the U.S. For example, Hasen himself argued in the NYT’s “Room for Debate” forum that elections should be nationalized. In response, Doug Chapin gave some reasons why not, citing both the (apparently normative) virtues of federalism and the perceived gridlock and incompetence of the feds right now.

Slightly more interesting is why or why not reforming election administration is “a thing” or not, in Chapin’s language. That is, why is it so remote from the policy agenda? It’s easy to say that it’s a non-starter and that state and local governments really don’t want to centralize, but that seems like begging the question. Indeed it’s even more mysterious when (to crib another item from Hasen’s blog) a poll finds 88% of Americans supporting a uniform system!

Perhaps it’s not hard to dig up other items where the public seems to speak with such a loud voice, and in contrast to what policy is or what elites think it should be. But the fact is that usually we laud correlations between mass opinion and public policy as a good thing, and decry low or negative correlations as a bad thing, for democracy. So it would seem inconsistent to just write this off as an anomaly.