Tag Archives: transparency

Supreme Court makes democracy (and social science research) harder

Last week, the U.S. Supreme Court unanimously ruled that the state of Virginia can refuse to entertain Freedom of Information Act requests from persons residing outside of Virginia. A New York Times article is here. I heard about the decision just by chance from turning on NPR during On the Media this weekend. But I knew the case was coming, and had paid attention to the arguments when they were heard in February.

The reason is that, as part of my dissertation research, I’m studying local elections in Virginia, and have been requesting that county election offices provide me with election results dating back to the early 1990s. A few counties outright refused to help me, citing the state law that says they don’t have to. The majority just did not respond. The state election office even decided to get in touch with me, apparently after some localities contacted them about my request, reminding me that their offices are under no obligation to help me. (Naturally, I tried to tailor my requests to be as polite as possible, and made clear that I was willing to reimburse local offices for any fees associated with my request.)

Of course I knew about this clause of the state law going into my request — probably I learned about it from the Virginia Coalition for Open Government web site, which was involved with the case and was on the side of the plaintiffs who argued the residency restriction was unconstitutional. All is not lost for me, anyway — I have since been in contact with a researcher in Virginia and we’re planning on completing the requests soon.

I have two comments about the On the Media report about the decision. First, I found it really strange that Justice Alito, writing for the Court, seems to believe that “Requiring noncitizens to conduct a few minutes’ of Internet research in lieu of using a relatively cumbersome state FOIA process cannot be said to impose any significant burden.” Assuming this is not horribly taken out of context, it’s just wrong — if government information was readily available on the Internet, then no one would be doing FOIA requests in the first place. Indeed, in my experience, the results of FOIA requests are often paper records — if you’re lucky, the person processing the request will scan them for you. So they often aren’t even digitized, let alone on a web site.

My second comment relates to a point raised by Mark Caramanica of Reporters Committee for Freedom of the Press. He points out that these type of residency requirements threaten evaluations of government policy — how well are different states doing on certain benchmarks:

For example, USA Today did a story where they compared how the No Child Left Behind initiative was impacting teacher behavior in a variety of jurisdictions and pulled records dealing whether teachers were disciplined for potentially coaching students and, you know, trying to cut corners on the standardized tests, and so forth, to boost up their scores. That was a case where, you know, you need access to Virginia records, obviously, to put Virginia in the mix.

ProPublica did a story where they investigated dialysis centers all across the country, their effectiveness and safety records, and so forth. To get that national perspective, you need Virginia. And, you know, it leaves a hole in the entire story and the complete picture, but it also shortchanges citizens of Virginia because they don’t get the benefit of knowing where their state stacks up on certain issues.

(Of course, this type of cross-state comparison is more commonly done by social scientists, including political scientists who study states but also, maybe even more often, by economists. While a large number of press organizations, including the New York Times, filed an amicus brief on the side of the plaintiffs in this case, did any social science organizations? Apparently not. SCOTUSblog has all the amica briefs for the case, and I don’t see any.)

Maybe it’s not surprising that the most insular branch, not especially friendly to sharing information in its own domain, decided this way. On the other hand, a famous Justice once spoke eloquently of the potential of states as generators of good public policy. With this decision, the Court has made this potential just a little bit harder to realize.

I want to be careful not to exaggerate. For one, already, workarounds are apparently in the works. For another, VA is apparently one of 8 states with such a clause. But I’ve found some states don’t specify residency — as a researcher, I worry that the next time I make a request to a state with an ambiguous law, they might decide to refuse me on residency grounds.

Is transparency of methods increasing in quality of research? 2 Education Studies

Two reports about education studies caught my eye in recent days, but more for what they have in common aside from education itself.

The first comes from the free daily Metro newspaper that subway riders in Boston read. Here is a link to the article, but in the past I’ve noticed that their URLs don’t stick around for too long.

The first study is reported to show that college students who live in dorms perform better academically than those who live off-campus. Worried about the selection bias problem, wherein the types of people who choose to live on or campus might just be more or less committed to their studies? Not to worry!

Plenty of studies have shown that students who live on campus tend to do slightly better academically. But a recent study out of Indiana University-Purdue University Indianapolis found that not only did on-campus students do significantly better academically, but that living on campus was a big reason why they did better.

The big question is causation. Does living in the dorms cause you to do better? Previous studies were just looking at those that live in the dorms versus those that don’t,” explains Dr. James Murray, who co-authored the study with Dr. Pedro De Araujo. “The problem is that students that care more about school may be more likely to want to live at school. So, they may perform better, because they happen to be the better students anyway. But we actually found that living on campus caused students to do better, at least in terms of GPA.”

The researchers also found that campus life had a positive effect even after the student moved out of the dorms. “Students who once lived on campus were more likely to study with their peers after they moved off campus, but not necessarily while they lived on campus,” explains Murray. “So, if you’re a freshman living on campus, you’re not more likely to study with your classmates, but you formed those relationships in the early years and were able to make use of them later.”

That’s the entire article (so sue me). It’s worth posting the whole thing to see that while the problem of selection bias and the claim of showing causation are both mentioned, exactly how causation is shown is completely unspecified.

(Now my usual disclaimer is that we should blame the journalist for the omission. However, I looked up the paper and, to put it tactfully, think that the authors could have done a better job explaining their research design. So it might not be a surprise that the journalist couldn’t figure it out. Also, the researchers are not from the Indiana University-Purdue University Indianapolis, as the first paragraph of the news article indicates; rather, the data for the study are from students at that university. Pretty sure this mistake is the journalist’s fault.)

Anyway, it boggles the mind that you can get away with stuff like this, even in the Metro.

Second study is from a NY Times blog post on education, and has a similar format: big question; possible selection bias problem; a vaguely specified fix.

Each year millions of middle-school students nationwide spend angst-filled months waiting to hear if they scored high enough on an entrance exam to attend a selective public high school. In New York City alone more than 27,000 students apply for precious spots in the three best-known schools: Stuyvesant, Brooklyn Technical and Bronx High School of Science. What Mr. Dobbie and Mr. Fryer wanted to know was just how much of a difference attending one of these high schools makes in the long run for students with similar equal admissions test scores. Some proponents say the benefits of an environment of high achievers, more advanced coursework and higher expectations are obvious. Skeptics counter that these gifted and motivated teenagers would have done well no matter where they went. Students in these schools with low class ranks may even be less competitive when it comes to college admissions. They might be better off in less competitive environments or in schools with a wider range of student abilities. Because there are so many applicants of similar ability and achievement who are not accepted to these New York institutions, the authors were able to compare the average outcomes of students who graduated from different high schools. “We provide the first causal estimate of the impact of attending an exam-high school in the United States on later outcomes,” they write.

(Also, what is this post doing on an Arts blog at the Times? [Shouldn't it be on the Fashion pages with the other research studies?])

Actually on second read that doesn’t sound so bad–much better than the first study. The abstract of the article is also nicely done. Basically what they are doing is comparing outcomes for admitted students who attended and admitted students who did not attend, adjusting for observable characteristics like family background and pre-treatment academic performance. Sounds good. But don’t we think that the choice to attend, even in this subsample, may be “endogenous”–that is, that those who choose to attend might just be better/worse academically than those who choose not to?

Update 8/19: I actually misread the blog post and I made a mistake on the link to the second study paper, and consequently misinterpret their design. Here is the actual link and here is the abstract, which makes it quite clear the authors are using regression discontinuity.

Publicly funded exam schools educate many of the world’s most talented students. These schools typically contain higher achieving peers, more rigorous instruction, and additional resources compared to regular public schools. This paper uses a sharp discontinuity in the admissions process at three prominent exam schools in New York City to provide the first causal estimate of the impact of attending an exam school in the United States on longer term academic outcomes. Attending an exam school increases the rigor of high school courses taken and the probability that a student graduates with an advanced high school degree. Surprisingly, however, attending an exam school has little impact on Scholastic Aptitude Test scores, college enrollment, or college graduation — casting doubt on their ultimate long term impact.

Potpourri

(With apologies to the Monkey Cage.)

  • The dark side of transparency laws?
  • Should parents of extremely obese children lose custody for not controlling their kids’ weight? (Who knew the Journal of the American Medical Association printed op/eds? [Some might say that's all they print. ;-) ])
  • Can Tax Credits Create Jobs? (Via Tax.com)
  • More fallout / back-and-forth in the wake of the Diane Ravitch op/ed piece I wrote about in this post. (Best line: “Saying you want a good teacher in every classroom and a well-rounded, rigorous curriculum is as trite as saying you’re for motherhood and apple pie. What would Ms. Ravitch say to John White and Cami Anderson, who just took over two of the toughest school systems in America, in New Orleans and Newark? What would be the top three to five things Ms. Ravitch would have them do in their first year?”)