Two reports about education studies caught my eye in recent days, but more for what they have in common aside from education itself.
The first comes from the free daily Metro newspaper that subway riders in Boston read. Here is a link to the article, but in the past I’ve noticed that their URLs don’t stick around for too long.
The first study is reported to show that college students who live in dorms perform better academically than those who live off-campus. Worried about the selection bias problem, wherein the types of people who choose to live on or campus might just be more or less committed to their studies? Not to worry!
Plenty of studies have shown that students who live on campus tend to do slightly better academically. But a recent study out of Indiana University-Purdue University Indianapolis found that not only did on-campus students do significantly better academically, but that living on campus was a big reason why they did better.
“The big question is causation. Does living in the dorms cause you to do better? Previous studies were just looking at those that live in the dorms versus those that don’t,” explains Dr. James Murray, who co-authored the study with Dr. Pedro De Araujo. “The problem is that students that care more about school may be more likely to want to live at school. So, they may perform better, because they happen to be the better students anyway. But we actually found that living on campus caused students to do better, at least in terms of GPA.”
The researchers also found that campus life had a positive effect even after the student moved out of the dorms. “Students who once lived on campus were more likely to study with their peers after they moved off campus, but not necessarily while they lived on campus,” explains Murray. “So, if you’re a freshman living on campus, you’re not more likely to study with your classmates, but you formed those relationships in the early years and were able to make use of them later.”
That’s the entire article (so sue me). It’s worth posting the whole thing to see that while the problem of selection bias and the claim of showing causation are both mentioned, exactly how causation is shown is completely unspecified.
(Now my usual disclaimer is that we should blame the journalist for the omission. However, I looked up the paper and, to put it tactfully, think that the authors could have done a better job explaining their research design. So it might not be a surprise that the journalist couldn’t figure it out. Also, the researchers are not from the Indiana University-Purdue University Indianapolis, as the first paragraph of the news article indicates; rather, the data for the study are from students at that university. Pretty sure this mistake is the journalist’s fault.)
Anyway, it boggles the mind that you can get away with stuff like this, even in the Metro.
Second study is from a NY Times blog post on education, and has a similar format: big question; possible selection bias problem; a vaguely specified fix.
Each year millions of middle-school students nationwide spend angst-filled months waiting to hear if they scored high enough on an entrance exam to attend a selective public high school. In New York City alone more than 27,000 students apply for precious spots in the three best-known schools: Stuyvesant, Brooklyn Technical and Bronx High School of Science. What Mr. Dobbie and Mr. Fryer wanted to know was just how much of a difference attending one of these high schools makes in the long run for students with similar equal admissions test scores. Some proponents say the benefits of an environment of high achievers, more advanced coursework and higher expectations are obvious. Skeptics counter that these gifted and motivated teenagers would have done well no matter where they went. Students in these schools with low class ranks may even be less competitive when it comes to college admissions. They might be better off in less competitive environments or in schools with a wider range of student abilities. Because there are so many applicants of similar ability and achievement who are not accepted to these New York institutions, the authors were able to compare the average outcomes of students who graduated from different high schools. “We provide the first causal estimate of the impact of attending an exam-high school in the United States on later outcomes,” they write.
(Also, what is this post doing on an Arts blog at the Times? [Shouldn't it be on the Fashion pages with the other research studies?])
Actually on second read that doesn’t sound so bad–much better than the first study. The abstract of the article is also nicely done. Basically what they are doing is comparing outcomes for admitted students who attended and admitted students who did not attend, adjusting for observable characteristics like family background and pre-treatment academic performance. Sounds good. But don’t we think that the choice to attend, even in this subsample, may be “endogenous”–that is, that those who choose to attend might just be better/worse academically than those who choose not to?
Update 8/19: I actually misread the blog post and I made a mistake on the link to the second study paper, and consequently misinterpret their design. Here is the actual link and here is the abstract, which makes it quite clear the authors are using regression discontinuity.
Publicly funded exam schools educate many of the world’s most talented students. These schools typically contain higher achieving peers, more rigorous instruction, and additional resources compared to regular public schools. This paper uses a sharp discontinuity in the admissions process at three prominent exam schools in New York City to provide the first causal estimate of the impact of attending an exam school in the United States on longer term academic outcomes. Attending an exam school increases the rigor of high school courses taken and the probability that a student graduates with an advanced high school degree. Surprisingly, however, attending an exam school has little impact on Scholastic Aptitude Test scores, college enrollment, or college graduation — casting doubt on their ultimate long term impact.