Recently, Brookings Institution revealed the results of the study that looked at the impact of vouchers on college enrollment rates. While the results of the study showed no overall increase in the rates of college enrollment, among African-American students the impact was quite pronounced — so pronounced that the authors felt confident enough to conclude that participation in the voucher program in question made African-American students more likely to enroll in an institution of higher learning.

Now, Sara Goldrick-Rab from University of Wisconsin-Madison, after looking at the data behind the Brookings report, is questioning the authors’ conclusions. Goldrick-Rab’s review was underwritten by the Think Twice think tank review project and was produced with help from the National Education Policy Center and funded by a grant from the Great Lakes Center for Education Research and Practice.

In her review, Goldrick-Rab questions the claim of a strong positive impact for a single subgroup of students. She observes that the study identifies no overall impacts of the voucher offer, but that the authors “report and emphasize large positive impacts for African American students, including increases in college attendance, full-time enrollment, and attendance at private, selective institutions of higher education.” This strong focus is not warranted.

Although the authors of the Brookings report, Matthew Chingos and Paul Peterson, admit that participating in the voucher program didn’t seem to have an impact on college enrollment, their focus on the African-American subgroup indicates that a statistically significant impact was detected there. According to Goldrick-Rab, that conclusion was unwarranted. She writes that the data didn’t show significant differences between the college enrollment rates of African-Americans and other students. Goldrick-Rab identified three further problems:

The authors fail to demonstrate any estimated negative effects that could help explain the average null results; and

There are previously existing differences between the African American treatment and control groups on factors known to matter for college attendance (e.g., parental education).
There is important but unmentioned measurement error in the dependent variables (college attendance outcomes) affecting the precision of those estimates and likely moving at least some of them out of the realm of statistical significance;

According to the report produced by Goldrick-Rab, the evidence seems to indicate that the voucher program studied in the Brookings report didn’t have an impact on college enrollment rates of all participants, of African-Americans or of any other sub-group.

There is important but unmentioned measurement error in the dependent variables (college attendance outcomes) affecting the precision of those estimates and likely moving at least some of them out of the realm of statistical significance;