Public funding of voucher programs is a front line debate on education reform, even in the Presidential arena. The Romney plan for K-12  public education would replace the current system with a $25 billion voucher program for use at any school – public, private, charter, or online.

Aside from accountability issues, one of the key questions surrounding vouchers was addressed by the conservative Brookings  Institution in a study released August 2012, “The Effects of School Vouchers on College Enrollment.”

Does a publically funded,  voucher choice program increase college enrollment in low-income, urban populations?

The Brookings study claims to identify a positive strategy that works – providing students with vouchers increases college enrollment of African-American students by 24%. A review of the Brookings study was published late last night by the National Education Policy Center (NEPC) at the University of Colorado, and written by Sara Goldrick-Rab at UW-Madison. Goldrick-Rab is a leading researcher on education reform. Badger Democracy received an advance copy of the review by Goldrick-Rab; and the analysis indicates a clear bias in the Brookings study – putting the desired conclusion in support of vouchers above quality research.

One key finding was emphasized in the Brookings study. Based on their analysis of the data from the 1997 New York City intervention study; vouchers had a “large and substantively important” impact on African-American college enrollment. The positive impact increased enrollment rates by 8.7%. The study found no significant benefit to the Hispanic student subgroup – the other student subgroup of focus.

The conclusion from the Brookings study drawn by the authors:

Effects of vouchers on African-American attendance are unusually large…vouchers have a greater impact than exposure to a good teacher…the next step for President Obama (is to) open private school doors for low-income students.

Goldrick-Rab’s review of this study raises serious concerns regarding research and method, from a highly regarded academician. Is it possible the conclusion above was foregone – the “research” conforming to the desired result? After all, recent Brookings papers on voucher policy are heavily supportive, being cited by conservatives nationwide. Here are key critiques from Goldrick-Rab:

1. The Brookings study assumes that “prior evidence” suggest heterogeneous impacts from school vouchers. The only “prior evidence” cited is that of the author. The Goldrick-Rab review points out one example of a significant study that disputes that claim (Krueger, Alan and Pei Zhu. 2004. “Another Look at the New York City School Voucher Experiment,” American Behavioral Scientist, Vol. 47, No. 5: 658-698). “The report leaves a less-informed reader without important knowledge of that debate. For example, discussion of a key issue—how students are classified by their race/ethnicity for analysis…” When significant research exists that disputes a foregone assumption, it is the author’s obligation to address the disputed data.

2. The Brookings study authors failed to test for differences in the two subgroups (African-American, Hispanic). “When testing for heterogeneous impacts, researchers should seek to rule out the possibility that there are no subgroup differences.”  From a statistical methodology:

“Findings for a specific subgroup should not be highlighted unless they differ statistically significantly from those for other sample members. If subgroup differences are not statistically significant, findings for the full study sample usually should be emphasized instead of those for the subgroup.”Bloom, H. & Michaelopoulos, C. 2011. “When is the Story in the Subgroups? : Strategies for Interpreting and Reporting Intervention Effects for Subgroups.” Prevention Science.  (see pg 5)

This is a critical “miss” in this study. If there is no difference between the subgroups, how can the effect be so pronounced in one, and not the other – unless other forces are at work? In 2000, Mathematica Policy Research issued a similar warning: “Because gains are so concentrated in this single group, one needs to be very cautious.” 

3. The Brookings study overstates the accuracy of the data from the original study in 1997. Important in a study of over 2,600 students. From the review:

…the NSC (National Student Clearinghouse) did not use SSNs (Social Security Numbers) to make the match—the matches were made based solely on name and date of birth, a process that may be more fallible for students with more complexity to their names, for students with very common names, and/or for student groups more likely to have missing data (e.g., racial/ethnic minority students). Thus, the report oversells the degree to which using the NSC solves the attrition problem confronted in prior studies of the program.

Goldrick-Rab’s review points out that “…if we inflate the standard errors even slightly, the estimates will be rendered statistically non-significant. Once again, this implies there is likely no story in the subgroups.”

4. Results for children who are non-African American or Hispanic are excluded from the Brookings study. In a prior study conducted by one of the authors (Peterson), accurate racial classification in the study presented a problem. This is not addressed in the recent Brookings study:

In a prior study of the same program by Peterson and his colleagues, a re-analysis found that estimated positive impacts for African-Americans were rendered null when children with an African-American non-Hispanic father were classified as African-Americans along with those born to an African-American mother. Might that be the case in this study as well? (Howell, W.G. and Peterson, P.E. 2004. “Uses of Theory in Randomized Field Trials: Lessons From School Voucher Research on Disaggregation, Missing Data, and the Generalization of Findings,” American Behavioral Scientist, Vol. 47, No. 5: 634-657.)

5. The most significant issue with the Brookings study, is that it downplays the significance of parental education attainment in the African-American subgroup. Goldrick-Rab addresses the problem:

On page 10, readers are assured that the treatment and control groups are similar on average, and that statistical testing indicates no problems for the two subgroups shown (African-Americans and Hispanics).

The authors, however, fail to statistically address this in their analysis, lumping the educational attainment into a catch-all “all controls” category.

Since parental education has well-established explanatory power for the dependent variable—college enrollment—this difference is critical. The treatment group of African-Americans has a higher rate of parental bachelor’s degree completion than the control group

The study itself is testing college enrollment impact of vouchers. The fact that parental degree attainment is higher in the treatment group, than in the control group , means that would be a more significant variable that “number of children in household.” That variable alone could be a large part of that 8.7% increase.

…readers cannot know whether any positive college enrollment differences for African-Americans are due to vouchers or due to pre-existing differences among students—such as whether their parents completed a bachelor’s degree.

Badger Democracy emailed Matthew Chingos at the Brookings Institute for a response to the Goldrick-Rab review. The link can be found here. Chingos’ response to the review does not address any of the analysis concerns, nor does it provide further supporting data. The response is largely anecdotal, and in essence, says “we know what we are doing…trust us.”

Policymakers and practitioners interested in the effectiveness of school voucher programs should indeed attend to the results of this study, which—contrary to the interpretation of the authors—convincingly demonstrates that in New York City a private voucher program failed to increase the college enrollment rates of students from low-income families.