Right-Wing Voucher Study – is sound research relevant to conservatives?

Public funding of voucher programs is a front line debate on education reform, even in the Presidential arena. The Romney plan for K-12  public education would replace the current system with a $25 billion voucher program for use at any school – public, private, charter, or online.

Aside from accountability issues, one of the key questions surrounding vouchers was addressed by the conservative Brookings  Institution in a study released August 2012, “The Effects of School Vouchers on College Enrollment.”

Does a publically funded,  voucher choice program increase college enrollment in low-income, urban populations?

The Brookings study claims to identify a positive strategy that works – providing students with vouchers increases college enrollment of African-American students by 24%. A review of the Brookings study was published late last night by the National Education Policy Center (NEPC) at the University of Colorado, and written by Sara Goldrick-Rab at UW-Madison. Goldrick-Rab is a leading researcher on education reform. Badger Democracy received an advance copy of the review by Goldrick-Rab; and the analysis indicates a clear bias in the Brookings study – putting the desired conclusion in support of vouchers above quality research.

One key finding was emphasized in the Brookings study. Based on their analysis of the data from the 1997 New York City intervention study; vouchers had a “large and substantively important” impact on African-American college enrollment. The positive impact increased enrollment rates by 8.7%. The study found no significant benefit to the Hispanic student subgroup – the other student subgroup of focus.

The conclusion from the Brookings study drawn by the authors:

Effects of vouchers on African-American attendance are unusually large…vouchers have a greater impact than exposure to a good teacher…the next step for President Obama (is to) open private school doors for low-income students.

Goldrick-Rab’s review of this study raises serious concerns regarding research and method, from a highly regarded academician. Is it possible the conclusion above was foregone – the “research” conforming to the desired result? After all, recent Brookings papers on voucher policy are heavily supportive, being cited by conservatives nationwide. Here are key critiques from Goldrick-Rab:

1. The Brookings study assumes that “prior evidence” suggest heterogeneous impacts from school vouchers. The only “prior evidence” cited is that of the author. The Goldrick-Rab review points out one example of a significant study that disputes that claim (Krueger, Alan and Pei Zhu. 2004. “Another Look at the New York City School Voucher Experiment,” American Behavioral Scientist, Vol. 47, No. 5: 658-698). “The report leaves a less-informed reader without important knowledge of that debate. For example, discussion of a key issue—how students are classified by their race/ethnicity for analysis…” When significant research exists that disputes a foregone assumption, it is the author’s obligation to address the disputed data.

2. The Brookings study authors failed to test for differences in the two subgroups (African-American, Hispanic). “When testing for heterogeneous impacts, researchers should seek to rule out the possibility that there are no subgroup differences.”  From a statistical methodology:

“Findings for a specific subgroup should not be highlighted unless they differ statistically significantly from those for other sample members. If subgroup differences are not statistically significant, findings for the full study sample usually should be emphasized instead of those for the subgroup.”Bloom, H. & Michaelopoulos, C. 2011. “When is the Story in the Subgroups? : Strategies for Interpreting and Reporting Intervention Effects for Subgroups.” Prevention Science.  (see pg 5)

This is a critical “miss” in this study. If there is no difference between the subgroups, how can the effect be so pronounced in one, and not the other – unless other forces are at work? In 2000, Mathematica Policy Research issued a similar warning: “Because gains are so concentrated in this single group, one needs to be very cautious.” 

3. The Brookings study overstates the accuracy of the data from the original study in 1997. Important in a study of over 2,600 students. From the review:

…the NSC (National Student Clearinghouse) did not use SSNs (Social Security Numbers) to make the match—the matches were made based solely on name and date of birth, a process that may be more fallible for students with more complexity to their names, for students with very common names, and/or for student groups more likely to have missing data (e.g., racial/ethnic minority students). Thus, the report oversells the degree to which using the NSC solves the attrition problem confronted in prior studies of the program.

Goldrick-Rab’s review points out that “…if we inflate the standard errors even slightly, the estimates will be rendered statistically non-significant. Once again, this implies there is likely no story in the subgroups.”

4. Results for children who are non-African American or Hispanic are excluded from the Brookings study. In a prior study conducted by one of the authors (Peterson), accurate racial classification in the study presented a problem. This is not addressed in the recent Brookings study:

In a prior study of the same program by Peterson and his colleagues, a re-analysis found that estimated positive impacts for African-Americans were rendered null when children with an African-American non-Hispanic father were classified as African-Americans along with those born to an African-American mother. Might that be the case in this study as well? (Howell, W.G. and Peterson, P.E. 2004. “Uses of Theory in Randomized Field Trials: Lessons From School Voucher Research on Disaggregation, Missing Data, and the Generalization of Findings,” American Behavioral Scientist, Vol. 47, No. 5: 634-657.)

5. The most significant issue with the Brookings study, is that it downplays the significance of parental education attainment in the African-American subgroup. Goldrick-Rab addresses the problem:

On page 10, readers are assured that the treatment and control groups are similar on average, and that statistical testing indicates no problems for the two subgroups shown (African-Americans and Hispanics).

The authors, however, fail to statistically address this in their analysis, lumping the educational attainment into a catch-all “all controls” category.

Since parental education has well-established explanatory power for the dependent variable—college enrollment—this difference is critical. The treatment group of African-Americans has a higher rate of parental bachelor’s degree completion than the control group

The study itself is testing college enrollment impact of vouchers. The fact that parental degree attainment is higher in the treatment group, than in the control group , means that would be a more significant variable that “number of children in household.” That variable alone could be a large part of that 8.7% increase.

…readers cannot know whether any positive college enrollment differences for African-Americans are due to vouchers or due to pre-existing differences among students—such as whether their parents completed a bachelor’s degree.

Badger Democracy emailed Matthew Chingos at the Brookings Institute for a response to the Goldrick-Rab review. The link can be found here. Chingos’ response to the review does not address any of the analysis concerns, nor does it provide further supporting data. The response is largely anecdotal, and in essence, says “we know what we are doing…trust us.”

Goldrick-Rab states the policy analysis value of the Brookings study:
Policymakers and practitioners interested in the effectiveness of school voucher programs should indeed attend to the results of this study, which—contrary to the interpretation of the authors—convincingly demonstrates that in New York City a private voucher program failed to increase the college enrollment rates of students from low-income families.
The focus of the Brookings study ought to be that there is no significant difference in the college enrollment of students given vouchers. Unfortunately, as has been evident over the past two decades, conservatives have become experts at using “research” to arrive at pre-determined conclusions. These conclusions are always politically (and in many cases fiscally) beneficial to a single, conservative corporate ideology. And damaging to a real discussion on educational policy. 
Help keep independent, progressive journalism alive!




4 thoughts on “Right-Wing Voucher Study – is sound research relevant to conservatives?

  1. What is well established in the literature is that voucher schools, almost universally, weed out difficult students. I wonder if this is taken in to consideration in the data? This is similar to the likelihood that voucher students have parents with a college degree. Moreover, not all voucher schools (at least in Wisconsin) are alike. For example, some of the traditional Catholic schools take a relatively small percentage of “voucher” students compared to some schools that are basically completely voucher students. I would hope that the students in traditional Catholic and Lutheran schools are not lumped in.

  2. “School Vouchers” is a program created to justify using tax dollars to pay for religious classes with tax dollars which is directly blocked by the Constitution.

    They should call it “indirectly funding school prayer with tax dollars” and be truthful for once about their motives to cherry pick the Constitution. Run by the evidence burning defrocked politician Scott Jensen whose stench permeates this movement.

    Motive? Stripping tax dollars out of the education process.

    Not to provide a better education for anyone – except the taxpayer who better be noticing by now. Because studies indicate public schools are the most effective education process and only by de-funding and quoting stacked “studies” and other manufactured statistics can they promote their agenda through a slew of paid bloggers. These tax siphoners have overplayed their hand and taxpayers and voters must shut them down.

    Reject Republicans, Reduce Corporate Influence and Arrest Temporary Governor Scott Walker !

  3. I want to make it very clear that in the city of Milwaukee and it is IN Walker’s “budget” that 30% of taxpayer money e directlt channeled FROM the Milwaukee public schools to voucher and charter schools. While PUBLIC schools and teachers are held to strict standards and toi teach in them requires a college degree, an educator in a voucher school does NOT, nor do they have to be certified to teach childen with learning disabilities such as mild-autism, special needs children who have been mainstreamed into the public schools as well. These institutions are NOT required by state and local governments to have their books audited like public schools are either so while they’re getting taxpayer money, they don’t have to be held accountable for what they are actually DOING with it since they’re protected under a loophole as “private”. I have all the lovely new p/w on how our WI public system and our teaachers, though ROBBED of 1.6 BILLION dollars last year is now expected to perform better and will have a REPORT CARD on their performance and guess who is NOT required to follow all the rules and expectations of our reich wing governor whose only higher education was his G.E.D. because he was a high school drop out and never got a diploma so finished with a G.E.D. and didn’t finish college either. All of this is the reich wing path to completly privatize America so everything from healthcare to getting an education will ultimately BE A PRIVILEGE because the only ones who eventually will be abe to afford them will be the RICH.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s