There have been state-commissioned evaluations of both the Milwaukee and Cleveland voucher programs. In Milwaukee, there were no appreciable academic gains at all. In Cleveland gains were found in one subject, science, for voucher students attending well-established private schools, with declines in test scores for students attending newly-opened private schools. Both sets of findings have been challenged by Harvard's Paul Peterson, whose methodology and objectivity has been criticized by many in the education research community. In the case of Milwaukee, a third researcher, Princeton economist Cecilia Rouse, has evaluated both the Peterson and Wisconsin studies, finding mixed results. Overall, there is no clear evidence that publicly funded vouchers have improved educational performance of voucher students.
Milwaukee Parental Choice Program
- In Milwaukee, the state commissioned the most thorough research yet conducted on vouchers. This research, conducted by a University of Wisconsin - Madison team led by John Witte over a five year period, found no appreciable academic gains in reading and math from vouchers.3 Attrition rates were high: 44% and 32% in the first two years.4 Reasons for leaving the program included application and fee problems, transportation, and the limitation on religious instruction (before the state Supreme Court allowed the expansion of the program in 1998).5 Among parents whose children remained in the program, satisfaction was high.6 Parents of children who remained in the voucher program had higher educational levels-a key determinant of a student's academic achievement-than those who did not.7
- Using Witte's data, Paul Peterson's team employed different assumptions and statistical techniques and claimed that there was a statistically significant gain for voucher students in the third and fourth years of the program.8 This finding was disputed by many in the research community, who argue that by the third year the control and experimental groups were not comparable. The 30% annual attrition rate-primarily students doing poorly in the voucher program-ensured that those who remained were an academically superior subset, not a random sample.9 Peterson's methodology has also been criticized on other points, including his reliance in some cases on tiny samples - in some cases as low as 26 students - and for dismissing group differences between voucher school students and the control group (non-choice public school students).10 Witte described Peterson's reanalysis of the Milwaukee data as a "confusing, tortured effort to try to find any evidence that students enrolled in private schools … do better than any students in the Milwaukee Public Schools."11 Peterson's unconventional reporting of statistical significance tests has drawn fire not only from Professor Alex Molnar, then at the University of Wisconsin-Milwaukee, who described Peterson's term "substantially significant" as an "important-sounding characterization with no precise research meaning," but also from the pro-voucher Wall Street Journal, which wrote that he had been "loose with his claims."12 Some have pointed to contradictions between Peterson's claims of academic achievement and his own statistical data,13 as well as a lack of adequate controls to take prior academic achievement of students into account. Other criticisms include a failure to adequately address variables that affect a child's success in school, including parental involvement and expectations, parental employment, marital status, family size and receipt of Aid to Families with Dependent Children.14
- A third analysis of the same data was conducted by Cecilia Rouse of Princeton University. Rouse found that the voucher students had made some small gains in math, and that "the [voucher] effects on the reading scores are as often negative as positive and are nearly always statistically indistinguishable from zero."15 The findings of a positive voucher effect for math were only for the subgroup of students who were in the voucher program over a four-year period, which again does not account for the significant number of dropouts from the voucher program. Moreover, Rouse showed that Milwaukee public schools serving low-income populations that have small class sizes and additional state funding keep pace with voucher schools in math gains and substantially outpace them in reading.16 Rouse does not assert that this proves that small classes are the causal factor but calls for more investigation; however, Alex Molnar and Charles Achilles of Eastern Michigan University do find that reducing class size is more effective than a voucher policy in helping at-risk students, as does Princeton economists Alan Krueger and Diane Whitmore.17 Referring to a different study in which Peterson found large improvements in test scores for African American students and claimed these gains outpace those observed in the Tennessee class size reduction experiment called STAR, Molnar and Achilles fault Peterson for making an apples-to-oranges comparison. They found that when the apples-to-apples comparison is made between similar students, "For those [minority] students, the STAR effects were approximately double the total effects [of vouchers]."18
- In 1995, Wisconsin lawmakers who support vouchers responded, not by changing the voucher program, but by eliminating any further state-sponsored research into the educational results of vouchers. The most recent Wisconsin state audit of the voucher program found that "some hopes for the program-most notably, that it would increase participating pupils' academic achievement-cannot be documented, largely because uniform testing is not required in participating schools."19 Wisconsin taxpayers thus have absolutely no current information on whether vouchers are having any positive effects on education.
Cleveland Scholarship and Tutoring Grant Program
- The state-sponsored study of the Cleveland voucher program, conducted by an Indiana University team led by Kim Metcalf, found that after two years there was no improvement in the overall test scores of those students using vouchers in established private schools. On a subject-by-subject basis, there were gains for voucher students only in science.20 Overall test scores and the other four subject scores revealed no differences between voucher and public school students. There were also students who used vouchers at the new HOPE Academy schools, created by David Brennan, an entrepreneur who helped push the legislature into adopting vouchers to receive voucher money and students. These students scored below their peers in both public schools and the established private schools in all five subjects tested.21
- Responding to Metcalf's first year report, Paul Peterson and his research team claimed that findings from his 1997 evaluation of the two HOPE Academy voucher schools showed significant academic gains.22 He was hired to evaluate these schools by their founder David Brennan, a prominent voucher advocate.23 Peterson criticizes the Indiana University study primarily for failing to include the HOPE Academy scores and for using second grade test scores taken prior to entry in voucher schools as a basis for comparison with third grade voucher scores.24
- Metcalf responded with a strong article entitled "Advocacy in the Guise of Science." In it, he suggested that the Peterson researchers "are strong supporters of vouchers and have done much to promote the implementation of voucher programs throughout the country. So it is possible that they are engaged in a deliberate effort to misrepresent the Cleveland data in order to influence educational policy."25 Specifically, he responded that he did include the HOPE scores but put them into a different section because those students took a different test. With regard to the use of second grade tests, he points out that assessing first year results of an experiment without a baseline is "a little like trying to determine who won a basketball game by looking only at the points scored in the second half of the game."26 The Ohio Legislative Committee on Education Oversight (LCEO), responsible for monitoring Cleveland's voucher program, further discredited Peterson's criticisms. The LCEO had appointed Peterson to its technical review committee of the Indiana study, and charged that Peterson released his critique to the Wall Street Journal, the Washington Post, and the World Wide Web because he didn't like the results, even though the study's methods "are viewed as appropriate and credible by disinterested scholars."27
Florida A+ Plan
- In Florida, vouchers are available to students living in the attendance zone of any schools designated as "failing" by the state any two years out of four. To date the Florida program includes just two elementary schools in Escambia County. It is just now entering its third year, and there has been no research on its impact on achievement among voucher students. There has been an effort by researchers to assess the impact of the "voucher threat" in Florida on public school student test scores.