Ironically, Lies and Distortions refers to researcher Jay Greene’s February 2001 report on Florida’s A+ Plan not as evidence that vouchers lead to better student performance in voucher schools—since there is no requirement that voucher schools participate in state testing, there is no way to know this. Instead, it cites Greene to contend that the voucher “threat” led to improvements in student performance in the low-performing ‘F’-rated public schools. This is a mantra of voucher advocates: voucher schools help public schools improve. But Greene’s report provides flimsy support, at best, for this notion. Indeed, Lies and Distortions conveniently ignores serious questions raised within the research community about Greene’s findings.
Significant methodological flaws in Greene’s analysis have been identified by Rutgers University researchers Gregory Camilli and Katrina Bulkley in a recent publication, including the fact that he uses school-level statistics rather than the more conventional student-level ones and yet asks in effect that his resultant “effect size” data be interpreted in the usual way. This leads to greatly inflated estimates of the gains made by the ‘F’-rated schools. Greene’s conclusions are “implausible,” according to Camilli and Bulkley.20
A second study, by Haggai Kupermintz of the University of Colorado at Boulder,21 demonstrates that the “gains” made by Florida’s ‘F’-rated schools may have been more apparent than real. Kupermintz’ analysis seems to support what had become the conventional wisdom in Florida at the time, shortly after the FCAT test results were announced in June 2000.22 This explanation was that the schools knew what was required to avoid an ‘F’ grade: they needed only to pass any one of the three sections on the FCAT, reading, math or writing. Of the three, writing was a far easier target for specific test preparation. As St. Petersburg Times reporter Stephen Hegarty observed: “Out of fear and necessity, Florida educators have figured out how the state's writing test works and are gearing instruction toward it—with constant writing and, in many cases, a shamelessly formulaic approach.”23 Schools did not want to be branded as “failing schools,” they knew the easiest route to escape that fate, and they succeeded—but without necessarily improving actual classroom learning. As Kupermintz concludes, Greene’s report provides a “a false sense of a dramatic success” with respect to vouchers.24
In addition to concerns about Greene’s methodology, People for the American Way Foundation pointed out in an earlier report that Greene failed to consider the significant impact of extra resources, both state and local, which were directed towards Florida’s ‘F’-rated schools. These resources enabled the schools to extend the school day, week, and year, as well as strengthen professional development activities for teachers, in order to help students raise their achievement. In short, these elements may well have been the real cause of improvements in these schools, along with testing and accountability.25 Indeed, Greene’s own research in Texas supports the conclusion that accountability, testing, and increased resources promote school improvement, independent of vouchers. In an earlier article on Texas, Greene found that the state’s Texas Assessment of Academic Skills (TAAS)—the set of tests which is the centerpiece of the Texas state accountability system—was critical to what he has called the “Texas school miracle” (the significant gains in some Texas achievement results during the 1990s).26 Texas, of course, does not have a private school voucher program.
Yet, Lies and Distortions is undeterred by the strong critiques of Greene’s Florida study. The booklet even borrows a quote from the August 2000 issue of the ardently pro-voucher School Reform News. In this issue, University of Florida professor David Figlio—who is researching the effect of vouchers under Florida’s A+ Plan—is quoted as saying: “All you need to have is the threat of vouchers” in order for public schools to improve. School Reform News gives no citation for the quotation, stating only that Figlio said this to the Tampa Tribune.27 According to Figlio, this quotation was taken out of context and was removed from the version of the article available from online search engines after Figlio “expressed dismay” to the Tribune.28 A February 2001 article in the St. Petersburg Times more accurately reflects Figlio’s assessment. Commenting on Greene’s report, Figlio stated that two more years of data would be needed before one could draw any conclusions about the impact of vouchers under the A+ program. “It’s such a charged political environment now; there's a tremendous hunger among policymakers and journalists to draw conclusions,” he told the St. Petersburg Times. “We will sell no wine before its time. It’s still grape juice, in my opinion.”29 In a recent e-mail message to PFAWF, Figlio reinforced his view that while “something happened” to public schools due to the threat of vouchers, “it’s not clear whether the ‘something real’ that happened is good or bad.” He added that, while he considers Greene’s analysis a reasonable first step, he feels that Greene’s conclusions suffer from “lots of confounding factors and it’s way too early to say anything about the threat of vouchers.”30
The authors of Lies and Distortion also reassert that voucher schools have raised the academic achievement of their students, but the supporting information the authors use is noteworthy for what it omits. For example, concerning Milwaukee vouchers, Lies and Distortions correctly reports that Princeton University researcher Cecilia Rouse found math gains for voucher students.31 Yet, the findings Rouse was characterizing were only for the subgroup of students who were in the voucher program over a four-year period. It should be noted that the University of Wisconsin’s John Witte found “that voucher students who left the [Milwaukee] program for various reasons had lower test scores than those who continued to participate [emphasis in original],”32 reminding the public that a full and accurate assessment of voucher schools considers not simply those students who use a voucher and remain in a voucher school, but, rather, all students who entered the voucher program. In other words, students who do well in voucher schools are more likely to stay—those doing poorly are more likely to leave or drop out. Moreover, Lies and Distortions neglects to mention Rouse’s conclusions that “the [voucher] effects on the reading scores are as often negative as positive and are nearly always statistically indistinguishable from zero.”33
Most significantly, Lies and Distortions ignores another finding by Rouse: “The results suggest that students in P-5 schools [Milwaukee public schools with small class sizes and supplemental funding] have math test score gains similar to those in the choice schools, and that students in the P-5 schools outperform students in the choice schools in reading.” Rouse went on to explain: “Given that the pupil-teacher ratios in the P-5 and choice schools are significantly smaller than those in the other public schools, one potential explanation for these results is that students perform well in schools with smaller class sizes [emphasis in original].”34 In other words, the gains in math for voucher students may very well not be due to the fact that they were in voucher or private schools, but to other factors such as class size.
In short, there is no compelling evidence from the Milwaukee voucher program that demonstrates student achievement gains. Indeed, Rouse’s analysis of P-5 schools strongly suggests that well-funded public schools with reduced class sizes produce test scores equal or superior to voucher schools.