Flaws and Failings

Unable to Evaluate as Required by Congress

DOE has already acknowledged that the voucher program cannot be evaluated this year in the manner required by Congress

Through the voucher legislation, Congress has required that DOE and the Mayor’s office “shall jointly select an independent entity to evaluate annually the performance of students who received scholarships under the 5-year program under this title, and shall make the evaluations public . . .” Sec. 309(a)(1). The legislation mandates that DOE “ensure that the evaluation is conducted using the strongest possible research design for determining the effectiveness of the programs funded under this title that addresses the issues described in paragraph 4,” which include a comparison of “the academic achievement of participating eligible students . . . to the achievement of students in the same grades in the District of Columbia public schools; and the eligible students in the same grades in the District of Columbia public schools who sought to participate in the scholarship program but were not selected.” Pub. L. No. 108-199, Div. C, Title III, Sec. 309(a)(2) and (a)(4)(emphasis added).

By DOE’s own admission, the voucher program cannot be evaluated this year as Congress has mandated. Because so few students applied for vouchers, evaluators cannot compare the academic achievement of students who received vouchers with students in the same grades who sought vouchers but did not receive them, because the latter control groups do not exist. Marsha Silverberg of DOE’s Institute of Education Sciences acknowledged this in a June 10, 2004 e-mail to Sally Sachar concerning expected press questions about the evaluation, and explained that WSF could reply:

the Department will be evaluating the program, including testing and tracking the progress of students who applied and received scholarships this year. If anyone asks, we would say that we will compare the performance of this year’s students to the performance of DCPS students, as required by the law. There is not an adequate control group this year for a comparison of applicants who did and did not receive scholarships, but we anticipate having one for next year’s cohort.

(Emphasis added.)38

The documents provided to us indicate that the small number of applicants created some tension between those who wanted to give out as many vouchers this first year as possible (for obvious political purposes), and the evaluators who were seeking to maximize slots for the next school year in the hopes of having a genuine control group and being able to evaluate the voucher program in a meaningful way.

For example, in an e-mail dated May 24, 2004, Marsha Silverberg wrote that “For the evaluation, we need to conserve as many slots and potential applicants for next year’s lottery as possible. . . I know there are conflicting priorities -- you, understandably, want to fill as many slots as possible this year, while we want as many available as possible for a lottery next year.” E-mail from Silverberg to Sally Sachar and Nina Rees (May 24, 2004). On June 1, 2004, Silverberg wrote, “Because so few students will be denied scholarships thru the lottery, it is not possible to use this year’s applicants for an evaluation of the program’s impact or effectiveness. In order to meet Congress’ intent, it will be VERY important that next year, WSF generate as close to 2,000 public school applicants as possible. In order to detect impacts, if there are any, we need 800-1000 students assigned to receive scholarships next year, and a similar number (or slightly less) assigned to not receive scholarships.” E-mail from Marsha Silverberg to Sally Sachar et al. (June 1, 2004).

Similarly, Marsha Silverberg wrote on June 3, 2004, “Our objective is to make sure we can actually do a rigorous evaluation with next year’s cohort; that means ensuring a large number of applicants (1600-2000) and 800-1000 scholarships (and slots) for public school lottery winners next year.” E-mail from Silverberg to Sally Sachar, et al. (June 3, 2004). According to Silverberg, “A waiting list or preferred treatment for this year’s non-recipients will contaminate the lottery and reduce the slots available to lottery winners next year. If we thought we’d have many more applicants and slots than needed next year, we could let the current non-recipients in outside the lottery. But we think it will be hard, as it is, to get 2000 applicants and 800-1000 slots for lottery winners, so we strongly suggest that you simply ask those families to re-apply and be part of the lottery next year.” Id.

On May 24 Silverberg wrote that if 1200 “scholarships” were given out this year: “[e]ven with attrition that barely leaves the funds available next year to have at least a 800 public school treatment group (who are awarded scholarships) and 800 control group students. Those are the numbers we need to make sure the evaluation can find any program impact. Fewer than that number means that, if there is any impact, we won’t be able to detect it statistically. To me, that would be the very worst possible scenario.” E-mail from Marsha Silverberg to Sally Sachar, Nina Rees, et al. (May 24, 2004).

The relatively few number of students who applied for vouchers for 2004-05 has presented DOE not only with a problem concerning the mandatory evaluation, but also with a public relations problem. For example, DJ Nordquist, the Deputy Director of DOE’s Office of Public Affairs, advised Sally Sachar in May 2004 to be “very, very careful with the national media (and CBS News in particular). We obviously have some vulnerabilities with the DC Choice program since it appears that it is going to be undersubscribed, which is a point that the media is certainly going to exploit . . .” E-mail from DJ Nordquist to Sally Sachar (on or about May 24, 2004) (emphasis added).

And in correspondence that Nina Rees had on May 21, 2004 with Clint Bolick, a founder of the Institute for Justice and one of the country’s leading proponents of school vouchers, Rees had to disabuse Bolick of the impression that all was going well with the applications. Bolick had written: “[a]t least for now, the reports we’re getting is [sic] that parent outreach is going very well -- lots more applications than slots, parents are getting info, etc.” E-mail from Clint Bolick to Nina Rees (May 21, 2004). Rees replied: “We should talk. It is good that you are hearing positive news. It’s just that most of the parents who have signed up are not actually at or below 185% of the poverty line or there are problems with their applications. I think (and I am low balling here) after all is said and done we will only place 800-1000 kids . . . I just can’t see how we will be able to do more. And just fyi, we don’t really have more applications than slots. We do in high school and middle schools b/c we have so few private school slots but we don’t have oversubscription in elementary schools.” E-mail from Nina Rees to Clint Bolick (May 21, 2004) (ellipsis in original).

DOE was careful to keep certain members of Congress informed about the implementation of the voucher program. For example, on May 25, 2004, Nina Rees told Sally Sachar that before WSF issued its press release announcing the number of voucher applicants and related data, “there are some members [of Congress] that we need to give advance notice to and I have promised them they would get these details before they read about them in the Post.” In a separate e-mail of the same day, Rees wrote to Sachar that “Spector (ugh) wants it and while I hate the guy, we need to be nice to him I am told.” E-mail from Rees to Sachar (May 25, 2004) (spelling as in original).

Share this page: Facebook Twitter Digg SU Digg Delicious