![]()
|
Major Areas State Education
Agency Calendars Columns Directories Organizations |
U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance Evaluation of the DC Opportunity Scholarship Program: First Year Report on ParticipationApril 2005 Patrick Wolf, Principal Investigator, Georgetown University Marsha Silverberg, Federal Project Officer U.S. Department of Education/Institute of Education Sciences U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance April 2005 This report was prepared for the Institute of Education Sciences under Contract No. ED-01-CO-0082/0016. The project officer was Marsha Silverberg in the National Center for Education Evaluation and Regional Assistance. The views expressed herein are those of the contractor. This publication is in the public domain. Authorization to reproduce it in whole or in part for educational purposes is granted. Suggested Citation Wolf, Patrick, Babette Gutmann, Nada Eissa, Michael Puma, and Marsha Silverberg. Evaluation of the DC Opportunity Scholarship Program: First Year Report on Participation. U.S. Department of Education, National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office, 2005. This report is available on the Institute of Education Sciences’ Web site at: http://www.ed.gov/ies/ncee ContentsAcknowledgements List of TablesTable ES-1 Number of DC Private Schools
Participating in the DC Opportunity Scholarship Program: 2004-05 List of FiguresFigure ES-1 Eligible Public School Applicants and Available Private School
Slots, by Grade-Level Band: Spring 2004 and Fall 2004 ACKNOWLEDGEMENTSThis report is the first of a series of annual reports, as mandated by Congress. We gratefully acknowledge the contributions of a significant number of individuals in its preparation and production. Staff from the U.S. Department of Education and the Mayor’s Office provided ongoing support throughout the process. Special recognition and thanks go to Marsha Silverberg at the Institute of Education Sciences’ National Center for Education Evaluation, the Contracting Officer’s Representative for this project, for her contributions and her encouragement. Guidance and comments were also received by Ricky Takai, Director of IES’ National Center for Education Evaluation and Phoebe Cottingham, Commissioner of IES’ National Center for Education Evaluation and Regional Assistance; Nina Rees, Assistant Deputy Secretary for the Office of Innovation and Improvement and Michael Petrilli, Associate Assistant Deputy Secretary for the Office of Innovation and Improvement; and Michelle Walker, Senior Advisor for Education, Executive Office of the Mayor of the District of Columbia. Staff from the Washington Scholarship Fund, the Public School Aid Service, and the District of Columbia Public Schools provided critical data and were always there to answer our many questions. We are also fortunate to have the advice of an Expert Advisory Panel. Members include: Julian Betts, University of California, San Diego; Thomas Cook, Northwestern University; Jeffrey Henig, Columbia University; William Howell, Harvard University; Guido Imbens, University of California; Rebecca Maynard, University of Pennsylvania; Larry Orr, Abt Associates. The challenging task of assembling the many data files was capably undertaken by Robert Schrack and Yong Lee at Westat. Providing invaluable assistance in the many analyses were Mike Ingram and Dan Hoople of Georgetown University. The management and conduct of the data collection was performed by Juanita Lucas-McLean and Kevin Jay of Westat. Expert editorial and production assistance was provided by Evarilla Cover, Saunders Freeland, Joan Murphy, and Beth Sinclair of Westat. Insightful comments throughout the course of the work were provided by Tom Stewart of Qwaku Productions. Administrative support for the Georgetown University project activities was provided ably by Stephen Cornman. EXECUTIVE SUMMARYThe District of Columbia School Choice Incentive Act of 2003 was passed by Congress in January 2004. The Act provided funds for District of Columbia Public Schools (DCPS) improvement activities and charter school facility acquisitions. Most notably, the statute established what is now called the DC Opportunity Scholarship Program―the first federal government initiative to provide K-12 education scholarships, or vouchers, to families to send their children to private schools of choice. The DC Opportunity Scholarship Program has the following programmatic elements:
The Act requires that this 5-year scholarship program be rigorously evaluated by an independent research team, using the "strongest possible research design for determining the effectiveness" of the program and addressing a specific set of student comparisons and topics (Section 309). The evaluation thus has several components: (1) an impact analysis, comparing outcomes of eligible applicants (students and their parents) from public schools randomly assigned to receive or not receive a scholarship through a lottery, and (2) a performance reporting analysis, comparing all students participating in the scholarship program to students in the same grades in DCPS. All participating students includes those randomly assigned scholarships and those who received scholarships automatically, those who were attending public schools and those attending private schools when they entered the scholarship program. Because DCPS students who did not apply to the scholarship program are likely to be quite different from those who applied and are participating, the impact analysis will be the source of the reliable, causal evidence on program effectiveness called for in the legislation. This document is the first of a series of annual reports from the evaluation team, as mandated by Congress. Because the initial cohort of program participants—those who applied in spring 2004 to receive scholarships for the 2004-05 school year—just recently matriculated at their new schools, no impact information is available at this time. Instead, the report describes the purposes and design of the scholarship program, the first-year implementation activities that generated 1,848 eligible applicants and 58 participating private schools, the process of awarding scholarships to 1,366 student applicants, and the characteristics of both applicants and scholarship users. The report provides an important foundation for the later examination of program impacts. Program Implementation in 2004: Recruitment and Applications (Chapter 2)The Washington Scholarship Fund (WSF) was awarded a grant by the U.S. Department of Education (ED) Office of Innovation and Improvement, in partnership with the DC Mayor’s Office, to implement the program, starting in March 2004. Despite the challenges stemming from the late start of the program, the implementers recruited 58 schools to participate in the program in some capacity in 2004-05 and obtained applications from 1,848 students deemed eligible for the program. Participating Schools The 58 private schools participating in the program during its inaugural year represent 53 percent of all private schools in the District (Table ES-1). All but four of the schools made new slots in their schools available to scholarship winners. Four schools were willing to enroll scholarship students only if they had been accepted to the school for the 2004-05 school year prior to the launch of the scholarship program. Table ES-1. Number of DC Private Schools Participating in the DC Opportunity Scholarship Program: 2004-05
NOTE: Detail may not sum to totals because of rounding. The characteristics of the private schools that chose to participate in the program the first year include the following:
Participating Families and Students The program implementer – WSF – conducted most of the outreach to and recruitment of families between March and May 2004. Perhaps as many as 40,000 DC children were eligible for the scholarship program, based on data from the U.S. Census (Table ES-2). Inquiries about the program were made on behalf of almost 6,000 students, and nearly 2,700 applications were submitted during the recruitment period. A total of 1,848 applicants (69 percent of those who applied) provided all of the required documents and were deemed eligible for the program. Seventy-two percent of those eligible applicants were attending public school during 2003-04, whereas 28 percent were already attending private schools but met the eligibility requirements in the statute. Table ES-2. Number and Percentage of Participants, by Application Status: Spring 2004
NOTE: Because the eligible base, inquiries, and applicants included an unknown combination of public and private school students, it would not be appropriate to express the number of public or private eligibles as a percentage of those bases. SOURCE: Figure for the "Eligible Base" is based on data from the U.S. Census, population of the District of Columbia age 5 to 17 under 185 percent of the federal poverty line in 2000. The exact number for 2004 is likely to differ somewhat from this 2000 figure. Figure for "Inquiries" provided by Fight For Children. Figures for "Applicants" and "Eligible Applicants" were drawn from the applicant database, with eligibility determined by Private School Aid Service (PSAS). Numbers of applicants from public or private schools were determined by cross-matching the name of the school each child was attending against a list of public and private schools in DC compiled from the DCPS web site and the NCES Common Core of Data. Scholarship and Placement Lotteries and Initial Use of Scholarships Awarded (Chapter 3)The program statute requires that scholarship recipients be randomly selected (e.g., by lottery), if the program or specific schools are “oversubscribed”―that is, have more demand for them than can be accommodated. The law also details congressional priorities to guide the award of scholarships and scarce seats to eligible applicants: (1) students attending a public school designated as in need of improvement (SINI) under the No Child Left Behind Act at the time of application to the program, and (2) families that lack the resources to take advantage of the educational choices available to them. A total of 79 eligible applicants (4 percent) were from one of the 15 SINI-designated schools in spring 2004 and were, therefore, given the highest priority in the lotteries.1 An additional 1,251 eligible applicants (68 percent) were attending non-SINI public schools and were assigned the second-highest priority in the lotteries. The 518 eligible applicants (28 percent) from private schools were given the lowest priority, because they were considered to meet neither of the congressional priorities. These priority groups were used both to award scholarships and, later, to place scholarship recipients in the participating private schools of their choice. The Scholarship Lottery The first lottery was to distribute scholarships to eligible applicants. For public school applicants, each student’s probability of obtaining a scholarship was dependent not only on his or her membership in a priority group but also on the availability of new private school seats at various grade levels. The new seats in participating private schools were highly concentrated in the K-5 elementary grades, meaning there were more available seats in those grades than there were eligible applicants (Figure ES-1). Thus, in the lottery, K-5 students would receive scholarships automatically. The middle school grades (6-8) were modestly oversubscribed, and the high school grades (9-12) were severely oversubscribed. Only in those grades was there random assignment as part of the lottery. Since eligible private school applicants already held slots in their private schools, they were not constrained by slot availability in the same way as public school applicants. Accounting for both the statutory priorities and the slot constraints within the grade-level bands, scholarship award probabilities were assigned to the various groups of eligible applicants and a custom-designed computer program awarded scholarships to students within each group (Figure ES-2). A total of 1,366 scholarships were awarded in June 2004, including the following:
In sum, the scholarship lottery produced two groups of students for purposes of meeting statutory requirements for the DC Choice Opportunity Scholarship program evaluation. The 1,366 scholarship recipients will be the subject of annual performance reporting and comparison to DCPS nonapplicants. The 492 public school applicants who were entering grades subject to random assignment (grades 6-12) will contribute to the annual impact analysis: these include 299 students assigned to the treatment group (and also included in the performance reporting sample) and 193 students assigned to the control group. The 289 private school applicants who were not awarded scholarships belong to neither group. Since they previously attended and presumably will continue to attend private school using resources outside of the scholarship program, following these students after baseline would not contribute meaningfully to the evaluation. To conserve resources, this group of initial applicants will not be part of the evaluation going forward. However, the impact sample in the first year of the program is not, on its own, large enough for the evaluation to reliably draw conclusions about any differences in achievement outcomes that might be expected from an intervention of this kind.3 Instead, the treatment and control groups from the first year lottery will be combined with those from the lottery for second year applicants, expected in April 2005, to provide a sufficient sample for the rigorous evaluation of program impacts.Placement Lottery and Follow Up After being notified of their scholarship offer, families were required to meet with officials at participating private schools and obtain conditional acceptance to the schools that they wanted their children to attend. Parents then submitted school preference forms to indicate and rank the top four private schools of their choice. These forms were used to place students, through a combination of a custom-designed computer placement lottery and followup case-by-case placements by WSF. There were a total of 1,366 scholarship winners:
Of the 1,040 who were placed, 1,027 had matriculated at their preferred private school by September 10, 2004. This represents an overall initial scholarship usage rate of 75 percent. The usage rate for the impact sample’s treatment group is lower—62 percent—because that group excludes students in grades K-5 and those who were already attending private schools when they applied to the program, subgroups that have significantly higher rates of scholarship use than do public school students in the middle and high school grades. Characteristics of Program Applicants (Chapter 4)In determining and interpreting program effectiveness later on, it is important to know how well the program is targeted to the disadvantaged families who are the focus of the program and, beyond that, what types of families and students apply, win scholarships in the lottery, and choose to use them to enroll in a private school. It is also useful to identify the extent to which public and private schools in the District are experiencing a significant loss or gain of students due to the first year of program implementation, because that information provides the foundation for our later examination of the impact of the program on DC schools. Public School Applicants Compared to Similar Public School Students in DC There are several reasons to examine the eligible public school applicants to the DC Opportunity Scholarship Program in relation to other DCPS students. Most clearly, the comparison provides a context for considering the kinds of students who might be attracted to the program in future years in the District or to a similar program in other locations. The income eligibility criteria for the program—family income within 185 percent of the federal poverty line—matches up quite closely with eligibility for the federal free or reduced-price lunch (FRL) program. Using this income indicator to compare public school program applicants to similarly low-income DCPS nonapplicants, we find some differences and similarities (Table ES–3): Table ES-3. Characteristics of DC Public School Free or Reduced-Price Lunch Program Students, Program Applicants Versus Nonapplicants: Spring 2004
* Statistically significant at the 95 percent
confidence level. 1. The sample size of 894 for the applicant group compared here differs from the total sample of 1343 public school applicants for two reasons. First, only 1,077 (80 percent) of the public school applicants could be identified conclusively in the DCPS accountability testing database, the only source of comparable data for both applicants and nonapplicants from DCPS. Most missing observations were in pre-K, first, or second grade, where accountability testing is optional. An additional 183 public school applicants in the accountability database were not enrolled in the free or reduced-price lunch (FRL) program. Because it is most analytically sound to compare similarly disadvantaged groups of applicants and DCPS nonapplicants, we included in this comparison only those students who had test scores in the database and who were confirmed eligible for FRL. To examine whether there was any substantial bias in our analysis, we compared the demographic characteristics of students who were tested by DCPS with those who were not tested, both within and across the applicant and nonapplicants samples, using a t test for difference of means to identify statistically significant differences. The testers did not differ significantly from the nontesters on any characteristic, except for grade, since testing is mandatory only in grades 3 and higher. Therefore, we are confident that tester/nontester bias does not affect the comparisons that we make here. 2. Test-score results are in terms of National Percentile Ranks, with 50 as the median score. 3. "Other race" includes students who were identified as white, Asian, American Indian, or Alaska Native. NOTE: Detail may not sum to totals because of rounding. Applicant sample includes all eligible applicants identified in the DCPS data base that were participating in the free- and reduced-price lunch program. SOURCE: Accountability testing database for DC public and charter schools, DCPS Office of Communications and Public Information.
Similar patterns are evident when we use a subset of the applicants—just the program participants—to compare to economically disadvantaged DCPS students in the same grades, as the program statute requires for performance reporting.4 Applicants in the Impact and Non-Impact Samples While it is important to examine who applies to the DC Opportunity Scholarship Program, for the evaluation it is equally critical to assure that the applicants who will be the focus of the impact analysis—the applicants who were randomly assigned to treatment and control groups randomly assigned to treatment by the lottery—are similar prior to the beginning of the program. It is on this similarity between the groups, not only in characteristics easily measured but also in those not observed, that the scientific benefits of the randomized control trial (RCT) approach rests. We find no statistically significant differences between the two groups on any major educational or family background measures, confirming that the assignment lottery was conducted appropriately. It is also useful to review how the characteristics of the impact sample differ from the characteristics of students who applied but will not be included in the analysis of program effectiveness (the "non-impact sample" of applicants). The existence of many significant differences between the impact and non-impact samples limits extrapolation of the results of the impact analysis to characterize overall program impact. There are a few differences between the two groups. Compared to their non-impact sample counterparts, members of the impact sample: (1) scored higher in reading in grades 9 through 12, (2) are more likely to have a learning disability, and (3) are less likely to be of Hispanic ethnicity. Applicants by Type of Previous School The schools students previously attended may be associated with students’ educational or background characteristics, their parents’ attitudes, and ultimately the extent to which the program is effective for students seeking scholarships. Eligible applicants to the program came from four different types of schools. Four percent came from SINI-designated public schools, 54 percent from non-SINI regular public schools, 14 percent from public charter schools, and 28 percent from various private schools. The most important differences among the applicants include the following:
Applicant Response Rates Among Public and Private Schools A central question in the debate surrounding school choice—and one of the topics the statute requires the evaluation to address—is whether a scholarship program has an impact on the larger public and private school systems. Such "systemic effects" could take place if significant percentages of students in the public school system or in specific schools apply for, receive, and use scholarships to transfer to private schools. With regard to the public schools, the DC Opportunity Scholarship Program could have either positive or negative effects. One theoretical argument suggests that scholarship programs will divert funding and the most motivated students from public schools to private schools, leaving the public school system with fewer resources with which to educate the remaining student population. Another theory is that schools behave in a manner similar to firms and will respond to competition by becoming more efficient. In the case of schools, it is the risk of losing students and subsequently funding that may provide an impetus for public schools to provide better services and produce better student outcomes. Private schools in jurisdictions with greater school choice may face similar incentives to improve or expand in order to retain as many of their current students and attract more. To provide the basis for a later analysis of the effects of the DC Opportunity Scholarship Program on schools, we first need to describe the extent to which public schools have so far been affected by program applications and scholarship users—a possible predictor of the level of competitive pressure the public schools may experience. The school-level scholarship application and use rates this first year suggest that relatively few public schools have experienced a significant loss of students as a result of the scholarship program (Table ES-4):
Table ES-4. Public School-Level Scholarship Application and Usage
Rates:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Applied for the Program |
Used a Scholarship |
|||
|
Percent of Student Body |
Number of Schools |
Percent of Schools |
Number of Schools |
Percent of Schools |
|
0 |
25 |
13 |
51 |
26 |
|
0.1 – 1.0 |
47 |
24 |
63 |
32 |
|
1.1 – 2.0 |
56 |
28 |
47 |
24 |
|
2.1 – 3.0 |
38 |
19 |
21 |
11 |
|
3.1 – 4.0 |
12 |
6 |
12 |
6 |
|
4.1 – |
19 |
11 |
3 |
2 |
|
Total |
197 |
100 |
197 |
100 |
NOTE: Detail may not sum to totals because of rounding.
SOURCES: Application and usage numbers by school generated from the Applicant Database and Washington Scholarship Fund (WSF) Placement Database. Enrollment figures for DCPS are from the 2002-03 school year and were obtained from Membership in the District of Columbia Public Schools by School and Grade, October 7, 2003, available on the DCPS web site, www.k12.dc.us/dcps/data/enrollment/membership-Oct.703_official_.pdf. Enrollment figures for DC public charter schools chartered by the District Board of Education are from the 2002-03 school year and were obtained from the Common Core of Data, National Center for Education Statistics. Enrollment figures for the DC public charter schools chartered by the DC Public Charter School Board are from the 2003-04 school year and were obtained from the Board’s web site at www.dcpubliccharter.com.
In contrast, a similar analysis of the participating private schools suggests that students using DC Opportunity Scholarships make up a significant share of their enrollments. In more than one-quarter of those private schools, nearly 20 percent of their students are using Opportunity Scholarships; in another 37 percent, scholarship students make up between 5 and 20 percent of their student populations in 2004-05.
The District of Columbia School Choice Incentive Act of 2003 was passed by Congress in January 2004. The Act provided funds for District of Columbia Public School (DCPS) improvement activities and charter school facility acquisitions. Most notably, the statute established what is now called the DC Opportunity Scholarship Program first federal government initiative to provide K-12 education scholarships, or vouchers, to families to send their children to private schools of choice.
The statute requires that this 5-year scholarship program be rigorously evaluated by an independent research team. This document is the first of a series of annual reports from that team, as mandated by Congress. The report describes the purposes and design of the scholarship program, the first-year implementation activities that generated 1,848 eligible applicants and 58 participating private schools, the process of randomly awarding scholarships to 1,366 student applicants, and the characteristics of both applicants and scholarship users. The report provides an important foundation for the later examination of program impacts.
In January 2004, the U.S. Congress passed the DC School Choice Incentive Act of 2003, Title III of Division C of the Consolidated Appropriations Act, 2004, P.L. 108-199. The statute established a new, 5-year school choice program for low-income residents of Washington, DC. The key elements of the program include the following:
To be eligible, students entering grades K-12 must reside in the District and have a family income at or below 185 percent of the federal poverty line.
Participating students will receive scholarships of up to $7,500 to cover the costs of tuition, school fees, and transportation to a participating private school of choice.
Scholarships are renewable for up to 5 years (as funds are appropriated), as long as students remain eligible for the program and remain in good academic standing at the private schools they are attending.
If there are more eligible applicants than available scholarships or open slots in private schools, applicants are to be awarded scholarships and admission to private schools by random selection (e.g., by a lottery).
Private schools participating in the program must be located in Washington, DC, and agree to program requirements regarding nondiscrimination in admissions, fiscal accountability, and cooperation with the evaluation.
Certain groups of students have priority in obtaining access to the program: (1) those coming from public schools identified as in need of improvement (SINI) under the federal Elementary and Secondary Education Act and (2) those whose families lack the financial resources to take advantage of available educational options.
The law also charged the U.S. Secretary of Education and the Mayor of the District of Columbia with selecting both a program implementer and an independent evaluator of the program. The Secretary designated the Office of Innovation and Improvement (OII) within the U.S. Department of Education (ED) as the lead agency for funding and monitoring the implementation of the program, and the Institute of Education Sciences (IES) to take the lead in funding and monitoring the independent evaluation. Under current appropriation levels—about $13 million annually—the program is likely to support 1,800 to 2,000 scholarships, depending on the tuition levels of the selected private schools.
Section 309 of the Act describes the requirements for an independent evaluation of the DC Opportunity Scholarship Program. The Secretary of Education is to ensure the following:
"The evaluation is conducted using the strongest possible research design for determining the effectiveness" of the school choice program;
The results of the evaluation regarding the impact of the program on the participating students and nonparticipating students and schools in the District are disseminated widely.
Early on, IES determined that the foundation of the DC Opportunity Scholarship Program evaluation would be a randomized controlled trial (RCT), comparing outcomes of eligible applicants (students and their parents) randomly assigned to receive or not receive a scholarship.6 This decision was based on the mandate to use rigorous evaluation methods, the expectation that there would be more applicants than funds and private school spaces available, and the requirement to use random selection to determine who receives a scholarship. In addition, the law clearly specified that such a comparison in outcomes be made.7 This component represents the impact analysis and will provide evidence on the effectiveness of the program.8
The law also called for the evaluation to track program progress in other ways. For example, the evaluation must compare students participating in the scholarship program to students in the same grades in the DC Public Schools. However, DCPS students who did not apply to the scholarship program are likely to be quite different from those who applied and are participating—in ways we can observe and ways we cannot. Comparing outcomes between participants and nonapplicants is, therefore, not a reliable measure of program effects. Instead, this type of performance reporting will be combined with other data collection and analysis that examines the context in which the program is operating.
In spring 2004, IES initiated a competitive bidding process to select an initial technical advising team as well as an entity to design and implement the 5-year impact evaluation of the program. The technical advising contract was competed quickly, so that a group of experts would be in place to advise ED and the program operator regarding the baseline data collection and the lotteries that are essential to both the effective launch of the program and the subsequent impact evaluation. In March 2004, the technical advising contract was awarded to a research consortium led by Westat and including Georgetown University and Chesapeake Research Associates. Later, in July 2004, the competition for the 5-year impact evaluation contract concluded with an award to the Westat-Georgetown-Chesapeake team.
Research Questions
Based on guidance in the statute, the research team plans to conduct a comprehensive and rigorous RCT evaluation of the impact of the scholarship program on participating students and families. Specifically, the impact analysis will address the following research questions:
What is the impact of the program on student academic achievement?
What is the impact of attending private versus public schools?
What is the impact of the program on other student measures?
What effect does the program have on student and parent satisfaction with the educational options available in the District and with children’s actual school experiences?
The evaluation will also address other issues posed in the law through program performance analysis:
How well are scholarship recipients performing relative to students in DCPS?
To what extent is the program influencing public schools and expanding choice options for parents in Washington, DC?
Data Collection
To answer these questions, the evaluation will draw on
different types of data—some available from DCPS, some collected for the
purposes of this study. These data will include preprogram (baseline)
measures of family background and student achievement. The baseline
measures allow us to verify that students randomly assigned to the
scholarship and nonscholarship groups were, in fact, similar before the
program; the measures also enable us to create subgroups of students whose
impacts we might want to examine separately, such as students with low
prior achievement. Additional data collected will include annual "in
program" measures (e.g., parents’ satisfaction with their children’s
school, students’ academic achievement), which will serve as outcomes
for the rigorous evaluation of program impacts (see
Table 1-1)
Impact Analysis
It is well known that the independent effects of school choice on student outcomes are difficult to estimate. Perhaps the most significant difficulty faced by researchers is selection bias self-selection of families to even seek out a new school choice for their child, and the mutual student/school decision process that selects students into different types of schools. Because this bias is generally a result of unmeasurable factors, most researchers have preferred the use of an RCT to a dependence on non-experimental (nonrandomized) statistical methods. Since the DC Opportunity Scholarship Program provides for the random distribution of scholarships through a lottery, we will, therefore, use RCT methods to estimate most program impacts.10
Table 1-1. Data Sources
|
Data Source |
Description |
|
Student assessments |
|
|
Parent surveys |
|
|
Student surveys |
|
|
Principal surveys |
|
1. Baseline achievement will be collected only for applicants from public schools because, as described in the Impact Analysis Sample section, applicants who were already attending private schools will not be included in the impact analysis. For public school applicants who did not participate in regular DCPS testing in the year they applied to the program (e.g., particularly children below grade 3), the study will administer the equivalent DCPS assessment to these students in the fall after application. All other data will be collected for all applicants, both from public and private schools.
Impact Analysis Sample
The RCT approach rests on random assignment or, in the case of the DC Opportunity Scholarship Program evaluation, a lottery to create two statistically equivalent groups of students from among program applicants: (1) a "treatment" group that receives a scholarship, and (2) a "control" group that does not receive a scholarship. Because the two groups are generated from the same pool of applicants, they are equally likely to be motivated to participate in the program and to reap any benefits from it. And as long as the pool of applicants is sufficiently large, the random assignment of students into treatment and control groups should produce groups that are similar in other characteristics, both those we can observe and measure (e.g., family income, prior academic achievement) and those we cannot (e.g., motivation to succeed). The random assignment assures that all observed and unobserved characteristics are equally represented in both groups.
However, according to the statute, the random assignment that is the means to create the treatment and control groups can only be used to help allocate scholarships under particular circumstances. Finally, it may not be appropriate to include some student applicants in the impact evaluation, even if they participate in the lottery. As a result of all of these conditions, the impact analysis sample will perform the following:
Exclude applicants who are already attending private schools. The statute contained no provision to exclude from the program students who were currently enrolled in private schools but otherwise eligible to participate.11 A substantial number of private school students did apply to the program, as described in Chapters 2 and 3. However, because those students intended to use the DC Opportunity Scholarship to continue to attend private schools, measuring the difference in outcomes between private school applicants who did and did not receive a scholarship through the lottery would likely only answer the question of whether a different type or amount of scholarship funds affect student outcomes. While that question is of some policy interest, it is not the main focus of the evaluation as specified in the legislation. Therefore, applicants currently enrolled in private schools will not be part of the impact analysis sample.
Include only public school applicants in grades where there are more applications than there are available private school slots. A lottery is a fair and efficient way of distributing scholarships when there are too many applicants, but the law specifies that random selection be used only when the program or particular grades are oversubscribed.
Exclude public school students who automatically receive a scholarship. It is possible that even if many students apply to participate in the program, there may not be oversubscription (i.e., more demand than slots) in some grade levels. In those grade levels, all applicants will receive scholarships and there will be no control group.
Thus, the impact evaluation of the DC Opportunity Scholarship Program depends on the extent to which large numbers of eligible DC families with public school students apply to the program. The treatment and control groups must be of a sufficient size to allow us to detect and measure any difference in outcomes between the two groups (the "impact") with statistical certainty. A procedure called "power analysis" is used to determine the sample sizes necessary to enable the study to answer the central research questions and to measure program effects that are large enough to be both meaningful in students’ lives and relevant to policy debates about the efficacy of school choice interventions. The power calculations conducted for the evaluation suggest that a total of at least 1,240 public school applicants must be randomly assigned, with over 800 assigned to the treatment (scholarship groups) and over 400 assigned to the control (non-scholarship) group in order for the impact analysis to be able to detect moderately large test score effects (see Appendix A for more detail).
General Statistical Approach: Estimating the Impact of the Offer of a Scholarship
Given appropriately sized treatment and control groups, the strategy for analyzing impacts is well established. To motivate the discussion of how we identify the effect of the scholarship program on test scores, it is useful to begin with a simple representation of the selection problem as a missing data problem, using the potential outcomes approach. This approach defines causal effects in terms of potential outcomes or counterfactuals. Conceptually, the causal effect of treatment—the scholarship—is defined as the difference between the "outcome for individuals assigned to the treatment group" and the "outcome for the treatment group if it had not received the treatment," or:
(E.1) "E(Yi| Xi, Ti =1)" - "E(Yi |Xi, Ti =0)"
In the case of scholarships, the treatment effect effect of the scholarships on academic achievement be defined as the difference between "test scores for program students" and "test scores for program students if they had not received a scholarship." The fundamental problem is that a student is never observed simultaneously in both states of the world. What is observed is a student in the treatment group (Ti =1) or in the control group (Ti =0). The outcome in the absence of treatment, E(Yi |Xi, Ti =0), is then the counterfactual would have occurred to those students receiving the scholarships if they had not received them.
If students receiving scholarships were identical to other students in both observable and unobservable characteristics, the counterfactual could be generated directly from an appropriately selected comparison group. Valid comparison groups are rarely found in practice, however. The random assignment of students into the program generates the counterfactual from the control group — applicants who did not receive a scholarship.12 If correctly implemented, random assignment yields statistically equivalent groups and allows estimation of the program impact through differences in mean outcomes between the two groups.
Consistent with this approach is the following basic analytic model of the effects of school choice scholarships on outcomes. Consider first the outcome equation for the test score of student i in year t. It is reasonable to assume that test scores (Yit ) are determined as follows:
(E.2) Yit =α+ τ Tit + Xi γ+ εit if t>k (period after program takes effect)
Equation (E.2) estimates the effect of the offer of a scholarship on student outcomes. Under this model, commonly referred to as the "Intent to Treat" (ITT) estimation, all students who were randomly assigned by virtue of the lottery are included in the analysis, regardless of whether a member of the treatment group uses the scholarship to attend a private school. In E.2, Tit is equal to one if the student has the opportunity to participate in the scholarship program (i.e., the award rather than the actual use of the scholarship) and equal to zero otherwise. Xi is a vector of student characteristics (measured at baseline) known to influence future academic achievement, such as prior test scores, mother’s level of education, family income, etc. In this model, τ represents the effect of scholarships on test scores for students in the program, conditional on Xi. With a properly designed RCT, using a concise and judiciously chosen set of statistical controls for characteristics that predict future achievement should improve the precision of the estimated impact.13 That treatment effect, τ, should be identical to the difference in mean outcomes between the treatment and the control groups.
Since the initial applicants were randomized within certain relevant subgroups, as described in Chapter 3, we will analyze program impacts using a randomized block design. We are interested in how academic achievement (Y) is affected by the assignment into the scholarship program within each block (B) or group of size n. The impacts are then estimated as:
(E.3) Yikt = μ+ τ Tikt +bj=2 ρj Bik+ Xik γ+ εik,t
where
i = 1,…..,n observations and k=1,….,b blocks(defined by grade and priority status);
Yji is the outcome for student i in block j, at time t;
μ is the overall mean outcome (e.g., test score);
τ is the treatment (scholarship program) effect;
ρj is the jth block effect;
Tit is assignment into the scholarship program;
Bji is the block assignment;
Xji represents observable characteristics, measured at baseline; and
εij is the random error; independent, Ν(0,σε2 ).
This analytical framework follows naturally from the group randomization and is easily implemented and interpreted. Y can be measured in several different dimensions, including test scores, school satisfaction, parental satisfaction, grade completion, including where appropriate, high school graduation, etc. μ is average outcome for all program members, ρj is the average block effect, and τ is the effect of scholarships on academic achievement.14
Estimating the Impact of the Use of Scholarships
Even with a properly implemented RCT, we may expect that not all applicants placed by random assignment into the treatment (scholarship offer) group will actually use the scholarship at a private school. That is, some scholarship recipients may choose not to use their scholarship and instead attend a public school. This type of non-participation or underutilization of treatment services has been observed across all RCT settings, including medical trials, job training and health insurance experiments, as well as in previous school scholarship RCTs such as the one of the Milwaukee Parental Choice Program.
Policymakers are typically interested in the effect of scholarship use on student achievement, in addition to the offer of the scholarship. To estimate the impact, we will use a model commonly referred to as the "Impact of the Treated" (IOT), which statistically estimates the impact of actual scholarship use. Instrumental variable analysis provides us with a well-established method to generate an unbiased estimate of the scholarship impact on the treated from the ITT estimator.15
Performance Reporting Analysis
To fulfill the requirements specified in the law, we will compare the outcomes of all participating students with the outcomes of similar DCPS students who did not apply to the program.
Performance Reporting Sample
The group of student applicants that will be examined as part of performance reporting differs from the impact analysis sample in several important ways. The impact analysis sample includes all public school applicants who were randomly assigned to receive or not receive a scholarship as part of the lottery (i.e., students in grade bands for which there were more applicants than there were private school slots available). In contrast, the performance reporting sample performs the following:
Excludes students who did not receive a scholarship as part of the lottery (the control group).
Includes scholarship recipients who were already attending private schools at the time of application.
Will focus on students who chose to use their scholarship; while the law does not define what it means to "participate," performance reporting in the evaluation’s later reports will examine the differences between students who were given the option of participating (i.e., received a scholarship) and those who exercised that option (i.e., who used the scholarship to attend a private school). Both groups will be compared to DCPS students in the same grades who did not apply to the DC Opportunity Scholarship Program.
Statistical Approach
The performance reporting comparisons will focus on student achievement, both as specified in the statute and because that is the only measure that will be similar for DCPS students and those participating in private schools through the DC Opportunity Scholarship program.16 In order to ensure comparability in student assessment, the evaluation will make every effort to administer the same test to program participants that is used by DCPS. DCPS will provide the evaluation team with test score and background data on public school students.
The analysis will be conducted by comparing the mean test scores of program participants and DCPS nonapplicants, testing for the statistical significance of the difference. To create the most relevant group of DCPS students for comparison, we will draw from the DCPS database the group of non-applicant students who qualify for the program (i.e., eligible for free/reduced-price lunch), stratified by grade level to match our scholarship performance reporting sample. We will present these comparative results as descriptive findings, since the absence of random-assignment to the scholarship or public school conditions would render any causal claims highly speculative.
Reports
The law requires the Secretary to submit to the Congress annual reports resulting from the independent evaluation by December 1 each year and a final report not more than a year after the 5-year program ends. These reports should provide the Congress, other policymakers, the research community, and the public at large with important new information about what happens to students, families, schools, and communities when educational options are expanded for urban low-income families through public policy.
This report is the first in the series of required evaluation reports to Congress. While the focus of the evaluation is on examining the effectiveness of the program, no impact information is available at this point because the initial cohort of program participants—those who applied in spring 2004 to receive scholarships for the 2004-05 school year—just recently matriculated at their new schools. Instead, this report examines the extent of student and school interest in the program and the characteristics of those participating.
Chapter 2 describes the recruitment and application activities that resulted in the submission of 2,692 initial applications for the program, of which 1,848 were deemed eligible. It also describes the characteristics of the group of 58 private schools in the District that agreed to participate in the first year of the program.
Chapter 4 describes the characteristics of program applicants, including their average test scores, family background, parental involvement in their education, type of school they attended previously, and parental assessments of their previous school. It also presents initial data regarding the extent to which DC schools are exposed to greater competition for students as a result of program implementation.
Any evaluation, particularly a randomized control trial (RCT), must take into consideration the number and flow of program applicants and participating institutions. In the case of the DC Opportunity Scholarship Program evaluation, the design was predicated on attracting a sufficient number of applicants to be able to create two sizable and randomly assigned groups of students whose outcomes could be compared: (1) eligible applicants who receive scholarships to attend participating private schools as part of the lottery and (2) eligible applicants who do not receive scholarships by virtue of the lottery.
The recruitment and application period for the program began at the end of March 2004, immediately after a partnership led by the Washington Scholarship Fund (WSF) was selected to implement the DC Opportunity Scholarship Program, and continued throughout the spring and early summer of 2004. The process produced agreements from 58 private schools in the District to participate in the federal program during the first year and applications from 2,692 students seeking scholarships, of whom 1,848 were eligible. This chapter describes in more detail the activities and results of these efforts to recruit schools and students to participate in the program.
An initial task of the WSF-led implementation team was to recruit DC private schools to participate in the program during the first year. The team directly contacted each of the 109 private schools in the District and also worked closely with umbrella organizations that represent groups of schools, such as the Catholic Archdiocese of Washington, the Council for American Private Education, and the Association of Christian Schools International. The team also hosted an information session for private school officials in April. Several representatives of independent private schools at that meeting said that their admissions process for 2004-05 had already concluded and their schools were fully enrolled. However, some of them expressed an interest in participating in the program anyway, and others suggested that they would participate in the future if the application and admissions schedule could be moved to earlier in the school year.
Despite the challenges stemming from the late start of the program, the implementers recruited 58 DC private schools to participate in the program in 2004-05, comprising just over one-half of the private schools in the District (Figure 2-1 and Table 2-1).17
|
Private Schools |
Number of Schools |
Percent |
|
In the District of Columbia |
109 |
100 |
|
Participating in the program in some capacity |
58 |
53 |
|
54 |
50 |
|
4 |
4 |
NOTE: Detail may not sum to totals because of rounding.
SOURCE: "School Directory, D.C. K-12 Scholarship Program, 2004-05 School Year," Washington Scholarship Fund, June 2004.
Fifty-four of the 58 schools made new slots available to scholarship students; the remaining four schools had already closed their admissions and were only willing to accept new students awarded Opportunity Scholarships who already had been accepted to those schools during their regular admissions period. This small subgroup of scholarship winners that we describe as private school "pre-admits" were attending public schools in 2003-04 but obtained acceptance to a private school independent of and prior to the launch of the scholarship program. Some data were obtained regarding 53 of the 58 participating schools.18
Religious Affiliation, Location, and Students Served by Participating Schools
The private schools that chose to participate in the DC Opportunity Scholarship Program in the first year are a diverse group but have some similar characteristics. Most are religiously affiliated and have been long established in the area. About half (27 of the 52 schools that reported their affiliation, or 51 percent) are affiliated with the Catholic Church; 11 (21 percent) are affiliated with a non-Catholic religion; and 15 (28 percent) are independent nonsectarian schools. More than three-quarters of the schools were established before 1983, with a large share of them founded before 1955. The most recently established private school to participate in the program opened in 2002.
The private schools that are participating in the DC Opportunity Scholarship Program are located in every ward of the District. The highest concentration of participating schools, nearly one-quarter of them, are in Ward 4. The remaining participating schools are almost evenly distributed throughout the other seven wards of the city.19
Most of the participating private schools already are serving a high proportion of students of color (Table2-2). On average among the schools, 82 percent of their students in the current year are African American, Latino, Asian, or Native American, compared to 95 percent average minority populations in regular DCPS schools. Sixty percent of the participating private schools have student bodies that are entirely from minority racial/ethnic groups, but a few schools serve fewer than 10 percent of students from those groups.
|
Characteristic |
Average |
Highest |
Lowest |
Valid N1 |
|
Enrollment (number of students) |
||||
|
Private schools |
206** |
1,056 |
12 |
52 |
|
DCPS schools2 |
414 |
1,442 |
127 |
139 |
|
Percentage of students from racial/ |
||||
|
Private schools |
82** |
100 |
8 |
52 |
|
DCPS schools |
95 |
100 |
29 |
139 |
|
Average student-teacher ratio3 |
|
|
||
|
Private schools |
11:1** |
20:1 |
5:1 |
47 |
|
DCPS schools |
14:1 |
21:1 |
7:1 |
133 |
* Statistically significant at the 95
percent confidence level.
** Statistically significant at the 99 percent confidence level.
1. "Valid N"
refers to the number of schools for which information on a particular
characteristic was available.
2. The comparison group of regular DCPS schools excludes
public charter schools, alternative schools, and learning centers.
3. Whenever ranges were given for student-teacher ratio,
the midpoint of the range was selected.
SOURCES: Data on participating private schools drawn from "School Directory, D.C. K-12 Scholarship Program, 2004-05 School Year," Washington Scholarship Fund, June 2004, supplemented by the National Center for Education Statistics, Common Core of Data. Data on DCPS drawn from the web sites of the District of Columbia government and DCPS.
When it comes to the size of overall enrollments or of individual classes, there are substantial differences across schools, although most of these stem from the different grade levels served. Nearly half of the participating private schools enroll students in either kindergarten through eighth grade or pre-K through eighth grade, with the other half enrolling various combinations of elementary-, middle-, and high-school-aged students. It is, therefore, no surprise that enrollments range from 1,000 in a relatively large high school to only 12 students in a very small school. The average enrollment of the participating private schools is about half the average enrollment of regular DCPS schools. The average ratio of students to teachers in the participating private schools is somewhat smaller than the average for DCPS schools. Five of the participating private schools (less than 10 percent) serve students of only one gender.
Although there is no data source that allows a systematic examination of the extent to which the participating private schools are representative of the entire population of private schools in the District, analysis of web sites suggests that there may be some differences. Schools not currently participating in the program appear to have slightly higher enrollments than the participating schools, probably because more nonparticipating schools serve students in the high school grades. Compared to participating schools, nonparticipating schools seem to enroll a substantially smaller share of minorities, have smaller student-teacher ratios, and are much less likely to be religiously affiliated. However, these comparisons should be interpreted with caution, because data were available from only 39 of the 51 nonparticipating schools and were obtained from difference sources and school years.
Programs and Services Available at Participating Schools
The participating private schools offer many but not all of the services and activities that are commonly available in public schools (Table 2-3). An overwhelming majority of participating schools offer computers, art, a library, tutoring, after-school care, music, and before-school care as part of their educational programs. Nearly three-quarters of the participating schools include religious instruction and/or worship as part of their school program, and a similar proportion require that applicants complete an exam to determine grade placement for enrollment. Nearly 12 percent require applicants to pass an entrance exam in order to be deemed admissible to their school. Only 60 percent have a gym and less than half have a school cafeteria. Gifted and talented programs are available in about 16 percent of the participating private schools.
|
Characteristic or Service |
Number of Schools |
Percent |
Valid N1 |
|
Computers available to students |
49 |
96 |
51 |
|
Art |
46 |
92 |
50 |
|
Tutoring |
45 |
88 |
51 |
|
After care |
45 |
87 |
52 |
|
Music |
40 |
85 |
47 |
|
Before care |
40 |
77 |
52 |