After-school tutoring in the context of no Child Left Behind: Effectiveness of two programs in the Pittsburgh Public Schools

https://doi.org/10.1016/j.econedurev.2009.02.005Get rights and content

Abstract

The No Child Left Behind (NCLB) legislation has created pressure for districts to improve their students’ proficiency levels on state tests. Districts that fail to meet their academic targets for 3 years must use their Title I funds to pay for supplemental education services (SES) that provide tutoring or other academic instruction. Many districts, including the Pittsburgh Public Schools (PPS), have also adopted additional tutoring programs designed to help students reach proficiency goals. This paper examines student participation and achievement in two PPS tutoring programs—the NCLB-mandated SES program and a state-developed tutoring program. We examine the characteristics of students participating in each program, the effects of participation on student achievement, and the program features that are associated with improved achievement.

Introduction

In recent decades, the availability of programs that provide academic tutoring services to children outside of school hours has grown tremendously. In the past, many of these programs were provided independently of the district, state, or federal government through non- and for-profit organizations (Bodilly and Beckett, 2005). These providers often did not coordinate their efforts with the school district and tried to attract students through their own means. However, in recent years, the world of out-of-school programs has been transformed through federal, state, and local legislation and programs—most notably through the No Child Left Behind Act (NCLB), which requires districts to provide funding for supplemental educational services (SES) to students in schools that fail to meet academic targets. Pennsylvania, like many states, has also enacted its own set of out-of-school programs, including a program known as the educational assistance program (EAP) which is intended to raise proficiency levels on state achievement tests.

In the 2004–2005 school year, Pittsburgh Public School (PPS) district began implementing these two new programs. Up to that point, out-of-school programs in Pittsburgh were not, for the most part, run by the district. Instead, they were managed and run by community-based organization such as churches, libraries, museums, recreation centers, and small non-profits. Many of these programs incorporated both academic and enrichment activities as part of their design. While these programs continue to play a strong role in the out-of-school time for students, SES and EAP programs have introduced new alternatives focused more heavily on academic activities such as tutoring.

As the role of the government has increased, so too has the need for greater level of scrutiny and accountability. Funders, district officials, and program planners have expressed a desire to know whether or not these programs are meeting their goals. In addition, the use of out-of-school programs is a strategy many urban districts are using to address the poor performance of their students, especially minority students. In the analysis, we examine the participation rates and the effects of these two programs on student achievement in PPS. While this is an evaluation of only one district and inferences may be restricted to this district, the analysis goes beyond previous analyses that have ignored the possibility that students participate in multiple out-of-school programs. This paper illustrates the need to avoid evaluating out-of-school programs in isolation, but to evaluate them as a collection of programs that may work together to improve student achievement. In the next section, we describe these programs in greater detail. In later sections, we describe our research approach, present the results, and draw conclusions and implications.

A key aim of the 2001 federal NCLB Act is to give parents of students in low-performing Title I schools new educational options, including the use of supplemental educational services (SES). Districts are required by federal law to use a portion of Title I funding to provide SES as an option for low-income students (as generally defined by eligibility for free or reduced-price lunch) attending schools that have failed to meet state goals for “adequate yearly progress” for three consecutive years.2

In PPS, SES were first offered to students in the 2004–2005 school year. These services are free to parents and students, must be provided outside the regular school day (after school or during the summer), and are intended to provide tutoring geared toward raising students’ proficiency in mathematics and reading. A variety of agencies may provide these services, including for-profit and non-profit entities, faith-based organizations, public or private schools, and school districts (provided the district is eligible to provide the services, which PPS was not until the 2006–2007 school year). Parents are permitted to select an SES provider from a state-approved provider list. Table 1 lists the SES providers in PPS in the 2005–2006 school year and the number of students they served. The table indicates that most SES organizations provided services to very few students and over 60 percent of the students received services from Huntington Learning Centers.

In 2003–2004, Pennsylvania established the educational assistance program (EAP), which provides extended learning opportunities for students by funding evidence-based tutoring3 for 82 of the state's most academically challenged school districts. Tutors are hired and managed by the districts and the services can be provided before school, after school, weekends, and/or during the summer. PPS's EAP tutoring sessions occur within the schools the students attend and the tutors are a combination of school teachers within the schools, retired teachers, and college students. In total, school districts must provide a minimum of 45 h per year of tutoring instruction per content area and required to begin the tutoring sessions no later than October 1 of the school year.

The EAP program is designed to target students who score below proficient on the state accountability test in tested grades or who score below a set score (which is approved by the Department of Education) on a district-administered test. However, some districts, including PPS, have expanded the services to a broader set of students beyond those who fail to meet test score thresholds and allowed any student interested in participating in the program to participate. Initially, the state only allowed students up to 9th grade to participate in the program, but in the 2005–2006 school year, the program was expanded to all students. Although the EAP program is not mandated by NCLB, it serves an important role in the state's efforts to help schools meet their NCLB targets. Both the SES and EAP programs represent a relatively new approach to the provision of after-school services, one that is explicitly intended to be aligned with the state's standards-based accountability provisions.

Section snippets

Previous literature

Because the world of out-of-school programs is generally informal and not highly monitored, only a fraction of all programs has been formally evaluated (Fashola, 1998, Roth et al., 1998, Bodilly and Beckett, 2005, Raley et al., 2005). In addition, because these programs are voluntary and because students who choose to participate in them may be quite different from the students who do not, it is challenging to accurately assess the programs’ effectiveness. Generally, previous research has not

Research questions

In this paper, we examine the effectiveness of the SES and EAP programs in PPS. Nearly 600 students received SES services and 6000 students received EAP services in the 2005–2006 school year. Our analysis not only examines the overall academic effectiveness of these programs, but also examines program factors that are associated with gains in student achievement. To provide greater context for the achievement analysis, we also examine characteristics of students participating in the programs.

Descriptive patterns of student participation

One of the key objectives of out-of-school programs is to improve student learning, which can only happen through participation. Therefore, our analysis first examines the participation patterns of students in the district's SES and EAP programs. Below, we display the overall participation rates by year, which indicate that these programs have generally been under utilized.

Achievement analysis

Ultimately, we are interested in the effect student participation in the SES and EAP programs have on student achievement. To estimate these effects, the counterfactual must be estimated. In other words, we must explore how students exposed to these programs would have performed in the absence of those options. Estimating an appropriate counterfactual is challenging, because participating students may be different in unobservable ways from non-participating students. Indeed, the mere fact that

Analytic details

As noted previously, outcome indicators for the achievement analyses are rank-based math and reading z-scores. The formal model to examine the impact of the programs on the outcomes is specified in Eq. (1).6 To examine achievement effects, we use achievement gains (Ajt  Ajt−1) as the outcome measure to guard against differences in achievement

Results

We present the results across the models in Table 6. For students participating in SES, the estimates are consistently positive and significant for math across the models. Only in the fixed-effect gains model in which we dropped student test scores prior to participation is the value not statistically significant, but has a coefficient estimate similar to the rest of the models. The lack of significance can be explained primarily by a reduced sample size. The effect sizes range from 0.14 to

Conclusions

In recent years, both state and federal governments have taken an increasingly larger role in out-of-school services. State-funded programs like EAP provide learning opportunities for students by funding tutoring, and federal Title I programs provide supplemental educational services (SES) to low-income students attending schools that miss school-wide academic targets. Both programs have been operating in Pittsburgh since the 2004–2005 school year, and they represent a significant change in the

References (29)

  • C. Scott-Little et al.

    Evaluations of after-school programs: A meta-evaluation of methodologies and narrative synthesis of findings

    American Journal of Evaluation

    (2002)
  • R.W. Zimmer et al.

    Charter school performance in two large urban districts

    Journal of Urban Economics

    (2006)
  • M. Arellano et al.

    Some tests of specification for panel data

    Review of Economic Studies

    (1991)
  • S. Bodilly et al.

    Making out-of-school time matter: Evidence for an action agenda

    (2005)
  • P. Burch

    Supplemental educational services under NCLB: Emerging evidence and policy issues

    (2007)
  • Chicago Public Schools, Office of Research, Evaluation and Accountability. Department of Evaluation and Data Analysis

    SES tutoring programs: An evaluation of the second year-part one of a two-part report

    (2005)
  • Chicago Public Schools, Office of Research, Evaluation and Accountability. Department of Evaluation and Data Analysis

    SES tutoring programs: An evaluation of year 3 in the Chicago Public Schools

    (2007)
  • O.S. Fashola

    Review of extended-day and after-school programs and their effectiveness

    (1998)
  • B. Gill et al.

    Inspiration, perspiration, and time: Operations and achievement in Edison Schools. RAND (MG-351-EDU)

    (2005)
  • GAO (Government Accountability Office) (2006). No Child Left Behind Act: Educational Actions Needed to Improve Local...
  • L. Hamilton

    Assessment as a policy tool

    Review of Research in Education

    (2003)
  • Hanushek, E. A., Kain, J. F., Rivkin, S. G., & Branch, G.F. (2005). Charter school quality and parental decision making...
  • Heinrich, C. J., Meyer, R. H., & Whitten, G. (2008) Supplemental education services under no child left behind: Who...
  • D. Heistad

    Analysis of 2005 supplemental educational services in Minneapolis public schools: An application of matched sample statistical design

    (2005)
  • Cited by (0)

    1

    Fax: +1 412 683 2800.

    View full text