Academic performance of subsequent schools and impacts of early interventions: Evidence from a randomized controlled trial in Head Start settings

Abstract

The role of subsequent school contexts in the long-term effects of early childhood interventions has received increasing attention, but has been understudied in the literature. Using data from the Chicago School Readiness Project (CSRP), a cluster-randomized controlled trial conducted in Head Start programs, we investigate whether the intervention had differential effects on academic and behavioral outcomes in kindergarten if children attended high- or low-performing schools subsequent to the preschool intervention year. To address the issue of selection bias, we adopt an innovative method, principal score matching, and control for a set of child, mother, and classroom covariates. We find that exposure to the CSRP intervention in the Head Start year had significant effects on academic and behavioral outcomes in kindergarten for children who subsequently attended high-performing schools, but no significant effects on children attending low-performing schools. Policy implications of the findings are discussed.

Introduction

The early school years, especially from kindergarten to third grade, are a critical transitional period not only for promoting children's scholastic and psychosocial development but also for preventing the dissipating effects of earlier interventions (Reynolds et al., 2006, Reynolds et al., 2010). Research has suggested that the benefits of high-quality early interventions can be sustained in later school years and even into adulthood for participants who attend continuing enrichment programs in the early school years, particularly so for those from low-income families, but that benefits tend to fade out by the second or third year of formal schooling for participants who subsequently attend inferior schools (Currie, 2001, Currie and Thomas, 1995, Currie and Thomas, 2000, Lee and Loeb, 1995, Magnuson et al., 2007, Reynolds et al., 2007, Reynolds et al., 2011, Takanishi and Bogard, 2007). Therefore, in the investigation of long-term effects of early childhood interventions, it is important to take into account the role of participants' subsequent school experiences, especially the quality and performance of the schools they attend. Research as to the mechanisms of how later schools promote or hinder the initial gains fostered by early interventions is likely to have important policy implications for the design and improvement of effective interventions targeting disadvantaged children.

However, largely due to the lack of data on the quality or performance of later schools, few empirical studies have examined directly their role in the long-term effects of early childhood interventions. When the data do exist, selection bias remains an important issue. On the one hand, economically disadvantaged children such as Head Start participants tend to attend low-quality or low-performing schools compared to their counterparts, which could undermine their earlier gains (Currie and Thomas, 1995, Currie and Thomas, 2000, Hastings and Weinstein, 2008, Lee and Loeb, 1995, Pigott and Israel, 2005). On the other hand, children's enrollment in higher versus lower performing schools may be endogenous to, or affected by, exposure to an initial treatment such as preschool interventions (Hong & Raudenbush, 2008). Evidence shows that enrollment in schools of different quality is associated with children's later outcomes (Hastings & Weinstein, 2008). As such, analyses that do not consider the role of subsequent schools may result in biased estimates of the long-term effects of early childhood interventions.

We use data from a cluster-randomized controlled trial conducted in Head Start programs, the Chicago School Readiness Project (CSRP), to investigate whether the academic performance of subsequent schools mattered in sustaining the effects of the CSRP intervention on children's academic and behavioral outcomes in kindergarten. As detailed below, in the analyses we employ a principal score matching method to address the issue of selection bias.

Studies conducted over the last two decades and beyond have consistently shown that the benefits of many high-quality early interventions, especially cognitive gains from programs targeting economically disadvantaged children such as Head Start, tend to dissipate after only a few years of formal schooling (see research and reviews by Barnett, 1995, Currie and Thomas, 1995, Currie and Thomas, 2000, Lee and Loeb, 1995, Magnuson et al., 2007, U.S. Department of Health and Human Services (USDHHS), Administration for Children and Families, 2010). A common explanation for the fade-out of initial gains from early interventions is the low-quality of schools that participants subsequently attend (Currie, 2001, Currie and Thomas, 2000, Lee and Loeb, 1995, Magnuson et al., 2007). Research has suggested that children in economically disadvantaged families who attended high-quality preschool programs such as Head Start were systematically more likely to attend low-quality and low-performing schools compared to their counterparts, which may be due to residential proximity, parental expectations of low return to education, and budget constraints in educational spending (Currie and Thomas, 1995, Currie and Thomas, 2000, Hastings and Weinstein, 2008, Lee and Loeb, 1995, Pigott and Israel, 2005). For example, evidence from the Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), a nationally representative sample, shows that compared to other children, Head Start participants tended to attend schools with lower average levels of socio-economic status, lower achievement in math and reading, more minority children, and more children eligible for free lunch (Pigott & Israel, 2005).

From the perspectives of life cycle skill formation and human capital accumulation, as well as developmental cascades theory, continuing and enriching school environments can facilitate the ongoing skill acquisition and school achievement of children with skill advantages gained from high-quality early interventions, and can also compensate for the skill deficits experienced by at-risk children at school entry (Cunha et al., 2006, Hamre and Pianta, 2005, Magnuson et al., 2007, Masten and Cicchetti, 2010). As a result, high-performing schools that children attend subsequent to preschool interventions may be able to reinforce their initial gains and make early interventions more effective (Cunha et al., 2006). For example, recent results from the Child-Parent Center (CPC) Early Education Program, which included services for low-income children from age 3 to third grade, demonstrated that continuing intervention strengthened learning gains from preschool and was independently associated with school performance leading to adult well-being (Reynolds et al., 2011). In contrast, low-quality learning environments and unchallenging programs may undermine children's earlier gains and the advantages fostered by initial interventions (Currie and Thomas, 2000, Lee and Loeb, 1995).

In these and some other studies on the role of schools, school academic performance has typically been indexed by the percentage of students meeting or exceeding performance benchmarks in standardized tests of math or reading (Currie and Thomas, 2000, Hanushek, 1986, Lee and Loeb, 1995). Families as well as educators recognize the importance of test-score-based ratings of school quality with academic performance of schools serving as one of the most important factors in parents' school choices (Schneider et al., 1998, Weidner and Herrington, 2000). Evidence from experimental studies shows that parents in low-income families were more likely to choose higher-performing schools if they received direct information on school test scores (Hastings & Weinstein, 2008). Other studies have shown that when low school performance information was publicly disseminated, students, especially those from low-income neighborhoods, were more likely to switch their schools (Friesen et al., 2011, Howell, 2006). Moreover, high-performing schools appear to make significant differences in the experiences and outcomes of low-income children. For example, high-performing schools, especially those in low-income neighborhoods, have been associated with a more caring and nurturing environment characterized by high expectations for staff and students, effective leadership, committed teachers, and a strong focus on academics, instruction, and student learning (Carter, 2000, Kannapel and Clements, 2005, McGee, 2004, Wolf and Hoople, 2006). Experimental studies found that attending high-performing schools significantly increased low-income students' test scores (Hastings & Weinstein, 2008).

Although the role of school academic performance in the long-term effects of early interventions has received increasing attention recently, few empirical studies have been conducted to investigate their role directly. One common challenge is that the chance of attending higher- versus lower-performing schools may differ for participants and non-participants, which may, in turn, contribute to the long-term outcomes of interest. As described above, the probability of children's subsequent enrollment in higher- versus lower-performing schools may be systematically different for economically disadvantaged children compared to their counterparts, and importantly, may also be a function of the initial intervention. For example, the cognitive and social-emotional benefits that participants gain from high-quality preschool interventions may motivate their parents to pursue better schools. In the CSRP, children in the intervention group had significantly better developmental outcomes than those in the control group at the end of the 9-month intervention during the Head Start year, including improvement in self-regulation and academic skills as well as reductions in emotional and behavioral difficulties (Raver et al., 2009, Raver et al., 2011). In addition, preliminary evidence suggests that children in the CSRP intervention group were significantly more likely to enroll in high-performing schools in kindergarten than children in the control group, even after adjusting for child and mother covariates as well as teacher and classroom characteristics in Head Start year (Zhai & Raver, 2010). Taken together, these results suggest that we run the risk of incorporating considerable bias into our estimates of the long-term effects of the initial intervention unless we also take the performance of those subsequent schools into account in our models. The present study therefore takes careful steps to minimize the risk of this bias, as we outline below.

In this study, we use data from the CSRP to investigate whether exposure to the intervention during the Head Start year had differential effects on children's academic (i.e., language, literacy, and math) and behavioral outcomes (i.e., internalizing and externalizing behavior problems) in kindergarten if children subsequently attended either high- or low-performing schools. To do this we conduct separate analyses for children who subsequently attended high-performing schools and for those who were enrolled in low-performing schools, and then examine whether the CSRP intervention effects were different between these two groups of children.

Building on prior research (Currie and Thomas, 2000, Hanushek, 1986, Lee and Loeb, 1995), we define low- and high-performing schools based on school-level aggregates of students' standardized test scores. As detailed below, we adopt a principal score matching method to address the issue of selection bias and control for a set of child, mother, and classroom covariates.