ECE quality indicators and child outcomes: Analyses of six large child care studies

Highlights

• Developmental theory suggests that children develop within complex systems, but most research on ECE focuses on discrete aspects of quality.

• This study uses a framework common in ECE policy to examine the association between ECE quality defined by multiple indicators and child outcomes.

• Replicated analyses were conducted with six large studies of ECE quality to relate structural and quality indicators to child outcomes.

• Small gains in child outcomes were observed related to quality of teacher–child interactions, curriculum, and teacher and director education.

• Combining quality indicators into a single index also predicted gains in preschoolers’ language and literacy scores, but effect sizes were small.

Abstract

Practice and policy in early childhood education rely heavily on professional expertise and guidelines developed from research on early care and education (ECE) quality, but these guidelines have not been extensively researched. Data from six large studies of ECE quality were analyzed to relate structural and process quality indicators based on professional guidelines to children’s language, literacy, math, and social outcomes. Results indicated small gains related to some quality indicators (e.g., quality of teacher–child interactions, curriculum, and teacher and director education) for some of the preschoolers’ outcomes, but not other indicators (e.g., global quality, group size). Combining quality indicators into a single index also predicted gains in preschoolers’ language and literacy scores, but effect sizes for the quality rating were smaller than for individual indicators.

Introduction

Early care and education (ECE) is viewed as one of the most promising means to increase opportunities for all children (Duncan and Magnuson, 2013, Heckman, 2011, Yoshikawa et al., 2013). Based on evidence from past literature that high quality ECE promotes children’s early learning and development, federal, state, and local governments in the U.S. have invested heavily in ECE programs and have greatly increased efforts to monitor the quality of these programs (Burchinal, Magnuson, Powell, & Soliday Hong, 2015). Federal and state policies typically regulate the characteristics of ECE classroom and center quality based on past research that defines what constitutes high quality. Empirical research to identify which of these aspects of ECE quality show consistent beneficial associations with children’s learning and development suggest that some of the widely accepted quality indicators show modest or no associations with child outcomes beyond minimum regulations (Mashburn et al., 2008; Sabol, Soliday Hong, Pianta, & Burchinal, 2013; Yoshikawa et al., 2013). This study was designed to further examine this issue, seeking to identify which aspects of ECE quality are consistently associated with preschool children’s developmental outcomes in center-based settings, and whether those associations persist when ECE quality variables are translated into the professional guidelines and summarized for use in practice.

Two major federal and state funded early childhood education programs – Head Start and state-funded public pre-kindergarten programs – serve over 2.3 million children each year and receive over $14 billion dollars in investment, representing an increase in over $4 billion from just one decade ago (Barnett, Hustedt, Robin, & Schulman, 2005; Barnett, Carolan, Squires, Clarke Brown, & Horowitz, 2015). Head Start has quality standards that apply to all Head Start programs, whereas each state and local public pre-kindergarten program has developed its own quality standards (Burchinal et al., 2015). Recently, states have been relying on a market-based policy to motivate providers to provide higher quality ECE. This effort, brought together under the Quality Rating and Improvement Systems (QRIS) umbrella, has been implemented in all or part of 38 states, and involves developing local quality standards (Build Initiative & Child Trends, 2014). All of these policy initiatives define quality standards that are regulated and monitored. Furthermore, professional organizations have issued guidelines and recommendations in an effort to translate research for policy-purposes, which have formed the foundation for regulating ECE quality in the past. These ECE efforts engaged in an important goal – to define program standards at levels of quality that show consistent associations with child outcomes – but further examination of the standards is needed because most program standards have limited to no validation of the standards themselves.

The accumulated scientific knowledge base of developmental psychology provides the foundation for which aspects of ECE quality are important for young children (Weinberg, 1979). Underlying the main premise of program standards is the belief, based on Attachment Theory, that sensitive and responsive interactions with adults provide a secure base that enables young children to feel safe and supported enough to explore their environment and develop relationships (Bowlby, 1969, Bowlby, 1982). As relationships with children and their teachers are co-constructed, children who experience their teacher as warm, responsive, and predictable develop a sense of trust which allows them seek proximity to their teacher, to welcome interactions where language exchanges and the construction of knowledge can occur, and are more likely to ask for help in solving problems (Howes and Spieker, 2008, Stuhlman and Pianta, 2002). However, when interactions are negative, punitive, or unpredictable children may become overly dependent on their teacher in an attempt to increase their sense of safety and connection, or engage in avoidant behaviors leaving them disengaged from the instructional context (Howes, Sidle Fuligni, Soliday Hong, Huang, & Lara-Cinisimo, 2013). Most ECE program standards include a measure of observed teacher warmth, sensitivity, and responsiveness toward children to ensure that children have access to interactions with adults that support their basic sense of safety and security from an attachment perspective.

Other aspects of teacher–child interactions and the classroom and center-level contexts are also thought to contribute to children’s learning. Founded on learning theories, like Piaget’s constructivist theory, program standards have been established to ensure that ECE classrooms provide active engagement in developmentally appropriate activities within enriched environments for children that allow them to construct their knowledge about the world (Gopnik, Meltzoff, & Kuhl, 1999). The degree to which children’s learning experiences are self-selected or designed and structured by an adult were also highlighted in the literature on Developmentally Appropriate Practice (DAP; Copple & Bredekamp, 2009; Fleer, 1995). The DAP literature highlighted the importance of children’s ability to co-construct their knowledge, and as such, many program standards and state policies incorporated requirements that ECE classroom environments be designed to promote children’s choice of learning activities through learning centers. Furthermore, sociocultural theory suggests that teacher–child interactions should be characterized by active engagement with a caregiver through the provision of scaffolded instruction designed to promote children’s learning (Bodrova & Leong, 2006; Vygotsky, 1994). These theories lead to greater attention to types of activities available to young children and the intentionality behind teacher–child interactions. Reflecting these beliefs about how children learn has led to program standards that have included requirements about the quantity and type of materials available to children in an arrangement that facilitates their ability to actively explore and co-create learning.

Connections between ECE teachers and programs and children’s families also play a role in supporting children’s developmental outcomes. Socio-ecological and transactional frameworks describe how developmental processes are shaped by bidirectional interactions between children and their environment (Bronfenbrenner, 1979; Bronfenbrenner & Morris, 1998; Sameroff, 1983, 1993; Sameroff & Chandler, 1975). These models focus on multiple systems, ranging from the proximal family system to the more distal influences of the community and society. According to this model, the quantity and quality of interactions between the young child and his/her primary caregivers, including the ECE providers, play a crucial role in early development, and provide the foundation for program standards related to aspects of the context at the program-level. As ECE programs are systems comprised of individuals at the child, family, teaching, and leadership levels, program standards have also considered the degree to which quality can be considered at the program level.

While there may be unique associations between individual measures of quality and specific outcomes, the degree to which any one quality indicator may be able to function is related to the quality of the other indicators in the environment. Educational production functions provide a paradigm to consider how individual, family, school, and other inputs integrate to produce an educational function with child outcomes as the output (Cohn & Millman, 1975; Hanushek, 1972). Since quality is expensive, parents and policy-makers may select different combinations of quality. Furthermore, production theory suggests that only the inclusion of indicators that define quality and the specific intended child outcomes should be retained in the quality function to the exclusion of related, but not predictive quality indicators.

Drawing from these theories, developmental research has documented that high-quality interactions with teachers within safe ECE environments and access to stimulating, age-appropriate activities are associated with long term impacts on child outcomes (see Burchinal et al., 2015 for review). These quality dimensions have been labeled “process” quality because they are believed to be the process through which ECE has its impact on child outcomes. Structural features, such as class size and curriculum, are posited to set the stage for warmer and more responsive interactions, and more scaffolding, and the provision of developmentally appropriate learning environments. There is an extensive literature that has examined both individual process and structural quality measures, and these findings have been translated into professional guidelines to ensure and improve the access that children have to high quality, stimulating environments and interactions. Some of the major findings used to support these professional guidelines are described below.

Many large multi-site and small single-site observational studies have examined the association between process quality and child outcomes, using a variety of measures of quality and child outcomes. Typically, these studies find that continuous measures of preschool process quality within individual classrooms are modestly related to language, academic, and social outcomes for children within those classrooms. Global measures of preschool classroom environmental quality (Early Childhood Environmental Rating Scale-Revised; ECERS-R, Harms, Clifford, & Cryer, 1998) have been related to gains in language, cognitive, and social skills (e.g., Howes et al., 2008, Peisner-Feinberg and Burchinal, 1997, Mashburn et al., 2008), although with some recent evidence suggesting smaller or nonsignificant effects (Gordon, Fujimoto, Kaestner, Korenman, & Abner, 2013; Sabol & Pianta, 2015). Ratings of caregiver sensitivity (Observation Record of the Caregiver Environment; ORCE; NICHD Early Child Care Research Network, 1999) in the large multi-site NICHD Study of Early Child Care and Youth Development, were significantly associated, albeit modestly, with concurrent language and academic skills for preschoolers (NICHD Early Child Care Research Network & Duncan, 2003). Observations of teacher–child interactions (Classroom Assessment Scoring System; CLASS; Pianta, LaParo, & Hamre, 2008) have been modestly related to children’s development over the preschool year and these results have replicated across studies (e.g., Howes et al., 2008, Mashburn et al., 2008, Weiland and Yoshikawa, 2013), with some evidence of differential prediction. The warmth and positivity of the classroom environment is related to gains in social skills, whereas the quality and complexity of the language and the cognitive stimulation in the classroom environment are related to gains in academic and language skills in the NCEDL pre-k study (e.g., Burchinal, Vandergrift, Pianta, & Mashburn, 2010; Mashburn et al., 2008). Finally, the productivity and management of the classroom has predicted gains in social skills (Jones, Bub, & Raver, 2013). During the past decade, meta-analyses of observational studies (Burchinal, Kainz, & Cai, 2011; Keys et al., 2013) have also concluded that consistent, but quite modest, linear associations exist between process quality and child outcomes, with a one standard deviation increase in quality scores yielding increases of .04 to .17 in standard units of child outcomes. Stronger associations were found for language and academic outcomes than for social and emotional development across all ages.

Other factors are thought to be important because they are necessary, but not sufficient, for teachers to be able to provide high process quality (e.g., NICHD Early Child Care Research Network, 2002). They include a wide range of factors including characteristics of the care providers, such as their education, certification, experience, and professional development; of the setting such the number of children (group size), the ratio of children in the classroom or family child care home to care providers, and health and safety practices; of the instruction, such as curriculum; and of the administration, such as the director’s education and training, wages and benefits provided to staff, and time for planning and meeting.

There was a greater focus on many aspects of structural quality earlier in the ECE research literature. Past research documented moderate to strong associations with global environmental quality and modest associations with child outcomes for continuous measures of classroom quality such as teacher education (Burchinal et al., 2000; Phillipsen, Burchinal, Howes, & Cryer, 1997; NICHD Early Child Care Research Network, 1999, NICHD Early Child Care Research Network, 2002), teacher–child ratios and group size (McCartney et al., 2010, NICHD Early Child Care Research Network, 1999, NICHD Early Child Care Research Network, 2002, Phillipsen et al., 1997). More recently, a growing research literature indicates that the use of evidence-based curricula, combined with aligned training or coaching, can relate to substantial gains in children’s literacy (e.g., Bierman et al., 2008, Neuman and Cunningham, 2009), math (e.g., Clements & Sarama, 2008), and social skills (Bierman et al., 2008) and with more focused, sequenced curricula producing larger gains in child outcomes (Weiland & Yoshikawa, 2013). It is worthy of note that few of these studies have examined the relative efficacy of one curriculum versus another, and although these results are promising more work needs to be conducted in this area.

In an effort to apply developmental theory and its resulting research, beliefs about what children need, and extensive ECE research findings led professional organizations to issue guidelines regarding ECE quality features (e.g., American Academy of Pediatrics, American Public Health Association, National Resource Center for Health and Safety in Child Care and Early Education). All of the guidelines defined standards for determining whether that quality feature was conceptually or theoretically rated as high, adequate, or less than adequate, with standards provided for characteristics, such as teacher/caregiver education, group size, and child:adult ratios that vary according to the child’s age (American Academy of Pediatrics, American Public Health Association, & National Resource Center for Health and Safety in Child Care and Early Education, 2011; National Association for the Education of Young Children, 2015), and global environmental quality (e.g., Harms et al., 1998; Improving Head Start for School Readiness Act, 2007). Although these standards are widely viewed as defining high quality, the level of evidence for ECE quality constructs vary widely and the level of evidence for quality standards is even more limited.

Professional guidelines for process quality measures have been provided by the instrument developers or through research and practice. The professional guidelines for the ECERS and CLASS divide the scales into low, medium, and high quality ranges (Burchinal et al., 2010), with some evidence supporting larger gains in higher quality classrooms based on these guidelines (Burchinal et al., 2000, Burchinal et al., 2010; Burchinal, Zaslow, & Tarullo, 2016; Hatfield, Burchinal, Pianta, & Sideris, 2016; Weiland & Yoshikawa, 2013), and larger, albeit small to moderate, effect sizes (d < .20) in high-quality than in low-quality classrooms.

Professional guidelines have been developed for most of these structural quality variables (National Association for the Education of Young Children, 2015), and related to child outcomes (Barnett et al., 2015; NICHD Early Child Care Research Network, 2000). However, positive associations have not always been observed between structural quality characteristics and children’s learning. For example, professional guidelines were not related to child outcomes when guidelines regarding teacher education were examined in a comprehensive analysis of seven child care studies (Early et al., 2007) or when multiple benchmarks were examined in analyses of a large pre-kindergarten study either individually (Mashburn et al., 2008) or jointly (Sabol et al., 2013). Also unsuccessful attempts have been made to relate child outcomes and QRIS composites of benchmark ratings across multiple ECE quality dimensions (Hestenes et al., 2014; Sabol & Pianta, 2015; Sabol et al., 2013; Soliday Hong, Howes, Marcella, Zucker, & Huang, 2014; Thornburg, Mayfield, Hawks, & Fuger, 2009; Zellman, Perlman, Le, & Setodji, 2008).

Publicly funded ECE programs such as Head Start and state-funded Pre-K also used quality standards to create quality standards based on quality benchmarks or professional guidelines. Head Start has 128 performance standards that reflect the whole-child orientation, including structural and process ECE quality, governance, and fiscal management (U.S. DHHS, ACF, & Office of Head Start, 2015). State pre-kindergarten centers and schools vary widely in terms of the structural and process ECE quality indicators that are regulated within their systems (Barnett et al., 2015). For example, out of 53 state-funded Pre-K systems reviewed as part of the NIEER 2014 State Pre-K Yearbook, all systems included state early learning standards, 30 required a bachelor’s degree, 45 required a group size of 20 or fewer and 46 required a teacher:child ratio of 1:10 or lower, but only 18 required at least a CDA for assistant teachers.

Two of the major goals of QRIS are to meaningfully distinguish between differing levels of quality and to improve the school readiness skills of young children. Some recent state QRIS validation studies have shown associations between quality rating tiers and measures of process quality (e.g., Lipscomb, Weber, Green, & Patterson, 2016; Quick et al., 2016; Tout et al., 2016). However, little evidence of associations between aggregate quality ratings in specific states and child outcomes (Hestenes et al., 2014; Karoly, Schwartz, Setodji, & Haas, 2016; Sabol & Pianta, 2015; Soliday Hong et al., 2014; Thornburg et al., 2009; Zellman et al., 2008) or between simulated QRIS ratings based on selected states’ established systems and child outcomes (Sabol et al., 2013).

Several methodological issues deserve further consideration when evaluating the extent to which ECE quality indicators relate to child outcomes. First, most of the research on ECE quality has focused on classroom assessments of quality, but the monitoring of quality in practice is typically assessed at the center level or program level. For example, monitoring of ECE quality within a QRIS or a Pre-K is evaluated at the center level and within Head Start at the grantee level. Second, many of the studies examining quality benchmarks have examined them jointly. At least some of the QRIS validation studies that examined the benchmarks jointly in their QRIS rating also involved decisions that may have obscured associations such as using local expertise rather than professional guidelines to develop their quality indicators and scoring based on the lowest score across all benchmarks (Sabol et al., 2013). Third, many of the older studies would not meet today’s standards for reducing selection bias (see NICHD Early Child Care Research Network & Duncan, 2003 for early discussion of this issue), and thus, those findings might not replicate if today’s more rigorous standards were applied (e.g., at a minimum, include the child’s initial skill levels as a covariate; see Keys et al., 2013 for full discussion). Fourth, many of the studies that examined the quality benchmarks, regardless of whether they are based on professional guidelines, included small samples of convenience so further examination with larger, more diverse samples is likely warranted. Finally, associations between child outcomes and both the continuous quality variable and the translated quality indicator need to estimated using the same data to be able evaluate the extent to which applying the professional guidelines provides similar patterns of association. To date, most work has either examined the continuous quality variable or the quality benchmarks, but not both.

Due to the lack of research on ECE program standards and their wide use for ensuring better outcomes for children, we need to examine the extent to which program standards and combinations of program standards relate to child outcomes. Some program monitoring is conducted using individual program standards, for example, in program monitoring standards and QRIS ratings based on block systems that restrict aggregate quality ratings to the lowest quality indicator score. Furthermore, combinations of quality indicators should be examined when they are summed to create quality indices (e.g., State of Preschool Ratings published by the National Institute for Early Education Research and QRIS ratings calculated by adding up scores across indicators). This study will also update research previously published using covariate adjusted regression models without adequately accounting for concerns regarding family selection bias and variability among ECE setting with respect to quality standards, sources of funding, and diversity in the children and families served. To address these concerns, we conducted analyses across six large childcare studies to examine the extent to which benchmarks or professional guidelines applied to widely used measures of structural and process ECE center quality relate to child outcomes during the preschool year in center-based settings.