Intervention fidelity of Getting Ready for School: Associations with classroom and teacher characteristics and preschooler’s school readiness skills

Highlights

• This study examined the intervention fidelity of Getting Ready for School (GRS).

• Classroom quality was associated with adherence to GRS lesson plan.

• Dosage and adherence were associated with growth in literacy, math, and self-regulation across the preschool year.

Abstract

Getting Ready for School (GRS) is a new school readiness intervention for teachers and parents, designed to help children develop early literacy, math, and self-regulation skills. GRS was implemented in 19 Head Start classrooms. In the present study we examined variability in different aspects of intervention fidelity including dosage, adherence, and child engagement. In addition, we studied the association among classroom, teacher and student characteristics and fidelity, and whether measures of fidelity were associated with children’s growth in math, early literacy, and self-regulation skills across the preschool year. Findings indicate that on average teachers reported completing almost 80% of the activities assigned, and that they were observed to adhere fairly well to the lessons. Child engagement was observed to be moderate to high across classrooms. Classroom quality, as measured by the CLASS, and age of children were positively associated with adherence. Teachers that had participated in GRS for two years were more likely to complete more activities. Different components of fidelity were associated with child outcomes. Percentage completion of math and literacy activities were positively associated with growth in math and literacy skills. Children in classrooms in which teachers adhered more faithfully to the curriculum made significantly greater gains in literacy, math, and self-regulation skills. Child engagement was positively associated with a measure of executive function. Results highlight the importance of examining implementation fidelity. Implications for preschool teachers are discussed.

Introduction

School readiness—the combination of skills that includes preacademic knowledge, such as basic literacy and math foundations, as well as self-regulatory skills, such as social–emotional regulation, patience, and focus—has become a critical component of Early Childhood Education (ECE) programs in the United States. Decades of research has shown that socioeconomically disadvantaged children tend to enter school behind their peers in terms of these skills (Bradley & Corwyn, 2002; Noble, Norman, & Farah, 2005; Noble et al., 2012), creating the foundation for an achievement gap that only grows over time (Reardon, 2011; West, Denton, & Germino-Hausken, 2001). Notably, a growing body of research suggests that the quality of ECE programs is a critical element in boosting school readiness (Duncan, 2003), yet “high-quality” is difficult to define and even more difficult to achieve in the resource-limited programs that serve the children who need the boost the most (Kagan, 2009). As an increasing number of states and cities expand their early childhood services, emphasis is placed not only on the number of programs and seats available to families, but also on their quality and success in helping children be prepared for school.

Recent studies have shown that returns on investment in ECE may only be significant for high-quality programs (Duncan & Magnuson, 2013), certain components of which (including class size reduction and teacher credentials) are difficult and expensive to infuse into existing programs. Thus, numerous interventions in recent years have focused on targeting specific school readiness skills, in the hopes that building teacher practice in terms of expanding these skills will be associated with increasing quality more generally (Clements & Sarama, 2013; Clements, Sarama, Wolfe, & Spitler, 2013; Wilson & Farran, 2012). This method seems to be particularly effective for relatively low-cost, easy-to-implement interventions.

Even so, it is only more recently becoming common for interventions that take this approach to discuss intervention fidelity when reporting outcomes in early childhood interventions (Clements, Sarama, Wolfe, & Spitler, 2015; Mendive, Weiland, Yoshikawa, & Snow, 2016). Intervention fidelity (IF) refers to the degree to which intervention practices and core components are implemented as intended; herein, the degree to which teachers implement activities assigned by coaches and adhere to the lesson plans is discussed. Without an understanding of the feasibility and fidelity of a new intervention, it is impossible for educators and policymakers to know whether or not an intervention could be brought to scale. In this paper, we will present IF findings from a novel school readiness intervention, Getting Ready for School, as well as associations among IF, classroom- and teacher-level variables, and child outcomes.

Getting Ready for School (GRS) is an intervention that aims to simultaneously target three main school readiness areas: literacy, math, and self-regulation. Based on the principle that children learn best when receiving high-quality, reinforced learning opportunities both at home and in school settings (Bronfenbrenner, 2005), GRS is a flexible, supplemental curriculum for teachers and parents, designed to help children develop the early literacy, math, and self-regulation skills they need to start primary school ready and eager to learn.

Initially designed as a math and literacy curriculum for parents in Central and Eastern Europe who lacked access to traditional preschools, the original GRS intervention was developed and implemented in partnership with the Open Society Foundation in several countries in Central and Eastern Europe, and aimed to prepare children for primary school by helping parents create more effective home learning environments. Parents attended monthly workshops and received printed materials with instructions for play-based learning activities that could be done with everyday materials. In 2010, the intervention was piloted in the USA and demonstrated potential to improve literacy and math skills in participating children relative to those receiving Head Start-as-usual (Noble et al., 2012). Therefore, the program sought to expand based on promising results and positive feedback from parents. The expansion involved three components: (1) design of a complementary classroom component, coupled with similar training for teachers, in order to reinforce skill building in both contexts and to foster parent–teacher communication, (2) the integration of self-regulation skills throughout both curricula (classroom and home), and (3) design of a video library of parent-based activities, to allow parents to observe demonstrations of home-based activities. This expansion was undertaken as a four-year process, with the first year dedicated to curriculum and intervention development, the second year dedicated to piloting the intervention with an iterative revision process and assessing its feasibility, and the third and fourth years aimed to pilot the preliminary efficacy of the GRS intervention collecting fidelity, child outcome data, and classroom quality measures. The three years of piloting the GRS intervention are the subject of the present paper.

School readiness skills such as early literacy, language development, math, and self-regulation do not develop in isolation from one another. Recent research has identified relevant correlations and predictions across domains, and the directionality of these relationships remains unclear. For instance, oral language has been shown to predict trajectories of self-regulation (Vallotton & Ayoub, 2011), and self-regulation at kindergarten has been associated with reading and math skills at 6th grade (McClelland, Acock, & Morrison, 2006; McClelland et al., 2014). There is also strong evidence showing correlations between math and literacy skills in preschool children (Purpura, Hume, Sims, & Lonigan, 2011). Yet, there is no clear evidence that one school readiness skill predicts academic development above and beyond the others (Bailey, Duncan, Watts, Clements, & Sarama, 2018). Therefore, it is crucial to offer teachers the tools to foster the development of all school readiness skills with equal emphasis. For these reasons, the primary aim of GRS is to comprehensively address school readiness by promoting literacy, math, and self-regulation skills with equal emphasis. GRS was designed as a supplemental curriculum that addresses multiple school readiness skills simultaneously, so that teachers can experience the benefits of content-focused goal-directed curricula that explicitly support skill development, without the additional burden of implementing three completely separate content curricula in addition to other programmatic requirements.

Indeed, results from the Preschool Curriculum Evaluation Research Initiative revealed that all but two comprehensive preschool curricula (out of 14 evaluated) failed to increase children’s academic readiness (Preschool Curriculum Evaluation Research Consortium, 2008). Recent work has also suggested that content-specific curricula (targeted at a single area such as math or literacy) may be more effective at generating gains in corresponding skills compared to global curricula such as High/Scope (Hohmann, Weikart, & Epstein, 2008) and Creative Curriculum (Dodge, Colker, Heroman, & Bickart, 2002) that focus more on holistic development through exploratoration than on explicit skill growth through a focus on content areas (Duncan et al., 2015). While the most effective approach to boosting school readiness skills might involve implementing multiple content-focused curricula at once, it is understandably quite difficult for teachers to incorporate multiple supplemental curricula into their daily routines (Duncan et al., 2015). Numerous interventions explicitly target one of these components of school readiness (Clements & Sarama, 2007; Jones, Bailey, & Jacob, 2014; Lonigan, Farver, Phillips, & Clancy-Menchetti, 2011), and several target two (Barnett et al., 2008; Bierman, Domitrovich et al., 2008; Bierman, Nix, Greenberg, Blair, & Domitrovich, 2008); however, GRS is one of the first interventions to take a fully integrated perspective, incorporating three specific components of skill-based content into a single package. Moreover, while some studies have found evidence of spillover effects into readiness domains outside of the targeted content area, e.g. improvement in literacy or executive function skills observed with a mathematics intervention; (Sarama, Lange, Clements, & Wolfe, 2012; Weiland & Yoshikawa, 2013), the sizes and sustainability of these effects are not yet clear. We posit that by allowing teachers to explicitly target multiple domains of school readiness simultaneously, it is more likely that gains will be larger across each of the addressed domains compared to the small spillover effects that may result from focusing on a single content area.

GRS is made up of classroom, home, and teacher–parent communication components. Broadly speaking, the classroom component consists of a series of supplemental classroom activities designed to promote literacy, math, and self-regulation development. Literacy and math activities include learning games (such as “Match the Letters” or “Racing Shapes”), movement activities (such as “Clap the Syllables” or “Jump to the Number”), or poster activities (involving tasks such as analyzing poems, talking about images, or sorting and counting objects).

Following the model originally constructed by Whitehurst and Lonigan (1998), GRS targets emergent literacy through the development of both “outside-in” and “inside-out” skills. “Outside-in” skills represent children's understanding of the linguistic context and include the semantic and conceptual knowledge that comprises oral language; knowledge of narrative structure and narrative-building skills; and print conventions (e.g. left-to-right and front-to-back). “Inside-out” skills comprise children's knowledge of the rules for translating the particular writing they are trying to read into sounds, and includes knowledge of graphemes, phonological awareness, phoneme–grapheme correspondence, and emergent writing. Experience with “outside-in” skills fosters development of oral language and vocabulary, whereas experience with “inside-out” skills fosters development of early reading skills (Bowyer-Crane et al., 2008; Connor, Morrison, & Slominski, 2006). GRS literacy activities include a “Morning Message” activity that grows in complexity in each unit to target conventions of print, as well as stand-alone literacy activities that build various important concepts.

To support the acquisition of math skills, GRS includes activities involving numbers and operations; geometry and spatial sense; measurement; algebra; and data analysis (Clements, Sarama, & DiBiase, 2004; National Association for the Education of Young Children & National Council of Teachers and Mathematics, 2002). These skills are taught through five main processes: problem solving, reasoning, communicating, connecting, and representing (National Association for the Education of Young Children & National Council of Teachers and Mathematics, 2002). For example, when children are prompted to compare the size of different objects in their environment, they learn about measurement. Similarly, when children count sides of shapes, they learn about numbers and geometry and make the connection between these two mathematical concepts (Butera et al., 2014). In GRS, each unit tackles a different math skill, including one-to-one correspondence, counting, sorting and comparison, pattern identification, and measurement.

To support the development of executive function and social emotional development, self-regulation activities were developed in partnership with the team of investigators who designed the Social Emotional Cognitive Understanding and Regulation (SECURe) curriculum (Jones et al., 2014). SECURe acts as a bridge between the self-regulation skills children need and the characteristics of high-quality teachers and classrooms (Bailey & Jones, 2013). Key themes and activities from SECURe’s comprehensive cognitive, emotional, and behavioral self-regulation intervention were carefully integrated into the GRS curriculum. GRS incorporates self-regulation skills into every unit, using both standalone activities and additions to literacy and math activities. The skills developed in these lessons include cognitive skills (working memory, inhibitory control, attention control, and cognitive flexibility) and social–emotional skills (emotional knowledge, expression, and management, empathy, prosocial behavior, coping skills, and conflict resolution). Classroom activities include daily “Brain Games,” which are fun, engaging, often-familiar games (such as a modified “Freeze Dance”), designed to build executive function skills by increasing in complexity over time. Standalone self-regulation lessons, which may be implemented up to twice a week, teach specific skills and strategies through signals, classroom tools, and teaching practices. These self-regulation tools can be integrated into other activities as well as normal classroom interactions and transitions.

These literacy, math, and self-regulation activities are organized into a single classroom manual composed of nine units, following a developmentally appropriate trajectory (see Table 1). It is important to emphasize that the GRS manual and the intervention more broadly were designed to provide high-quality, content-focused activities that can be easily integrated with the requirements of other widely used, more global curricula. All activities last under 15 min; can be done with the whole group, in small groups, or one-on-one; have suggestions for implementation and scaffolding; are designed to be repeated; and are flexible to maximize integration with any primary curriculum or schedule (see Appendices for a sample lesson). Moreover, the manual includes additional tools to help teachers choose activities that will easily integrate with their chosen themes or target skills, such as corresponding standards (from the Head Start Quality Framework) for each activity, and an Appendix of additional tools such as skill and content indices, to assist in customization and integration.

Teachers are asked to implement two to three GRS activities each day, for a total of about 30–45 min. The full classroom manual includes 79 literacy activities, 47 math activities, and 50 self-regulation activities. In a given week, “Morning Message” and “Brain Games” were designed to be implemented daily, along with two to three stand-alone activities that could be repeated over the course of the week. To the extent possible, each activity was designed to synthesize multiple readiness skills, with the goal of being truly integrative across targeted content areas.

Throughout program implementation, teachers received a full-day introductory training and individualized support from a classroom coach, who was also a member of the research team. Coaches met with teachers once a week (alternating between weeks of coaching and weeks of observation) for approximately eight months. During coaching meetings, which typically lasted between 30 and 60 min, teachers planned the implementation of GRS activities and reflected on previous activities implemented. On alternate weeks, coaches observed classroom activity implementation, with modeling and live coaching support sometimes accompanying these sessions. Coaching procedures were modeled after the principles of Practice-Based Coaching (PBC; National Center on Quality Teaching and Learning, 2015), whereby a cyclical model of teacher–coach planning, focused observation, and reflection is used to guide classroom practices. All classroom coaches either held or were pursuing a Masters or PhD degree in educational or psychological fields. Coaches received rigorous training at the beginning of the year, including on principles of PBC, and participated in biweekly group meetings.

The home component of the curriculum consists of literacy, math, and self-regulation activities that are very simple and can be done with everyday household materials. Activities are meant to integrate targeted developmentally appropriate learning into everyday family life. As with the classroom component, these activities are organized into nine units following a developmentally appropriate trajectory. Parents have the flexibility to work through this book on their own or to follow along with parallel activities suggested by the child’s teacher in the weekly letters (the primary instrument of teacher–parent communication in the program). We also provide one tablet per classroom that has a collection of videos demonstrating how to implement the program’s activities at home by showing other parents and their children doing the activities. Tablets are available in every classroom and can be checked out for a week, like a library book. Parents also have access to a program website where they can see all the videos and download all the activities. All materials are available in both Spanish and English and were designed with special attention to culturally diverse and low-literacy parents.

Parent knowledge and practice is supplemented with twelve events, led by GRS staff, over the course of the school year. Parents are invited to attend workshops where they can learn about learning and skill development, as well as in-class events where teachers and GRS staff work with parent–child dyads in the classroom on activities from the parent book.

A wealth of research suggests that the level of intervention fidelity, or the degree to which intervention practices and core components are implemented as intended, affects the program’s outcomes (for a review; Durlak & DuPre, 2008). Recent research has differentiated intervention fidelity from fidelity of implementation, which refers to the degree to which any procedures—such as coaching, in-service training, or any other kind of evidence-based professional development practice—are used as intended to support the adoption and use of core intervention practices (Dunst, Trivette, & Raab, 2013; Hulleman, Rimm-Kaufman, & Abry, 2013). The current study focuses on intervention fidelity, which is most commonly measured by two constructs: dosage and adherence (Darrow, 2010). Dosage refers to the amount of time that the intervention is actually implemented, while adherence refers to the degree to which implementation by the teacher matches the design of the intervention (Durlak, 2010). Evidence is mixed about the degree to which dosage and/or adherence is linked to student outcomes. For example, Hamre et al. (2010) found significant associations with dosage but not adherence, and others have found similar mixed effects of one measure of IF or the other (see Mendive et al., 2016).

The lack of consistent findings is due in part to the lack of consensus measuring IF. Researchers measure these constructs in a variety of different ways. Dosage is most commonly measured by counting the number of intervention activities completed by the teacher in the classroom or number of minutes spent on these activities in a given time period (Hamre et al., 2010). Adherence is typically measured by observing the actual implementation of activities to determine fidelity to the intervention’s supported practices (Stein et al., 2008). Measures of adherence vary widely because they are informed by the intervention. Some researchers advocate for directly observing implementation rather than relying on teacher-reported measures of fidelity (Schulte, Easton, & Parker, 2009), though this is more challenging methodologically.

In addition to dosage and fidelity, research has pointed to other dimensions of implementation, such as participant responsiveness, which has been conceptualized as participants’ reaction to the intervention, including sustained interest and attention (Domitrovich et al., 2008; Domitrovich, Gest, Jones, Gill, & DeRousie, 2010; Dusenbury, Brannigan, Hansen, Walsh, & Falco, 2005). This has been operationalized by measuring children’s affective and behavioral responses to the intervention (e.g. to what extent is the child engaged in and enjoying the activity). Although participant responsiveness has been often omitted in IF research, some studies have linked it to child outcomes. For example, in the implementation study of the Head Start REDI intervention, child engagement was positively associated with a range of social–emotional skills, including teachers’ ratings of social competence and aggression, and quality of children’s problem-solving (Domitrovich et al., 2010). In the present study, we use multiple dimensions of IF (dosage, adherence, and child engagement) to operationalize and disentangle the various aspects of IF.

Studies show that numerous ecological factors affect IF (Durlak & DuPre, 2008), and as such it is critical to examine both IF and program outcomes through a contextual lens. However, evidence is mixed across studies about the degree to which ecological factors matter. For example, it is unclear whether or not teacher experience and education levels influence IF (Baker, Kupersmidt, Voegler-Lee, Arnold, & Willoughby, 2010; Knoche, Sheridan, Edwards, & Osborn, 2010), or whether even more proximal measures such as perceived need for a program are more predictive (Durlak, 2010). Classroom characteristics such as larger classroom size and teacher–child ratio have a negative impact on classroom quality (Barnett, Schulman, & Shore, 2004; Pianta, La Paro, Payne, Cox, & Bradley, 2002), and recent research suggests that larger classroom size is associated with lower adherence fidelity (Zvoch, 2009). In the present study, we examine both classroom-level factors such as classroom size, teacher–child ratios, child age distribution, and classroom quality as measured by the CLASS (Pianta, Karen, Paro, & Hamre, 2008) and teacher-level factors (including years of education and experience, as well as years in the center) in relation to fidelity to GRS.

Although GRS incorporates both classroom and home components, in the present paper, only teacher fidelity to the classroom component of the curriculum will be analyzed. The results regarding the home component are presented in a separate study.

In order to draw conclusions about the potential promise of this integrated intervention, we examined the feasibility and fidelity of the classroom component of the intervention. Our research questions in this study were as follows:

• How feasible is a supplemental, integrated school readiness intervention that taps skills across the domains of literacy, math, and self-regulation?

• How do classroom and teacher characteristics inform intervention fidelity?

• Are different measures of IF (dosage, adherence, and child responsiveness) associated with growth on child's self-regulatory and preacademic skills during the academic year?