Elsevier

Early Childhood Research Quarterly

Volume 37, 4th Quarter 2016, Pages 165-174
Early Childhood Research Quarterly

Fidelity of implementation for an early-literacy intervention: Dimensionality and contribution to children’s intervention outcomes

https://doi.org/10.1016/j.ecresq.2016.06.001Get rights and content

Highlights

  • We examined fidelity of implementation (FOI) in the context of an early-literacy intervention.

  • Adherence, dosage, participant responsiveness, and program differentiation were assessed.

  • The three-factor FOI model was the best fit for the data.

  • Program differentiation predicted children’s literacy gains.

  • Program differentiation fully mediated the intervention effects.

Abstract

This study examined fidelity of implementation (FOI) in the context of an early-literacy intervention involving 83 early childhood special education (ECSE) teachers and 291 three- to five-year old children with disabilities in their classrooms. Adherence, dosage, participant responsiveness, and program differentiation were assessed as multiple dimensions of FOI. Results demonstrated that a three-factor model of adherence and dosage, participant responsiveness, and program differentiation offered the best fit to the data to represent FOI. Further, program differentiation significantly related to children’s early-literacy gains, and the effects of the intervention on children’s gains in early literacy were fully mediated by program differentiation. Findings have implications for the design of effective early-literacy interventions and also for theorizing the construct of FOI.

Introduction

Many young children with disabilities face significant risks in becoming proficient readers (Morgan, Farkas, & Wu, 2011). Consequently, teachers in early childhood special education (ECSE) programs are increasingly required to implement interventions that systematically and explicitly promote the early-literacy development of children in their classrooms (Foorman & Nixon, 2006). Exposure to such interventions may reduce the prevalence of reading problems among preschool children with disabilities and enhance the effectiveness of ECSE programs. While researchers have conducted experimental investigations of early-literacy interventions to assess their impacts on children with disabilities (Justice, Chow, Capellini, Flanigan, & Colton, 2003; Justice, Logan, Kaderavek, & Dynia, 2015; Lovelace & Stewart, 2007; Wilcox, Gray, Guimond, & Lafferty, 2011), they often overlook the importance of measuring and describing fidelity of implementation (FOI; O’Donnell, 2008; Swanson, Wanzek, Haring, Ciullo, & McCulley, 2011) as a potentially important contributor to intervention effects. Thus, the purpose of this study is to examine the fidelity of implementation of an early-literacy intervention for preschool children with disabilities in ECSE classrooms.

FOI within the field of education generally represents the extent to which teachers enact an intervention as intended by the developers (Keller-Margulis, 2012) and can be an influential factor governing the impacts of interventions on child outcomes (Abry, Rimm-Kaufman, Larsen, & Brewer, 2013; Dane and Schneider, 1998, O’Donnell, 2008). Examination of FOI is important as it represents the extent to which the “intended intervention” as implemented leads to positive benefits to children’s learning (Century, Rudnick, & Freeman, 2010). It is generally accepted that high levels of FOI (e.g., dosage or adherence) can result in greater benefits of an intervention for children (Clements and Sarama, 2008, Hamre et al., 2010), whereas low levels of FOI can dampen treatment effects (Hulleman & Cordray, 2009). Thus the measurement of FOI is crucial to understanding and interpreting the outcomes of intervention studies because of its theoretical importance as an active ingredient of an intervention. Furthermore, in the absence of intervention effects, measures of FOI can assist researchers in determining whether the null results reflect a failure of implementation or a failure of intervention theory (Century et al., 2010, Raudenbush, 2007). Interestingly, despite the growing recognition of the importance of understanding and documenting FOI, there are only a handful of studies examining FOI within the early-intervention literature (Mishara & Ystgaard, 2006; Zvoch, Letourneau, & Parker, 2007).

To contribute to this line of research, the present study examined FOI within the context of a teacher-implemented, early-literacy intervention featuring use of print referencing during repeated readings of storybooks (Justice et al., 2015). The defining feature of print-referencing intervention is that teachers systematically and explicitly draw young children’s attention towards print within the context of shared book reading (Justice, Kaderavek, Fan, Sofka, & Hunt, 2009; Justice, McGinty, Piasta, Kaderavek, & Fan, 2010; Piasta, Justice, McGinty, & Kaderavek, 2012). Several efficacy studies suggest that exposure to this intervention can increase children’s print knowledge, an important dimension of early-literacy development, for typically developing preschool children (Justice & Ezell, 2002), children who are at risk due to poverty (Justice et al., 2009, Piasta et al., 2012), and children with disabilities (Justice et al., 2015; Justice, Skibbe, McGinty, Piasta, & Petrill, 2011; Lovelace & Stewart, 2007).

Although print-referencing at-large demonstrates significant positive effects on children’s print knowledge, two recent larger-scale studies of print-referencing intervention implemented within preschool settings have shown that the fidelity with which teachers implement print referencing as intended is variable across fidelity measures (Justice et al., 2009, Justice et al., 2015). Specifically, in Justice et al. (2009), research staff coded one video at three time points (fall, winter, spring) of teachers’ references to two print targets prescribed per session during shared book reading. Results indicated that of the 14 teachers in the intervention condition, 57% hit all six targets, 36% hit four or five targets, and 7% hit no targets (Justice et al., 2009). In a more recent large-scale study of print referencing specifically in early childhood special education classrooms, teachers’ logging of prescribed print targets per session showed high and relatively consistent fidelity to the targets. Of the 55 teachers assigned to the intervention condition, teachers reported addressing the two assigned print targets between 92% and 93.5% of the time.

Despite the fact that examination of individual differences among teachers in their implementation of print-referencing intervention was not the subject of these investigations, data drawn from these studies provides important insights regarding empirical differences in fidelity measures as well as theoretical conceptualizations of fidelity as a multidimensional construct. To this end, our first research aim was to examine the factor structure of FOI, so as to understand the dimensionality of the construct, whereas the second aim was to assess whether FOI may mediate the effects of early-literacy intervention on children’s early-literacy skills.

Many researchers conceptualize FOI as a multidimensional construct (Dane & Schneider, 1998; Dusenbury, Brannigan, Falco, & Hansen, 2003; Greenberg, Domitrovich, Graczyk, & Zins, 2005; Mowbray, Holter, Teague, & Bybee, 2003; O’Donnell, 2008; Pence, Justice, & Wiggins, 2008). However, according to Nelson, Cordray, Hulleman, Darrow, and Sommer (2012), it is important to note that the conceptualization of FOI and its core components in general vary across research disciplines and researchers. For example, Breitenstein et al. (2010) in their examination of community-based interventions used two dimensions, competence and adherence, to measure implementation fidelity whereas Domitrovich, Gest, Jones, Gill, and DeRousie (2010) employed four dimensions of implementation (dosage, fidelity to curriculum components, generalization, and child engagement) to their efficacy study of a preschool curriculum. Thus, specific dimensions of fidelity are not yet universal despite the growth in implementation fidelity research.

As a result, for the purposes of the present study, we chose to use an often-cited work by O’Donnell (2008) who applied Dane and Schneider’s (1998) proposed five dimensions to the exploration of FOI. Dane and Schneider’s five dimensions offer one of the most comprehensive examinations of FOI as it relates to the early literacy intervention under study. The five dimensions are: (a) adherence, (b) dosage, (c) quality of delivery, (d) participant responsiveness, and (e) program differentiation.

In regards to adherence, the definition as proposed by Dane and Schneider (1998) refers to whether or not implementers did what was expected (Dane and Schneider, 1998, Nelson et al., 2012). Given this broad description of what constitutes adhering to an intervention, researchers examine adherence different ways. Many researchers measure intervention adherence as the presence or absence of certain instructional practices via checklists or observations of implementers (Hamre et al., 2010; Kasari, Gulsrud, Wong, Kwon, & Locke, 2010; Piasta, Justice, McGinty, Mashburn, & Slocum, 2015; Mendive, Weiland, Yoshikawa, & Snow, 2016). Other researchers measure intervention adherence as the completion of study requirements or more often as the following of study procedures (Jordan, Snow, & Porche, 2000; Odom, 2009). Studies measuring adherence to the completion of study requirements or procedures track participants’ attendence (Jordan et al., 2000, Odom, 2009) and examine the extent to which participants follow the agenda as prescribed in the intervention (Forgatch, Patterson, & DeGarmo, 2006). In this study, we measured adherence as the completion of study procedures via teachers’ submission or nonsubmission of weekly reading logs. In this way, we could measure whether or not implementers did what was expected of them in following study procedures as the definition of adherence posits (Dane and Schneider, 1998, Nelson et al., 2012).

Dosage refers to the amount of intervention participants received and reflects whether or not participants received as much of the intervention as was prescribed (O’Donnell, 2008). Dosage may include (a) the number of sessions completed, (b) the length of each session, and (c) the frequency with which strategies are implemented (O’Donnell, 2008). Measurement of dosage includes teacher report of the number of sessions implemented and session length (Piasta et al., 2015) and direct observation of the frequency of intervention (Mendive et al., 2016). In this study, dosage was assessed through teacher report of the number of sessions completed over the academic year.

Nelson et al. (2012) defined quality of delivery as the manner in which implementers execute the activities of the intervention. Others refer to it as the quality of interaction or good teaching that may not be prescribed by the intervention, but may impact intervention outcomes (Dusenbury et al., 2003, O’Donnell, 2008). Quality of delivery can also include contextual factors that exist outside the intervention itself, such as teachers’ comfort with the curriculum and perceived effectiveness of the curriculum (Dane and Schneider, 1998, Pence et al., 2008). The nature and quality of methods used to measure quality of delivery vary widely among studies capturing this dimension of FOI. Some studies capture the manner in which teachers deliver the key intervention components using observational measures (Fogarty et al., 2014; Knoche, Sheridan, Edwards, & Osborn, 2010; Piasta et al., 2015). However, other studies utilize a measure of the global quality of teacher-child interactions (Hamre et al., 2010, Wasik and Hindman, 2011). While some measures such as the Classroom Assessment Scoring System (CLASS: Pianta, La Paro, & Hamre, 2008) have helped the field begin to measure quality of delivery, the definition of quality of delivery includes a global look at intervention implementation which, “can always shift depending on the boundaries drawn around the intervention itself” (Century et al., 2010, p. 10). Thus, a measure of this dimension was not included in the present study.

Participant responsiveness indicates the extent to which participants are engaged in and responsive to interventions in terms of participation and enthusiasm (O’Donnell, 2008). Measures of participant responsiveness may include examination of engagement and enthusiasm on the part of the teacher or the child (Darrow, 2013). In this study, we used a participant satisfaction survey that captured enthusiasm for the intervention to measure participant responsiveness.

Program differentiation describes whether critical features of an intervention distinguish the treatment and comparison groups during the implementation of the intervention (O’Donnell, 2008). This dimension is not typically assessed in intervention studies. One recent study however measured program differentiation using a checklist to document reading comprehension strategies in both the intervention and comparison group to distinguish the intervention from the typical practice condition (Fogarty et al., 2014). Indeed, accurate differentiation between the intervention and comparison or control condition is crucial in interpreting intervention effects (Vaughn et al., 2013). In the present study, we used a checklist to code the raw frequency of early literacy intervention strategies used as a measure of program differentiation.

To a large extent, reference to these dimensions of FOI within the literature is theoretically rather than empirically driven, and there is a great need to understand how best to represent FOI (Berkel, Mauricio, Schoenfelder, & Sandler, 2011). In the present study, we applied O’Donnell’s framework to a prior efficacy study (Justice et al., 2015) examining ECSE teachers’ implementation of an early-literacy intervention over an academic year. This study aims to move the science of implementation forward by analyzing the dimensionality of the construct of FOI. Specifically, some researchers suggest that dimensions of FOI (adherence, dosage, participant responsiveness, program differentiation) represent a single underlying factor representing the overall level of fidelity to an intervention (Fogarty et al., 2014). Yet, others suggest that FOI represents at least three distinct dimensions: fidelity to adherence and dosage, fidelity to participant responsiveness, and fidelity to program differentiation (Darrow, 2013, Dynia and Logan, 2014). In this study, we examined whether FOI is best represented as a one-factor model specifying that the common variance among four dimensions is attributable to a single factor, or as a three-factor model indicating that FOI represents three distinct factors of adherence and dosage, participant responsiveness, and program differentiation.

A number of research studies have examined the relations between individual dimensions of FOI and child outcomes with mixed results (Allinder, Bolling, Oats, & Gagnon, 2000; Benner, Nelson, Stage, & Ralston, 2010; Clements and Sarama, 2008, Dane and Schneider, 1998, Hamre et al., 2010; Justice, Mashburn, Pence, & Wiggins, 2008). For example, one intervention study reported statistically significant relations between teachers’ adherence to a literacy intervention and the literacy skills of preschool children (Justice et al., 2008). Other intervention studies, however, in early childhood settings have found no significant relations between teachers’ adherence and preschool children’s language and literacy gains (Hamre et al., 2010, Piasta et al., 2015). Dosage (i.e., the number of minutes teacher implemented the intervention) has been shown to be significantly associated with children’s language and literacy gains within early childhood settings (Hamre et al., 2010, Mendive et al., 2016) and early elementary classrooms (Wolgemuth et al., 2014); however one recent study on strategy-focused reading intervention showed that dosage was not significantly related to middle school students’ reading outcomes (Cantrell, Almasi, Carter, & Rintamaa, 2010). In regards to participant responsiveness and program differentiation, very few intervention studies have included measures of these individual dimensions as they relate to children’s outcomes. Only one recent study that we are aware of included metrics of FOI for adherence, dosage, quality of delivery, participant responsiveness, and program differentiation (Fogarty et al., 2014). In this study, the researchers examined a reading comprehension intervention for middle school students. They found that FOI, as a single factor, was significantly associated with students’ reading outcomes. Overall, despite some inconsistencies related to individual dimensions of FOI, general consensus, as pointed out by Durlak and DuPre (2008), suggests that interventions implemented with high levels of fidelity exhibit impacts with effect sizes two to three times greater with respect to student outcomes, on average, as compared to interventions implemented with low levels of fidelity.

The above-referenced studies provide empirical evidence regarding the potential relations between FOI and child outcomes. However, our understanding of how FOI influences the effectiveness of interventions is far from complete. For instance, it is possible that the impact of an intervention on child outcomes might be mediated through FOI because FOI as conceptualized should represent the critical components of an intervention. In essence, the degree to which these critical components are enacted by implementers should influence their practices which would relate to child outcomes (Girolametto, Weitzman, Lefebvre, & Greenberg, 2007; Hindman and Wasik, 2012, Justice et al., 2009, Pence et al., 2008). However, few empirical studies have explored the mechanisms by which an intervention is effective by testing hypothesized mediators such as FOI specified in the conceptual model of the intervention. For example, most studies examining the importance of FOI have looked at either the relationship between intervention effects and FOI (Girolametto et al., 2007, Pence et al., 2008) or the relationship between FOI and child outcomes (Fogarty et al., 2014, Hamre et al., 2010, Wolgemuth et al., 2014). Instead, it may be possible that an intervention exerts effects on child outcomes through FOI. Hence, this study examined the direct effects of FOI on child outcomes as well as the indirect effects of the intervention on child outcomes via teachers’ FOI within the context of a print-referencing intervention in ECSE classrooms. To this end, two aims were addressed: (1) to examine the factor structure (i.e., dimensionality) of FOI when ECSE teachers implemented an early-literacy intervention, and (2) to examine the direct and indirect relations among the intervention, fidelity of implementation, and children’s early-literacy skills.

Section snippets

Participants

Three cohorts of teachers and children from ECSE programs in the Midwest participated in the present study during the 2008–2011 academic school years. A majority of the classrooms were half-day inclusion classrooms (90%) and the average class size was 13 children (SD = 4 children, Range 5–21 children). Teachers were randomly assigned to one of three conditions; each condition also included a caregiver condition (teacher/caregiver): (a) print-focused teacher/print-focused caregiver (PF/PF: n = 28

Results

Descriptive statistics for the four FOI measures are shown in Table 1, separately by whether teachers were in the comparison or treatment group. Descriptive statistics for children’s early-literacy composite scores are displayed in Table 2. Table 2 demonstrates that, on average, the full sample of 278 children gained 1.73 composite score points from fall to spring, a moderate to large effect (effect size, Cohen’s d = 0.69). To test for missing data, we ran Little’s MCAR test on all included

Discussion

The overarching aims of this study were to examine the factor structure of Fidelity of Intervention (FOI), specifically in the context of an early-literacy intervention implemented in ECSE classrooms, and to determine whether and to what extent teachers’ FOI may serve to mediate treatment impacts. Three major findings of this work, each of which are discussed in turn, include the following: (a) the three-factor FOI model of adherence and dosage, participant responsiveness, and program

Conclusion

In conclusion, the importance of assessing fidelity of implementation (FOI) and examining its role in an intervention effectiveness in research on early-literacy intervention and preschool special education has been emphasized (Odom and Strain, 2002, Wolery, 2011). Experts have argued that delivery context can influence dimensions of implementation (Durlak & DuPre, 2008). The present study contributes to current efforts to increase the understanding of how FOI may impact intervention outcomes

Acknowledgements

The research was supported by U.S. Department of Education, Institute of Education Sciences Grant R324A080037. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Institute of Education Sciences or U.S. Department of Education. We would like to thank STAR 2 staff, as well as the many administrators, teachers, and children without whom this study would not be possible.

References (68)

  • M.J. Wilcox et al.

    Efficacy of the TELL language and literacy curriculum for preschoolers with developmental speech and/or language impairment

    Early Childhood Research Quarterly

    (2011)
  • R.M. Allinder et al.

    Effects of teacher self-monitoring on implementation of curriculum-based measurement and mathematics computation achievement of students with disabilities

    Remedial and Special Education

    (2000)
  • G.J. Benner et al.

    The influence of fidelity of implementation on the reading outcomes of middle school students experiencing reading difficulties

    Remedial and Special Education

    (2010)
  • C. Berkel et al.

    Putting the pieces together: an integrated model of program implementation

    Prevention Science

    (2011)
  • S.M. Breitenstein et al.

    Implementation fidelity in community-based interventions

    Research in Nursing & Health

    (2010)
  • S.C. Cantrell et al.

    Striving readers year 3 evaluation report

    (2010)
  • J. Century et al.

    A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge

    American Journal of Evaluation

    (2010)
  • D.H. Clements et al.

    Experimental evaluation of the effects of a research-based preschool mathematics curriculum

    American Educational Research Journal

    (2008)
  • C.L. Darrow

    The effectiveness and precision of intervention fidelity measures in preschool intervention research

    Early Education and Development

    (2013)
  • C.E. Domitrovich et al.

    The study of implementation: current findings from effective programs that prevent mental disorders in school-aged children

    Journal of Educational and Psychological Consultation

    (2000)
  • J. Durlak et al.

    Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation

    American Journal of Community Psychology

    (2008)
  • L. Dusenbury et al.

    A review of research on fidelity of implementation: implications for drug abuse prevention in school settings

    Health Education Research

    (2003)
  • J.M. Dynia et al.

    An empirical investigation of the dimensionality of fidelity

  • M. Fogarty et al.

    Examining the effectiveness of a multicomponent reading comprehension intervention in middle schools: a focus on treatment fidelity

    Educational Psychology Review

    (2014)
  • B.R. Foorman et al.

    The influence of public policy on reading research and practice

    Topics in Language Disorders

    (2006)
  • L. Girolametto et al.

    The effects of in-service education to promote emergent literacy in child care centers: a feasibility study

    Language, Speech, and Hearing Services in Schools

    (2007)
  • M.T. Greenberg et al.

    The study of implementation in school-based preventive interventions: theory, research, and practice

    Promotion of mental health and prevention of mental and behavioral disorders 2005 Series Vol. 3

    (2005)
  • Y. Guo et al.

    Behavioral engagement and reading achievement in elementary-school-age children: a longitudinal cross-lagged analysis

    Journal of Educational Psychology

    (2015)
  • S. Hayes

    Nine ducks nine

    (1990)
  • A.H. Hindman et al.

    Unpacking an effective language and literacy coaching intervention in Head Start

    The Elementary School Journal

    (2012)
  • C.S. Hulleman et al.

    Moving from the lab to the field: the role of fidelity and achieved relative intervention strength

    Journal of Research on Educational Effectiveness

    (2009)
  • M.A. Invernizzi et al.

    Phonological awareness literacy screening for preschool: teacher's manual

    (2004)
  • G.E. Jordan et al.

    Project EASE: the effect of a family literacy project on kindergarten students' early literacy skills

    Reading Research Quarterly

    (2000)
  • L.M. Justice et al.

    Use of storybook reading to increase print awareness in at-risk children

    American Journal of Speech-Language Pathology

    (2002)
  • Cited by (18)

    • Assessing the Implementation Fidelity of Early Interventions: Data Collection Methods

      2020, Evaluation and Program Planning
      Citation Excerpt :

      With respect to fidelity of intervention practices, in two studies (Bierman et al., 2014; Pence et al., 2008) teachers completed a questionnaire on the quality of implementation of their practices. In 6 other studies (Dunlap, Strain, Lee, Joseph, & Leech, 2017; Fox et al., 2011; Guo et al., 2016; Holzinger, Laister, Vivanti, Barbaresi, & Fellinger, 2019; Shire et al., 2017; Strain & Bovey, 2011) a questionnaire was used to examine the degree of satisfaction or teachers’ perceived relevance of intervention practices. Data on duration or number of contacts was collected in 5 studies that look at fidelity of professional development.

    • Intervention fidelity of Getting Ready for School: Associations with classroom and teacher characteristics and preschooler's school readiness skills

      2018, Early Childhood Research Quarterly
      Citation Excerpt :

      Future research should explore how to define quality of coaching and what the implications may be for fidelity to school readiness interventions such as GRS. There has been a growing interest in connecting intervention fidelity and child outcomes in early childhood research (Clements et al., 2013; Domitrovich et al., 2010; Guo et al., 2016; Knoche et al., 2010). Despite research that shows a generally positive relationship between intervention fidelity and child outcomes, these relationships varied in the present study depending on the fidelity measure used (Clements & Sarama, 2008; Hamre et al., 2010; Domitrovich et al., 2008; Justice, Mashburn, Hamre, & Pianta, 2008).

    • Narrative dialogic reading with wordless picture books: A cluster-randomized intervention study

      2020, Early Childhood Research Quarterly
      Citation Excerpt :

      To assess treatment fidelity, instructors completed a short questionnaire directly after each of the 42 sessions. In line with early literacy research (e.g., Guo et al., 2016), we assessed adherence to the intervention program on the group level (5 = very good, 4 = good, 3 = satisfactory, 2 = unsatisfactory, 1 = poor), dosage (number of sessions attended by each child), and participant responsiveness on the individual level (quantity and quality of child involvement in the program: 5 = very good, 4 = good, 3 = satisfactory, 2 = unsatisfactory, 1 = poor). Adherence to the intervention program was good (dialogic reading: M = 4.5, SD = 0.3; alternative treatment: M = 3.9, SD = 0.9).

    View all citing articles on Scopus
    View full text