Elsevier

Evaluation and Program Planning

Volume 60, February 2017, Pages 24-36
Evaluation and Program Planning

Process evaluation of a preschool physical activity intervention using web-based delivery

https://doi.org/10.1016/j.evalprogplan.2016.08.022Get rights and content

Highlights

  • Online delivery of SHAPES-D to preschool teachers was at least as effective as in-person delivery of SHAPES.

  • Translating and adapting an effective intervention that used in-person delivery for an online format has the potential for broader dissemination and greater reach and public health impact.

  • The SHAPES-D study may be the first structural intervention to monitor change agent dose received through web metrics.

Abstract

Background and purpose

Preschool/childcare settings offer a practical target for physical activity interventions. Online learning programs have the potential for greater public health reach and impact. The SHAPES-Dissemination (SHAPES-D) project adapted the original SHAPES in-person intervention for online delivery to teachers. The purpose of this paper is to describe the implementation monitoring and process evaluation for the SHAPES-D project.

Methods

Nine preschools with 26 classrooms participated. A total of 41 teachers were trained via online learning to implement the SHAPES-D program in their classrooms. The dose received, completeness, and fidelity of implementation were assessed through website metrics, teacher surveys and interviews, and classroom observations.

Results

Dose received was adequate (73%). Observed completeness and physical activity enjoyment fidelity were high (100%), although moderate-to-vigorous physical activity fidelity and social environment fidelity were low (25% each). Overall implementation was high (91%).

Discussion

Results indicate that the online method of delivery is viable for dissemination. The online delivery system provides an easy method of monitoring dose received. This may be the first structural intervention to monitor dose received through web metrics.

Conclusion

The adaptation of an in-person intervention to an online delivery system increases the potential for dissemination of a successful program to increase physical activity in preschool settings.

Introduction

Physical activity in early childhood is associated with multiple health benefits (Hinkley et al., 2014, Timmons et al., 2012). Specifically in this population, moderate-to-vigorous physical activity (MVPA) has been shown to improve bone health later in life (Janz et al., 2010), and PA during early childhood is associated with weight status and may have a protective effect on adiposity (Janz et al., 2009, Moore et al., 2003, Reilly, 2008). Additionally, higher levels of MVPA in early childhood may protect against chronic disease throughout the life-course (Tremblay et al., 2015). Despite compelling evidence for the benefits of PA, current levels of PA among preschool students fall far below the recommended standards (Cardon & De Bourdeaudhuij, 2008; Hinkley, Salmon, Okely, Crawford, & Hesketh, 2012; Pate, McIver, Dowda, Brown, & Addy, 2008; Reilly, 2010). Therefore, increasing PA in preschool children should be a public health priority (Lessard and Breck, 2015, Sallis et al., 2012, Wright et al., 2015).

Preschool and childcare settings pose a practical target for intervention. In the United States, 61% of 3- to 6-year-old children not yet in kindergarten are enrolled in preschool or childcare outside the home (Federal Interagency Forum on Child & Family Statistics, 2015). The characteristics of a preschool, including policies and practices regarding PA, can significantly influence children’s activity levels (Dowda et al., 2009, Pate et al., 2008; Pate, Pfeiffer, Trost, Ziegler, & Dowda, 2004).

Results from studies in preschool settings show inconsistent effects for increasing moderate-to-vigorous physical activity (MVPA) (Annesi, Smith, & Tennant, 2013; De Bock, Genser, Raat, Fischer, & Renz-Polster, 2013; De Craemer et al., 2014, Fitzgibbon et al., 2011, Hannon and Brown, 2008), and few studies reported process evaluation in an effort to explain intervention effects. Poor implementation may contribute to inconsistent findings as level of implementation has been shown to affect results (Durlak and DuPre, 2008, Wilson et al., 2009). Those that have monitored implementation have shown that low implementation could be one reason for the lack of significant results. For example, Trost, Fees, and Dzewaltowski (2008) utilized process evaluation methods in order to explore the frequency and contexts in which their intervention was delivered by preschool teachers. Only seventy-four percent of the movement activities met the intervention time requirements, and classroom observations revealed that preschool teachers were not implementing the intervention as it was intended, thus explaining the null findings during the first half of the study (Trost et al., 2008). Bonvin and colleagues noted that while there was no significant change in outcomes, not all child care centers delivered all the program components (Bonvin et al., 2013). Alhassan and Whitt-Glover noted teachers indicated barriers that limited the implementation of the intervention and no PA changes were noted (Alhassan & Whitt-Glover, 2014).

Intervention delivery may also impact the results. Some studies aiming to increase PA in preschools have employed trained research staff for implementation (Finch, Jones, Yoong, Wiggers, & Wolfenden, 2016; Fitzgibbon et al., 2011, Hannon and Brown, 2008), whereas others have trained teachers (Alhassan and Whitt-Glover, 2014, De Craemer et al., 2014, Finch et al., 2014, Finch et al., 2016, Trost et al., 2008) and/or involved parents (De Bock et al., 2013, Finch et al., 2016) to deliver the intervention. Research indicates that in-person interventions delivered by research staff may not effectively reach teacher change agents, who deliver the intervention, or children, who are the priority audience. For example, Finch and colleagues (2014) found that only 41% of preschool staff attended the intervention’s training workshop, and at one site, only 18% of the teachers were in attendance (Finch et al., 2014). In a community-based preschool PA intervention, De Bock and others (2013) found that, although the intervention implementation and adoption rates were relatively high (80% and 83%, respectively), only 33% of eligible children were reached. Methods must be tested that enable evidence-based programs to reach a wider audience in order to have a greater public health impact (Brownson, Jacobs, Tabak, Hoehner, & Stamatakis, 2013; Neta et al., 2015; Riley, Glasgow, Etheredge, & Abernethy, 2013) and new methods for addressing barriers to implementation should be explored.

The use of eTechnology (web applications or mobile apps) in interventions is growing (Barretto, Bingham, Goh, & Shope, 2011; Rosa, Campbell, Miele, Brunner, & Winstanley, 2015; Santoro, Nicolis, Franzosi, & Tognoni, 1999), and it has the potential to promote widespread reach. Research indicates that while gaps still exist within some demographics, 84% of American adults are using the internet (Perrin & Duggan, 2015). Obstacles to translating research to practice have been identified. For example, failure to design interventions for dissemination creates stumbling blocks toward bringing research into practice-based settings (Brownson et al., 2013, Cohen et al., 2008; Glasgow, Lichtenstein, & Marcus, 2003). Interventions that use technology in their delivery design may help to diminish the research-to-practice gap and may address barriers to implementation of PA programs within preschool settings. Additionally, online learning programs allow for a greater public health reach and impact if programs can be delivered as effectively as in face-to-face interventions (Rosa et al., 2015, Santoro et al., 1999).

The SHAPES (Study of Health and Activity in Preschool Environments) intervention was a flexible and adaptive environmental (i.e., structural) intervention delivered by teachers (change agents) to change instructional practices and classroom social environments in an effort to increase preschool children’s MVPA (Howie et al., 2014, Pate et al., 2016, Pfeiffer et al., 2013). The results indicated that students in the intervention schools accumulated significantly more MVPA than students in control schools, with a stronger intervention effect for girls than for boys (Pate et al., 2016). The original SHAPES intervention (Howie et al., 2014, Pate et al., 2016, Pfeiffer et al., 2013) consisted of four essential elements: indoor PA opportunities (“Move Inside”), outdoor free play and structured PA opportunities (“Move Outside”), PA integrated with pre-academic lessons (“Move to Learn”), and enhanced social support (i.e., teacher participation, teacher encouragement, child enjoyment).

In an effort to increase the reach of this intervention, the original SHAPES in-person delivery methods were adapted for web-based delivery through the SHAPES-Dissemination (SHAPES-D) project. The intervention team incorporated the key training components from the original SHAPES intervention (i.e., SHAPES philosophy, PA definitions, essential elements, self-assessment, etc.) into six online training modules. The modules included an introduction to SHAPES-D, PA concepts, PA components (Move Inside, Move Outside, and Move to Learn), strategies for enhancing PA quality, self-assessments, and a review of the course. In addition to the online training modules, teachers received a hard copy of SHAPES-D materials, including a guidebook detailing the information presented in each module, sample activities, and laminated activity cards with actions to facilitate activity implementation.

Studies delivered in real world preschool/childcare settings are needed to influence future policy changes (Finch et al., 2016), and process evaluation and implementation monitoring are needed to explain intervention results (Bopp, Saunders, & Lattimore, 2013; Durlak & DuPre, 2008; Saunders, Evans, & Joshi, 2005). Triangulating measures of teacher dose received, implementation completeness, and implementation fidelity provide insight into the level of implementation of an intervention (Bopp et al., 2013, Saunders, 2015, Wilson et al., 2009). Furthermore, investigating teacher perceived barriers provides insights to reduce/prevent barriers to implementation in future studies. The purpose of this paper is to describe the implementation monitoring and process evaluation for the SHAPES-D project.

Section snippets

Participants

In an effort to include all types of preschools, a combination of public, commercial, and religious preschools that had at least two classrooms with 3- to 5-year-old children within a moderately-sized southern city were identified and approached to gauge interest in participating in the intervention. The exact number of preschools available within this city cannot be identified; however, all schools and districts who were approached agreed to participate and helped to identify the classrooms

Results

SHAPES-D process data, based on lead teacher report, are summarized by classroom in Table 3. Within the nine schools, 41 lead and assistant teachers in 26 classrooms completed the course. Each school had different methods for using lead, assistant, or co-teachers within the classroom. To assess classroom implementation, the lead teacher’s data from the surveys was used with two exceptions. One school (Classrooms N and O in Table 2) used a co-teaching system, with four teachers in two

Summary of major findings

Overall, implementation was very high, with much higher levels of implementation than those currently reported in the literature (Bice, Brown, & Parry, 2014; Carroll et al., 2007, Durlak and DuPre, 2008). Carroll et al. indicate that success for interventions relies on high implementation fidelity (Carroll et al., 2007). Durlak and Dupree postulate that perfect implementation for any intervention is unrealistic and significant results can be seen at 60% implementation; few studies report higher

Lessons learned

First, we learned how to translate and adapt an in-person intervention for online delivery. Additionally, we learned that web-based delivery is a viable method to support the implementation of PA programs in childcare settings. This allows for broader dissemination and the opportunity for greater reach, thus decreasing the gap between research and practice.

Second, we learned about minor content changes to the program which could be beneficial to preschool teachers and the research team. For

Conclusion

The comprehensive process evaluation conducted for the SHAPES-D program allowed for greater understanding of how SHAPES-D was utilized and received by participating preschool teachers. The process evaluation shed light on portions of the program that could benefit from revisions. One of the most important lessons learned was that, from an implementation point-of-view, the web-based program was well received and the use of technology to deliver the program online has the potential for great

Acknowledgments

The authors would like to acknowledge Dale Murrie, SHAPES-D Project Coordinator, Samantha McDonald, research assistant, Gaye Groover Christmus, MPH, for technical and editorial assistance in the development of the manuscript, and the website development team for their contributions to the SHAPES-D program. We would also like to express our sincere appreciation to the preschool directors and teachers who participated in SHAPES-D.

Ann Blair Kennedy, DrPH is a postdoctoral fellow at the University of South Carolina School of Medicine Greenville. She became interested in implementation monitoring and process evaluation during her doctoral studies.

References (61)

  • E. Santoro et al.

    Internet for clinical trials: Past, present, and future

    Controlled Clinical Trials

    (1999)
  • D.R. Wright et al.

    Modeling the cost effectiveness of child care policy changes in the U.S

    American Journal of Preventive Medicine

    (2015)
  • J.J. Annesi et al.

    Effects of a cognitive-behaviorally based physical activity treatment for 4- and 5-year-old children attending US preschools

    International Journal of Behavioral Medicine

    (2013)
  • A.I. Barretto et al.

    Developing a Web-Based Health Promotion Intervention: A Case Study From a Brief Motivational Alcohol Program

    Health Promotion Practice

    (2011)
  • M.R. Bice et al.

    Retrospective evaluation of factors that influence the implementation of CATCH in southern illinois schools

    Health Promotion Practice

    (2014)
  • L. Bond et al.

    Building capacity for system-level change in schools: Lessons from the Gatehouse Project

    Health Education & Behavior: The Official Publication of the Society for Public Health Education

    (2001)
  • A. Bonvin et al.

    Effect of a governmentally-led physical activity program on motor skills in young children attending child care centers: A cluster randomized controlled trial

    The International Journal of Behavioral Nutrition and Physical Activity

    (2013)
  • M. Bopp et al.

    The tug-of-War: Fidelity versus adaptation throughout the health promotion program life cycle

    The Journal of Primary Prevention

    (2013)
  • S.S. Brady et al.

    An interactive website to reduce sexual risk behavior: Process evaluation of TeensTalkHealth

    JMIR Research Protocols

    (2015)
  • R.C. Brownson et al.

    Designing for dissemination among public health researchers: Findings from a national survey in the United States

    American Journal of Public Health

    (2013)
  • G.M. Cardon et al.

    Are preschool children active enough? objectively measured physical activity levels

    Research Quarterly for Exercise and Sport

    (2008)
  • C. Carroll et al.

    A conceptual framework for implementation fidelity

    Implementation Science: IS

    (2007)
  • D.J. Cohen et al.

    Fidelity versus flexibility: Translating evidence-based research into practice

    American Journal of Preventive Medicine

    (2008)
  • M. De Craemer et al.

    The effect of a kindergarten-based, family-involved intervention on objectively measured physical activity in Belgian preschool boys and girls of high and low SES: The ToyBox-study

    The International Journal of Behavioral Nutrition and Physical Activity

    (2014)
  • M. Dowda et al.

    Policies and characteristics of the preschool environment and physical activity of young children

    Pediatrics

    (2009)
  • J.A. Durlak et al.

    Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation

    American Journal of Community Psychology

    (2008)
  • S. Elo et al.

    The qualitative content analysis process

    Journal of Advanced Nursing

    (2008)
  • Federal Interagency Forum on Child and Family Statistics

    Federal interagency forum on child and family statistics

    (2015)
  • M. Finch et al.

    Effectiveness of centre-based childcare interventions in increasing child physical activity: A systematic review and meta-analysis for policymakers and practitioners

    Obesity Reviews: An Official Journal of the International Association for the Study of Obesity

    (2016)
  • M.L. Fitzgibbon et al.

    Hip-Hop to health jr. obesity prevention effectiveness trial: Postintervention results

    Obesity (Silver Spring, Md.)

    (2011)
  • Cited by (18)

    • Comparison between the Healthy Start-Départ Santé online and in-person training of childcare educators to improve healthy eating and physical activity practices and knowledge of physical activity and fundamental movement skills: A controlled trial

      2020, Preventive Medicine Reports
      Citation Excerpt :

      Difficulties with coordination of training sessions and high turn-over rate of personnel in ECCs also pose a challenge (Ward et al., 2018). Online education has been identified as a possible option for professional learning in early childhood education (Peden et al., 2018; Kennedy et al., 2017), with one randomized pilot study suggesting that in-person programs can effectively be translated to an online format and be successfully implemented by ECC directors (Ward et al., 2017). Thus, online training could reduce costs associated with human resources and onsite training, allow educators to complete the training at their own pace, and provide all new personnel with the opportunity to be trained.

    • CHAMP: A cluster randomized-control trial to prevent obesity in child care centers

      2019, Contemporary Clinical Trials
      Citation Excerpt :

      In addition, trained data collectors conduct in-person visits to the center to observe at least two lessons throughout the intervention. The in-person data collection informs data reported by child care staff over the phone using a paper instrument modified from previous childhood interventions [111–113]. Only centers assigned to CHAMP and CHAMP+ arms are assessed for process evaluation and information gathered reflects the entire classroom, extending beyond the child evaluation sample.

    • The theoretical foundation, fidelity, feasibility, and acceptability of a teacher training to promote physical activity among preschoolers in child care: A pilot study

      2019, Preventive Medicine Reports
      Citation Excerpt :

      Web-based training to support teacher PA promotion in child care is a promising, new approach that could result in a large reach and has the potential for a public health impact. Indeed, we are aware of only one other study that has examined online training in this area, and that study also reported high levels of implementation fidelity (Kennedy et al., 2017). Due to the small sample size of the current study, a larger study of WE PLAY with more teachers in Head Start and non-Head Start programs and direct measurement of teachers' behaviors is warranted.

    • Physical activity and nutrition program for adults with metabolic syndrome: Process evaluation

      2017, Evaluation and Program Planning
      Citation Excerpt :

      At the end of the six month intervention program, participants showed significant improvements in moderate intensity activity and walking, as well as reductions in the consumption of animal internal organs and cooking oil for daily meal preparation (Tran et al., 2016a). It is important to monitor the implementation of community-based trials, so as to gain insights into why a program may succeed or fail (Blackford et al., 2016; Burke et al., 2012; Kennedy, Schenkelberg, Moyer, Pate, & Saunders, 2017; Wilson et al., 2009). Process evaluation offers an opportunity to determine how the program was conducted and the characteristics or mechanisms associated with its implementation, which potentially affected the outcomes (Aarestrup, Jørgensen, Due, & Krølner, 2014; Blackford et al., 2016; Burke et al., 2012; Wilson et al., 2009).

    View all citing articles on Scopus

    Ann Blair Kennedy, DrPH is a postdoctoral fellow at the University of South Carolina School of Medicine Greenville. She became interested in implementation monitoring and process evaluation during her doctoral studies.

    Michaela Schenkelberg, MPH is a research assistant and doctoral candidate in Exercise Science at the University of South Carolina. Her research is focused on creating healthful environments in early childhood settings and physical activity promotion among young children with and without developmental disabilities, especially those with Autism Spectrum Disorders.

    Christina Moyer, M.S., works with the Children's Physical Activity Research Group at the University of South Carolina's Arnold School of Public Health. She is currently the intervention coordinator for ongoing dissemination of SHAPES.

    Russell R. Pate, PhD is a Professor in the Department of Exercise Science and Director of the Children’s Physical Activity Research Group at the University of South Carolina. Dr. Pate is an exercise physiologist and an international authority on physical activity and fitness in children, health implications of physical activity, and public policies to promote physical activity. He has published more than 300 scholarly papers and authored or edited eight books. Dr. Pate’s research currently is supported by the National Institutes of Health, Centers for Disease Control and Prevention, and foundation and private funders.

    Ruth P. Saunders is a Professor Emerita in the Arnold School of Public Health at the University of South Carolina. She has conducted process evaluation in eight large-scale intervention trials and oversaw process evaluation in SHAPES-D.

    This research was supported by a grant from The Duke Endowment. The funding agency had no input into the study design; collection, analysis or interpretation of data; writing the report; or in deciding to submit this article for publication.

    View full text