Skip to main content
SpringerLink
Log in

Quality is Critical for Meaningful Synthesis of Afterschool Program Effects: A Systematic Review and Meta-analysis

  • Empirical Research
  • Published:
Journal of Youth and Adolescence Aims and scope Submit manuscript

Abstract

Afterschool programs are designed to promote academic achievement and youth development as well as mitigate risk for students who attend. Systematic reviews have reported that afterschool programs are associated with increased academic proficiency, school bonding, prosocial behavior, and decreased problem behavior. However, the majority of meta-analyses that report on these outcomes limit their samples to only rigorously conducted studies, which is not representative of the larger literature base. This systematic review and meta-analysis is designed to extend the literature by meta-analyzing a comprehensive sample of studies and examine the impact of study quality on meta-analytical results. A comprehensive literature and effect size extraction process search yielded 130 effect sizes from 30 studies that examined the effects of afterschool programs on secondary students’ academic achievement and social / behavioral development. The majority of studies were unpublished reports, and study quality in the sample represented a high risk of bias. The results confirm the hypothesis that the inclusion of lower quality studies significantly impacts overall outcomes in comparison to prior meta-analyses that have limited samples. As such, it is imperative that researchers and practitioners increase the rigor of study design across individual program evaluations so that fewer reports will be excluded. This will improve future meta-analytic outcomes by ensuring a more representative sample of the literature base. Only through improved rigor of study design will a more clear understanding of program impacts on secondary students be possible.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • * Indicate studies included in the meta-analysis

  • Alliance, A. (2014). America after 3PM: Afterschool programs in demand. Washington, DC: Afterschool Alliance.

    Google Scholar 

  • Alliance, A. (2016). America after 3pm special report: afterschool in communities of concentrated poverty. Washington, DC: Afterschool Alliance.

    Google Scholar 

  • Bohnert, A., Fredericks, J., & Randall, E. (2010). Capturing unique dimensions of youth organized activity involvement: Theoretical and methodological considerations. Review of Educational Research, 80(4), 576–610.

    Article  Google Scholar 

  • *Boyer, K. A. M. (2010). Investigating differences among Asian American youth participating and not participating in grant-funded high school after -school programs. Fresno, Ann Arbor: California State University. http://proxy.library.vcu.edu/login?url=https://search.proquest.com/docview/756466418?accountid=1478.

  • *Bryner, S. L. (2004). Is gender mediating the effects of after-school programs? College Park, Ann Arbor: University of Maryland. http://proxy.library.vcu.edu/login?url=https://search.proquest.com/docview/305175724?accountid=14780.

  • *Carbone, P. M. (2010). The effects of an after-school tutoring program on the Pennsylvania System of Assessment. US: ProQuest Information & Learning.

    Google Scholar 

  • *Carr, E. M. (2015). Afterschool program interventions that support the academic achievement, behavior, and engagement of at-risk student populations. Ann Arbor: University of Houston-Clear Lake. http://proxy.library.vcu.edu/login?url=https://search.proquest.com/docview/1831587154?accountid=14780.

  • Carver, P. R., & Iruka, I. U. (2006). National household education surveys program of 2005 after-school programs and activities (NCES 2006–07). Washington, DC: National Center for Education Statistics.

  • *Chittum, J. R., Jones, B. D., Akalin, S., & Schram, Á. B. (2017). The effects of an afterschool STEM program on students’ motivation and engagement. International Journal of STEM Education, 4(1), 1–16. https://doi.org/10.1186/s40594-017-0065-4.

    Article  Google Scholar 

  • Chow, J. C. (2018). Prevalence of publication bias tests in speech, language, and hearing research. Journal of Speech, Language, and Hearing Research, 61(12), 3055–3063.

    Article  Google Scholar 

  • Chow, J. C., & Ekholm, E. (2018). Do published studies yield larger effect sizes than unpublished studies in education and special education? A meta-review. Educational Psychology Review, 30(3), 727–744.

    Article  Google Scholar 

  • *Cross, A. B., Gottfredson, D. C., Wilson, D. M., Rode, M., & Connell, N. (2009). The impact of after-school programs on the routine activities of middle-school students: Results from a randomized, controlled trial. Criminology & Public Policy, 8(2), 391–391.

    Article  Google Scholar 

  • *Delucchi, G. R. (2010). An evaluation of an after -school program for low -income elementary and middle school students. Ann Arbor: Fordham University. http://proxy.library.vcu.edu/login?url=https://search.proquest.com/docview/750474087?accountid=14780.

  • *Dreyer, K. J. (2011). An examination of academic outcomes for students who attend a school-based afterschool program. Ann Arbor: University of Pittsburgh. http://proxy.library.vcu.edu/login?url=https://search.proquest.com/docview/746583937?accountid=14780.

  • Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after‐school that seek to promote personal and social skills in children and adolescents. American journal of community psychology, 45(3-4), 294–309.

    Article  Google Scholar 

  • Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463.

    Article  Google Scholar 

  • Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629–634.

    Article  Google Scholar 

  • *Fink, B. L. (2011). The effect of a seventh grade after school leadership program on the developmental assets, academic achievement, and behavior of non-thriving students. US: ProQuest Information & Learning.

    Google Scholar 

  • *Franklin, C. (2017). Effects of the Afterschool Program on Student Achievement of Students with Disabilities in a Rural Georgia Middle School. Ann Arbor: Liberty University. http://proxy.library.vcu.edu/login?url=https://search.proquest.com/docview/1952045712?accountid=14780.

  • Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71(2), 149–164.

    Article  Google Scholar 

  • *Girod, M., Martineau, J., & Yong, Z. (2004). After-School Computer Clubhouses and At-risk Teens. American Secondary Education, 32(3), 63–76.

    Google Scholar 

  • *Gottfredson, D. C., Cross, A. B., Wilson, D. M., Connell, N., & Rorie, M. (2010a). A randomized trial of the effects of an enhanced after-school program for middle-school students. Final Report Submitted to the US Department of Education Institute for Educational Sciences.

  • *Gottfredson, D. C., Cross, A., Wilson, D., Rorie, M., & Connell, N. (2010b). An experimental evaluation of the all stars prevention curriculum in a community after school setting. Prevention Science, 11(2), 142–154.

    Article  Google Scholar 

  • *Grassi, A. M. (2012). Quality after-school programming and its relationship to achievement-related behaviors and academic performance. Ann Arbor: Cleveland State University.

    Google Scholar 

  • *Grolnick, W. S., Farkas, M. S., Sohmer, R., Michaels, S., & Valsiner, J. (2007). Facilitating motivation in adolescents: effects of an after-school program. Journal of Applied Developmental Psychology, 28(4), 332–344.

    Article  Google Scholar 

  • Higgins, J. P., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., & Sterne, J. A. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Bmj, 343, d5928.

    Article  Google Scholar 

  • *Hobbs, C. L. (2012). Effects of an afterschool program on elementary and middle school math achievement in Georgia schools (Ed.D.). United States—Virginia: Liberty University.

    Google Scholar 

  • *Hu, X., Craig, S. D., Bargagliotti, A. E., Graesser, A. C., Okwumabua, T., Anderson, C., & Sterbinsky, A. (2012). The Effects of a traditional and technology-based after-school program on 6th grade student’s mathematics skills. Journal of Computers in Mathematics and Science Teaching, 31(1), 17–38.

    Google Scholar 

  • *Jarratt, K. (2014). Mathematics Achievement Outcomes for Middle School Students Attending School-Based Afterschool Mathematics Programs (Ed.D.). United States—Tennessee: Union University.

    Google Scholar 

  • *Jones, B. R. (2014). An after-school program and its effect on the math and reading performance levels of the standardized testing and reporting (STAR) for identified at-risk students (Ph.D.). United States—Minnesota: Capella University. https://search.proquest.com/docview/1497967944/abstract/F46A90DBC46A47D6PQ/1.

  • Kremer, K. P., Maynard, B. R., Polanin, J. R., Vaughn, M. G., & Sarteschi, C. M. (2015). Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: a systematic review and meta-analysis. Journal of youth and adolescence, 44(3), 616–636.

    Article  Google Scholar 

  • Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Green, M. (2006). Out-of school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.

    Article  Google Scholar 

  • *Le, T. N., Arifuku, I., Vuong, L., Tran, G., Lustig, D. F., & Zimring, F. (2011). Community mobilization and community-based participatory research to prevent youth violence among Asian and immigrant populations. American Journal of Community Psychology, 48(1–2), 77–88.

    PubMed  Google Scholar 

  • *Luce, F. C. (2018). The Effects of an After-School Program on Sixth-Grade Students’ PARCC Scores in a High-Poverty Urban Location in New Jersey (Ed.D.). United States—New Jersey: Saint Peter’s University. http://search.proquest.com/docview/2092265378/abstract/934CF09625EA4E4FPQ/1.

  • *Manior, S. H. (1997). Effects of a structured after school program on middle school students (Ed.D.). United States—South Carolina: South Carolina State University. https://search.proquest.com/docview/252319191/abstract/6DD97E4D71CB49DAPQ/1.

  • *Morrison, G. M., Storino, M. H., Robertson, L. M., Weissglass, T., & Dondero, A. (2000). The protective of afterschool programming and parent education and support for students at risk for substance abuse. Evaluation and Program Planning, 23(3), 365–371.

    Article  Google Scholar 

  • *Nears, K. (2007). The achievement gap: Effects of a resilience-based after school program on indicators of academic achievement (Ph.D.). United States—North Carolina: North Carolina State University. https://search.proquest.com/docview/304843101/abstract/A2E25ADD631E476APQ/1.

  • *Pfefferle, K. (2011). The Effects of the After School Program Kids Unlimited on Students’ Academic Performance on the Ohio Achievement Assessment (Ed.D.). United States—Ohio: The University of Toledo. https://search.proquest.com/docview/926418129/abstract/1DB25C17C8544E50PQ/1.

  • Pierce, K. M., Bolt, D. M., & Vandell, D. L. (2010). Specific features of after-school program quality: Associations with children’s functioning in middle childhood. American Journal of Community Psychology, 45(3-4), 381–393.

    Article  Google Scholar 

  • Pigott, T. D. (2012). Advances in Meta-Analysis. Boston, MA: Springer.

  • *Prenovost, J. K. E. (2001). A first-year evaluation of after -school learning programs in four urban middle schools in the Santa Ana Unified School District (Ed.D.). Irvine, United States—California: University of California. https://search.proquest.com/docview/276296103/abstract/4738F5C4E90B43A2PQ/1.

  • Riggs, N. R., & Greenberg, M. T. (2004). After-school youth development programs: a developmental-ecologicalmodel of current research. Clinical Child and Family Psychology Review, 7(3), 177–190.

    Article  Google Scholar 

  • Roth, J. L., Malone, L. M., & Brooks-Gunn, J. (2010). Does the amount of participation in afterschool programs relate to developmental outcomes? a review of the literature. American journal of community psychology, 45(3-4), 310–324.

    Article  Google Scholar 

  • *Roukema, R. A. (2005). The impact of the Support Our Students (SOS) after-school program on the achievement of middle-grade students at risk of academic failure (Ed.D.). United States—North Carolina: North Carolina State University. https://search.proquest.com/docview/305397555/abstract/C15240D4A4EF48C1PQ/1.

  • *Ryal, R. R. (2016). The effects of an after school program on at-risk tenth grade students’ achievement in English/language arts and mathematics (Ph.D.). United States—Minnesota: Capella University. https://search.proquest.com/docview/1774020011/abstract/35D3ABDFDA07441APQ/1.

  • Shernoff, D. J. (2010). Engagement in after-school programs as a predictor of social competence and academic performance. American Journal of Community Psychology, 45(3–4), 325–337.

    Article  Google Scholar 

  • Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 Statement: updated guidelines for reporting parallel group randomized trials. Open Medicine, 4(1), e60.

    PubMed  PubMed Central  Google Scholar 

  • *Texas 21st Century Community: Learning Centers 2014–15 Evaluation Report. (2016). https://tea.texas.gov/Reports_and_Data/Program_Evaluations/Out-of-School_Learning_Opportunities/Program_Evaluation__Out-of-School_Learning_Opportunities/.

  • *Towler, S. M. (2015). A comparison of academic achievement of urban students in 21st CCLC after school programs to students not enrolled (Ph.D.). United States—Minnesota: Capella University. https://search.proquest.com/docview/1666453749/abstract/A46F1E47C03640C4PQ/1.

  • Zief, S., Lauver, S., Maynard, R. A., Zief, S. G., & Park, A. (2006). Impacts of after-school programs on student outcomes. Campbell Systematic Reviews, 3, 1–56.

    Article  Google Scholar 

Download references

Acknowledgements

Support for this project is provided by Virginia Commonwealth University—School of Education. Additionally, thank you to our lab mates and colleagues for their feedback and moral support.

Authors’ Contributions

AL conceived of the study, participated in its design and analysis, and drafted the manuscript; JC advised the design, participating in the analysis, and interpretation of the data; TM participated in data coding and helped to draft the manuscript. All authors participated in the statistical analysis, drafting, revision, and approval of the final manuscript.

Data Sharing and Declaration

The datasets generated and/or analyzed during the current study are not publicly available but are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

  1. Virginia Commonwealth University, Oliver Hall, 1015 W. Main St., P.O. Box 842020, Richmond, VA, 23284-2020, USA

    Ashlee M. Lester & Jason C. Chow

  2. University of Virginia, Curry School of Education, 405 Emmet Street South, P.O. Box 400281, Charlottesville, VA, 22904-4260, USA

    Theresa N. Melton

Authors
  1. Ashlee M. Lester
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jason C. Chow
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Theresa N. Melton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashlee M. Lester.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

Given that this research was a quantitative synthesis of publically available literature and data, ethical approval from the authors’ Institutional Review Board was not required.

Informed Consent

This research did not involve human participants and therefore informed consent was not obtained.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lester, A.M., Chow, J.C. & Melton, T.N. Quality is Critical for Meaningful Synthesis of Afterschool Program Effects: A Systematic Review and Meta-analysis. J Youth Adolescence 49, 369–382 (2020). https://doi.org/10.1007/s10964-019-01188-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10964-019-01188-8

Keywords

Search

Navigation