Skip to main content
SpringerLink
Log in

Teacher Competence of Delivery of BEST in CLASS as a Mediator of Treatment Effects

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

This study investigated if training and practice-based coaching in an evidence-based program was associated with higher observed treatment integrity (adherence and competence) and if these treatment integrity components were associated with teacher report of child behavioral outcomes in the BEST in CLASS efficacy trial. Participants were 462 children (M = 4.32 years, SD = 0.53; 65% male; 17.0% Caucasian, 66.0% African-American, 5.0% Hispanic, and 12.0% other) identified as having problem behavior and their 185 teachers (M = 12.09 years teaching experience; 99% female; 47.0% Caucasian, 48.0% African-American, 1.0% Asian/Pacific Islander, 3.0% Hispanic and 1.0% other). Teachers and focal children were randomly assigned to the intervention (teacher n = 92, children n = 230) or control condition (teacher n = 93, child n = 232). Results of a multilevel mediation analysis indicated that the BEST in CLASS intervention had a positive effect on teacher report of child problem behavior (SSIS-RS) and externalizing problems (C-TRF), as well as having a positive effect on teachers’ adherence and competence of delivery of the intervention. There was an indirect effect through competence of delivery for externalizing problems, but not problem behavior. No indirect effects for adherence were found. Implications of these findings and directions for future research are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Two items assess Rules: 3–5 rules are visible in classroom and Teacher reviews rules, addresses rule violations. Only the second item was included in the current study as we focused on teacher-delivered practices.

References

  • Abry, T., Rimm-Kaufman, S. E., Larsen, R. A., & Brewer, A. J. (2013). The influence of fidelity of implementation on teacher–student interaction quality in the context of a randomized controlled trial of the Responsive Classroom approach. Journal of School Psychology, 51(4), 437–453. doi:10.1016/j.jsp.2013.03.001.

    Article  PubMed  Google Scholar 

  • Achenbach, T. M., & Rescorla, L. A. (2000). ASEBA preschool forms & profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth and Families.

    Google Scholar 

  • Allen, J. D., Linnan, L. A., & Emmons, K. M. (2012). Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. Dissemination and Implementation Research in Health: Translating Science to Practice. doi:10.1093/acprof:oso/9780199751877.003.0014.

    Google Scholar 

  • Barbarin, O. A. (2007). Mental health screening of preschool children: validity and reliability of ABLE. American Journal of Orthopsychiatry, 77(3), 402–408. doi:10.1037/0002-9432.77.3.402.

    Article  PubMed  Google Scholar 

  • Barber, J. P., Crits-Christoph, P., & Luborsky, L. (1996). Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology, 64(3), 619–622.

    Article  PubMed  Google Scholar 

  • Berry, V., Axford, N., Blower, S., Taylor, R. S., Edwards, R. T., Tobin, K., et al. (2016). The effectiveness and micro-costing analysis of a universal, school-based, social–emotional learning programme in the UK: A cluster-randomised controlled trial. School Mental Health, 8(2), 238–256. doi:10.1007/s12310-015-9160-1.

    Article  Google Scholar 

  • Bierman, K. L., Coie, J., Dodge, K., Greenberg, M., Lochman, J., McMohan, R., et al. (2013). School outcomes of aggressive–disruptive children: Prediction from kindergarten risk factors and impact of the fast track prevention program. Aggressive Behavior, 39(2), 114–130. doi:10.1002/ab.21467.

    Article  PubMed  Google Scholar 

  • Brauner, C. B., & Stephens, C. B. (2006). Estimating the prevalence of early childhood serious emotional/behavioral disorders: Challenges and recommendations. Public Health Reports, 121(3), 303–310. doi:10.1177/003335490612100314.

    Article  PubMed  PubMed Central  Google Scholar 

  • Brennan, E. M., Bradley, J. R., Allen, M. D., & Perry, D. F. (2008). The evidence base for mental health consultation in early childhood settings: Research synthesis addressing staff and program outcomes. Early Education and Development, 19(6), 982–1022. doi:10.1080/10409280801975834.

    Article  Google Scholar 

  • Bulotsky-Shearer, R. J., Bell, E. R., & Domínguez, X. (2012). Latent profiles of problem behavior within learning, peer, and teacher contexts: Identifying subgroups of children at academic risk across the preschool year. Journal of School Psychology, 50(6), 775–798. doi:10.1016/j.jsp.2012.08.001.

    Article  PubMed  Google Scholar 

  • Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57(3), 225–238. doi:10.1016/S0376-8716(99)00049-6.

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. doi:10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  • Conroy, M. A., & Sutherland, K. S. (2011). Promoting social, emotional, and behavioral competence in young high-risk children: A preventative classroom-based early intervention model. Unpublished manual, Virginia Commonwealth University, Richmond, VA.

  • Conroy, M. A., Sutherland, K. S., Algina, J. J., Wilson, R. E., Martinez, J. R., & Whalon, K. J. (2015). Measuring teacher implementation of the intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23(3), 144–155.

  • Conroy, M. A., Sutherland, K. S., Vo, A. K., Carr, S., & Ogston, P. L. (2014). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions, 16(2), 81–92. doi:10.1177/1098300713478666.

  • Crits-Christoph, P., Siqueland, L., Chittams, J., Barber, J. P., Beck, A. T., Frank, A., et al. (1998). Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. Journal of Consulting and Clinical Psychology, 66(3), 484–492. doi:10.1037//0022-006X.66.3.484.

    Article  PubMed  Google Scholar 

  • Domitrovich, C. E., Cortes, R. C., & Greenberg, M. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. The Journal of Primary Prevention, 28(2), 67–91. doi:10.1007/s10935-007-0081-0.

    Article  PubMed  Google Scholar 

  • Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & DeRousie, R. M. S. (2010). Implementation quality: Lessons learned in the context of the Head Start REDI trial. Early Childhood Research Quarterly, 25(3), 284–298. doi:10.1016/j.ecresq.2010.04.001.

    Article  PubMed  PubMed Central  Google Scholar 

  • Doumen, S., Verschueren, K., Buyse, E., Germeijs, V., Luyckx, K., & Soenens, B. (2008). Reciprocal relations between teacher–child conflict and aggressive behavior in kindergarten: A three-wave longitudinal study. Journal of Clinical Child & Adolescent Psychology, 37(3), 588–599. doi:10.1016/j.jsp.2007.06.009.

    Article  Google Scholar 

  • Durlak, J. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education”. Early Childhood Research Quarterly, 25(3), 348–357. doi:10.1016/j.ecresq.2010.03.003.

    Article  Google Scholar 

  • Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127. doi:10.1007/s11121-015-0606-3.

    Article  PubMed  Google Scholar 

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. doi:10.1007/s10464-008-9165-0.

    Article  PubMed  Google Scholar 

  • Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. doi:10.1111/j.1467-8624.2010.01564.x.

    Article  PubMed  Google Scholar 

  • Feeley, M., DeRubeis, R. J., & Gelfand, L. A. (1999). The temporal relation of adherence and alliance to symptom change in cognitive therapy for depression. Journal of Consulting and Clinical Psychology, 67(4), 578–582.

    Article  PubMed  Google Scholar 

  • Feil, E. G., Severson, H. H., & Walker, H. M. (1998). Screening for emotional and behavioral delays the Early Screening Project. Journal of Early Intervention, 21(3), 252–266. doi:10.1177/105381519802100306.

    Article  Google Scholar 

  • Feil, E. G., Small, J. W., Forness, S. R., Serna, L. A., Kaiser, A. P., Hancock, T. B., et al. (2005). Using different measures, informants, and clinical cut-off points to estimate prevalence of emotional or behavioral disorders in preschoolers: Effects on age, gender, and ethnicity. Behavioral Disorders, 30(4), 375–391.

    Article  Google Scholar 

  • Fox, L., Hemmeter, M. L., Snyder, P., Binder, D. P., & Clarke, S. (2011). Coaching early childhood special educators to implement a comprehensive model for promoting young children’s social competence. Topics in Early Childhood Special Education, 31(3), 178–192. doi:10.1177/0271121411404440.

    Article  Google Scholar 

  • Gilliam, W. S., & Marchesseault, C. M. (2005). From capitols to classrooms, policy to practice: State funded Pre-k at the classroom level. New Haven, CT: Yale University Child Study Center.

    Google Scholar 

  • Goncy, E. A., Sutherland, K. S., Farrell, A. D., Sullivan, T. N., & Doyle, S. T. (2015). Measuring teacher implementation in delivery of a bullying prevention program: The impact of instructional and procedural adherence and competence on student responsiveness. Prevention Science, 16(3), 440–450. doi:10.1007/s11121-014-0508-9.

    Article  PubMed  Google Scholar 

  • Greenberg, M. T. (2010). School-based prevention: Current status and future challenges. Effective Education, 2(1), 27–52. doi:10.1080/19415531003616862.

    Article  Google Scholar 

  • Gresham, F., & Elliott, S. N. (2008). Social skills improvement system (SSIS) rating scales. Bloomington, MN: Pearson Assessments.

    Google Scholar 

  • Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., et al. (2010). Implementation fidelity of My Teaching Partner literacy and language activities: Association with preschoolers’ language and literacy growth. Early Childhood Research Quarterly, 25(3), 329–347. doi:10.1016/j.ecresq.2009.07.002.

    Article  Google Scholar 

  • Hamre, B. K., & Pianta, R. C. (2001). Early teacher–child relationships and the trajectory of children’s school outcomes through eighth grade. Child Development, 72(2), 625–638. doi:10.1111/1467-8624.00301.

    Article  PubMed  Google Scholar 

  • Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children, 79(2), 181–193. doi:10.1177/001440291307900204.

    Article  Google Scholar 

  • Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. doi:10.3102/1076998606298043.

    Article  Google Scholar 

  • Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76(4), 544–555. doi:10.1037/0022-006X.76.4.544.

    Article  PubMed  PubMed Central  Google Scholar 

  • LoCasale-Crouch, J., Cabell, S. Q., Jimenez, M., & Taylor, M. (2014). Research on consultation in early childhood programs. Handbook of research in school consultation. doi:10.4324/9780203133170.ch19.

    Google Scholar 

  • Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthen, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. doi:10.1037/a0012869.

    Article  PubMed  Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73–84. doi:10.1037/a0018121.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., Rodriguez, A., Quinoy, A., Arnold, C., Kendall, P. C., et al. (2016). Development and initial psychometrics for a therapist competence instrument for CBT for youth anxiety. Journal of Clinical Child and Adolescent Psychology. doi:10.1080/15374416.2016.1253018.

    PubMed  Google Scholar 

  • Mendel, P., Meredith, L. S., Schoenbaum, M., Sherbourne, C. D., & Wells, K. B. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 21–37. doi:10.1007/s10488-007-0144-9.

    Article  PubMed  Google Scholar 

  • Muthén, L. K. & Muthén, B. O. (1998–2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

  • Newborg, J. (2005). Battelle developmental inventory, Second edition: Examiner’s manual. Itasca, IL: Riverside.

    Google Scholar 

  • Pituch, K. A., Stapleton, L. M., & Kang, J. Y. (2006). A comparison of single sample and bootstrap methods to assess mediation in cluster randomized trials. Multivariate Behavioral Research, 41(3), 367–400. doi:10.1207/s15327906mbr4103_5.

    Article  PubMed  Google Scholar 

  • Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. doi:10.1007/s10488-010-0319-7.

    Article  PubMed  Google Scholar 

  • Qi, C. H., & Kaiser, A. P. (2003). Behavior problems of preschool children from low-income families: Review of the literature. Topics in Early Childhood Special Education, 23(4), 188–216. doi:10.1177/02711214030230040201.

    Article  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Thousand Oaks: Sage.

    Google Scholar 

  • Roorda, D. L., Verschueren, K., Vancraeyveldt, C., Van Craeyevelt, S., & Colpin, H. (2014). Teacher–child relationships and behavioral adjustment: Transactional links for preschool boys at risk. Journal of School Psychology, 52(5), 495–510. doi:10.1016/j.jsp.2014.06.004.

    Article  PubMed  Google Scholar 

  • Sameroff, A. J., & Mackenzie, M. J. (2003). Research strategies for capturing transactional models of development: The limits of the possible. Development and Psychopathology, 15(03), 613–640. doi:10.1017/S0954579403000312.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K. (2012). The transport and diffusion of multisystemic therapy. In Dissemination and implementation of evidence-based psychological interventions (pp. 227–246). doi:10.1093/med:psych/9780195389050.003.0012.

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. doi:10.1007/s10488-010-0321-0.

    Article  PubMed  Google Scholar 

  • Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., et al. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78(5), 751. doi:10.1037/a0020569.

    Article  PubMed  Google Scholar 

  • Snyder, P. A., Hemmeter, M. L., & Fox, L. (2015). Supporting implementation of evidence-based practices through practice-based coaching. Topics in Early Childhood Special Education, 35(3), 133–143. doi:10.1177/0271121415594925.

    Article  Google Scholar 

  • Snyder, P., Hemmeter, M. L., Meeker, K. A., Kinder, K., Pasia, C., & McLaughlin, T. (2012). Characterizing key features of the early childhood professional development literature. Infants & Young Children, 25(3), 188–212. doi:10.1097/IYC.0b013e31825a1ebf.

    Article  Google Scholar 

  • Stapleton, L. M., Pituch, K. A., & Dion, E. (2015). Standardized effect size measures for mediation analysis in cluster-randomized trials. Journal of Experimental Education, 83(4), 547–582. doi:10.1080/00220973.2014.919569.

    Article  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2017). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Manuscript submitted for publication.

  • Sutherland, K. S., Conroy, M. A., Vo, A., & Ladwig, C. (2015). Implementation integrity of practice-based coaching: Preliminary results from the BEST in CLASS efficacy trial. School Mental Health, 7(1), 21–33.

  • Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L. M., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS adherence and competence scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259.

  • Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. doi:10.1177/1053815113515025.

    Article  Google Scholar 

  • Tofighi, D., & MacKinnon, D. P. (2011). RMediation: An R package for mediation analysis confidence intervals. Behavior Research Methods, 43(3), 692–700. doi:10.3758/s13428-011-0076-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wanless, S. B., & Domitrovich, C. E. (2015). Readiness to implement school-based social–emotional learning interventions: Using research on factors related to implementation to maximize quality. Prevention Science, 16(8), 1037–1043. doi:10.1007/s11121-015-0612-5.

    Article  PubMed  Google Scholar 

  • Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200–211. doi:10.1037/a0018912.

    Article  PubMed  PubMed Central  Google Scholar 

  • Webster-Stratton, C. (1997). From parent training to community building. Families in Society: The Journal of Contemporary Social Services, 78(2), 156–171. doi:10.1606/1044-3894.755.

    Article  Google Scholar 

  • Webster-Stratton, C., Reid, M. J., & Stoolmiller, M. (2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. Journal of Child Psychology and Psychiatry, 49(5), 471–488. doi:10.1111/j.1469-7610.2007.01861.x.

    Article  PubMed  Google Scholar 

Download references

Funding

This research was supported by a Grant (R324A110173) from the U. S. Department of Education, Institute for Education Sciences, with additional support from an NIH/NCATS Clinical and Translational Science Award to the University of Florida UL1 TR000064. The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U. S. Department of Education.

Author information

Authors and Affiliations

  1. Virginia Commonwealth University, Richmond, VA, USA

    Kevin S. Sutherland, Bryce D. McLeod & Eleanor Wu

  2. University of Florida, Gainesville, FL, USA

    Maureen A. Conroy & James Algina

Authors
  1. Kevin S. Sutherland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maureen A. Conroy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bryce D. McLeod
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. James Algina
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eleanor Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kevin S. Sutherland.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All study procedures involving human participants were in accordance with the ethical standards of the researchers’ Institutional Review Boards and with the 1964 Helsinki declaration and its later amendments of comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sutherland, K.S., Conroy, M.A., McLeod, B.D. et al. Teacher Competence of Delivery of BEST in CLASS as a Mediator of Treatment Effects. School Mental Health 10, 214–225 (2018). https://doi.org/10.1007/s12310-017-9224-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-017-9224-5

Keywords

Search

Navigation