Skip to main content

Advertisement

SpringerLink
Log in

A Pilot Study Examining Outcomes Associated with the Implementation of Progress Monitoring at a Substance Use Disorder Treatment Program for Adolescents

  • Original Paper
  • Published:
Child & Youth Care Forum Aims and scope Submit manuscript

Abstract

Background

Progress monitoring (PM) is the periodic and reliable assessment of client progress to evaluate and inform psychotherapeutic treatment. PM’s use in a variety of treatment contexts show improved treatment outcome, dropout rate reduction, moderated treatment deterioration, and more efficient treatment delivery.

Objective

This study reports the preliminary results of a PM implementation initiative at a youth care program that focuses on substance use disorder treatment.

Methods

Participants were 70 adolescent clients (mean age 16.6 years, 56% male, 58% White, 30% Aboriginal or Mètis). A time series evaluation design guided the study, where data were collected at admission, periodic intervals during treatment, and at discharge. Descriptive analysis was performed on all demographic variables. Pairwise t tests and effect sizes were calculated to determine clinically significant change.

Results

PM was successfully implemented in the youth care program and the outcomes associated with PM were statistically and clinically significant with large effect sizes reported. Changes in the development of mindfulness related-skills were positively associated with changes in outcome. Case studies illustrate how PM can identify clients who are not responding to treatment.

Conclusion

PM in adolescent substance use treatment programs is an important consideration for program providers. Implementation needs to involve staff, be flexible, empower clients, and be integrated into the culture of programs. The successful implementation of PM can help identify clients who are not responding to treatment and generate useful and reliable outcome data. Recommendations for how PM can be implemented in similar youth care programs are also discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • American Psychological Association. (2009). Criteria for the evaluation of quality improvement programs and the use of quality improvement data. American Psychologist, 64, 551–557. https://doi.org/10.1037/a0016744.

    Article  Google Scholar 

  • Base Camp. (2017). Retrieved November 20, 2017, from http://www.enviros.org/programmenu/base-camp/.

  • Bedard, R. M., Rosen, L. A., & Vacha-Haase, T. (2003). Wilderness therapy programs for juvenile delinquents: A meta-analysis. Journal of Therapeutic Wilderness Camping, 3, 7–13.

    Google Scholar 

  • Bettmann, J. E., Gillis, H. L., Speelman, E. A., Parry, K. J., & Case, J. M. (2016). A meta-analysis of wilderness therapy outcomes for private pay clients. Journal of Child and Family Studies, 25(9), 2659–2673.

    Article  Google Scholar 

  • Bickman, L., Kelley, S. D., Breda, C., Regina de Andrade, A., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011.

    Article  PubMed  Google Scholar 

  • Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research: Journal of the Society for Psychotherapy Research. https://doi.org/10.1080/10503307.2013.817696.

    Google Scholar 

  • Bowen, D. J., & Neill, J. T. (2013). A meta-analysis of adventure therapy outcomes and moderators. The Open Psychology Journal, 6, 28–53.

    Article  Google Scholar 

  • Burlingame, G., Cox, J., Wells, G., Latkowski, M., Justice, D., Carter, C., et al. (2005). The administration and scoring manual of the Youth Outcome Questionnaire. Salt Lake City, Utah: OQ Measures.

    Google Scholar 

  • Carlier, I. V., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J. A., & Zitman, F. G. (2012). ROM and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18, 104–110. https://doi.org/10.1111/j.1365-2753.2010.01543.x.

    Article  PubMed  Google Scholar 

  • Cason, D., & Gillis, H. L. (1994). A meta-analysis of outdoor adventure programming with adolescents. Journal of Experiential Education, 17, 40–47.

    Article  Google Scholar 

  • Duncan, B. L., & Shaw, G. B. (2012). The partners for change outcome management system (PCOMS): The heart and soul of change project. Canadian Psychology/Psychologie Canadienne, 53(2), 93–104. https://doi.org/10.1037/a0027762.

    Article  Google Scholar 

  • Fernandez, A. C., Wood, M. D., Stein, L. A. R., & Rossi, J. S. (2010). Measuring mindfulness and examining its relationship with alcohol use and negative consequences. Psychology of Addictive Behaviors : Journal of the Society of Psychologists in Addictive Behaviors, 24(4), 608–616.

    Article  Google Scholar 

  • Gass, M., Gillis, H. L., & Russell, K. C. (2012). Adventure therapy: Theory, practice and research. New York: Routledge Mental Health Publishing.

    Google Scholar 

  • Gillis, H. L., & Speelman, E. (2008). Are challenge (ropes) courses an effective tool? A meta-analysis. Journal of Experiential Education, 31(2), 111–135.

    Article  Google Scholar 

  • Gillis, H. L., Speelman, E., Linville, N., Bailey, E., Kalle, A., Oglesbee, N., et al. (2016). Meta-analysis of treatment outcomes measured by the Y-OQ and Y-OQ SR comparing wilderness and non-wilderness treatment programs. Child & Youth Care Forum, 45(6), 851–863. https://doi.org/10.1007/s10566-016-9360-3.

    Article  Google Scholar 

  • Gironda, M. A. (2000). A validity study of the Youth-Outcome Questionnaire and the Ohio Scales. Dissertation Abstracts International: Section B: The Sciences & Engineering, 61(5-B).

  • Goldberg, S. B., Babins-Wagner, R., Rousmaniere, T., Berzins, S., Hoyt, W. T., Whipple, J. L., et al. (2016). Creating a climate for therapist improvement: A case study of an agency focused on outcomes and deliberate practice. Psychotherapy, 53(3), 367–375. https://doi.org/10.1037/pst0000060.

    Article  PubMed  Google Scholar 

  • Goodman, J. D., McKay, J. R., & DePhilippis, D. (2013). Progress monitoring in mental health and addiction treatment: A means of improving care. Professional Psychology: Research and Practice, 44(4), 231–246. https://doi.org/10.1037/a0032605.

    Article  Google Scholar 

  • Greco, L., Baer, R. A., & Smith, G. T. (2011). Assessing mindfulness in children and adolescents: Development and validation of the child and adolescent mindfulness measure (CAMM). Psychological Assessment, 23, 606–614.

    Article  PubMed  Google Scholar 

  • Harris, N., Brazeau, J. N. R., Rawana, E. P., Brownlee, K., & Mazmanian, D. (2016). A preliminary examination of a strengths-based treatment for adolescent substance use issues. Canadian Journal of Counselling and Psychotherapy (Online), 50(2), 145–163.

    Google Scholar 

  • Hatfield, D., McCullough, L., Frantz, S. H., & Krieger, K. (2010). Do we know when our clients get worse? An investigation of therapists’ ability to detect negative client change. Clinical Psychology and Psychotherapy, 17, 25–32. https://doi.org/10.1002/cpp.656.

    PubMed  Google Scholar 

  • Hattie, J., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and outward bound: Out-of-class experiences that make a lasting difference. Review of Educational Research, 67, 43–87.

    Article  Google Scholar 

  • Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51(10), 1059–1064.

    Article  PubMed  Google Scholar 

  • IBM. (2017). IBM: How to cite IBM SPSS statistics or earlier versions of SPSS. http://www-01.ibm.com/support/docview.wss?uid=swg21476197. Accessed June 21, 2017.

  • Jacobsen, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19.

    Article  Google Scholar 

  • Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, and feedback in clinical practice. Washington, DC: American Psychological Association.

    Book  Google Scholar 

  • Lambert, M. J. (2017). Maximizing psychotherapy outcome beyond evidence-based medicine. Psychotherapy and Psychosematics, 86, 80–89. https://doi.org/10.1159/000455170.

    Article  Google Scholar 

  • Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback. Psychotherapy, 48(1), 72–79. https://doi.org/10.1037/a0022238.

    Article  PubMed  Google Scholar 

  • Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301.

    Google Scholar 

  • Lee, B. R., & Barth, R. (2011). Defining group care programs: An index of reporting standards. Child & Youth Care Forum, 40, 253–266. https://doi.org/10.1007/s10566-011-9143-9.

    Article  Google Scholar 

  • Levin, M. E., Dalrymple, K., & Zimmerman, M. (2014). Which facets of mindfulness predict the presence of substance use disorders in an outpatient psychiatric sample? Psychology of Addictive Behaviors, 28(2), 498–506.

    Article  PubMed  Google Scholar 

  • Miller, S. D., & Hubble, M. (2011). The road to mastery. Psychotherapy Networker, 35, 22–60.

    Google Scholar 

  • Miller, S. D., Hubble, M. A., Chow, D., & Seidel, J. (2015). Beyond measures and monitoring: Realizing the potential of feedback-informed treatment. Psychotherapy, 52(4), 449–457.

    Article  PubMed  Google Scholar 

  • Park, N. (2009). Building strengths of character: Keys to positive youth development. Reclaiming Children and Youth, 18, 42–47.

    Google Scholar 

  • Patton, M. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Posten, J. M., & Hanson, W. E. (2010). Meta-analysis of psychological assessment as a therapeutic intervention. Psychological Assessment, 22(2), 203–212. https://doi.org/10.1037/a0018679.

    Article  Google Scholar 

  • Russell, K. C., & Farnum, J. (2004). A concurrent model of the wilderness therapy process. Journal of Adventure Education and Outdoor Learning, 4, 39–55.

    Article  Google Scholar 

  • Russell, K. C., & Gillis, H. L. (2017). The Adventure Therapy Experience Scale (ATES): The psychometric properties of a scale to measure the unique factors moderating an adventure therapy experience. Journal of Experiential Education. https://doi.org/10.1177/10538259.

    Google Scholar 

  • Russell, K. C., Gillis, H. L., & Heppner, W. (2015). An examination of mindfulness-based experiences through adventure in substance use disorder treatment for young adult males: A pilot study. Mindfulness, 7(2), 320–328. https://doi.org/10.1007/s12671-015-0441-4.

    Article  Google Scholar 

  • Russell, K. C., Gillis, H. L., & Kivligan, D. (2017, August). Process factors explaining psycho-social outcomes in adventure therapy. Psychotherapy, http://dx.doi.org/10.1037/pst0000047.

  • Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78(3), 298–311. https://doi.org/10.1037/a0019247.

    Article  PubMed  Google Scholar 

  • Solstad, S. M., Castonguay, L. G., & Moltu, C. (2017). Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research. https://doi.org/10.1080/10503307.2017.1326645.

    PubMed  Google Scholar 

  • Sundet, R. (2012). Therapist perspectives on the use of feedback on process and outcome: Patient-focused research in practice. Canadian Psychology/Psychologie Canadienne, 53(2), 122–130. https://doi.org/10.1037/a0027776.

    Article  Google Scholar 

  • Toback, R. L., Graham-Bermann, S. A., & Paresh, D. P. (2016). Outcomes of a character strengths–based intervention on self-esteem and self-efficacy of psychiatrically hospitalized youths. Psychiatric Services, 67(5), 574–577.

    Article  PubMed  Google Scholar 

  • Valenstein, M., Adler, D. A., Berlant, J., Dixon, L. B., Dulit, R. A., Goldman, B., et al. (2009). Implementing standardized assessments in clinical care: Now’s the time. Psychiatric Services, 60, 1372–1375. https://doi.org/10.1176/appi.ps.60.10.1372.

    Article  PubMed  Google Scholar 

  • van Sonsbeek, M. A. M. S., Hutschemaekers, G. G. J. M., Veerman, J. W., & Tiemens, B. B. G. (2014). Effective components of feedback from routine outcome monitoring (ROM) in youth mental health care: Study protocol of a three-arm parallel-group randomized controlled trial. BMC Psychiatry, 14(1), 3. https://doi.org/10.1186/1471-244X-14-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • Weems, C. F. (2011). Guidelines for empirical papers on group care programs. Child & Youth Care Forum, 40, 251. https://doi.org/10.1007/s10566-011-9147-5.

    Article  Google Scholar 

  • Wilson, S. J., & Lipsey, M. W. (2000). Wilderness challenge programs for delinquent youth: A meta-analysis of outcome evaluations. Evaluation and Program Planning, 23, 1–12.

    Article  Google Scholar 

  • Winters, K. C., & Henly, G. A. (1989). Personal experience inventory and manual. Los Angeles: Western Psychological Services.

    Google Scholar 

Download references

Funding

This study was funded by a grant from Health Canada and Alberta Health Services (Grant# 005470). K.C. Russell and H.L. Gillis, received funding to conduct this evaluation from Health Canada and Alberta Health Services (Grant# 005470). L. Law and J. Couillard both work for ENVIROS, the Organization with which the research was coordinated and conducted.

Author information

Authors and Affiliations

  1. Western Washington University, Bellingham, WA, USA

    Keith C. Russell

  2. Georgia College, Milledgeville, GA, USA

    Harold L. Gillis

  3. ENVIROS, Calgary, AB, Canada

    Liam Law & Jeff Couillard

Authors
  1. Keith C. Russell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harold L. Gillis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liam Law
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeff Couillard
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Keith C. Russell.

Ethics declarations

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study and the study was approved by the Institutional Review Board of Georgia College and State University (#656206: Evaluation of Enviros Base Camp Program Co-Principal Investigator: Lee Gillis, Ph.D.).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Russell, K.C., Gillis, H.L., Law, L. et al. A Pilot Study Examining Outcomes Associated with the Implementation of Progress Monitoring at a Substance Use Disorder Treatment Program for Adolescents. Child Youth Care Forum 47, 403–419 (2018). https://doi.org/10.1007/s10566-018-9437-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10566-018-9437-2

Keywords

Search

Navigation