Abstract
Two studies examined inter-rater reliability and content-related validity of an addiction treatment level of care determination tool currently in use in New York, the LOCADTR 3.0. The studies occurred after tool implementation. In study 1, 139 providers used the LOCADTR 3.0 to determine level of care for four case vignettes. Inter-rater reliability coefficients were calculated. In study 2, 387,338 state records from existing data were analyzed to determine how often providers opted to override the LOCADTR 3.0 level of care determination by choosing an alternative level of care. In study 1, an acceptable inter-rater reliability (IRR = .57–.59) was found. Good indication of content-related validity was also found; participants chose the same level of care the study team chose for each vignette 80% of the time. In study 2, the override option was selected only 10% of the time, further establishing the content validity of the tool. These studies provide evidence for acceptable preliminary reliability and validity of the LOCADTR 3.0.
This is a preview of subscription content, log in via an institution to check access.
References
Smith VK, Gifford K, Ellis E, et al. Medicaid Reforms to Expand Coverage, Control Costs and Improve Care: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2015 and 2016. Health Mangagment Services and Kaiser Permanente, 2015.
Quinn AE, Reif S, Merrick EL, et al. How Do Private Health Plans Manage Specialty Behavioral Health Treatment Entry and Continuing Care? Psychiatric Services. 2017;68(9):931–937.
Alam DA, Martorana A. Addiction treatment: level of care determination. Primary Care. 2011;38(1):125–136.
Bickman L, Karver MS, Schut LJ. Clinician reliability and accuracy in judging appropriate level of care. Journal of Consulting and Clinical Psychology. 1997;65(3):515–520.
Sowers W, Pumariega A, Huffine C, et al. Level-of-care decision making in behavioral health services: the LOCUS and the CALOCUS. Psychiatric Services. 2003;54(11):1461–1463.
Merkx MJ, Schippers GM, Koeter MW, et al. Predictive validity of treatment allocation guidelines on drinking outcome in alcohol-dependent patients. Addictive Behaviors. 2013;38(3):1691–1698.
Gregoire TK. Factors associated with level of care assignment in substance abuse treatment. Journal of Subtance Abuse Treatment. 2000;18(3):241–248.
Lyons JS, Abraham ME. Designing Level of Care Criteria. In: LJ Kiser, PM Lefkovitz, LL Kennedy (Eds). The Integrated Behavioral Health Continuum: Theory and Practice. Arlington, VA: America Psychiatric Association, 2001, pp. 91–107.
Rohrig J, Buchholz A, Wahl S, et al. Placement matching for patients with alcohol use disorders using standardized assessment: a pilot study. Journal of Substance Use. 2015;20(2):97–105.
Lopez-Goni JJ, Fernandez-Montalvo J, Arteaga A, et al. Searching objective criteria for patient assignment in addiction treatment. Journal of Substance Abuse Treatment. 2017;76:28–35.
Magura S, Staines G, Kosanke N, et al. Predictive validity of the ASAM Patient Placement Criteria for naturalistically matched vs. mismatched alcoholism patients. The American Journal of Addictions. 2003;12(5):386–397.
Stallvik M, Gastfriend DR, Nordahl HM. Matching patients with substance use disorder to optimal level of care with the ASAM Criteria software. J Subst Use. 2015;20(6):389–398.
Galanter M, Keller DS, Dermatis H, et al. The impact of managed care on substance abuse treatment: a report of the American Society of Addiction Medicine. Journal of Addictive Diseases. 2000;19(3):13–34.
Coleman M, Schnapp W, Hurwitz D, et al. Overview of publicly funded managed behavioral health care. Adminstration and Policy in Mental Health and Mental Health Services Research. 2005;32(4):321–340.
Chang CF, Kiser LJ, Bailey JE, et al. Tennessee’s failed managed care program for mental health and substance abuse services. The Journal of American Medical Association. 1998;279(11):864–869.
McFarland BH, McCamant LE, Barron NM. Outcomes for clients of public substance abuse treatment programs before and after Medicaid managed care. Journal of Substance Abuse Treatment. 2005;28(2):149–157.
Ettner SL, Argeriou M, McCarty D, et al. How did the introduction of managed care for the uninsured in Iowa affect the use of substance abuse services? The Journal of Behavioral Health Services and Research.. 2003;30(1):26-40.
Sowers W, George C, Thompson K. Level of care utilization system for psychiatric and addiction services (LOCUS): a preliminary assessment of reliability and validity. Community Health Journal. 1999;35(6):545–563.
Baker SL, Gastfriend DR. Reliability of multidimensional substance abuse treatment matching: implementing the ASAM Patient Placement Criteria. Journal of Addictive Diseases. 2003;22 Suppl 1:45–60.
Willis GB. Cognitive Interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: SAGE Publications; 2004.
Kearney MH. Under-Appreciated Steps in Instrument Development, Part I: Starting With Validity. Research in Nursing and Health. 2016;39:81–83.
Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology. 2012;8(1):23–34.
Hayes AF, Hayes AF, Krippendorff K. Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures. 2007;1(1):77–89.
AgreeStat 2015.1 for Excel Windows/Mac User’s Guide [computer program]. Maryland, 2015.
Fleiss JL. Statistical Methods for Weights and Proportions. Hoboken, New Jersey: John Wiley & Sons; 1981.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest Statement
The authors declare that they have no conflicts of interest.
Rights and permissions
About this article
Cite this article
O’Grady, M.A., Neighbors, C.J., Lincourt, P. et al. Development and Testing of an Addiction Treatment Level of Care Determination Tool. J Behav Health Serv Res 46, 487–496 (2019). https://doi.org/10.1007/s11414-018-9623-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11414-018-9623-x