Abstract
Pragmatic instruments with psychometric support are important to advance dissemination and implementation (D&I) research, but few well-researched D&I instruments exist. Item response theory (IRT), an approach that is underutilized in D&I, can help with the development of actionable and brief instruments. This paper provides an overview of IRT for D&I researchers and examines an instrument of therapist attitudes using IRT measurement models. Eight items of the Attitudes Towards Individualized Assessment—Monitoring and Feedback (AIA-MF) Clinical Utility scale were fit to the Graded Response Model in a national sample of master’s level therapists. Various IRT model characteristics including item threshold and discrimination parameters, information, and item and person fit were examined. Discrimination and thresholds parameters showed significant variability across the eight items. Item information curves also showed that each item contributed variably to the total test information, suggesting that items 4 and 5 reliably measure therapist attitudes across the latent continuum and items 3 and 6 warrant further investigation. Results suggest that IRT models can help D&I researchers examine existing instruments with greater specificity than traditional measurement methods, thus increasing measurement precision while lowering response burden, both important considerations for the field.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Researchealth Services Research, 6(2), 61–74. https://doi.org/10.1023/b:mhsr.0000024351.12294.65
Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47(10), 1114. https://doi.org/10.1097/CHI.0b013e3181825af8
Chen, W.-H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289
Cone, J. D. (1997). Issues in functional analysis in behavioral assessment. Behaviour Research and Therapy, 35(3), 259–275. https://doi.org/10.1016/s0005-7967(96)00101-5
de Ayala, R. J. (2009). The theory and practice of item response theory. Guilford Press.
Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x
Drasgow, F., & Lissak, R. I. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied psychology, 68(3), 363
Eccles, M. P., & Mittman, B. S. (2006). Welcome to implementation science. Implementation Science. https://doi.org/10.1186/1748-5908-1-1
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
Embretson, S. E., & Reise, S. P. (2013). Item response theory. Taylor & Francis. https://books.google.com/books?id=9Xm0AAAAQBAJ
Fortney, J. C., Unützer, J., Wrenn, G., Pyne, J. M., Smith, G. R., Schoenbaum, M., & Harbin, H. T. (2017). A tipping point for measurement-based care. Psychiatric Services, 68(2), 179–188. https://doi.org/10.1176/appi.ps.201500439
Glasgow, R. E., & Riley, W. T. (2013). Pragmatic measures: What they are and why we need them. American Journal of Preventive Medicine, 45(2), 237–243. https://doi.org/10.1016/j.amepre.2013.03.010
Glasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012). National Institutes of Health approaches to dissemination and implementation science: Current and future directions. American Journal of Public Health, 102(7), 1274–1281. https://doi.org/10.2105/AJPH.2012.300755
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage Publications Inc.
Haynes, S. N., Mumma, G. H., & Pinson, C. (2009). Idiographic assessment: Conceptual and psychometric foundations of individualized behavioral assessment. Clinical Psychology Review, 29(2), 179–191. https://doi.org/10.1016/j.cpr.2008.12.003
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55
Jensen-Doss, A., Haimes, E. M. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., & Hawley, K. M. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0
Jensen-Doss, A., Smith, A. M., Becker-Haimes, E. M., Ringle, V. M., Walsh, L. M., Nanda, M., Walsh, S. L., Maxwell, C. A., & Lyon, A. R. (2018). Individualized progress measures are more acceptable to clinicians than standardized measures: Results of a national survey. Administration and Policy in Mental Health and Mental Health Services Research, 45(3), 392–403. https://doi.org/10.1007/s10488-017-0833-y
Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301. https://doi.org/10.1093/clipsy.bpg025
Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., & Douglas, S. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.3329
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018). An updated protocol for a systematic review of implementation-related measures. Systematic Reviews, 7(1), 66. https://doi.org/10.1186/s13643-018-0728-3
Lewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2015). The society for implementation research collaboration instrument review project: A methodology to promote rigorous evaluation. Implementation Science, 10(1), 2. https://doi.org/10.1186/s13012-014-0193-x
Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198–1202. https://doi.org/10.1080/01621459.1988.10478722
Lord, F. M. (2012). Applications of item response theory to practical testing problems. Taylor & Francis. https://books.google.com/books?id=1aOchqcSIaUC
Lyon, A. R., Ludwig, K., Wasse, J. K., Bergstrom, A., Hendrix, E., & McCauley, E. (2016). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 122–134. https://doi.org/10.1007/s10488-015-0626-0
Martinez, R. G., Lewis, C. C., & Weiner, B. J. (2014). Instrumentation issues in implementation science. Implementation Science, 9(1), 118. https://doi.org/10.1186/s13012-014-0118-8
Mash, E. J., & Hunsley, J. (2005). Evidence-based assessment of child and adolescent disorders: Issues and challenges. Journal of Clinical Child and Adolescent Psychology, 34(3), 362–379. https://doi.org/10.1207/s15374424jccp3403_1
Orlando, M., & Thissen, D. (2003). Further investigation of the performance of S-X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27(4), 289–298. https://doi.org/10.1177/0146621603027004004
Rabin, B. A., Lewis, C. C., Norton, W. E., Neta, G., Chambers, D., Tobin, J. N., Brownson, R. C., & Glasgow, R. E. (2015). Measurement resources for dissemination and implementation research in health. Implementation Science, 11(1), 42. https://doi.org/10.1186/s13012-016-0401-y
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement. https://doi.org/10.1002/j.2333-8504.1968.tb00153.x
Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice, 22(1), 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010
Thissen, D. (2000). Reliability and measurement precision. In Computerized adaptive testing: A primer (2nd ed., pp. 159–184). Lawrence Erlbaum Associates Publishers.
Toland, M. D. (2014). Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence, 34(1), 120–151. https://doi.org/10.1177/0272431613511332
Weisz, J. R., Chorpita, B. F., Frye, A., Ng, M. Y., Lau, N., Bearman, S. K., Ugueto, A. M., Langer, D. A., & Hoagwood, K. E. (2011). Youth top problems: Using idiographic, consumer-guided assessment to identify treatment needs and to track change during psychotherapy. Journal of Consulting and Clinical Psychology, 79(3), 369. https://doi.org/10.1037/a0023307
Willis, G. B. (2010). Cognitive interviewing: A tool for improving questionnaire design. Sage.
Acknowledgements
We thank the reviewer of this manuscript, Jason E. Chapman, Ph.D., Senior Research Scientist at the Oregon Social Learning Center, whose detailed comments and suggestions helped to substantially improve this work. The peer review process was double blinded, and the authors learned the reviewer’s identity after the manuscript had been accepted for publication.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
None of the authors have conflicts of interest to declare.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Patel, Z.S., Jensen-Doss, A. & Zopluoglu, C. Illustrating the Applicability of IRT to Implementation Science: Examining an Instrument of Therapist Attitudes. Adm Policy Ment Health 48, 921–935 (2021). https://doi.org/10.1007/s10488-021-01139-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-021-01139-1