Skip to main content

Advertisement

SpringerLink
Log in

Validity of the Work Assessment Triage Tool for Selecting Rehabilitation Interventions for Workers’ Compensation Claimants with Musculoskeletal Conditions

  • Published:
Journal of Occupational Rehabilitation Aims and scope Submit manuscript

Abstract

Purpose The Work Assessment Triage Tool (WATT) is a clinical decision support tool developed using machine learning to help select interventions for patients with musculoskeletal disorders. The WATT categorizes patients based on individual characteristics according to likelihood of successful return to work following rehabilitation. A previous validation showed acceptable classification accuracy, but we re-examined accuracy using a new dataset drawn from the same system 2 years later. Methods A population-based cohort design was used, with data extracted from a Canadian compensation database on workers considered for rehabilitation between January 2013 and December 2016. Data were obtained on demographic, clinical, and occupational characteristics, type of rehabilitation undertaken, and return to work outcomes. Analysis included classification accuracy statistics of WATT recommendations. Results The sample included 28,919 workers (mean age 43.9 years, median duration 56 days), of whom 23,124 experienced a positive outcome within 30 days following return to work assessment. Sensitivity of the WATT for selecting successful programs was 0.13 while specificity was 0.87. Overall accuracy was 0.60 while human recommendations were higher at 0.72. Conclusions Overall accuracy of the WATT for selecting successful rehabilitation programs declined in a more recent cohort and proved less accurate than human clinical recommendations. Algorithm revision and further validation is needed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Hartvigsen J, Hancock MJ, Kongsted A, Louw Q, Ferreira ML, Genevay S, et al. What low back pain is and why we need to pay attention. Lancet. 2018;391(10137):2356–2367. https://doi.org/10.1016/S0140-6736(18)30480-X.

    Article  PubMed  Google Scholar 

  2. DALYs GBD, Collaborators H. Global, regional, and national disability-adjusted life-years (DALYs) for 315 diseases and injuries and healthy life expectancy (HALE), 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet. 2016;388(10053):1603–1658. https://doi.org/10.1016/S0140-6736(16)31460-X.

    Article  Google Scholar 

  3. Foster NE, Anema JR, Cherkin D, Chou R, Cohen SP, Gross DP, et al. Prevention and treatment of low back pain: evidence, challenges, and promising directions. Lancet. 2018;391(10137):2368–2383. https://doi.org/10.1016/S0140-6736(18)30489-6.

    Article  PubMed  Google Scholar 

  4. Collie A, Di Donato M, Iles R. Work disability in Australia: an overview of prevalence, expenditure, support systems and services. J Occup Rehabil. 2018. https://doi.org/10.1007/s10926-018-9816-4.

    Article  PubMed  Google Scholar 

  5. Shaw WS, van der Windt DA, Main CJ, Loisel P, Linton SJ. Early patient screening and intervention to address individual-level occupational factors (“blue flags”) in back disability. J Occup Rehabil. 2009;19(1):64–80. https://doi.org/10.1007/s10926-008-9159-7.

    Article  PubMed  Google Scholar 

  6. Steenstra IA, Ibrahim SA, Franche RL, Hogg-Johnson S, Shaw WS, Pransky GS. Validation of a risk factor-based intervention strategy model using data from the readiness for return to work cohort study. J Occup Rehabil. 2009;20(3):394–405. https://doi.org/10.1007/s10926-009-9218-8.

    Article  Google Scholar 

  7. Shaw WS, Linton SJ, Pransky G. Reducing sickness absence from work due to low back pain: how well do intervention strategies match modifiable risk factors? J Occup Rehabil. 2006;16(4):591–605. https://doi.org/10.1007/s10926-006-9061-0.

    Article  PubMed  Google Scholar 

  8. Cote P, Wong JJ, Sutton D, Shearer HM, Mior S, Randhawa K, et al. Management of neck pain and associated disorders: a clinical practice guideline from the Ontario Protocol for Traffic Injury Management (OPTIMa) Collaboration. Eur Spine J. 2016;25(7):2000–2022. https://doi.org/10.1007/s00586-016-4467-7.

    Article  PubMed  Google Scholar 

  9. Wong JJ, Cote P, Shearer HM, Carroll LJ, Yu H, Varatharajan S, et al. Clinical practice guidelines for the management of conditions related to traffic collisions: a systematic review by the OPTIMa Collaboration. Disabil Rehabil. 2015;37(6):471–489. https://doi.org/10.3109/09638288.2014.932448.

    Article  PubMed  Google Scholar 

  10. Qaseem A, Wilt TJ, McLean RM, Forciea MA, Clinical Guidelines Committee of the American College of Physicians. Noninvasive treatments for acute, subacute, and chronic low back pain: a clinical practice guideline from the American College of Physicians. Ann Intern Med. 2017;166(7):514–530. https://doi.org/10.7326/m16-2367.

    Article  PubMed  Google Scholar 

  11. Almeida M, Saragiotto B, Richards B, Maher CG. Primary care management of non-specific low back pain: key messages from recent clinical guidelines. Med J Aust. 2018;208(6):272–275.

    Article  Google Scholar 

  12. Schaafsma F, Schonstein E, Whelan KM, Ulvestad E, Kenny DT, Verbeek JH. Physical conditioning programs for improving work outcomes in workers with back pain. Cochrane Database Syst Rev. 2010;(1):CD001822. https://doi.org/10.1002/14651858.cd001822.pub2.

  13. van Oostrom SH, Driessen MT, de Vet HC, Franche RL, Schonstein E, Loisel P, et al. Workplace interventions for preventing work disability. Cochrane Database Syst Rev. 2009;(1):CD006955. https://doi.org/10.1002/14651858.cd006955.pub2.

  14. Cullen KL, Irvin E, Collie A, Clay F, Gensby U, Jennings PA, et al. Effectiveness of workplace interventions in return-to-work for musculoskeletal, pain-related and mental health conditions: an update of the evidence and messages for practitioners. J Occup Rehabil. 2018;28(1):1–15. https://doi.org/10.1007/s10926-016-9690-x.

    Article  CAS  PubMed  Google Scholar 

  15. Saragiotto BT, de Almeida MO, Yamato TP, Maher CG. Multidisciplinary biopsychosocial rehabilitation for nonspecific chronic low back pain. Phys Ther. 2016;96(6):759–763. https://doi.org/10.2522/ptj.20150359.

    Article  PubMed  Google Scholar 

  16. Kamper SJ, Apeldoorn AT, Chiarotto A, Smeets RJ, Ostelo RW, Guzman J, et al. Multidisciplinary biopsychosocial rehabilitation for chronic low back pain: cochrane systematic review and meta-analysis. BMJ. 2015;350:h444. https://doi.org/10.1136/bmj.h444.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Gross DP, Armijo-Olivo S, Shaw WS, Williams-Whitt K, Shaw NT, Hartvigsen J, et al. Clinical decision support tools for selecting interventions for patients with disabling musculoskeletal disorders: a scoping review. J Occup Rehabil. 2016;26(3):286–318. https://doi.org/10.1007/s10926-015-9614-1.

    Article  PubMed  Google Scholar 

  18. Gross DP, Zhang J, Steenstra I, Barnsley S, Haws C, Amell T, et al. Development of a computer-based clinical decision support tool for selecting appropriate rehabilitation interventions for injured workers. J Occup Rehabil. 2013;23(4):597–609. https://doi.org/10.1007/s10926-013-9430-4.

    Article  PubMed  Google Scholar 

  19. Collins GS, de Groot JA, Dutton S, Omar O, Shanyinde M, Tajar A, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014;14:40. https://doi.org/10.1186/1471-2288-14-40.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Qin Z, Armijo-Olivo S, Woodhouse LJ, Gross DP. An investigation of the validity of the Work Assessment Triage Tool clinical decision support tool for selecting optimal rehabilitation interventions for workers with musculoskeletal injuries. Clin Rehabil. 2016;30(3):277–287. https://doi.org/10.1177/0269215515578696.

    Article  PubMed  Google Scholar 

  21. Johnson AEW, Pollard TJ, Mark RG. Reproducibility in critical care: a mortality prediction case study. Proc Mach Learn Res. 2017;68:361–376.

    Google Scholar 

  22. Pollard CA. Preliminary validity study of the Pain Disability Index. Percept Mot Skills. 1984;59(3):974.

    Article  CAS  Google Scholar 

  23. Finch E, Brooks D, Stratford P, Mayo N. Physical rehabilitation outcome measures: a guide to enhanced clinical decision making. 2nd ed. Toronto: Canadian Physiotherapy Association; 2002.

    Google Scholar 

  24. McHorney CA, Ware JE Jr, Raczek AE. The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Med Care. 1993;31(3):247–263.

    Article  CAS  Google Scholar 

  25. Zhao M. K-fold cross-validation for improving medical classification accuracy and model selection in K-nearest neighbors classifiers. Basic Clin Pharmacol. 2016;118(Suppl 1):107.

    Google Scholar 

  26. Žliobaitė I, Pechenizkiy M, Gama J. An overview of concept drift applications. In: Japkowicz N, Stefanowski J, editors. Big data analysis: new algorithms for a new society. Cham: Springer; 2016. p. 91–114.

    Chapter  Google Scholar 

  27. Gama J, Žliobaitė I, Bifet A, Pechenizkiy M, Bouchachia A. A survey on concept drift adaptation. ACM Comput Surv. 2014;46(4):1–37. https://doi.org/10.1145/2523813.

    Article  Google Scholar 

  28. Webb GI, Lee LK, Goethals B, Petitjean F. Analyzing concept drift and shift from sample data. Data Min Knowl Discov. 2018;32(5):1179–1199. https://doi.org/10.1007/s10618-018-0554-1.

    Article  Google Scholar 

  29. Widmer G, Kubat M. Learning in the presence of concept drift and hidden contexts. Mach Learn. 1996;23(1):69–101. https://doi.org/10.1023/a:1018046501280.

    Article  Google Scholar 

  30. Gama J, Castillo G. Learning with local drift detection. In: Li X, Zaïane OR, Li Z, editors. Advanced data mining and applications. ADMA 2006. Lecture notes in computer science. Vol. 4093. Springer, Berlin, Heidelberg; 2006.

  31. Ambrosino R, Buchanan BG. The use of physician domain knowledge to improve the learning of rule-based models for decision-support. Proc AMIA Symp. 1999;192–196.

    Google Scholar 

  32. Lee CH, Yoon HJ. Medical big data: promise and challenges. Kidney Res Clin Pract. 2017;36(1):3–11. https://doi.org/10.23876/j.krcp.2017.36.1.3.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Dinov ID. Methodological challenges and analytic opportunities for modeling and interpreting Big Healthcare Data. Gigascience. 2016. https://doi.org/10.1186/s13742-016-0117-6.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Sinha A, Hripcsak G, Markatou M. Large datasets in biomedicine: a discussion of salient analytic issues. J Am Med Inform Assoc. 2009;16(6):759–767. https://doi.org/10.1197/jamia.M2780.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Taranu I. Data mining in healthcare: decision making and precision. Database Syst J. 2015;6(4):33–40.

    Google Scholar 

  36. Iles RA, Davidson M, Taylor NF, O’Halloran P. Systematic review of the ability of recovery expectations to predict outcomes in non-chronic non-specific low back pain. J Occup Rehabil. 2009;19(1):25–40. https://doi.org/10.1007/s10926-008-9161-0.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The Workers’ Compensation Board of Alberta assisted with data collection.

Funding

Funding was provided by the Workers’ Compensation Board of Alberta.

Author information

Authors and Affiliations

  1. Department of Physical Therapy, University of Alberta, 2-50 Corbett Hall, Edmonton, AB, T6G 2G4, Canada

    Douglas P. Gross

  2. Morneau Shepell, Toronto, Canada

    Ivan A. Steenstra

  3. Department of Medicine, University of Connecticut School of Medicine, Farmington, CT, USA

    William Shaw

  4. Department of Computing Science, University of Alberta, Edmonton, Canada

    Parnian Yousefi & Osmar Zaïane

  5. National Research Council of Canada, Ottawa, Canada

    Colin Bellinger

Authors
  1. Douglas P. Gross
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivan A. Steenstra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William Shaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Parnian Yousefi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Colin Bellinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Osmar Zaïane
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Douglas P. Gross.

Ethics declarations

Conflict of interest

The research team received a research grant from the Workers’ Compensation Board of Alberta but otherwise declares that they have no relevant conflicts of interest. Douglas Gross declares that he has no conflict of interest. Ivan A. Steenstra declares that he has no conflict of interest. William Shaw declares that he has no conflict of interest. Parnian Yousefi declares that she has no conflict of interest. Colin Bellinger declares that he has no conflict of interest. Osmar Zaïane declares that he has no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

This study was an analysis of an administrative dataset therefore informed consent for the study was not obtained.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 15 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gross, D.P., Steenstra, I.A., Shaw, W. et al. Validity of the Work Assessment Triage Tool for Selecting Rehabilitation Interventions for Workers’ Compensation Claimants with Musculoskeletal Conditions. J Occup Rehabil 30, 318–330 (2020). https://doi.org/10.1007/s10926-019-09843-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10926-019-09843-4

Keywords

Search

Navigation