Download PDF
CC BY-NC-ND 4.0 · Appl Clin Inform 2022; 13(01): 315-321
DOI: 10.1055/s-0042-1743241
State of the Art/Best Practice Paper

Building a Learning Health System: Creating an Analytical Workflow for Evidence Generation to Inform Institutional Clinical Care Guidelines

Dev Dash‡   *
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Arjun Gokhale‡   *
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Birju S. Patel
2   Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
,
Alison Callahan
2   Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
,
Jose Posada
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Gomathi Krishnan
2   Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
,
William Collins
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Ron Li
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Kevin Schulman
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Lily Ren
1   Department of Medicine, Stanford University School of Medicine Stanford, California, United States
,
Nigam H. Shah
2   Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
› Author Affiliations
› Further Information
   Permissions and Reprints

Abstract

Background One key aspect of a learning health system (LHS) is utilizing data generated during care delivery to inform clinical care. However, institutional guidelines that utilize observational data are rare and require months to create, making current processes impractical for more urgent scenarios such as those posed by the COVID-19 pandemic. There exists a need to rapidly analyze institutional data to drive guideline creation where evidence from randomized control trials are unavailable.

Objectives This article provides a background on the current state of observational data generation in institutional guideline creation and details our institution's experience in creating a novel workflow to (1) demonstrate the value of such a workflow, (2) demonstrate a real-world example, and (3) discuss difficulties encountered and future directions.

Methods Utilizing a multidisciplinary team of database specialists, clinicians, and informaticists, we created a workflow for identifying and translating a clinical need into a queryable format in our clinical data warehouse, creating data summaries and feeding this information back into clinical guideline creation.

Results Clinical questions posed by the hospital medicine division were answered in a rapid time frame and informed creation of institutional guidelines for the care of patients with COVID-19. The cost of setting up a workflow, answering the questions, and producing data summaries required around 300 hours of effort and $300,000 USD.

Conclusion A key component of an LHS is the ability to learn from data generated during care delivery. There are rare examples in the literature and we demonstrate one such example along with proposed thoughts of ideal multidisciplinary team formation and deployment.

Keywords

learning health system - COVID-19 - practice guidelines

Protection of Human and Animal Subjects

This manuscript discusses development of an analytic process and reports QA results which do not constitute human subjects research.


Author Contributions

N.H.S. oversaw the study. R.L., K.S., and W.C. helped formulate clinical questions and update practice guidelines. G.K. and J.P. performed database queries. B.S.P. and A.C. performed data analysis and summarization. L.R. devised the literature search strategy. D.D. and A.G. performed data validation.


* These authors contributed equally to this work.


Supplementary Material



Publication History

Received: 27 September 2021

Accepted: 06 January 2022

Article published online:
02 March 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Institute of Medicine. Integrating Research and Practice: Health System Leaders Working Toward High-Value Care: Workshop Summary. Washington, DC: The National Academies Press; 2015
    Google Scholar
  • 2 Van Spall HGC, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA 2007; 297 (11) 1233-1240
    CrossrefPubMedGoogle Scholar
  • 3 Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials 2015; 16 (01) 495
    CrossrefPubMedGoogle Scholar
  • 4 Ebell MH, Sokol R, Lee A, Simons C, Early J. How good is the evidence to support primary care practice?. Evid Based Med 2017; 22 (03) 88-92
    CrossrefPubMedGoogle Scholar
  • 5 Tricoci P, Allen JM, Kramer JM, Califf RM, Smith Jr SC. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA 2009; 301 (08) 831-841
    CrossrefPubMedGoogle Scholar
  • 6 Fanaroff AC, Califf RM, Windecker S, Smith Jr SC, Lopes RD. Levels of evidence supporting American College of Cardiology/American Heart Association and European Society of Cardiology Guidelines, 2008-2018. JAMA 2019; 321 (11) 1069-1080
    CrossrefPubMedGoogle Scholar
  • 7 Stewart WF, Shah NR, Selna MJ, Paulus RA, Walker JM. Bridging the inferential gap: the electronic health record and clinical evidence. Health Aff (Millwood) 2007; 26 (02, Suppl 1): w181-w191
    CrossrefPubMedGoogle Scholar
  • 8 Friedman C, Rubin J, Brown J. et al. Toward a science of learning systems: a research agenda for the high-functioning learning health system. J Am Med Inform Assoc 2015; 22 (01) 43-50
    CrossrefPubMedGoogle Scholar
  • 9 Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press; 2013
    Google Scholar
  • 10 HealthIT.gov. Non-federal acute care hospital electronic health record adoption. Accessed August 17, 2021 at: https://www.healthit.gov/data/quickstats/non-federal-acute-care-hospital-electronic-health-record-adoption
    PubMedGoogle Scholar
  • 11 Calzoni L, Clermont G, Cooper GF, Visweswaran S, Hochheiser H. Graphical presentations of clinical data in a learning electronic medical record. Appl Clin Inform 2020; 11 (04) 680-691
    Article in Thieme ConnectPubMedGoogle Scholar
  • 12 Nimmich EB, Bookstaver PB, Kohn J. et al. Development of institutional guidelines for management of gram-negative bloodstream infections: incorporating local evidence. Hosp Pharm 2017; 52 (10) 691-697
    CrossrefPubMedGoogle Scholar
  • 13 Wong-Beringer A, Nguyen LH, Lee M, Shriner KA, Pallares J. An antimicrobial stewardship program with a focus on reducing fluoroquinolone overuse. Pharmacotherapy 2009; 29 (06) 736-743
    CrossrefPubMedGoogle Scholar
  • 14 AHRQ. How learning health systems learn: lessons from the field. Accessed May 10, 2021 at: http://www.ahrq.gov/learning-health-systems/how-lhs-learn.html
    PubMedGoogle Scholar
  • 15 McGinnis JM, Fineberg HV, Dzau VJ. Advancing the learning health system. N Engl J Med 2021; 385 (01) 1-5
    CrossrefPubMedGoogle Scholar
  • 16 Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med 2011; 365 (19) 1758-1759
    CrossrefPubMedGoogle Scholar
  • 17 Longhurst CA, Harrington RA, Shah NHA. A ‘green button’ for using aggregate patient data at the point of care. Health Aff (Millwood) 2014; 33 (07) 1229-1235
    CrossrefPubMedGoogle Scholar
  • 18 Gombar S, Callahan A, Califf R, Harrington R, Shah NH. It is time to learn from patients like mine. NPJ Digit Med 2019; 2 (01) 16
    CrossrefPubMedGoogle Scholar
  • 19 Callahan A, Gombar S, Cahan EM. et al. Using aggregate patient data at the bedside via an on-demand consultation service. NEJM Catal 2021; DOI: 10.1056/CAT.21.0224.
    CrossrefPubMedGoogle Scholar
  • 20 Institute of Medicine. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Washington, DC: The National Academies Press; 2011
    Google Scholar
  • 21 Ostropolets A, Zachariah P, Ryan P, Chen R, Hripcsak G. Data consult service: can we use observational data to address immediate clinical needs?. J Am Med Inform Assoc 2021; 28 (10) 2139-2146
    CrossrefPubMedGoogle Scholar
  • 22 Knighton AJ, Ranade-Kharkar P, Brunisholz KD. et al. Rapid implementation of a complex, multimodal technology response to COVID-19 at an integrated community-based health care system. Appl Clin Inform 2020; 11 (05) 825-838
    Article in Thieme ConnectPubMedGoogle Scholar
  • 23 Datta S, Posada J, Olson G. et al. A new paradigm for accelerating clinical data science at Stanford Medicine. 44
    PubMedGoogle Scholar
  • 24 OHDSI. Observational Health Data Sciences and Informatics GitHub Library. Accessed August 18, 2021 at: https://github.com/OHDSI/
    PubMedGoogle Scholar
  • 25 STARR OMOP | Observational Medical Outcomes Partnership | Stanford Medicine. Accessed August 17, 2021 at: https://med.stanford.edu/starr-omop.html
    PubMedGoogle Scholar
  • 26 Wongvibulsin S, Zeger SL. Enabling individualised health in learning healthcare systems. BMJ Evid Based Med 2020; 25 (04) 125-129
    CrossrefPubMedGoogle Scholar
  • 27 Sertkaya A, Birkenbach A, Berlind A, Eyraud J. Examination of Clinical Trial Costs and Barriers for Drug Development: Report to the Assistant Secretary of PLanning and Evaluation (ASPE). Washington, DC: Department of Health and Human Services; 2014
    Google Scholar
  • 28 Sendak MP, Balu S, Schulman KA. Barriers to achieving economies of scale in analysis of EHR data. A cautionary tale. Appl Clin Inform 2017; 8 (03) 826-831
    Article in Thieme ConnectPubMedGoogle Scholar
  • 29 Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med 2010; 2 (57) 57cm29
    CrossrefPubMedGoogle Scholar
  • 30 Gould MK, Sharp AL, Nguyen HQ. et al. Embedded research in the learning healthcare system: ongoing challenges and recommendations for researchers, clinicians, and health system leaders. J Gen Intern Med 2020; 35 (12) 3675-3680
    CrossrefPubMedGoogle Scholar