Figures
Abstract
Background
Large amounts of scientific evidence are generated, but not implemented into patient care (the ‘knowledge-to-care’ gap). We identified and prioritized knowledge-to-care gaps in critical care as opportunities to improve the quality and value of healthcare.
Methods
We used a multi-method community-based participatory research approach to engage a Network of all adult (n = 14) and pediatric (n = 2) medical-surgical intensive care units (ICUs) in a fully integrated geographically defined healthcare system serving 4 million residents. Participants included Network oversight committee members (n = 38) and frontline providers (n = 1,790). Network committee members used a modified RAND/University of California Appropriateness Methodology, to serially propose, rate (validated 9 point scale) and revise potential knowledge-to-care gaps as priorities for improvement. The priorities were sent to frontline providers for evaluation. Results were relayed back to all frontline providers for feedback.
Results
Initially, 68 knowledge-to-care gaps were proposed, rated and revised by the committee (n = 32 participants) over 3 rounds of review and resulted in 13 proposed priorities for improvement. Then, 1,103 providers (62% response rate) evaluated the priorities, and rated 9 as ‘necessary’ (median score 7–9). Several factors were associated with rating priorities as necessary in multivariable logistic regression, related to the provider (experience, teaching status of ICU) and topic (strength of supporting evidence, potential to benefit the patient, potential to improve patient/family experience, potential to decrease costs).
Conclusions
A community-based participatory research approach engaged a diverse group of stakeholders to identify 9 priorities for improving the quality and value of critical care. The approach was time and cost efficient and could serve as a model to prioritize areas for research quality improvement across other settings.
Citation: Stelfox HT, Niven DJ, Clement FM, Bagshaw SM, Cook DJ, McKenzie E, et al. (2015) Stakeholder Engagement to Identify Priorities for Improving the Quality and Value of Critical Care. PLoS ONE 10(10): e0140141. https://doi.org/10.1371/journal.pone.0140141
Editor: Umberto Simeoni, Centre Hospitalier Universitaire Vaudois, FRANCE
Received: June 11, 2015; Accepted: September 21, 2015; Published: October 22, 2015
Copyright: © 2015 Stelfox et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The project was supported by a Partnerships for Research and Innovation in Health System Improvement Grant (201309 AIHS PRIHS, www.aihealthsolutions.ca) from Alberta Innovates Health Solutions. Funding sources had no role in the design, conduct, or reporting of this study.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Large amounts of scientific evidence are generated, yet not effectively implemented into patient care (the ‘knowledge-to-care’ gap).[1] Healthcare systems routinely fail to make optimal use of evidence, which results in suboptimal, low value patient care (i.e., underuse, overuse and misuse of therapies).[2] Identifying and addressing knowledge-to-care gaps is an opportunity to improve the quality and value of healthcare by implementing high value patient care practices.[3]
The best approach to identify and prioritize knowledge-to-care gaps for intervention is unknown.[4] Historically, priorities have been independently established by individuals and organizations engaged in research and quality improvement.[5] However, because of growing concerns about the relevance of research and quality improvement to human health, healthcare systems are increasingly looking to engage stakeholders in establishing priorities.[5] Employing research-practice partnerships holds the promise of better matching stakeholders’ needs with research and quality improvement efforts, while engaging stakeholders to lead and sustain practice change.[6]
In 2013, Alberta Health Services, a fully integrated geographically defined healthcare system serving a population of 4 million residents, launched a program with Alberta Innovates Health Solutions (research funder) to improve value for money in healthcare by encouraging de-adoption of low value patient care practices and adoption of high value patient care practices.[7] In response, a network (Critical Care Strategic Clinical Network)[8] of all adult (n = 14) and pediatric (n = 2) medical-surgical intensive care units (ICUs) in the healthcare system conducted a research project to evaluate the feasibility of using a research-practice partnership to identify and prioritize knowledge-to-care gaps in the care of critically ill patients admitted to ICU as opportunities to improve the quality and value of healthcare.
Methods
Study Design and Population
Identifying and addressing knowledge-to-care gaps requires a coordinated multi-step longitudinal process that includes identifying problems, developing and implementing locally tailored interventions, evaluating the outcomes and monitoring the sustainability of the practice change.[9] The complexity and applied nature of the problem requires a research-practice partnership.[10] We selected a community-based participatory research approach where researchers and ICU stakeholders are equitably partnered, because it balances scientific rigor with community engagement so that practice change can be informed by science, but implemented, scaled and sustained by stakeholders.[6, 11] We used the ‘knowledge-to-action’ (KTA) cycle developed by Graham et al.[9] as the guiding framework for the study given its review of more than 30 theories of planned action and its successful use in similar clinical contexts.[12, 13]
We conducted a multi-method study to identify and prioritize common patient care practices (e.g., tests, treatments, processes of care) that may be underused, overused or misused in medical-surgical ICUs. The goal was to inform a subsequent audit of patient care, as part of the Critical Care Strategic Clinical Network’s program to systematically de-adopt low value patient care practices and adopt high value patient care practices. First, we used a modified RAND/University of California appropriateness method, a valid and reliable deliberative process, to identify common patient care practices with perceived knowledge-to-care gaps as priorities for improvement.[14, 15] Second, we surveyed frontline providers to evaluate the proposed priorities for improvement.[16] Third, we relayed the results back to frontline providers for feedback and to ascertain their engagement to participate in resulting quality improvement initiatives. We defined a knowledge-to-care gap as a discrepancy between what current scientific evidence suggests is the best practice, and the actual care provided.[17] We defined ICU stakeholders as providers, managers and decision-makers responsible for patient care. To be inclusive, ICUs that serve both adult and pediatric patients were included.
Identification of Priorities for Improvement
We invited members of the Critical Care Strategic Clinical Network oversight committee to participate in a modified RAND/University of California appropriateness method to identify common patient care practices with perceived knowledge-to-care gaps as priorities for improvement.[15] The committee is a standing body comprised of multidisciplinary providers, managers and decision-makers representing all ICUs in the healthcare system and meets monthly to provide oversight of the Network. Committee members were divided into two separate focus groups (March 20, 2014) and asked to consider three common types of knowledge-to-care gaps: (1) over use—when science shows a patient care practice is ineffective or harmful, but is prescribed (e.g., tight glycemic control in critically ill patients);[18, 19] (2) under use—when science shows a practice is effective, but is not prescribed (e.g., venous thromboembolism prophylaxis);[20, 21] and (3) misuse—when science shows a practice to be effective, but is prescribed for the wrong patients, wrong reason or wrong time (e.g., prescription of albumin in patients with traumatic brain injury, but not in patients with cirrhosis and spontaneous bacterial peritonitis)[22, 23] (S1 Appendix–focus group guide). We included suggestions spanning the spectrum of scientific evidence (i.e., quality of evidence did not serve as an inclusion or exclusion criterion). In addition, participants were asked what criteria should be considered when evaluating patient care practices as potential priorities for improvement. Suggestions were recorded by a note taker, reviewed with the participants and collated. Audiotaped recordings of the proceedings were reviewed to ensure all suggestions were documented. Two members of the research team independently reviewed the lists of practices and criteria from the two focus groups, combined suggestions and refined terminology for consistency. Disagreements were resolved with discussion.
The rating process consisted of two rounds of sequential review and revision. Committee members were presented with a list of the priorities and asked to independently rate the priorities according to their importance as opportunities for improving the quality and value of healthcare using the validated nine-point RAND/University of California appropriateness method scale.[15] Committee members were asked to provide written comments. Round one was performed using a secure, web-based survey. Round two was performed using a half-day in-person workshop (May 15, 2014, Edmonton, Canada). Committee members were provided with personalized rating summaries from round one (i.e., individual priority ratings assigned by the committee member, anonymous distribution of ratings by other committee members, summary of committee member comments). Each priority was presented by a moderator, discussed and independently rerated by committee members.
Evaluation of Priorities for Improvement
We surveyed all frontline providers working in adult (n = 14 ICUs) and pediatric (n = 2 ICUs) medical-surgical (medical, surgical, trauma, transplant) ICUs in Alberta, Canada. Providers were identified using login identifications for electronic medical record systems and patient care manager staff lists. We sent an email cover letter explaining the purpose of the study and a link to a secure web-based survey. The survey included an evaluation framework comprised of six criteria proposed by committee members. Members of the research team, working independently in groups of two, produced a synthesis of the criteria for each priority using a modified Grades of Recommendation Assessment, Development, and Evaluation (GRADE) system: strength of supporting evidence (searched literature), potential to benefit the patient (searched literature), potential to improve patient/family experience (solicited perspectives of patient/family network advisors), potential to decrease costs (reviewed costing data), ability to easily measure (reviewed electronic medical record data elements), and ability to act upon (solicited perspectives of network decision-makers).[24] We asked participants to rate the priorities according to their importance as opportunities for improving the quality and value of healthcare using the same validated nine-point scale and to suggest additional priorities. Participation in the study was voluntary and anonymous; written consent was not requested, but inferred with survey completion. Reminders were sent weekly (survey launched June 4, 2014 and closed July 2, 2014). Participation was encouraged by widely advertising the survey, recruiting local champions to promote participation and offering weekly draws for coffee cards.[16]
Community Engagement
The results of the provider surveys were collated and relayed back to all providers (September 17, 2014). An attached survey asked providers to indicate whether the priorities rated as necessary were reasonable choices for quality improvement initiatives, whether they were supportive of working on future initiatives targeting the priorities, and whether they would be willing to act as a local champion for the initiatives. We provided no reminders as the primary purpose of the communication was to relay results back to providers (survey closed October 2, 2014) and we wanted to minimize participant burden given the programmatic nature of the work.
Statistical Analysis
The primary analysis was to describe priority ratings by committee members and providers. The median rating was used to classify each priority as unnecessary (median score 1–3), supplementary (median score 4–6) or necessary (median score 7–9). Priorities rated as necessary by committee members and providers were considered potential priorities for improvement. We tested for associations between provider and priority characteristics, and the ratings provided by providers, using mixed effects logistic regression with clustering by respondent (i.e., to account for the rating of multiple priorities by each respondent).[25–28] The unit of analysis was each respondent’s rating of each priority (responses dichotomized necessary vs. unnecessary/supplementary). We evaluated for potential interactions between significant provider and priority characteristics. All analyses were conducted using Stata version 13.1 (Stata Corp, College Station, TX), with a p-value < 0.05 defining statistical significance. Ethics approval was obtained from the Conjoint Health Research Ethics Board at the University of Calgary.
Results
Identification of Priorities for Improvement
Of the 38 network committee members, 32 (84%) participated in the consensus process (Table 1). Fig 1 summarizes the flow of priorities for improvement through the study. Committee members proposed 121 priorities between two focus groups. After combining 40 duplicates and deleting 13 priorities that did not satisfy inclusion criteria (e.g., not a direct patient care practice) a total of 68 priorities were proposed. Committee members were presented with all 68 priorities at the beginning of the review process (S2 Appendix). Two priorities, strategies to facilitate patient day/night cycle and strategies to preserve patient sleep, were merged. Nineteen priorities were rated as unnecessary. Table 2 summarizes the 13 priorities for improvement rated as necessary.
Committee members proposed six criteria to guide selection of priorities during the focus groups and rated the criteria according to importance during the first round of reviews. Committee members rated potential to benefit patient (median score 9, interquartile range [IQR] 8–9), ability to easily measure the practice (median score 8, IQR 7–9), ability to take action to change the practice (median score 8, IQR 7–9), strength of supporting evidence (median score 8, IQR 7–9), and potential to improve patient/family experience (median score 7, IQR 6–8) as necessary criteria, and potential to decrease costs (median score 6, IQR 5–8) as a supplementary criterion.
Evaluation of Priorities for Improvement
We surveyed 1,790 providers and received 1,103 responses (62% response rate) from providers who reported working in 16 medical-surgical ICUs in seven cities. The majority of respondents were nurses (61%) followed by respiratory therapists (18%), allied health professionals (13%) and physicians (7%) (Table 1). Most provided direct patient care (93%) to adult patients (88%) in teaching (77%) ICUs. Respondents worked a median of 7 years in critical care.
Table 2 summarizes provider ratings (S3 Appendix). Of the 13 priorities for improvement rated as necessary by network committee members, nine were rated as necessary by providers (i.e., median score 7–9): end-of-life care, early mobilization, strategies to preserve patient sleep, transition of patient care from ICU to hospital ward, transition of patient care between providers within the ICU, daily sedation interruption, delirium screening & diagnosis, and temperature control in patients after resuscitation from cardiac arrest. One hundred and thirty six providers proposed 55 additional unique priorities for improvement, of which 19 had not been previously considered by the network committee (S4 Appendix).
Ratings were consistent across provider and institution characteristics with a few notable exceptions (S5 Appendix). Among the priorities rated as supplementary, nurses and respiratory therapists rated the use of physical and pharmacological restraints as necessary, physicians rated the duration of empiric antimicrobial prescriptions as necessary and allied health providers rated patient and family participation in daily rounds as necessary.
Characteristics of the Priorities for Improvement
Table 3 summarizes the multivariable adjusted odds ratios of provider and priority characteristics associated with provider ratings. The odds of physicians, nurses and respiratory therapists rating a priority as necessary were not significantly different. Provider years of experience were associated with an increased likelihood of priorities being rated as necessary. Providers working in non-teaching ICUs were 20 percent (95% confidence interval, 3% to 40%) more likely to rate priorities as necessary than providers working in teaching ICUs. The characteristics of the priorities associated with those rated as necessary included strength of supporting evidence, potential to benefit the patient, potential to improve patient/family experience, and potential to decrease costs. The perceived ability to easily measure the practice was not associated with priorities rated as necessary. The perceived ability to take action to change the practice was inversely associated with priorities rated as necessary. There was a significant interaction between respondent profession and the potential to decrease costs (p < 0.001)(S6 Appendix). For priorities with a neutral effect on costs, nurses, respiratory therapists, and allied health practitioners were more likely than physicians to rate the priority as necessary. For priorities with potential to decrease costs, there was no difference across respondent professions.
Community Engagement
The results of the provider survey were relayed back to providers. A total of 627 providers responded (35% response rate) and indicated that the nine priorities rated as necessary were reasonable choices for quality improvement initiatives (87%) and that they were highly supportive of working on future initiatives targeting the priorities (61%). Ninety-two individuals representing nurses, respiratory therapists, physicians and allied health professionals working in adult, pediatric, teaching and non-teaching ICUs indicated that they would be willing to act as local champions for future initiatives targeting these priorities.
Discussion
In response to a request from a healthcare system, we tested the feasibility of using a community-based participatory research approach to engage a diverse group of stakeholders to identify and prioritize opportunities to improve the quality and value of critical care. We trialed the use of technically robust techniques as part of the approach using a modification of the RAND/UCLA Appropriateness Methodology to identify priorities and survey of frontline providers to evaluate them. We proposed and tested six evaluation criteria that can be used to frame future prioritization activities. The approach, methods and evaluation criteria we employed can be used as a template for identifying priorities across other settings.
Different approaches have been used to identify priorities for research and quality improvement. Scientists have traditionally advocated the use of valid and reliable tools with robust evaluative methodologies (e.g., consensus methods), but have been criticized for moving too slowly and having ‘ivory tower’ perspectives.[29] Healthcare decision-makers have traditionally advocated efficient and nimble quality improvement strategies (e.g., Plan-Do-Study-Act cycles), but have been criticized for employing methodologies at risk of bias.[30] The strength of the research-practice partnership employed in our study is that it combines the strengths of these two approaches. For example, the Choosing Wisely® initiative to identify and publicize low value tests or procedures has resulted in over 50 professional societies publishing top five lists and spread internationally to other healthcare systems.[31, 32] Such initiatives have been criticized for a lack of rigor and transparency in the methods employed in selecting priorities.[33, 34] Our methodology demonstrates the feasibility of using a community-based participatory research approach that is scientifically robust and engages all stakeholders involved in multidisciplinary patient care. This approach has been successfully used in public health, but infrequently in hospital-based healthcare.[11] For example, a search of the National Library of Medicine identified 18 articles indexed with the exploded Medical Subject Headings ‘hospitals’ and ‘community-based participatory research’. We were able to identify two articles within the clinical area of critical care, one review article[35] and one commentary[36], but no original research reports. In contrast, we identified over seventeen hundred articles of community-based participatory research in public health. Similar approaches have been adopted in non-healthcare industries to guide quality improvement efforts.[37]
Our research highlights a different conceptual approach to improving quality and value in healthcare. Traditionally, knowledge transfer activities have focused on facilitating the adoption of evidence informed technologies into practice.[17] Conversely, the Choosing Wisely® initiative has focused on eliminating waste from practice.[31] We employed a modification of these two approaches, reasoning that closing knowledge-to-care gaps by reducing the underuse, overuse or misuse of patient care practices will improve the quality and value of care. The Choosing Wisely® Top 5 List in Critical Care Medicine, nested within the priorities considered in our approach, highlights this nicely.[38] In addition to independently considering these items, providers in our project selected patient care practices perceived to be underused (e.g., early mobilization) or misused (e.g., transitions of patient care). Of the Choosing Wisely® Top 5 List, two items were rated as necessary priorities for improving the quality and value of care by stakeholders within our critical care community (end-of-life care, daily sedation interruption), two as supplementary priorities (routine blood tests/diagnostic imaging studies, limiting blood transfusion in non-bleeding patients) and one as an unnecessary priority (providing total parenteral nutrition).[38] This highlights that priorities for improvement are likely to vary across jurisdictions and institutions, and underscores the importance of engaging local stakeholders to ensure initiatives match local needs.
We found that consideration needs to be given to both stakeholder characteristics and patient care practice characteristics when selecting priorities for improvement. Although stakeholders of different backgrounds demonstrate similar patterns of selection, those working in non-teaching hospitals were more likely to rate priorities for improvement as necessary than those working in teaching hospitals, suggesting a greater openness or perceived need for improvement. Providers selected priorities supported by scientific evidence and potential to benefit the patient/family. Surprisingly, the logistical practicalities of the ability to easily measure the practice and to take action to change the practice were not positively associated with priority selection. Cost only modestly influenced providers. These data reinforce previous observations that providers may be more motivated to improve care of their patients than to act as stewards of the healthcare system.[39, 40] Future efforts to improve value of healthcare may benefit from accompanying educational initiatives that highlight the importance of stewardship in improving and sustaining quality of care.[41]
Our study has both important strengths and limitations. First, our study reports perceived priorities for improvement, but does not document actual deficiencies in care. This is an important distinction because quality improvement initiatives have previously been initiated in response to stakeholder priorities only to discover that opportunities for improvement were smaller than anticipated.[42] An audit of patient care is required to document gaps as well as establish facilitators and barriers to practice change before designing and implementing interventions. Second, our project reports anonymous data from multiple institutions across a geographically defined healthcare system. Although the rating of priorities was similar across institutions with different characteristics (e.g., urban vs. regional, non-teaching vs. teaching, adult vs. pediatric), differences may exist between individual institutions and should be considered during healthcare system wide initiatives. Finally, our study does not provide any information about patient and family priorities, which may differ from those of providers, managers and decision-makers, and were solicited through a separate process[43]. Future work is needed to bring together the perspectives of the different stakeholders into a ‘reconciled list of provider, decision-maker, patient and family priorities’ to inform the resulting audit and improvement initiatives. The strengths of our study include its applied nature, population-based design and community-based participatory research approach.
Conclusions
In summary, we successfully used a community-based participatory research approach to engage a diverse group of multidisciplinary stakeholders to identify and prioritize knowledge-to-care gaps in the care of critically ill patients. We tested the use of valid and reliable scientific methods as part of the approach. We proposed and tested six evaluation criteria that can be used to frame future prioritization activities. The nine priorities rated as necessary by stakeholders will inform initiatives to improve the quality and value of care. The approach, methods and evaluation criteria we employed can be used as a template for identifying priorities across other settings.
Supporting Information
S2 Appendix. Critical Care Network Core Committee Ratings of Priorities for Improvement.
https://doi.org/10.1371/journal.pone.0140141.s002
(PDF)
S3 Appendix. Histograms of Provider Ratings of Priorities for Improvement.
https://doi.org/10.1371/journal.pone.0140141.s003
(PDF)
S4 Appendix. Additional Priorities Proposed by Providers for Quality Improvement.
https://doi.org/10.1371/journal.pone.0140141.s004
(PDF)
S5 Appendix. Provider Ratings of Priorities for Improvement According to Provider Characteristics.
https://doi.org/10.1371/journal.pone.0140141.s005
(PDF)
S6 Appendix. Selection of Priorities According to Survey Respondent Profession and Potential to Decrease Costs.
https://doi.org/10.1371/journal.pone.0140141.s006
(PDF)
Acknowledgments
We thank all of the critical care providers, managers and decision-makers who participated in this study. We thank Patty Wickson, Alberta Health Services, and Michelle Salesse, Alberta Health Services, for their help with the core committee focus groups. We thank members of the Critical Care Strategic Clinical Network Core Committee who contributed to the study; Brenda Ashman, Sylvia Barron, Sandra Beida, Luc Berthiaume, Paul Boucher, Kelly Coutts, Jonathan Davidow, Roberta Dubois, Elaine Gilfoyle, Nancy Guebert, Laurie Harding, Caroline Hatcher, Pam Hruska, Barry Kushner, Pat Lynkowski, Mike Meier, Simon Parsons, Carmen Petersen, Heather Richardson, Janet Schimpf, Colleen Sokolowski, Dan Stollery, Teddie Tanguay, Yolandra Westra, Jill Woodward, Bryan Yipp and Dan Zuege.
Author Contributions
Conceived and designed the experiments: HTS DJN FMC SMB DJC EM MLP CJD BO DZ. Performed the experiments: HTS DJN EM MLP. Analyzed the data: HTS DJN. Contributed reagents/materials/analysis tools: HTS. Wrote the paper: HTS. Interpreted the data: HTS DJN FMC SMB DJC EM MLP CJD BO DZ. Revised the manuscript critically for important intellectual content: HTS DJN FMC SMB DJC EM MLP CJD BO DZ.
References
- 1. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implementation science: IS. 2012;7:50. pmid:22651257; PubMed Central PMCID: PMC3462671.
- 2.
Institute of Medicine Committee on the Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC: 2001.
- 3. Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, et al. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003;327(7405):33–5. pmid:12842955; PubMed Central PMCID: PMCPMC164240.
- 4.
Kitson AL, Straus SE. Identifying knowledge to action gaps. In: Straus SE, Tetroe J, Graham ID, editors. Knowledge Translation in Health Care. Oxford: BMJ Books; 2013. p. 97–109.
- 5. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. Epub 2014 Jan 8. pmid:24411644
- 6. Horowitz CR, Robinson M, Seifer S. Community-based participatory research from the margin to the mainstream: are researchers prepared? Circulation. 2009;119(19):2633–42. pmid:19451365; PubMed Central PMCID: PMC2796448.
- 7.
Alberta Innovates Health Solutions. Partnerships for Research and Innovation in the Health System Edmonton, Canada: Alberta Innovates Health Solutions; 2013 [cited 2014 December 4]. Available from: http://www.aihealthsolutions.ca/funding/health-research-funding/prihs-partnerships-for-research-and-innovation-in-the-health-system/.
- 8.
Alberta Health Services. Critical Care Strategic Clinical Network Edmonton, Canada: Alberta Health Services; 2014 [cited 2014 December 4]. Available from: http://www.albertahealthservices.ca/9437.asp.
- 9. Graham ID, Logan J. Innovations in knowledge transfer and continuity of care. Can J Nurs Res. 2004;36(2):89–103. pmid:15369167.
- 10. Ovretveit J, Hempel S, Magnabosco JL, Mittman BS, Rubenstein LV, Ganz DA. Guidance for research-practice partnerships (R-PPs) and collaborative research. J Health Organ Manag. 2014;28(1):115–26. pmid:24783669.
- 11. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annual review of public health. 1998;19:173–202. pmid:9611617.
- 12. Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001;285(22):2871–9. pmid:11401608.
- 13. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969–77. pmid:15758007.
- 14. Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. The New England journal of medicine. 1998;338(26):1888–95. pmid:9637810.
- 15.
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, P L , et al. The RAND/UCLA appropriateness method user's manual. Santa Monica, USA: RAND; 2001.
- 16. Burns KE, Duffett M, Kho ME, Meade MO, Adhikari NK, Sinuff T, et al. A guide for the design and conduct of self-administered surveys of clinicians. CMAJ: Canadian Medical Association journal = journal de l'Association medicale canadienne. 2008;179(3):245–52. pmid:18663204; PubMed Central PMCID: PMC2474876.
- 17.
Kitson AL, Straus S. Identifying knowledge to action gaps. In: Straus S, Tetroe J, Graham I, editors. Knowledge Translation in Health Care. Second Edition ed. Oxford, United Kingdom: Wiley Blackwell; 2013. p. 97–109.
- 18. Finfer S, Liu B, Chittock DR, Norton R, Myburgh JA, et al. Hypoglycemia and risk of death in critically ill patients. N Engl J Med. 2012;367(12):1108–18. pmid:22992074.
- 19. Niven DJ, Rubenfeld GD, Kramer AA, Stelfox HT. Effect of published scientific evidence on glycemic control in adult intensive care units. JAMA Intern Med. 2015;175(5):801–9. pmid:25775163.
- 20. Fowler RA, Mittmann N, Geerts W, Heels-Ansdell D, Gould MK, Guyatt G, et al. Cost-effectiveness of dalteparin vs unfractionated heparin for the prevention of venous thromboembolism in critically ill patients. Jama. 2014;312(20):2135–45. pmid:25362228.
- 21. Lauzier F, Muscedere J, Deland E, Kutsogiannis DJ, Jacka M, Heels-Ansdell D, et al. Thromboprophylaxis patterns and determinants in critically ill patients: a multicenter audit. Critical care. 2014;18(2):R82. pmid:24766968; PubMed Central PMCID: PMC4057024.
- 22. Myburgh J, Cooper DJ, Finfer S, Bellomo R, Norton R, Bishop N, et al. Saline or albumin for fluid resuscitation in patients with traumatic brain injury. The New England journal of medicine. 2007;357(9):874–84. pmid:17761591.
- 23. Sort P, Navasa M, Arroyo V, Aldeguer X, Planas R, Ruiz-del-Arbol L, et al. Effect of intravenous albumin on renal impairment and mortality in patients with cirrhosis and spontaneous bacterial peritonitis. The New England journal of medicine. 1999;341(6):403–9. pmid:10432325.
- 24. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Bmj. 2004;328(7454):1490. pmid:15205295; PubMed Central PMCID: PMC428525.
- 25. Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986;42(1):121–30. pmid:3719049
- 26. Liang KY, S.L. Z. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:13–22.
- 27. Carlin JB, Wolfe R, Brown CH, Gelman A. A case study on the choice, interpretation and checking of multilevel models for longitudinal binary outcomes. Biostatistics. 2001;2(4):397–416. pmid:12933632.
- 28. Hu FB, Goldberg J, Hedeker D, Flay BR, Pentz MA. Comparison of population-averaged and subject-specific approaches for analyzing repeated binary outcomes. Am J Epidemiol. 1998;147(7):694–703. pmid:9554609
- 29. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74(9):979–83. pmid:6380323; PubMed Central PMCID: PMCPMC1651783.
- 30.
Langley GL, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. Second Edition ed. San Francisco: Jossey-Bass Publishers; 2009.
- 31.
America Board of Internal Medicine. Choosing Wisely Philadelphia, United States of America: American Board of Internal Medicine; 2014 [cited 2014 December 4]. Available from: http://www.choosingwisely.org.
- 32.
Canadian Medical Association. Choosing Wisely Canada Ottawa, Canada: Canadian Medical Association; 2014 [cited 2014 December 5]. Available from: http://www.choosingwiselycanada.org.
- 33. Grady D, Redberg RF, Mallon WK. How should top-five lists be developed?: what is the next step? JAMA internal medicine. 2014;174(4):498–9. pmid:24534809.
- 34. Schuur JD, Carney DP, Lyn ET, Raja AS, Michael JA, Ross NG, et al. A top-five list for emergency medicine: a pilot project to improve the value of emergency care. JAMA internal medicine. 2014;174(4):509–15. pmid:24534899.
- 35. Bartlett SJ, Barnes T, McIvor RA. Integrating patients into meaningful real-world research. Annals of the American Thoracic Society. 2014;11 Suppl 2:S112–7. pmid:24559023.
- 36. Kahn JM. Disseminating clinical trial results in critical care. Critical care medicine. 2009;37(1 Suppl):S147–53. pmid:19104215.
- 37. Atkinson AA, Waterhouse JH, Wells RB. A stakeholder approach to strategic performance measurement. Sloan Management Review. 1997;38(3):25–37.
- 38. Halpern SD, Becker D, Curtis JR, Fowler R, Hyzy R, Kaplan LJ, et al. An official American Thoracic Society/American Association of Critical-Care Nurses/American College of Chest Physicians/Society of Critical Care Medicine policy statement: the Choosing Wisely(R) Top 5 list in Critical Care Medicine. American journal of respiratory and critical care medicine. 2014;190(7):818–26. pmid:25271745.
- 39. Levinsky NG. The doctor's master. The New England journal of medicine. 1984;311(24):1573–5. pmid:6438510.
- 40. Tilburt JC, Wynia MK, Sheeler RD, Thorsteinsdottir B, James KM, Egginton JS, et al. Views of US physicians about controlling health care costs. Jama. 2013;310(4):380–8. pmid:23917288.
- 41. Brody H. Medicine's ethical responsibility for health care reform—the Top Five list. The New England journal of medicine. 2010;362(4):283–5. pmid:20032315.
- 42. Scales DC, Dainty K, Hales B, Pinto R, Fowler RA, Adhikari NK, et al. A multifaceted intervention for quality improvement in a network of intensive care units: a cluster randomized trial. Jama. 2011;305(4):363–72. pmid:21248161.
- 43. Gill M, Bagshaw SM, McKenzie E, Oxland P, Oswell D, Boulton D, et al. Patient and Family Researchers in the Intensive Care Unit: A Qualitative Approach to Patient-Centered Care. Manuscript Under Peer Review. 2015.