Abstract
Implementation science is concerned with the effective deployment and sustainment of evidence-based practices in service delivery systems, with the ultimate goal of providing services to bring about optimal outcomes for clients. As the field of implementation science matures and research moves towards sustainment of implementation strategies within complex dynamic systems of care, the integration of implementation research into real-life practice settings will become increasingly important. Such an integrated approach relies on the ability to measure implementation success over time and to learn about causal mechanisms of implementation in service delivery systems. However, despite a continuing emphasis on the importance of high-quality data to support successful implementation efforts, data collection for implementation practice remains an under-researched frontier of implementation science. To this end, this chapter describes a causal approach of implementation research predicated on recognising implementation as a system component, with interventions and their implementation forming integral parts of dynamic systems of care with multiple stakeholders. This approach is facilitated by a 5-step implementation research framework, with a high-quality data system that integrates research with operational components to enable a holistic view of stakeholder incentives in what we denote as the Implementation Space. By considering the full implementation space from the beginning, data can be purposefully collected, stored and used. Such data systems will enable a process of learning by supporting a cascading and dynamic model of continuous quality improvement and practice optimisation through the Plan-Do-Study-Act (PDSA) cycle across all domains of the implementation space.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
For example, most customer relationship management systems today are built on a modular basis that support the flexibility to expand the functionalities of data systems based on the needs of businesses. While standard solutions are most likely not able to fully cater to the multifaceted needs of a service delivery systems described here, module-based designs are easily implementable in data systems specially designed for the purposes described here.
- 3.
A fourth group of stakeholders are funding bodies and government. However, we defer the discussion on this group to the end of this section as in most cases, it may not be directly involved in the implementation project itself. Generally, the role of this group is more likely to be that of a policy maker or a data user with an interest in impact assessment or policy.
- 4.
Concerns about anonymity for such measurements could be easily accommodated by restricting the content of reporting functions to aggregate measures or to report individual-level measures with de-identified data only. Alternatively, measures could be administered without linking the information to a particular identification key, that is, the data are truly anonymous but are still stored within the same data system and therefore available for CQI endeavours.
- 5.
In this context, costs are not restricted to monetary costs but also include opportunity costs as defined in the economic literature.
References
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7
Bollen, K. A., & Pearl, J. (2013). Eight myths about causality and structural equation models. In S. L. Morgan (Ed.), Handbook of causal analysis for social research (pp. 301–328). Dordrecht: Springer Netherlands. https://doi.org/10.1007/978-94-007-6094-3_15
Chambers, D. (2012). Foreword. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. vii–vix). Oxford: Oxford University Press.
Chambers, D., Wilson, P., Thompson, C., Harden, M., Coiera, E. (2012) Social Network Analysis in Healthcare Settings: A Systematic Scoping Review. PLoS ONE 7(8):e41911
Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013). Measuring factors affecting implementation of health innovations: A systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science, 8(1), 22. https://doi.org/10.1186/1748-5908-8-22
Cook, T. J., & Dobson, L. D. (1982). Reaction to reexamination: More on type III error in program evaluation. Evaluation and Program Planning, 5(2), 119–121. https://doi.org/10.1016/0149-7189(82)90018-0
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217–226. https://doi.org/10.1097/MLR.0b013e3182408812
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50
Deming, W. E. (1950). Elementary principles of the statistical control of quality: A series of lectures. Nippon Kagaku Gijutsu Remmei. Retrieved from https://books.google.com.au/books?id=8k5DGQAACAAJ
Deming, W. E. (1986). Out of the crisis. Cambridge, MA: Cambridge University Press. Retrieved from https://books.google.com.au/books?id=4qw8AAAAIAAJ
Dobson, D., & Cook, T. J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3(4), 269–276. https://doi.org/10.1016/0149-7189(80)90042-7
Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 157. https://doi.org/10.1186/s13012-014-0157-1
Fernandez, M. E., Walker, T. J., Weiner, B. J., Calo, W. A., Liang, S., Risendal, B., … Kegler, M. C. (2018). Developing measures to assess constructs from the inner setting domain of the consolidated framework for implementation research. Implementation Science, 13(1), 52. https://doi.org/10.1186/s13012-018-0736-7
Heckman, J. J., & Pinto, R. (2015). Causal analysis after Haavelmo. Econometric Theory, 31(1), 115–151. https://doi.org/10.1017/S026646661400022X
Heckman, J. J., & Vytlacil, E. (2005). Structural equations, treatment effects, and econometric policy evaluation. Econometrica, 73(3), 669–738. https://doi.org/10.1111/j.1468-0262.2005.00594.x
Heckman, J. J., & Vytlacil, E. J. (2007). Chapter 70 Econometric evaluation of social programs, part I: Causal models, structural models and econometric policy evaluation. In J. J. Heckman & E. E. Leamer (Eds.), Handbook of econometrics, Vol. 6 (pp. 4779–4874). North Holland: Elsevier. https://doi.org/10.1016/S1573-4412(07)06070-9
Holmes, B.J., Finegood, D.T., Riley, B.L., Best, A. (2014). Systems thinking in dissemination and implementation research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and Implementation Research in Health: Translating Science to Practice (pp. 175–91). New York; Oxford: Oxford University Press.
Kimball, R., Ross, M., Thornthwaite, W., Mundy, J., & Becker, B. (2008). In Tom (Ed.), The data warehouse lifecycle toolkit: Practical techniques for building data warehouse and business intelligence systems (2nd ed.). Indianapolis: Wiley.
Krcmar, H. (2005). Informations management (4th ed.). Berlin, Germany: Springer.
Langley, G. J., Nolan, K. M., & Nolan, T. W. (1994). The Foundation of Improvement. Quality Progress, 27(6), 81–86.
Lewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2015). The society for implementation research collaboration instrument review project: A methodology to promote rigorous evaluation. Implementation Science, 10(1), 2. https://doi.org/10.1186/s13012-014-0193-x
Maglio, P.P., & Mabry, P.L. (2011). Agent-based models and systems science approaches to public health. Am J Prev Med, 40(3):392–394.
Maier, R. (2007). Knowledge management systems: Information and communication technologies for knowledge management (3rd ed.). Berlin, Germany: Springer.
Moen, R. D., & Norman, C. L. (2010). Circling back. Quality Progress, 11, 20.
Mas-Colell, A., Whinston, M. D., & Green, J. R. (1995). Microeconomic theory. New York: Oxford University Press.
Parolini A., Tan W.W., Shlonsky A. (2019). Decision-based models of the implementation of interventions in systems of healthcare: Implementation outcomes and intervention effectiveness in complex service environments. PLOS ONE 14(10): e0223129. https://doi.org/10.1371/journal.pone.0223129
Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). New York: Cambridge University Press.
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002). Data quality assessment. Communications of the ACM, 45(4). https://doi.org/10.1145/505248.506010
Proctor, E. (2014). Dissemination and Implementation Research. In Encyclopedia of social work. Oxford University Press. Retrieved from http://socialwork.oxfordre.com/view/10.1093/acrefore/9780199975839.001.0001/acrefore-9780199975839-e-900
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7
Rezmovic, E. L. (1982). Program implementation and evaluation results: A reexamination of type III error in a field experiment. Evaluation and Program Planning, 5(2), 111–118. https://doi.org/10.1016/0149-7189(82)90017-9
Roberts, M. S. (2015) Dynamic Simulation in Health Care Comes of Age. Value in Health 18 (2):143–144
Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Boca Raton, FL: Chapman & Hall/CRC.
Sterman, J. D. (1994). Learning in and about complex systems. System Dynamics Review, 10(2–3), 291–330. https://doi.org/10.1002/sdr.4260100214
Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505–514. https://doi.org/10.2105/AJPH.2005.066043
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice. Models for dissemination and implementation research. American Journal of Preventive Medicine, 43, 337–350.
Wand, Y., & Wang, R. Y. (1996). Anchoring data quality dimensions in ontological foundations. Communications of the ACM, 39(11), 86–95. https://doi.org/10.1145/240455.240479
Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. https://doi.org/10.1080/07421222.1996.11518099
White, H., & Chalak, K. (2013). Identification and identification failure for treatment effects using structural systems. Econometric Reviews, 32(3), 273–317. https://doi.org/10.1080/07474938.2012.690664
White, H., & Lu, X. (2011). Causal diagrams for treatment effect estimation with application to efficient covariate selection. The Review of Economics and Statistics, 93(4), 1453–1459. https://doi.org/10.1162/REST_a_00153
Wulczyn, F., Clinch, R., Coulton, C., Keller, S., Moore, J., Muschkin, C., … Barghaus, K. (2017). Establishing a standard data model for large-scale IDS use (actionable intelligence for social policy, expert panel report). Actionable intelligence for social policy, University of Pennsylvania. Retrieved from https://www.aisp.upenn.edu/wp-content/uploads/2016/07/Data-Standards.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Tan, W.W., Jeffreys, C., Parolini, A. (2020). Implementing Implementation: Integrating the Measurement of Implementation and Effectiveness in Complex Service Systems. In: Albers, B., Shlonsky, A., Mildon, R. (eds) Implementation Science 3.0. Springer, Cham. https://doi.org/10.1007/978-3-030-03874-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-03874-8_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03873-1
Online ISBN: 978-3-030-03874-8
eBook Packages: Social SciencesSocial Sciences (R0)