Building evaluation capacity: Definitional and practical implications from an Australian case study
Introduction
Internationally, there is growing emphasis on evaluation capacity building within public sector organizations that provide social, educational and health-related programs for communities (Beere, 2005; Horton et al., 2003; McDonald, Rogers, & Kefford, 2003; Preskill & Russ-Eft, 2004; Stevenson, Florin, Mill, & Andrade, 2002; Stockdill, Baizerman, & Compton, 2002; Taut, 2007b). Australia is no exception, with considerable attention focused on evaluation capacity building in the general practice and primary health care sector (Commonwealth Department of Health and Aged Care, 2000; Dunbar, Comino, Mehmet, & Harris, 2002; Naccarella, Tacticos, & Potiriadis, 2005).
This interest in evaluation capacity building has occurred for a number of inter-related reasons. There is an increasing expectation on the part of funders and communities that organizations demonstrate effectiveness, value for money, appropriateness and accessibility—both for the purposes of modifying individual services and programs so they are delivered optimally, and in order to contribute to broader knowledge that can feed in to higher level policy decisions (Batterbury, 2002). External evaluators are sometimes brought in to conduct evaluations in this regard, but there is a growing recognition that this may not always be the best model. One criticism, for example, is that external evaluators do not always have a full understanding of the given service or program, and do not adequately capture what is going on. In many cases, it may make more sense for organizations to conduct their own evaluations, but this requires that they be appropriately equipped to do so. As a result, it is now quite common for funding bodies to demand that organizations develop their own internal evaluation capacity (Stevenson et al., 2002), and an area of expertise has developed in assisting organizations to build this capacity (Preskill & Russ-Eft, 2004).
There is still a lack of clarity, however, in what the term “evaluation capacity” means. Definitions of evaluation capacity have ranged from the narrow to very broad. At the former end of the spectrum, Milstein and Cotton (2000) conceptualize evaluation capacity as “the ability to conduct an effective evaluation”. At the latter end, McDonald et al. (2003) argue that it is insufficient to focus on the capacity to supply a credible evaluation, and that steps must be taken to ensure that the organizational culture nurtures and values evaluation skills in a manner that allows them to become sustainable over time. For this reason, they choose the term “evaluation capability” in preference to “evaluation capacity”.
Differing definitions of evaluation capacity result in varying conceptualizations of evaluation capacity building. This is not surprising—if there are different views about what is being built, there will inevitably be different views about how to build it. Some express concern about these differing definitions, arguing that they need to be refined and made explicit in order that there can be a common understanding about the evaluation capacity building endeavor (Compton & Baizerman, 2007). Others contend that the diversity in conceptualization is necessary in a field that is highly dependent on the local context, and that a definition that is applicable in one setting may not be applicable in another (Taut, 2007a).
Having said this, there are some common threads in most definitions of evaluation capacity building. Most emphasize that it involves equipping staff within organizations with the appropriate skills to conduct rigorous evaluations, and doing so in a manner that acknowledges the local context and ensures that such evaluations become part of routine practice. Stockdill et al. (2002), for example, describe “evaluation capacity building as “… a context-dependent, intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or between one or more organizations/programs/sites.”
Most definitions also stress that in order for evaluation capacity building to achieve the outcome of enshrining high-quality evaluations in routine practice, it needs to foster a culture of organizational support for evaluation activities, which includes an appropriate learning environment and a sufficient level of resources (Milstein & Cotton, 2000; Stevenson et al., 2002; Stockdill et al., 2002). It also needs to recognize that different organizations (and individuals within those organizations) will be starting from different points (Milstein & Cotton, 2000; Stevenson et al., 2002; Stockdill et al., 2002). Cousins and Lee (2004), for example, argue that evaluation capacity building is connected to organizational development and organizational capacity to learn. Similarly, Taut (2007b) uses the term “self-evaluation capacity building” in describing her efforts to build the evaluation capacity of staff within development organizations with limited readiness for learning.
Some definitions go beyond the notion of equipping organizations to conduct evaluations, and explicitly consider evaluation capacity building to have a role in facilitating the use of evaluations. This is evident, for example, in the definition of Stockdill et al. (2002), above. Stockdill et al. (2002) are referring to the capacity of organizations to use the evaluation findings they generate to modify their services or programs. Batterbury (2002) goes further still and considers broader use within a policy context, defining evaluation capacity building as “… a multidimensional phenomenon that involves the ability to apply knowledge to policy as well as the ability to generate useable evaluation findings.”
The remainder of this paper presents a case study of evaluation capacity building, taken from our own work, and discusses the findings from the case study with reference to definitional, conceptual and practical issues in evaluation capacity building.
Section snippets
Background
In Australia, Divisions of General Practice (Divisions) have been the focus of various evaluation capacity building exercises because they are organizations that receive government funding to conduct a range of local projects targeting different health issues. Divisions are legally incorporated entities which draw their membership from the general practitioners in the geographically defined area which they serve. They provide a range of services for these general practitioners, including the
Implications for defining and conceptualizing evaluation capacity building
Our case study highlights some features of our evaluation capacity-building approach that may contribute to the debate on how the term is defined. We would certainly agree, for example, that any definition should emphasize equipping staff within organizations with the appropriate skills to conduct rigorous evaluations in a routine, ongoing fashion. We would also agree that an acknowledgement of the local context is crucial, and that this requires the fostering of a culture of organizational
Implications for the practice of evaluation capacity building
Our case study may also provide some practical lessons for others wishing to build evaluation capacity. Firstly, the roles, perspective and approaches taken by those charged with the task of building evaluation capacity need to align with the program or project context. Secondly, the variability in existing capacity needs to be taken into account, as does the readiness to develop this capacity. Thirdly, active ‘buy-in’ must be sought from those who are implementing the program or project to be
Summary and conclusions
Evaluation capacity building is becoming more and more common as calls for programs to demonstrate their effectiveness become louder and louder. Despite this, evaluation capacity building has not been clearly defined. Our case study teases out some of the crucial elements of the evaluation capacity building endeavor, suggesting that any definition should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which
Acknowledgment
This work was funded by the Australian Government Department of Health and Ageing.
References (38)
- et al.
Building evaluation capacity in human service organisations: A case study
Evaluation and Program Planning
(2002) Building evaluation capacity: Linking programme theory, evidence, and learning
(2002)Evaluation capacity building: A tale of value adding
Evaluation Journal of Australisia
(2005)Primary health care research, evaluation and development
(2000)- et al.
Defining evaluation capacity building
American Journal of Evaluation
(2007) - et al.
Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base
The Canadian Journal of Program Evaluation
(2004) - et al.
Research capacity in general practice: Opinions from the field
Australian Family Physician
(2002) - et al.
Primary care-led mental health service reform: An outline of the better outcomes in mental health care initiative
Australasian Psychiatry
(2002) - et al.
General practitioners’ response to depression and anxiety in the Australian community: A preliminary analysis
Medical Journal of Australia
(2004) - Horton, D., Alexaki, A., Bennett-Lartey, S., Noële Brice, K., & Campilan, D., et al. (2003). Evaluating capacity...
Evaluating the access to allied health services component of the better outcomes in mental health care initiative: Fourth interim evaluation report
Knowledge utilization: What is it?
Science Communication
Research on the utilization of evaluations: A review and synthesis
Evaluation Review
Teaching people to fish? Building the evaluation capability of public sector organisations
Evaluation
Defining concepts for the presidential strand on building evaluation capacity
Cited by (50)
Exploring data use in nonprofit organizations
2023, Evaluation and Program PlanningCitation Excerpt :As noted, the construct remains undefined in other research, yet the items used in Carman and Fredericks (2010) focus on difficulties in identifying and capturing data, a focus common elsewhere in the literature (e.g., Andrews et al., 2016; Leake et al., 2007). The implications of sustainability of evaluation and data practices are discussed briefly in Naccarella et al. (2007), however, evaluation capacity has yet to receive a formal treatment which addresses all such concerns. In general, the theories prominent in this literature are centered around features external to the nonprofit organization, such as funders, norms, and contracts, thereby failing to adequately conceptualize the organization.
Putting evaluation capacity building in context: Reflections on the Ontario Brain Institute's Evaluation Support Program
2020, Evaluation and Program PlanningEvaluation capacity building in the nonformal education context: Challenges and strategies
2020, Evaluation and Program PlanningCitation Excerpt :However, none of the literature linked ECB challenges to specific strategies. Our study, documented the challenges and strategies of ECB similar to those discussed in multiple literature sources (e.g., Hudib et al., 2016; Khan, 1998; King & Volkov, 2005; Naccarella et al., 2007), and provides a unique framework (see Table 4) elaborating what specific strategies are available to overcome each of the identified ECB challenges. ECB professionals can use this table as a framework to identify specific strategies to overcome the ECB challenges they are facing in their organizations.
Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector
2019, Evaluation and Program PlanningNarrative review of strategies by organizations for building evaluation capacity
2016, Evaluation and Program PlanningApplication of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned
2015, Evaluation and Program Planning