Organizational Evaluation Capacity Research Articles

Capturing Evaluation Capacity: Findings from a Mapping of Evaluation Capacity Instruments

This article surveys the literature on organizational evaluation capacity (EC) instruments. Over the past 25 years, articles have been published in four waves covering the development of organizational evaluation capacity models, surveys of the evaluation capacity and practice landscape, the development of EC instruments, and their replication and adaptation. Altogether, the authors identified 20 studies applying 16 EC instruments. Several studies applied variants of the same instrument. We found instruments of three types: checklists, rubrics, and questionnaires. The instruments vary in terms of type, purpose, scoring, dimensions, number of items, and validation processes. In general, the validation studies demonstrated acceptable face, content, construct validity, and internal consistency. Only two studies tested for predictive validity. No studies covered concurrent validity and test–retest, or inter-rater reliability.

Evaluation Literacy and Use in Community Organizations: Insights and Implications for Evaluation Capacity Building

Past research on evaluation capacity in community organizations has focused on their capacity to conduct rather than use evaluation. This study sought to address this gap by exploring evaluation use and the supporting processes, systems, and structures in the community sector. The authors administered an online survey to 102 individuals working or involved in community organizations to measure their perceptions of their organization’s capacity to use evaluation. The survey was based on an adaptation of Bourgeois and Cousins’ evaluation capacity framework and a related assessment tool, the Organizational Evaluation Capacity Assessment Instrument, for the community sector context. Their findings offer empirical insights into three critical dimensions of the adapted framework associated with evaluation use in community organizations: evaluation literacy, organizational decision-making, and learning benefits. This exploratory study generated several key findings, including the need for tools and policies that establish a strong link between evaluation and decision-making, expanding the scope of engagement in data interpretation beyond primary users, and extending the dissemination of findings to external stakeholders. Implications for practice and research are discussed.

Interorganizational evaluation capacity building in the public, health and community sectors

Evaluation capacity building is generally conceptualized as occurring either at the individual or the organizational levels. However, ongoing societal crises require organizations within or across sectors to work together to find solutions to complex problems and to evaluate joint initiatives. Interorganizational evaluation capacity is required to ensure the ongoing conduct and use of evaluations to support interorganizational decision-making and improvement. This exploratory study describes and analyzes four cases of interorganizational evaluation capacity building initiatives in the public, health and community sectors in Canada and Denmark to identify their key dimensions. Preliminary findings highlight the importance of developing individual and organizational evaluation capacity as well as the need to provide stakeholders with interorganizational evaluation training and projects in which they can work together to better learn about each other’s organizations and challenges and find solutions to common problems.

Exploring the value of pro bono evaluation work as a method to build evaluator competencies and strengthen organizational capacity

The Cultivating Evaluation Education and Development (CEED) program engages early career evaluators in an experiential learning experience by supporting them as they lead a pro bono evaluation for a local nonprofit community organization. We explored the value of this professional development to the early career evaluators, to the sponsor organization, and to the nonprofit organizations receiving CEED services by examining findings from six CEED projects. We found that early career evaluators self-reported gains in confidence and competence especially in four American Evaluation Association Evaluator Competencies (2018) domains - professional practice, methodology, planning and management, and interpersonal skills. The initiative allowed the sponsor organization to expand connections with community organizations, implement work consistent with the mission, and provide new mentoring opportunities. Representatives from the recipient nonprofit organizations reported they gained a deeper understanding of evaluation, improved their dissemination efforts, and built organizational evaluation capacity.

Developing evaluation capacities in integrated care projects: Lessons from a scientific support mission implemented in Belgium.

The capacity of self-assessment, to learn from experience, to make information-based decisions, and to adapt over time are essential drivers of success for any project aiming at healthcare system change. Yet, many of those projects are managed by healthcare providers' teams with little evaluation capacity. In this article, we describe the support mission delivered by an interdisciplinary scientific team to 12 integrated care pilot projects in Belgium, mobilizing a set of tools and methods: a dashboard gathering population health indicators, a significant event reporting method, an annual report, and the development of a sustainable "learning community." The article provides a reflexive return on the design and implementation of such interventions aimed at building organizational evaluation capacity. Some lessons were drawn from our experience, in comparison with the broader evaluation literature: The provided support should be adapted to the various needs and contexts of the beneficiary organizations, and it has to foster experience-based learning and requires all stakeholders to adopt a learning posture. A long-time, secure perspective should be provided for organizations, and the availability of data and other resources is an essential precondition for successful work.

Evaluation policy and organizational evaluation capacity building: A study of international aid agency evaluation policies

AbstractResearch and theory on evaluation capacity building (ECB) and organizational evaluation capacity have been developing at a good pace over the past decade. On the other hand, there is a paucity of research on the nature and consequences of organizational evaluation policy. Evaluation policies are developed and implemented ultimately to inform and shape evaluation practice and its consequences. It is therefore natural to consider the interface between evaluation policy and ECB. The present exploratory descriptive study examines 52 evaluation policies from bilateral and multilateral aid agencies to explore connections between evaluation policy content and ECB principles and considerations. The results shed light on some interesting relationships; they are discussed in terms of evaluation use, evaluation purposes, and organizational leadership. The study also resulted in a revision of Trochim's (2009) definition of evaluation policy and a refinement and expansion of his eight‐category taxonomy. Implications for ongoing inquiry are considered and practical implications are offered to organization members and evaluation policy developers.

Toward an Evidence-Based Approach to Building Evaluation Capacity

Abstract: Organizations are required to evaluate their programs for both learning and accountability purposes, which has increased the need to build their internal evaluation capacity. A remaining challenge is access to tools that lead to valid evidence supporting internal capacity development. The authors share practical insights from the development and use of the Evaluation Capacity Needs Assessment tool and framework and implications for using its data to make concrete decisions within Canadian contexts. The article refers to validity evidence generated from factor analyses and structural equation modelling and describes how applying the framework can be used to identify individual and organizational evaluation capacity strengths and gaps, concluding with practice considerations and future directions for this work.

Evaluation Policy and Organizational Evaluation Capacity Building: Application of an Ecological Framework across Cultural Contexts

Background: Research on the role and effects of evaluation policy is limited. Some research on the policy’s role in enhancing organizational evaluation capacity (EC) is beginning to accrue but to date it has been limited largely to global Western evaluation contexts.
 Purpose: We employed an ecological conceptual framework arising from our own empirical research to explore the interface between evaluation policy and EC in non-western contexts. We asked—To what extent does this framework resonate across these contexts? In the selected non-Western context, what are the salient variables moderating the relationship between policy and EC in the selected contexts? Are there differences across countries?
 Setting: The present research is focused on perceptions about evaluation culture and experiences in two countries situated in the Middle East and North Africa (MENA) region, namely Turkey and Jordan.
 Intervention: Not applicable.
 Research design: We conducted focus groups within the respective countries with a combined total of 18 participants associated with country-level voluntary organizations for professional evaluation (VOPE). Participants worked in government, non-governmental aid agencies, universities and private sector organizations.
 Data collection and analysis: We introduced the focus group participants to our ecological framework and then guided the conversation using semi-structured questions. Data were audio-recorded, transcribed and subsequently thematically analyzed using NVivo.
 Findings: The ecological framework was found to resonate well but the findings were weighted heavily toward macro-level contextual variables. Even though important contextual and cultural differences between Turkey and Jordan were evident, leadership emerged as a significant meso-level moderating variable in both settings. The discussion of the results included implications for ongoing research.
 Keywords: Evaluation capacity building; evaluation policy; program evaluation; cultural context

Application of a Multisite Empowerment Evaluation Approach to Increase Evaluation Capacity Among HIV Services Providers: Results From Project Pride in Chicago

In 2015, the Centers for Disease Control and Prevention (CDC) funded Project PrIDE, a national initiative to implement and evaluate demonstration projects to increase PrEP uptake among HIV-negative individuals and to re-engage HIV-positive individuals in HIV care. Our team served as the Evaluation Center for Project PrIDE organizations in Chicago and used an empowerment evaluation (EE) approach to enhance evaluation capacity at these organizations. To evaluate our approach, we assessed organizations' evaluation capacity and engagement in technical assistance and capacity building activities in 2016 and 2018. Respondents who self-reported higher engagement with the Evaluation Center and who spent a greater number of hours engaged with our evaluators experienced greater increases in evaluation capacity tied to implementation of evaluation activities and technical assistance utilization. These findings demonstrate that multisite EE can be successfully applied to increase the evaluation capacity of organizations providing both HIV prevention and care services.

Issues in Japanese Development Cooperation: Japan's ODA Evaluation and Evaluation Criteria

Specific criteria are important for conducting evaluations and determining donor stances on Official Development Assistance (ODA), as well as for ascertaining how donors intend to use the evaluation results. This study examined how such criteria have been used in Japan’s project-level ex-post evaluations; the following paper discusses the related challenges and concerns to identify opportunities for improving the current system. A literature review revealed that, when conducting ODA evaluations, most donor agencies adopt the five Development Assistance Committee criteria. The results suggest that a conceptual framework should be used to conduct these evaluations. This will improve quality while also delivering information that meets the needs of the evaluation users. This framework should be further developed using information from research based on practical case studies of meta-level and organizational evaluation capacity development.

2397

2397

Exploring Organizational Evaluation Capacity and Evaluation Capacity Building

Researchers have conducted numerous empirical studies on evaluation capacity (EC) and evaluation capacity building (ECB) in Western cultural settings. However, little is known about these practices in non-Western contexts. To that end, this study identified the major dimensions of EC and feasible ECB approaches in Taiwanese elementary and junior high schools. Using a Delphi technique with 23 experts, the research sought consensus on the components of EC organized in three categories (evaluation culture, evaluation infrastructure, and human resources) and on approaches to building it in Taiwanese schools. The study also identified school-driven and government-driven approaches to ECB in this context. Although the findings support the major dimensions and approaches identified in the Western literature, unique differences emerged in the Taiwanese context. The article concludes with implications for theory and practice.

Performance measurement as precursor to organizational evaluation capacity building

The widespread use of performance measurement and program evaluation in public administrations worldwide has been met with varying degrees of success, in terms of increased transparency and effectiveness. In many cases, the lack of impact of these functions is attributed to insufficient organizational knowledge and capacity to implement proper monitoring and evaluation systems. Both of these management tools are recognized in their own right as having the potential to contribute to ongoing decision-making and budgetary allocations within public organizations; however, they have complementary roles in terms of producing ongoing versus periodic information, and focusing on outputs and early outcomes versus longer-term program objectives. This paper attempts to bridge these two functions by proposing that performance measurement can act as a precursor to the development of organizational evaluation capacity, by providing some of the building blocks required to develop an evaluative culture within an organization. Five models of organizational evaluation capacity were analyzed to identify the extent to which performance measurement contributes to evaluation capacity building.

The "Usability" of Evaluation Reports: A Precursor to Evaluation Use in Government Organizations

Background: According to the Treasury Board of Canada’s Policy on Evaluation (2009), evaluations produced by federal government departments must contribute to decision-making at an organizational level (mainly summative) as well as a program level (mainly formative). Previous research shows that although the formative objectives of evaluation are generally reached, the use of evaluation for broader, budgetary management is limited. However, little research has been conducted thus far on this issue. Purpose: This study investigates the extent to which program evaluation is used in the Canadian federal government for budgetary management purposes. Setting: This paper outlines the results obtained following the first component of a two-pronged research strategy focusing on evaluation use in Canadian federal government organizations. Intervention: N/A Research Design: Two federal agencies were recruited to participate in organizational case studies aiming to identify the factors that facilitate the use of evaluation for budgetary reallocation exercises. Data Collection and Analysis: This report presents the findings from a detailed analysis of evaluation reports published by both agencies between 2010-2013. The data were collected from public evaluation reports and analyzed using NVivo. Findings: The preliminary findings of the study show that instrumental use has occurred or can be expected to occur, based on the types of recommendations outlined in the reports reviewed and on the responses to the evaluations produced by program managers. Keywords: evaluation use; organizational evaluation capacity; instrumental use; evaluation reports; Canadian federal government; document review; organizational decision-making.

Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned

Organizational evaluation capacity (EC) has received significant attention in the evaluation research literature in the past decade. Much of the focus has been on defining organizational evaluation capacity, which can be thought of as the competencies and structures required to conduct high-quality evaluation studies (capacity to do), as well as the organization's ability to integrate evaluation findings into its decision-making processes (capacity to use). This paper seeks to contribute to this growing body of knowledge through a multiple case study of EC across three different organizations (e.g., non-profit, provincial government and federal government, herein named sectors); the novelty of this particular study is that each case study is based on the use of a common measurement tool developed by Bourgeois, Toews, Whynot and Lamarche (2013). The cross-case analysis presented in the paper reveals that evaluation capacity tends to be higher, both in terms of capacity to do and capacity to use, in organizations that have developed systematic mechanisms to institute an evaluation culture within their walls. Interestingly, however, we also found that capacity to use does not first require capacity to do, as evidenced in the non-profit organization under study.

Cross‐Case Analysis and Implications for Research, Theory, and Practice

AbstractChapter 2 of this volume provided a close look at the evaluation capacity of eight distinct organizations. The organizations were selected on the basis of their interest in and commitment to evaluation as leverage for program and organizational change. Having examined each of the organizations in terms of context, evaluation drivers, evaluation capacity to do and use evaluation, and forces and factors that affect such capacity, this chapter looks across them in order to learn more about the complexities of evaluation capacity. Beginning with a cross‐case lens, we recapitulate what we found from the cases about organizational evaluation capacity and then present an emergent thematic analysis and discussion. The chapter ends with some implications for research, theory, and practice in the area.

Multiple Case Study Methods and Findings

AbstractResearch on organizational evaluation capacity building (ECB) has focused very much on the capacity to do evaluation, neglecting organizational demand for evaluation and the capacity to use it. This qualitative multiple case study comprises a systematic examination of organizational capacity within eight distinct organizations guided by a common conceptual framework. Described in this chapter are the rationale and methods for the study and then the sequential presentation of findings for each of the eight case organizations. Data collection and analyses for these studies occurred six years ago; findings are cross‐sectional and do not reflect changes in organizations or their capacity for evaluation since that time. The format for presenting the findings was standardized so as to foster cross‐case analyses, the focus for the next and final chapter of this volume.

Framing the Capacity to Do and Use Evaluation

AbstractThe construct of organizational evaluation capacity is a concept that is receiving increasing attention in theoretical and research‐based literature. It is situated within a stream of inquiry that has come to be known as evaluation capacity building (ECB). This chapter reviews evolving conceptions of ECB and recent research and theory in the area. A conceptualization of organizational capacity for evaluation is explicated. The framework addresses not only the capacity to do but also the capacity to use evaluation. This framework has evolved within our ongoing research program and has also informed other research activities focusing on the integration of evaluation into organizational culture. This chapter concludes with a discussion of implications for ongoing research and practice.

Measuring Organizational Evaluation Capacity in the Canadian Federal Government

Abstract: The development of organizational evaluation capacity has emerged in recent years as one mechanism through which evaluators can extend their influence and foster evaluation utilization. However, organizational evaluation capacity is not always easy to define, and internal evaluators sometimes struggle with the identification of concrete activities that might increase their organization’s evaluation capacity. This article describes an organizational self-assessment instrument developed for Canadian federal government organizations. The instrument is presented and described, and further details regarding its use and next steps for this area of evaluation research are also provided.

Understanding Dimensions of Organizational Evaluation Capacity

Organizational evaluation capacity building has been a topic of increasing interest in recent years. However, the actual dimensions of evaluation capacity have not been clearly articulated through empirical research. This study sought to address this gap by identifying the key dimensions of evaluation capacity in Canadian federal government organizations. The methodology used, based on Leithwood and Montgomery’s Innovation Profile approach, featured semistructured interviews with evaluation experts and a validating exercise conducted in four government organizations. The framework developed as a result of the study identifies six main dimensions of evaluation capacity (human resources, organizational resources, evaluation planning and activities, evaluation literacy, organizational decision making, and learning benefits), each one broken down into further subdimensions. The evaluation capacity of organizations on each of these dimensions and subdimensions can be described using four levels: low, developing, intermediate, and exemplary. The study found that government organizations vary in terms of their capacity from one dimension to the next, and indeed, from one subdimension to the next.