Evaluation Capacity Building Research Articles

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.

Building a sustainable organizational energy evaluation system in the Asia Pacific

In the face of climate change and with the vision for the future of humanity represented by the Sustainability Development Goals, sustainability has become associated with the major and rapid transformation of global systems and the resilience of transformed systems to adapt over time. This dynamic view of transformation has implications for both the design of transformational energy initiatives and their evaluation. As there is an urgent need to evaluate the scale, scope, nature, and sufficiency of transformation toward planetary sustainability, a framework (model) is proposed to determine the most effective energy evaluation capacity building strategies for achieving various individual and group level outcomes, most effective strategies for certain types of participants, and design of different strategies to maximize their impact in a sustainable way. Furthermore, a new energy evaluation organization is identified, which is expected to lead us into this new era.

Evaluation competencies and challenges faced by early career extension professionals: developing a competency model through consensus building

ABSTRACTPurposeTo examine the evaluation competencies extension professionals need to develop and the challenges faced by early career extension professionals when they evaluate their extension education programs.DesignWe conducted two studies that utilized a three round Delphi technique to come to consensus on the core program evaluation competencies for extension professionals and the most pervasive program evaluation challenges early-career extension professionals face. These studies were conducted in the spring and summer of 2018. We used two Delphi panels: (1) 46 evaluation specialists and (2) 30 new county extension professionals from various program areas. We inputted the final list of program evaluation competencies and challenges into a spreadsheet for comparative analysis to achieve the research purpose.FindingsThe panel of extension evaluation specialists reached consensus on 36 competencies and the panel of early career extension professionals had consensus on 26 program evaluation challenges. Comparative analysis of the results of the two Delphi studies effectively linked early career extension program evaluation challenges with the evaluation specialists identified core evaluation competencies.Practical ImplicationsWe provide a framework for designing needs-based, proactive in-service trainings to build the evaluation capacity of early-career extension professionals.Theoretical ImplicationsThis program evaluation competency model for early-career extension professionals adds to the overall theoretical body of literature focused on evaluation capacity building of extension professionals.OriginalityIt presents the US Extension Evaluation Specialists’ national consensus of the core program evaluation competencies for extension professionals. It is also the first study to compare early-career extension program evaluation challenges with a relevant evaluation competency model.

Taking a systems approach to explore the impacts and outcomes of a research and evaluation capacity building partnership: a protocol

IntroductionPartnership models that bring researchers, policymakers and service providers closer together are gaining traction as a strategy to improve public health practice. Yet, there is little evidence of how these...

Strategies to Improve Capacity for Policy Monitoring and Evaluation in the Public Sector

Scholars around the globe have contested the inadequate infrastructure and tools used by public sector agencies to monitor and evaluate public policies and programmes. Some of the urgent issues of concern deal with the inadequate human capacity in public agencies and departments to conduct fair and credible evaluations in the public sector. South Africa is not the only country that has adopted a government wide monitoring and evaluation system, other countries like Ghana, Kenya, Benin and Uganda have also endorsed formal monitoring and evaluation practice in the public sector. This article argues that monitoring and evaluation must not just measure the effectiveness and efficiency of public programmes and processes, but it must create a sustainable process whereby participants and evaluators can learn from the process. Capacity building in monitoring and evaluation must be fairly and continuous conducted to offer credible and valid information and knowledge on M&E by training agencies and institutions like universities. This theoretical paper adopted document analysis strategy to review and evaluate documents used as data source. Lessons learnt from this article contribute towards the existing strategies to enhance monitoring and evaluation.

Program Evaluation Challenges and Obstacles Faced by New Extension Agents: Implications for Capacity Building

In this era of accountability, Extension agents are expected to evaluate their programs for accountability. New Extension agents are not exempt from this expectation. If they lack evaluation capacity, this scenario can contribute to frustration and burnout. Therefore, it is paramount to explore new Extension agents' evaluation challenges and obstacles to find ways to help them build evaluation capacity. We used a modified Delphi study approach to identify and describe the most important challenges and obstacles faced by early-career Extension agents. The study panel demonstrated consensus on 26 program evaluation challenges and seven program evaluation obstacles. The findings may inform regional collaboration for evaluation competency building and promote meaningful discussions that move support beyond the status quo.

Using Action Research to Build Evaluation Capacity in Public Health Organizations


 
 
 
 Background: New practice standards in Ontario require the ongoing evaluation of public health programs. However, public health units have limited capacity to conduct and use evaluations.
 
 Purpose: The purpose of this study was to assess an action research approach as a means to build evaluation capacity in public health units.
 
 Setting: 36 Canadian public health units in Ontario.
 
 Intervention: Action research for evaluation capacity building.
 
 
 
 Research Design: Multiple-case study.
 
 Data Collection and Analysis: Qualitative, semi-structured interviews were held with study participants after the design and implementation of evaluation capacity building strategies in their organizations. Analysis was conducted using the general inductive approach (Thomas, 2006).
 
 Findings: Evaluation capacity building is well-supported by an action research approach.
 
 
 

How Evaluation Capacity Building Grows Credible and Actionable Evidence for Cooperative Extension Programs

Evaluation capacity building (ECB) is an essential element for generating credible and actionable evidence on Extension programs. This paper offers a discussion of ECB efforts in Cooperative Extension and how such efforts enable Extension professionals to collect and use credible and actionable evidence on the quality and impacts of programs. Sufficient investments in ECB, both at the individual and organizational levels, can better equip Extension to advocate for and make changes to programs, advance as a learning organization, and have a more powerful impact on communities. Furthermore, as Extension program stakeholders often have varying perspectives on the credibility of evidence, these perspectives must also be accounted for in efforts to build Extension’s evaluation capacity. Intentional investments in ECB efforts provide an opportunity for Extension to further deepen and expand impact, positioning programs to most effectively and positively benefit individuals and communities.

Evaluation competencies and functions in advertised evaluation roles in Australia

This study explores the functions of professional evaluators outlined in online job advertisements. A total of 97 job advertisements were reviewed in the study. A content analysis using a Coding Analysis Toolkit developed by Shulman was conducted to identify six main evaluation functions based on the collected data. These functions are (1) evaluation and reporting, (2) providing evaluation advice, (3) evaluation capacity building, (4) communication and engagement, (5) forming partnerships and (6) leading, managing and influencing. These functions were then compared to the Australian Evaluation Society’s (AES) Core Competency Domains. Overall, there is a broad alignment between these functions and the AES Core Competency Domains. However, the analysis shows that the delivery of culturally competent evaluations and evaluation utilisation received no mention in advertised evaluation roles. The delivery of culturally competent evaluation is essential from the perspective of ethics, validity and theory, while the utilisation of evaluation findings is important for the benefit of society.

An Evaluation of Capacity Building of SHGs in Hamirpur and Una District of Himachal Pradesh

An Evaluation of Capacity Building of SHGs in Hamirpur and Una District of Himachal Pradesh

Training as strategy for school self-evaluation capacity building

The aim of the paper is to discuss the importance of school (self-) evaluation capacity building within quality assurance system in education in Slovenia. The research is focused on self-evaluation training program in 245 educational institutions. The results demonstrate its importance and pinpoints several challenges, i.e.: “how to build organizational (leaders’) evaluation capacity”, “learning transfer in practice”, and “how to choose among various evaluation capacity building strategies”. Building upon empirical findings, the paper suggests that the training as a sole strategy for capacity building does not suffice in the competitive, quality-driven educational context. Correspondingly, the model of self-evaluation capacity building for schools is presented and discussed.

Institutionalising the evaluation function: A South African study of impartiality, use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both. ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation. FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity. Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach. Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.

Mobilising Q methodology within a realist evaluation: Lessons from an empirical study

Realist evaluation and Q methodology are established approaches in social science. However, integration of Q methodology within a realist evaluation is scarce. This article attempts to illustrate (through a recent evaluation) how Q methodology can support a realist evaluation. The article attempts to capture the philosophical compatibility of the two approaches creating an argument for Q’s integration within realist evaluation. Through the case study selected (a realist evaluation of an evaluation capacity building framework), the iterative methodological process is presented, capturing a snapshot of the findings from the evaluation. This illuminates how Q met the philosophical aims of a realist evaluation to make sense of how, why and under what circumstances a programme or intervention works. It is argued that Q methodology is entirely suitable for capturing the three key stages in a realist evaluation of developing, testing and refining programme theory. It is intended that this article can contribute to inspire other realist evaluators and methodologists to make use of Q as a tool in their evaluation.

Evaluation Capacity Building and Improvement of Monitoring and Evaluation practice among Non- Governmental Organizations in Central Eastern Counties of Kenya

Monitoring and evaluation (M&E) in projects management is an expensive but critical process that needs careful implementation because of its role in providing information necessary for decision making, planning, and policy formulation. This importance has occasioned the need to look into the quality of M&E process and a lot of effort has been put into building capacity of M&E stakeholders in projects and programs. The goal of this study was to establish evaluation capacity building (ECB) efforts and the influence of the same on general M&E practice among non-governmental organization in central eastern counties of Kenya. The study was guided by pragmatism paradigm to conduct a descriptive survey. Stratified random sampling was used to obtain the sample studied. A structured questionnaire with Likert-type questions, anchored on a five-point scale was used to collect primary data which was triangulated using data from interviews. The findings were that organisations in the region are doing a number of unstructured activities to build evaluation capacity which are done in varying degrees, these activities had influence on M&E practice. The study recommends organizations to invest in ECB activities especially those that build capacity in M&E professional development and building M&E support structures because these were perceived to contribute more to improved M&E practice. However they need to be systematized and balanced in addressing the M&E capacity need. Organizations needs to put more resources into. There is also a need to establish and test an ECB model that would be used in a simple in the region.

공공부문 프로그램평가를 위한 평가역량구축 방안의 모색

공공부문 프로그램평가를 위한 평가역량구축 방안의 모색

Evaluation Capacity Building for Informal STEM Education: Working for Success Across the Field

AbstractInformal STEM education (ISE) organizations, especially museums, have used evaluation productively but unevenly. We argue that advancing evaluation in ISE requires that evaluation capacity building (ECB) broadens to include not only professional evaluators but also other professionals such as educators, exhibit developers, activity facilitators, and institutional leaders. We identify four categories of evaluation capacity: evaluation skill and knowledge, use of evaluation, organizational systems related to conducting or integrating evaluation, and values related to evaluation. We studied a field‐wide effort to build evaluation capacity across a network of organizations and found it important to address individuals’ evaluation capacities as well as capacities at the organizational level. Organizational factors that support ECB included redundancy of evaluation capacities across multiple people in an organization, institutional coherence around the value of evaluation, and recognition that ECB can be led from multiple levels of an organizational hierarchy. We argue that the increasing emphasis on evaluation in the ISE field represents an exciting opportunity and that, with targeted strategies and investments, ECB holds great promise for the future of ISE and ISE evaluation.

Synthesis of Issues and Future Directions

AbstractIn this final chapter, we reflect on four themes that cut across the preceding chapters on evaluating outcomes in informal STEM education (ISE): validity, context, technology, and evaluation capacity building. We also note several topics that we hope will be addressed more extensively in future conversations about evaluation in ISE. These include the ethical responsibilities of evaluators, theory‐based approaches to evaluation, formal training and professional development pathways for evaluators, and funding opportunities and the structural landscape of the field.

The Funding, Administrative, and Policy Influences on the Evaluation of Primary Prevention Programs in Australia.

Evaluation of primary prevention and health promotion programs contributes necessary information to the evidence base for prevention programs. There is increasing demand for high-quality evaluation of program impact and effectiveness for use in public health decision making. Despite the demand for evidence and known benefits, evaluation of prevention programs can be challenging and organizations face barriers to conducting rigorous evaluation. Evaluation capacity building efforts are gaining attention in the prevention field; however, there is limited knowledge about how components of the health promotion and primary prevention system (e.g., funding, administrative arrangements, and the policy environment) may facilitate or hinder this work. We sought to identify the important influences on evaluation practice within the Australian primary prevention and health promotion system. We conducted in-depth semi-structured interviews with experienced practitioners and managers (n = 40) from government and non-government organizations, and used thematic analysis to identify the main factors that impact on prevention program evaluation. Firstly, accountability and reporting requirements impacted on evaluation, especially if expectations were poorly aligned between the funding body and prevention organization. Secondly, the funding and political context was found to directly and indirectly affect the resources available and evaluation approach. Finally, it was found that participants made use of various strategies to modify the prevention system for more favorable conditions for evaluation. We highlight the opportunities to address barriers to evaluation in the prevention system, and argue that through targeted investment, there is potential for widespread gain through improved evaluation capacity.

Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector

There is a lack of clarity around intra-organisational evaluation roles and pathways into these roles in non-government organisations (NGOs). This article presents three auto-narratives from the authors who are working as internal evaluators in the NGO sector. We examine this phenomenon of role ambiguity by exploring our evaluation journeys and struggles to find identities in the formal evaluation community. Findings from the auto-narratives identify implications for the evaluation field regarding professionalisation. This article explores how aspects of professionalisation, such as clarification of roles and tasks of internal evaluators, could facilitate their recruitment, assess credibility and guide career trajectory. Elucidating internal evaluation career pathways contributes to the evaluation discipline by providing information relevant for evaluation capacity building, evaluator training, and the professionalisation movement.