Evaluation Capacity Development Research Articles

Challenging democracy, human rights, and governance evaluation capacity development through a transformative lens

AbstractDemocracy, human rights, and governance (DRG) evaluation practice faces diverse challenges, not only because of the complexity of intervention designs and implementation but also due to the complexity of the fragile contexts in which DRG evaluation is generally applied. Evaluating in fragile contexts demands responsiveness to the complexity and provides challenges to define what capacities and competencies a DRG evaluator should have. In 2019, the International Organization for Cooperation in Evaluation (IOCE) was funded to design a set of tools to develop individual and institutional DRG evaluation capacities in an online training course and an associated toolkit. These materials aim to advocate for more DRG evaluation for a stronger democracy, advancing human rights, and increased governance. Using a transformative lens, this chapter critically reviews the challenges and constraints faced while developing DRG evaluators’ capacities. This critical analysis of complexity with a transformative lens adds insights into the practice of evaluation in the pursuit of human rights and social, economic, and environmental justice.

Developing Evaluation Approaches for an Anti-Human Trafficking Housing Program

The increasing number of anti-trafficking organizations and funding for anti-trafficking services have greatly out-paced evaluative efforts resulting in critical knowledge gaps, which have been underscored by recent recommendations for the development of greater evaluation capacity in the anti-trafficking field. In response to these calls, this paper reports on the development and feasibility testing of an evaluation protocol to generate practice-based evidence for an anti-trafficking transitional housing program. Guided by formative evaluation and evaluability frameworks, our practitioner-researcher team had two aims: (1) develop an evaluation protocol, and (2) test the protocol with a feasibility trial. To the best of our knowledge, this is one of only a few reports concerning anti-trafficking housing program evaluations, particularly one with many foreign-national survivors as evaluation participants. In addition to presenting evaluation findings, the team herein documented decisions and strategies related to conceptualizing, designing, and conducting the evaluation to offer approaches for future evaluations.

Strengthening Anglophone Africa M&E systems: A CLEAR-AA perspective on guiding principles, challenges and emerging lessons

Background: Evaluation capacity development (ECD) is evolving to adapt to 21st century governance and development contexts across the world. Consequently, the ECD community is seized with processes of developing, implementing and sharing best ECD practices that are able to build strong and resilient individual and institutional ME capacities.Objectives: The article seeks to contribute to the on-going discourse and practice regarding evaluation capacity development (ECD) approaches and interventions in Anglophone Africa, Africa, the Global South and global best practice.Method: The article’s methodology is essentially centred on action research pursued during the course of co-planning and designing ECD interventions across English-speaking African countries, empirical data as well as the authors’ experiences and insights gained from leading ECD interventions across African countries.Results: The Centre for Learning on Evaluation and Results – Anglophone Africa’s (CLEAR-AA) principle of partnering and engaging governments and development partners throughout the life cycle of ECD interventions is paramount to cultivating country-owned and led national monitoring and evaluation (ME) systems that are sustainable and are able to facilitate a shift towards evidence-based decision-making.Conclusion: The value of this CLEAR-AA mission is that it offers the possibility of improving decision-making, policymaking and implementation and better service delivery and development outcomes in English-speaking Africa. While noting well-known ME challenges that impact the success of ECD interventions in Africa, the article highlights forward-looking CLEAR-AA strategies that seek to improve the success and impact of its ECD interventions in Anglophone Africa. Such strategies include the identification and use of ME champions in each country; working to integrate the ME function within public sector decision-making, development planning, implementation and budgeting processes; and building ECD partnerships with governments, non-governmental institutions and development partners who have intricate knowledge of country ME systems.

Grounding evaluation capacity development in systems theory

While “systemic thinking” is popular in the context of capacity development and evaluation, there is currently a lack of understanding about the benefits to employing systems theory in evaluation capacity development. Systems theory provides a useful orientation to the work involved in complex systems (e.g. national evaluation systems). This article illustrates how evaluation capacity development practitioners can use systems theory as a conceptual tool to gain a better understanding of the functional aspects and interrelationships present within a given evaluation system. Specifically, the systems theory perspective can help elucidate the reasons for the success or failure of a given evaluation capacity development program or activity. With the goal of motivating evaluation capacity development practitioners to use systems theory in their work, this article presents a systems theory framework for evaluation capacity development and offers practical examples of how it can be adopted.

Empowering Indigenous Communities Through a Participatory, Culturally Responsive Evaluation of a Federal Program for Older Americans.

This article describes our experience of conducting a 5-year, culturally responsive evaluation of a federal program with Indigenous communities. It describes how we adapted tenets from "participatory evaluation models" to ensure cultural relevance and empowerment. We provide recommendations for evaluators engaged in similar efforts. The evaluation included stakeholder engagement through a Steering Committee and an Evaluation Working Group in designing and implementing the evaluation. That engagement facilitated attention to Indigenous cultural values in developing a program logic model and medicine wheel and in gathering local perspectives through storytelling to facilitate understanding of community traditions. Our ongoing assessment of program grantees' needs shaped our approach to evaluation capacity building and development of a diverse array of experiential learning opportunities and user-friendly tools and resources. We present practical strategies from lessons learned during the evaluation design and implementation phases of our project that might be useful for other evaluators.

Capacidades en evaluación en los gobiernos locales para la implementación de Agenda 2030. Tres experiencias de medición en México

The 2030 Agenda has, at least in the discourse, a marked orientation towards the local scope as the immediate space for development. This has been one of the main challenges on the international agenda since, in turn, at the local levels fewer capacities are available for the implementation of sustainable development. The essay proposes the strengthening capacities in evaluation approach as an alternative within the implementation of the 2030 Agenda, since it is oriented towards the generation of evidence to inform decision-making and, even further, to improve local conditions for sustainable development. First, a review of this international agenda framework and the place of the local scope within which it is carried out. Secondly, the adoption of the Evaluation Capacity Development (ECD) approach is suggested to be implemented at the local level. Finally, three relevant experiences of measuring capacities in the local governments of Mexico are presented.

Can massive open online courses fill African evaluation capacity gaps?

Background: Theory of Change for Development is a free online course developed at an African institution to strengthen evaluation capacity in the region. Massive Open Online Courses (MOOCs) provide a platform for building skills at scale in the region. Scholars of evaluation have long pointed to a gap between supply and demand that frustrates both evaluation practitioners and commissioners. This article explores the possibilities and limitations of MOOCs to bridge this gap. Objectives: This article seeks to explore the role open online training can play in evaluation capacity development. It considers the implications of one course’s experience of evaluation-sector-blended learning and instructional design in Africa for strengthening regional communities of practice, and for evaluation capacity development. Method: This article draws on current theories on online learning, specific course data from the Theory of Change course as well as experiential learning from two processes of revision and delivery of the MOOC to explore open online learning as an approach to evaluation capacity development. Results: This study has found that MOOCs can play important roles in evaluation capacity development. They can raise awareness of methods and tools that are not part of traditional monitoring and evaluation curriculum, their reach for evaluation practitioners is wide and they build much needed networks in the sector. However, their role in skills development has limits, particularly if they are delivered in isolation of other accessible contact learning options. Conclusion: Massive Open Online Courses should be one part of an evaluation capacity development strategy. They support blended learning, expand access to certain core practical skills and link evaluation practitioners to other real outlets of learning and exchange.

Issues in Japanese Development Cooperation: Japan's ODA Evaluation and Evaluation Criteria

Specific criteria are important for conducting evaluations and determining donor stances on Official Development Assistance (ODA), as well as for ascertaining how donors intend to use the evaluation results. This study examined how such criteria have been used in Japan’s project-level ex-post evaluations; the following paper discusses the related challenges and concerns to identify opportunities for improving the current system. A literature review revealed that, when conducting ODA evaluations, most donor agencies adopt the five Development Assistance Committee criteria. The results suggest that a conceptual framework should be used to conduct these evaluations. This will improve quality while also delivering information that meets the needs of the evaluation users. This framework should be further developed using information from research based on practical case studies of meta-level and organizational evaluation capacity development.

The six-sphere framework: A practical tool for assessing monitoring and evaluation systems

Background: Successful evaluation capacity development (ECD) at regional, national and institutional levels has been built on a sound understanding of the opportunities and constraints in establishing and sustaining a monitoring and evaluation system. Diagnostics are one of the tools that ECD agents can use to better understand the nature of the ECD environment. Conventional diagnostics have typically focused on issues related to technical capacity and the ‘bridging of the gap’ between evaluation supply and demand. In so doing, they risk overlooking the more subtle organisational and environmental factors that lie outside the conventional diagnostic lens.Method: As a result of programming and dialogue carried out by the Centre for Learning on Evaluation and Results Anglophone Africa engaging with government planners, evaluators, civil society groups and voluntary organisations, the author has developed a modified diagnostic tool that extends the scope of conventional analysis.Results: This article outlines the six-sphere framework that can be used to extend the scope of such diagnostics to include considerations of the political environment, trust and collaboration between key stakeholders and the principles and values that underpin the whole system. The framework employs a graphic device that allows the capture and organisation of structural knowledge relating to the ECD environment.Conclusion: The article describes the framework in relation to other organisational development tools and gives some examples of how it can be used to make sense of the ECD environment. It highlights the potential of the framework to contribute to a more nuanced understanding of the ECD environment using a structured diagnostic approach and to move beyond conventional supply and demand models.

Application of game theory and new institutional economics in establishing a National Voluntary Organisation for Professional Evaluation in Nigeria

Background: In Nigeria, there is a plethora of evaluators found in the over 90 universities, specialised educational institutions and private research organisations. However, there is limited or no opportunity for networking among the evaluators from similar and different programmatic specialisations. After applying the determinant framework to assess the evaluation capacity development situation in Nigeria, we agree on the importance of supporting the establishment of a National Voluntary Organisation for Professional Evaluation (VOPE).Objectives: Several leaders in evaluation were competing recently for occupying the national space reserved for a National VOPE. The main objective was to encourage partnership.Methods: We used a powerful theoretical framework provided by game theory and new institutional economics. We analysed the situation and identified the challenge they are facing as a Nash Equilibrium-of-a-Game View of Institutions: each player knows the equilibrium strategies of the other players and no player has anything to gain by changing only his or her own strategy. This explains why Nigerian evaluation leaders were not able to cooperate for the last two decades.Results and conclusion: To break this barrier, we proposed a new deal to the leaders that had the advantages of reshaping the ‘rules of the game’. We proposed a federation of associations, akin to a coalition in game theory. The result was that all leaders came together under this umbrella organisation, to celebrate the evaluation year in 2015 and committed under the Abuja Declaration on Evaluation to register and establish an association, with an elected board, a written constitution and election bylaws. The association is governed by a Board of Trustees, which is chaired by the former Minister of Planning. Elections are planned for the end of 2017.

Evaluation capacity assessment of the transport sector in South Africa: An innovative approach

Background: This article was based on the study on the assessment of evaluation capacity in the transport sector in South Africa. The purpose of the study was to test the Six Sphere Framework (SSF), which is an innovative evaluation capacity diagnostic tool developed by the Centre for Learning on Evaluation and Results (CLEAR-AA) based in South Africa.Objectives: The article presents the findings emanating from the study and new knowledge on how evaluation capacity assessments can be conducted in the transport sector in South Africa.Method: A variety of methods including a survey, semi-structured interviews, document review and focus group discussions were used to collect data from primary and secondary sources. In all the cases, research questions were structured around the six components of the framework. Similarly, the presentation of the findings was arranged in themes that mirror these components.Results: The article locates the SSF within the current evaluation capacity development literature and argues that existing evaluation capacity assessment tools are inadequate to understand pertinent issues affecting the use of evidence in the transport sector in South Africa.Conclusion: In this regard, the framework is recommended as an innovative tool to assist evaluation practitioners and scholars to better understand evaluation capacity constraints within a broader context that involves logistical, technical, contextual, social and political dimensions. It also offers an important insight on how these components interfaced to shape the organisational value system that impacts the use of evidence in the transport sector in South Africa.

Monitoring and Evaluation Capacity Development of Municipalities in the Province of KwaZulu-Natal, South Africa

The purpose of the study was to assess the monitoring and evaluation capacity development in local municipalities in the KwaZulu-Natal Province. The research method included the case study and the ...

Evaluation Capacity Development for Educational Development: Case Study of the Policy-level Evaluations of Japan’s ODA in the Education Sector

Evaluation Capacity Development for Educational Development: Case Study of the Policy-level Evaluations of Japan’s ODA in the Education Sector

Stretching between learning and accountability: Experiences of South African non-governmental organisations

This article contributes to knowledge around organisational learning in relation to Monitoring and Evaluation (M&E) processes of non-governmental organisations (NGOs).While learning and accountability are two fundamental purposes of M&E, in practice NGOs often perceive donor accountability as the only function of their organisation’s M&E system. Learning through meaningful monitoring of actions is a necessary process to satisfy effective functioning of organisations working on social change.The article is based on action research, which included qualitative methods such as case studies, focus group interview(s), semi-structured interviews and questionnaires with participating organisations from an 18-month Monitoring and Evaluation Capacity Development Programme of the Community Development Resource Association, which involved nine South African NGOs. Organisations benefited from the programme through peer learning, consulting and research linked to the programme.The research revealed that NGOs have a perception of M&E as an accountability procedure that has been imposed on them by donor communities and are resistant to it as well as to rigid data collection.Organisations require specific capacities to realise their power and to be able to negotiate more developmental approaches to M&E within their organisations and with donors. The model is suggested that fosters learning in M&E systems and includes two interlinked processes: self-awareness (a sense of core organisational values and intuitive ability) and awareness about the outside world and the effects of organisations’ work.The model for M&E systems can be explored further and help those organisations who are working towards better balance between learning and accountability.

Performance measurement as precursor to organizational evaluation capacity building

The widespread use of performance measurement and program evaluation in public administrations worldwide has been met with varying degrees of success, in terms of increased transparency and effectiveness. In many cases, the lack of impact of these functions is attributed to insufficient organizational knowledge and capacity to implement proper monitoring and evaluation systems. Both of these management tools are recognized in their own right as having the potential to contribute to ongoing decision-making and budgetary allocations within public organizations; however, they have complementary roles in terms of producing ongoing versus periodic information, and focusing on outputs and early outcomes versus longer-term program objectives. This paper attempts to bridge these two functions by proposing that performance measurement can act as a precursor to the development of organizational evaluation capacity, by providing some of the building blocks required to develop an evaluative culture within an organization. Five models of organizational evaluation capacity were analyzed to identify the extent to which performance measurement contributes to evaluation capacity building.

Improving Nutritional Habits to Decrease Childhood Obesity through Development of Monitoring and Evaluation Capacity in a Community Based Organization in Chicago

Improving Nutritional Habits to Decrease Childhood Obesity through Development of Monitoring and Evaluation Capacity in a Community Based Organization in Chicago

A reflection on the partnership between government and South African Monitoring and Evaluation Association

Established in 2005, the South African Monitoring and Evaluation Association (SAMEA) is celebrating its tenth anniversary as a voluntary organisation for professional evaluation (VOPE). This VOPE occupies a special space in South African society as an independent and credible voice of monitoring and evaluation practitioners, drawing its membership from individuals working for government, academia, civil society, consultancies, donor agencies and private sector foundations, including students and the international community. As the biggest user of evaluations, government has become a natural partner for SAMEA since its first conferences organised in partnership with the Public Service Commission. The establishment of the Department of Performance (now Planning,) Monitoring and Evaluation (DPME) in the Presidency of South Africa in 2010 created new impetus for the institutionalisation of evaluation in government through the establishment of the National Evaluation System (NES). In 2012, DPME entered into a formal partnership with SAMEA to enhance monitoring and evaluation (M&E) capacity and continuous professional development. Working with the PSC and the DPME as well as other key partners like universities and development agencies, the role of SAMEA has been enhanced via the establishment of provincial associations, direct input to the development of government policy, evaluation standards, competencies and training – and co-hosting of high-profile conferences and workshops. The partnership between SAMEA and the government of South Africa is extremely useful for ensuring evaluation capacity development and institutionalisation of the practice in government and society. There have been tensions, especially as government’s role in evaluation has strengthened. However, the significance of the partnership between SAMEA and the DPME has gained international interest and appreciation because of its structured nature, mutual respect, collaboration on capacity development and practical implementation of innovative projects.

A holistic, learning-centred approach to building evaluation capacity in development organizations

There is growing pressure on development organizations to improve their evaluation systems and capacities. This presents considerable challenges for time- and resource-poor organizations in developing countries. Evaluation capacity development (ECD) approaches are needed that are appropriate and effective for such organizations. We argue that this requires a long-term, holistic, participatory, learning-centred approach that aims to develop learning organizations and build the capacity of whole organizations and their stakeholders. It also needs to incorporate local knowledge and ideas and ongoing meta-evaluation of ECD activities. We describe this approach and how it was applied in a four-year action research project with a non-governmental organization in Nepal. Drawing on findings from this project and various follow up activities, we suggest some principles and strategies for designing and implementing an effective and sustainable approach to ECD that can help to address the many challenges and issues we have identified.

Tensions, Challenges and Issues in Evaluating Communication for Development

AbstractThe complexity of development and social change and growing tensions between dominant results-based and emerging learning and improvement-based approaches to evaluating development interventions have created major challenges for the evaluation of communication for development (C4D). Drawing on our recent research, we identify significant tensions, challenges and issues in evaluating C4D. They include contextual and institutional challenges, problems with attribution and unrealistic timeframes, a lack of capacities in both evaluation and C4D, and a lack of appreciation, funding and support for approaches that are more appropriate for the evaluation of C4D.We propose various strategies that can help to address these challenges and issues, including using a rigorous mixed methods approach, and implementing long-term, holistic evaluation capacity development at all levels and our new framework for evaluating C4D. These and other strategies can help to create a supportive environment in which new ideas and approaches can flourish, more sustainable outcomes of C4D can be achieved, and C4D organisations can become more sustainable and effective. The implications for C4D policy are considered.

Case study of an evaluation coaching model: Exploring the role of the evaluator

This study examined the role of the external evaluator as a coach. More specifically, using an evaluative inquiry framework (Preskill & Torres, 1999a; Preskill & Torres, 1999b), it explored the types of coaching that an evaluator employed to promote individual, team and organizational learning. The study demonstrated that evaluation coaching provided a viable means for an organization with a limited budget to conduct evaluations through support of a coach. It also demonstrated how the coaching processes supported the development of evaluation capacity within the organization. By examining coaching models outside of the field of evaluation, this study identified two forms of coaching — results coaching and developmental coaching — that promoted evaluation capacity building and have not been previously discussed in the evaluation literature.