Theory-based Evaluation Approach Research Articles

Getting Inspired by Contribution Analysis. Impact Evaluation of the European Commission’s Social Business Initiative

This article addresses the question of how complex interventions can be adequately assessed with limited resources. In 2011, the European Commission presented the Social Business Initiative (SBI), a framework initiative to develop a favourable environment for social enterprises and the social economy. The complexity of the programme stems from its numerous components and the long timeframe for its implementation. The evaluation of the impact of the SBI was commissioned in 2019 and the final report was presented in November 2020. The article presents the study and the evaluation methodology used to analyse the impact contribution influenced by the SBI. A theory-based evaluation approach was chosen, inspired by contribution analysis. The article describes the implementation of the evaluation and discusses the suitability of the chosen method, identifying both its strengths and shortcomings in this specific context.

Combining realist evaluation and appreciative inquiry: A participatory, learning-focused methodology for tourism interventions

Useful lessons can be drawn from realist evaluation and appreciative inquiry for the evaluation of tourism interventions in complex contexts. We combine the strengths of these two approaches to design a participatory learning-focused evaluation with an emphasis on co-created intervention improvement. We apply this proposed ‘Realist Appreciative Evaluation’ on a government-led intervention aimed at supporting tourism firms to improve their sustainability practices. While realist evaluation introduces causal thinking within a framework of contexts-mechanisms-outcomes, appreciative inquiry enables participants to be solution-oriented. The study shows how evaluation can constitute a constructive, dynamic learning process for participants, rather than something to fear. Our methodology promotes collaboration and real-time change, within the rigorous theory-based realist evaluation approach, to improve the evaluation of tourism interventions.

The Use of Contribution Analysis in Evaluating Health Interventions: A Scoping Review.

Contribution Analysis (CA) is a promising theory-based evaluation approach for complex interventions, yet its application in health interventions remains largely unexplored. To bridge this gap, we conducted a scoping review to examine the extent of such applications and the methodologies, strengths, and limitations of this approach in health programming. Our comprehensive search strategy was developed and used in 15 databases to identify peer-reviewed articles from 1999 to 2023 that focused on using CA to evaluate health interventions. We then implemented rigorous double- and triple-screening processes for abstracts and full-text papers, respectively. Data were extracted and narratively summarized. Our review found seven relevant studies, which showed that CA has been employed in health promotion programs, health policies, and targeted health issues such as nutrition, cardiovascular disease, substance misuse, and suicide prevention. The studies identified strengths of using CA, including its flexible impact evaluation approach, capacity to inform decision-making, and potential to enhance understanding of health programs and policies. However, challenges such as how to determine suitable evidence levels and how to best manage resource intensity were also identified. The limited number of studies indicates that CA is still a novel approach, whereas the variation in the reporting of the studies suggests that this approach could benefit from more standardized methods and detailed stakeholder engagement strategies.

Bridging Evaluation Theory and Practice: The Contributions of John Mayne to Canadian Federal Evaluation

This paper provides a thematic review of Mayne’s contributions to Canadian federal evaluation in order to describe his influence on how the function has evolved over the past 40 years. Key themes explored in the paper include Mayne’s influence on the development of a results-based culture in federal departments and agencies, as well as his conceptual grounding of evaluation activities through theory-based evaluation approaches and, most notably, Contribution Analysis. We draw on government publications to link policy directions to Mayne’s work and we provide a prospective view on his continued influence on the function for years to come.

The logic behind the initiatives of national governing bodies in Flanders to improve organised youth sport: A theory-based evaluation approach

ABSTRACT Regular involvement in organised youth sport boosts the likelihood of lifelong sport participation. However, increasing dropout rates in sport clubs and alarming levels of young people not meeting physical activity guidelines may suggest that the organised sport sector’s efforts have been insufficient to date. This situation calls for evaluation of the applicable youth sport policy to determine how organised youth sport is intended to operate and to articulate whether and how the policy achieves its stated goals. Accordingly, this paper identifies and analyses initiatives of national governing bodies (NGBs) to increase participation rates and improve the quality of youth sport in their affiliated sport clubs, using Flanders as a case study. A theory-based evaluation was applied working with qualitative data derived from extensive document analysis and five focus groups with representatives of 19 Flemish NGBs. The results reveal that the focus of NGBs is narrowed to increasing participation rates of youth members and building a sound organisational structure in sport clubs, rather than investing in the quality of provision to maximise enjoyment. Furthermore, most of the initiatives lack a clearly articulated strategy and are based on assumed mechanisms, rather than on prior understanding of effective processes. This study adds to the scarce literature on sport policy evaluation research by mapping out NGBs’ youth sport initiatives and using a theoretical lens through which to evaluate them. For practitioners, the results demonstrate the usefulness of a logic model to evaluate youth sport policy and to optimise the effectiveness of the initiatives employed.

A theory-based evaluation of the Leadership for Universal Health Coverage Programme: insights for multisectoral leadership development in global health

BackgroundLeadership to manage the complex political and technical challenges of moving towards universal health coverage (UHC) is widely recognized as critical, but there are few studies which evaluate how to expand capacities in this area. This article aims to fill some of this gap by presenting the methods and findings of an evaluation of the Leadership for UHC (L4UHC) programme in 2019–2020.MethodsGiven the complexity of the intervention and environment, we adopted a theory-driven evaluation approach that allowed us to understand the role of the programme, amongst other factors. Data from a range of sources and tools were compared with a programme theory of change, with analysis structured using an evaluation matrix organized according to the Organisation for Economic Co-operation and Development–Development Assistance Committee (OECD-DAC) criteria. Data sources included key informant (KI) interviews (89 in total); surveys of the 80 workshop participants; a range of secondary data sources; case studies in two countries; and observation of activities and modules by the evaluator.ResultsParticipants and KIs at the global and country levels reported high relevance of the programme and a lack of alternatives aiming at similar goals. In relation to effectiveness, at the individual level, there was an increase in some competencies, particularly for those with less experience at the baseline. Less change was observed in commitment to UHC as that started at a relatively high level. Understanding of UHC complexity grew, particularly for those coming from a non-health background. Connections across institutional divides for team members in-country increased, although variably across the countries, but the programme has not as yet had a major impact on national coalitions for UHC. Impacts on health policy and practice outcomes were evident in two out of seven countries. We examined factors favouring success and explanatory factors. We identified positive but no negative unintended effects.ConclusionsWhile noting methodological constraints, the theory-based evaluation approach is found suitable for assessing and learning lessons from complex global programmes. We conclude that L4UHC is an important addition to the global and national health ecosystem, addressing a relevant need with some strong results, and also highlight challenges which can inform other programmes with similar objectives.

Students-As-Teachers: Fostering medical educators.

While mounting evidence supports various benefits of Students-As-Teachers (SAT) curricula in preparing students to teach, limited SAT electives are offered across Canada. We developed a 4-week SAT selective for fourth-year medical students at the University of Toronto to enhance medical education knowledge and teaching skills. This study aimed to evaluate the SAT programme and its impact on students' development as educators, their experience as learners and educators, and their future plans for involvement with medical education. Students participated in highly interactive small group seminars and teaching opportunities in nonclinical and clinical environments. Course evaluation consisted of pre-selective and post-selective surveys and written reflections on the selective experience and future career aspirations. A theory-based evaluation approach was utilized to compare the SAT programme's theory with course outcomes. Post-SAT selective, students self-reported greater knowledge and confidence in teaching methods, provision of feedback, medical education scholarship, and interest in further medical education training. Student reflections highlighted three key themes. Identity formation as educators and the importance of mentorship in medical education aligned with our programme theory, while an unexpected outcome included a shifting perception on teaching and feedback from a learner to an educator lens. This study's findings demonstrate the ability of SAT curricula to build capacity for future medical educators. Positive factors contributing to the programme's outcomes included cohort size, course and seminar structure, and active group participation. Future iterations may explore use of flipped classroom models, additional clinical teaching opportunities, and near-peer teaching.

Evaluating behavioural changes for climate adaptation planning

Most climate adaptation plans expect stakeholders to change their behavior as part of building resilience. Given its long-term and complex nature, monitoring and evaluation is a key requisite for climate adaptation planning. So far, behavioral aspects have received only limited attention in the evaluation approaches for climate adaptation planning. This article proposes a theory-based evaluation approach based on the theory of planned behavior, for the evaluation of climate adaptation. A local climate adaptation programme for watershed development in rural India provides an illustrative case for this approach. For this case, the approach helped to uncover important factors that influence behavioral intentions, which were different for different groups in the farming community. Additionally, it helped to put behavioral change in a longer-term perspective. The illustrative case also suggests certain improvements for evaluations based on the theory of planned behavior.

Strengthening decision-making on sustainable agricultural intensification through multi-stakeholder social learning in sub-Saharan Africa

ABSTRACT Increasing and competing demands on agriculture in sub-Saharan Africa mean that policy and investment decisions become more complex. Despite growing consensus on the need for sustainable agricultural intensification, there is limited agreement on how to achieve this in practice. Governments and societies face uncertainty and complex choices. This paper explores the potential of Multi-Stakeholder, Social Learning (MSL) approaches, facilitated by National Learning Alliances (NLAs), to improve policy and investment decisions. Comparative evidence from a donor-supported research and learning programme in Ethiopia, Ghana, Malawi, Tanzania and Zambia is used in a theory-based evaluation approach to assess the contribution of the NLAs to capacity and practice change amongst individuals, networks and senior decision-makers. Ten outcome cases are explored, including their contribution to systemic changes in the governance of evidence. Key lessons included: the value to decision-makers of engaging with informal networks; importance of combining dialogue, deliberation and experiential learning; the need to create safe spaces in national level MSL processes; the demanding combination of facilitation skills and commitment; and appropriately flexible support. This suggests a need not only for the production of quality research, but crucially support for MSL as a means of contributing to the good governance of evidence and sustainable change.

Concepts for the design and evaluation of change-oriented research

Research funding bodies, and society in general, have high expectations for research to contribute solutions that have positive societal and environmental impact. This desire to increase societal benefits has prompted an evolution in how research is conducted, and a corresponding need for appropriate research evaluation tools and approaches. The Sustainability Research Effectiveness Program, led by Professor Brian Belcher at Royal Roads University, has responded to this need by developing a Transdisciplinary Research Quality Assessment Framework to evaluate research design and implementation and a theory-based evaluation approach to assess research contributions to outcomes and impacts.

Trends and future studies on policies to improve renewable energy share

Every country needs to set up robust policies to reach its future targets in renewable energy development. The objective of this article is to explore the trends and research opportunities in this subject area. Systematic Literature Review (SLR) is applied together with bibliometric analysis to satisfy the objective. The database used in this study is Scopus considering its size and wide acceptance by major publishers. This study applies structural literature review and bibliometric analysis within one of its stages. A simple meta-analysis is cooperated prior to other analyses in this study. Various models are exercised to analyse important aspects in renewable energy policies, namely Input-output models, CGE models, Theory-based evaluation, multi-criteria analysis, and hybrid approach. Nonetheless, renewable energy is considered as a complex system. Therefore, advanced simulation techniques such as the discrete event simulation model, the agent-based model, system dynamics, and mixed methods are considered. Due to its characteristics, the mixed method is the most comprehensive approach. However, because mix methods have not been supported by proper tools or simulation software, agent-based model and system dynamics are proposed individually. Future research also should consider other than a market-based approach to develop robust policies to reach the ideal renewable energy share.

Impact assessment of Vision 2023 defense technology foresight

PurposeThe purpose of this paper is to provide information about the long-term ex-post impacts of Vision 2023 technology foresight (TF) on the defense sector and to identify critical success factors (CSFs) of impactful foresight.Design/methodology/approachIn the present research, a theory-based evaluation approach was adopted with the logic-model of defense TF to identify the intended outcomes, impacts and leading mechanisms. The impact assessment framework developed by Johnston, R. (2012) was adopted to identify (un)intended impacts and possible measures.FindingsTF had some effects on technological developments, foresight capacity and capability and skills on the sector. The overall impact was assessed at the “some contribution” level with 2.9 out of 5.0 points. It contributed to the development of science technology and innovation (STI) policies and research and development programs, awareness-raising in STI, increase in cooperation between government-university-industry and the development of foresight culture. However, the impacts were more visible in the early years of TF. Additionally, country/sector-specific CSFs were identified. In consequence, it was proposed to measure the maturity of strategic technologies with technology readiness level as a tangible indicator.Originality/valueAccording to the authors, this is the first study to assess the long-term ex-post impact of TF in defense. An instrument was developed to assess TF’s contribution to impact measures. The constructs and weights of the instrument differentiated from the adopted framework/schema reflecting the national/sectoral context of TF. Additionally, the study revealed country/sector-specific CSFs and new tangible impact measures.

Improving Monitoring and Evaluation in the Civic Tech Ecosystem

For nearly a decade, civic tech stakeholders have been creating technology-supported solutions to civic challenges. Globally, the civic tech movement is rapidly professionalizing but has a limited history of documenting evidence of successes and challenges. Robust monitoring and evaluation in the civic tech ecosystem are necessary to create a foundation of knowledge for future initiatives. Monitoring plays a key role in improving services, pivoting approaches and guiding more efficient resource allocation. Evaluation highlights what is working, what is not working, and critically, why? In a sector that merges data, design and technology with user-centred principles, monitoring and evaluation in the civic tech ecosystem have several inherent challenges. This paper suggests that a theory-based evaluation approach called Contribution Analysis has the necessary sophistication and agility to support comprehensive monitoring and evaluation to support the growth and sustainability of the movement. This paper applies the early steps of contribution analysis to two Canadian civic tech projects to demonstrate its feasibility for civic tech.

Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations

AbstractResearchers and research organizations are under increasing pressure to demonstrate that their work contributes to positive change and helps solve pressing societal challenges. There is a simultaneous trend towards more engaged transdisciplinary research that is complexity-aware and appreciates that change happens through systems transformation, not only through technological innovation. Appropriate evaluation approaches are needed to evidence research impact and generate learning for continual improvement. This is challenging in any research field, but especially for research that crosses disciplinary boundaries and intervenes in complex systems. Moreover, evaluation challenges at the project scale are compounded at the programme scale. The Forest, Trees and Agroforestry (FTA) research programme serves as an example of this evolution in research approach and the resulting evaluation challenges. FTA research is responding to the demand for greater impact with more engaged research following multiple pathways. However, research impact assessment in the CGIAR (Consultative Group on International Agricultural Research) was developed in a technology-centric context where counterfactual approaches of causal inference (experimental and quasi-experimental) predominate. Relying solely on such approaches is inappropriate for evaluating research contributions that target policy and institutional change and systems transformation. Instead, we propose a multifaceted, multi-scale, theory-based evaluation approach. This includes nested project- and programme-scale theories of change (ToCs); research quality assessment; theory-based outcome evaluations to empirically test ToCs and assess policy, institutional, and practice influence; experimental and quasi-experimental impact of FTA-informed ‘large n’ innovations; ex ante impact assessment to estimate potential impacts at scale; and logically and plausibly linking programme-level outcomes to secondary data on development and conservation status.

Chinese martial arts and the Olympics: Analysing the policy of the International Wushu Federation

The International Wushu Federation (IWUF) has been striving to get wushu (Chinese martial arts) into the Olympics for over three decades. As these efforts have not been successful to date, a study was set up to identify and evaluate the IWUF’s underlying ‘Olympic policy’ assumptions. A theory-based evaluation approach was used which was based on data collected through 20 semi-structured interviews including senior IWUF officials, coaches, national federation officers and scholars. Findings suggest that the IWUF’s Olympic policy is characterised by a number of implementation and theory failures. Regarding the former is the unequal continental breakdown of board members, overrepresentation of Chinese nationals in specific positions and the use of mostly Chinese social media. And the latter, among other things, relates to simultaneously trying to enrich the cultural diversity of the Olympic programme by introducing a traditional Chinese sport, as well as detraditionalising the sport in an attempt to comply more with Olympic standards and facilitating comparison between performances. Results also showed that different perspectives exist among respondents with regard to China’s dominance in the Olympic policy of the IWUF. It is suggested to use specific policy analysis theories to further investigate this aspect.

Evaluating gender equality effects in research and innovation systems

Despite the fact that the topic of “women in research and innovation” has been on the agenda for decades and numerous measures have been implemented at both national and supranational levels to improve gender equality in research and innovation systems, it is still unclear which measures and under which conditions these measures are most effective. Even less research has been carried out on the effects of better representation of women in terms of research and innovation. This paper is based on the application of an innovative evaluation framework, which encompasses complexity and theory of change approaches and aims at exploring the link between interventions and their subsequent effects to two case studies. We discuss two major German flagship programmes aiming at increasing the participation of female researchers in the science system, the “Women Professorship Programme” and the “Pact for Research and Innovation”. Through the two programmes, we tested and validated the evaluation framework and its indicators. As part of the validation process, a theory of change has been developed for each of the programmes. The theory-based evaluation approach helped not only to identify gender equality impacts but also broader effects on research and innovation that might have otherwise remained undetected. We studied the effects of the two programmes on the number of women in leadership positions and analysed whether an increase in the proportion of women leaders influences publication patterns. Although linear linkages are challenging to establish due to the complexity of the process, the findings suggest that the flagship programmes have contributed not only to higher shares of women researchers but also to improved female publication and citation rates. There are clear benefits for Germany in terms of scientific results from the increased proportion of female researchers in research and innovation.

What works, for whom and under what circumstances? Using realist methodology to evaluate complex interventions in nursing: A scoping review

BackgroundThe Medical Research Council's (MRC) framework recommends basing every evaluation of a complex intervention on a theory that explains WHY an intervention works instead of focusing exclusively on the identification IF an intervention works. Theory-based evaluation approaches such as realist evaluations are promising in this respect. The aim of this scoping review is to examine current scientific literature with regard to the use of realist methodology in studies that evaluate complex nursing interventions. We also seek to answer the question whether realist evaluation studies of complex nursing interventions are embedded in the Medical Research Council's framework. MethodsA systematic database search in MEDLINE and CINAHL supplemented with snowballing techniques and a hand search in relevant nursing journals were performed between January and February 2018 and updated in July 2019. Both authors independently screened the title / abstract of studies that were initially identified and appraised the full texts if inclusion criteria were fulfilled. Inclusion criteria were the explicit use of realist methodology and a focus on the development or evaluation of complex nursing interventions. Results were summarized narratively. ResultsA total of 28 articles from 22 studies, published between 2010 and 2019, were included. The majority of the studies included originate in the UK. Realist methodology was used by a variety of nursing disciplines, including midwifery, community / public health, critical care, palliative care, gerontological nursing, psychiatric nursing / mental health, and infection control. Study designs included case studies and realist reviews; two studies were conducted alongside a randomized controlled trial. All studies included used more than one method to develop their results; the majority of methods were qualitative by nature. Thirteen of the studies included aimed to develop a theory about what works, for whom and under what circumstances regarding the investigated complex nursing intervention. One study was clearly embedded in the Medical Research Council's framework. ConclusionRealist evaluation broadens the methodologies that have traditionally been used in nursing. Conceptual clarity on its principal tenets is still needed, as well as the methods used to assess them. Nursing scientists should be more critical when using novel methodology and contribute to academic debates about limitations. Embedding realist evaluations into the Medical Research Council's framework is in its infancy in nursing and needs more contributions in order to develop a common academic position within the profession.Tweetable abstractRealist methodology in nursing – review of the use, methods, limitations and embedding into the Medical Research Council's framework

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.

Using Actor-Based Theories of Change to Conduct Robust Evaluation in Complex Settings

Abstract: The use of theories of change (ToCs) is a hallmark of sound evaluation practice. As interventions have become more complex, the development of ToCs that adequately unpack this complexity has become more challenging. Equally important is the development of evaluable ToCs, necessary for conducting robust theory-based evaluation approaches such as contribution analysis (CA). This article explores one approach to tackling these challenges through the use of nested actor-based ToCs using the case of an impact evaluation of a complex police-reform program in the Democratic Republic of Congo, describing how evaluable nested actor-based ToCs were built to structure the evaluation.

Featured risk evaluation of nautical navigational environment using a risk cloud model

In this paper, a risk cloud model (RCM) based on cloud model theory is proposed for evaluating risks in the nautical navigational environment (NNEt). To validate the method, the proposed strategy is applied to a risk evaluation of NNEt of the waters around the Hangzhou Bay Sea-Crossing Bridge and the results are compared with those of a conventional fuzzy theory-based evaluation approach. The results show that risk evaluations based on the proposed RCM are robust, can accommodate uncertainty, can leverage multi-channel information, and provide a full-featured analysis of NNEt. In addition, plots of the RCM droplets provide an intuitive depiction of the risk level, which is extremely beneficial in analyses of the results. Thus, the proposed RCM and implementation method is well-suited for use in conducting risk evaluation of NNEt.