Participatory Evaluation Approach Research Articles

Partnering with People with Lived/Living Experience of Mental Health and substance Use in Participatory Evaluation of an integrated virtual navigation support service for people experiencing substance use challenges.

Introduction/Background: This presentation shares how the Mid-West Toronto-Ontario Health Team (MWT-OHT) in Toronto, Canada involved patients and partners in the design, implementation, and evaluation of the “In Your Corner Program” to support people with substance use challenges. The experience of embedding the voices of people with living/lived experience in co-designing and evaluating the program will be shared.
 Who is it for? The “In Your Corner” program represents the actualization of key recommendations from people with lived/living experience with substance use challenges in MWT-OHT that identified a need for service navigation supports that helps to close transition gaps. The ‘In Your Corner’ program necessitated the MWT-OHT to take a collaborative, team, and consensus-based approach to coordinating services across the region, re-designing care pathways that better meet the needs of a dynamic and diverse population within the constraints of existing resources and structures.
 Who did you involve & engage with? The MWT-OHT brings together partners across health and social sectors in the mid-west downtown region of Toronto Canada to work together for better, more integrated care. In 2019, MWT-OHT completed co-design work with people with living/lived experiences of substance use and mental health focused on improving the care experience and outcomes for people with challenges related to substance use. The In Your Corner program was implemented in 2021 through a 5-month implementation funding and extended for an additional 4 months through in-kind support from implementation partners. During this time, a participatory evaluation approach was employed with partners as well as clients and providers of the program to inform next steps, scale, and sustainability.
 What results did you get? What impact did you have? Initial implementation findings of the program demonstrated that the model of care capitalizes on existing programs and was able to create a more robust scaffolding for integrated care for the clients served by the program. The opportunity to embed Peer support workers into the model was highlighted along with the need for follow-up supports over a longer period. The program was a first step toward working together with people with living and lived experiences from design to evaluation from a partnered approach that is hoped to be embedded in how the MWT-OHT approaches all initiatives.
 What is the learning for the international audience? The learnings from this program demonstrates the processes and supports needed to meaningfully embed the voices of patients and providers from design phase through implementation, evaluation, and scale at various points in the journey from a systems perspective. Considerations for scale are important, as well as minimizing and preventing tokenism and supporting true partnering.
 What are the next steps? Next steps identified from the co-design evaluation sessions with clients and providers of the In Your Corner Program have informed how to leverage, change, and scale the program to reach more people in the region with substance use challenges. More importantly, this process has become a blueprint for the MWT-OHT in how to embed the voices of patients, families, and caregivers in the designing and evaluating initiatives.

Improving Student Learning by Building Infrastructure, Capacities and Competencies: A Multinational e-Learning Partnership and Initiative

The COVID-19 pandemic disrupted education systems worldwide, requiring many higher education institutions (HEIs) to make immediate shifts towards online course and program delivery. While leaders simultaneously sought innovative solutions, many lacked the infrastructure to make such time-sensitive and resource-intensive changes. With substantive foundation funding, one private university in Africa partnered with one public university in the United States of America (USA) and co-designed and coimplemented an e-Learning initiative to deliver high-quality, inclusive e-Learning. Using a participatory evaluation approach and an evaluation framework that accounts for structural and institutional inequities in education, researchers representing both universities also co-examined this e-Learning initiative’s impacts, including its successes and challenges, using survey instruments, interviews, and focus groups. In this research paper, the authors provide some background to contextualize the research project, present details on the methodology used to conduct the research project and present the results via four key themes: participants’ experiences, successes, challenges, and implications and recommendations. The paper concludes with a discussion on the key findings of the research project and how they impact theory and practice in e-Learning.

Influence of participatory monitoring and evaluation on decision-making in maternal and newborn health programs in Mombasa County, Kenya

The study assessed the utilization of participatory monitoring and evaluation approaches (PM&E) and their influence on decision-making in maternal and newborn health programs (MNH) in Mombasa County, Kenya. We conducted a cross-sectional study with a sample of 390 participants and used a structured questionnaire, modified Quality of Decision-Making Orientation Scheme, and an interview guide to collect data. We analyzed quantitative responses using descriptive statistics and binary logistic regression (at significance level of 0.05) and qualitative responses using content analysis. The study established that quality decision-making was more likely to occur in MNH programs in Mombasa County that used PM&E approaches at the initiation (OR 1.728, p < 0.05), design and planning (OR 2.977, p < 0.05), and implementation (OR 5.665, p < 0.05) phases of MNH programs than in those that did not use the PM&E approaches. This study shapes a persuasive case for improving the provision of maternal and newborn health services.

Principal Leadership in the Al-Qur'an Tahfizh Program (Multicase Study at Al-Muhafizhah Middle School Blitar and Sunan Gunung Jati Tulungagung Middle School)

This study aims to find answers to three fundamental questions; 1) How is the leadership of the principal in defining the vision and mission of the tahfizh al-Qur’an program?; 2) How is the leadership of the school principal in mobilizing school members to carry out the tahfizh Al-Qur’an program?; 3) How is the leadership of the school principal in controlling the tahfizh Al-Qur’an program at SMP Al Muhafizhah Blitar and SMP Sunan Gunung Jati Tulungagung? This research used a qualitative approach with a case study type at SMP Al-Muhafizhah Blitar and SMP Sunan Gunungjati Tulungagung. Sources of informant data, events, locations and documents both primary and secondary. Data collection techniques with participant observation, in-depth interviews and documentation. Data analysis using Huberman data analysis techniques. While checking the validity of the data is done through peer discussions, triangulation of data sources and triangulation of methods. Research result; 1) The leadership of the school principal in defining the vision and mission of the Al-Qur’an tahfizh program is an affirmation of historical reasons, community aspirations, character, characteristics of excellence and competitiveness of schools as Islamic educational institutions; 2) The leadership of the school principal in mobilizing school members to implement the tahfizh program is carried out using an awareness, participatory and delegative approach by prioritizing a collaborative work system; 3) The leadership of the school principal in controlling the Al-Qur’an tahfizh program is carried out using a model of supervision, evaluation and problem solving using a participatory evaluation approach, as well as coaching programs using a delegative approach. The results of the research strengthen Mulyasa’s concept that leadership is the ability to direct, mobilize and foster members of the school organization to achieve common goals. Strengthening the concept of Selznik’s leadership function, namely defining vision and mission, maintaining organizational integrity and control. Strengthen Blanchard’s leadership concept of participatory and delegative approaches in the relationship between leaders and subordinates.

Empowerment Evaluation of Programs Involving Youth

Background: Participatory and collaborative evaluation approaches, including Empowerment Evaluation (EE), are useful for evaluating programs involving youth. Empowerment evaluation involves stakeholders in the evaluation process through a set of structured steps. It is primarily concerned with empowering, illuminating, and building program beneficiaries’ self-determination. Given the emphasis that EE places on inclusivity of stakeholders, it appears to be a good fit for evaluating programs that involve youth.&#x0D; Purpose: To explore the extent to which evaluators use EE to evaluate programs involving youth as well as what factor(s) facilitate and hinder their use of EE in these programs.&#x0D; Setting: The study involved evaluators associated with the Collaborative, Participatory and Empowerment Evaluation and Youth-Focused Evaluation Targeted Interest Groups (TIGs) of the American Evaluation Association (AEA) who are involved in evaluating programs targeted at youth.&#x0D; Intervention: Not applicable.&#x0D; Research Design: We used a two-phase sequential mixed-methods research design. In Phase 1, we surveyed evaluators. In Phase 2, we interviewed a sample of evaluators from Phase 1. &#x0D; Findings: In Phase 1, 41 (53.9%) respondents indicated not using EE to evaluate programs involving youth, 30 (39.5%) had used EE and 5 (6.6%) were unsure. Of those who used EE, they used it to teach youth program stakeholders about evaluation (n=8, 24.2%), produce more authentic results by engaging youth as experts of their lived experience (n=7, 21.2%) or produce more useful results for stakeholders to use (n=6, 18.2%), as well as other less popular reasons. In Phase 2, 12 interviewees raised five factors that facilitate or hinder the use of EE to evaluate programs involving youth including, evaluator perceptions, type of evaluation experience, evaluator knowledge and professional training, guidelines from organizations and funders, and stakeholders and time. Factors that some interviewees viewed as facilitators others viewed as hinderances.&#x0D; &#x0D; Keywords: empowerment evaluation, program evaluation, youth-focused evaluation&#x0D; &#x0D;

Quality assurance indicators for environmental citizen science. Development of indicators for volunteer-based biodiversity monitoring.

Volunteer-based biodiversity monitoring schemes are currently developed and tested for feasibility in and for agricultural landscapes in Germany. For the assessment of the effectiveness and efficiency of volunteer-based biodiversity monitoring schemes, indicators are required but so far, such indicators have neither been developed nor tested. Here, sets of indicators are developed and presented based on evidence from scientific literature and from the volunteers’ perspectives. As a starting point of the development of indicators, challenges for volunteers need to be identified that may hinder them from taking part in the schemes and from achieving project goals. On the basis of formulated actions to overcome these challenges, three sets of indicators are derived, covering the areas of i) capacity building for volunteer-based engagement, (ii) appreciation and valuing of volunteer commitment, and (iii) education and learning in volunteer-based approaches. Indicators are developed to potentially serve internal and external communication and act as project quality assurance measures. At the same time, the presented indicators may potentially also be applied by decision-makers in policy as well as by funding agencies. In a next step, indicators are co-developed using participatory evaluation approaches to combine conventionally developed indicators with indicators developed with members of the community of practitioners. Implementing indicators in practice as well as regular reflections and revisions will ensure an adaptive quality assurance system for volunteer-based biodiversity monitoring and beyond.

Evaluation of the curriculum of the teaching in the multigrade classrooms course: Participatory evaluation approach

Evaluation of the curriculum of the teaching in the multigrade classrooms course: Participatory evaluation approach

Fundamental Teaching Activities in Family Medicine Framework: Analysis of Awareness and Utilization.

In 2015, the College of Family Physicians of Canada, in performing their commitment to supporting its members in their educational roles, created the Family Medicine Framework (FTA). It was designed to assist family medicine educators with an understanding of the core activities of educators: precepting, coaching, and teaching within or beyond clinical settings. Given that an examination of member awareness of FTA has not been previously undertaken, our primary objective was to conduct an evaluation on its utility and application. In partnership with College of Family Physicians of Canada Faculty Development Education Committee members, we used a practical participatory evaluation approach to conduct a two-phase mixed-methods evaluation of the FTA. We distributed an electronic survey in French and English languages to Canadian faculty development, program, and site directors in family medicine. We then conducted follow-up interviews with self-selected participants. Of the target populations, 12/15 (80%) faculty development directors (FDDs), 12/18 (66.7%) program directors, and 34/174 (19.5%) site directors completed the electronic survey. Subsequently, 6 FDDs, 3 program directors, and 3 site directors completed an interview (n = 12). Findings indicate that awareness of the FTA was highest among FDDs. Facilitators who encourage teachers to use the FTA and barriers for low uptake were also identified. This evaluation illuminated that varied levels of awareness of the FTA may contribute to the low uptake among education leaders. We also suggest future research to address possible barriers that hinder effective applications of the FTA in faculty development initiatives.

Breaking barriers and building bridges: Increasing community engagement in program evaluation

BackgroundBetween September 2015 and 2018, The Michigan Department of Health and Human Services (MDHHS) focused its Project PrIDE efforts on increasing PrEP awareness through a media campaign, and community engagement in. Wayne County, MI. A strained relationship between MDHHS and community has created barriers to sustained and effective community engagement. The local evaluation (LE) was developed to determine the barriers and facilitators to engagement in program activities. MethodsTo understand the process and the factors that impacted the project’s goal, a mixed methods participatory evaluation approach was utilized. Data collection included quantitative data from meeting surveys, qualitative data from focus groups, and a photovoice project. Analysis and synthesis included descriptive statistics and deductive thematic coding. ResultsCAB engagement increased collaboration, enhanced feasibility and utility, appropriateness, sustainability of activities, and increased transparency between MDHHS and CAB. Thematic coding identified Barriers and facilitators to engagement with PrEP activites included restrictive funding, comprehensive services delivery and CAB feedback underutilized and resulting in ineffective campaign messages. ConclusionsAlthough a strained relationship was evident in the beginning, encouraging interagency collaboration in the development, implementation, and utilization of program and evaluation activities, resulted in the improvement of relationships between the community and MDHHS.

Practice Notes from a Participatory Impact Evaluation of a Leadership Development Program for People Living with HIV

Abstract: Interest in participatory evaluation and other collaborative inquiry approaches has risen substantially over the past few decades. However, there appears to be a lack of practical information about using and applying participatory evaluation approaches on the ground. This Practice Note uses a participatory impact evaluation of a leadership development program for people living with HIV and findings from a meta-evaluation of this work to describe the participatory evaluation approach adopted, challenges and lessons learned related to conducting a participatory evaluation, and some key factors and implications to consider for maximizing the success of future participatory evaluations.

A Participatory Evaluation of Transforming First Year LLB Into Block Mode

This paper adopts a participatory evaluation approach to identify successes and challenges experienced by a group of academics in redesigning and delivering the first year of the Law degree (LLB) in block mode in an Australian university. This transformation into an intensive four-week block necessitated an overhaul of curricula, assessment regimes, teaching practices and delivery processes. The evaluation of the academics staff experience focused on five themes: (1) course design and development, (2) satisfying learning outcomes and accreditation standards, (3) implementing student engagement strategies, (4) embedding block mode principles, and (5) establishing effective assessment regimes. For each theme, participants identified the successes, challenges, and lessons learned. The study highlighted the value of early design-specialist involvement and collaboration in course design including the importance of being aware of the legal accreditation body requirements. The criticality of focused active learning and the need to balance lecture-style delivery with knowledge-consolidating practical legal analysis exercises was emphasised. Also stressed was wellbeing, including sensitivity to individual student circumstances and strategies that successfully manage staff and student workloads alongside academic integrity. Unintended benefits such as professional learning through the collaboration, refining personal thinking related to pedagogy in the block were consequences of this evaluative approach.

Findings on the role of cooperatives in local economic development in Buffalo City Metropolitan Municipality: A participatory monitoring and evaluation approach

Background: This article outlines the role of cooperatives in local economic development (LED) within participatory monitoring and evaluation (PM&amp;E) approach in the Buffalo City Metropolitan Municipality (BCMM), South Africa. The Department of Trade and Industry acknowledged the inadequacy of monitoring and evaluation (M&amp;E) of cooperatives. This lack of M&amp;E of cooperatives’ role in LED has also affected cooperatives in BCMM. Aim: The study sought to establish the contributory role of cooperatives in promoting LED outcomes and to determine the extent to which the PM&amp;E was capable of capacitating cooperatives. Setting: The study is based in East London and the target population is primary cooperatives. Primary cooperatives are defined by Buffalo City Integrated Cooperative Development Strategy (ICDS) as cooperatives that have at least five members, with an aim of providing employment to members. Methods: The research design for the study is exploratory and inductive. The study adopted qualitative research approach that combines participatory action research (PAR) and PM&amp;E. Two sets of data collection such as focus group interviews (FGIs) and e-mailed questionnaire (EQ) were used. Results: It was established that cooperatives have not yet achieved their contributory role of promoting LED, specifically in terms of job creation. Conclusion: Based on the findings it was recommended that capacity building and training with regard to M&amp;E should not focus on conventional M&amp;E, but rather on PM&amp;E, where cooperatives become involved in the M&amp;E of their business.

Localizing the Water-Energy Nexus: The Relationship between Solar Thermal Power Plants and Future Developments in Local Water Demand

Water availability plays an important role in the expansion planning of utility-scale solar power plants, especially in the arid regions of the Middle East and North Africa. Although these power plants usually account for only a small fraction of local water demand, competition for water resources between communities, farmers, companies, and power suppliers is already emerging and is likely to intensify in future. Despite this, to date there has been a lack of comprehensive studies analyzing interdependencies and potential conflicts between energy and water at local level. This study addresses this research gap and examines the linkages between water resources and energy technologies at local level based on a case study conducted in Ouarzazate, Morocco, where one of the largest solar power complexes in the world was recently completed. To better understand the challenges faced by the region in light of increased water demand and diminishing water supply, a mixed-method research design was applied to integrate the knowledge of local stakeholders through a series of workshops. In a first step, regional socio-economic water demand scenarios were developed and, in a second step, water saving measures to avoid critical development pathways were systematically evaluated using a participatory multi-criteria evaluation approach. The results are a set of water demand scenarios for the region and a preferential ranking of water saving measures that could be drawn upon to support decision-making relating to energy and water development in the region.

PNS244 Public Preferences on the Disinvestment of Healthcare Interventions and Services: An Application of the Participatory VALUE Evaluation Approach in a Health Economics Context

PNS244 Public Preferences on the Disinvestment of Healthcare Interventions and Services: An Application of the Participatory VALUE Evaluation Approach in a Health Economics Context

A scoping review about social and emotional wellbeing programs and services targeting Aboriginal and Torres Strait Islander young people in Australia: understanding the principles guiding promising practice

BackgroundMultiple culturally-oriented programs, services, and frameworks have emerged in recent decades to support the social and emotional wellbeing (SEWB) of Aboriginal and Torres Strait Islander (Aboriginal) people in Australia. Although there are some common elements, principles, and methods, few attempts have been made to integrate them into a set of guidelines for policy and practice settings. This review aims to identify key practices adopted by programs and services that align with the principles of the National Strategic Framework for Aboriginal and Torres Strait Islander Peoples’ Mental Health and Social and Emotional Wellbeing 2017–2023.MethodsA comprehensive review of electronic databases and organisational websites was conducted to retrieve studies of relevance. Twenty-seven publications were included in the review. Next, we identified promising practices through a collaborative review process. We then used the principles articulated in the above-mentioned framework as the basis to complete a framework analysis. This enabled us to explore the alignment between current scholarship about SEWB programs and services with respect to the principles of the framework.ResultsWe found there was a strong alignment, with selected principles being effectively incorporated into most SEWB program and service delivery contexts. However, only one study incorporated all nine principles, using them as conceptual framework. Additionally, ‘capacity building’, ‘individual skill development’, and ‘development of maladaptive coping mechanisms’ were identified as common factors in SEWB program planning and delivery for Aboriginal people.ConclusionWe argue the selective application of nationally agreed principles in SEWB programs and services, alongside a paucity of scholarship relating to promising practices in young people-oriented SEWB programs and services, are two areas that need the urgent attention of commissioners and service providers tasked with funding, planning, and implementing SEWB programs and services for Aboriginal people. Embedding robust participatory action research and evaluation approaches into the design of such services and programs will help to build the necessary evidence-base to achieve improved SEWB health outcomes among Aboriginal people, particularly young people with severe and complex mental health needs.

The Use of Ripple Effect Mapping to Understand Successes of the SC Pregnancy Assistance Fund: A Participatory Evaluation Approach

ObjectivesThe South Carolina Pregnancy Assistance Fund (SCPAF) funded four counties to increase the amount, quality, and awareness of services for young parents; increase educational attainment among expectant and parenting youth; reduce the number of repeat teen pregnancies among youth; and improve parenting skills. The purpose of this paper is twofold: (1) to describe our application of the Ripple Effect Mapping (REM) technique as an innovative evaluation strategy to gather perspectives from SCPAF stakeholders and (2) to share key findings generated by participants in REM sessions on the perceived success of local SCPAF community collaboratives.MethodsREM, an innovative evaluation strategy, was used to gather perspectives from SCPAF stakeholders. Five REM sessions were conducted with 52 participants. REM sessions included partner interviews and collective development of visual maps to illustrate stakeholder perspectives of program successes. Visual maps, as well as transcripts of discussions, were analyzed using an inductive approach.ResultsStakeholders reported that the connections to resources, supports, and services provided through SCPAF had the potential to alter the life trajectories of expectant and parenting teens (EPT). Stakeholders also described that SCPAF fostered growth in collaboration among partners and reduced duplication of services in funded communitiesConclusions for PracticeThis paper describes how an innovative evaluation strategy was used to provide a space for stakeholders to dialogue, synthesize their experiences, and construct a collective narrative of key program successes. This paper also illustrates how such approaches can be applied to complex community initiatives.

Closing the Gap: Participatory Formative Evaluation to Reduce Cancer Screening Disparities among Patients with Limited English Proficiency.

Patients with limited English proficiency (LEP) have disproportionately lower rates of cancer screening than English-proficient patients. Given the multifactorial nature of screening disparities, strategies to improve screening rates must address barriers within and outside of the clinic setting. The objectives of this study were to understand local barriers from multiple stakeholder perspectives, to identify potential multilevel intervention approaches, and to mobilize community-engaged intervention decision making and planning. This participatory formative evaluation approach employed needs assessment and user engagement in order to enhance intervention usefulness and relevance. The study took place in several stages and involved clinic and community partners in a small metropolitan area of the Midwest USA. Interviews were conducted with LEP patients (n = 9) who had not completed three recommended screenings (breast, cervical, and colorectal), primary care providers (n = 5), medical interpreter (n = 5), and community members (n = 3). These highlighted multilevel barriers including limited patient understanding of preventive health, time and cost constraints, and variable roles of language interpreters. The literature was also reviewed to identify interventions used with similar populations. Findings from this review suggest that interventions are largely focused on single population groups or address single screening barriers. Finally, a community-academic summit (n = 48 participants) was held to review results and develop recommendations for community and clinic interventions. Findings from this study indicate that it is possible to engage a diverse group of stakeholders in strategies that are responsive to health care providers and patients, including LEP patients from heterogeneous backgrounds.

A Community-Academic Partnership for School-Based Nonviolence Education: The Healthy Power Program.

Youth violence is a significant problem affecting community health. Community-academic partnerships can advance youth nonviolence education by synergizing the strengths of collaborators while working toward a common goal. We describe a collaboration between an urban public middle school, community nonprofit, and university-based graduate school of nursing in implementing and evaluating the Healthy Power program, a school-based youth nonviolence program for middle-school boys. A participatory program evaluation approach was used to plan and implement evaluation of the Healthy Power program with a cohort of 8 students. Collaborative planning allowed for the selection of measures that reflected program objectives and were of value to community partners while also scientifically sound. A mixed-methods approach included a focus group and a pretest-posttest with quantitative items and open-ended questions. While the quantitative pre-posttest did not show any significant change, the open-ended questions and focus group suggested that students had advanced their understanding and application of conflict resolution skills. The findings support the usefulness of community-academic partnerships for peace/conflict resolution education and program evaluation. Such programs may benefit from mixed methods of evaluation.

Evaluation of the Environmental Engineering Study Programme at University

Abstract The European Union has placed competence-based teaching and competence-based education as one of its highly relevant goals. Due to mass higher education, the assessment of effectiveness and relevance evaluation of environmental engineering study programmes should become an important issue. Presently the focus of the evaluation on multi-disciplinary study programmes varies from the evaluation of attitudes, impacts or effectiveness of utilisation-focused evaluation, summative evaluation and participatory evaluation approaches. The objective of this study was to propose an effective framework to evaluate the Environmental Engineering Master study programmes. During the research, the evaluation of existing study programmes on environmental engineering in Europe was conducted, information about the study courses, teaching methods, assessment methods and competences was used for the analysis. The results obtained showed that lectures, site visits, group coursework, practical laboratories and role-plays allows to reach the necessary knowledge, skills and competences and to provide an effective and relevant education to the Environmental Engineering Master programme students. The proposed evaluation framework was tested and approbated on new Riga Technical University Master study programmes on Environmental Engineering and Bioeconomy.

Between power and perception: Understanding youth perspectives in participatory monitoring and evaluation (PM&E) in Ghana

Participatory monitoring and evaluation (PM&E) approaches have been touted as an important concept for local participation in social interventions programmes. Utilizing a process analysis approaches, the PM&E data gathered through key informant interviews, formal surveys, and policy frameworks were analyzed. The study revealed three dominant power struggles between youth and programme implementers in three districts at the Greater Accra region, Ghana. The first and pervasive form of power dynamics involves the youth and programme implementers. The second involves a variety of arrangements with the government on one hand and implementers and youth on the other side. The third is who qualifies to be a beneficiary and for that matter participate in the PM&E. In these three forms of power struggles the paper reveals superior implementers control of who participate in the programme PM&E and at what stage in the process. The power imbalance between programme implementer and target beneficiaries impeded the level of beneficiaries’ participation in the PM&E.