Participatory Process Evaluation Research Articles

Can communities be mobilised to build capacity to respond to the COVID-19 pandemic? A qualitative process evaluation

ObjectivesGovernment guidance to manage COVID-19 was challenged by low levels of health and digital literacy and lack of information in different languages. ‘Covid Confidence’ sessions (CC-sessions) were evaluated to assess...

Participatory Process Evaluation of Flexible and Integrative Treatment (FIT) Models in German Psychiatry – A Mixed Method Study

Flexible and integrative treatment (FIT) services in Germany make it possible to shift mental health inpatient care to day- and outpatient care. This paper presents the results of the mixed methods, participatory process evaluation (= PE) of the PsychCare clinical trial, which compares the outcomes of nine FIT departments to those of eight standard mental health care services. This PE integrates diverse data using a program theory and data transformation approach. It shows various implementation types of FIT services and how their processes and structures are distinct from those of control conditions, also experienced by service users. It contributes to mixed methods research by showing that the PE methodological frame is a valuable tool to effectively involve service users and other stakeholders.

Understanding the what, how, and why in advocacy: Assessing the applicability of participatory process evaluation methodology in an advocacy context

While evaluations are critical for non-governmental organizations to strengthen their advocacy strategies, evaluators and advocates encounter many difficulties evaluating such efforts. This article discusses the contribution of the participatory process evaluation methodology to advocacy evaluation, using a Dutch global health advocacy program as a case study. As participatory process evaluation is a novel methodology in the field of advocacy, the article’s primary focus concerns the application and utility of the methodology. Findings suggest that participatory process evaluation in an advocacy context can provide insights into the implementation of advocacy tools and activities, encouraging reflection and leading to ideas and practical tools to strengthen advocacy efforts. While participatory process evaluation can help overcome some of the often-experienced barriers in advocacy evaluation, further research is needed to consolidate advocacy evaluation theory and practice.

Evaluating the Impact of Public Participation Processes on Participants in Smart City Development: A Scoping Review

This paper provides an overview of the current state of research on the evaluation methods of participatory processes in smart cities. Specifically, it aims to identify and analyze existing evaluation methods and frameworks for public participation (PP) in smart city development. The study focuses on the evaluation of participatory processes to find key indicators and enable an assessment of PP from multiple perspectives. A scoping literature review was conducted to analyze the past ten years of scientific literature on the topic. Relevant literature was retrieved from Scopus, Web of Science, and Google Scholar, and articles were selected based on a set of criteria to ensure quality and relevance. We found that digital participation, also known as “e-participation”, is the most used method either exclusively or mixed with in-person methods. The level of participation achieved was mostly limited to the first two degrees of PP (i.e., information and consultation), and only a few papers addressed the highest degrees (i.e., agenda-setting and co-management). The impact on participants was mostly related to knowledge and skills, awareness raising, and satisfaction with the process and method. This paper highlights the potential to upskill citizens and enhance their understanding of sustainable urban development, fostering their commitment to achieving the United Nations’ sustainability goals for climate change mitigation in the urban context.

Using natural capital and ecosystem services to facilitate participatory environmental decision making: Results from a systematic map

Abstract The rights all people have for involvement in environmental decision making has long been established yet collaborative resource management has had mixed success. Natural capital; the renewable and non‐renewable natural assets that benefit societies, and the flow of ecosystem services these assets provide, are increasingly promoted as approaches that ensure consideration of the environment in decision making. Natural capital and ecosystem services concepts can facilitate participation in decision making by explicitly describing the role of the environment in sustaining society. Increased promotion of these approaches requires consideration on how best to involve stakeholders, those involved and affected by a decision, in the process. We conducted a systematic search to identify where stakeholders have participated in natural capital, ecosystem services and nature’s contributions to people decision making, creating a systematic map of 56 case studies. While many papers discussing stakeholders and these concepts were found, few actively engaged stakeholders in a decision‐making process that used the concepts and therefore were included in the map. Where stakeholders were involved, engagement methods included focus group discussion, stakeholder negotiation and scenario development, as well as ecosystem service ranking and mapping. Ranking for prioritisation of ecosystem services was common, with a bias towards using services with a direct tangible economic benefit; food production and tourism, are both prominent examples. A limited number of case studies performed robust participatory methods evaluations, offering little indication of how best to use natural capital or ecosystem services in participatory approaches. Therefore, the work highlights need for greater evaluation of participatory processes involving natural capital to ensure stakeholder engagement is efficient, productive and useful to all involved. Read the free Plain Language Summary for this article on the Journal blog.

Multiperspective and Multimethod Evaluation of Flexible and Integrative Psychiatric Care Models in Germany: Study Protocol of a Prospective, Controlled Multicenter Observational Study (PsychCare).

Background: New cross-sectoral mental health care models have been initiated in Germany to overcome the fragmentation of the German health care system. Starting in 2013, flexible and integrative psychiatric care model projects according to §64b SGB V German Social Law (FIT64b) have been implemented. The study “PsychCare” combines quantitative and qualitative primary data with routine health insurance data for the evaluation of these models. Effects, costs and cost-effectiveness from the perspectives of patients, relatives and care providers are compared with standard care. Additionally, quality indicators for a modern, flexible and integrated care are developed. This article describes the rationale, design and methods of the project.Methods: “PsychCare” is built on a multiperspective and multimethod design. A controlled prospective multicenter cohort study is conducted with three data collection points (baseline assessment, follow-up after 9 and 15 months). A total of 18 hospitals (10 FIT64b model and 8 matched control hospitals) have consecutively recruited in- and outpatients with pre-specified common and/or severe psychiatric disorders. Primary endpoints are differences in change of health-related quality of life and treatment satisfaction. Sociodemographic and service receipt data of the primary data collection are linked with routine health insurance data. A cost-effectiveness analysis, a mixed method, participatory process evaluation by means of qualitative surveys and the development of quality indicators are further elements of “PsychCare.”Discussion and Practical Implications: The results based on data from different methodological approaches will provide essential conclusions for the improvement of hospital based mental health care in Germany. This should result in the identification of key FIT64b elements that can be efficiently implemented into standard care in Germany and re-structure the care strongly aligned to patient needs.Clinical Trial Registration: German Clinical Trial Register, identifier DRKS 00022535.

Assessment of adaptation, policy, and capacity building outcomes from 14 processes

In the US alone, there are over 200 tools that support climate adaptation planning, along with a large number of case studies documenting their use. Case studies frequently document positive results. Systematic assessment of processes can provide important benefits, including justification for action and expenditures, promotion of learning and adaptive management, accountability, and ensuring “fit” with other goals. There are very few such assessments in the context of climate adaptation planning, despite of the emphasis on the development and use of planning tools by federal and state agencies, university researchers, and non-profit organizations.We undertook an effort to assess the outcomes resulting from fourteen applications of the Vulnerability, Consequences, and Adaptation Planning Scenarios (VCAPS) process, which we helped develop and implement. VCAPS is designed to facilitate information exchange, co-production of knowledge, and stakeholder collaboration while helping communities appraise climate change-related risks and devise strategies to manage them. Using qualitative interviews we explored the perceived value and the measurable performance outcomes of VCAPS at both individual and community scales occurring 3–10 years after the processes were conducted, allowing participants to take a broader view of “success” and reflect on how different forms of success emerged over time. Although the assessment of each case is based on a small number of interviews, we learned that VCAPS informed plans and decisions of municipalities, informed actions and decisions of other public and private actors, generated broader support for subsequent actions, helped efforts to secure/seek funding for climate adaptation actions, developed material resources to support planning, and promoted learning among participants.This assessment also reinforces prior work showing that deliberative planning tools/processes are conducive to developing adaptive capacities; processes should be closely coordinated with regular governance activities to impact policy and action; adequate time for deliberation needs to be budgeted; participants need support to “think outside the box” and consider adaptation strategies that are both incremental and transformational as well as highlight potential undesirable consequences of adaptation; and processes, like VCAPS, produce actionable outcomes when participants agree on the immediacy of the issue. We conclude with observations about the need for evaluation of participatory processes and the challenges of defining success of tools to support municipal climate change adaptation planning.

Formulating the success of citizen participation in urban regeneration: Insights and perplexities from Lisbon

ABSTRACT Worldwide, participatory processes have expanded by covering a wide range of policy areas, including urban regeneration. Acknowledging the formulation of success as the first methodological step for the evaluation of participatory processes, the article discusses insights and perplexities emerging from the BipZip Programme in Lisbon. The proposal of an alternative agenda builds on the need to provide the formulation of success with information on the macro, meso and micro contexts, to be collected through the participation of local partners and political/administrative agents. Theoretical and empirical knowledge concur to the international debate on the evaluation of citizen participation in urban regeneration.

Process evaluation of a National Primary Eye Care Programme in Rwanda

BackgroundVisual impairment is a global public health problem, with an estimated 285 million affected globally, of which 43% are due to refractive error. A lack of specialist eye care in low and middle-income countries indicates a new model of care would support a task-shifting model and address this urgent need. We describe the features and results of the process evaluation of a national primary eye care (PEC) programme in Rwanda.MethodsWe used the Medical Research Council process evaluation framework to examine the implementation of the PEC programme, and to determine enablers and challenges to implementation. The process evaluation uses a mixed methods approach, drawing on results from several sources including a survey of 574 attendees at 50 PEC clinics, structured clinical observations of 30 PEC nurses, in-depth interviews with 19 key stakeholders, documentary review and a participatory process evaluation workshop with key stakeholders to review collated evidence and contextualize the results.ResultsStructured clinical assessment indicated that the PEC provided is consistent with the PEC curriculum, with over 90% of the clinical examination processes conducted correctly. In 4 years, programme monitoring data showed that nearly a million PEC eye examinations had been conducted in every health centre in Rwanda, with 2707 nurses trained. The development of the eye health system was an important enabler in the implementation of PEC, where political support allowed key developments such as inclusion of eye-drops on the essential medicines list, the inclusion of PEC on insurance benefits, the integration of PEC indicators on the health management information systems and integration of the PEC curriculum into the general nursing school curriculum. Challenges included high turnover of primary care nurses, lack of clarity and communication on the future funding of the programme, competing priorities for the health sector and sustained supervision to assure quality of care.ConclusionsA model of a national primary eye care programme is presented, with service delivery to all areas in Rwanda. Key learning from this evaluation is the importance of strengthening the eye health care system, together with a strong focus on training primary care nurses using a PEC curriculum.

How to Find out What’s Really Going On: Understanding Impact through Participatory Process Evaluation

This article considers the contribution participatory process evaluation can make to impact assessment, using a case study of a study carried out to evaluate how a Kenyan nutrition education program had brought about change in the nutritional status of children and in their and their parents’ understanding and practices. Using Bhola’s three dimensions of impact—“impact by design”, “impact by interaction”, and “impact by emergence”—focuses not just on what changes as an intended result of an intervention, but on how change happens and how positive changes can be sustained. The principal focus of the article is methodological and as such it describes in some detail the development of a sequence of participatory visualization and discussion methods and their application with a range of stakeholders, from program staff in the headquarters of the implementing agency, to local government officials, front-line program workers, and beneficiaries. It suggests that the use of a participatory approach can enable researchers and evaluators to gain a fuller picture of incidental and unintended outcomes arising from interventions, making participatory process evaluation a valuable complement to other impact assessment methodologies.

The Modes of Participation: A Revised Frame for Identifying and Analyzing Participatory Budgeting Practices

Participatory budgeting is a democratic process through which citizens make decisions about budgeting. It has received attention in public administration literature. Many scholars apply “participatory budgeting” too broadly. They conflate nominally participatory practices with those that emanated from the Global South and now expand throughout the United States. This article explores various applications of “participatory budgeting.” It also presents a heuristic framework, developed out of the work of Arnstein and Fung, which assists practitioners and researchers in identification and evaluation of participatory processes, including participatory budgeting. This framework is applied to four cases to illustrate determination of participatory budgeting practices.

Using Participatory Process Evaluation to Understand the Dynamics of Change in a Nutrition Education Programme

SummaryWith roots in approaches to popular education and participatory action research that place the learner and the ‘beneficiary’ of development at the centre of enquiry and action, the participatory visualisation methods associated with Participatory Rural Appraisal have been widely used as tools for learning and accountability. In this article, I reflect on lessons learnt from using these methods in a participatory process evaluation of an educational programme aimed at addressing chronic malnutrition in an East African country. Building on this experience, I explore the educative and empowering dimensions of participatory visualisation methods, and consider the contribution that these methods can make to effective evaluation.

Aboriginal Community-Centered Injury Surveillance: A Community-Based Participatory Process Evaluation

While injuries are a leading health concern for Aboriginal populations, injury rates and types vary substantially across bands. The uniqueness of Aboriginal communities highlights the importance of collecting community-level injury surveillance data to assist with identifying local injury patterns, setting priorities for action and evaluating programs. Secwepemc First Nations communities in British Columbia, Canada, implemented the Injury Surveillance Project using the Aboriginal Community-Centered Injury Surveillance System. This paper presents findings from a community-based participatory process evaluation of the Injury Surveillance Project. Qualitative data collection methods were informed by OCAP (Ownership, Control, Access, and Possession) principles and included focus groups, interviews and document review. Results focused on lessons learned through the planning, implementation and management of the Injury Surveillance Project identifying lessons related to: project leadership and staff, training, project funding, initial project outcomes, and community readiness. Key findings included the central importance of a community-based and paced approach guided by OCAP principles, the key role of leadership and project champions, and the strongly collaborative relationships between the project communities. Findings may assist with successful implementation of community-based health surveillance in other settings and with other health issues and illustrate another path to self-determination for Aboriginal communities. The evaluation methods represent an example of a collaborative community-driven approach guided by OCAP principles necessary for work with Aboriginal communities.

Book Review: A model health system approach to gender-based violence

A model health system approach to gender-based violence VIOLENCE AGAINST WOMEN: The Health sector Responds. Velzeboer M, Ellsberg M, Areas CC, Garcia-Moreno C Pan American Health Organization, Washington 2003. Occasional Publication No. 12. 131 pp. THIS EXCELLENT REPORT complements and extends World Health Organisation efforts to emphasise the importance of health sector reform to combat gender-based violence, outlined in the 2002 WHO World Report on Violence and Health. In this case the Pan American Health Organization (PAHO) describes the process, experience and outcomes of model reform undertaken by a collaboration of government and non-government organisations in Central America to improve the health sector as part of an overall response to gender-based violence. For Australian readers interested in Australian health care reform, the value of this report lies both in the proposed model and processes used to drive the reform and the excellent resources section and bibliography. While the report cites the UN definition of violence against women (acts of gender-based violence resulting in harm to women) as the basis for its action, in fact the book addresses mainly responses to intimate partner violence. Section 1, The health sector responds to genderbased violence, profiles the evidence, rationale and processes of model development. The first chapter reviews the health outcomes of violence against women. The second describes findings of what is described as the 'Critical Path' studies in 10 countries. These summarise in-depth interviews with women and service providers in a range of government and non-government sectors about pathways for women and the services they access in order to leave a violent relationship behind. The analysis (unsurprisingly) identified, among other things, that the success of interventions depended on the availability, quality, and coordination of services and most of all on the commitment of providers. Chapter three then goes on to describe important factors (coalition building, legal reform, research dissemination, screening, surveillance and training strategies, etc.) utilised in the participating countries within the PAHO integrated model. This includes suggested strategies (eg, symposia, campaigns) to bring community, legal, education and health sectors together at national and regional levels. The authors argue for the overall incorporation of 'cross-cutting values' of gender equity, participation, and partnerships. Section 2, Lessons learnt from Central America, presents highlights from a participatory process evaluation in four of the countries. Chapter four reviews the legislative reform and other governmental policy changes and their limitations in these countries, which will be of limited interest to Australian readers. Chapter five is most useful, highlighting lessons from the PAHO evaluation of this integrated approach within the health system. The barriers to change are very familiar to Australian readers familiar with health system responses: unhelpful provider attitudes shaped by prevailing cultural norms; victimised women's mistrust of the health system, women's shame and embarrassment about disclosure; providers worried that screening would add another burden to their overstretched capacity; and some providers acknowledging that they have been the victims and abusers themselves. …

Evaluating participation processes in community development

In this study evaluation methodology is discussed in the context of a participatory community health development project. The paper presents a study of the participatory dynamics of the project and explores the implications thereof for the evaluation of participation. The principal source of data is interviews conducted with 13 selected participants in the project. Analysis of interview data is done using a structured hermeneutic method specifically developed for the purpose. The methodology and findings of this study are discussed in relation to the need to find ways of evaluating participatory processes. The authors develop a set of objectives which they suggest should be central to the evaluation of participatory processes in such contexts.