Participatory Evaluation Approach Research Articles

Evaluation of competence-based teaching in higher education: From theory to practice

Competence-based teaching in higher education institutions and its evaluation have become a prevalent topic especially in the European Union. However, evaluation instruments are often limited, for example to single student competencies or specific elements of the teaching process. The present paper provides a more comprehensive evaluation concept that contributes to sustainable improvement of competence-based teaching in higher education institutions. The evaluation concept considers competence research developments as well as the participatory evaluation approach. The evaluation concept consists of three stages. The first stage evaluates whether the competencies students are supposed to acquire within the curriculum (ideal situation) are well defined. The second stage evaluates the teaching process and the competencies students have actually acquired (real situation). The third stage evaluates concrete aspects of the teaching process. Additionally, an implementation strategy is introduced to support the transfer from the theoretical evaluation concept to practice. The evaluation concept and its implementation strategy are designed for internal evaluations in higher education and primarily address higher education institutions that have already developed and conducted a competence-based curriculum.

Editorial: Participatory research and evaluation approaches in developing contexts: reviewing evidence on professional practice and capacity development

We embarked on the development of this special issue with a clear intention to curate a collection of articles for Educational Assessment Evaluation and Accountability (EAEA) and to capture the current theory, practice and debates associated with participatory evaluation and research in developing contexts. This special issue coincides with a growing international and national interest in participatory methods for research, evaluation and policy implementation. However, like many others, we remain concerned that across many contexts ‘participation’ constitutes little more than a oneoff consultation along with an exclusion from decision-making processes. We firmly believe, rooted in the evidence presented in this issue and beyond, that participatory processes must be continuous from the onset and consider the personal and collective experience of participants. The discussions and debates that have inspired this special issue arose from our longstanding professional collaboration on participatory approaches. Our partnership has centred on applying these methodologies in collaboration with community practitioners, policy decision-makers and academic researchers. With this in mind, we introduce the overarching themes emerging from the three papers and two commentaries. In the first article, ‘The journey from rhetoric to reality: Participatory evaluation in a development context’, Choinard and Cousins provide a systematic review of 40 studies published over the past 16 years. The authors define participatory evaluation as the process of the co-production of evaluative knowledge derived from a partnership Educ Asse Eval Acc (2015) 27:1–3 DOI 10.1007/s11092-015-9217-6

A participatory methodological framework for paving alternative local tourist development paths—the case of Sterea Ellada Region

The focus of the present paper is on the development of a participatory methodological framework, based on the future workshop participatory approach and participatory evaluation tools for planning the integrated development of a specific region, the Region of Sterea Ellada—Greece. Towards this end, particular emphasis is placed upon the sustainable use of natural and cultural resources for the spatial planning of alternative tourist development paths, which are effectively integrated into the local economic structure and its future perspectives. The proposed framework results in the building of scenario-specific policy guidelines which, by taking into consideration developments of the internal and external environment of the study region and the specific decision contexts these outline, support policy makers by providing a range of policy directions and policy measures that can serve effective decision-making within each specific decision context. Moreover, the participatory evaluation approach adopted in the proposed framework supports public and stakeholders’ engagement in the decision-making processes, rendering thus these processes more pluralistic, credible, legitimized and transparent, which in turn are to the benefit of the planning process, the final policy decisions and their successful implementation at the local level.

A Course Model for Building Evaluation Capacity Through a University–Community Partnership

Program evaluation is recognized as an essential skill set for practitioners in service-related fields, such as education, nonprofit management, social work, and public health. Recently, the need for a public workforce trained in evaluation has increased and is driven primarily by our nation’s emphasis on accountability during a time when financial resources are limited. However, many of these professionals lack the necessary skills to conduct evaluations of their programs and often rely on a much smaller number of evaluation consultants to perform this task. How then can we educate students and professionals in ways that build evaluation capacity among service organizations and meet these growing needs? A novel course design that integrates principles of adult learning, participatory evaluation approaches, and experiential forms of learning to build evaluation capacity among students and a nonprofit organization is presented. Evidence is provided to demonstrate student learning and the impact of the course on the nonprofit service organization’s evaluation capacity.

Mission, messages, and measures: Engaging zoo educators in environmental education program evaluation

Here we describe the development and implementation of a large-scale monitoring system to systematically evaluate various Chicago Zoological Society (CZS) education programs. Our primary goal was to engage program staff in developing a consistent measurement and evaluation strategy across and within education programs. We did this by using the CZS mission as a framework and incorporating participatory, theory-based, and utilization-focused evaluation approaches into our education programs. As we carried out the process, we learned several lessons that helped us to succeed. This process has allowed us to begin building the perspective among our staff and leaders that evaluation is an ongoing process that occurs alongside program delivery to inform cycles of reflection and improvement and measure program performance over time.

Using the Plan-Do-Study-Act model: Pacesetters experiences

This article aims to analytically review the Plan-Do-Study-Act (PDSA) model used in the Pacesetters national programme evaluation. The evaluation team's two-fold role is outlined: supporting project teams to develop PDSA plans and collect evaluation data. Four case studies are used to show the PDSA model's application and effect in a participatory action learning approach. Despite limitations, it is clear that all four case studies illustrate the PDSA model's potential benefits in a participatory evaluation approach, which involves public and patients. The model's effectiveness is premised on several enabling factors such as: teams appreciating the model; a climate that values all learning and open to re-planning; engaging any external evaluators whose role is clearly communicated to all project stakeholders in a timely fashion. There are clear intentions to promote evidence-based commissioning in the UK that values patient and public involvement. The PDSA model has the potential to test and implement changes in real work-place settings and to involve the public in evaluation. Case study analysis identifies new reflections on the PDSA model's use to support innovative NHS practice development with public involvement in a participatory approach.

Supporting a Community-Based Participatory Evaluation Approach to Violence Prevention in Kansas City

Disparities in rates of firearm-related homicide exist for some segments of the population and by geographic area. There are interrelated factors across multiple social–ecological levels that increase the risk of violence for some individuals and groups, which may suggest the importance of comprehensive community intervention approaches for addressing violence. Participatory approaches can aid in the implementation of community interventions by engaging community and researcher partners in collaboratively addressing community-identified concerns. The purpose of the present study is to demonstrate the application of a participatory evaluation framework used to support the Aim4Peace Violence Prevention Project in Kansas City, Missouri. The study presents data from the second year of program implementation to examine the contributions of the program in addressing violence.

Who views what? Impact assessment through the eyes of farmers, development organization staff and researchers

The present study assesses the impact of a rural development project on farmers' livelihoods, as perceived by farmers, development organization staff and researchers. The project concerned, aimed to increase small-scale farmers' financial benefits by promoting the commercialization of Ricinodendron heudelotii (Baill.) Pierre ex Pax kernels (njansang) in Cameroon. The three parties evaluated the impacts of the project, over the 2005–2010 period, through indicators embedded in the Sustainable Livelihood Framework (natural, financial, human, social and physical assets). Project households were compared with control households. Results show that farmers' and development organization staff's views were aligned regarding the relative importance of the indicators to measure success (with overlaps > 85%). The three stakeholders evaluated changes of farmers' livelihood indicators over the 2005–2010 period on 5-point Likert items. All three stated that most indicators improved significantly more in project than in control households (p < 0.001). Development organization staff probably overestimated changes induced by project interventions as they perceived significantly larger changes as compared with farmers and researchers (p < 0.05). Our study highlights the differences between impact perceived by farmers, development organization staff and researchers and helps to build the knowledge base of the potential and reliability of participatory evaluation approaches. Furthermore, an approach to assess impacts on people's livelihood is proposed, combining the strengths of participatory evaluation with those of classic evaluation methods.

The Case for Participatory Evaluation in an Era of Accountability

Evaluation occurs within a specific context and is influenced by the economic, political, historical, and social forces that shape that context. The culture of evaluation is thus very much embedded in the culture of accountability that currently prevails in public sector institutions, policies, and program. As such, our understanding of the reception and use of participatory approaches to evaluation must include an understanding of the practices of new public management and of the concomitant call for accountability and performance measurement standards that currently prevail. In this article, the author discusses how accountability has been defined and understood in the context of government policies, programs, and services and provides a brief discussion of participatory and collaborative approaches to evaluation and the interrelationship between participatory evaluation and technical approaches to evaluation. The main part of the article is a critical look at key tensions between participatory and technocratic approaches to evaluation. The article concludes with a focus on the epistemological and cultural implications of the current culture of public accountability.

Capturing stories of change: Evaluating media practices of Papua New Guinean journalists during and after media attachment in Australia

Abstract In recent years, there has been an increased recognition of the value of developing the professional capacity of journalists in developing countries and emerging democracies, and of evaluating these developments more effectively. In addition to the quantitative assessment of the media training programs along pre-defined indicators, this article argues for a participatory evaluation approach to assessing donor-funded media capacity building program to capture stories of change from the perspectives of the participants. Drawing on a case study of an evaluation of a media attachment program, ‘MDI Social Journalism Award’, it highlights the most significant change stories from the perspectives of two journalists from Papua New Guinea. The article outlines the outcomes and impacts of the program for the participating journalists, along with the facilitators and barriers that can influence the success of such media capacity building programs.

Participatory Monitoring and Evaluation: Reviewing an Inclusive Approach in the South Africa's Government Wide Monitoring and Evaluation

The premise of this paper is to canvass for the inclusion of stakeholders in participating in the Government Wide Monitoring and Evaluation (GWM&amp;amp;E) using the participatory monitoring and evaluation (PM&amp;amp;E) approach. The focus is mainly on the practicality of using participatory methods as part of the agenda to cascade the implement the M&amp;amp;E into different stake-holders using various methods in South Africa. A variety of public participation methods should be recognised to benchmark for the inclusion of stakeholders at the local level in the evaluation and monitoring of public programmes and projects in order to allow more space for civil society engagement in development policies and social accountability. Participation is increasingly recognised by countries as an integral part of M&amp;amp;E process while it offers environment for civic participation in a more inclusive and responsive approach towards service delivery and budget allocation in South Africa. Using a qualitative approach, conclusions and recommendations from this study contribute towards monitoring and evaluation of policies, programmes and projects in the field of Public Management. This study intends to address the paucity of literature on Participatory Monitoring and Evaluation in South Africa.

Clinicians’ Expectations and Early Experiences of a New Comprehensive Rehabilitation Case Management Model in a Specialist Brain Injury Rehabilitation Unit

This article presents early findings from the implementation and ongoing evaluation of a new model of care that employs a comprehensive approach to rehabilitation case management for clients with traumatic brain injury, with the aim of improving transition from inpatient rehabilitation to community settings. The evaluation explores the design, implementation, utility and acceptability of the new model using clinicians’ perceptions and experiences. Method: The evaluation framework employs a participatory evaluation approach, drawing on semistructured interview data. Interviews were conducted with brain injury unit clinicians, rehabilitation case managers and external stakeholders at the model's implementation and four months later, as part of a 12-month evaluation period. The data were descriptively organised, then coded and subjected to interpretative analysis to identify key issues. Results: Early findings suggest that the new model provides increased consistency for staff, clients and carers; promotes efficiency in discharge planning and facilitates a more streamlined and seamless transition between inpatient rehabilitation and community services. Data gathered across the remainder of the implementation trial will extend understanding of this comprehensive rehabilitation case management model and its potential utility in other services and settings.

Collaborative Evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike “distanced” evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar “participant-oriented” evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos (2005, 2008) brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.

Program Evaluation in Family-Centred Pediatric Rehabilitation Settings

Many health care professionals in pediatric rehabilitation settings are under pressure to evaluate programs that adhere to the philosophy of family-centred service (FCS). Although there is rhetoric about the importance of upholding this philosophy in program evaluation, the literature reveals that evaluation practices within these settings are often incongruent. This article provides a brief overview of the current evaluation approaches and methods found in the pediatric rehabilitation literature and explores the applicability and potential use of a variety of participatory and collaborative evaluation approaches that, in theory, are congruent with the philosophy of FCS.

Forms and Functions of Participatory Evaluation in International Development: A Review of the Empirical and Theoretical Literature

Background: Since the late 1970s participatory approaches have been widely promoted to evaluate international development programs. However, there is no universal agreement of what is meant by participatory evaluation. For some evaluators, participatory evaluations involve the extensive participation of all stakeholder groups (from donor to non-recipients) in every phase of the evaluation (from design to dissemination). For others, the participation of donors in the design constitutes a participatory evaluation approach. Participatory evaluation approaches are best considered on a continuum. In other words, there are many gradations to participation and evaluations should be classified accordingly. Purpose: The lack of shared meaning of participatory evaluation approaches also impedes serious discussion on their use including their merits and demerits, suggestions for their improvement, and their overall effectiveness. The purpose of this article is to present an examination of the literature on participatory evaluation approaches to highlight commonalities and differences. Setting: Not applicable Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Desk review. Findings: This article clearly demonstrates how broadly participatory evaluation is conceptualized and practiced and underscores the clear need for specification and precision when discussing what is meant by participatory evaluation. Recommendations for how evaluators should describe participatory evaluations are provided. Keywords: participatory evaluation; collaborative evaluation; empowerment evaluation; stakeholder-based evaluation

The Politics and Consequences of Including Stakeholders in International Development Evaluation

Participatory evaluation approaches have a relatively long history of advocacy and application in the international development evaluation community. Despite widespread use and apparent resonance with practitioners and donors alike, very little empirical research exists on why and how participatory evaluation approaches are used in international development settings. In this article, we present results derived from a mixed method investigation of a sample of practicing international development evaluators regarding their perceptions of how and why stakeholders are included in international development evaluations. Findings suggest that participatory evaluation approaches are interpreted and practiced in widely differing ways. Implications for international development evaluation practice and future research are discussed.

Developing the Native People for Cancer Control Telehealth Network.

We aimed to develop a telehealth network to deliver postdiagnosis cancer care clinical services and education to American Indian and Alaska Native patients, their families, and their healthcare providers. We also sought to identify the challenges and opportunities of implementing such a telehealth-based application for this rural and underserved population. We followed a participatory formative evaluation approach to engage all stakeholders in the telehealth network design and implementation. This approach allowed us to identify and address technical and infrastructure barriers, lack of previous experience with telehealth, and political, legal, and historical challenges. Between September 2006 and August 2009, nine tribal clinics in Washington and 26 clinical sites in Alaska had participated in the telehealth network activities. Network programming included cancer education presentations, case conferences, and cancer survivor support groups. Twenty-seven cancer education presentations were held, with a total provider attendance of 369. Forty-four case conferences were held, with a total of 129 cases discussed. In total, 513 patient encounters took place. Keys to success included gaining provider and community acceptance, working closely with respected tribal members, understanding tribal sovereignty and governance, and working in partnership with cultural liaisons. The telehealth network exceeded expectations in terms of the number of participating sites and the number of patients served. Following a participatory formative evaluation approach contributed to the success of this telehealth network and demonstrated the importance of community involvement in all stages of telehealth system design and implementation.

Applications in Bridging the Gap: A Community-Campus Partnership to Address Sexual Health Disparities Among African-American Youth in the South

Risky sexual behavior among African-American youth increases risks for sexually transmitted diseases and unintended pregnancy. This article describes a community-academic partnership to assess The 2 HYPE Abstinence Club, a program combining abstinence education with stress management and creative arts promotion for African-American youth ages 12-18. Bi-directional learning and communication systems were established to facilitate culturally relevant evaluation approaches, quality assurance in data collection, and action-based protocols for on-going improvement. Assessment tools included self-administered surveys and focus groups to gauge intervention effectiveness and perceptions regarding abstinence, sexual peer norms and intervention characteristics. Statistically significant increases in the understanding of abstinence benefits and sexual activity risks were observed and youth identified goal-setting and refusal skills as most important program components. Youth-instructor relationships and the integration of hip-hop were reasons cited for sustained participation. This assessment partnership represents a rapport with minority youth and a participatory evaluation approach adding programmatic and evidence-based value to intervention efforts.

Phoenix Rising: Use of a participatory approach to evaluate a federally funded HIV, Hepatitis and substance abuse prevention program

This paper highlights the value of utilizing a participatory evaluation approach when working with community agencies receiving federal funding for prevention and intervention services. Drawing from our experience as evaluators of a SAMHSA-funded substance abuse, HIV and Hepatitis prevention program targeting homeless young adults, we describe the importance of and strategies for creating a participatory evaluation partnership with program implementers. By participatory evaluation we mean the active involvement of program implementers in defining the evaluation, developing instruments, collecting data, discussing findings, and disseminating results. There are a number of challenges faced when using this approach with federally funded programs that require the use of standardized measurement tools and data collection procedures. Strategies we used to strike a balance between federal requirements and local needs are presented. By increasing the understanding of and participation in the evaluation process, program implementers have greater support for data collection requirements and are appreciably more interested in learning from the evaluation data. This approach has helped to build the capacity of a program and stimulated new possibilities for learning, growing, and ultimately improving the services offered to those the program strives to reach.