American Evaluation Association Research Articles

What Can Be Accomplished in 25 Years: In Memoriam of Dr. Stafford Hood

Dr. Stafford Hood left this earthly realm on Sunday, January 15th, 2023. His sudden passing created ripples of grief and rumination throughout the evaluation community. Hood was an evaluation and assessment expert, scholar, practitioner, teacher, and innovator. Last year, 2023, marked the 25th anniversary of his talk “Responsive Evaluation Amistad Style,” where he introduced the term “culturally responsive evaluation.” There have been a plethora of tributes to him (see the Hood Obituary, Center for Culturally Responsive Evaluation and Assessment [CREA] Statement, University of Illinois, CREA–Dublin, M. Q. Patton Video, American Evaluation Association Blog). Thus, as a board member of the Canadian Journal of Program Evaluation, CREA affiliate faculty member, and mentee of Dr. Hood, the author humbly offers some brief professional and personal reflections on his legacy 1 year after his passing. How much time is needed to uplift unheard voices, rectify years of erasure, change theory, shift praxis, connect scholars, and mentor the next generation? Dr. Stafford Hood accomplished all of this and transformed the landscape of our field in a mere 25 years. He introduced a revolution, rectified years of erasure, created a movement, and solidified his legacy.

Developing Evaluation Capacity Building Competencies: Participant Reflections From the Evaluation Capacity Case Challenge

In 2023, McGill University’s Max Bell School of Public Policy hosted the inaugural Evaluation Capacity Case Challenge (EC3) competition with a cohort of 19 selected graduate students and early-career professionals studying or working in Canada or the United States. It was a multifaceted learning opportunity for participants to expand evaluation capacity building (ECB) competencies and served as a bridge between formal education and real-world practice. This practice note offers reflections from five students representing all teams and one coach on how EC3 supported competency development as outlined by the Canadian Evaluation Society and the American Evaluation Association. Focused on domains related to professional reflection, technical and methodological skills, situational context, as well as management and interpersonal skills, this article explores the role of EC3 in honing skills specific to ECB, preparing evaluators to excel in their roles and champion ECB in diverse and evolving contexts.

La sazón de liderazgo among Latine evaluators to advance LatCREE

AbstractThe pool of Latine evaluators advancing to leadership is significantly narrow. Investing in the pipeline of Latine talent in the evaluation field who can authentically practice Latine Culturally Responsive and Equitable Evaluation (LatCREE) is essential for working in partnership with Latine communities, advancing theory, and the profession. This article summarizes the findings from testimonios (testimonials) with Latine evaluation líderes (leaders) in the American Evaluation Association (AEA) using a collective critical reflection approach. First, we describe the mentorship and liderazgo (leadership) experiences of Latine evaluators. We also define the central tenets for applying LatCREE as viewed through the lens of Latine evaluators as well as the challenges and lessons learned for conducting LatCREE with diverse Latine communities. Lastly, we discuss the liderazgo (leadership) characteristics that Latine evaluators recognized as being influential to their own leadership journeys as advocates of LatCREE.

Co‐constructing comunidad: A collective critical reflection on the co‐creation of La RED

AbstractThis article describes the co‐creation of the American Evaluation Association's Latinx Responsive Evaluation Discourse (La RED) Topical Interest Group by using a collective critical reflection approach to analyze the evolution of La RED as a platform for theory building and a community of practice during its foundational years (2014–2017). Ten years post instituting the TIG, the authors illustrate how the positionality of inaugural La RED Latine evaluators, their lived experiences, and their engagement in discourse on Critical Race Theory, Latine Critical Race Theory (LatCrit), culturally responsive evaluation, and community participatory approaches informed the conceptualization of Latine culturally responsive and equitable evaluation (LatCREE). LatCREE is introduced as an effective and appropriate framework for the study and practice of evaluation with Latine communities. We emphasize the value of discourse in comunidad as an essential actionable process for practicing LatCREE and offer epistemological, ontological, and axiological considerations for co‐constructing and sustaining comunidad as La RED Latine evaluators.

Meta-evaluation: Validating program evaluation standards through the United Nations Evaluation Quality Assessment (EQAs)

The 2015 United Nation’s ‘Eval Year’ declaration heightened program evaluation’s significance in achieving the Sustainable Development Goals. However, uncertainties persist regarding evaluating high-quality evaluations and addressing social justice concerns in meta-evaluations. The field lacks consensus on both conducting meta-evaluation and the standards to use. To address this, we reviewed meta-evaluation literature, mapped the American Evaluation Association’s foundational documents with the United Nations Evaluation Group’s Norms and Standards to explore their intersectionality on social justice, and analysed 62 United Nations Population Fund evaluation reports alongside their management responses. Our findings indicate that addressing social justice concerns in meta-evaluation is contingent on context rather than established standards. Thus, it’s crucial for evaluators to prioritise social justice in evaluation design and implementation, and to select quality assurance tools that match the evaluation context and professional association guidelines, especially in the absence of standardised guidelines.

Latine evaluators and evaluation: Advancing representation and collective agency

AbstractThis article details how connection to the late Dr. Stafford Hood, Culturally Responsive Evaluation (CRE) and Culturally Responsive Evaluation and Assessment (CREA) provided conceptual, methodological, and affirming foundations of our collective work as Latina evaluators, knowledge co‐creators, and mobilizers. The introduction of CRE & CREA with strategic sponsorship provided by Dr. Hood are key historic moments in the legitimization and advancement of Latine evaluators within and across the American Evaluation Association (AEA) and CREA. The co‐authors, who are multiethnic and multicultural Latinas, consider LatCrit theory, collective agency, and legitimacy in relation to key moments in the evolution of Latine evaluation discourse over the last decade.

Developing and Validating the Evaluator Competencies Assessment Tool (ECAT) Cultural Competencies Subscale

This study aimed to develop and validate the Evaluator Competencies Assessment Tool (ECAT) Cultural Competencies Subscale, which measures cultural competence among evaluators. By addressing the scarcity of validated tools in this area, the study offers professionals a valuable resource to assess their strengths and areas for improvement. The subscale comprises 11 items rated on a seven-point Likert scale, derived from the American Evaluation Association (AEA) Evaluator Competencies. Validation involved a survey of 116 AEA members, employing multiple validity procedures. The subscale demonstrated excellent internal consistency (α=.96) and significant correlation with the Cultural Competence of Program Evaluators scale, confirming its convergent validity. However, the subscale's structural validity yielded mixed results, indicating the need for further exploration. Moreover, the demographic analysis highlighted underrepresentation of Asian and younger members. Overall, the ECAT Cultural Competencies Subscale shows promise for assessing cultural competency, but refinement of its factor structure and additional research are needed.

We Can’t Hear You – You’re on Mute: Findings From a Review of Evaluation Capacity Building (ECB) Practice Online

Background: In her presidential address to the American Evaluation Association (AEA) in 2007, Hallie Preskill (2008) highlighted the potential role of technology to promote learning from evaluation, noting the increased use of computers, the internet, and social media as untapped ways to facilitate evaluation. More than ten years later in the context of the COVID-19 pandemic, evaluators and evaluation capacity building (ECB) practitioners found themselves needing to shift to online modalities to conduct evaluation and build capacity. The COVID-19 pandemic, technological advancements, and the rapid shift to remote work have changed our way of working (Gratton, 2021; Kane et al., 2021). Building evaluation capacity is no exception to this trend. Purpose: This study aimed to examine ways that practitioners have built evaluation capacity online or have used technology to do so, to capture lessons learned that can be applied in a COVID and post-normal context. Setting: Findings from this study can be applied in online contexts for developing evaluation capacity. Intervention: Not applicable. Research Design: The study design consisted of a rapid review of the ECB literature published from 2000 to 2019 in eight academic journals focused on evaluation research and practice. Data Collection and Analysis: Twenty-nine case applications of ECB practice that: 1) mentioned use of technology as a strategy for building evaluation capacity or 2) noted that at least one component of the ECB intervention was carried out online or virtually were reviewed for this study. Quantitative data were analyzed via descriptive statistics. Qualitative data were coded in MAXQDA using conventional content analysis (Hsieh & Shannon, 2005). Findings: More diverse online interventions have increased over time. Less than half (45%) of ECB interventions made use of both asynchronous and synchronous strategies for building capacity while more than one-third (38%) made use of asynchronous only strategies. Key barriers to implementing ECB strategies online included lack of social connections to other participants during the capacity building activity, technical malfunctions, lack of access to or familiarity with the technology in use, and limited resources for carrying out evaluation activities. Key facilitators for enhancing implementation included facilitating participant interaction and relationship-building both on and off-line, tailoring ECB activities to participant work contexts, and providing tutorials for accessing and using the technology in play.

Exploring the value of pro bono evaluation work as a method to build evaluator competencies and strengthen organizational capacity

The Cultivating Evaluation Education and Development (CEED) program engages early career evaluators in an experiential learning experience by supporting them as they lead a pro bono evaluation for a local nonprofit community organization. We explored the value of this professional development to the early career evaluators, to the sponsor organization, and to the nonprofit organizations receiving CEED services by examining findings from six CEED projects. We found that early career evaluators self-reported gains in confidence and competence especially in four American Evaluation Association Evaluator Competencies (2018) domains - professional practice, methodology, planning and management, and interpersonal skills. The initiative allowed the sponsor organization to expand connections with community organizations, implement work consistent with the mission, and provide new mentoring opportunities. Representatives from the recipient nonprofit organizations reported they gained a deeper understanding of evaluation, improved their dissemination efforts, and built organizational evaluation capacity.

Gatekeeping's Influence on Equitable Evaluation Practice

The ethical guidelines for the American Evaluation Association and the principles of community-based participatory evaluation both state the importance of equitable stakeholder involvement. Regardless of the evaluation approach, however, evaluators are often confronted with gatekeepers, or those who control the access to stakeholders, information, or resources. Gatekeepers limit both the participation of key community members and, therefore, the exchange of relevant information related to the evaluation—a process called gatekeeping. Little research attention has been placed on studying gatekeeping, resulting in a dearth of knowledge about the influence of gatekeeping on stakeholder-engaged evaluations and social-structural dynamics that potentially perpetuate gatekeeping practices. In this article, we propose a gatekeeping influence theory grounded in the findings from 14 interviews. With a constructed theory of gatekeeping, we document the emergent social-structural and relational dynamics involved in stakeholder-engaged evaluation, with a focus on evaluations that include community partners and members.

Finding a safe zone in the highlands: Exploring evaluator competencies in the world of AI

AbstractSince the public launch of ChatGPT in November 2022, disciplines across the globe have grappled with questions about how emerging artificial intelligence will impact their fields. In this article I explore a set of foundational concepts in artificial intelligence (AI), then apply them to the field of evaluation broadly, and the American Evaluation Association's evaluator competencies more specifically. Given recent developments in narrow AI, I then explore two potential frameworks for considering which evaluation competencies are most likely to be impacted—and potentially replaced—by emerging AI tools. Building on Moravec's Landscape of Human Competencies and Lee's Risk of Replacement Matrix I create an exploratory Landscape of Evaluator Competencies and an Evaluation‐Specific Risk of Replacement Matrix to help conceptualize which evaluator competencies may be more likely to contribute to long‐term sustainability for the field. Overall, I argue that the interpersonal, and contextually‐responsive aspects of evaluation work—in contrast to the more technical, program management, or methodological aspects of the field—may be the competencies least likely to be impacted or replaced by AI. As such, these may be the competencies we continue to emphasize, both in the day‐to‐day aspects of our operations, and in the training of new and emerging evaluators. This article is intended to be a starting point for discussions that continue throughout the remainder of this issue.

Program Evaluation Standards in Evaluation Scholarship and Practice

Background: The Program Evaluation Standards that were developed and approved by the Joint Committee on Standards for Educational Evaluation have served as a resource to the broader evaluation field for over four decades. However, little evidence has been collected regarding the extent to which the standards have influenced the field through scholarship or professional practice. Purpose: This study seeks to estimate the prevalence of the Program Evaluation Standards in evaluation scholarship and professional practice. Setting: Not applicable. Intervention: Not applicable. Research Design: The study combines a systematic review of evaluation literature with a survey of American Evaluation Association (AEA) and Canadian Evaluation Society (CES) members. Data Collection and Analysis: A systematic review of articles published in 14 evaluation-specific journals from 2010 to 2020 was conducted to identify and typify articles citing the standards. Additionally, AEA and CES members were surveyed, with a focus on knowledge and use of the standards. Descriptive analyses are presented to quantify the prevalence of the standards in evaluation scholarship and practice, respectively. Findings: The systematic review revealed that 4.48% of the 4,460 articles published in 14 evaluation-specific journals from 2010 to 2020 contained some use of the standards. Survey results show that 53.14% of AEA members and 67.12% of CES members are familiar with the standards and that, among those with knowledge of the standards, most AEA (67.67%) and CES (71.74%) members use them at least “occasionally” in their professional work, education, and scholarship activities. Keywords: program evaluation standards; Joint Committee on Standards for Educational Evaluation; American Evaluation Association; Canadian Evaluation Society; systematic review; research on evaluation

Using Dissemination Research Approaches to Understand the Awareness, Adoption, and Use of The Program Evaluation Standards

Background: The adoption and use of effective, legally defensible, and ethically sound practices relies on the successful dissemination of evidence-based practices and professional standards. The field of program evaluation has standards, competencies, and principles, yet little is known about how these are utilized by education-focused program evaluators. Purpose: The purpose of this study is to examine the dissemination and use of the program evaluation standards established by the Joint Committee on Standards for Educational Evaluation, relative to the dissemination and use of the American Evaluation Association’s (AEA’s) guiding principles and AEA’s evaluator competencies. Setting: The SIGnetwork, a network of evaluators of State Personnel Development Grants (SPDGs) funded by the U.S. Department of Education, Office for Special Education Programs (OSEP). Intervention: NA Research Design: Descriptive research. Data Collection and Analysis: Data collection involved administering an online survey to members designated as evaluators in the SIGnetwork directory. Descriptive statistics were used to summarize the data collected via the online survey. Findings: Using the formative audience research approach to understanding dissemination, the results of the study support previous findings that awareness of the standards was inconsistent among a sample of AEA members. Respondents self-reported low to moderate levels of familiarity with The Program Evaluation Standards and the other two guidance documents: Guiding Principles for Evaluators and AEA Evaluator Competencies. Using the audience segmentation research approach to understanding dissemination, the results of this study indicate that participants who were AEA members were more likely than those who were not members of AEA to report being familiar with the standards and to have earned an advanced degree related to their role as an evaluator. Keywords: Joint Committee on Standards for Educational Evaluation, American Evaluation Association, program evaluation standards

¡Milwaukee Evaluation! Inc.: Design principles for recentering social justice in evaluation training and disrupting whiteness and neoliberalism

AbstractAcross the nation, the American Evaluation Association (AEA) recognizes over thirty volunteer‐led organizations called local affiliates. These affiliates provide professional development, networking, and field building opportunities that influence the local evaluation marketplace and ecosystem in ways that have not been systematically studied or understood within the larger discourse of continuing education for evaluators. In this chapter, we present a single case study on the long‐term efforts of ¡Milwaukee Evaluation! Inc., the AEA Local Affiliate in Wisconsin, to recenter social justice in evaluator training and education. The case study is presented in the form of two design principles that have shaped the affiliate's work over the past 10 years and helped move the local evaluation marketplace and infrastructure toward deeper expressions of social justice. The affiliate uses a two‐generation approach, creates liminal spaces as the site for critical consciousness raising and emancipatory capacity building, validates (working class) evaluators of color, and covers topics such as reparations and neoliberalism in its educational offerings. The rationale for this approach is discussed.

Cultural Competence: 10-Year Comparison of Program Evaluators’ Perceptions

As part of its Guiding Principles for Evaluators, the American Evaluation Association (AEA) requires that evaluators develop cultural competencies. Using a successive-independent-samples design, the researchers sought to compare perceptions of cultural competence across a duration of 10 years. Qualitative data were collected via online surveying, which included 168 program evaluators in 2009 and 110 program evaluators in 2019. Content analysis was utilized, and content categories were identified and quantified for both data collections. The data reflect that, from 2009 to 2019, there has been an increased recognition of what cultural competence entails and a closer alignment between what the Guiding Principles for Evaluators promotes and what evaluators demonstrate. However, the data also indicate that perhaps preferences have evolved past the current cultural competence paradigm as well as the term “cultural competence” itself. These findings and implications are discussed in further detail.

Empowerment Evaluation of Programs Involving Youth

Background: Participatory and collaborative evaluation approaches, including Empowerment Evaluation (EE), are useful for evaluating programs involving youth. Empowerment evaluation involves stakeholders in the evaluation process through a set of structured steps. It is primarily concerned with empowering, illuminating, and building program beneficiaries’ self-determination. Given the emphasis that EE places on inclusivity of stakeholders, it appears to be a good fit for evaluating programs that involve youth.
 Purpose: To explore the extent to which evaluators use EE to evaluate programs involving youth as well as what factor(s) facilitate and hinder their use of EE in these programs.
 Setting: The study involved evaluators associated with the Collaborative, Participatory and Empowerment Evaluation and Youth-Focused Evaluation Targeted Interest Groups (TIGs) of the American Evaluation Association (AEA) who are involved in evaluating programs targeted at youth.
 Intervention: Not applicable.
 Research Design: We used a two-phase sequential mixed-methods research design. In Phase 1, we surveyed evaluators. In Phase 2, we interviewed a sample of evaluators from Phase 1. 
 Findings: In Phase 1, 41 (53.9%) respondents indicated not using EE to evaluate programs involving youth, 30 (39.5%) had used EE and 5 (6.6%) were unsure. Of those who used EE, they used it to teach youth program stakeholders about evaluation (n=8, 24.2%), produce more authentic results by engaging youth as experts of their lived experience (n=7, 21.2%) or produce more useful results for stakeholders to use (n=6, 18.2%), as well as other less popular reasons. In Phase 2, 12 interviewees raised five factors that facilitate or hinder the use of EE to evaluate programs involving youth including, evaluator perceptions, type of evaluation experience, evaluator knowledge and professional training, guidelines from organizations and funders, and stakeholders and time. Factors that some interviewees viewed as facilitators others viewed as hinderances.
 
 Keywords: empowerment evaluation, program evaluation, youth-focused evaluation
 

Measuring Evaluator Competencies: Developing and Validating the Evaluator Competencies Assessment Tool

Evaluation scholars have committed decades of work to the development of evaluator competencies. The 2018 American Evaluation Association (AEA) Evaluator Competencies may be useful for evaluators to identify their strengths and weaknesses to improve their practice; however, a few empirically validated self-assessment tools based on the competencies exist. Two studies were conducted to develop a validated tool. The first study ( N = 170) developed the Evaluator Competencies Assessment Tool (ECAT), a self-assessment tool based on the AEA, 2018 Evaluator Competencies. This study provided evidence for structural validity via confirmatory factor analysis. The second study ( N = 142) reconfirmed structural validity with a new sample and examined variables that are associated with evaluator competencies through correlation and t-test analyses. Having a mentor, years of evaluation experience, age, evaluation training, and education level were positively related to evaluator competencies. The ECAT can be used to foster self-reflection for practitioners to improve evaluation competence.

The Prevalence of Evaluator Anxiety in Practice: An Empirical Examination

Anxiety is a multifaceted force that can negatively impact the ability of evaluators to succeed in practice. In the evaluation literature, discussions concerning anxiety have primarily been limited to strategies to reduce stakeholder anxiety to encourage positive and productive working relationships with evaluators. This study was among the first to explicitly recognize and empirically assess the anxiety evaluators feel during practice. Using a mixed-methods design, a random sample of 109 American Evaluation Association members was surveyed. There was overwhelming agreement (96.30%) that evaluator anxiety is a commonly experienced phenomenon, often citing reasons such as lack of experience, imposter syndrome, unreasonable scope, and difficulties with stakeholder interactions as key contributors to increased anxiety. Evaluator anxiety varied across stages of the evaluation process, with some stages eliciting more anxiety than others. Further discussion and training concerning how to reduce evaluator anxiety could be useful in the preparation of evaluators.

Supporting Evaluation Practice Through Mindfulness

This article explores the utility of mindfulness in the field of evaluation. Mindfulness is a translation of the ancient Indian word, Sati, which means awareness, attention, and remembering. While definitions vary, a practical definition of mindfulness is present-moment awareness in an open and nonjudgmental manner. Mindfulness-based interventions have been employed by a wide variety of professions. Although it has received limited attention in the writings of evaluators, we argue that mindfulness can improve the practice of evaluation and support the development of the professional practice and interpersonal domains of American Evaluation Association (AEA) evaluator competencies. We review several mindfulness-based practices and how they can be used by evaluators in their work. Thus, we posit that far from being an esoteric concept, mindfulness practices can serve the pragmatic end of improving our discipline. We also discuss the limits of mindfulness and propose recommendations for future efforts.

Professionalizing Evaluation: A Time-Bound Comparison of the American Evaluation Association's Foundational Documents

This article compares the purposes and content of the four foundational documents of the American Evaluation Association (AEA): the Program Evaluation Standards, the AEA Public Statement on Cultural Competence in Evaluation, the AEA Evaluator Competencies, and the AEA Guiding Principles. This reflection on alignment is an early effort in the third step of professionalization: defining how to use and recognize evaluator competencies. The analysis intentionally focuses on content and reflects on the implications of the differences and similarities across documents. The comparison reveals important questions of interest at both the micro level (individual evaluator) and the macro level (evaluation). The article concludes with challenges, learnings, and proposed next steps of AEA's Professionalization and Competencies Working Group.