Evaluation Technical Assistance Research Articles

Centralizing Evaluation to Amplify the Impact of Focused Grantmaking and Capacity Building Across a Southern US Large-Scale HIV Initiative.

Education, Training and Research Associates (ETR) employed a centralized evaluation (CE) approach to monitor progress toward COMPASS (COMmitment to Partnership in Addressing HIV/AIDS in Southern States) goals and demonstrate the impact of grantmaking, capacity-building, and HIV-related service delivery. Evaluators 1) developed streamlined surveys to capture people served and impact; 2) held routine meetings with partners to contextualize data; 3) implemented partner feedback loops for refinement; and 4) provided partners with evaluation technical assistance. This streamlined the evaluation process for more than 350 organizations serving people living with HIV (PLWH), which permitted timely outcomes and impact data. Centralized evaluation has been pivotal in offering considerations for data use, informing strategic funding, and facilitating alignment across the initiative. By standardizing measures, evaluators developed a cross-site data collection strategy measuring impact on PLWH. This adoption of CE produced vital recommendations for social impact evaluation and replicability for other large-scale projects.

NACCHO announces funding and TA for SSPs

Via federal grants, the harm‐reduction workforce may take off for the first time (see story, p. 1), with the help of funding for staffing, technical assistance (TA), and more. Most recently, the Centers for Disease Control and Prevention and the University of Washington have partnered with the National Association of County and City Health Officials (NACCHO) to fund monitoring and evaluation of syringe services programs (SSPs), and to support the use of data for SSP program improvement. The funding will go to local health departments and community‐based SSPs. SSPs, originally created to help prevent HIV transmission via needle sharing, have become part of the harm‐reduction landscape with the opioid overdose epidemic, as a place for drug users to turn instead of the street for help. Monitoring and evaluations will include routine (e.g., intake or encounter‐level) and periodic (e.g., surveys) data collection, as well as management, analysis, evaluation and reporting. Up to $98,750 can be requested per application. Applications are due May 21. For more information, go to the Request for Applications (RFA) at https://www.naccho.org/uploads/body‐images/NACCHO‐Harm‐Reduction‐Monitoring‐and‐Evaluation‐RFA‐FINAL.pdf. The RFA includes an appendix that characterizes common software used by SSPs. NACCHO and UW will hold “office hours” Tuesdays from 1:00 to 2:00 p.m. ET from April 27 to May 18. Applicants are encouraged to sign up for a specific time slot. Go to the Zoom link (https://washington.zoom.us/meeting/register/tJwqceiupzgiHtUY8mVISv2eopBnk9PHwmLD) and also go to the sign‐up link (https://airtable.com/shrDeGHAbUO5K4mq8). If you have questions, submit them in advance via form or by emailing SSP Measurement and Evaluation Technical Assistance Program Manager Kelly Knudtson (kellyknu@uw.edu). Kat Kelley, senior program analyst at NACCHO for HIV, STI, and viral hepatitis, will need all of the documents for the grant, titled “Building Capacity for Harm Reduction Monitoring and Evaluation.” State health departments are eligible to apply for the funding. SSPs that are currently operating are eligible as well. In states that don't allow SSPs, there may be no opportunity for funding. The grant money cannot be used for lobbying. However, if SSPs are legally allowed to operate in a certain jurisdiction but face restrictions such as exchange only with no secondary syringes allowed, the grant could be used for data collection and analysis to demonstrate the benefits of changing policy and practice. Funds cannot be used for syringes, naloxone or fentanyl test strips. Funding can be used for staff time to conduct monitoring and evaluation. Approaches such as vending machines, marketing promotion of SSPs and other issues may be funded as well, providing the applicant describes how these activities would enhance SSP monitoring and evaluation, or the use of SSP monitoring and evaluation data. Funds could not be used to purchase a vending machine. The number of awards depends on the funding amounts requested by the organizations with the highest scores in their applications. The smallest number of programs that would be funded is 16, but the amount could be higher if applicants do not request the maximum funding amount ($98,750). “Equity” is important as well; grantees who find that they primarily serve cisgender men and women, for example, may use the funds to conduct data‐collection activities with trans, nonbinary or gender‐nonconforming people to see how the SSP could better meet their needs.

Lessons From the Social Innovation Fund: A Tested Evaluation Technical Assistance Approach to Strengthening Evaluation Practice and Building a Body of Evidence.

The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service that received funding from 2010 to 2016, is one of a set of tiered evidence initiatives that was designed and implemented at the federal level during President Obama's administration. The key objectives of the initiative were to (1) invest in promising interventions that address social and community challenges and grow their impact and (2) invest in evaluation and capacity building in order to support the development and use of rigorous evidence to measure the effectiveness of each funded intervention (i.e., to "move the evidence needle") and inform decision making. The SIF proved successful in strengthening and sustaining the capacity of its implementing partners to conduct rigorous evaluations when put through a robust impact evaluation of its own at the national level. It has also spurred high-quality local evaluations that are building knowledge and a body of evidence across the supported program models to inform practice. The SIF's evaluation technical assistance program was critical to its success, and as such, its design and approach holds interesting lessons for the larger field. This article discusses the structure and key features of the SIF as a grant making model, its evaluation requirements, and embedded approach and process for evaluation capacity building and the delivery of technical assistance, the tools and resources that it generated to support its goals, the evidence supporting its success, and how those lessons can inform other organizations and initiatives.

Teen Pregnancy Prevention Evaluation Technical Assistance: A Case Study.

This case study discusses Mathematica's experience providing large-scale evaluation technical assistance (ETA) to 65 grantees across two cohorts of Teen Pregnancy Prevention (TPP) Program grants. The grantees were required to conduct rigorous evaluations with specific evaluation benchmarks. This case study provides an overview of the TPP grant program, the evaluation requirements, the ETA provider, and other key stakeholders and the ETA provided to the grantees. Finally, it discusses the successes, challenges, and lessons learned from the effort. One important lesson learned is that there are two related evaluation features, strong counterfactuals and insufficient target sample sizes, that funders should attend to prior to selecting awardees because they are not easy to change through ETA. In addition, if focused on particular outcomes (for TPP, the goal was to improve sexual behavior outcomes), the funder should prioritize studies with an opportunity to observe differences in these outcomes across conditions; several TPP grantees served young populations, and sexual behavior outcomes were not observed or were rare, limiting the opportunity to observe impacts. Unless funders are attentive to weaning out evaluations with critical limitations during the funding process, requiring grantees to conduct impact evaluations supported by ETA might unintentionally foster internally valid, yet underpowered studies that show nonsignificant program impacts. The TPP funder was able to overcome some of the limitations of the grantee evaluations by funding additional evidence-building activities, including federally led evaluations and a large meta-analysis of the effort, as part of a broader learning agenda.

Application of a Multisite Empowerment Evaluation Approach to Increase Evaluation Capacity Among HIV Services Providers: Results From Project Pride in Chicago

In 2015, the Centers for Disease Control and Prevention (CDC) funded Project PrIDE, a national initiative to implement and evaluate demonstration projects to increase PrEP uptake among HIV-negative individuals and to re-engage HIV-positive individuals in HIV care. Our team served as the Evaluation Center for Project PrIDE organizations in Chicago and used an empowerment evaluation (EE) approach to enhance evaluation capacity at these organizations. To evaluate our approach, we assessed organizations' evaluation capacity and engagement in technical assistance and capacity building activities in 2016 and 2018. Respondents who self-reported higher engagement with the Evaluation Center and who spent a greater number of hours engaged with our evaluators experienced greater increases in evaluation capacity tied to implementation of evaluation activities and technical assistance utilization. These findings demonstrate that multisite EE can be successfully applied to increase the evaluation capacity of organizations providing both HIV prevention and care services.

Challenges and Solutions in Evaluation Technical Assistance During Design and Early Implementation.

This case study discusses two federal grant initiatives and the evaluation technical assistance (ETA) provided to the grantees. One program is a multiphase program funded by the Children's Bureau (CB) within the U.S. Department of Health and Human Services, Administration for Children and Families. The program funded communities interested in preventing homelessness among youth and young adults with child welfare histories. The funder, grantees, and ETA provider call it Youth At-Risk of Homelessness (YARH). Six federal agencies-the U.S. Departments of Education, Health and Human Services, and Labor, the Corporation for National and Community Service, and the Institute for Museum and Library Services-supported the second initiative known as Performance Partnership Pilots for Disconnected Youth (P3). We discuss the programs together as they share common features including a focus on building evidence, a focus on disconnected youth, use of liaisons to work with grantees who are developing interventions and planning or conducting evaluations at the same time, and having the same liaisons provide ETA to both the grantee/service providing group and the local evaluator. The article discusses (1) the grant programs, (2) details about ETA including its funding and provision, (3) details about the success and utilization of ETA, (4) benefits of ETA, and (5) balancing program or initiative requirements and evaluation needs and concludes with (6) thoughts on how to make ETA successful.

Nutrition Embedding Evaluation Programme: An Evaluation Technical Assistance Model for Supporting Civil Society Organizations to Conduct Quality Nutrition Impact Evaluations.

The Nutrition Embedding Evaluation Programme (NEEP) was a global 4-year program (2013-2017) funded by the United Kingdom Department for International Development created to respond to gaps in the nutrition evidence base. The NEEP implementing agency-PATH-provided grants and evaluation technical assistance (ETA) to civil society organizations (CSOs) from 12 countries to conduct robust nutrition-related impact evaluations. The programmatic approach of having an intermediary agent to manage the funding and ETA mechanisms for nutrition impact evaluations is rare and therefore provides a unique opportunity to understand its effectiveness. Over the program duration, NEEP collected lessons learned that were analyzed and disaggregated into key themes considered critical for the completion of high-quality impact evaluations. From these lessons learned, NEEP provides an ETA program model that can be replicated or adapted to other international development sectors. This model highlights the key role of the three tiers (donor, ETA manager, and CSOs) in ensuring the best value for money and effective technical support for conducting impact evaluations and fostering the importance of knowledge uptake and evaluative culture for maximum knowledge diffusion. In this way, global research can be targeted to approaches that provide options to collaborate with the program implementers and contribute to a holistic evidence base to inform policy and programmatic decisions.

Evaluating Cross-Cutting Approaches to Chronic Disease Prevention and Management: Developing a Comprehensive Evaluation.

We provide an overview of the comprehensive evaluation of State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health (State Public Health Actions). State Public Health Actions is a program funded by the Centers for Disease Control and Prevention to support the statewide implementation of cross-cutting approaches to promote health and prevent and control chronic diseases. The evaluation addresses the relevance, quality, and impact of the program by using 4 components: a national evaluation, performance measures, state evaluations, and evaluation technical assistance to states. Challenges of the evaluation included assessing the extent to which the program contributed to changes in the outcomes of interest and the variability in the states’ capacity to conduct evaluations and track performance measures. Given the investment in implementing collaborative approaches at both the state and national level, achieving meaningful findings from the evaluation is critical.

Lessons From the Social Innovation Fund

Tiered evidence initiatives are an important federal strategy to incentivize and accelerate the use of rigorous evidence in planning, implementing, and assessing social service investments. The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service, adopted a public–private partnership approach to tiered evidence. What was learned from implementing this ambitious program? How can large funding initiatives promote evaluation capacity in smaller organizations and evidence building in a sector broadly, increasing knowledge about how to address important social problems? And what can evaluators and evaluation technical assistance providers not working within a tiered evidence framework learn from the SIF? We provide an overview of the SIF model and describe how the fund operationalized “evidence building.” Materials developed to support SIF grantees represent practical, best practice strategies for successfully completing rigorous, relevant evaluations. Key lessons from overseeing over 130 evaluations—and their utility for other local evaluators—are discussed.

A Framework for Evaluation Technical Assistance.

The authors reflect on the U.S. Department of Health and Human Services (HHS) Teen Pregnancy Prevention (TPP) evidence review standards and the framework for evaluation technical assistance (TA) as of 2016. Approval for funded grantees' proposed evaluation designs is examined, along with the monitoring of the application of evaluations plans to real-world problems. Group TA presentations are assessed, along with the evaluation TA contract funded by the HHS Office of Adolescent Health (OAH).

Challenges and Lessons Learned From Providing Large-Scale Evaluation Technical Assistance to Build the Adolescent Pregnancy Evidence Base.

The authors reflect on the challenges and lessons that can be learned from providing large-scale evaluation technical assistance (TA) to construct an adolescent pregnancy prevention evidence base as of 2016, and it mentions the U.S. Department of Health and Human Services' evidence standards. U.S. government contracts are addressed, along with formal and informal training programs and the development of written documents to assist grantees and evaluators. Benchmarks are also assessed.

Establishing an Evaluation Technical Assistance Contract to Support Studies in Meeting the US Department of Health and Human Services Evidence Standards.

An introduction is presented in which the authors discuss the special issue of the journal which examines evidence review standards and the U.S. Department of Health and Human Services' Office of Adolescent Health's investment in rigorous evaluations of teen pregnancy prevention programs.

Foundation engagement in healthy aging initiatives and evidence-based programs for older adults.

Foundation engagement in healthy aging initiatives and evidence-based programs for older adults.

A Method for Building Evaluation Competency Among Community-Based Organizations

Community-based organizations often lack the capacity (e.g., time, staff, skills) to effectively evaluate programs, policies, and environmental changes. Providing evaluation technical assistance and training can be an effective and feasible way to build individual evaluation competency. The purpose of this article is to present a practical approach and related tools that can be used by evaluators and others (e.g., academic partners, funders) providing assistance to build evaluation skills in community organizations. The approach described was developed in collaboration with local universities and a regional health foundation to provide intensive technical support to 19 community-based organizations awarded funding to implement obesity prevention projects. Technical assistance processes and tools were designed to be tailored to organizations' capacity and needs and can be used as templates by others who provide technical assistance. Evaluators, funders, and academic partners can use lessons learned from this experience to help shape and implement evaluation technical assistance approaches with community-based organizations.

Accommodating diverse clients’ needs in evaluation capacity building: A case study of the Tobacco Control Evaluation Center

In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a “blended approach” and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004–2007 and 2007–2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.

Building the Evaluation Capacity of California’s Local Tobacco Control Programs

Successful evaluation capacity building requires a dynamic balance between responding to local agency needs and ensuring that local staff have appropriate skills to conduct rigorous evaluations. In 2004, the California Tobacco Control Program established the Tobacco Control Evaluation Center (TCEC), based at a public research university, to provide evaluation technical assistance to approximately 100 local agencies implementing tobacco control programs. TCEC has been responsive to local needs, for instance, by answering 512 technical assistance requests in the first 5 years of operation and by tailoring training according to needs assessment results. About 50% of the technical assistance requests were for new data collection instruments (n = 255). TCEC has sought proactively to improve local evaluation skills, most recently in a data analysis and report writing skill building campaign that included a webinar, newsletter, and seven regional training meetings. Preliminary analysis suggests a 20% improvement in scores for the local final evaluation reports as a result of this campaign. It is concluded that evaluation technical assistance can be provided effectively by a university as long as the local context is kept in mind, and a balance of responsive and proactive technical assistance is provided.

Building a community of evaluation practice within a multisite program

AbstractNew and novel uses of evaluation processes and findings are possible when a community of practice develops as evaluation stakeholders participate in multisite evaluations in multiple ways. Developing such communities takes advantage of what makes multisite evaluations special. This chapter uses the example of the Innovative Technology Experiences for Students and Teachers Program's Learning Resource Center, funded by the National Science Foundation, to articulate how the provision of evaluation technical assistance to a large, multisite program and its funded projects can contribute to evaluation use. © Wiley Periodicals, Inc., and the American Evaluation Association.

Garnering Grantee Buy-In on a National Cross-Site Evaluation: The Case of ConnectHIV

Grantees on national projects are often required by their funders to take part in evaluation activities. While the requirement can encourage compliance, grantees can feel overburdened and disempowered with their evaluation experience. Evaluators in a national cross-site evaluation utilized multiple strategies for obtaining buy-in of participating grantees: (1) an initial evaluation needs assessment to foster a collaborative partnership and inform a plan for capacity building; (2) an ‘evaluation summit’ to facilitate input on the evaluation framework and cross-site measures, encourage relationship-building and evaluation ownership; (3) an Evaluation Advisory Committee to further on-going input and negotiations on evaluation methods and measures; (4) tailored and timely evaluation technical assistance to build capacity and promote peer exchange; and (5) integration of technology (PDAs) to alleviate data collection burden and improve data quality. Use of these methods increased overall grantee trust in the evaluators and buy-in on the evaluation, their increased capacity in evaluation, their expected future investment in evaluation and the availability of data to benefit their agency and clients. It also resulted in the development of acceptable outcomes, measures and instruments for the evaluation, timely IRB approval and increased quality and timeliness of collected data over time. While labor-intensive and challenging, the evaluators found that frequent and creative engagement of stakeholders in the evaluation yielded important benefits both for the evaluators and for grantees. Although these methodologies have direct relevance for evaluators working on large multisite evaluations, they are also applicable to evaluators who are working on smaller, single-site assessments.

Empowering Programs With Evaluation Technical Assistance

Although empowerment evaluation has gained widespread currency, few reports have described its outcomes. This article combines perspectives of participants and technical assistance providers to describe the process and outcomes of the Evaluation Assistance for Sexual Violence Programs project. Participating programs reported substantial enhancements in evaluation capacity, resources devoted to evaluation, and the extent and sophistication of their evaluation practice, as well as numerous examples of the application of evaluation findings to program improvement. Experiences from evaluation technical assistance providers identified aspects of the process that were particularly useful in achieving these outcomes, including investing in collaborative relationships, maximizing participation among program staff, tailoring the content and form of technical assistance to program preferences, and combining structured learning with program-specific technical assistance.

Participatory research and evaluation: from best practices for all states to achievable practices within each state in the context of the Master Settlement Agreement.

Because the Centers for Disease Control and Prevention (CDC) already assisted state tobacco control initiatives, many state health departments turned to CDC for guidance on how to use Master Settlement Agreement monies. This article describes how CDC funded participatory research to establish local evidence and provided technical assistance for participatory program evaluation. The article (a) presents a telephone interview study of principal investigators (PIs) to determine how CDC could best facilitate participatory research and (b) identifies factors CDC considered to devise evaluation technical assistance that reflected local context. The interview study identified 8 areas where PIs needed CDC's support to undertake participatory research: continuity, timeliness, flexibility, clear and explicit expectations, appropriate and instructive accountability, creation of a vision for participatory research, tailored technical assistance, and enhancement of partner collaboration and support. These findings are being used to inform CDC's research funding. Meanwhile, user-driven technical assistance for participatory evaluation is being accessed by many states.